Factor analytic and item response theory evaluation of the Penn State Worry Questionnaire in women with cancer

Abstract

Purpose

Cancer survivors frequently experience worry about a variety of topics, including fear of recurrence. However, general measures of worry still require examination of reliability for this vulnerable population. This study utilized modern psychometric methods to examine the reliability of a worry measure in women with breast or gynecologic cancer.

Methods

Women with cancer (n = 332) completed the 16-item Penn State Worry Questionnaire (PSWQ), which has an abbreviated 8-item version (PSWQ-A). Categorical confirmatory factor analysis (CCFA) was used to determine the factor structure and item response theory (IRT) was used to examine score reliability.

Results

CCFA supported a two-factor structure with 11 positively worded items and the 5 negatively worded items loading on different factors. IRT analysis of the 11 positively worded items showed that each was contributing meaningful information to the overall scores. The 11 positively worded items and the PSWQ-A produced the most reliable scores for levels of worry ranging from one θ below to two θ above the mean.

Conclusions

The 11 positively worded items of the PSWQ and the 8-item PSWQ-A were suitable for use in cancer patients while the full PSWQ was unsuitable due to inclusion of the negatively worded items. Future research should consider measuring worry when examining distress in cancer survivors.

Introduction

Worry is the cognitive component of the emotional experience of anxiety and the key symptom of generalized anxiety disorder (GAD; [1]). Worry cognitions are focused on the anticipation of future problems and may include thoughts of solutions for problems [2, 3]. Though worry reduces anxiety in the short term, it prevents concrete emotional processing of distress and often leads to continued distress [2]. The focus on potential problems also results in heightened levels of anxiety [4, 5].

Worry is also linked to a variety of negative effects. It is more strongly related to anxiety and depression than rumination and is also rated as more negative [6]. Worry also mediates the relationship between personality factors, such as neuroticism and depression and anxiety [7]. In healthy individuals, worry is linked to dysregulation of the autonomic nervous system [8-10], immune disruption following exposure to a feared stimulus [11], increased cortisol output [12] and reduced antibody production and elevated interleukin-6 responses following influenza vaccination [11, 13].

While heightened worry is problematic for any population, worry is especially problematic for cancer survivors. Heightened worry is prevalent in cancer survivors, reported by at least 30 % of patients [14, 15]. As many as 80 % of patients at risk (low social support and economic problems) for distress report worry as well [14, 15]. Worry is linked to heightened psychological distress [16], poorer quality of life [17], a more threatening perception of illness [18], poorer coping with physical symptoms [19] and heightened depression and anxiety [20]. Specific sources of worry, such as worry about cancer recurrence or disease progression, occur for a majority of survivors [21-23]. Concerns regarding being a burden to loved ones are also common [24]. However, worry content tends to change over the disease trajectory in cancer survivors [25]. In short, worry content varies across cancer survivors and across time, indicating the utility of a general worry measure for both research and symptom management purposes.

Consistent with development of most psychological measures, measures of worry have been developed and validated in non-clinical samples or ones from which generalization to cancer survivors may be limited [26]. For example, the factor structure of one of the most commonly used measures of worry, the Pennsylvania State Worry Questionnaire (PSWQ), varies between samples even though the PSWQ was developed to measure a general, unidimensional factor of worry [27]. Some studies have identified a two-factor structure based on item wording and keying. Researchers have described the first factor as containing 11 positively worded and scored items (e.g., “I worry all the time”) and the second factor containing five negatively worded and scored items (e.g., “If I do not have enough time to do everything, I do not worry about it”) [28-30]. Previous literature [31, 32] has called the first factor “Worry engagement” and the second factor “Absence of worry”. In response to studies supporting this two-factor structure, an abbreviated form was recommended (PSWQ-A [26]), which uses 8 of the 11 positively worded items. Thus far the PSWQ literature does not have any assessment of reliability using modern psychometrics, such as item response theory (IRT). The advantage of IRT analysis is testing the reliability of each item and the reliability of the total scores at differing levels of the construct (in this case, worry). IRT can also generate scores of the construct that may be more precise than traditional sum scores.

To address these issues, we studied the PSWQ, one of the most commonly used measures of worry, with a cancer sample [33]. This study had two aims: (1) examine the PSWQ factor structure via categorical confirmatory factor analysis (CCFA [34]) as currently the factor structure of the PSWQ when used with cancer patients is unknown; and (2) examine the reliability of the PSWQ scores using item response theory (IRT) to ensure each item contributes meaningful information. As cancer patients or other populations who frequently suffer from fatigue may experience difficulties completing long assessments, questionnaires in which each item contributes meaningfully are preferable. Thus, the present study was conducted to contribute to the broader literature on the psychometric properties of the PSWQ but simultaneously examine the measure in a clinical sample in which worry is common and disruptive.

Methods

Participants and procedures

Data were collected through a retrospective chart review on women (N = 332) with breast (n = 97) or gynecologic (n = 235; 46 % ovarian, 31 % endometrial, 23 % other gynecologic site) cancer (average age 56, SD = 11, 87.1 % Caucasian, 64.6 % married). Similar to other retrospective chart reviews [35, 36], limited demographic data were available. Women had completed the PSWQ as part of a screen for distress at two university-affiliated outpatient oncology clinics. The majority of patients were within 2 years of diagnosis (mean months since diagnosis 11.6, SD = 24.9). The majority (57 %) had completed treatment with the remainder still receiving adjuvant chemotherapy or radiation therapy. Measures were completed as women waited in the clinic for follow-up appointments. The study was conducted following approval by the institutional review board.

Measures

The PSWQ contains 16 items (see Table 1). Each item is rated on a 1–5 scale, with anchors at 1 (Not at all typical), 3 (Somewhat typical) and 5 (Very typical). Eleven items are positively worded (e.g., “My worries overwhelm me”) and five items are negatively worded (e.g., “I never worry about anything”). All negatively worded items are reverse scored and then all item scores are summed to create a total score, with the total score ranging from 16 to 80 with higher scores reflecting greater worry. The literature shows that the PSWQ cut points suggestive of heightened worry or GAD have varied across studies, with a score of 50 used in medically healthy older adults [37] and scores of 62 [32] and 65 [38] used for general population samples. The PSWQ-A includes 8 of the 16 items [26], all positively worded. Scores for the PSWQ-A range from 8 to 40. Data suggest a score of 22 on the PSWQ-A as having acceptable sensitivity and specificity for detecting GAD in healthy older adults [37].

Table 1.

Model fit indices for Model 1 (all items load on one factor), Model 2 (two factors, all items load on one factor, only negatively worded items load on a second method factor) and Model 3 (two factors, positively worded items load on the first factor, negatively worded items only load on the second factor)

Model	Fit indices
	χ2 (df)	χ2, p value	RMSEA (90 % CI)	ECVI	CFI	SRMR
Model 1 (1 factor)	792.74 (104)	<0.01	0.09 (0.08, 0.10)	1.38	0.97	0.09
Model 2 (1 factor plus method factor)	469.61 (99)	<0.01	0.06 (0.05, 0.07)	0.88	0.99	0.04
Model 3 (2 factors)	499.01 (103)	<0.01	0.06 (0.05, 0.07)	0.89	0.99	0.05

Acceptable fit includes the following: RMSEA <0.08, CFI >0.96. Lower values on the ECVI and SRMR indicate better model fit

RMSEA root mean square error of approximation, ECVI expected cross-validation index, CFI comparative fit index, SRMR root mean square residual

Analytic strategy

Categorical confirmatory factor analysis (CCFA) was conducted using LISREL 8.8 [39] and the diagonally weighted least squares estimator. Given the categorical nature of the data, we used a matrix of polychoric correlations as the data for the analyses. Three CCFA models were tested based on prior research [30, 32]. First, a one-factor solution was tested in which all 16 items loaded on one common factor (Model 1). Second, a one-factor solution including a method factor was tested. In this model, all items loaded on a primary factor and the negatively worded items also loaded on a second orthogonal method factor (Model 2). Third, a two-factor solution with correlated factors was tested in which all 11 positively worded items loaded on the first factor and all five negatively worded items loaded on the second factor (Model 3). In Model 2, the additional covariation among the negatively worded items was viewed as an artifact of wording and keying. In Model 3, the negatively worded items were viewed as a different aspect of worry. To evaluate the CCFA models, factor loadings and measures of model fit were examined [40]: root mean square error of approximation (RMSEA; [41]), comparative fit index (CFI [42]), the expected cross-validation index (ECVI), a measure of expected model fit if the model was tested in another sample and the standardized root mean square residual (SRMR). Acceptable fit includes the following: RMSEA <0.08, CFI >0.96. Lower values on the ECVI and SRMR indicate better model fit.

For the second analysis, IRT was used to analyze participant responses to the PSWQ items. Samejima’s [43] graded response model (GRM) was chosen since the response scale of the PSWQ was intended to be a graded scale. Unidimensionality and local independence are assumed by the GRM [44]. Local independence implies that, conditional on the latent variable (i.e., worry), the item responses are independent from one another. The GRM estimates two types of parameters for each item: severity and slope. The severity parameters are related to the probability that an individual at a particular level of the construct will choose a particular response alternative. The GRM estimates C-1 severity parameters for each item, where C is the number of response categories. As the PSWQ has five response categories, four severity parameters were estimated for each item. Ideally, a measure contains items for one of two purposes: symptom monitoring, which requires items with a range of severity parameters; or distress screening, which requires items with severity parameters (measuring symptom severity) near potential screening cut points for significant distress. The slope parameter indicates how precisely or reliably an item measures the construct. A higher slope indicates a stronger relationship between the item and the construct. Higher slope parameters indicate better discrimination or more reliable measurement [45]. The slope and severity parameters can also be used to generate IRT-based scale scores, which are standardized (mean of zero and standard deviation of 1). Based on the severity and slope parameters, item characteristic curves (ICC) can be constructed. Each ICC indicates the probability of a response category being endorsed at a given level of worry. Ideally, each category is most likely to be endorsed at some level of worry. For items with higher slope parameters, the curves become more steep and distinct reflecting the increased discrimination.

Using the item parameters, it is possible to determine the amount of information a scale provides at different levels of the construct (in this case, worry). This test information function (TIF) can be easily converted to a standard error curve (SEC; standard error is the inverse of information). The standard error is also related to the concept of reliability. Standard errors vary as a function of score magnitude in IRT and this indicates that reliability varies depending on the magnitude of the score. This is different from the standard classical test theory measures of reliability (e.g., alpha), in which reliability is assumed to be constant across all levels of the construct.

Results

Preliminary analyses

The negatively worded items (1, 3, 8, 10 and 11) were reverse scored. All response categories were maintained as sufficient numbers of participants endorsed each response option. The mean PSWQ score was 43.3 (SD = 13.1, range 16–80), lower than that for a sample of undergraduate students, reported by Meyer et al. [33] as 48.8 (SD = 13.8). The mean PSWQ-A score was 20.0 (SD = 8.3), higher than the mean for a sample of healthy, older adults (mean age 71.6 years; M = 14.9, SD = 6.8) and comparable to that from healthy, undergraduate students (mean age 21.3 years; M = 21.8, SD = 8.2) [46]. Although the sample was not particularly distressed on average, the variability of the scores has a greater impact on IRT analyses and, as scores covered the full range of the PSWQ and were normally distributed, we proceeded with the planned analyses.

Categorical confirmatory factor analyses

Three models were fit to the data (see Table 1 for fit indices and Table 2 for factor loadings). For Model 1 (one factor, 16 items), factor loadings for the five negatively worded items were low (0.26–0.48) and fit indices were not within adequate ranges (RMSEA = 0.09). For Model 2, fit indices were within acceptable ranges (RMSEA = 0.06, CFI = 0.99). Positively worded items had high factor loadings on the first factor (0.73–0.92). Examination of the factor loadings from Model 2 indicated that the negatively worded items did not load sufficiently on the first factor (0.20–0.44). Factor loadings for the five negatively worded items were moderate for the method factor in Model 2 (0.42–0.63). For Model 3, fit indices were within acceptable ranges (RMSEA = 0.06, CFI = 0.99). Factor loadings for the 11 positively worded items were high for the first factor in Model 3 (0.73–0.92). The factor loadings for the five negatively worded items on the second factor were moderate to high (0.48–0.84). The two factors in Model 3 were correlated at 0.48 (SE = 0.06). In summary, fit indices indicated that both Models 2 and 3 adequately accounted for the observed data. However, the relatively low factor loadings of the negatively worded items on the primary factor in Model 2 (all in 0.2 ranges except one) indicated that the negatively worded items measured a different aspect of worry than the 11 positively worded items. Therefore, a two-factor correlated solution (Model 3) was chosen.

Table 2.

Item content and factor loadings of the PSWQ across three confirmatory factor analytic models

Item	Model 1	Model 2		Model 3
	Factor 1	Factor 1	Factor 2	Factor 1	Factor 2
15. I worry all the time	0.91	0.92		0.92
7. I am always worrying about somethinga	0.90	0.90		0.90
5. I know I should not worry about things, but I just cannot help ita	0.87	0.87		0.87
13. I have been worrying about thingsa	0.87	0.88		0.88
14. Once I start worrying, I cannot stop	0.87	0.87		0.87
6. When I am under pressure, I worry a lota	0.84	0.85		0.85
4. Many situations make me worrya	0.83	0.83		0.83
9. As soon as I finish one task, I start to worry about everything else I must doa	0.78	0.78		0.78
2. My worries overwhelm mea	0.76	0.77		0.77
12. I have been a worrier all my lifea	0.76	0.76		0.76
16. I worry about projects until they’re done	0.72	0.73		0.73
10. I never worry about anythingb	0.48	0.44	0.54		0.84
8. I find it easy to dismiss worrisome thoughtsb	0.32	0.28	0.42		0.54
11. When there is nothing more I can do about a concern, I don’t worry anymoreb	0.32	0.26	0.59		0.58
3. I do not tend to worry about thingsb	0.30	0.23	0.63		0.56
1. If I do not have time to do everything, I do not worry about itb	0.26	0.20	0.53		0.48

Model 1 allows all items to load on the one factor. Model 2 has all items load on one factor and the five negatively worded items load on a method factor. Model 3 has the positively worded and negatively worded items load on two different factors

^aPSWQ-abbreviated items

^bNegatively worded items

Item response theory analyses

The CCFA results suggested that the negatively worded items measured a different aspect of worry and hence, indicate a departure from unidimensionality. Other data suggest that, when used alone, the negatively worded items do not predict GAD diagnosis [47]. As this study was attempting to find a measure for heightened worry and GAD in cancer patients, the five items were excluded from the IRT analyses.

The graded response model (GRM) was fit to the data from the 11 positively worded items and parameter estimates were obtained using maximum marginal likelihood estimation (with an EM algorithm) in the Multilog program [48]. Severity and slope parameter estimates are reported in Table 3. Slope parameters ranged from 1.95 to 4.76, indicating reasonable associations between the items and the construct of worry. Severity parameters ranged from −1.06 to 1.99, indicating most items measured moderate to severe levels of worry. Figure 1a, b provides item characteristic curve (ICC) examples, displaying items 2 and 15. Figure 1a shows a typical ICC for a reliable item. Each curve represents a different response category and each curve (line) is the highest at some level of worry. Figure 1b (item 15) shows an ICC for an item with a high slope parameter (4.76). The steeper peaks of the curves in Fig. 1b compared with the shallower peaks of Fig. 1a reflects the greater slope for item 15 (“I worry all the time”) compared with item 2 (“My worries overwhelm me”).

Table 3.

Parameter estimates for the Penn State Worry Questionnaire (PSWQ) using the graded response model

Item	Slope (SE)	1st severity parameter (SE)	2nd severity parameter (SE)	3rd severity parameter (SE)	4th severity parameter (SE)
2. My worries overwhelm mea	2.17 (0.23)	−0.37 (0.10)	0.31 (0.09)	1.19 (0.13)	1.99 (0.22)
4. Many situations make me worrya	2.70 (0.24)	−0.73 (0.10)	0.08 (0.08)	0.90 (0.10)	1.59 (0.15)
5. I know I should not worry about things, but I just cannot help ita	3.14 (0.33)	−0.75 (0.10)	−0.06 (0.07)	0.67 (0.09)	1.24 (0.11)
6. When I am under pressure, I worry a lota	2.80 (0.27)	−1.06 (0.11)	−0.30 (0.08)	0.62 (0.09)	1.25 (0.12)
7. I am always worrying about somethinga	4.06 (0.41)	−0.24 (0.06)	0.28 (0.07)	0.98 (0.08)	1.54 (0.12)
9. As soon as I finish one task, I start to worry about everything else I must doa	2.27 (0.23)	−0.33 (0.10)	0.40 (0.09)	1.18 (0.12)	1.74 (0.18)
12. I have been a worrier all my lifea	2.31 (0.22)	−0.57 (0.11)	0.03 (0.09)	0.94 (0.11)	1.38 (0.14)
13. I have been worrying about thingsa	3.32 (0.33)	−0.86 (0.10)	−0.01 (0.07)	0.84 (0.09)	1.37 (0.11)
14. Once I start worrying, I cannot stop	3.42 (0.35)	−0.36 (0.07)	0.37 (0.08)	1.21 (0.10)	1.64 (0.15)
15. I worry all the time	4.76 (0.48)	−0.05 (0.06)	0.58 (0.06)	1.12 (0.09)	1.64 (0.12)
16. I worry about projects until they’re done	1.95 (0.20)	−1.01 (0.15)	−0.04 (−0.10)	1.17 (0.15)	1.78 (0.20)

SE standard error

^aPSWQ-abbreviated items

Fig. 1.

Item characteristic curves. Each line indicates the probability of a respondent with a particular level of worry endorsing a response category for the item. The x-axis corresponds to standardized (mean of 0, standard deviation of 1) worry scores and the y-axis corresponds to probability of endorsing the response category. Ideally, each category is the most likely category to be endorsed at a specific level of worry: a Item 2 follows the ideal pattern. b Item 15 has a particularly high slope and the curves are exceptionally peaked compared with other items (item 2)

Figure 2 shows the standard error curves for the 11 positively worded items and the 8 PSWQ-A. Error appeared to be lowest (and conversely, reliability highest) from one θ below to two θ above the mean. While the PSWQ-A had the expected lower reliability due to fewer items, the error was still lowest from one θ below to two θ above the mean. Error can be transformed into reliability, thereby estimating reliability for a particular score value. For example, at a scale score value of 0 (the mean), the error for the 11 positively worded items was 0.18 and the corresponding reliability was 0.97. For the PSWQ-A, the error was 0.23 and the corresponding reliability was 0.95. This indicates suitability of both the 11- and 8-item versions of the PSWQ for worry screening. Reliability at the mean is minimally (0.02) decreased for the PSWQ-A scores compared with the 11 positively worded item scores.

Fig. 2.

Standard error curve. The lines indicate where the error is lowest and hence, reliability highest for various scores. The x-axis corresponds to standardized (mean of 0, standard deviation of 1) scores of worry and the y-axis refers to amount of error. The solid line plots the error of the 11 positively worded items (PSWQ-11) and the dashed line plots the error of the PSWQ-A (PSWQ-8)

Discussion

This study used confirmatory factor analyses and item response theory to examine the psychometric properties of a commonly used measure of worry, the PSWQ, in a sample of women with cancer. Factor analysis supported the body of research finding a two-factor solution rather than a one-factor solution [28, 30]. The first factor consisted of positively worded and keyed items and the second consisted of negatively worded and keyed items. IRT analyses of the first PSWQ factor (positively worded items) and the PSWQ-A indicated suitable reliability between average and high levels of worry (one standard deviation below to two standard deviations above the mean). Overall, this study provides support for the use of the 11 positively worded PSWQ items and the PSWQ-A as measures for heightened worry in cancer survivors.

The two-factor solution for the PSWQ has been debated in prior literature. While studies of undergraduates and the general population [31, 47] have advocated a one-factor solution, numerous other studies have suggested a two-factor solution. Our data specifically support a two-factor solution for cancer survivors, with one factor of positively worded items and the other factor of negatively worded items. These disparate results regarding factor structure may stem from differences in responding to negatively worded items when compared with positively worded items. Differential response styles have been shown for negatively worded items versus positively worded items across a range of anxiety measures, including social anxiety [49] and post-traumatic stress symptoms [50]. Negatively worded items have also been shown to have lower reliability and more frequent inconsistent responding for a measure of depression [51]. For some respondents, including cancer survivors, the content of negatively worded items may be difficult to interpret. For example, one PSWQ item reads, “If I do not have enough time to do everything, I do not worry about it.” To indicate the greatest level of worry, a respondent must endorse the “not at all” category. Thus, the meaning of such an item and selecting the correct response option likely requires greater cognitive effort than is the case for positively worded items. Such interpretive problems may result in lower reliability for negatively worded items, especially for cancer survivors who may be fatigued or pre-occupied by other matters when completing a measure.

IRT analyses indicated sufficiently strong relationships between the positively worded items and the construct of worry. Individually, each positively worded item showed sufficient discrimination of worry and contributed meaningful information. The scores from the positively worded items of the PSWQ and the PSWQ-A had higher reliabilities for average to high levels of worry. The PSWQ-A may be optimal for worry measurement in cancer survivors due to its brevity. Moreover, computer administration of screening measures is feasible [52] and would allow use of the IRT model to generate IRT scores, offering greater precision than traditional sum scores and cut points and is therefore advantageous with time pressed or fatigued cancer survivors. Thus, the positively worded items of the PSWQ or the PSWQ-A should be utilized for assessing heightened worry in medical populations, instead of the full PSWQ due to inclusion of the negatively worded items in the full PSWQ score.

As noted above, worry is prevalent in cancer survivors [14, 15]. Survivors with early stage disease may experience primarily fears of recurrence [22, 53, 54] that does not necessarily decrease over time [21], while other survivors experience different worries, such as worry about ability to care for themselves [23]. Worrisome thoughts predict psychological distress and quality of life decrements in cancer survivors [55]. Worry may contribute to continued distress in cancer survivors through disruption of processing emotional stimuli [2], and a short and reliable measure of general worry, as outlined in this study, would be useful for both research and symptom monitoring, both for comparing across heterogeneous survivors and for monitoring survivors across different points in the disease trajectory. Through the combination of CCFA and IRT, this study provided clarity on the PSWQ factor structure for cancer survivors and was the first IRT analysis of the PSWQ. This study provides a reliable and shorter measure for a cognitive component of distress, either using the 11 positively worded items of the PSWQ or the 8-item PSWQ-A.

These data support several directions for future research. First, validation studies would be important to assess correspondence of the PSWQ-A scores with criterion measures, including diagnoses obtained with a clinical interview such as the Structured Clinical Interview of the DSM-IV [56]. Second, research is needed to replicate these results in different samples as this study only assessed women, although the prevalence of GAD in women is higher than in men [57]. Third, future research could examine measurement invariance of the PSWQ between different cancer samples and between cancer samples and samples of healthy participants. Overall, the brief versions of the PSWQ may provide an economical and efficient strategy for assessing worry in cancer survivors.