Confirmatory factor analysis of the Insomnia Severity Index (ISI) and invariance across race: A pooled analysis of MsFLASH data

Abstract

Objective:

Women's sleep at menopause is widely reported to be problematic. The Insomnia Severity Index is a commonly used tool for quantifying sleep problems in clinical and research settings, but psychometric properties in menopausal women have not been reported. Our study aim was to examine the factor structure of the Insomnia Severity Index in a large and diverse sample of midlife women with hot flashes.

Methods:

Baseline data were from 899 women enrolled in one of the three clinical trials using similar entry criteria conducted by the Menopause Strategies Finding Lasting Answers to Symptoms and Health (MsFLASH) research network. We conducted confirmatory factor analyses for the total sample and within strata defined by race/ethnicity (Black and White women).

Results:

The Insomnia Severity Index had two factors in the total sample. The 2-factor structure was consistent across Black and White women, with the exception of one item “Difficulty falling asleep”.

Conclusions:

The Insomnia Severity Index in midlife women with hot flashes is composed of two factors that capture dimensions of the insomnia severity and daytime impact. The instrument is a psychometrically sound scale appropriate for use in research and clinical practice to capture the severity and daytime impact of insomnia symptoms in diverse samples of midlife women with hot flashes. An abbreviated screening of two items could be considered to determine if further evaluation is needed of sleep complaints.

Introduction

Extensive research shows that midlife women report poor sleep during various stages of the menopause transition and postmenopause.^1–6 For example, at least 48–50% of women with hot flashes experience problems with nighttime sleep during late menopause and early post-menopause.^3,5–7 Self-reported sleep problems are often characterized as symptoms of insomnia such as the inability to fall asleep, remain asleep, and feel rested during the day.^8–11 Insomnia symptoms are highly associated with daytime fatigue and overall poor quality of life. ^10–14 Thus, it is imperative that clinicians and researchers have well-validated measures of insomnia symptoms to clinically diagnose and treat primary and secondary insomnia disorders.¹³

The Insomnia Severity Index (ISI) is widely-used to quantify self-reported insomnia symptoms in clinical practice and research. It is a 7-item scale that requires under ten minutes to complete and score. However, use in clinical practice can be limited due to limited resources, competing demands, and time constraints. Scores indicate no, mild, moderate, or severe insomnia over the past 2 weeks, with higher scores indicating more severe insomnia.¹⁵ The scale has been used in adults^16,17 and teenagers¹⁸ and has been translated to over 50 languages such as Chinese,¹⁹ Spanish²⁰, and Arabic.²¹

The research team that created the ISI did not conduct an exploratory factor analysis in the initial psychometric evaluation studies which limited the depth of understanding regarding of the theoretical structure of insomnia. There are only a small number of studies that have examined ISI psychometrics and/or factor structure with the original English-language version of the scale, and none of these studies were specific to menopausal women.^{15,18,22–24} Factor structures reported contained models of two or three-factor structures. Two factor models included a factor of insomnia symptoms and factor of the impact of insomnia symptoms.^18,22,23 Three factor models include a factor structure of severity of symptoms, impact of poor sleep, and satisfaction with sleep.^15,24 Because the factor structures of validated questionnaires can vary by population (e.g., age, gender, race), the purpose of this study was to examine the factor structure of the ISI using data from midlife women with hot flashes enrolled in the MsFLASH menopausal symptom treatment trials to determine the variable dimensions on insomnia in this population. We sought to evaluate the factor structure in the overall sample as well as whether it varied between women according to race (White vs. Black). Although women were not selected for the first three MsFLASH trials based on a-priori self-reports of sleep problems, 70% of the sample reported sleep problems at baseline, further supporting the relationship between hot flashes and sleep complaints. Based on the factor structure we will examine if an abbreviated scale is possible to use in clinical practice when assessing insomnia symptoms in this population.

Methods

Design

We collated baseline data from 899 women enrolled in the MsFLASH 01, 02, and 03 trials.^25–32 Common inclusion criteria across studies were 14 or more hot flashes per week, peri- or post-menopausal, age 40–62, and in good general health. Women were recruited from July, 2009 through October, 2012. Recruitment methods included mass mailings using purchased mailing lists and health plan enrollment files. Interested women maintained a daily hot flash diary for two weeks; if they continued to be eligible, they took part in two baseline assessments scheduled one week apart. All trials were reviewed by institutional review boards and informed consent was obtained for all participants. The details of these trials have been extensively described in other reports.^25–32

Measures

Women reported basic demographic information at baseline. These data included race, ethnicity, age, education, and smoking history. Study staff measured women’s height and weight in the clinics for calculations of body mass index. Women completed the ISI as part of a larger packet of baseline questionnaires.

Vasomotor frequency, severity, and bother were assessed using daily diaries. Participants completed the paper-based diaries upon awakening and before falling asleep. Severity of daytime and nighttime hot flashes was rated from 0 (mild) to 2 (severe). Bother of daytime and nighttime hot flashes was rated from 0 (not at all) to 3 (a lot).

The 7-item ISI evaluates perceived severity of insomnia over the past 2 weeks. Items are scored using a Likert-type scale with 5 response options (0=not at all to 4=very much). Total scores are obtained by summing items, with total scores ranging from 0 to 28. Higher scores indicate greater perception of insomnia severity. Reliability and validity have been established in healthy individuals (n=145 Cronbach’s alpha=0.74) and those with chronic illnesses (n=1634, Cronbach’s alpha=0.90).^15,23

Statistical analysis

For demographic variables, we used absolute and relative frequencies to describe categorical variable and median and interquartile range for continuous variables. To compare demographic variables between Black and White participants, we used Pearson’s chi-squared test for categorical variables and the Mann-Whitney test for continuous variables..

We performed a series of confirmatory factor analyses based on unweighted least squares (ULS) estimation of the model parameters. We chose ULS estimation because the items of the ISI scale were ordinal variables and thus the usual maximum likelihood estimation was not applicable. Other popular approaches to deal with ordinal items are weighted least squares (WLS) and diagonally weighted least squares (DWLS) estimation methods. However, ULS performance has been shown to be superior in finite samples to that of WLS and DWLS estimation.³³ Good model fit was defined as values of both Tucker-Lewis fit Index (TLI) and Comparative Fit Index (CFI) greater than 0.95, a Root Mean Square Error of Approximation (RMSEA) value smaller than 0.06, and a Standardized Root Mean Square Residual (SRMR) value smaller than 0.08.³⁴

The assessment of measurement invariance between Black and White women was based on a sequence of nested-model comparisons using scale items. Following the recommendation by Cheung and Rensvold (2002),³⁵ we used the CFI measure for comparing nested models between Black and White women. However, unlike Cheung and Rensvold, we performed bootstrap hypothesis tests to statistically evaluate the potential differences between the CFI measures for pairs of nested models that were considered in the invariance analysis.³⁶ This choice was crucial because, to our knowledge, the asymptotic distribution of the CFI estimator has not been studied, and therefore there is no available parametric approach for hypothesis testing. To compare differences in the intercepts of individual items between Black and White women we performed formal Wald tests for the differences between each item’s intercepts, using the appropriate variance components of the estimated variance covariance matrix for the fitted model for the strong/scalar invariance and applied the Bonferroni correction for type I error inflation due to multiple comparisons.

All statistical analyses except for the confirmatory factor analysis were performed using STATA 14 for Windows (STATACorp LLC, College Station, TX). The free statistical software package called R was used to perform the confirmatory factor analysis and the evaluation of construct invariance.³⁷

Results

In total, 899 participants from 3 MsFLASH studies were included in the analysis. Of them, 530 (59.0%) were White, 288 (32.0%) were Black, 78 (8.7%) were American-Indian, Asia/Pacific Islanders or other, while ethnicity information for 3 (0.3%) participants was missing. For this overall sample, most participants were educated beyond high school, were never smokers, were overweight, and had subthreshold or higher insomnia (see Table 1). When comparing only the subgroups of White and Black women, Black women were younger, had higher BMI, higher VMS symptomatology, and higher ISI severity (see Table 1).

Table 1:

Sample Demographics

		Group of interest
	Overall (N=899) N (%)	Black (N=288) N (%)	White (N=530) N (%)	p
Age at screening				<0.001
< 50	73 (8.1)	39 (13.5)	29 (5.5)
50 - 54	404 (44.9)	139 (48.3)	234 (44.2)
55 - 59	319 (35.5)	88 (30.6)	199 (37.5)
>= 60	103 (11.5)	22 (7.6)	68 (12.8)
Education				<0.001
<=High school diploma or general education degree	114 (12.7)	66 (22.9)	37 (7.0)
Post-high school	310 (34.6)	128 (44.4)	150 (28.4)
College graduate	473 (52.7)	94 (32.6)	341 (64.6)
Smoking				<0.001
Never	507 (56.6)	136 (47.2)	325 (61.7)
Past	255 (28.5)	66 (22.9)	161 (30.6)
Current	134 (15.0)	86 (29.9)	41 (7.8)
BMI (kg/m2)				<0.001
< 25	295 (33.1)	45 (15.7)	219 (41.8)
25 - 29.9	323 (36.3)	96 (33.6)	198 (37.8)
>= 30	273 (30.6)	145 (50.7)	107 (20.4)
ISI Sleep Score				0.006
None (0-7)	251 (28.4)	84 (30.2)	140 (26.7)
Subthreshold (8-14)	358 (40.5)	101 (36.3)	240 (45.7)
Moderate (15-21)	235 (26.6)	73 (26.3)	130 (24.8)
Severe (22-28)	40 (4.5)	20 (7.2)	15 (2.9)
	Median (IQR)	Median (IQR)	Median (IQR)	p
Age at screening	54.0 (52.0, 57.0)	53.0 (51.0, 56.0)	55.0 (52.0, 58.0)	<0.001
BMI (kg/m2)	27.1 (23.7, 31.1)	30.1 (26.8, 33.4)	25.9 (22.9, 28.9)	<0.001
VMS Frequency	7.1 (5.2, 10.1)	7.4 (5.4, 10.5)	7.1 (5.0, 9.9)	0.032
VMS Severity	1.0 (0.7, 1.3)	1.2 (0.9, 1.6)	0.9 (0.7, 1.2)	<0.001
VMS Bother	2.0 (1.6, 2.3)	2.2 (1.9, 2.6)	1.9 (1.5, 2.2)	<0.001
ISI Sleep Score	11.0 (7.0, 16.0)	12.0 (6.0, 16.0)	11.0 (7.0, 15.0)	0.468

BMI=body mass index; ISI=Insomnia Severity Index; IQR= interquartile range; VMS=vasomotor symptoms (hot flashes, night sweats); VMS severity rated 0 mild to 2 severe; VMS bother rated 0 not at all to 3 a lot.

Results of the factor analysis in the overall sample are shown in Table 2 and Figure 1. Fit indices shown in Table 2 indicate that a one-factor model did not provide good fit according to the RMSEA. In contrast, a 2-factor model met all the criteria of a good fit. The 2-factor model (severity and impact of symptoms) is depicted along with the estimated factor loadings, correlation between the factors, and the residual standard errors in Figure 1.

Table 2:

Fit Indices of Different Insomnia Severity Index Factor Models

Model	CFI	TLI	RMSEA	RMSEA 90% CI	SRMR
1-Factor	0.990	0.986	0.076	(0.061, 0.091)	0.050
2-Factor	0.997	0.994	0.047	(0.030, 0.064)	0.033

CFI= Comparative Fit Index; TLI= Tucker-Lewis fit Index; RMSEA=Root Mean Square Error of Approximation; SRMR= Standardized Root Mean Square Residual

Figure 1:

The 2-factor model of the Insomnia Severity Index in midlife women

Results of the ISI measurement invariance are shown in Table 3. The CFI for configural invariance was excellent for a 2-group, 2-factor model with all model parameters allowed to freely vary between groups. When the factor loadings were constrained to be equal for Black and White women, the CFI did not vary significantly. These results show the ISI factor structure did not differ by race.

Table 3:

Evaluation of Invariance of Insomnia Severity Index Between Black and White women

	CFI	ΔCFI	p
Invariance
Configural	0.997	NA	NA
Loadings	0.990	0.007	0.080
Intercepts	0.986	0.004	0.010
Means	0.985	0.001	0.396

CFI= Comparative Fit Index; ΔCFI=Change in Comparative Fit Index

When the intercepts were constrained to be equal across the two groups, however, the model fit was significantly inferior, suggesting a difference in at least one item intercept. Additional analyses not shown in the table showed one difference between racial groups. When we compared differences of the intercepts of the individual items between Black and White women, the item “difficulty falling asleep” (p<0.001) showed a partial strong/scalar non-invariance of the ISI construct across race. We further considered the fit and the invariance properties of the 2-factor model for the reduced 6-item ISI scale, with the item “difficulty falling asleep” omitted. The fit of the 2-factor model based on the reduced scale was good (CFI=0.996, TLI=0.993, RMSEA=0.055, SRMR=0.034). More importantly, there was no evidence for non-invariance of that model between Black and White women, based on the reduced 6-item scale meaning the scale performed the same in both groups.

Discussion

The study results show that a 2-factor model (Symptom Severity and Daytime Impact) is a good fit for this sample of midlife women with hot flashes. Although this is the first analysis to solely focus on the ISI factor structure in women, our findings are similar to previous research studies that found similar 2-factor models that included mixed gender samples.^19,22,23 This is the first paper to focus solely on women in this specific age group with insomnia symptoms and hot flashes with more extensive analyses to determine factor structure.

The ISI was developed by the original authors to provide a reliable and valid brief scale to screen for insomnia and provide a tool for evaluation of treatment outcomes.¹⁶ However no factor structure was presented in that original article. Typically the instrument is presented using the established cutoff scores and clinically significant change scores to determine incidence of insomnia and treatment response. Subsequent researchers have added to the dimensionality of the measure by conducting factor analysis suggesting that the measure can perform differently in various populations.²² Specifically, the items from the questionnaire fall into the following 2-factor model. Factor 1 includes items 1–4 of the questionnaire that focus on the symptoms of insomnia: difficulty falling asleep, staying asleep, waking too early, and satisfaction with current sleep. Factor 2 includes natural content on the impact of the insomnia symptoms. This is an important outcome as clinicians use both symptoms and impact of symptoms to diagnose and recommend treatment.

We found that the 2-factor structure and factor inter-correlations were consistent between Black and White women, meaning that the scale performed similarly in both groups. However, comparisons showed one difference between the models. The ‘difficulty falling asleep’ item did not perform the same across two racial groups. This particular item was the weakest loading item on the Symptom Severity factor, suggesting the other three items within this factor were better indicators of that construct. After removing this item, the resulting model showed good fit and performed consistently across racial groups. This suggests consideration should be given to using the 6-item version of the ISI to allow for meaningful comparisons of insomnia symptoms between Black and White women. The goal is to have ISI models that measure insomnia symptoms equally well in both racial groups in order to enable cross-group quantitative comparisons and ensure proper interpretation of results. Clinically, this item omission is unlikely to affect the accurate assessment of insomnia symptoms. The 6-item version of the ISI is flexible and appropriate for use in both groups of women.

Study findings have important implications for scoring the ISI in clinical practice and research. For clinical purposes, it would be acceptable to report a single ISI score. The high correlation between the two underlying factors suggests that items might not need to be scored and reported as two separate scores to be clinically meaningful. Clinically, having an abbreviated scale or single item questions could be beneficial as it reduces the time of administering and scoring the scale. However, for research purposes, it is more descriptive to report scores for each of the two factors. In particular, in intervention studies, it may be beneficial to report scores for each factor to determine the differential impact of interventions on each factor. This is especially important when there could be a disconnect between symptom severity and the impact on daytime outcomes such as individuals that did not sleep but do not always rate daytime outcomes to be detrimentally impacted or vice versa.

Lastly, for clinicians looking for an abbreviated scale or single items to use during a clinical visit to assess sleep, two items emerged that could provide an initial screening for father exploration into sleep complaints. Looking at the items within each factor separately, single items emerged within each factor that could provide some indication of insomnia severity and daytime impact. For symptom severity, item 1 ‘difficulty staying asleep’ had the highest correlation with the factor score. For impact of insomnia, the item 6 that evaluates how worried/distressed the person is about sleep could be the best indicator of how daytime functioning is impacted. In the prior research studies that supported a 3-factor structure, the sleep satisfaction item loaded into a different factor.^15,24 In our analyses, the sleep satisfaction loaded onto the nighttime symptoms severity factor. It is unclear why this particular item loads differently compared to previous literature but could be attributed to the particular sample and how that item is interpreted.

Conclusion

The incidence and prevalence of sleep complaints in women continue to be high, especially in midlife women with vasomotor symptoms, although women continue to be under-represented in sleep research.⁹ Using existing data, we performed the first factor analysis of the ISI in midlife women. Findings suggest the ISI performed equally well in both Black and White women. Clinicians can also consider using an abbreviated screening of the two items suggested above in this population of women (items 1 “Difficulty staying asleep” and item 6 “How Worried/Distressed are you about your current sleep problem”) to determine if further exploration is needed regarding sleep complaints. For research, we recommend scoring based on a two-factor structure to understand both symptom severity and daytime impact, two important outcomes in descriptive and intervention research. In the clinical setting, we recommend scoring based on a one-factor structure (traditional scoring) or considering screening for insomnia using only the two items most highly correlated with each factor score. Clinically having an overall severity rating using the one factor or total score is optimal for ease of use and continues to provide information on insomnia severity for treatment.