Factor analyses of a social support scale using two methods

Abstract

Purpose

Evaluation and comparison of the factor structure of the Medical Outcomes Study Social Support Survey (MOS-SSS) using both confirmatory factor analysis (CFA) and exploratory factor analysis (EFA) with two samples of people living with HIV/AIDS in China.

Methods

Secondary analyses were conducted with data from two comparable samples of 320 people living with HIV/AIDS from the same hospital using the same inclusion criteria. The first sample of 120 was collected in 2006, and the second sample of 200 was collected in 2012. For each sample, CFA was first performed on the original four-factor structure to check model fit, followed by EFA to explore other factor structures and a subsequent CFA for model fit statistics to be compared to the original four-factor CFA.

Results

In both samples, CFA on the originally hypothesized four-factor structure yielded an acceptable model fit. The EFA yielded a two-factor solution in both samples, with different items included in each factor for the two samples. Comparison of CFA on the a priori four-factor structure and the new two-factor structure in both samples indicated that both factor structures were of acceptable model fit, with the four-factor model performing slightly better than the two-factor model.

Conclusion

Factor structure of the MOS-SSS is method-dependent, with CFA supporting a four-factor structure, while EFA yielded a two-factor structure in two separate samples. We need to be careful in selecting the analytic method when applying the MOS-SSS to various samples and choose the factor structure that best fits the theoretical model.

Introduction

The key role of social support in buffering stress [1], improving psychosocial health [2–6], enhancing self-care [7, 8], improving quality of life [9–11], and reducing mortality [3, 12] has been extensively reported. In order to better understand and evaluate the health-promoting effects of social support, a psychometrically sound measurement tool of social support is needed.

The Medical Outcomes Study Social Support Survey (MOS-SSS) is a brief, multidimensional, self-administered questionnaire developed by Sherbourne and Stewart [13] in 1991 to evaluate social support in patients with chronic illness. It was originally designed to measure five dimensions including emotional support, information support, tangible support, affectionate support, and positive social interaction. Subsequent confirmatory factor analysis (CFA) produced a four-factor structure, aggregating emotional, and informational dimensions of social support [13], which has been widely adopted in various studies [14, 15]. Being simple, short, and easy to understand, the MOS-SSS has also been translated into various languages including Portuguese [16, 17], Spanish [18, 19], French [20, 21], Malay [22], and Chinese [23–28] and has been widely used among different sub-populations in different countries.

While versions of the MOS-SSS in various languages have shown good general reliability and concurrent and criterion construct validity, examination of the factorial validity has produced conflicting results. For instance, when the MOS-SSS was first translated into Chinese in 2004, Yu et al. [23] conducted a CFA on data from a sample of 110 patients with heart failure to test its factor structure. They found that although the four-factor structure was confirmed, the extremely high Cronbach’s alpha and inter-factor correlations implied that there might be some redundancy of items. Gjesfjeld et al. [29] further compared the original 18-item version of MOS-SSS with the abbreviated versions of MOS-SSS (with 12 items and 4 items) using CFA and found that the abbreviated versions had much better model fit. The factorial validity of the MOS-SSS was later examined with a sample of English-and French-speaking Canadians aged 55 years or older [21]. The authors conducted CFAs on both English and French versions of the MOS-SSS. Although they found acceptable fit indices for the four-factor structure, some items appeared to be cross-loading on more than one factor in both versions. However, two recent CFAs on the MOS-SSS using two samples with different chronic diseases in different parts of China have found excellent model fit with the four-factor structure and reported no item redundancy problems [26, 28].

While CFA generally supported the four-factor structure of various language versions of MOS-SSS with diverse samples, exploratory factor analysis (EFA) have not produced the same four factors as CFA. One study using EFA on the MOS-SSS with a sample of 263 Black diabetes mellitus outpatients in South Africa yielded a two-factor structure: tangible support and socio-emotional support [30]. Interestingly, the same finding was reported in another study using EFA with a Taiwanese sample of cancer patients’ caregivers, which found that the two-factor model was better than the original five-factor model [25]. Using a sample of civil servants and Hodgkin’s lymphoma survivors in Brazil, Griep et al. [16] and Soares et al. [17] ran EFAs on the Portuguese version of MOS-SSS and both found a three-factor structure. A similar three-factor solution was also produced by Costa-Requena et al. [18] using an EFA on the Spanish version of MOS-SSS with a sample of 400 oncology outpatients [18]. Factor analyses of MOS-SSS using CFA and EFA on various samples are summarized in Tables 1 and 2, respectively.

Table 1.

Factor structure of MOS-SSS using CFA

Study	Location	Language	Sample characteristics	Model fit index	Problem
Yu et al. [23]	China (Hong Kong)	Mandarin	110 patients with heart failure	χ2/df 2.69 NFI 0.87 NNFI 0.90 CFI 0.91	Strong inter-factor correlation (0.88–0.99) Goodness-of-fit criteria marginally met
Gjesfjeld et al. [28]	USA	English	330 mothers of children with mental illness	χ2/df 2.07 CFI 0.96 GFI 0.88 AGFI 0.85 RMSEA 0.06 SRMR 0.14	12/4-item has better fitting model
Robitaille et al. [21]	USA	English	2,642 residents aged 55 above	CFI 0.96 TLI 0.99 RMSEA 0.076 CFI 0.96 TLI 0.99 RMSEA 0.047	Item cross-loading
	Canada	French	489 residents aged 55 above		Item cross-loading
Li [26]	China (Mainland)	Mandarin	297 people living with HIV/AIDS	χ2/df 2.34 CFI 0.935 PNFI 0.761 PCFI 0.798 RMSEA 0.067	None
Wang et al. [28]	China (Mainland)	Mandarin	200 CHD patients	χ2/df 2.79 IFI 0.92 NNFI 0.90 CFI 0.91 RMSEA0.064	None

Table 2.

Factor structure of MOS-SSS using EFA

Study	Location	Language	Sample characteristics	Factor number	Factor contenta
Westaway et al. [30]	South Africa	Various local languages	263 black patients with diabetes	2	EIAP/Tb
Shyu et al. [25]	China (Taiwan)	Mandarin	265 caregivers of cancer patients	2	EIAP/T
Griep et al. [16]	Brazil	Portuguese	4,030 civil servants	3	EI/AP/T
Soares et al. [17]	Brazil	Portuguese	200 Hodgkin’s lymphoma survivors	3	EI/AP/T
Costa-Requena et al. [18]	Spain	Spanish	400 oncology outpatients	3	EIP/A/T

^aE = emotional support, I = informational support, A = affectionate support, P = positive social interaction, T = tangible support

^bHere, T is a combination of all items from the original tangible support and three items from the emotional/informational support

In summary, our literature review of factor analyses on the MOS-SSS revealed that differences in factor structures seem to be related to the type of analyses employed instead of samples used. CFA leads to good fitting four-factor models, while EFA leads to three or two-factor models. As the names suggest, CFA is generally used to confirm a predetermined factor structure with a priori theory, while EFA is mainly applied to explore a factor structure when there is no assumption about it. No known study has compared model fit of both CFA and EFA on the same sample. It is important to understand how factor structure is related to analytic method employed.

In the present study, we conducted both CFA and EFA of the MOS-SSS on two populations of Chinese people living with HIV/AIDS. Specifically, we first performed CFA with the original four-factor instrument then ran EFA to see whether we could determine an even better fitting model that could be confirmed with CFA and compared to the original four-factor CFA.

Methods

Participants

This analysis included a total of 320 Chinese people living with HIV/AIDS with complete MOS-SSS data from two individual study populations. These data were all collected at Beijing Ditan Hospital, the premier treatment center for infectious diseases in China. The first sample of 120 people living with HIV/AIDS was collected in 2006 as part of a randomized controlled trial evaluating a nurse-delivered intervention for antiretroviral medication adherence [31]. The second sample of 200 people living with HIV/AIDS was collected in 2012 as part of a study to characterize mental health symptomatology of Chinese HIV-positive individuals. For both studies, convenience samples were recruited via a poster describing the study posted in the waiting room and physician referral. Interested clinic patients were referred to study staff that described the study and obtained consent with participants who met the eligibility criteria of (a) ability to read and write in Chinese, (b) above the age of 18, and (c) not cognitively impaired or actively psychotic. Both samples completed a 1–1.5 hour long paper-and-pencil baseline survey; the first survey was interviewer–administered, the second survey was self-administered. All study protocols and patient consent forms were approved by the institutional review boards of University of Washington and Ditan Hospital.

Measures

Socio-demographic characteristics and social support scores listed in Table 3 and described below were assessed at baseline by face-to-face interviews and self-administered questionnaires.

Table 3.

Socio-demographics by study population^a

Characteristics	First sample (n = 120)	Second sample (n = 200)
	N (%)	N (%)
Gender
Male	98 (81.7)	162 (81.0)
Female	22 (18.3)	38 (19.0)
Age (years)
18–29	29 (24.3)	45 (22.5)
30–49	84 (70.0)	133 (66.5)
50–100	7 (5.8)	20 (10.0)
Ethnicity
Han	112 (93.3)	179 (89.5)
Non-han	8 (6.7)	21 (10.5)
Educationb
Middle school or less	46 (38.3)	35 (17.5)
High school or less	45 (37.5)	64 (32.0)
College and above	29 (24.2)	98 (49.0)
Employment
Unemployed	53 (44.2)	78 (39.0)
Part-time	13 (10.8)	22 (11.0)
Full-time	49 (40.8)	96 (48.0)
Incomeb (RMB/month)
1,999 or less	69 (57.5)	70 (35.0)
2,000–3,999	28 (23.3)	67 (33.5)
4,000 or greater	22 (18.3)	49 (24.5)
Marital statusb
Married	66 (55.0)	69 (34.5)
Divorced/separated/widowed	19 (15.8)	47 (23.5)
Never married	35 (29.2)	84 (42.0)
Sexual identityb
Gay/homosexual/bisexual	50 (41.7)	95 (47.5)
Heterosexual	57 (47.5)	67 (33.5)
Unknown	13 (10.8)	38 (19.0)

RMB Ren Ming Bi

^aSome percent doesn’t add up to 1 due to missing values

^bSignificant difference between the first sample and the second sample at α = 0.05

Socio-demographic characteristics

Demographic and socioeconomic variables of interest included age, gender, ethnicity, education, employment, income, marital status, and sexual identity. Age in years was categorized as <30, 30–49, >49; ethnicity as Han or non-Han; education as middle school or less, high school or less, college or above; employment as unemployed, part-time, full-time; income as <2,000 RMB per month, 2,000–3,999, >3,999; marital status as married, divorced/separated/widowed, never married; and sexual identity as gay/homosexual/bisexual, heterosexual, unknown.

MOS-SSS

The MOS-SSS is a 19-item survey originally designed to assess five different dimensions of social support including emotional, informational, tangible, and affectionate support as well as positive social interaction. Respondents are asked to choose how often each kind of support is available to them on a 5-point Likert scale from 0-”none of the time,” 1-”a little of the time,” 2-”some of the time,” 3-”most of the time,” to 4-”all of the time”, with higher scores indicating better perceived support. Administered to a sample of 2,987 patients with various chronic illness in the United States (US), the MOS-SSS proved to have good psychometric properties, including high internal consistency (Cronbach’s α = 0.97), high test–retest stability (intra-class correlation = 0.78), good convergent, and discriminant validities, as well as good construct validity [13]. In the present study, the Chinese version of MOS-SSS showed acceptable internal consistency with Cronbach’s α of 0.90 for the first sample and 0.97 for the second sample.

Data analysis

All data were analyzed using STATA version 12. Socio-demographic characteristics of the sample were examined using descriptive statistics such as count, percent, and median. Factorial validity was first evaluated by CFA to test the a priori four-factor structure of MOS-SSS for each sample. EFA was then performed using the principle components method for factor extraction with oblique rotation to explore other possible models that could be confirmed with CFA and compared to the original four-factor CFA.

In EFA, factors retained were those with an eigenvalue above 1. Factor loadings equal to or >0.4 were considered appropriate [32]. CFA goodness-of-fit measures for comparison to the a priori four-factor solution included relative chi-square (χ²/df), Comparative Fit Index (CFI), Tucker–Lewis Index (TLI), root mean square error of approximation (RMSEA), and standardized root mean square residual (SRMR) [33]. Relative chi-square is the ratio of chi-square to degrees of freedom, and its recommended reference value is <3 for acceptable model fit [34]. Values for CFI and TLI range between 0 and 1, with values closer to 1 indicative of data fitness [35]. An RMSEA ranging from 0.08 to 0.10 shows a mediocre fit and below 0.08 indicates a good fit [36]. The acceptable value for SRMR is <0.10, with values<0.08 indicating adequate fit, and values below 0.05 indicating good fit [37, 38].

Results

Sample characteristics

The socio-demographic characteristics for each sample are provided in Table 3. The two samples were comparable in basic demographic information including age, gender, ethnicity, and employment status. Most of the participants in both samples were young with a median age of 35.5, male, of Han nationality, and employed.

However, there were significant differences between the two samples with regards to education, income, marriage, and sexual identity. Compared to the first sample, the second sample reported higher education (college and above degree: 24 vs. 49 %); lower poverty (monthly income below 1,999 RMB per month: 58 vs. 35 %); were less likely to be married (married: 55 vs. 35 %); more homosexual/bisexual (41.7 vs. 47.5 %).

CFA on the original four-factor structure

Results of CFA analyses generally supported the a priori specified four-factor structure across both populations (see Table 4). For both samples, the relative chi-squares (χ²/df) were lower than 3 (1.74 vs. 2.66) indicating the fitness of the model [34]; the values for CFI and TLI were close to 1.0 (0.87/0.85 vs. 0.93/0.91), showing the goodness-of-fit for the data [35]; the RMSEAs were between 0.08 and 1.00 (0.083 vs. 0.097) suggesting a mediocre fit [36]; the SRMRs not exceeding 0.08 (0.08 vs. 0.05) further confirmed an adequate fit for the model [37, 38]. In general, CFA suggested an acceptable four-factor structure with good fit according to the relative chi-square, CFI and TLI, and mediocre to adequate fit according to the RMSEA and SRMR.

Table 4.

CFA on the four-factor and two-factor structure on both samples

	Study population
	First sample (n = 120)		Second sample (n = 200)
	Four-factor	Two-factor	Four-factor	Two-factor
χ2/df	1.74	1.94	2.66	2.68
RMSEA	0.083	0.093	0.097	0.098
SRMR	0.080	0.091	0.050	0.051
CFI	0.873	0.836	0.926	0.924
TLI	0.851	0.811	0.913	0.912

χ²/df relative chi-square, RMSEA root mean square error of approximation, SRMR standardized root mean square residual, CFI Comparative Fit Index, TLI Tucker–Lewis Index

EFA

In both samples, EFA yielded a two-factor solution with two initial eigenvalues above 1 (6.46/1.78 vs. 11.92/1.13) generated by a principle analysis (Table 5). The two-factor structure accounted for 82.5 % of the total variance in the first sample and 93.63 % in the second sample. However, the items included in each factor varied slightly between the two samples. For the first sample, the first factor included all items from the originally hypothesized sub-scales of informational support, emotional support (except for item 17: Someone who understands your problems), and positive social interaction, with another item from the affectionate subscale (item 4: Someone who shows you love and affection). The second factor included all items from the tangible and affectionate subscales (except for item 4). For the second sample, the original tangible support subscale loaded as one factor, while the rest subscales were aggregated into another factor. The inter-factor correlation was 0.78 for the first sample and 0.83 for the second sample, suggesting overall high inter-correlations between the two factors.

Table 5.

EFA of the MOS-SSS-CM on both samples

	Study population
	First sample (n = 120)a	Second sample (n = 200)b
Factor 1 item loadings	0.49–0.77	0.44–0.88
Factor 2 item loadings	0.45–0.82	0.49–0.95
Eigen value	6.46/1.78	11.92/1.13
Proportional variance	0.65/0.18	0.86/0.08
Cumulative variance	0.65/0.83	0.86/0.94
Inter-factor correlation	0.78	0.83

^aFactor 1 includes item 1, 2, 4–7, 9–11, 14–16; factor 2 includes item 3, 8, 12, 13, 17–19

^bFactor 1 includes item 1, 2, 4–11, 14–18; factor 2 includes item 3, 12, 13, 19

Comparison of CFA on the original four-factor structure with CFA on the new two-factor structure

After a two-factor solution was generated with EFA on both samples, further CFA was performed on the new two-factor structure for each sample. The model fit indices including χ²/df, CFI, TLI, RMSEA, and SRMR of each factor structure for each sample are listed in Table 4. Notably, although EFA supported a two-factor solution, comparison of CFA on both the four-factor structure and two-factor structure showed that both were of acceptable model fit, with fit indices of the original four-factor structure being actually slightly better than the two-factor structure. For example, the RMSEA was marginally lower for the four-factor than the two-factor structure for both samples (0.083/0.093 vs. 0.097/0.098), while the CFI was marginally higher in four-factor structure than in two-factor structure for both samples (0.873/0.836 vs. 0.926/0.924), further suggesting the superiority of the four-factor structure over the two-factor structure.

Discussion

Compelling health research requires psychometrically sound assessment measures. To our knowledge, this is the first study comparing both CFA and EFA across two populations to explore the reasons for the different factor structures often found for the commonly used MOS-SSS. Our findings demonstrate that the factor structure of the MOS-SSS is method-dependent. Specifically, CFA on both samples generally supported the a priori four-factor structure; however, the EFA yielded a two-factor structure for both samples. A comparison of CFA on the original four-factor structure and the new two-factor structure indicated the four-factor structure had a slightly better fit than two-factor structure.

These results extend previous psychometric testing of MOS-SSS not only across two samples but also spanning two analytic methods. EFA is a data-driven analytic approach, aiming to explore the underlying factor structures that can explain as much item variation as possible with no or low between-factor correlation. The error terms are assumed to be normally distributed and independent from each other. In sharp contrast, CFA aims to test a factor structure specified a priori and theory driven. CFA often employs structural equation modeling that allows inter-correlation between factors and between residuals of different items [39]. In general, EFA will be considered when there is very limited prior information about the factor structure among the items in a given population. EFA is then applied to extrapolate the numbers of the factors and estimate factor loadings of each item on all the factors retained. In other words, cross-loading of an item on different factors often occurs. CFA, on the other hand, is usually chosen when the aim is to replicate the specified factor structure documented in the literature. As the major purpose is to confirm rather than explore factor structure, factor loading of each item is often fixed at one particular factor with minimum cross-loadings. However, as the two approaches parameterize the latent factors as well as the relationships between items and factors in different ways, they may or may not yield similar results.

Although CFA on the a priori four-factor structure of the MOS-SSS showed somewhat acceptable model fit in both samples, the model fit indices were less favorable compared to most of the previous psychometric testing of MOS-SSS. For instance, the RMSEA of the MOS-SSS in a sample of 297 people living with HIV/AIDS in Hunan province [26] and a sample of 200 coronary heart disease patients in Xi’an [28] were both lower than 0.08(0.067, 0.064, respectively), demonstrating good fit, while the RMSEA in the present study were between 0.08 and 1.00 across both samples (0.083, 0.097, respectively), indicating only mediocre fit [36]. Interestingly, similar results were also reported in Robitaille’s study [21], where the RMSEA of the English version of MOS-SSS was slightly high (0.076); and in Yu’s study [23] on a sample of 110 heart failure patients in Hong Kong, where they found the goodness-of-fit criteria were only marginally met. It seems that CFA supports the a priori four-factor structure with a different degree of stability across different populations.

Contrary to a three-factor structure produced by EFA on three different populations with two different language versions of MOS-SSS [16–18], EFA in the present study yielded a two-factor solution with different items included in each factor across both samples. It is noteworthy that for the second sample, the two-factor structure was exactly the same as a previous EFA on a sample of 265 Taiwanese [25], which aggregated the original hypothesized affectionate, emotional, informational support, and positive social interaction into one dimension called emotional support, while keeping the original tangible support as the other dimension. For the first sample in our study, the two-factor structure seems to be less consistent with the original hypothesized dimensions, with each factor containing part of the original subscale items, which is similar to the EFA result in a sample of South African diabetic outpatients [30]. The reduced dimensions of MOS-SSS in Portuguese, Spanish, and Chinese may be caused by the high correlation among various social support dimensions, or the relatively narrow sources of social support in those cultures, which have been discussed in detail elsewhere [16–18, 21, 23, 30].

The main innovation of this study is that it is the first study reviewing and comparing both EFA and CFA across two populations and extending previous psychometric testing of MOS-SSS not only to different samples but also to different analytic methods. It is the first study exploring the relationship between factor structure and analytic method with the conclusion that factor structure is method-dependent, thereby recommending caution when choosing the appropriate analytic method to yield a factor structure that best fits the theoretical model of interest.

One limitation of our results is the relatively small sample size of 120 for the first study and 200 for the second study. Compared to the sample size of over 3,000 in Robitaille’s [21] study on English and French version of MOS-SSS and over 4,000 in Griep’s [16] study on Portuguese version, the sample size in our study is relatively small. However, according to Tamaka’s recommendation [40], a sample size of at least five participants for each item to conduct a factor analysis is acceptable, showing that our sample size would not compromise the power of our analysis. Furthermore, another study [17] using a sample of 200 to explore the factor structure of the Portuguese version of MOS-SSS reported the same result as Griep’s sample of 4,000, further demonstrating that sample size above a threshold is not an issue in the factor analysis.

An additional possible limitation is the comparability of two samples. Although the two samples are comparable in basic demographic information such as gender, age and ethnicity, significant differences exist in education, income, marital status and sexual identity. Given that the two samples were of people living with HIV/AIDS with the same inclusion criteria drawn from the same hospital, with the only structural difference being data collection of a 6-year gap, these samples may be considered to be drawn from the same population. Indeed, as researchers, it is impossible to return to the same time period to repeat exactly the same study on the same population; thus, these samples should theoretically represent enough similarity for replication. Furthermore, a review of previous studies also shows that the same analytic method always produced similar, if not the same result regardless of the various samples used, which further corroborates our hypothesis that it is the analytic method instead of the sample that drove the different factor structures of the MOS-SSS in our study.

Another limitation is that the study populations were restricted to two convenience samples of people living with HIV/AIDS in China’s capital of Beijing, which may not be generalizable to populations with other diseases, or living in other parts of China or in other countries. However, as mentioned above, sample characteristics do not affect factor structure as severely as analytic methods and therefore may not significantly impact generalizability.

In conclusion, our study indicates that there is no unified standard on the dimensions of the MOS-SSS thus necessitating choosing the factor structure that best fits the theoretical model at hand.