Construct Validity of the SF-12 among American Indian and Alaska Native People Using Two Known Scoring Methods

Abstract

Objective

This study evaluated the construct validity of the 12-Item Short Form Survey Instrument (SF-12) in a cohort of American Indian and Alaska Native (AIAN) people. We evaluated two scoring methods to determine their utility in this population.

Methods

Participants (N=11,127) were aged 18 and older, self-identified as AIAN, and had complete SF-12 interview data. Physical and mental health summary scores were calculated using traditional SF-12 (PCS12 and MCS12) and RAND-12 (PHC and MHC) scoring methods.

Results

Women scored lower than men on the PHC, PCS12, MHC, and MCS12, as did those with more medical conditions versus none. Those aged 55 and older scored lower on the PHC and PCS12 than younger people. There was no difference in the mean MCS12 score by age and for those 31–55 and aged older than 55 for the MHC.

Conclusions

This study demonstrates the construct validity of the PCS12/MCS12 and PHC/MHC in a cohort of AIAN people.

Health status questionnaires are commonly used in clinical and epidemiologic studies to measure health-related quality of life and its relationship to health outcomes.¹ Given the importance of accurately measuring health status, it is critical that we have valid questionnaires to access the health status of American Indian and Alaska Native (AIAN) people in light of increasing obesity and chronic disease rates among AIAN.^2–4 A recent study of almost 4,000 AIAN adults living in Alaska found a higher prevalence of chronic disease risk factors compared to the U.S. population based on the National Health and Nutrition Examination Survey (NHANES) data.⁵ Comparison of AIAN and U.S. non-Hispanic White populations who were 55–64 years and died between 2002–2004 showed AIAN populations had higher death rates for numerous conditions. American Indian and Alaska Native people had higher death rates from heart disease, diabetes, chronic liver disease, injuries, respiratory diseases, kidney diseases, and stoke than their non-Hispanic White counterparts.⁶

The Medical Outcomes Study Short-Form (SF) set of measures (SF-36, SF-12) developed by Ware et al. are commonly used measures of health-related quality of life. To separate the physical and mental dimensions, Ware et al. posited that the physical and mental health summary scales should not be correlated. To do this, the summary scores were developed using orthogonal factor rotations within principle component factor analysis.^7–10 Factor score coefficients or loadings were based on how much each of the eight subscales contributed to a either the physical or mental dimension factor.¹¹ The result of this analysis yielded factor score coefficients for the Physical Component Summary (PCS) that were positive for the summary scores in the physical domain (physical functioning, role-physical, bodily pain, and general health) and negative for the summary scores in the mental health domain (vitality, social functioning, role-emotional, and mental health). Conversely, the factor score coefficients for the Mental Component Summary (MCS) were positive for the summary scores in the mental health domain and negative for summary scores in the physical domain. Hays and Morales report that the scoring method developed by Ware et al. leads to counterintuitive results where high mental health scale scores lower the PCS and high physical health scores lower the MCS scores.¹²

An alternative scoring method for the SF-36 and SF-12 questionnaire items has been developed by Hays et al.¹³ This method utilized item response theory (IRT) scoring to produce equal measurement intervals for each of the eight subscales and incorporated a correlated or oblique factor solution.¹² As a result, the physical and mental health summary scores are allowed to be correlated in this scoring method known as the RAND-36 and, in its short form, the RAND-12. The RAND Physical Health Composite (PHC) score uses only physical functioning, role-physical, bodily pain, and general health summary scores. No negative weighting is applied to those summary scores from the mental health domain. Similarly the RAND Mental Health Composite (MHC) score uses only the vitality, social functioning, role-emotional, and mental health summary scores in its calculations and no negative weighting is applied to summary scores in the mental health domain.

The results of these alternative scoring methods have very different outcomes in the final scores. In the Ware scoring to keep the PHS and MCS uncorrelated, a high score within the mental health domain will actually lower the PHS and a high score in the physical health domain will lower the MCS. The RAND scoring method utilized a correlated factor solution and does use the summary scores from one domain in the calculations of the final score for the other domain.

The standard scoring of the SF-36 and SF-12 questionnaire developed by Ware and the RAND-36 and RAND-12 method of scoring have been compared in various populations. The RAND-36 scoring performed similarly to the SF-36 scoring in a mostly non-Hispanic White population of general medical patients in the Western U.S. However, for each chronic medical condition, the RAND-36 showed a slightly larger decrement in mental health than did the SF-36.¹⁴ Both the SF-12 and RAND-12 scoring methods performed well in a rural population in Australia, although, the range of scores from the RAND-12 was wider than those from the standard SF-12. The authors recommended use of the standard SF-12 scoring when distinct uncorrelated physical and mental constructs are required, while the RAND-12 scoring method be used if correlated physical and mental constructs are needed.¹⁵ An Australian community-based survey found that those with high SF-12 physical health scores also had low mental health scores as a result of the SF-12 scoring method. They advise researchers to use caution when using uncorrelated factor scores if in fact the constructs are associated.¹⁶ A study of people with type II diabetes found both the RAND-12 physical and mental health component scores to be significantly associated with measures of disease severity while the standard SF-12 scores were not.¹⁷

To our knowledge, only two studies have been conducted among AIAN populations to evaluate the utility of the Short Form Health Survey measures. The SF-36 was used to assess health-related quality of life among 54 Pima Indian and showed favorable internal consistencies for SF-36 scales.¹⁸ In an effort to understand the psychometric characteristics and factor structure of the SF-36 for an older American Indian population, Beals et al. conducted an analysis of the questionnaire items, using both exploratory and confirmatory factor analysis.¹⁹ The results indicated that the SF-36 performed adequately in this population of older American Indians, but cautioned that “the use of summary scores assuming a differentiated physical/mental functioning structure is likely improper in at least some populations.”^19[p.208]

The primary objective of this study was to evaluate the construct validity of the 12-Item Short Form Survey (SF-12) questionnaire in a cohort of AIAN people using two different known and recognized scoring methods, 1) the standard SF-12 scoring developed by Ware and 2) the alternative method of scoring, RAND-12.^10,13 Construct validity refers to the ability of a questionnaire to detect previously hypothesized associations to external criteria such as demographic or clinical variables. These associations have been widely evaluated for the SF-12 and to a lesser extent the RAND-12 in studies involving English speaking general populations ^20–22 in studies of different disease-specific populations,^17,23–27 and in studies of underserved populations^28–30 as well as studies involving non-English speaking populations.^31–33 To evaluate construct validity, we assessed the ability of the summary scales from both methods to detect hypothesized associations within the study population with health and lifestyle variables. Based on existing literature, we hypothesized that 1) women would have lower physical and mental health summary scores than men;^28,31 2) older participants would have lower physical health summary scores than younger ones;^28,31 3) higher physical and mental summary scores would be associated with better perceived general health status;²⁸ and 4) those reporting specific chronic medical conditions as well as multiple chronic medical conditions would have both poorer physical and mental health summary scores although this would differ by scoring methods with the RAND-12 scores lower than the SF-12.^{7,14,28,29,31,34} Furthermore, the study examines the strength of the association of the sub-scores with various health and lifestyle variables by comparing the extremely low sub-scale score group to the extremely high sub-scale group on various health and lifestyle variables to evaluate differences within the population.

Methods

Study background

The study was part of the Education and Research Toward Health (EARTH) study, a pilot study to explore the feasibility of establishing a cohort of AIAN people. The purpose of the EARTH study was to determine how diet, physical activity, and other lifestyle and cultural factors relate to the development of chronic diseases. The study protocol was approved by the Alaska Area Institutional Review Board (IRB), the Navajo Nation IRB, Indian Health Services IRB, the University of Utah IRB, the research and ethics committees and governing boards of each of the participating regional health corporations, and the tribal councils of the participating communities. Study participants were 18 years of age or older, self-identified as American Indian or Alaska Native, gave informed consent, understood English or one of the tribal languages included in the study, and were eligible to receive care at the Indian Health Service. All participants signed an informed consent form before participating in the study.

Data collection

Detailed study methods have been described elsewhere.³⁵ Convenience sampling within communities was used with enrollment open all tribal members. Briefly, the baseline study visit consisted of an interviewer-administered Intake Questionnaire (usually administered by AIAN staff) that included demographic data such as education, employment status, age, and marital status; medical measurements that included height, weight, waist circumference, fasting glucose, triglycerides, total cholesterol, high density lipoprotein cholesterol, and very low density lipoprotein cholesterol; touch-screen audio computer-assisted self-interview (ACASI) diet history questionnaire (DHQ) that included use of alcohol and beverages containing caffeine; and a health, lifestyle, and physical activity questionnaire (HLPA), that included the SF12 questions, use of non-steroidal anti-inflammatory drugs (NSAIDS) and aspirin, physical activity, and information on a variety of self-reported medical conditions diagnosed by a doctor or other health care provider prior to the baseline study visit (see Table 1).^36,37 As the SF-12 had not been translated and validated for administration in Yup’ik or Navajo, study protocol restricted the inclusion of the SF-12 questionnaire to only those participants whose baseline study visit was conducted in English. A total of 11,127 participants enrolled in the study from March 2004 through October 2007 had complete SF-12 data.

Table 1.

Characteristics of participants who completed the SF-12 questionnaire, earth (Education and Research Toward Health) study, 2005–2007

	Na	%
Gender
Male	4188	37.6
Female	6939	62.4
Age (years)
Age 30 or less	3476	31.2
Age 31 to 55	6034	54.2
Over 55	1616	14.5
Marital Status
Marriedb	4824	43.4
Widowed	524	4.7
Divorced	1052	9.5
Separated	642	5.8
Never married	4062	36.6
Education
Less than high school	3200	28.9
High school and beyondc	7295	65.9
College degree or more	570	5.2
Income
25,000 or less	6282	65.7
25,001–50,000	2282	23.9
More than 50,000	1004	10.5
Employment
Employed	4900	44.1
Not employed	3099	27.9
Homemaker	1096	9.9
Student	836	7.5
Retired	678	6.1
Other	502	4.5
Language at home
Tribal language	1107	9.9
English	4697	42.2
Both	5272	47.4
Other	47	0.4
General Health (Self Report)
Poor	392	3.5
Fair	2596	23.3
Good	4708	42.3
Very good	2434	21.9
Excellent	997	9.0
Number of Medical Conditionsd
None	3950	35.5
1 to 2	4764	42.8
3 or More	2413	21.7
Depression (self-reported)
Yes	1833	16.9
No	9045	83.1
Vigorous Activity
None	3515	31.7
0.49 hr/day or less	4928	44.4
0.5 to 1 hr/day	1131	10.2
>1 hr /day	1513	13.6

^aNumbers vary slightly due to item nonresponse.

^bIncludes married and living as married.

^cHigh school and beyond includes high school graduate or GED, some college/technical school, technical school degree, and associate degree.

^dMedical conditions included self-reported cancer, high blood pressure, high cholesterol, chronic obstructive pulmonary disease (COPD), asthma, liver disease, stroke, any heart disease, diabetes, gallbladder disease, kidney disease, any thyroid condition, any bone fracture after 18, and depression.

Statistics

Body mass index (BMI) was calculated using measured height and weight (kg/m²). The presence of metabolic syndrome was defined using the definition of the Third Report of the National Cholesterol Education Program.^3,38 Perceived general health status was based on the general health question of the SF-12 “In general, would you say that your health is excellent, very good, good, fair or poor?” and defined as excellent, very good, good, fair, and poor. A count of the medical conditions was created by summing the number of self-reported medical conditions from the HLPA. Dichotomous variables were created for gender, traditional medicine use (yes versus none), education (less than high school versus high school or more), aspirin or other nonsteroidal anti-inflammatory drug (NSAID) use (yes versus none), other marital status (widowed, divorced, separated) versus married, other employed status (not employed, homemaker, retired, other) versus employed, income (lower versus higher [this analysis did not include the group in the middle]), vigorous activity (one hour vigorous activity per day versus none), alcohol (two or more drinks per day versus none), and caffeine (two or more drinks per day versus none).

The scoring protocol for the SF-12v1 was written in SAS following the scoring protocol recommended by Ware et al.¹⁰ The summary scores based on the Ware et al. scoring method using 1998 population norms are referred to as the SF-12 Physical Component Summary (PCS12) or the SF12 Mental Component Summary (MCS12). An algorithm also was written in SAS using regression equations developed by RAND for the overall age-stratified sample.¹³ Scores using the RAND scoring method are referred to as the RAND Physical Health Composite T score (PHC) or the base RAND Mental Health Composite T score (MHC). The SF-12v1 scoring allows three missing items for each subscale, whereas the RAND scoring method allows only one missing item for each subscale. Due to this discrepancy and the low frequency of incomplete questionnaires (115 out of 11,242: 1.0%), only those with complete SF-12 data (n=11,127) were included in the analysis for both scoring methods.

Statistical analyses were performed using SAS version 9.1.3. (SAS Institute Inc., Cary, NC) Spearman correlations were performed for each of the four subscales: 1) PCS12, 2) MCS12, 3) PHC, and 4) MHC. (The RAND General Health subscore was not considered in this analysis.) Differences in means for each subscale were calculated using analysis of variance (a common method used for such testing) for gender, age differences, number of chronic diseases, self-reported depression, and general health. Type I error rate for pair-wise comparisons was controlled using Tukey’s HSD (Honestly Significantly Different) test.

To compare the SF-12 and RAND-12 scoring methods, those who scored in the lowest decile for each component summary score were compared to those scoring in the highest decile Given the cross-sectional nature of the data, age-adjusted prevalence relative risks (PRR) for having a score in the lowest decile were calculated for various health conditions and lifestyle factors. These PRRs were calculated under a log-binomial model using the genmod procedure (SAS 9.1.3).

Results

Population characteristics

Table 1 describes the 11,127 study participants. Participants tended to be female (62.4% versus 37.6%), between 31 and 55 years of age (54.2%), currently employed (44.1%), and had at least a high school education (71.1%). Less than half (43.4%) were currently married. The majority (73.2%) perceived their health status as good, very good, or excellent, and 64.5% percent self-reported having one or more health care provider-diagnosed medical conditions.

Comparison of PHC, PCS12, MHC, and MCS12 to gender, age, medical conditions, self-reported depression, and general health

The correlation between the PCS12 and MCS12 was near 0, r= −0.097 (p<.0001), while the PHC and MHC were positively correlated, r=0.500 (p<.0001). Differences in mean scores for the PHC, PCS12, MHC, and MCS12 for male and female and for number of medical conditions (none, 1–2, 3 or more) were statistically significant (Table 2). Females scored significantly lower than men on the PHC, PCS12, MHC, and MCS12. Those older than age 55 scored lower on both the PHC and PCS12 than younger people, there were no significant differences in MCS12 scores across age categories, and those 30 or younger had significantly higher MHC scores than the other two age groups. Individuals without self-reported depression scored significantly higher on the PHC, PCS12, MHC, and MCS12 with a greater difference in the means for the MHC and MCS12. The mean scores for the PHC, PCS12, MHC, and MCS12 for those reporting better general health were significantly higher.

Table 2.

PCS12, MCS12, PHC, and RAND MH scores by gender, age, and medical condition, earth study 2005–2007

		PCS12	MCS12	PHC		MHC
	N	Mean (SD)	Mean	Mean		Mean
Gender
Male	4090	50.08 (9.47)	46.89 (8.02)	45.32 (9.33)		49.02 (10.05)
Female	6787	49.01 (9.56)	45.50 (8.56)	44.22 (9.29)		46.17 (10.50)
Age (years)
30 or less	3385	51.33 (8.22)	46.21 (8.01)|*	46.64	7.68	48.24 (9.79)
31 to 55	5912	49.27 (9.79)	45.84 (8.50)|	44.4	9.63	46.84 (10.66)|*
Over 55	1580	45.83 (10.11)	46.31 (8.73)|	41.22	10.21	46.64 (10.70)|
Number of Medical Conditions
None	3853	51.51 (8.44)	47.49 (7.47)	47.28	7.72	50.02 (9.15)
1–2	4673	50.09 (9.05)	45.65 (8.45)	45.23	8.72	47.03 (10.35)
3 or more	2351	44.62 (10.51)	44.38 (9.26)	39.13	10.52	43.13 (11.09)
Self-reported depression
Yes	1833	46.36 (10.78)	40.85 (9.27)	40.55	10.61	39.44 (10.62)
No	9044	50.03 (9.15)	47.08 (7.79)	45.46	8.81	48.83 (9.64)
General health
Poor	377	40.56 (10.41)	40.76 (9.70)	32.06	9.12	36.61 (10.49)
Fair	2520	45.5 (10.04)	44.31 (9.05)	38.32	8.82	42.68 (10.37)
Good	4604	49.88 (8.83)	46.38 (8.13)	44.69	7.58	47.62 (9.65)
Very good	2393	52.61 (8.14)	47.14 (7.65)	50.09	6.98	50.48 (9.21)
Excellent	983	52.84 (8.61)	48.09 (7.38)	52.09	7.20	53.41 (9.01)

^*All means except those joined by a | are significant at p <.001

PCS = Physical Component Summary

MCS = Mental Component Summary

PHC = Physical Health Composite

Comparison of PHC, PCS12, MHC, and MCS12 to perceived health status

The relationship of perceived health status to the PHC, PCS12, MHC, and MCS12 is shown in Figure 1. There were differences between the two scoring methods although the mean for each subscale increased directly with better perceived health status. However, the MHC and PHC scores showed a sharper percentage increase with better perceived health status than the PCS12 and MCS12 scores. The change in mean MHC and PHC scores between the excellent and poor categories increased by 46% and 62%, respectively, while MCS12 and PCS12 increased by 18% and 29%, respectively.

Figure 1.

Comparison of PCS12, MCS12, PHC, and MHC to perceived general health status, EARTH Study 2005–2007.

Prevalence relative risks

Figure 2 shows the PRRs for reported medical conditions and the PHC, PCS12, MHC, and MCS12 scores. Arthritis and number of medical conditions had the highest impact on risk ratios for both low PCS12 and PHC. Figure 3 shows the PRR for the MCS12 and MHC for medical conditions and lifestyle and socio-demographic factors. Self-reported depression and number of medical conditions had the greatest impact on risks for low MCS12 and MHC while lifestyle factors had a greater impact on risks associated with the MHC than the MCS12.

Figure 2.

Prevalence ratios for being in the lowest decile for PCS12 and PHC scores for self-reported medical conditions, EARTH Study 2005–2007.

Figure 3.

Prevalence ratios for being in the lowest decile for MCS12 and MHC scores for self-reported medical conditions and lifestyle factors, EARTH Study 2005–2007.

Discussion

This study supports construct validity from both the traditional SF-12 (PCS12 and MCS12) scoring method and the RAND-12 (PHC and MHC) method of scoring the SF12 questionnaire for this population of AIAN people. The four subscales used to score the SF-12 detected previously hypothesized differences and associations with other health and lifestyle variables. Female participants scored lower on all four sub-scales than men. Participants aged 30 or younger scored higher on both the PCS12 and PHC than older individuals. Those reporting more medical conditions had lower scores on all four subscales than those who self-reported fewer medical conditions. Participants who self-reported depression scored lower than others on all mental and physical subscales.

These summary subscales, which were originally validated in a predominantly non-Hispanic White population, are often used to measure health outcomes. While the literature examining the construct validly of the SF-12 in diverse populations is growing, no studies to date have compared the SF-12 scoring to the RAND-12 scoring in AIAN populations. In our data, PRRs were significant for all medical condition and lifestyle variables for the MHC and PHC. Using the SF-12 scoring all PRRs for medical conditions were significant for the PHS12, but there were few significant PRRs for the MCS12 for medical condition and lifestyle variables.

Though both scoring methods were associated with general health status, subtle differences were observed in those relationships. Those who reported poor general health had similar PCS12 and MCS12 scores, while those who reported excellent general health had higher PCS12 than MCS12 scores. For the RAND scoring method, those reporting poor general health had lower PHC than MHC scores, while those reporting excellent general health, had comparable PHC and MHC scores. Additionally, those who reported more medical conditions were more likely to be in the lowest decile for the MHC than the MCS12. This finding is consistence with the literature,^19,39 indicating that physical and mental health interact and the choice of scoring methods should be considered carefully.

This is the largest study to date looking at health-related quality of life measures and health-related factors in AIAN populations; however, there are limitations. We only were able to administer the questionnaire to those participants who could speak English, since the questionnaire was not validated into Navajo or other AIAN languages and could not be translated. Thus, these results only apply to AIAN people who speak English. Given the cross-sectional nature of our data, we are unable to evaluate causal associations. Finally, we were limited to comparisons within the study population at the request of the participating tribes.

The objective of this study was to evaluate the construct validity of the SF-12 using two scoring methods. We have demonstrated the construct validly of both scoring methods. However, our data suggest that some differences in associations with other health and lifestyle factors result from the scoring method used in this AIAN population.

Abbreviations

ACASI

Audio computer-assisted self-interview

AIAN

American Indian and Alaska Native

BMI

Body mass index

COPD

Chronic obstructive pulmonary disease

DHQ

Diet history questionnaire

EARTH

Education and Research Toward Health

HLPA

Health, lifestyle, and physical activity questionnaire

HSD

Honestly Significantly Different

IRB

Institutional Review Board

MCS, MCS12

Mental Component Summary, Mental Component Summary of the SF-12

MHC

RAND Mental Health Composite T score of the SF-12

NSAIDS

Nonsteroidal anti-inflammatory drugs

PCS, PCS12

Physical Component Summary, Physical Component Summary of the SF-12

PHC

RAND Physical Health Composite T score of the SF-12

PRR

Prevalence Relative Risks

SF

Short Form Health Survey

SF-12

12-item Short Form Health Survey

SF-36

36-item Short-Form Health Survey