A Comparative Analysis of the Validity of US State- and County-Level Social Capital Measures and Their Associations with Population Health

Abstract

The goals of this study were to validate a number of available collective social capital measures at the U.S. state and county levels, and to examine the relative extent to which these social capital measures are associated with population health outcomes. Measures of social capital at the U.S. state level included aggregate indices based on the Annenberg National Health Communication Survey (ANHCS) and the Behavioral Risk Factor Surveillance System (BRFSS), Petris Social Capital Index (PSCI), Putnam’s index, and Kim et al.’s scales. County-level measures consisted of Rupasingha et al.’s social capital index (RGFI) and a BRFSS-derived measure. These measures, except for the PSCI, showed evidence of acceptable validity. Moreover, we observed differences across the social capital measures in their associations with population health outcomes. The implications of the findings for future research in this area are discussed.

INTRODUCTION

The concept of social capital was developed by sociologists and political scientists and has become one of the most widely employed concepts in social sciences to account for “how citizens within certain communities cooperate with each other to overcome the dilemmas of collective action” (Lochner, Kawachi, & Kennedy, 1999, p. 259). Since Wilkinson (1996) introduced the concept of social capital to the domain of public health, it has received tremendous attention as an environmental factor of importance to health. Based on numerous empirical studies, many scholars have argued that social capital is beneficial to population health (Islam, Merlo, Kawachi, Lindstrom, & Gerdtham, 2006; Kawachi, Subramanian, & Kim, 2008; Putnam, 2000).

However, some epidemiologic studies have failed to detect associations of social capital with health. The lack of evidence for beneficial effects of social capital may in part be because of the possibility that some social capital measures may be more valid than others, and that the effects of social capital may vary largely according to the kinds of social capital indices employed (Harpham, Grant, & Thomas, 2002; Narayan & Cassidy, 2002). It is therefore possible that scholars may come to different conclusions depending on which social capital measures are used. Despite the fact that a plethora of social capital measurement tools have been utilized in public health, little research has been conducted to evaluate the validity of social capital measures (De Silva et al., 2006; Roberts & Roche, 2001; van Deth, 2003).

We first aim to validate a group of social capital measures for U.S. states and counties, some of which are publicly available and others of which have been developed in this study following the procedures set by previous studies. One of the goals of this study is to demonstrate that by using existing data creatively and relying on publicly available indices, researchers can validly explore social capital processes. Therefore, we do not include burgeoning efforts to develop original social capital items for surveys (e.g., Narayan & Cassidy, 2002). We then examine whether there are differences across measures of social capital in their associations with health outcomes. In sum, this paper has three goals: (1) to introduce social capital researchers to key publicly available social capital measures at the U.S. state and county levels; (2) to test the validity of these social capital measures, so that researchers can strike a balance between their availability/convenience and theoretical/empirical rigours; and (3) to compare the associations between these social capital measures with health outcomes.

Social Capital as a Macro-Social Construct

Social capital has many definitions even within the field of public health. Despite the large variation in the conceptual definitions of social capital across studies, two common dimensions of social capital emerge (Kawachi et al., 2008). First, social capital is a social network of relationships either at an individual level or collective level. This is a structural dimension of social capital. Second, social capital can be defined by the extent to which the social network involves interpersonal trust and reciprocity. That is, social capital also has a cognitive dimension.

It should be noted that there are two separate approaches to social capital because each line of research requires a different measurement strategy (Harpham et al., 2002; Kawachi et al., 2008; van Deth, 2003). One camp treats social capital as an individual-level construct, viewing social capital as a similar concept to a social network (e.g., Bourdieu, 1986; Lin, 2001). According to this perspective, scholars measure individuals’ social capital by tapping the extent and nature of their social networks. These measures include the numbers of group memberships in voluntary associations, and the degrees of social trust and perceived reciprocity. A second group of scholars argues that social capital is a feature of collectives, and is a trans-individual, ecological construct, comprising actual or potential resources which can be mobilized by collectives to solve common problems (e.g., Coleman, 1988; Lochner et al., 1999; Putnam, 2000).

We view social capital as the latter macro-social construct. One of the major criticisms of social capital is that it simply puts a new label on previously developed sociological concepts i.e., “putting new wine in old bottles”. For example, many social scientists have argued that social capital is essentially synonymous with classic sociological concepts such as social networks and social support (Edwards & Foley, 1998; Portes, 1998). In fact, considering that these traditional concepts have been observed and analyzed primarily at an individual level, the conceptualization of social capital by the first school of social capital researchers is not much different from these traditional concepts. Therefore, we believe that the role of social capital as an aggregate property of collectives is more novel than its role as an individual attribute. Furthermore, collective social capital may be inherently more valuable from a public health perspective, since intervening at a population level can have a more powerful and widespread effect on improving health than intervening at an individual level.

Validating Measures of Social Capital at the Collective Level

Social capital researchers have measured collective social capital in three different ways, i.e., as aggregate vs. integral vs. hybrid measures (Diez Roux, 2002; Raudenbush, 2003). In tapping macro-level social capital, many scholars have aggregated individuals’ responses to the collective level (aggregate measures) by taking the mean level of reported formal group activities or social trust of individuals in a sample who live in a community as the measure of community-level social capital (e.g., Sampson, Raudenbush, & Earls, 1997). Rather than relying on survey responses of individuals who constitute the community, for which response bias and recall bias may be issues in constructing measures of collective social capital (e.g., Brown, Scheffler, Seo, & Reed, 2006; Rupasingha & Goetz, 2008), other scholars have used administrative records and official statistics (integral measures). Neither approach is inherently wrong. In some sense, it can be argued that both aggregate and integral measures of social capital are necessary (Raudenbush, 2003). In fact, many scholars including Putnam (2000) have constructed hybrid measures of social capital by using both survey data and administrative data sources.

Researchers in this field have not reached a consensus about which social capital measures should be used. To make an informed decision, rigorous validation tests should be conducted regardless of the methods to create collective social capital measures. This study therefore attempts to evaluate some key publicly available social capital measures on the basis of face validity, content validity, convergent validity, and nomological validity (Cronbach & Meehl, 1955; Schutt, 2009).

Face Validity

To examine whether social capital measures have face validity, one should compare each item comprising social capital indices with the conceptual definition of social capital. The concept of social capital is so abstract and complex (Narayan & Cassidy, 2002; Portes, 1998) that many scholars have tried instead to use easily accessible proxies, such as newspaper readership, voting turnout, blood donation, life expectancy, suicide rate, crime rate, and informal social control (Harpham et al., 2002; van Deth, 2003). Despite being practical and convenient, this practice has the risk of “logical circularity” or “tautology.” This is because these proxies should be considered as either predictors or outcomes of social capital.

Content Validity

Content validity refers to the extent to which the scales or indices “cover the full range of the concept’s meaning” (Schutt, 2009, p. 132; italics added for emphasis). Given that social capital is a complex and multi-dimensional concept (Narayan & Cassidy, 2002), it is important to examine the extent to which social capital indices encompass multiple aspects or dimensions of social capital.

Convergent Validity

Convergent validity is achieved when different measures intended to capture the same underlying phenomena are highly correlated with one another.

Nomological Validity

Nomological validity refers to the extent to which the concept under study is closely related to other constructs theorized and found to predict or result from that construct in previous studies (Crobach & Meehl, 1955; Shadish, Cook, & Campbell, 2002). To identify the constructs, we used the following three standards: (1) whether strong theoretical arguments have been presented, (2) whether there is a strong body of empirical studies, and (3) whether more than one group of scholars has produced supporting evidence. We used income inequality and violent crime rate as variables for the nomological validity tests, based on their consistent and significant associations with collective social capital found in previous studies (unlike area-level income, education, and urban sprawl, for which past evidence was less consistent).

First, social capital has been widely used as an explanatory factor underlying the effects of income inequality on health. Wilkinson (1996) found that more egalitarian societies have rich stocks of social capital, making them good for population health. Wilkinson proposed that low social capital could provide a psychosocial mechanism linking income inequality to poorer individual health. Following Wilkinson’s lead, many scholars have shown that income inequality is detrimental to the social cohesion of communities (e.g., Islam et al., 2006; Kennedy, Kawachi, Prothrow-Stith, Lochner, & Gupta, 1998).

Second, we use crime rates in this study to validate social capital measures because the lack of social capital has been widely accepted as a cause of violent crime in a community. Many social capital scholars including Sampson et al. (1997) and Kennedy et al. (1998) have empirically demonstrated that social capital may keep communities free from violent crimes.

Predicting Health Outcomes

A critical question is whether the size or direction of associations between the measures of social capital and health outcomes differs according to which social capital measure is used. No empirical studies, to our knowledge, have been conducted to compare the performances of different collective social capital measures in predicting health outcomes. Therefore, we compared the associations between each social capital measure and overall health outcome indices at both the U.S. state and county levels. We used the most recent, comprehensive indices of health outcomes at both geographic levels.

METHODS

State-Level Social Capital Measures

Putnam’s social capital index and Kim et al.’s updated measures

Putnam (2000) created a comprehensive index of state-level social capital based on a series of social surveys and administrative data during the time period 1974-1994 (Cronbach’s α = .80). Kim, Subramanian, Gortmaker, & Kawachi (2006) later updated Putnam’s social capital measure to the 1990s using updated data sources, and through factor analysis assigned ten of the indicators to two scales with high internal consistency reliability (Scale 1: Cronbach’s α = .83; Scale 2: Cronbach’s α = .90). The Putnam index and Kim et al. measures are both hybrid measures of social capital (see Table 1).

Table 1.

Components and Data Sources for Putnam’s and Kim et al.’s Social Capital Measures

Components of Comprehensive Social Capital Index		Data Sources
(1)	Measures of community organizational life
	bServed on committee of local organization in last year (percent)	The Roper Social and Political Trends Archive
	bServed as officer of some club or organization in last year (percent)	The Roper Social and Political Trends Archive
	Civic and social organizations per 1,000 population	U.S. Commerce Department
	aMean number of club meetings attended in last year	The DDB Needham Lifestyle Archive
	Mean number of group memberships	General Social Survey
(2)	Measures of engagement in public affairs
	bTurnout in presidential elections, 1988 and 1992	U.S. Statistical Abstract
	bAttended public meeting on town or school affairs in last year (percent)	The Roper Social and Political Trends Archive
(3)	Measures of community volunteerism
	Number of nonprofit organizations per 1,000 population	The Non-profit Almanac
	aMean number of times worked on community project in last year	The DDB Needham Lifestyle Archive
	aMean number of times did volunteer in last year	The DDB Needham Lifestyle Archive
(4)	Measures of informal sociability
	Agree that “I spend a lot of time visiting friends”	The DDB Needham Lifestyle Archive
	aMean number of times entertained at home in last year	The DDB Needham Lifestyle Archive
(5)	Measures of social trust
	aAgree that “Most people can be trusted”	General Social Survey
	aAgree that “Most people are honest”	The DDB Needham Lifestyle Archive

Notes:

^aSix indicators comprising social capital scale 1 by Kim et al. (2006), based on loadings from factor analysis.

^bFour indicators for social capital scale 2 by Kim et al.

Petris Social Capital Index (PSCI, hereafter)

The state-level PSCI was calculated by dividing the number of employees hired at voluntary organizations (NAICS 813 from the County Business Pattern (CBP) dataset) in each state by the total population of that state (Brown et al., 2006; Scheffler, Brown, & Rice, 2007). The CBP data used to construct PSCI was collected by the U.S. Census (2006). PSCI is an integral measure of social capital.

ANHCS social capital index

We also created an aggregate measure of social capital at the state level using a publicly available survey dataset, entitled the Annenberg National Health Communication Survey (ANHCS). ANHCS was conducted by Knowledge Networks (KN) for the Annenberg Schools for Communication at the University of Pennsylvania and the University of Southern California, with support from the Annenberg Trust at Sunnylands (ANHCS, 2008). ANHCS was designed to collect nationally representative data over the Internet every month about the American public’s health-related media exposures, behaviors, knowledge and beliefs, and health policy preferences and beliefs. In total, 13,487 civilian, non-institutionalized adults (ages 18 and above) were surveyed through a national probability sample between January 2005 and December 2008. Knowledge Networks identified respondents using random digit dialing (RDD) procedures and provided selected households who did not have home internet access with free hardware (Web TV) and internet access. Of those recruited for the panel, 29.9 percent agreed to participate. Of those who were in the panel and asked to participate, 68.3 percent agreed to participate in the ANHCS questionnaire. Thus, the overall response rate was 20.4 percent, the product of the recruitment rate and the cooperation rate.

We used the ANHCS respondents’ formal group membership measures to create a state-level social capital index. Respondents were asked to indicate if they actively participated in the following 15 types of organizations or groups: (1) service club or fraternal organization; (2) veteran’s group; (3) religious group; (4) senior citizen center or group; (5) women’s group; (6) issue-oriented political organization; (7) non-partisan civic organization; (8) school club or association; (9) hobby, sports team, or youth group; (10) neighborhood association or community group; (11) group representing racial/ethnic interests; (12) a community group meeting; (13) a charity; (14) a community board; and (15) working with others to solve a community problem. We used the mean scores of respondents’ number of formal group memberships for each state as the ANHCS state-level social capital index. There was acceptable internal consistency reliability (Kuder-Richardson Formula 20 = .71).

BRFSS measure

We also used the percentage of adults with social/emotional support in each state as an aggregate social capital measure. This was calculated originally at the US county level, using the Behavioral Risk Factor Surveillance System (BRFSS) data, 2005-2009, based on the County Health Rankings (2011). We derived the state-level BRFSS measures by aggregating the county-level measures (see the county-level social capital measures section below).

County-Level Social Capital Measures

At the county level, we evaluated (1) the county-level PSCI, which was constructed following a similar procedure to that described for the state-level index; (2) the BRFSS measure (County Health Rankings, 2011), and (3) a broad social capital index constructed by Rupasingha, Goetz, & Freshwater (2006). The BRFSS data enabled the calculation of the percentage of respondents with social/emotional support for 2,293 U.S. counties (County Health Rankings, 2011). Like the PSCI, Rupasingha et al. (2006) used the County Business Pattern (CBP) data, which provided the numbers of a variety of organizations for each U.S. county (e.g., civic organizations, bowling centers, golf clubs). In addition to associational density, Rupasingha et al. (2006) included the percentages who voted in presidential elections; the county-level response rates to the Census Bureau’s decennial census; and the numbers of tax-exempt non-profit organizations from the National Center for Charitable Statistics. Based on principal component analysis, they created overall social capital indices (RGFI, hereafter) from these sub-dimensions for the years of 1990, 1997, and 2005. We report only the RGFI for 2005 because the three indices were highly correlated one another (r > 0.8) and because the results reported in that paper were virtually the same across the three indices.

We chose not to validate county-level measures based on the ANHCS or Kim et al.’s (2006) county-level social capital scales. Both the ANHCS and DDB Needham Lifestyle Survey (included in the Kim et al. measure) lacked adequate numbers of respondents per county in many counties, which threatened the reliability of these ecological measures (Raudenbush, 2003); the relatively small proportion of counties with sufficient numbers of respondents prevented the data from being nationally representative.

Population Health Outcome Measures

County-level health outcome indices were obtained from the County Health Rankings (2011). We used the following four indices: premature death, poor physical health days, poor mental health days, and self-rated health. Premature death, which refers to the loss of years of productive life due to death before age 75, is based on 2005-2007 mortality data from the National Center for Health Statistics. Poor physical (mental) health days refers to the average number of days in the previous 30 days that a person could not perform work or household tasks due to physical (mental) illness. These measures are based on 2003-2009 BRFSS data. Self-rated health was measured as the percentage of adults who reported “fair” or “poor” overall health in the 2003-2009 BRFSS. The existence of inverse associations between county-level social capital and summary health outcomes would be evidence of the beneficial effects of county-level social capital on population health. See County Health Rankings (2011) for details.

At the state level, six health outcome measures were employed. First, we used the following five indices created by the America’s Health Rankings (2010): premature death, poor physical health days, poor mental health days, cardiovascular deaths, and cancer deaths. Premature death is based on 2007 data from the CDC. Poor physical (mental) health days were calculated using 2009 BRFSS data. Cardiovascular deaths were measured using a three-year average of the age-adjusted death rate due to heart disease, strokes, and other cardiovascular diseases. Likewise, cancer deaths were measured using a three-year average of the age-adjusted death rate due to cancer. These two measures were based on 2005-2007 CDC data. Second, we derived a state-level self-rated health measure from the county-level BRFSS self-rated health measure. Inverse associations between social capital and the health outcomes would support beneficial effects of state-level social capital on population health. See America’s Health Rankings (2010) and County Health Rankings (2011) for details.

Analysis Procedure

Face Validity

Among the three established methods of validity testing (i.e., logical analysis, correlation analysis, and experiment), logical analysis was adopted to conduct face validity tests of each social capital measure (Nunnally, 1978). That is, we examined face validity of the collective social capital measures based on the conceptual definitions of social capital. Even though some scholars (e.g., Edwards & Foley, 1998) have argued that social trust belongs to cultural capital rather than social capital or that social trust should be antecedent to social capital rather than social capital itself, we treated social trust as a valid component of social capital in keeping with the majority of scholars in public health (Harpham et al., 2002).

Content Validity

As outlined earlier, social capital consists of structural and cognitive dimensions (Narayan & Cassidy, 2002; van Deth, 2003). The structural dimension encompasses informal voluntary activities as well as community residents’ more organized or formal group activities. Many scholars (e.g., Kwak, Shah, & Holbert, 2004; Putnam, 2000) have argued that informal socializing, such as attending a dinner party and entertaining people in one’s home, also helps to build resources that can be used to solve common problems of a community. The cognitive dimension taps the average levels of social trust and norms of reciprocity of community members. We thus examine whether each social capital measure includes both dimensions and contains multiple aspects of each dimension.

Convergent Validity

A case could be made for the convergent validity of social capital measures if there are strong correlations among the social capital measures (Nunnally, 1978). In this way, we test whether these measures actually represent the same latent phenomenon – social capital.

Nomological Validity

We conducted correlation analyses to examine whether each social capital index shows expected associations with social phenomena that were found to be closely related to social capital in previous studies (Nunnally, 1978). The variables used to evaluate the nomological validity for the state- and county-level social capital measures were as follows:

State-level variables

The state-level Gini coefficient, a measure of income inequality, ranges from 0 (perfect equality) to 1 (perfect inequality), and was based on 2000 U.S. Census data on household income. Data on the number of violent crimes (i.e., murder, rape, robbery, aggravated assault) per 100,000 persons at the state level was obtained from the Bureau of Justice Statistics (2007).

County-level variables

The Gini coefficient corresponded to the county level, and was based on 2000 U.S. Census income data. County-level crime rates were taken from the Inter-University Consortium for Political and Social Research (ICPSR)’s data, entitled “County Characteristics, 2000-2007” (ICPSR, 2007). In this data, we compiled the numbers of murders, robberies, rapes, and aggravated assaults, to estimate the total numbers of violent crimes for each county in 2004. We divided these total numbers of violent crimes by the 2004 county population and multiplied them by 100,000.

Predicting Health Outcomes

Using multivariable ordinary least squares (OLS) regression analyses, we regressed each population health outcome on each social capital measure, while controlling for median household income and the percentage of African Americans, which have been shown to have strong effects on health of community residents (Morenoff et al., 2007; Szreter & Woolcock, 2004). This allowed us to examine whether the social capital measures differed in their ability to predict health outcomes.

RESULTS

For descriptive statistics for the U.S. state- and county-level social capital measures, see Table 2.

Table 2.

Descriptive Statistics for Social Capital Measures

	N	M	SD	Min.	Max.
State-Level Social Capital Measures
Putnam’s Index	48	.02	.78	−1.43	1.71
Kim et al.’s Scale 1	49	−.0000004	1.00	−2.23	2.17
Kim et al.’s Scale 2	49	−.0000002	.99	−1.94	2.57
ANHCS Social Capital Index	51	1.94	.27	1.40	2.78
Petris Social Capital Index	50	.92	.17	.45	1.24
BRFSS Measure	51	80.94	2.62	75.63	87.18
County-Level Social Capital Measures
Petris Social Capital Index	3,123	.62	.51	.00	6.44
Rupasingha, Goetz, & Freshwater Index	3,107	.00	1.65	−3.80	15.22
BRFSS Measure	2,293	80.55	5.02	46.60	94.60

Notes: (1) Putnam’s index includes 48 U.S. states (except Alaska and Hawaii). (2) Kim et al.’s scales include 48 U.S. states and Washington DC. (3) ANHCS social capital index and BRFSS measure include 50 U.S. states and Washington DC. (4) Petris Social Capital Index includes 50 U.S. states. Washington DC was excluded because it was an extreme outlier.

Validating State-Level Social Capital Measures

Face validity

All state-level social capital measures are somewhat limited in terms of face validity. The social capital index constructed using the ANHCS data and the BRFSS data rely only on survey data. There are a few problems with aggregate measures of social capital such as the ANHCS and BRFSS measures. First, respondents’ self-reports about characteristics of their living environment can be affected by respondents’ personalities, affect, and psychological adjustment (Glaeser, Laibson, Scheinkman, & Soutter, 2000; Raudenbush, 2003). Also, survey responses may change according to how the questions are asked and who administers the questions (Glaeser et al., 2000). If this is the case, the average scores of social trust or associational memberships may not be valid social capital measures. Second, survey results depend on how respondents interpret the question items and how accurately they recall their associational activities (De Silva et al., 2006).

Some of these problems also apply to Putnam’s (2000) and Kim et al.’s (2006) measures because they incorporate DDB Needham Lifestyle Survey data and General Social Survey (GSS, hereafter) data. Yet, because the measures of Putnam and Kim et al. contain official statistics (i.e., integral items) in addition to aggregate data, they are less problematic than the ANHCS and BRFSS social capital measures.

Unlike the aggregate social capital measures and hybrid measures, PSCI is free from the limitations inherent to survey data because it is solely based on administrative data sources.

Next, one needs to consider whether social capital indices include any invalid indicators. Indices of Putnam (200) and Kim et al. (2006) suffer from this problem because they contain “turnout in presidential elections” from the U.S. Statistical Abstract. Some scholars have argued that voter turnout is the outcome of social capital rather than social capital itself (e.g., Harpham et al., 2002; van Deth, 2003). Moreover, PSCI is also limited because all organizations included in PSCI do not necessarily create resources available to the members of organizations or to community residents. It has been argued that activities conducive to social capital should involve direct face-to-face communication among individuals in the locality (Putnam, 2000). In this vein, interest groups should be excluded from group membership because the former may be bureaucracies rather than associations in which real communication among citizens happens. Because most administrative records and official statistics do not classify organizations in this way, integral social capital measures are limited in this regard.

Content validity

To assess the content validity of social capital measures, we evaluated whether the operationalization of each social capital measure covers its diverse dimensions. Although PSCI captures structural dimension of social capital, PSCI does not capture informal socializing, which is assumed to be a critical element of associational activities. In addition, because PSCI is an integral measure constructed based on administrative data, it does not contain trust and norms of reciprocity of citizens.

Putnam’s (2000) and Kim et al.’s (2006) hybrid measures fare better than the PSCI, ANHCS, and BRFSS measures in terms of content validity. This is because the former two measures contain both cognitive and structural dimensions of social capital, while the PSCI and ANHCS index tap only the structural dimension and the BRFSS measure captures only the cognitive dimension. In addition, Putnam and Kim et al. include informal volunteering items from the DDB Needham Lifestyle Archives as well as formal group memberships.

The ANHCS and BRFSS social capital measures, as aggregate measures, have additional limitations in terms of content validity. A simple aggregation of individuals cannot provide exhaustive information about the whole because a group or community may be more than the sum of its parts (Diez Roux, 2002; Lochner et al., 1999). In this sense, scholars should try to employ integral items as well as aggregate ones. Therefore, relying solely on survey or official statistics can be limiting because both aggregate and integral measures are needed for content validity. In this regard, hybrid measures such as the indices of Putnam (2000) and Kim et al. (2006) are favourable.

Nomological validity

Both Putnam’s index (2000) and Kim et al.’s measure (2006) are very strong in their nomological validity; overall, both measures were correlated with the Gini coefficient and violent crime rate as expected (see Table 3). The ANHCS and BRFSS measures also appear to be valid based on expected patterns of associations. In contrast, the PSCI was weakly and non-significantly associated with the Gini coefficient, and was unassociated with violent crime.

Table 3.

Pearson Correlation Coefficients for Associations between State-Level Social Capital Measures with Income Inequality and Crime

		Gini Coefficient	Violent Crime Rate
Putnam’s Index	r	−.65**	−.64**
	N	48	48
Kim et al.’s Scale 1	r	−.47**	−.38**
	N	49	49
Kim et al.’s Scale 2	r	−.57**	−.60**
	N	49	49
ANHCS Social Capital Index	r	−.52**	−.27+
	N	51	51
Petris Social Capital Index	r	−.13	−.06
	N	50	50
BRFSS Measure	r	−.51**	−.43**
	N	51	51

Note:

⁺p < .10;

* p < .05;

^**p < .01.

Convergent validity

As can be seen in Table 4, except for PSCI, other indices were positively correlated with one another. Although PSCI was weakly associated with Putnam index, the correlation was not statistically significant. In addition, PSCI was weakly correlated with one dimension of Kim et al.’s scale.

Table 4.

Pearson Correlation Coefficient Matrix for State-Level Social Capital Measures

		(1)	(2)	(3)	(4)	(5)
(1) Putnam’s Index	r	-	-	-	-	-
	N	-	-	-	-	-
(2) Kim et al.’s Scale 1	r	.82**	-	-	-	-
	N	48	-	-	-	-
(3) Kim et al.’s Scale 2	r	.76**	.51**	-	-	-
	N	48	49	-	-	-
(4) ANHCS Social Capital Index	r	.41**	.42**	.44**	-	-
	N	48	49	49	-	-
(5) Petris Social Capital Index	r	.24	.01	.29*	−.04	-
	N	48	48	48	50	-
(6) BRFSS Measure	r	.57**	.36*	.65**	.39**	.32*
	N	48	49	49	51	50

Note:

^*p < .05;

^**p < .01.

Validating County-Level Social Capital Measures

Face validity

All three county-level measures were somewhat limited in terms of their face validity. As in the case of state-level PSCI, county-level PSCI may include organizations in which no direct face-to-face interactions occur among members. The RGFI includes some proxy measures of social capital, such as the county-level response rate to the Census Bureau’s decennial census and the percentage of voters who voted in presidential elections. These items may be interpreted as the outcomes of social capital rather than social capital itself. Like the state-level ANHCS and BRFSS measures, county-level BRFSS measure has the same limitations from relying solely on survey data, such as reporting bias.

Content validity

All three measures were limited in that they captured only one dimension of social capital: The BRFSS measure does not tap the structural dimension of social capital, whereas the RGFI and PSCI do not include the cognitive dimension of social capital. However, RGFI is more inclusive than PSCI. This is because the RGFI encompasses items tapping some informal organizations, like golf clubs, fitness centers, and sports organizations, whereas the PSCI measure only includes formal group activities.

Nomological validity

The RGFI and BRFSS measures demonstrated expected patterns of relations with the Gini coefficient and violent crime rate. In contrast, PSCI was not correlated with the Gini coefficient. In addition, PSCI was positively related to the violent crime, which was contrary to expectation. Therefore, PSCI is problematic in terms of nomological validity.

Convergent validity

The RGFI was positively correlated with the PSCI (r = .23, N = 3,085) and the BRFSS measure (r = .36, N = 2,263). The PSCI and BRFSS measures were also positively correlated with each other, although only weakly (r = .09, N = 2,287). Thus, PSCI’s convergent validity seems to be limited (table not shown here).

Predicting Health Outcomes

At the state level, Putnam’s and Kim et al.’s measures showed moderate to strong associations with health outcomes (see Table 6). Except for cancer deaths, Putnam’s index demonstrated expected relationships with health outcomes. Across all health outcomes, at least one scale of Kim et al.’s measures predicted population health. Likewise, the ANHCS index was associated with all health outcomes except for premature death, although the magnitude of the associations was smaller than those for the Putnam’s and Kim et al.’s indices.

Table 6.

Coefficient Estimates from Multivariable Linear Regression Models of State-Level Social Capital as Predictors of Population Health

		Putnam	Kim et al.	1 Kim et al. 2	ANHCS	PSCI	BRFSS
Premature Death	β	−.37**	−.11	−.18*	−.02	−.18*	−.19*
	N	48	49	49	51	50	51
Average Poor Physical Health Days	β	−.79**	−.47**	−.35**	−.30*	−.22+	−.27*
	N	48	49	49	51	50	51
Average Poor Mental Health Days	β	−.90**	−.63**	−.30*	−.20	−.30*	−.29*
	N	48	49	49	51	50	51
Average Self-Rated Health	β	−.79**	−.41**	−.43**	−.22+	−.30*	−.44**
	N	48	49	49	51	50	51
Age-adjusted Cardiovascular	β	−.40**	−.48**	−.11	−.28**	−.05	−.13
Mortality Rates	N	48	49	49	51	50	51
Age-adjusted Cancer	β	−.14	−.39*	.12	−.37**	.02	.06
Mortality Rates	N	48	49	49	51	50	51

Notes: (1)

⁺p < .10;

^*p < .05;

^**p < .01.

(2) PSCI refers to Petris Social Capital Index. (3) Premature death refers to the loss of years of productive life due to death before age 75. (4) Cell entries refer to standardized regression coefficients after controlling for median household income and the percentage of African Americans. (5) When calculating regression coefficients, two scales of Kim et al. were entered in the same regression model.

The patterns of associations between the PSCI and BRFSS measures with health outcomes were similar. Both measures were weakly associated with premature death and poor physical health days, moderately associated with poor mental health days and self-rated health, and unassociated with cardiovascular and cancer mortality rates.

County-level associations are provided in Table 7. The RGFI and BRFSS measures showed stronger associations with all health outcomes compared with the PSCI. The RGFI and BRFSS measures were weakly to moderately related to all health outcomes. In contrast, the PSCI was very weakly related to premature death, physical health, and self-rated health. Also, the PSCI was not related to mental health.

Table 7.

Coefficient Estimates from Multivariable Linear Regression Models of County-Level Social Capital as Predictors of Population Health

		PSCI	RGFI	BRFSS
Premature Death	β	−.09**	−.22**	−.24**
	N	3038	3016	2254
Average Poor Physical Health Days	β	−.11**	−.43**	−.28**
	N	2926	2909	2292
Average Poor Mental Health Days	β	−.04*	−.40**	−.32**
	N	2929	2912	2292
Average Self-Rated Health	β	−.12**	−.47**	−.37**
	N	2727	2707	2257

Notes: (1)

^*p < .05;

^**p < .01.

(2) RGFI refers to Rupasingha, Goetz, & Freshwater Index. (3) PSCI refers to Petris Social Capital Index. (4) Premature death refers to the loss of years of productive life due to death before age 75. (5) Cell entries represent standardized regression coefficients after controlling for median household income and the percentage of African Americans.

DISCUSSION

Social capital can be adequately utilized and developed into a scientific concept only to the extent that its conceptual and operational definitions are clear (Portes, 1998; Putnam, 2004). This study contributes to the knowledge in this area by evaluating the validity of generally available social capital measures, and carefully discussing measurement issues revolving around the concept of social capital. Moreover, we demonstrated that there are some differences in the size of associations between social capital and health outcomes across different measures of social capital, which supports the assumption that mixed findings in this area of research is partly because scholars have variably defined and operationalized social capital.

Study Limitations

Before discussing the implications of our findings, several limitations of this study should be mentioned. First, although we discussed three types of social capital measures (i.e., aggregate, integral, hybrid), we examined the validity of examples from each category. Thus, with this study, one cannot reach a definitive conclusion about which measurement approach should be taken. Second, recent studies have divided social capital according to other dimensions, such as bridging, bonding, and linking social capital (Szreter & Woolcock, 2004). Because most social capital indices based on publicly available data have not adopted this typology, we could not discuss the validity of social capital measures according to these dimensions. Third, this study focused only on survey data and official statistics. However, there are other methods, such as field observation and creative use of administrative data, which have been used to construct measures of social capital (e.g., Sampson & Raudenbush, 1999). Fourth, this study discussed measures of social capital only at the county and state levels. There are large variations in terms of which geographic units scholars pay attention to in measuring social capital and assessing its effects (Islam et al., 2006; Kawachi et al., 2008). A number of studies have been conducted at smaller areas, such as zip codes, census tract, and census block groups, which were not addressed in this study (e.g., Sampson et al., 1997). Finally, we provided only cross-sectional, ecological analyses in testing the associations between social capital measures and population health outcomes. Thus, causal order between social capital and health outcomes cannot be confirmed in this study. Also, because ecological analyses cannot separate contextual effects from compositional effects and do not allow for inferences about effects of collective social capital on individual-level health (Kawachi et al, 2008; Schwartz, 1994), future research should adopt validated social capital measures and conduct multilevel analyses.

A Comparative Look at the Validity of Social Capital Measures

Despite these limitations, this study provides the first comparative look at the validity of several key measures of social capital and their analytical utility in predicting health. Several interesting findings should be highlighted in this context. First, it appears that both Putnam’s (2000) and Kim et al.’s (2006) indices are advantageous across all validity tests. Nearly all items comprising these indices (except the voter turn-out rate, arguably) appropriately reflect the conceptual definition of social capital. They also comprehensively capture diverse aspects of social capital by including informal voluntary activities and the cognitive dimension of social capital. Furthermore, they showed good evidence of nomological validity, and were strongly correlated with other measures of social capital. In addition, our results indicate that these measures were significantly associated with state-level health outcomes, even after controlling for state-level income and the percentage of African Americans.

Second, the PSCI has both advantages and disadvantages. The PSCI is somewhat limited because it covers only a partial aspect of the structural dimension of social capital, i.e., more organized, formal group activities. Also, the PSCI does not show expected relationships with other constructs that can be regarded as predictors or outcomes of social capital. In addition, the PSCI was not closely related to other measures of social capital. Therefore, if the concerns of nomological validity, content validity, and convergent validity loom large, Putnam’s (2000) and Kim et al.’s (2006) indices offer some important advantages to using the PSCI at the state level. At the county level, the RGFI and BRFSS measure could be very useful alternatives to the PSCI. However, the PSCI is favourable in that it can be created at any geographic units in the United States, from zip code to higher levels such as county, MSA, region, and state (Brown et al., 2006). This flexibility is advantageous, considering that social capital processes can be better understood when explored at multiple levels simultaneously (Kawachi et al., 2008). Yet, because of little evidence for the validity of the PSCI and its relatively weak and unstable associations with overall health outcomes, scholars may want to triangulate their findings by simultaneously using other measures of social capital.

Third, the aggregate index based on the ANHCS data seems to be valid based on its associations with income inequality and violent crime rate, as well as its strong positive correlations with other measures of social capital at the state level. Even though its associations with health outcomes were weaker than those of Putnam’s and Kim et al.’s measures, the ANHCS index performs very well in predicting cardiovascular deaths and cancer deaths. However, researchers should keep in mind the inherent problems of aggregate measures of ecological characteristics and the limited scope of the ANHCS index. That is, because the ANHCS index might risk providing biased estimates for the effects of state-level social capital, and because it captures only formal group activities, researchers may need to employ other state-level measures of social capital and try to examine whether these additional measures produce similar results as the ones yielded by the ANHCS index.

Fourth, the aggregate measure derived from the BRFSS data showed some evidence of validity at both the state and county levels. The BRFSS measures were strong in terms of construct and convergent validity. Also, the BRFSS measure performed very well in predicting county-level health outcomes. In addition, at the state level, the PSCI’s and BRFSS measures’ patterns of associations with health outcomes were essentially the same across all health outcomes employed in this paper. Considering that the PSCI is somewhat limited in terms of validity and that the BRFSS measure and PSCI capture different dimensions of social capital (cognitive and structural dimensions, respectively), the BRFSS measure can be a practical alternative or complement to PSCI. Of course, researchers should consider the inherent limitations of the BRFSS measures as aggregate measures. Also, because the BRFSS measures do not encompass a structural dimension of social capital, other more comprehensive social capital indices should ideally be used together.

Finally, this study provides compelling evidence that the RGFI is a very useful, valid tool for researchers interested in social capital processes at the U.S. county level. Moreover, it was found that the RGFI was associated with county-level health outcomes as expected. However, if effects of the cognitive dimension of social capital on health outcomes are the focus of a study, the RGFI is not the proper index to use. Because the cognitive dimension of social capital is hard to capture by such integral measures as RGFI, researchers should come up with creative methods to tap social trust and reciprocity other than through prohibitively expensive local surveys. Galassi (2001), for example, showed that it is possible to measure social trust by using official statistics, such as Italian co-operative membership.

Practical Guidelines for Aggregate Measures of Social Capital

While we have pointed out many limitations of aggregate measures of social capital throughout the paper, we are not arguing that aggregate indices should be avoided in constructing social capital measures and exploring the effects of collective social capital on health outcomes. As we mentioned earlier, both integral and aggregate measures are needed to study social capital effects. From a practical standpoint, researchers sometimes have no choice but to rely solely on aggregate indices especially when they use existing large-scale survey data, such as ANHCS, BRFSS, and GSS, to measure collective social capital. In light of this, a few guidelines may help researchers redress some of the potential problems with aggregate measures of social capital.

First, when testing the effects of aggregate social capital measures on any health outcomes, researchers should control for individual-level social capital-related items. That is, researchers should try to show that collective social capital has effects over and above the individual-level items that were used to construct collective social capital. This is because of the possibility that the observed association between aggregate social capital and population health outcomes may be confounded by individual-level factors, including demographic and socioeconomic variables. Analytically, multilevel models are useful in enabling modeling of both aggregate social capital measures and individual-level factors, thereby disentangling the contextual effects of social capital from compositional effects.

Also, it is a convention that scholars calculate the mean of raw scores of survey respondents’ reports of social capital-related items or derive the proportions of those with high levels of social trust or formal group activities to construct aggregate measures of social capital. However, a few scholars (e.g., Kawachi et al., 2008; Subramanian, Lochner, & Kawachi, 2003) have contended that the differences in aggregate social capital among collectives obtained in such a way might be the artifact of the characteristics of residents who make up the collectives. Therefore, they have suggested that aggregate measures should be constructed after adjusting for individual-level compositional factors by employing multilevel models (“predicted” or “posterior” residuals; for details, see Subramanian et al., 2003). This permits a less confounded score that can arguably be regarded as a more faithful representation of collective social capital. However, because residents’ characteristics may be largely influenced by the social capital of the community in which they live, researchers should provide strong theoretical and empirical rationales for separating collective social capital from certain individual-level factors to adopt this approach (Macintyre & Ellaway, 2000).

Moreover, to obtain valid and reliable measures of aggregate social capital, researchers should achieve a large sample size for each ecological unit. Researchers should first establish reliability and validity of their measures from the perspective of psychometrics if they aggregate individuals’ responses to a group level. The quality of measurement for ecological units, however, requires more than just statistical procedures associated with psychometrics. The reliability from the perspective of ecometrics depends on the degree of inter-subjective agreement among community members and the number of community members sampled as well as item consistency and the number of items (Raudenbush, 2003, p. 115). Even if researchers detect large random variation among the responses of residents within the community, sampling a large number of respondents per each community can overcome such a problem. If existing survey data provide researchers with only a small number of respondents for some ecological units, those units should be dropped from the analyses. In that case, researchers should explicitly mention that their findings have limited generalizability.

In addition, as a practical strategy to obtain a large sample size for each ecological unit, many researchers combine cross-sectional survey datasets employing the same social capital measures over a long period of time (e.g., across years). This practice is based on the assumption that social capital-related measures tend to remain constant over time. However, this assumption has rarely been empirically tested. Therefore, researchers should justify their decisions to pool survey data over time by showing that social capital-related survey measures do not change significantly over time.

Finally, although large-scale survey data are typically intended to be nationally representative, this does not necessarily mean that each respondent is representative of his or her residential area. Thus, aggregate measures of social capital derived from non-representative samples may limit generalizability. To redress this limitation and generate more generalizable data at more local levels, researchers may consider developing post-stratification weights (see Kawachi, Kennedy, Lochner, & Prothrow-Stith, 1997, p. 1492). For example, using demographic data (e.g., age, gender, race/ethnicity, education, income) from the U.S. census for each state and county, researchers can develop post-stratification weights to account for oversampling, differences in response rates by population groups, and to better generalize their results to state and county populations.

Conclusions

This study validated a number of available social capital measures at the U.S. state and county levels, and then examined the relative extent to which the measures of social capital are associated with population health outcomes. These measures, except for the PSCI, show some evidence of validity. Moreover, we see notable differences across social capital measures in their associations with population health outcomes. These findings clearly illustrate the importance of measurement issues related to social capital in public health research. By showing that both conceptually valid and practically robust measures of social capital exist at the U.S. state and county levels, this study may usefully guide researchers’ future endeavours to explore the contextual effects of collective social capital on health.

Table 5.

Pearson Correlation Coefficients for Associations between County-Level Social Capital Measures with Income Inequality and Crime

		Gini Coefficient	Violent Crime Rate
Petris Social Capital Index	r	−.04*	.20**
	N	3117	2937
Rupasingha, Goetz, & Freshwater Index	r	−.24**	−.27**
	N	3085	2950
BRFSS Measure	r	−.39**	−.33**
	N	2286	2185

Note:

^*p < .05;

^**p < .01.