The Use of Small Area Estimates in Place-Based Health Research

Abstract

Interest in the impact of the built environment on health behaviors, outcomes, and disparities is increasing, and the growing development of statistical modeling techniques has allowed researchers to better investigate these relationships. However, without enough data that are identifiable at smaller geographic levels (e.g., census tract), place-based health researchers are unable to reliably estimate the prevalence of a health outcome at these more granular and potentially more salient neighborhood levels.

When reliable direct survey estimates cannot be produced because of small samples or a lack of samples, estimates based on small area estimation techniques are often used. As place-based health research and the production and secondary use of small area estimates increase, it is critical that researchers understand both the underlying methods used to create these estimates and their limitations. Without this foundation, researchers may fit inappropriate models, or interpret findings inaccurately.

As a demonstrative example, we focus this discussion on the small area health indicator estimates recently produced through the 500 Cities Project by the Robert Wood Johnson Foundation, the Centers for Disease Control and Prevention (CDC), and the CDC Foundation.

Interest by both researchers and communities in the impact of neighborhood characteristics on health behaviors and outcomes is flourishing. Increased interest in health inequities, policies regulating the built environment, and the dissemination of geospatial technologies and data has greatly contributed to this trend.¹ Furthermore, increasing statistical computing power and the growing popularity and feasibility of implementing multilevel statistical models have allowed researchers to more accurately determine whether compositional or contextual variables are associated with health behaviors and outcomes.² Generally, compositional effects represent those that are due to variation in characteristics of individuals, whereas contextual effects represent those that are due to variation in the neighborhood environment that individuals may reside in.^2,3 For example, a researcher may be interested in investigating whether individual-level factors or built environment factors are driving physical activity levels. One could fit multilevel models to examine whether an individual’s self-efficacy to exercise (compositional variable) or their accessibility to parks (contextual variable) is predictive of physical activity levels. Where individual-level data are not available, researchers often rely on aggregate group-level data to test associations between group- or area-level variables.²

A major difficulty of place-based health research is defining a salient neighborhood to the priority population or health phenomena, as well as accessing data that include geoidentifiers at the salient neighborhood level.¹ To date, administratively defined areas, such as census block groups, census tracts, and zip codes, are commonly used to operationalize neighborhoods.¹ However, without enough data that are identifiable at smaller geographic scales, place-based health researchers are unable to reliably estimate the prevalence of an outcome at more granular neighborhood levels.^4–6 Furthermore, local jurisdictions with limited resources to conduct community health assessments may only have access to estimates of health indicators at larger geographic levels (e.g., state, county) that are not relevant to or representative of what may be occurring in their local neighborhoods.

Small area estimates represent those that are indirectly estimated for a geographic area for which direct survey estimates cannot produce valid and reliable estimates because of factors such as low response rate, limited sample sizes, or a lack of samples.^5,6 Small area estimates can be a useful indicator for local health behavior and disease surveillance purposes, and they have also been used in secondary analyses to identify potential community-level factors that may be associated with health behaviors or disease prevalence.^7–9 Additionally, small area estimates are more precise than those provided by direct survey estimates.^5,6,10,11 As neighborhood and health research continues to grow, and the release and use of small area estimates increases, it is critical that researchers understand both the underlying methods used to create these estimates and their limitations.

The purpose of this commentary is to provide an overview of the methods used to generate small area estimates and to discuss the limitations of their use in answering research questions. Although there are several methods to create small area estimates, most have similar procedures requiring the combination of individual-level data, area group composition population estimates, and multilevel mixed effects regression techniques. As a demonstrative example, we focus our discussion, which may broadly be applied to other small area estimates, to those recently produced through the 500 Cities Project.

PRODUCING 500 CITIES SMALL AREA ESTIMATES

Through a partnership between the Robert Wood Johnson Foundation (RWJF), the Centers for Disease Control and Prevention (CDC), and the CDC Foundation, this publicly available data set includes 27 chronic disease health behavior and health outcome small area estimates.¹² Researchers produced city- and census tract–level estimates for 500 cities, including the largest 497 US cities and three additional cities, to ensure representation from all states.¹² Together, these cities represent 33.4% of the total US population.¹² The 500 Cities Project (500 Cities) is innovative because of its national urban representation and inclusion of an array of health indicator prevalence estimates at small geographic area levels. The data set is intended to provide descriptive estimates of health indicators in an effort to supplement existing surveillance data, help local jurisdictions expand community engagement to identify potential health issues, and establish key health objectives.¹²

Researchers created the small area estimates from 500 Cities by combining data from the CDC’s Behavioral Risk Factor Surveillance System (BRFSS)¹³ with Census Bureau and American Community Survey¹⁴ population estimates. The BRFSS is a national surveillance system that uses random digit dialing to administer a telephone survey (cellphone and landline) to collect demographic and behavioral health risk data for US adults in all 50 states, the District of Colombia, and three US territories.¹³ Nearly half a million adult BRFSS surveys are completed annually.¹³ Although data are collected at the individual level, only county-level indicators are included, and direct area prevalence estimates are available only at the state level and for some large metropolitan areas.¹³ Researchers used a multilevel regression and poststratification approach to generate estimates in 500 Cities and the census tracts within these cities.¹⁰ Details of this methodology are briefly described on the CDC Web site, which also refers readers to several journal articles that first used this method.^10,15,16 Here, we summarize this methodology with a focus on the conceptual interpretation of this approach to aid in understanding what these model estimates represent.

To produce small area model estimates in 500 Cities, researchers fit multilevel logistic models to national BRFSS data to determine an individual’s probability of having a health behavior or outcome as a function of several compositional sociodemographic attributes, including a person’s age, gender, and race/ethnicity.¹⁰ Models included county and census tract poverty estimates from the American Community Survey and county and state random effects. County and state random effects statistically account for the correlation of individuals living in the same county (or state). These random effects also represent unmeasured contextual factors that may affect an individual’s probability of having a health indicator. However, the impacts of measured contextual effects (other than area poverty) are not directly incorporated into model estimates. Other contextual variables were not included as predictors of model estimates because, although one contextual variable may be an important predictor for one geographic area, it may not be appropriate for another.¹⁰ For example, two cities may have a similarly high obesity prevalence. However, the factors driving the high prevalence may differ: perhaps lower accessibility to parks and recreation centers in one city and high caloric intake in the other city. Finally, to estimate city- and tract-level prevalence, the models were poststratified to their respective area group composition population estimates using 2010 Census demographic data.¹⁰ In other words, the produced 500 Cities small area estimates represent the expected prevalence estimate of a health indicator for census tracts and cities given the sociodemographic characteristics (age, gender, and race/ethnicity) of the individuals in the census tract or city, census tract–level poverty, and unmeasured contextual factors in the county and state. The produced values are therefore akin to synthetic estimates, and they do not represent the actual observed or direct prevalence in a census tract or city.^5,6 Finally, because a global statistical model is fit to create these model estimates, some estimates may be more valid for certain health indicators or geographic areas.

VALIDATING 500 CITIES SMALL AREA ESTIMATES

Prior studies comparing estimates produced using the 500 Cities methodology to the “gold standard” of direct survey estimates have found validity to vary by health indicator and geographic scale. In a national study, internal validation of county-level small area estimates of chronic obstructive pulmonary disease compared with BRFSS direct survey estimates indicated a high Pearson’s correlation of 0.88.¹⁰ In Missouri, Pearson’s correlations between BRFSS direct survey estimates and county-level small area estimates of chronic obstructive pulmonary disease, current smoking, diabetes, obesity, and uninsured status ranged from 0.731 to 0.960.¹⁵ External validation (i.e., comparing direct survey estimates to those not used to produce small area estimates) with locally collected data indicated correlations of 0.28 for obesity and no health insurance, 0.40 for cigarette smoking, 0.51 for diabetes, and 0.69 for chronic obstructive pulmonary disease.¹⁵ In a sample of cities in Massachusetts, external validation comparing city-level MDPHnet (an electronic health record–based surveillance platform) disease prevalence with 500 Cities small area estimates indicated Pearson’s correlations ranging from 0.65 (obesity prevalence) to 0.89 (diabetes prevalence).¹⁷ Finally, an external validation study comparing 500 Cities estimates to Boston BRFSS direct survey estimates indicated good to strong validity for several city-level factors (e.g., cigarette smoking, high blood pressure, diabetes), but not for others (e.g., binge drinking, obesity, mental distress, physical distress); additionally, validity was not as strong for subcity levels.¹⁶

Validation with small area estimates is not an easy task, as external sources (such as health surveys conducted by local jurisdictions) may also suffer from nonresponse, bias, and suppression. The validity of health indicators may also depend on the actual predictors entered into the multilevel prediction model. Additionally, at finer geographic scales, direct survey estimates may be less reliable and have greater bias, and it also may become more difficult to incorporate local variations in public health interventions and social, economic, and cultural contexts into small area estimates.¹⁶ In this way, it is difficult to know what the “true” or “valid” prevalence of a neighborhood health indicator is for comparison purposes. When interpreting and disseminating study results, researchers using small area estimates, and specifically those produced by 500 Cities, should be acutely aware that validation of these estimates may vary¹⁸ substantially by health indicator, geographic area, and geographic scale.

OTHER CONSIDERATIONS

With the release of this wealth of small area health data, several publications have appeared in which the authors use regression methods to conduct secondary analyses of the 500 Cities small area estimates. To date, researchers have used 500 Cities to investigate associations of gentrification⁷ and park quality⁸ with physical health, and to examine neighborhood-level income and racial disparities in the prevalence of smoking,¹⁹ obesity,⁹ and poor mental health.²⁰ Researchers conducting secondary regression analyses of small area estimates should be cognizant of several additional interpretation issues. First, while the county and state random effects model some county- and state-level variation in the estimates, those neighborhood-level or other local contextual factors that may have a large impact on a health indicator will not be directly accounted for in model estimates. For example, if a tobacco control policy is implemented in a city that has substantially reduced smoking prevalence, this policy effect will not be directly accounted for in the 500 Cities small area estimate, resulting in an area smoking prevalence estimate that may be biased upward.¹⁵ Furthermore, the fact that a tract or city has a high prevalence of a health indicator does not imply that there is something about the environment or context that is driving this prevalence estimate; rather, there may be a large number of individuals with a high probability of having the health indicator residing in this tract or city. Thus, researchers should be aware that small area estimates could potentially introduce substantial bias in the evaluation of the actual impact of contextual variables, such as local interventions and policies.^15,16

Second, the CDC states about 500 Cities, “The SAE [small area estimate] for each city is dependent mainly upon the sociodemographic characteristics of that city.”¹² Existing studies that statistically test associations between tract– or city- level demographic characteristics with small area estimate indicators should be aware that significant associations may be due to the poststratification estimation process that uses an area’s group composition of age, gender, and race/ethnicity data to produce model estimates. Regressing small area estimates on the same variables that were used to create them will result in associations that reflect the group compositional variables used to estimate the small area estimate. Likewise, regressing small area estimates on one another (e.g., physical activity with obesity) will also likely reflect significant associations. Although researchers may choose to include some area-level demographic characteristics that were used to generate the small area estimates as control variables, caution should be taken when interpreting these specific effects as meaningful.

Finally, researchers using small area estimates should be aware of error associated with these estimates, which are often not acknowledged. First, the BRFSS data itself may have measurement error or self-reported bias associated with it. Additionally, the model estimates produced in 500 Cities have varying margins of errors that differ substantially by health indicators, geographic areas, and geographic scale. This error may be especially large in neighborhoods with small population sizes and small county and state BRFSS samples. Although there are various statistical methodologies^21,22 that may be used to account for this measurement error, these methods can be complex, and researchers often do not account for this measurement error in their secondary analyses. Therefore, standard errors reported are more than likely biased, potentially increasing type I error. We urge researchers to examine and report the margins of error of 500 Cities health indicators and discuss the impact and limitations of their findings, especially if this error is not explicitly modeled in subsequent analyses.

CONCLUSIONS

500 Cities is an exciting, promising, and novel project that has spurred continued interest in understanding the relationship between neighborhood characteristics on health behaviors and outcomes. Although the estimates are so far reflective of 2010 population estimates, the CDC hopes to provide census tract– and city-level estimates using more recent intercensal population data.¹² The small area estimates generated by 500 Cities are intended to “allow cities and local health departments to better understand the burden and geographic distribution of health-related variables in their jurisdictions, and assist them in planning public health interventions.”¹² To support these efforts, New York University Lagone Health, in partnership with the RWJF, released an interactive Web-based resource known as the City Health Dashboard.²³ The City Health Dashboard includes measures and data visualization from 500 Cities, as well as several other indicators of socioeconomic, built environment, and health behavior and outcome factors from other surveillance systems (e.g., National Vital Statistics System), small area estimates (e.g., US Small Area Life Expectancy Estimates Project), and data sources (e.g., US Department of Agriculture Economic Research Service’s Food Access Research Atlas).²³ Small area estimates may provide preliminary data to communities about important health indicators and may also supplement local community health assessments and surveillance efforts, which may help jurisdictions to prioritize health needs, set community health goals, and allocate funding to local health policy and intervention strategies. As researchers use this data to statistically test hypotheses, they must also be mindful of how these estimates were produced when interpreting research findings, especially as localities and policymakers increasingly look to researchers for guidance and expertise.