ZIP Code-Level Estimates from a Local Health Survey: Added Value and Limitations

Abstract

We assessed the added value and limitations of generating directly estimated ZIP Code-level estimates by aggregating 5 years of data from an annual cross-sectional survey, the New York City Community Health Survey (n = 44,886) from 2009 to 2013, that were designed to provide reliable estimates only of larger geographies. Survey weights generated directly-observed ZIP Code (n = 128) level estimates. We assessed the heterogeneity of ZIP Code-level estimates within coarser United Hospital Fund (UHF) neighborhood areas (n = 34) by using the Rao-Scott Chi-Square test and one-way ANOVA. Orthogonal linear contrasts assessed whether there were linear trends at the UHF level from 2009 to 2013. 22 of 37 health indicators were reliable in over 50% of ZIP Codes. 14 of the 22 variables showed heterogeneity in ≥4 UHFs. Variables for drinking, nutrition, and HIV testing showed heterogeneity in the most UHFs (9–24 UHFs). In half of the 32 UHFs, >20% variables had within-UHF heterogeneity. Flu vaccination and sugary beverage consumption showed significant time trends in the largest number of UHFs (12 or more UHFs). Overall, heterogeneity of ZIP Code-level estimates suggests that there is value in aggregating 5 years of data to make direct small area estimates.

Electronic supplementary material

The online version of this article (10.1007/s11524-020-00423-z) contains supplementary material, which is available to authorized users.

Introduction

The NYC Community Health Survey (CHS) is an annual survey that has provided key health surveillance data to the New York City Department of Health and Mental Hygiene (DOHMH) since 2002. The survey covers approximately 10,000 residents across New York City each year. From the inception of the survey, the annual data were designed to be analyzed at the level of the United Hospital Fund (UHF) neighborhoods, which are 42 geographic areas comprised of aggregated ZIP Codes. Survey data collection at the UHF level is appealing because survey respondents generally know the ZIP Code that they live in, and therefore survey data can be aggregated into neighborhoods comprised of multiple ZIP Codes that have enough respondents to generate reliable estimates. The neighborhood level data have enabled the Health Department in the ensuing years to identify unique health challenges within neighborhoods [1] and highlight disparities between neighborhoods [2–4]. The data have provided the basis for various targeted interventions [5]. Although collecting data at the level of 42 neighborhoods is valuable, there may be substantial variation in health conditions within neighborhoods that cannot be captured at the coarser neighborhood level when the neighborhoods used for analysis have an average of 200,000 residential adults. Several data gathering projects [6, 7] have made health estimates at the level as small as census tracts, as public health professionals seek information about increasingly smaller areas. Understanding areas of highest need is a key goal of public health, and estimates at smaller geographies, or of subpopulations, are needed to provide this information so that interventions can be appropriately targeted.

Survey designs require sample sizes to be sufficiently large to yield direct estimates that are statistically reliable in small areas or subpopulations. For example, CDC recommends not reporting direct estimates based on a sample size of fewer than 50 people in the nationwide Behavioral Risk Factor Surveillance System (BRFSS) [8]. For this reason, NYC DOHMH has not produced ZIP Code-level direct estimates of single years of data. The challenge of unreliable estimates could be resolved with a larger sample sizes, however, increasing sample sizes in small areas is costly. One approach to making small area estimations without expanding sample size is to apply indirect estimators that use auxiliary population data about the variable of interest from related areas and/or time periods [9]. Indirect estimators have been applied to obtain estimates at various geographic units [10–14] using data sources such as BRFSS and the California Health Interview Survey (CHIS). However, there are limitations in using these data for program evaluation or to evaluate changes over time [15].

New York City Department of Health and Mental Hygiene (DOHMH) aggregated CHS data over 5 years (2009–2013) and applied survey weights to directly observed estimates at the ZIP Code level. Direct estimation by aggregation takes advantage of the continuous data collection of the CHS, and is similar to the approach taken by the American Community Survey (ACS) to create multi-year estimates [16, 17]. Without statistical modeling, this direct estimator is cost-efficient, easy to implement, and easy to interpret, though at the expense of sacrificing temporality of the data. This study aims to understand the tradeoff of gaining spatial granularity at the ZIP Code level while losing the temporal granularity. To assess how much additional information ZIP Code-level estimates present in comparison to the coarser UHF level estimates under the context of CHS, we assessed the heterogeneity of five-year aggregated estimates at the ZIP Code level within 34 larger UHF neighborhoods. We also indirectly assessed whether combining 5 years of data may have masked changes over time.

Methods

Study Design

The CHS is a cross-sectional telephone survey that has been implemented annually since 2002 by the New York City Department of Health and Mental Hygiene (DOHMH). It was conducted via landline telephone until 2009, when cellular telephones were added to the sample frame. Live interviewers implement the questionnaire in English, Spanish, Russian, Mandarin, and Cantonese. The survey is designed to produce estimates at the UHF, borough, and city levels. For single years of data, the survey can provide generally statistically reliable estimates at the level of 34 UHF neighborhoods; with two or more years of data, it can provide estimates at the level of all 42 UHF neighborhoods [18]. Adults in group quarters or institutional housing are not included, nor are adults who cannot be reached by either a landline or cellular telephone. Cooperation rates during 2009–2013 ranged from 86.6% to 89.4%, and response rate ranged from 37.7% to 40.0% [18]. The study was approved by the New York City Department of Health and Mental Hygiene Institutional Review Board.

Since the single year data at the ZIP Code-level have insufficient records for producing reliable estimates, data from 2009 to 2013 were pooled to create ZIP Code estimates. The pooled 2009–2013 ZIP Code dataset contains a total of 44,886 household-dwelling adult respondents aged 18 and older living in Brooklyn, the Bronx, Manhattan, Queens, or Staten Island. Respondents in the 56 ZIP Codes with fewer than 30,000 residents according to the 2010 Census were combined with other (generally adjacent) ZIP Codes within the same UHF. ZIP Codes with fewer than 30,000 residents have an average population size of 18,000 ranging from approximately 3000 to 30,000 residents. After the combination of ZIP Codes, there were 128 modified ZIP Code groups that had an average population size of 64,000, ranging from approximately 30,000 to 110,000 residents. Weights were applied that incorporated data from the 2010 Census (race/ethnicity, and age), the 2009–2013 American Community Survey (ACS) (marital status, education, number of adults in the household, presence of children in the household), and the 2008 and 2011 New York City Housing and Vacancy Survey (the distribution of adults accessible only by landline telephone, only by cellular telephone, or by both) [19].

Study Variables

The pooled 2009–2013 CHS ZIP Code dataset included 37 measures asked in all 5 years falling within 13 health topics including general health status, healthcare access, cardiovascular health, diabetes, asthma, nutrition, physical activity, smoking, immunizations, cancer screening, HIV, sexual behavior, and alcohol consumption (Table S1). We defined reliable estimates as having a Relative Standard Error less than 30%, a 95% confidence interval (CI) half-width less than or equal to 10%, and a sample size greater than or equal to 50, which is a standard definition used by NYC DOHMH and CDC for survey reporting [20, 21]. Analysis was limited to questions that met reliability criteria in at least 50% of the 128 ZIP Code groups. Binary variables were considered reliable when estimates from both categories (e.g., “yes”, and “no”) were reliable. We dichotomized categorical variables with more than two levels to avoid only some levels showing reliable estimates.

Statistical Methods

We used the Rao-Scott Chi-Square test to assess the equality of crude ZIP Code-level prevalence estimates within each UHF. We used one-way ANOVA to assess the equality of means of continuous variables. We rejected the null hypotheses at a significance level of α = 0.05. Continuous variables that strongly violated the normality assumption according to visual assessments of the residual plots were log-transformed. We performed one-way ANOVA using proc surveyreg, and performed second order (Satterthwaite) Rao-Scott Chi-Square analyses [22–24] for categorical variables using proc surveyfreq in SAS.

Due to a limited sample size within each ZIP Code group, we did not try to determine if yearly ZIP Code-level estimates were changing over time. As a proxy for change that might have occurred, we assessed whether yearly prevalence estimates and means of continuous variables from 2009 to 2013 showed significant linear trends within each of 34 UHFs by constructing orthogonal linear contrasts. We rejected the null hypothesis that a contrast was equal to zero (i.e., no linear trend) at the significance level of α = 0.05. For variables where we found linear trends within UHFs, pooling years of data to develop estimates at the ZIP Code level was likely masking changes over time. All analyzes were conducted using SUDAAN 9 (RTI, Research Triangle Park, NC) and SAS EG 7.1 (SAS Institute, Cary, NC). Figures and maps were produced in R.

Results

The pooled 2009–2013 CHS ZIP Code-level dataset had a sample size of 44,886, and an average sample size of 345 per ZIP Code group (IQR = 257, 425; range = 148, 744). In comparison, the average sample size per UHF neighborhood in the yearly CHS dataset from 2009 through 2013 ranged from 250 (IQR = 183, 280; range = 101, 736) in 2009 to 291(IQR = 238, 308; range = 127, 629) in 2013. Each UHF neighborhood contained on average 3.8 ZIP Code groups (IQR = 3, 5; range = 1, 9). The average survey design effect was 2.49 (IQR = 1.89, 2.62) for the 34 UHFs and 2.41 (IQR = 2.02, 2.65) for the ZIP Code groups.

Three continuous variables and 19 binary variables across 13 health topics met reliability criteria in at least 50% of the 128 ZIP Code groups, so they were included in the subsequent analyses (Table S2). ZIP Code-level estimates using 5 years of data were more consistently reliable (at the ZIP Code level) than were UHF estimates using a single year of CHS data (e.g., 2011) based on the reliability criteria used by NYC DOHMH for survey reporting; for each of these variables, the percentages of UHFs meeting reliable criteria in at least 50% of the 34 UHFs in the yearly CHS survey were lower than the percentages of ZIP Code groups meeting the same criteria in the 2009–2013 CHS ZIP Code-level dataset (Table S2).

For each UHF, we calculated the proportion of variables whose ZIP Code-level estimates were significantly different from each other (Table S3, Fig. 1). In half of the 32 UHFs, >20% variables had within-UHF heterogeneity. In the Union Square-Lower Manhattan area, where the most variables showed ZIP Code-level heterogeneity (12/22 = 54.5%), the heterogeneous variables covered the health topics of alcohol consumption, diabetes, cancer screening, physical activity, sexual behavior, general health status, and cardiovascular health. Other UHFs showing within-UHF heterogeneity for 8 (36.4%) or more of the 22 variables include Ridgewood, Pelham-Throgs Neck, and Flushing. In East Harlem and Washington Heights, where the fewest variables had ZIP Code-level heterogeneity, only asthma prevalence in East Harlem and vegetable/fruits consumption in Washington Heights had significant differences between ZIP Code groups (Table S3, Fig. 1). We could not assess heterogeneity of variables in two UHFs - Kingsbridge and Sunset Park - because the ZIP Codes in the two UHF neighborhoods had fewer than 30,000 residents according to the 2010 Census, and thus were combined to a single group.

Fig. 1.

A UHF neighborhood level map showing the percentage of variables whose ZIP Code-level estimates within the same UHF were statistically different from each other. Heterogeneity of variables in two UHFs - Kingsbridge and Sunset Park – could not be assessed because the ZIP Codes in the two UHF neighborhoods had fewer than 30,000 residents according to the 2010 Census, and thus were combined with each other to make a single grouped estimate. Source: NYC Community Health Survey, 2009–2013. 2009–2013 data were weighted to the NYC adult residential population as per the American Community Survey, 2009–2013

In addition, for each variable, we calculated the percentage of UHFs that had heterogeneous ZIP Code estimates within them (Table S2, Fig. 2). HIV testing, as well as questions related to fruits and vegetable consumption, health insurance coverage, cycling, and alcohol consumption, showed heterogeneity of ZIP Code-level estimates within the same UHFs in at least one quarter of the 32 UHF neighborhoods that had more than one ZIP Code group (Fig. S1, Fig. S2). 14 of the 22 reliable variables showed heterogeneity in at least 4 UHFs.

Fig. 2.

A UHF neighborhood level map showing the proportion of variables showing significant linear trends over time. Source: NYC Community Health Survey, 2009, 2010, 2011, 2013, and 2013

For each UHF neighborhood, we calculated the number of variables whose yearly data from 2009 to 2013 showed significant linear trends. None of the 22 variables showed significant linear trends from 2009 to 2013 in West Queens, Chelsea Village, or Ridgewood, while 6 (27.3%) showed significant linear trends in Flatbush and Pelham-Throgs Neck, followed by 5 (22.7%) in Southern Staten Island (Table S3, Fig. 2). Two variables related to flu vaccination history showed significant linear trends in over 40% of UHFs (41.2% for flu vaccination coverage in the past year, and 44.1% for flu vaccination coverage in the past flu season) (Table S2). Sugary beverage consumption and percentage reporting exercise in the past 30 days showed a significant linear trend in more than 20% of UHFs.

Discussion

Five-year aggregated ZIP Code level estimates showed heterogeneity across UHF neighborhoods in NYC for health outcomes examined in this study. As a result, producing estimates for health outcomes at smaller geographies like ZIP Code helps capture differences that are obscured at the UHF level. UHFs are large geographies, and understanding variation within them can help target interventions. For example, placement of stations for NYC’s bike-share program, Citi Bike, would likely be better guided by health estimates at the ZIP Code level than by UHF.

Since single year data at the ZIP Code level were insufficient to produce reliable estimates, we aggregated data over time, pooling annual data collected at a less granular level to create ZIP Code-level estimates that had sufficient records. This approach presents unique benefits in creating estimates for health indicators at fine spatial scales. It is cost-efficient, easy to implement, and scalable, showing potential for meeting the high demand for unbiased estimates at a finer geographic scale. ZIP Code weights were also similar in efficiency to single year UHF weights, as evidenced by the design effects due to unequal weighting. An alternative method is to create estimates from models, which could produce statistically reliable estimates at the ZIP Code-level without combining years of data, enabling analysts to obtain ZIP Code-specific estimates and observe change over time [25]. Although powerful, the model-based approach has several drawbacks compared to aggregation. Models often involve additional assumptions to create unbiased estimates. They are generally more complex, with separate equations for specific measures, requiring additional statistical resources. Modeled estimates are also more difficult to calibrate to multiple geographies for a single year, e.g., estimates for a county overall may differ from the combined estimates of its constituent ZIP Codes. For these reasons, the use of model-based estimators may be less desirable for certain scenarios.

Both model-based indirect estimators and our direct estimators highlight challenges in evaluating changes over time for large scale community surveys. The indirect estimates in the CDC’s 500 Cities project are based on expected prevalence rather than on actual samples, so they can be inappropriate for measuring changes over time [12, 26]. With our direct estimators, rolling ZIP Code-level multi-year estimates can be used to infer ZIP Code-level trends once they become available. However, the limitation of this approach lies in its inability to detect abrupt changes. When the impact of intervention programs is believed to be gradual, rolling multi-year estimates are particularly relevant as they accentuate long-term trends by smoothing out short-term fluctuations that can be the results of sampling variation [27].

A clear hypothesis of the timing of the effect of local intervention is important when deciding whether using multi-year smoothed direct estimates is appropriate for evaluating trends over time. Detailed discussions of this methodology can be found in the context of the ACS, which has been producing 5-year estimates since 1999 [28]. Here, we examined secular trends at the UHF level as a proxy for changes at the ZIP Code level. Of note, trends within ZIP Codes may still exist when there is an absence of a significant linear trend at the UHF level. Additionally, analysis at the ZIP code level may fail to detect meaningful geographic variation observed at the level of Census block group or tract [29].

In conclusion, aggregating multiple years of data to directly obtain ZIP Code-level estimates is a simple and cost-efficient method for producing small area estimates for regularly collected community surveys, and is generalizable to large community surveys, albeit with the loss of temporal granularity. Significant heterogeneity of estimates within UHFs suggests the added value of releasing multi-year ZIP Code-level estimates. More work is needed to identify trends over time using rolling multi-year estimates.