Lessons Learned From the Environmental Public Health Tracking Sub-County Data Pilot Project

Abstract

Objective:

Small area data are key to better understanding the complex relationships between environmental health, health outcomes, and risk factors at a local level. In 2014, the Centers for Disease Control and Prevention’s National Environmental Public Health Tracking Program (Tracking Program) conducted the Sub-County Data Pilot Project with grantees to consider integration of sub-county data into the National Environmental Public Health Tracking Network (Tracking Network).

Design:

The Tracking Program and grantees developed sub-county-level data for several data sets during this pilot project, working to standardize processes for submitting data and creating required geographies. Grantees documented challenges they encountered during the pilot project and documented decisions.

Results:

This article covers the challenges revealed during the project. It includes insights into geocoding, aggregation, population estimates, and data stability and provides recommendations for moving forward.

Conclusion:

National standards for generating, analyzing, and sharing sub-county data should be established to build a system of sub-county data that allow for comparison of outcomes, geographies, and time. Increasing the availability and accessibility of small area data will not only enhance the Tracking Network’s capabilities but also contribute to an improved understanding of environmental health and informed decision making at a local level.

Small area data are essential to better understand complex environmental health problems at a local level, with higher-resolution data necessary for uncovering local rate variation. Smaller areas, unlike county level, generally preserve the geographic variability of data.¹ Small area analyses can increase our knowledge of local health issues. They can help inform public policy, have the potential to serve as surveillance tools, identify geographic areas where care can be improved, and provide information for creating and targeting specific interventions.^2,3 Small area data can also be used to help attribute causality to observed health outcomes.⁴

The use of spatial analysis methods has grown rapidly in recent years,² as has the use of small area data. This growth has promoted increased availability and accessibility of these data, as well as an increasing demand to examine relationships between the environment, risk factors, and health outcomes in local areas. This has prompted the Centers for Disease Control and Prevention’s (CDC’s) National Environmental Public Health Tracking Program (Tracking Program) to consider building a system of sub-county data within the National Environmental Public Health Tracking Network (Tracking Network).

The Tracking Program was established in 2002 to track exposures and health effects associated with environmental hazards and to bridge existing data gaps.⁵ Generally, fewer data are available at the sub-county level.^6,7 The current geographic resolution of data collected by the Tracking Program is limited mostly to the county or state level. The Tracking Program is working to add more sub-county data to fill this void, creating a system of sub-county data. Considerations specific to the Tracking Program’s decision-making process include temporality (eg, cross-sectional vs longitudinal), compatibility between data and measures, dramatic increases in data that must be managed, and communications and technical issues with displaying data.

To address this gap, the Tracking Program had to identify the challenges of increasing the availability of standardized sub-county data and propose solutions to overcome these challenges. A pilot project was launched to shed light on these issues and gain insights into future work at the sub-county level. State and local Tracking Programs were to develop or enhance standards as part of the Nationally Consistent Data and Measures (NCDM) for sub-county data calculation, dissemination, and display. Three overarching questions were identified:

1. How can sub-county data be displayed on the Tracking Network to ensure maximum utility and protection of confidentiality?

2. What are the barriers to accessing and sharing sub-county data through the Tracking Network?

3. What are the challenges to implementing data submission standards and display of sub-county data on the Tracking Network?

Increasing the availability of standardized sub-county data is important for enhancing the capability of the Tracking Network, improving our understanding of environmental health, and informing local-level decision making. This article describes the project and then discusses the lessons learned and recommendations to increase the availability of sub-county data within the Tracking Network and to guide others working with similar data.

Project Overview

In 2014, the Tracking Program proposed the Sub-County Data Pilot Project. Tracking Program grantees were invited to apply for funding to work on this topic. Florida, Maine, New York, Washington, and Wisconsin were selected to develop sub-county-level data from August 1, 2014, to July 31, 2015. Through individual and monthly conference calls, the team evaluated the availability of sub-county data within each state and developed data standards. The team heard expert presentations on the Geographic Aggregation Tool (GAT),⁸ developed by the New York State Department of Health, and calculations for small area data. At the end of the project, grantees submitted sub-county data to CDC. The Figure presents an overview of the process.

FIGURE.

Overview of the Sub-County Data Pilot Project Process

The team reviewed available sub-county data of interest to each grantee. Two data sets were selected for which the Tracking Program already had corresponding county-level NCDM and which were available to 4 of the 5 grantees. These included data sets on acute myocardial infarction (AMI) from hospitalization data and low birth weight (LBW) from birth certificate data. Maine selected childhood lead and private well water data. Grantees submitted data at varying geographic levels, depending on availability.

Existing NCDM standards were extended by developing new standards for geographic and temporal aggregation, suppression, and calculation methods. The team developed standardized indicator measures, data dictionaries, and how-to guides, based on geographic and temporal aggregation levels and state suppression policies for data at set geographic levels. This allowed for standardized extraction of required data elements, identification of cases and events, and calculation of measures from sub-county data. Because of inconsistencies in available geographies, a standard geography was not created. The team did standardize the processes for creating geographies and submitting data.

Each health data set had a variable to uniquely identify sub-county units, which were linkable to geographic data developed by grantees. Geographic data were based on individual or aggregated geographies by census tract, zip code, or town. The Table details the data submission options for each grantee. Data were submitted by all grantees, resulting in a successful data call. Grantees were instructed to document their decisions and challenges in working with sub-county data over time to better inform future sub-county data use and processes within the Tracking Program and more broadly. At the end of the project, CDC and the grantees reviewed lessons learned and recommendations resulting from this work.

TABLE.

Overview of the Data Submitted by Each Grantee for the Pilot Project

	Florida	Maine	New York State	Washington	Wisconsin
Geography level	CT: LBW Zip code: AMI	HMP: CL HMP and town: PWW	CT: AMI and LBW	CT: LBW Zip code: AMI	Zip code: AMI and LBW
Need for geographic aggregation	No	N/Aa	Yes (using GAT)	Either geographic or temporal (eg, single year with CT aggregation)	Yes (using GAT)
Need for temporal aggregation	Yes (5 y, 2009–2013)	Yes (5 y for CL, 2009–2013)	Yes (5 y, 2008–2012)	Yes (5 y, 2009–2013)	Yes (5 y, 2009–2013)
State suppression policy	None	Suppress counts <6	No sub-county-level data sharing (must submit rates)	Small numbers guidelinesb	Suppress counts <5

Abbreviations: AMI, acute myocardial infarction; CL, childhood lead; CT, census tract; GAT, Geographic Aggregation Tool; HMP, Healthy Maine Partnership service areas; LBW, low birth weight; N/A, not applicable; PWW, private well water.

^aHMPs are aggregations of towns.

^bWashington State Department of Health Guidelines for Working with Small Numbers can be accessed at http://www.doh.wa.gov/Portals/1/Documents/5500/SmallNumbers.pdf

Lessons Learned

The needs of state, local, and community-based programs need to be weighed against the challenges of using finer resolution data, which is a trade-off between geographic resolution and statistical stability.⁹ Small area data are more challenging to work with than are larger-scale data because of changing boundaries, lack of historical patterns of change to serve as a basis for estimation, reliability issues, and location-specific factors that can greatly affect calculations.^10,11 The team encountered many of these issues during the pilot project. Several main challenges emerged throughout the process, including during geocoding and assessing aggregation needs. Challenges identified as “lessons learned” included inconsistent geographic levels, aggregation, geocoding data, and data stability issues.

Available geographic level

For this project, no one data set was available for which all grantees had the same geographic resolution. Census tract, town, and zip code were used for different data sets. Zip codes are identifiable for the public and available in many data sets. However, using zip codes has disadvantages.^12–15 Zip code boundaries are spatially and temporally dynamic, differing from other geographies in terms of stability.^14,15 Many grantee data sets were only available by zip code, which must be converted to zip code tabulation areas (ZCTAs). ZCTAs are used to address the difficulties in defining areas covered by zip codes and to assign population estimates.^15,16 However, ZCTAs do not match zip codes exactly and the areas they cover might differ, so typical public health surveillance data cannot be easily linked to ZCTAs.^12,15,17 The notion that ZCTAs represent finer levels of geographic resolution is certainly not the case for exurban and rural areas.^14,18

Census tracts belong to a hierarchical structure of spatial units created by the US Census Bureau. In comparison with zip code areas and ZCTAs, census tracts change less frequently, allowing for longitudinal analyses.^13,19,20 Krieger et al²¹ have promoted the use of census tract–level data due to the relatively homogenous populations and that they comprise administrative units used by various agencies. Some studies concluded that census tract (or block group) units should be used, as these performed better when detecting socioeconomic gradients across various health outcomes than zip codes.^12,22 Census tracts, however, are a less recognizable unit when displaying data. In addition, census tracts are considered coarse spatial units when aggregating certain health outcomes (eg, cancer) and estimating exposures.²³ Geocoding to this level is often based on billing addresses, which might not reflect place of residence. Some data sets, such as vital records, are geocoded using residential address.²⁴

Creation of a national standard geography would, in some ways, reduce the time, effort, and resources needed to ensure consistency across the Tracking Network. The selection of sub-county geographies should provide a compromise between confidentiality issues and the demand for finer geographic resolution data.¹⁸ However, data availability at this resolution has limitations. The selection of a set of geographies ultimately should allow for comparison across outcomes, over time, and between places. For monitoring purposes, dissimilarities of area-based measures across various geographic levels hinder their use for comparisons.¹² The use of such different geographies (eg, census tract, town, zip code) does not allow for comparability and also creates display issues and inconsistency for display on the Tracking Network. Hence, the need for uniform measures and geographies.

Aggregation

Most grantees found that sub-county data were too sparse to present stable, unsuppressed rates. Temporal aggregation can be used, in which a certain number of years are combined for a given area. Grantees noted the need for aggregation over a 3- or 5-year period, depending on the data set. A 5-year period was used for standardization. Temporal aggregation is generally used to estimate stable rates over time, particularly with rare outcomes. It is also used to ensure confidentiality.²⁵ However, any estimates relying on data aggregated temporally will not be able to show trend differences over time for smaller areas.²⁶ The suitability and necessity for using temporal aggregation will be highly dependent on the data set.

Oftentimes, geographic (spatial) aggregation is needed both to obtain meaningful units for analysis⁹ and to protect confidentiality.^25,27 Several grantees used the GAT⁸ to facilitate this, which gave the stability needed to monitor outcomes over time and for grantees to generate the best geography for temporal data points. However, visualization and analysis of long-term trends can be difficult with inconsistent geographies. Grantees noted that aggregations should be meaningful to the public, if possible (eg, zip code aggregation eliminates the ability to recognize the geography). When using geographic aggregation, numerator and denominator rules should be established to have cut points for aggregation or suppression. If, for example, areas are too small, spurious spatial patterns from random variation are likely.^28,29

Although geographic aggregation is often necessary, it presents the modifiable areal unit problem (MAUP), where “conclusions based on data aggregated to a particular set of districts may change if one aggregates the same underlying data to a different set of districts.”^9(p104) The scale (aggregation) effect occurs when the same data are grouped into larger areal units, resulting in different effects. The grouping (zoning) effect occurs when results differ because of alternative areal unit formations at similar scale.³⁰ MAUP cannot be solved; however, it is a problem that should be recognized, understood, and planned for when working with spatial data and geospatial analyses.^27,31

For either form of aggregation, the minimum percentage of data that can be suppressed needs to be established. This could be based either on the percentage of geographies (eg, percentage of census tract aggregation suppressed) or on the percentage of total cases (eg, percentage of hospitalizations out of the total number of cases for that outcome across all geographies).

Geocoding

Geocoding occurs when a spatial location is assigned to an address record.⁹ Thus, records are matched in at least 2 databases: one with address information and one with a reference file containing addresses and geographic coordinates.⁹ Multiple errors can arise with geocoding, including records that have incorrectly recorded addresses or records that have correctly recorded addresses but incorrect geocodes.³² While it would be ideal to have point data to improve our analyses of potential environmental exposures at the sub-county level, many institutions do not share these data or have access restriction policies to ensure confidentiality.⁹ The Tracking Program will be geocoding to census tracts rather than relying on point data. In turn, this means that studies (and the conclusions drawn from those studies) can be limited.⁹

The team suggested establishing a set of guidelines for geocoding data, particularly for dealing with records that are more difficult to geocode. This will allow for more accurate, consistent geocoding. Some grantees suggested exploring alternative reference files so that the number of accurately geocoded records can be increased, reducing the need for manual geocoding. Geocoding to the zip code level should be avoided, if possible, as exact spatial boundaries are often unknown, and frequent boundary changes mean that a geocoding file could be outdated.³³

Nongeocoded records would not be mapped and aggregated. If these data are ignored, this could result in bias toward urban areas, resulting in misleading rates. To deal with missing geography, some grantees used imputation to assign census tracts to nongeocoded records. For example, a zip code contained within a census tract could be assigned to that tract. When a point zip code (eg, for a high-volume address or PO box) is given, an enclosing zip code for the area should be assigned and assessed to see whether it is within a census tract. If it is not, it should be imputed using race or age.

Issues also arose with geocoding geographic units that had a population of zero or that were listed as a PO box. Where zip codes must be geocoded, the team suggested geocoding addresses to the zip code centroid. This is typically done when using commercially available zip code boundary approximations.³³ It is also important to know which, if any, zip codes enclose a PO box–only zip code (ie, a “P” classification zip code).

Population data–related challenges

Accurate population estimates are crucial for accurate rate calculation. However, using data from the census, conducted every 10 years, is not sufficient for these purposes, so estimation methods are required.³⁴ Available sub-county population estimate data were discussed during this project to determine which were most suitable. Estimates are available from the American Community Survey, state and local government agencies, and private organizations.³⁵ However, certain estimates worked better for some grantees than others. Therefore, different estimates and estimation methods were used for this pilot project. Key factors to consider when selecting population estimation methods are the quality and type of data available.³⁵ Several criteria, including estimation error and uncertainty, necessary detail, validity, plausibility, cost, timeliness, and ease of application, should be used to evaluate estimates.³⁶

Other population-related data issues were identified during this project, including the following:

• Inaccurate or misleading data, such as significant census population miscounts in some areas;

• Challenges with the underlying population (eg, prisons, universities); and

• Geographies that had cases but no populations.

Some states have large areas of federal lands that cannot logically be aggregated into nearby areas; therefore, the map has to accurately reflect no populations or rates for these areas. Rather than assign zero population to these areas, it is better to classify these as no measurement so that calculations are not distorted.³⁷ Some patients were also double-counted in hospital transfers. This happened more often in rural areas than in urban areas. Double-counting of patients can be problematic, possibly introducing bias into a data set.³⁸

While the geographies discussed earlier focus on administratively defined units, another consideration for future work is the use of gridded population data. This would eliminate the need for determining which geographies grantees have available and allow for normalizing geographic space, addressing the demand for up-to-date population data at a higher resolution, and obtaining population and environmental data at mutually compatible scales.³⁹ There are currently several high-resolution gridded population data sets that have been used in epidemiological studies (Gridded Population of the World, Global Rural-Urban Mapping Project, LandScan)⁴⁰; however, there are differences between these databases and between these databases and reference data.⁴¹ For example, LandScan effectively handled gridded areas where no one lives while the others did not.⁴¹ Such data sets would have to be examined to determine whether they are suitable for Tracking’s needs.

Data stability and other data challenges

Data stability is an important issue. Data aggregation using smaller geographic units might be more accurate, but less stable rates can arise because the area includes fewer at-risk persons.²⁷ This can also occur when using rarer health outcomes at finer resolutions.⁴² Data stability is of particular concern when rates are based on small numbers, which is likelier in rural areas. In these cases, suppression and aggregation rules do not necessarily guarantee rate stability and rates can vary widely by chance. The team also found lower rates near state borders. Hospitals near state borders often treat patients from neighboring states,⁴³ which requires neighboring states’ data to ensure accurate calculations.

The standard approach to calculating 95% confidence intervals (CIs) can perform poorly with small area data due to non-normal data distribution. The reliability of estimates can also be assessed using relative standard error (RSE), a measure indicating the extent to which estimates deviate from true values, expressed as a fraction of the estimate.⁴⁴ Typically, an RSE greater than 25% indicates high sampling error.⁴⁴ For this project, alternatives considered for calculating CIs included ABC intervals and Dobson, Kuulasmaa, Eberle, and Scherer intervals.⁴⁵ A third alternative, the adjusted gamma CI estimator, was used for rates based on small numbers.

The potential for rate variation means there can be misinterpretation in mapping displays. This should be considered when data are shared. Data visualization, such as the Tracking Network’s maps on the Internet, is a means of sharing information, getting the viewer’s attention, and attracting interest in the data.^28,46 Specific to small area analyses, rates that are mapped in smaller areas are often somewhat misleading because of unstable rates, which is why smoothing techniques are often preferred in these scenarios.²⁸ Other visualization considerations include choosing appropriate administrative units, data classification methods, and color schemes or hatching patterns.²⁸

In cases where risk estimates are unstable, Bayesian hierarchical models have been used to address sparseness in populations and cases, allowing for an adaptive smoothing approach.⁴⁷ This can, however, create overly smoothed maps, masking true risk distribution.⁴⁷ The degree of smoothing used is a trade-off between high sensitivity and high specificity.²⁹ Some have suggested that a numerator of 20 or more is needed to produce fairly stable estimates, approximating a normal distribution and allowing for simpler CI calculations.⁴⁸ There are also other analyses and modeling approaches to consider, including spatial regression techniques. One such example is geographically weighted regression, which allows for regression coefficients to vary over space.⁴⁹ An underlying principle of this approach is that it is expected that places that are closer together are more similar than those greater apart (ie, they are spatially autocorrelated).⁵⁰ Approaches such as this would allow the Tracking Program to explore, analyze, and model relationships between covariates and outcomes of interest while accounting for spatial variation.

For maps, uncertainty might need to be displayed to provide a clearer picture of the data, which is not unique to small area data but important to consider. Several options can be used to display these uncertainties alongside risk estimates.⁴⁷ Estimates and uncertainties can be displayed on a bivariate choropleth map,⁵¹ for example. They can be shown using opacity to represent uncertainty,⁵² decreasing boundary crispness where more uncertainty exists, or using posterior probability values for the interpretation of areas with excess risk.^47,53 Ideally, a standard method for displaying and communicating these uncertainties should be used to ensure consistency.

Another challenge noted during this project was the lack of consistent state and national standards for analyzing, sharing, and displaying sub-county data. Therefore, for this project, different outcome data (eg, AMI, LBW, childhood lead), geographic boundaries (eg, zip code, census tract), and denominator data were used. Establishing standards would improve the maintainability and scalability of data on the Tracking Network’s public portal and increase the comparability between data sets to allow for comparison across studies, time, and places.

Recommendations

The Tracking Program undertook this project to increase the availability and accessibility of sub-county data on the Tracking Network while considering the Tracking Program’s unique needs for creating a system of sub-county geography over time. The considerations and lessons learned from this project led to several key recommendations. These are important steps for building this system of sub-county data and using the data in a way that provides meaningful analyses, particularly longitudinal analyses.

First, data stewards should be engaged to develop new data sharing agreements and policies to allow for the analysis and dissemination of sub-county data for surveillance purposes. Collaboration with data stewards will also include efforts to geocode address-level data to census tracts to allow for standardization. Tracking Program and grantees need guidelines for how to handle difficult records and missing geographies so that all use accurate, consistent methods. Alternative reference files also could be considered, which might reduce the time and resources needed for manual geocoding. Perhaps, using the same software or company for geocoding might ensure consistency among grantees.

Second, the Tracking Program and grantees, in collaboration with data stewards and data users, should develop standardized sub-county geographies. This would be done by aggregating census tracts to allow for comparison and tracking of data over time while meeting the needs of data users. This might include a more conservative aggregation scheme for rarer outcomes (eg, birth defects) and a less conservative aggregation scheme for more common outcomes (eg, emergency department visits for asthma). These schemes should balance the need for sub-county geographies and stable rates using minimal suppression. Potential solutions should be evaluated for identified issues, including handling of census tracts with zero population and zero cases or with zero population and cases, managing and validating census tract–level data, and recognizing and planning for the MAUP. Methods are required both to deal with changes to census tracts over time (ie, creating a consistent geography) and to understand how proposed aggregations could be affected by these changing boundaries.

Third, available population estimates and estimation methods should be evaluated to identify the best ones for these purposes. Guidelines should be established to direct how to incorporate estimate uncertainties into analyses and displays. Appropriate methods should be outlined for calculating sub-county rates, RSE, and CIs. Guidelines also should be established for dealing with reliability issues posed by small numbers frequently encountered in sub-county data sets. Associated uncertainties should be factored into rate calculations and data displays.

Finally, collaboration is key to advancing these efforts and making sub-county data available on a larger scale. The Tracking Network and grantees cannot undertake this endeavor alone. Data stewards need to be involved to work toward the goal of increasing the availability and accessibility of sub-county data. Other experts should be engaged to help develop standards and evaluate the selected methods. In addition, training is required for all aspects of sub-county data use and analysis to ensure successful integration, analysis, dissemination, interpretation, and communication of sub-county data.

The use of sub-county data can also increase public awareness of place-based factors and understanding how these relate to health. Examining socioeconomic disparities at this finer geographic resolution can help guide resource allocation and set and evaluate health objectives.²¹ Having an available system of sub-county data through the Tracking Network will allow users to better understand local health outcomes and risk factors over time. Incorporating these data into the Tracking Network requires collaboration with data stewards and adequate training of public health practitioners so that the benefits of using these data can be fully realized and identified challenges resolved.

Implications for Policy & Practice.

Future use of sub-county data can have important implications for public health. It can support

• ■Identification and monitoring of health disparity hotspots;
• ■Investigation of contributing behavioral, social, and environmental factors; and
• ■Examination of health outcome variations across time, populations, and places.