States’ Use of Local Population Health Data: Comparing the Behavioral Risk Factor Surveillance System and Independent State Health Surveys

Abstract

Objectives

To identify and compare key features of independent comprehensive state health surveys (SHS) with those of the Behavioral Risk Factor Surveillance System (BRFSS) for addressing the need for statewide and local population health data.

Methods

We developed inclusion criteria, systematically collected information about federal and SHS that met these criteria, and obtained supplemental information from SHS leaders.

Results

We identified comprehensive independent SHS in 11 states and BRFSS surveys in all 50 states. The independent SHS meet important statewide and local data needs, filling 3 key health data gaps in the BRFSS: lack of adequate data on special populations such as children, lack of data on specific localities, and limited depth and scope of health topics surveyed on key issues such as health insurance coverage. Unlike BRFSS, independent SHS have limited comparability with each other.

Conclusions

The BRFSS and independent SHS each meet some key state and local data needs but result in data gaps and inefficient use of resources. Surveys could more effectively and efficiently meet future needs for comparable data to monitor health care reform and address health disparities if they were coordinated across states and at the national, state, and local levels.

Since 1984, the Behavioral Risk Factor Surveillance System (BRFSS) administered by the Centers for Disease Control and Prevention (CDC) has been the primary source of state-level population health survey data for states across the United States. In 2008, researchers at Mathematica Policy Research identified all state data sources that could be used to track 10 Healthy People 2010 “leading health indicators” and found a proliferation of state and local population health survey data collection across the United States between 2000 and 2006.¹ Had the BRFSS left critical gaps in the availability of state- and local-level health-related data, leading some states to field separate comprehensive state health surveys (SHS)? The Mathematica study suggested this possibility but did not systematically assess the availability of comprehensive state and local health survey data.

The purpose of this study was to systematically document and characterize the state and local population health surveys that emerged and the features they shared that supplement the BRFSS. We began by developing a set of criteria for inclusion and then identified the number of state surveys that collected comprehensive health information on localities. Finally, we evaluated all the surveys we identified for notable characteristics that distinguish them from the BRFSS to understand why these surveys would emerge, given the difficulty and expense of developing and administering state and local health surveys independent of the BRFSS.

Methods

To be included in our study, surveys had to

• be conducted within the previous 5 years;

• collect self-reported information from a probability sample of the civilian noninstitutionalized population;

• include questions on demographics, income, health status, health insurance, chronic conditions, health behaviors and risk factors;

• produce state and at least one local population estimate; and

• be repeated periodically.

Surveys of limited scope or content, such as those focused only on tobacco or health insurance and access to health care, were excluded. National surveys, such as the National Health Interview Survey (NHIS), were not included because they are not designed to routinely produce state or local estimates for the range of variables they collect. The BRFSS is the only federally administered health survey that met study criteria and released uniform data on all states and some localities.

To identify surveys that met our criteria, we used Google to conduct an Internet search using the key words “health survey” and the name of each state. We scanned Web sites and documents that mentioned an SHS and used more refined key words for subsequent state-specific searches. We also examined health agency Web sites in all 50 states for references or links to SHS Web sites, surveys listed in the Mathematica report or on the State Health Access and Data Assistance Center Web site, and publications that used or cited SHS data.² In addition, we visited all available state BRFSS Web sites.

Web sites were systematically reviewed for survey descriptions; questionnaires; technical reports on methods; online data tables, data files, or query engines; and policy publications using survey data. Information was collected on survey topics and question sets, subpopulations measured, sample design and mode of data collection, dissemination strategies, and innovation. We checked participant lists from an annual meeting of SHS leaders and leaders from states planning or conducting such surveys without a Web presence. For independent surveys that provided little information online, we conducted a limited number of follow-up attempts by e-mail or telephone to obtain additional information. Data were collected from January to May 2009. The study design was reviewed and approved by the UCLA General Campus institutional review board.

Results

All US states and territories use the BRFSS to collect health data on adults aged 18 years and older. We identified 11 states that also collected health data using independent SHS that met our study criteria.

Behavioral Risk Factor Surveillance System

The BRFSS surveys are conducted continuously throughout the year by all 50 states, the District of Columbia, and most US territories with oversight, technical support, and major funding from the CDC. The CDC provides a sampling frame of telephone numbers, an annual core questionnaire, and optional survey questions. Optional questions and additional sample may be added at the state’s own expense.

The BRFSS enables state-specific estimates on chronic conditions, use of preventive services, injuries, and health behaviors for noninstitutionalized adults. The CDC releases state-level BRFSS estimates annually, and states can also conduct their own analyses. However, many states need data on children, consistent local estimates for both geographic substate and subpopulation characteristics such as race and ethnicity, or more in-depth information on key topics than the BRFSS provides.

Independent State Health Surveys

The 11 independent SHS we identified varied widely with regard to populations measured, survey design, and topical emphasis. With one exception, these surveys collected data on children, adolescents, or both (Table). Also, with one exception, they sampled local or other substate areas. Finally, most independent SHS collect more in-depth information than the BRFSS on topics of interest to each state.

TABLE.

Independent State Health Surveys (SHS) Included in Our Study^a

State Survey	Year of First Survey	Frequency of Administration	Population Sampled	Number of Strata	Number of Households
Arizona Health Survey (AHS)	2008	Biennial	Adult, child (parent respondent), and adolescent (interviewed directly)	2	4000
California Health Interview Survey (CHIS)	2001	Biennial	Adult, child (parent respondent), and adolescent (interviewed directly)	44	50 000
Colorado Child Health Survey	2004	Annual	Child/adolescent (parent respondent)b	14	2000
Hawaii Health Survey (HHS)	1968	Annual	Adult (for self and household)	4	7000
Illinois Health Survey	2008 (pilot)	Biennial	Child (parent respondent) and adolescent (interviewed directly)b	1	500
Iowa Child and Family Household Health Survey (IHHS)	2000	Every 5 years	Child/adolescent (parent respondent)	8	4000
Missouri County Level Study (CLS)	2003	2003, 2007 and 2011	Adult	115	50 000
New Jersey Family Health Survey	2001	2001, 2009	Adult (for self and household)c	5	2500
North Carolina Child Assessment and Monitoring Program (CHAMP)	2005	Annual	Child/adolescent (parent respondent)b	d	2500
Ohio Family Health Survey (OFHS)	1998	1998, 2004, 2008, 2010	Adult (for self and child/adolescent)	88	51 000
Utah Healthcare Access Survey (UHAS)	1986	1986, 1991, 1996, 2001 and annually since 2003e	Adult (for self and household)	61	3000

^aAll surveys include demographics, health status, health insurance, risk factors, health behaviors, and health conditions.

^bLinked to the BRFSS parent data.

^cNew Jersey also interviews the adult about any dependent young adults aged 18 to 30 years living outside the household.

^dNorth Carolina draws its CHAMP sample from the stratified BRFSS sample but does not release stratified estimates for CHAMP.

^eIn 2009, Utah had discontinued its independent survey.

Strategies to address major data needs

Special populations

Children and adolescents

Local health departments are responsible for the well-being of children and adolescents as well as adults. Some state BRFSS surveys have added 1 or 2 questions on children, but no state BRFSS collects comprehensive data on children. Existing nationwide surveys focusing on children, including the National Survey of Children’s Health and the Youth Risk Behavior Surveillance System, do not meet our study’s inclusion criteria, nor do they fully address the need for more comprehensive data on children. To fill this data gap in the BRFSS and other federally administered surveys, 10 of the 11 independent SHS collect data on the health of children and adolescents as well as linkable data on their parents.

The range of designs that independent SHS use to collect these data is striking. California, Ohio, and Arizona sample and interview an adult in each participating household and sample a related child and/or adolescent for a separate interview either with the parent or directly with the adolescent after obtaining parental consent. The Iowa Child and Family Household Health Survey collects data on the health of 1 child, but limited data on parents. Hawaii and Utah interview 1 adult about the health of the entire household of adults and children. New Jersey fields a survey to 1 adult per household who reports for the entire family, including dependent young adults aged 18 to 30 years who live outside the household. Colorado, Illinois, and North Carolina administer child and/or adolescent health surveys to parents identified from the BRFSS sample but the surveys are conducted separately from the BRFSS. This strategy reduces the cost of identifying the subset of adults who are parents and enables parent data from the BRFSS to be linked with data on their children from the independent survey. Missouri is the only state with an independent SHS that does not collect data on children.

Racial/ethnic and other minorities

The BRFSS and independent surveys can produce state-level health estimates for some minority populations. These estimates are needed to reflect the increasing diversity of many states, to monitor and address health disparities, and to assess the impact of health care reform. Some samples of subpopulations are too small to support bivariate analyses or local estimates for these groups, and it can be especially difficult to generate state-level estimates for racial and ethnic populations that are relatively small proportions of the state’s population or are concentrated in specific localities. For these reasons, some states oversample racial and ethnic minorities in their BRFSS or their independent SHS, an enhancement that typically requires larger samples, adding to a survey’s total cost.

As an example, the California Health Interview Survey (CHIS) draws a large sample from its state’s very large and racially/ethnically diverse population. The CHIS sampling strategy is designed to include large numbers of Latinos, Asians, African Americans, and American Indians/Alaska Natives and is administered in 6 languages. Samples are sufficient to provide estimates for subgroups of Latinos and Asians. Follow-back surveys have been conducted with CHIS subsamples of low-prevalence populations, including American Indian women, sexual minorities, cancer survivors, and other groups.

Specific localities

The BRFSS has responded to the demand for local health data by giving states the option to request samples that are stratified to be representative at the county or regional level. We found 41 states that used this option to stratify their sampling.³ It is not clear how many states actually created robust substate or local samples using their BRFSS-stratified sampling frame, but at least 16 produced basic local estimates from their BRFSS sample. Stratified sampling usually increases the size and complexity of the survey sample design. States must pay both for the added data collection costs for larger samples and for technical expertise needed to produce substate estimates.

Colorado (2007), Idaho (2007), Mississippi (2007), Virginia (2009), and Wisconsin (2007) designed their enhanced BRFSS sample to provide estimates for their state health planning districts; statewide samples ranged from 5000 to 12 000 in 2008–2009. Texas interviewed a sample of more than 17 000 respondents annually to assure stable estimates in 6 metropolitan areas and occasionally oversampled additional areas to address the needs of specific programs or local health initiatives. The Illinois BRFSS added 12 000 respondents, at state expense, for a total sample of 17 000 to produce estimates for each of 102 counties once every 5 years. North Carolina used its enhanced BRFSS sample of about 15 000 adult respondents to provide estimates for about 20 individual counties and several regions (see the Table for year of sample).

Another option available to states for obtaining substate estimates and estimates for low-prevalence populations is to pool data across years. While the BRFSS’s continuous data collection schedule makes this possible, it is an option that few states appeared to have exercised.

The CDC annually produced local BRFSS estimates for approximately 30 health indicators, reweighting data aggregated from all BRFSS surveys to represent local area populations. These estimates of the selected metropolitan/micropolitan area risk trends (SMART) are generated for metropolitan and micropolitan statistical areas with at least 500 completed interviews and for counties with at least 250 interviews that meet the minimum criteria for reweighting. In 2009, the CDC generated estimates for 180 metropolitan and micropolitan statistical areas and 283 counties nationwide where sample met these criteria. In 2009, there were 3033 organized counties or county-equivalents in the United States (http://en.wikipedia.org/wiki/County [United States], accessed August 10, 2012).

Data collected from the SMART are valuable because they provide basic local estimates for specified health indicators, but they are limited in their ability to meet state and local needs for consistent trend data to inform planning and monitoring of programs and policies relevant to health care reform.⁴ The availability of SMART estimates depends on meeting sample thresholds that rural counties with low population densities are unlikely to ever meet.

All independent SHS studied, except North Carolina, design sample to produce representative city, county, or region-specific estimates cross-sectionally and over time. The number of substate samples per state ranged from one in Illinois (Chicago Metropolitan area) to every county and some cities in Missouri. Sample sizes ranged from about 50 000 households in California, Missouri, and Ohio to 2000 children in Colorado. To facilitate local estimates, the CHIS has converted its biennial survey to a continuous design to allow aggregation across multiple years. North Carolina’s Child Health Assessment and Monitoring Program did not release local estimates because it drew children from the state’s BRFSS sample of parents, which was not designed to produce local estimates.

Health topics covered

The CDC requires all states to administer the same BRFSS core questionnaire; states can add optional CDC-provided content modules or their own questions.⁵ The requirement that all states field the core component assures comparability of core questions across states, an important strength of the BRFSS. The CDC and states jointly approve new questions.

The CDC limits the duration of the BRFSS core survey to a maximum of 15 to 20 minutes to allow room for topics of interest to each state. The total BRFSS interview time, including the core and optional modules, ranges from 15 to 40 minutes (personal correspondence from Lina Balluz, ScD, MPH, Office of Surveillance, Epidemiology and Laboratory Services, the CDC, March 31, 2011). Some states divide their BRFSS sample in half or thirds to administer more questions. This approach⁶ expands the number of questions but reduces sample sizes for the questions that are administered to a partial sample, and may yield insufficient sample size to generate estimates in any 1 year for local areas, rural areas, or subpopulations.

Most of the independent SHS collect more detailed information than the BRFSS on health insurance, behaviors and conditions, and other key issues for monitoring health care reform, whether state-initiated or federal. The emphasis and depth of topical information collected vary by state. Arizona’s survey, for example, collected information on social determinants of health (eg, neighborhood safety, housing, and employment status). California, Utah, and Ohio collected detailed information on health insurance coverage and access to health care, as well as health behaviors, chronic conditions, and mental health. The CHIS also collected detailed information on race and ethnicity, immigration status, sexual orientation, and other demographic and income characteristics, enabling studies ranging from analyzing various disparities in population health and health care to detailed modeling of the impacts of public policies.

The independent SHS used or adapted most of their questions from federal surveys such as the NHIS, the National Survey of Children’s Health, or the BRFSS. At least 3 independent SHS have drawn questions from the CHIS, which adopts (sometimes adapting for telephone administration) questions from the NHIS. Adopting questions used in other surveys enables benchmarking against national surveys or comparison across states. Independent SHS that follow this “best practice” share with the BRFSS the advantage of comparability of measures and estimates. Questions from the NHIS undergo testing at the National Center for Health Statistics Cognitive Testing Laboratory. Using them improves the quality of survey questions and increases the likelihood that different surveys are fielding similar questions for topics of common interest.

Independent SHS examined in this study have different emphases and scope, and they target different age groups. These differences contribute to wide variation in survey length and content.

Data dissemination

SHS data are used by government agencies, advocacy groups, the media, policy makers, researchers, not-for-profit organizations, and businesses. However, the extent to which a given survey’s data are used by these stakeholders depends on the effectiveness of the survey organization in making the data accessible and highlighting important findings through analyses and publications.

Most SHS and BRFSS produce fact sheets, policy briefs, and reports on the basis of analyses of their survey data; some also produce peer-reviewed journal articles. Most independent SHS employ innovative strategies to disseminate survey results. Independent surveys in Arizona, California, Colorado, Hawaii, North Carolina, Missouri, Ohio, and Utah post data tables on their Web sites. Arizona and Ohio create online visual displays of survey research findings. California, Colorado, and Utah have Web-based query tools that enable users to tailor descriptive analyses to their own specifications. For example, California’s AskCHIS query tool has been used by more than 30 000 policy stakeholders and researchers.⁷

Some independent SHS make detailed information about their survey instruments, methods, and results publicly available. Arizona, California, Colorado, Illinois, North Carolina, Ohio, and Utah post their survey instruments online. California, North Carolina, and Ohio create public-use data sets stripped of sensitive and identifying information that can be downloaded from their public Web site. The CDC BRFSS Web site makes survey instruments, data sets, and documentation available online for all surveys. Arizona, California, and Ohio enable authorized researchers to analyze confidential data sets through special user arrangements and requirements. California and Ohio provide detailed reports on survey methodology online. Data of SHS are widely used in research; as of 2009, more than 1500 peer-reviewed journal articles were published using BRFSS data since 1984, and more 230 articles using CHIS data were published in peer-reviewed journals since the data became available in 2002.

Two independent SHS provide resources to promote use of the data by stakeholders with low technical capacity. The Hawaii Health Survey offers statistical consulting on how to analyze and present survey data. The CHIS conducts community and online trainings on how to use and interpret CHIS data and offers technical assistance to all data users. The CHIS makes data widely available to researchers, program directors, and policy makers; advocates at the state and local level; and monitors how data are used and impact policy.⁷ The experience of the CHIS and other SHS provides evidence that high-quality state and local health survey data can stimulate and support public health action.^8–10

Discussion

State and local health departments require local population health survey data to conduct community health needs assessments, measure program impact, and meet new accreditation standards. The 2010 Patient Protection and Affordable Care Act (ACA) mandates that health data be collected and reported “at the smallest geographic level” to address health disparities based on race, ethnicity, and other social characteristics.¹¹ The recent decision of the US Supreme Court upholding this legislation adds urgency to the need for state and local data. Moreover, state and local health departments are being encouraged to seek accreditation from the Public Health Accreditation Board (PHAB), a process intended to improve their quality and performance. The PHAB, a nonprofit organization supported by the CDC and the Robert Wood Johnson Foundation, made the collection and maintenance of “reliable, comparable, and valid data that provide information on conditions of public health importance and on the health status of the population” a key standard that local health departments must meet.¹² The PHAB requirement and the Congressional ACA mandate for better state and especially local health data capped more than a decade of reports and recommendations by the Institute of Medicine, government agencies, and health advocacy organizations arguing that local- and state-level population health data collection and evaluation is essential to national progress in improving health and eliminating health disparities.^13–15

The BRFSS provides information important to the federal government in making decisions about public health policy and funding allocations. It also provides information to states to develop, prioritize, and measure the impact of public health interventions. A key strength of the BRFSS surveys is their comparability with each other. However, needs for local population health survey data are met only partially by what the federal BRFSS program currently supports. We identified 3 key limitations to the BRFSS. First, core BRFSS does not produce reliable estimates for smaller population groups needed for evaluation of the ACA and other policies. Second, although the BRFSS offers substate and local-area sampling, federal funds do not cover the cost of larger samples or analyses that would be needed for states to take full advantage of this option. The CDC’s SMART estimates do not seem to fully meet local jurisdictions’ needs for consistent local estimates. We found only 16 states that had produced substate estimates from their BRFSS data. It is unclear whether this is because states lack the analytic capacity to perform substate analyses, because sample sizes are insufficient, or for some other reason.

Finally, the BRFSS core and most optional modules offer limited depth and breadth on some issues of key interest to states. States have expressed their desire for greater detail on health insurance coverage and access to health care, chronic disease and health status, tobacco prevention and control, environmental factors, and mental health. The BRFSS offers follow-up surveys such as one on asthma, but fielding them is dependent on additional funding.

These key limitations in the BRFSS have helped motivate the development of independent SHS in nearly one-fourth of states. We identified 11 states in which independent SHS play an important role in meeting statewide and local data needs. The SHS developed by 10 of these states systematically sample substate areas and produce local samples and estimates from them. Ten states sample children and collect comprehensive information about them, 3 by “piggybacking” on the BRFSS adult sample. All 11 independent surveys we reviewed either cover topics not addressed in the BRFSS or go into greater depth on specific topics of interest, especially health insurance coverage. Several are specifically designed to produce estimates for minority populations needed to address health disparities.

The CHIS provides an example of how SHS data are deployed. In 2007, the National Institutes of Health sponsored a study to understand the impact of CHIS including the types of users, the way in which the survey was being used, and some examples of successful application of the data. State and national policy makers, advocacy organizations, philanthropic health foundations, private hospitals and health care organizations, and state and county public health agencies use the CHIS to promote education and awareness of an issue, support a policy position taken by the organization, identify a specific population and its characteristics, and to perform complex modeling to meet organizational goals and objectives. Successful application of CHIS data led to significant outcomes in several domains of interest including health policy development, health policy advocacy, grant making, grant seeking and service, and program planning.

While a major strength of the independent SHS is their innovative approach to meeting state and local needs, a key weakness is their limited comparability to each other. This problem of design differences that limit comparability also characterizes federal surveys that measure important indicators in different ways. Key indicators expected to be dramatically impacted by the ACA, for example, are health insurance and access to care. Health insurance coverage, which the BRFSS core addresses with a single question, is measured with different questions and in different time frames in the CDC’s NHIS, the Census Bureau’s Current Population Survey (CPS), and the Medical Expenditure Panel Survey conducted by the Agency for Healthcare Quality and Research. The CPS provides detailed state-level data on health insurance coverage but does not collect information on access to health care, health behaviors, health status, service use, or chronic conditions.¹⁶ Grants from the Health Resources and Services Administration between 2000 and 2005 supported a number of state health insurance and access surveys;¹⁷ many continue with other funding. However, like the CPS, most of these surveys do not collect information on risk factors, health behaviors, or health conditions. The lack of comparability leaves policy audiences to interpret the implications of different estimates produced by these surveys.

An important limitation of our study is that although the data reported here represent the information available on the Internet in early 2009, some independent SHS may not have had any Web presence and are therefore not represented. Also, some SHS may have made significant changes since we conducted our inventory. One of the surveys (Utah) has ceased to exist, while others have been launched. Another limitation of this study is that it excludes SHS that are not repeated periodically but still contribute useful data. We also were not able to collect and compare costs associated with collection and dissemination among the SHS or compared to the BRFSS. Finally, evaluating the quality or detailed content of surveys falls outside the scope of this study.

State health surveys provide crucial information on population health and on how it is affected by state and local conditions, policies, and programs. Independent SHS and/or expanded BRFFS may help state and local health departments obtain PHAB accreditation. Systematic efforts by federal agencies and state BRFSS units to coordinate and collaborate with each other and with independent SHS could begin to integrate existing SHS into a national framework of state-based and national health surveys, possibly framed by the NHIS. Surveys could more effectively and efficiently meet future needs for comparable data to monitor health and health care reform and address health disparities if they were coordinated across states and at the national, state, and local levels. We hope that bringing SHS to the attention of readers may help generate additional funding to explore what data SHS are collecting, how they are using it, and what public health action has resulted when state and local population-based health evidence is available to researchers, policy makers, and advocates.