Meeting the Need for State-Level Estimates of Health Insurance Coverage: Use of State and Federal Survey Data

Abstract

Objective

Critically review estimates of health insurance coverage available from different sources, including the federal government, state survey initiatives, and foundation-sponsored surveys for use in state policy research.

Study Setting and Design

We review the surveys in an attempt to flesh out the current weaknesses of survey data for state policy uses. The main data sources assessed in this analysis are federal government surveys (such as the Current Population Survey's Annual Social and Economic Supplement, and the National Health Interview Survey), foundation-supported surveys (National Survey of America's Families, and the Community Tracking Survey), and state-sponsored surveys.

Principal Findings

Despite information on estimates of health insurance coverage from six federal surveys, states find the data lacking for state policy purposes. We document the need for state representative data on the uninsured and the recent history of state data collection efforts spurred in part by the Health Resources Services Administration State Planning Grant program. We assess the state estimates of uninsurance from the Current Population Survey and make recommendations for a new consolidated federal survey with better state representative data.

Conclusions

We think there are several options to consider for coordinating a federal and state data collection strategy to inform state and national policy on coverage and access.

States have collected their own survey data for many years but only recently has their need for data been more acute. This is due in part to new pressures to expand health insurance coverage through the State Children's Health Insurance Program (SCHIP) and to demonstrate the effectiveness of these new programs in reducing the number of uninsured children (Balanced Budget Act 1997). In addition, state analysts and policymakers have now been exposed to state-level data and the type of information it can provide. State-level data are routinely used in state discussions about health policy, coverage options, and on the impact of state budget decisions. Policymakers are asking state analysts for more specific information about the characteristics of state populations and more technical detail about the data and from where it comes. States need good state-level data to inform policymakers interested in access, coverage, and trends in health insurance coverage.

The need and interest for state-specific data are not going away. A recent report by the Health and Human Services Inspector General (IG) presents concerns about the states' ability to evaluate their SCHIP expansion efforts on reducing the number of uninsured children as required by law (U.S. DHHS 2004). Section 2108 of the Social Security Act requires all participating states to document SCHIP achievements and program effectiveness. Each year, states are required to submit a report to the Secretary of the Department of Health and Human Services (DHHS) that documents the progress made in reducing the number of uninsured low-income children. States are using a variety data sources and methods to meet this evaluation requirement including use of administrative enrollment data, the Current Population Survey (CPS), and state surveys. The 2004 IG report is a follow-up to a report issued by DHHS in February 2001, which found ongoing problems with state-mandated SCHIP evaluations including “technical and conceptual weaknesses” (DHHS 2001). States continue to struggle to meet these evaluation requirements as estimates of reductions in the number of uninsured low-income children are not typically found in program administrative data, but in outside sources of state and national survey data. Furthermore, the need for state-specific survey data may increase if, in response to the IG's recommendations, enrollment data proxies are no longer accepted by CMS.

This paper outlines the role of states in collecting state-level data, supported in part by the federal Health Resources Services Administration (HRSA) State Planning Grant (SPG) program. While there have been ongoing state-level data collections over time, the SPG program stimulated unprecedented state-level survey activity. Most states now have experience with state-level data and an increased interest in maintaining it. In this paper, we present an overview of the states' need for data followed by a description of how states have met those needs over time. This includes the use of federal survey data and the development of state-initiated surveys. We present concerns both with the current state estimates of the CPS and the HRSA-funded state surveys. We assess the various options that could be used to meet both state and national needs for data on health insurance coverage and recommend improvements in federal-state partnerships to support state-level data collection and evaluation efforts. We have limited our discussion in this paper to state implementation of household surveys but acknowledge additional, although more limited, state-initiated employer survey activity.

HOW STATES HAVE MET THEIR DATA NEEDS

States have met their data needs through a variety of mechanisms and funding sources and both have changed over time. Some states use multiple sources of federal data available at both the state and national levels to understand the characteristics of their state population, including the CPS, Behavioral Risk Factor Surveillance System (BRFSS), and Medical Expenditure Panel Survey—Insurance Component (MEPS-IC). The most widely used is the CPS (Blewett et al. 2004). Over time, many states have developed their own state household surveys to fill the gaps in federal data and produce a more detailed understanding of state populations, programs, and unmet insurance coverage needs. These state-specific data collection strategies have been funded through a combination of state general fund revenue, private foundation funds, primarily—but not limited to—the Robert Wood Johnson Foundation (RWJF), and federal funding, primarily through the HRSA SPG program.

In this section we outline the key federal surveys used by states to measure and monitor health insurance coverage at the state level and then describe the recent history of state survey activity that has been initiated to supplement federal data. We also outline concerns regarding the key state data surveys with the CPS and the state household surveys.

Federal Survey Data

Based on a survey of state analysts to assess the type of data and information they needed from federal surveys, four key criteria were identified (Blewett et al. 2004): (1) sample that is representative of the state population; (2) a large enough sample design that provides for reliable estimates of health insurance coverage for subpopulations, low-income children, by race/ethnicity and by geographic area such as county or region; (3) timely release of data including tabulated estimates of health insurance coverage released within 1 year of data collection; and (4) access to micro-data through readily available public use files with state identifiers to allow states do conduct their own analysis and policy simulations.

Of the five federal household surveys that measure health insurance coverage, three met the first, and perhaps most important, criteria of having a state-representative sample; the Current Population Survey Annual Social and Economic Supplement (CPS), the Behavior Risk Factor Surveillance System (BRFSS), and the State and Local Area Integrated Telephone Survey (SLAITS). The CPS is the primary source of information on health insurance coverage and provides annual state-level estimates of coverage. While the BRFSS does have a state representative sample, its focus is on health and health behavior and has only one health insurance item. In addition, the BRFSS was designed for those 18 and older and while a child component does exist, not all states use it. The SLAITS also provides a state-representative sample. Yet, SLAITS is not an ongoing federally funded survey but a mechanism that includes a sampling frame that can be used to field different survey modules (Blewett et al. 2004).

The other federal surveys clearly offer less to the states because they do not include a state-representative sample. These include the Survey of Income and Program Dynamics (SIPP), the Medical Expenditure Survey—Household Component (MEPS-HC), and the National Health Interview Survey (NHIS). This brief assessment of the monitoring capabilities of presently available national survey data highlights the challenges facing states as they try to track the uninsured.

Technical Issues with CPS from a State Policy Perspective

Although the CPS is the most widely used to monitor health insurance coverage rates over time, states have long been dissatisfied with the CPS and the state estimates of health insurance coverage (SHADAC 2001). This section describes three specific technical issues with the CPS from a state policy perspective; sample size, sampling frame, and imputation of missing values.

Sample Size

The CPS is the only federal survey to produce state-level estimates of health insurance coverage for people of all ages on an annual basis (Farley-Short 2001). The Census Bureau recommends pooling 2–3 years of data to increase state sample size and reduce the variance. Even with pooling 3 years of data, the standard errors for estimates of health insurance coverage for subpopulations, such as the number of low-income uninsured children used in the SCHIP funding formula, are quite large (Davern et al. 2003). In an attempt to remedy this problem, Congress appropriated $10 million annually to the Census Bureau to produce more precise annual state estimates (Balanced Budget Act 1997). The legislation specifically required the Census Bureau to increase the CPS sample size to more accurately estimate the number of uninsured low-income children who are potentially eligible for SCHIP. As a result, the number of households that are interviewed for the CPS increased from approximately 49,596 to 78,000 in 2001. Table 1 compares the increased sample size for select states with the sample size for state-initiated household surveys.

Table 1.

Sample Size Comparisons between the 2003 Current Population Survey's (CPS) Annual Social and Economic Supplement and Select State Surveys

	2003 CPS		State Surveys
Select States	Number of Households Interviewed	Number of Persons in Survey	Number of Households (H) Interviewed or Persons (P) in Survey	Survey Year(s)
Alabama	1,169	3,030	7,299 (H)	2002–2003
Arkansas	903	2,367	6,596 (P)	2001
California	5,600	16,779	73,821 (P)	2001
Colorado	1,626	4,462	10,000 (H)	2001
Connecticut	1,330	3,680	3,985 (H)	2001
Florida	3,511	9,229	38,000 (P)	1999
Georgia	1,136	2,973	10,088 (H)	2002–2003
Hawaii	1,058	3,013	6,000 (H)	Annually
Illinois	2,736	7,588	25,735 (P)	2001
Indiana	1,480	4,103	28,475 (P)	2003
Iowa	1,242	3,454	1,500 (P)	2001
Kansas	1,351	3,712	22,691 (P)	2001
Kentucky	1,055	2,899	5–6,000 (P)	Annually
Maine	1,204	3,160	987 (H)	2003
Maryland	1,242	3,298	5,000 (H)	2001
Massachusetts	1,443	3,958	7,069 (P)	2002
Minnesota	1,398	3,957	29,395 (P)	2001
Montana	856	2,240	4,000 (H)	2003
New Hampshire	1,264	3,664	5,177 (P)	2001
New Jersey	1,981	5,638	650 (H)	2003
New Mexico	1,011	2,689	3,389 (H)	1998
North Dakota	1,068	2,808	1,571(H)	2000
Ohio	2,404	6,455	16,000 (H)	2001–2002
Oregon	1,220	3,377	10,271 (P)	2002
Rhode Island	1,330	3,542	6,583 (P)	2000
South Carolina	1,014	2,687	1,600 (H)	2003
South Dakota	1,171	3,419	1,502 (H)	2001
Tennessee	987	2,663	5,000 (H)	2001
Utah	972	3,345	24,088 (P)	2002
Vermont	1,071	2,903	22,258 (P)	2000
Virginia	1,340	3,599	4,801 (P)	2001
Washington	1,387	3,711	6,726 (H)	2000
West Virginia	1,125	2,975	16,493 (P)	2001
Wisconsin	1,495	4,133	6,368 (P)	Annually
Wyoming	1,039	2,748	5,586 (H)	2002–2003
Total U.S.	78,310	216,424

Although the 2003 CPS included data on 78,310 households across the 50 states and the District of Columbia, the sample varies widely from 903 interviewed households in Arkansas to 5,600 in California. When developing estimates for a more refined substate population, such as the number of uninsured children living in each state below 200 percent of poverty, the sample size is still small (Davern et al. 2003). The sample size (for most states) is also still too small to detect significant change in rates of uninsurance from year to year and to provide estimates for subpopulations—either geographic area or by race/ethnicity (Davern et al. 2003). State surveys typically have larger sample than the CPS (Table 1). For example, the 2001 Minnesota Health Access Survey oversampled populations by race and ethnicity, along with rural residents, for an expanded sample size of 29,395, compared with the 2003 expanded CPS sample size of 3,957. Sample sizes of this magnitude permit analysts to estimate health insurance coverage rates by subregion and by race/ethnicity, and facilitate the development of targeted approaches to expand health insurance coverage.

State Sample Design—Primary Sampling Units (PSUs)

In the first stage of sampling, the CPS divides states into Primary Sampling Units (PSUs) composed of a metropolitan area, a large county, or a group of neighboring smaller counties. For the 2002 CPS sample the 3,141 counties or county equivalents in the U.S. are divided into 2,007 PSUs, 754 of these are selected to be within the CPS sample. Within each state, PSUs are then grouped into strata with other PSUs that have similar labor force characteristics and then one PSU within each strata is randomly selected for inclusion in the CPS sample (U.S. Census Bureau and Bureau of Labor Statistics 2002). PSUs consisting of major metropolitan areas are always included in the CPS sample. This method of sampling ensures that a state's CPS sample is representative of the statewide population but it means that all counties within a state are not included in the CPS sample. The sampling methodology plus the relatively small sample sizes limits the ability to conduct analysis at the county or regional level in many states.

Imputation and Potential Bias

The concern regarding bias in income and health insurance estimates for states is based on the way in which the U.S. Census Bureau imputes its missing data. The Census Bureau uses a process called “hotdeck imputation” to replace missing data with reported values in the CPS. This process segments the population based on a variety of characteristics associated with the variable being imputed. The objective of segmenting the population into homogenous groups is to try to create the closest match between the respondent with the missing values and a “data donor.” If a respondent within a homogeneous segment has a missing value for the variable being imputed, then another respondent from within the same population segment is chosen at random to share their value. In the end, the two respondents will share the same value and the respondent who originally had a valid value is called a “donor.”

Seventeen percent of the income data and 11.3 percent of the health insurance coverage data have missing values and must be imputed (Davern et al. 2004). Notably, in imputing missing CPS values, “state of residence” is not one of the variables used to segment the population for either income or health insurance coverage imputation. As a result, respondents from Texas (the state with the highest uninsurance rate) can donate health insurance coverage responses to recipients in Rhode Island (the state with the lowest uninsurance rate) (Mills 2001). A similar mismatch can occur with the income imputation, although only within the four Census regions, as Census region is used as a segmentation variable for income imputation. This leads to potential bias in estimating health insurance coverage and income estimates for subpopulations at the state level, such as low-income uninsured children (Davern et al. 2004).

STATE SURVEY DATA

States have developed and implemented their own surveys to meet the growing need for better state-level data over time. We use four time periods to describe the history of state survey activity: pre-1990, 1990–1995, 1996–2000, and 2001–2005. Most states have had some experience with state household surveys over the past 25 years and several states have ongoing state surveys. The remainder of this section describes state surveys during each of the four time periods (see Table 2).

Table 2.

State-Level Household Surveys, Pre-1990 to 2004

Pre-1990 (5 States)		1990–1995 (18 States)			1996–2000 (26 States)			2000–2004 (45 States/4 Territories)
	State Funded		RWJF (1993)	State Funded		NSAF (1997, 1999, 2002)	State Funded		HRSA SPG	State Funded
Hawaii	✓	Colorado	✓		Alabama	✓		Alabama	✓
Nebraska	✓	Connecticut		✓	California	✓	✓	Arkansas	✓
Rhode Island	✓	Florida	✓		Colorado	✓		California	✓	✓
Utah	✓	Hawaii		✓	Florida	✓	✓	Colorado	✓
Wisconsin	✓	Minnesota	✓	✓	Hawaii		✓	Connecticut	✓
		Nebraska		✓	Indiana		✓	D.C.	✓
		New Mexico	✓		Kentucky		✓	Florida	✓
		New York	✓		Maine		✓	Georgia	✓
		North Dakota	✓		Massachusetts	✓	✓	Hawaii		✓
		Oklahoma	✓		Michigan	✓		Illinois	✓
		Oregon	✓		Minnesota	✓		Indiana	✓
		Rhode Island		✓	Mississippi	✓		Iowa	✓
		Tennessee		✓	Nebraska		✓	Kansas	✓
		Utah		✓	New Hampshire		✓	Kentucky		✓
		Vermont	✓	✓	New Jersey	✓		Louisiana	✓
		Virginia		✓	New Mexico		✓	Maine	✓
		Washington	✓		New York	✓		Maryland	✓
		Wisconsin		✓	North Dakota		✓	Massachusetts	✓
					Ohio		✓	Michigan	✓
					Oregon		✓	Minnesota	✓
					Rhode Island		✓	Mississippi	✓
					Tennessee		✓	Missouri	✓
					Texas	✓		Montana	✓
					Utah		✓	Nebraska	✓	✓
					Washington	✓	✓	New Hampshire	✓
					Wisconsin	✓	✓	New Jersey		✓
								New Mexico	✓
								North Carolina	✓
								North Dakota	✓
								Ohio		✓
								Oklahoma	✓
								Oregon	✓	✓
								Pennsylvania		✓
								Rhode Island	✓
								South Carolina	✓
								South Dakota	✓
								Tennessee		✓
								Texas	✓
								Utah	✓
								Vermont	✓
								Virginia	✓
								Washington	✓
								West Virginia	✓
								Wisconsin	✓
								Wyoming	✓
								American Samoa	✓
								Guam	✓
								Puerto Rico	✓
								Virgin Islands	✓

Period 1: Pre-1990

Five states—Hawaii, Nebraska, Rhode Island, Utah, and Wisconson—have had long-standing state-funded household surveys that have included questions on health insurance coverage and served multiple needs for information on the characteristics of the states' population. The importance and use of these surveys has changed over time with increased focus on the ability of states to collect information on the distribution of health insurance coverage. These states have had and have used state-level data for over a decade. State general fund revenue and local foundations have primarily funded these surveys.

Period 2: 1990–1995

The health reform of the early 1990s brought new interest in developing options to significantly transform the U.S. health care system. The development of the Clinton Health Plan stimulated the involvement of researchers, academics and analysts in using data and producing research findings to guide decision making at the national level. This reform agenda trickled down to the state level and several states pursued comprehensive reform strategies and implemented state-initiated health coverage expansions including Minnesota, Washington State, Tennessee, and Oregon. The RWJF, through its State Initiatives in Health Care Reform grants in the early 1990s, provided needed state infrastructure support to facilitate state-level planning and implementation activities. As part of their initial planning grant proposals to RWJF, several states requested the funding of state-specific data collection activities. In 1993, RWJF provided funds to the RAND Corporation to coordinate and implement data collection activities in ten states (Cantor, Long, and Marquis 1998). The intent was to collect uniform data that could be used both for individual state planning activities as well as to evaluate program initiatives across states. States were allowed to add a few state-specific questions to the survey, but the core surveys and sampling strategies were developed by RAND.

The 1993 RWJF Household Survey was a one-time investment. The data collection efforts were sound and the quality of data was considered very good. Yet the final weighted data file was viewed by states as coming too late in the state planning process to have any immediate effect. What the survey did provide for the first time, however, was more detail on the characteristics of the uninsured at the state level including estimates of the uninsured by region, race, and ethnicity. The data were also considered helpful as public use files were made available for state use and were archived at the Inter-University Consortium for Political and Social Research (ICPSR) at the University of Michigan for public use. The health reform era stimulated state interest in more detailed data on the characteristics of the uninsured and several states provided new funds to field state surveys (Blewett et al. 2004).

Period 3: 1996–2000

The late 1990s can be characterized by a large increase in state-initiated survey activity. This was a period of healthy state budgets and continued interest in access expansions. Seventeen states funded their own state household surveys during this time period. In addition, the Urban Institute developed and fielded the National Survey of American Families (NSAF) that provided detailed information on health insurance coverage, employment, income, education and housing for 13 states in 1997, 1999, and again in 2001. Five states had both NSAF and state-funded surveys during this time period. The NSAF data were rated highly in meeting state data criteria, and again, the availability of public use files increased their usefulness at the state level (Blewett et al. 2004).

Period 4: 2001–2005

State data collection activity was stimulated during the early 2000s by funding from the HRSA SPG program established as part of the FY 2000 Health and Human Services Appropriations bill. The “Obey Amendment,” as it was commonly called, was introduced by Wisconsin Congressman David Obey (ranking minority on House Appropriations) to provide grants to states to plan for providing access to health insurance for all citizens (H.R. 3194). HRSA awarded competitive 1-year planning grants averaging $1.03 million to fund data collection and analysis, development of coverage options, and program design to cover all uninsured citizens through expanded state, federal, and private partnerships (U.S. DHHS 2000).

Funds were initially targeted to states with the lowest number of uninsured or those that “otherwise could plan to dramatically reduce the number of uninsured within the next few years” (Baumgartner 2000). Table 3 provides details on the SPG award amounts and 3-year average uninsurance rates for SPG grant recipients. With the exception of Arkansas, the first round of grants went to states with relatively low uninsurance rates, Minnesota (8.2 percent), Iowa (9.5 percent), and New Hampshire (9.9 percent). By 2003, the awards included states with relatively high uninsurance rates, New Mexico (21.3 percent), Oklahoma (18.7 percent), and Florida (17.6 percent).1 Through FY2004, HRSA has awarded SPG grants to 46 states, the District of Columbia, and four territories. In FY 2005 HRSA awarded a new planning grant to Alaska for $964,000 plus 11 new pilot project planning grants of up to $400,000 each.

Table 3.

Health Resources and Services Administration (HRSA) State Planning Grant Awards

State	Uninsured*(%)	FY2000	FY2001	FY2002	FY2003	FY2004
Arkansas	16.6	$1,393,322
Delaware	10.1	800,900
Illinois	14.0	1,200,000
Iowa	9.5	1,303,731
Kansas	10.9	1,298,205
Massachusetts	9.6	1,069,195
Minnesota	8.2	1,630,931
New Hampshire	9.9	1,033,315
Oregon	14.8	1,253,264
Vermont	9.9	1,288,892
Wisconsin	9.5	1,240,846
Arizona	17.3		$1,162,879
California	18.7		1,197,000
Colorado	16.3		1,300,000
Connecticut	10.4		668,110
Idaho	17.5		1,119,421
South Dakota	11.0		1,056,812
Texas	24.6		1,350,735
Utah	13.6		1,102,000
Washington	14.3		1,320,400
Alabama	13.3			$766,534
Georgia	16.4			1,170,518
Hawaii	9.9			1,116,429
Indiana	12.9			1,012,798
Maine	10.7			1,283,426
Maryland	13.2			1,232,301
Montana	16.1			721,377
New Jersey	13.7			982,075
South Carolina	13.1			1,042,472
U.S. Virgin Islands	NA			930,992
West Virginia	14.8			1,197,074
Wyoming	16.5			1,023,472
District of Columbia	13.3				$990,000
Florida	17.6				975,000
Mississippi	17.0				1,245,669
Missouri	10.9				938,489
Nebraska	10.3				776,522
New Mexico	21.3				905,000
North Dakota	10.5				781,889
Oklahoma	18.7				874,360
Rhode Island	9.3				961,156
Virginia	12.5				969,729
American Samoa	NA					$868,841
Guam	NA					373,955
Kentucky	13.3					713,619
Louisiana	19.4					801,319
Michigan	11.0					900,000
North Carolina	16.1					864,598
Pennsylvania	10.7					900,000
Puerto Rico	NA					712,811
Tennessee	11.8					900,000

^*Source: U.S. Census Bureau, Current Population Survey, 2002–2004 Annual Social and Economic Supplements

State-level data collection and analysis were key components of the SPG program. Of the 46 state SPG grantees, 42 used funds to conduct a state household survey. HRSA SPG grants funded states with a wide range of data collection experience, from those with ten years of state-initiated survey work (e.g., Wisconsin and California) to states fielding their first household health insurance survey (e.g., Arkansas and Colorado).

Use and Value of State Survey Data

State survey data are used to monitor the uninsured and to produce health insurance coverage estimates that are relevant to the state policy-making process. According to analysts and officials in states that have conducted their own household surveys, state-specific survey results have been more useful in health policy deliberations than national estimates for a variety of reasons, as discussed below. State analysts repeatedly highlight the importance of having state-specific information for state policy discussions (Blewett and Johnson 2004).

State Surveys Support Subpopulation Analyses

Not only do state surveys have larger sample sizes, they also have sample designs that can be used to derive estimates for specific subpopulations. Typical state sampling frames included over-sampling for rural or other geographic areas, racial and ethnic groups, low-income populations, and children. States use these subpopulation analyses to examine disparities in insurance coverage by race, income level, and geographic region. Local communities are interested in these data for planning purposes and many state analysts are required to provide it to CMS for SCHIP evaluation. The design and size of national surveys' samples do not support these subanalyses.

State Survey Data Add Value to the Political Process

State analysts prefer collecting their own data because they can tailor the survey instrument to address unique policy interests, control what analyses are conducted and when the results are released, paying attention to the state legislative calendar when beneficial.

With state-specific survey results, state analysts and officials are also able to uncover local nuances with respect to health insurance coverage issues, answer more detailed questions from policymakers and/or the public, and provide better input into policy deliberations than they had previously been able to do with national survey data. Several states also noted that local groups engaged in the policy making process used the information from the state-level household survey to do things they would not have been able to do with national data sets.

A select number of examples of how state-level household survey results improved state analyses of the uninsured and/or better-informed policy development and legislative processes are listed below (Blewett and Johnson 2004):

• Researchers from Georgia discovered that although rural residents throughout the state were more likely to be uninsured than those in urban areas, the patterns of coverage—and thus the potential policy interventions—for northern and southern rural areas of the state were quite different.

• Hawaii's state-level survey oversampled less populated counties and was crucial in producing county-level estimates as well as estimates related to the state's unique ethnic mix. These details were important to key policymakers and community groups.

• Data from Illinois' state-level survey helped to focus coverage expansion efforts on the design of an affordable product for small business owners with 25 or less employees who did not provide health coverage. After a series of statewide meetings, a pilot project in St. Clair County, one of the most economically disadvantaged areas in the state, was authorized.

• With the ability to look at state survey results at smaller geographic levels, local communities in Massachusetts used their data to provide specific information in grant applications and to pinpoint local needs.

• Detailed survey data from Montana have been used to apply and receive funding for community health centers. Three new community health centers have been funded since the inception of Montana's SPG.

• Researchers in Minnesota used the demographic, employer, and public program questions in their survey to determine potential employer coverage, public program knowledge, and the extent to which eligible children would enroll in public programs if their parents knew they were eligible. Having a state-specific survey also allowed the state to add questions of particular policy interest from year to year (e.g., dental insurance, supplemental insurance, prescription drug coverage).

• Policy analysts from Washington emphasized the political value of local data collection efforts and results. State-specific data, and detailed information at the county level, had more credibility with policy makers and other stakeholders than national data that rarely matched local administrative records.

Ability to Add Policy-Relevant Questions

Another advantage of state surveys is the ability to add policy-relevant questions with short turnaround time for results. For example, Alabama added questions regarding dental coverage and questions related to the “worried insured” (SHADAC 2004a). Indiana added questions to its state survey to get additional information about people with chronic illness and disability regarding their insurance coverage and barriers to care. Finally, the U.S. Virgin Islands were able to get information about the distribution of health insurance coverage for the first time as the CPS is not fielded in the U.S. Territories (SHADAC 2004b).

Problems with Independently Developed and Implemented State Surveys

In this section, we address several of the issues associated with state-initiated household surveys of health insurance coverage. A fundamental underlying problem with state surveys, particularly those funded by HRSA SPG, is that they have been developed and implemented with a limited budget and a tight time frame. As a result, although the information from the state-initiated surveys can be quite influential in the state policy process, there is very little documentation of the potential problems that might arise using state survey data.

The following discussion represents general problems with state survey implementation and is not meant to be a critique of all state surveys. There are several long-standing state surveys that are well documented and would meet high-quality survey research standards (e.g., California Health Interview Survey, Survey of Health Insurance Status of Massachusetts Residents, and the Wisconsin Family Health Survey). Although this technical discussion of state surveys was developed from a survey research perspective, the states themselves continue to find the data of high value for state policy work. State analysts are in need of information and make the tradeoff between having no information and relying on less-than-perfect state survey data information.

Survey Documentation and Public Use Files

Even though the information is widely used in forecasting and budget decisions, there is often very little documentation of state survey methodology. This includes the lack of documentation when the survey instrument is modified or when problems are encountered in the field. Because of time and resource constraints, these changes are often not documented, even though they may affect the results of certain analysis. In addition, many states, or their vendors, do not maintain a master data file which should include any changes or edits to the data. Without proper database management resources, data files can become altered in significant ways and not allow consistent analysis over time for key items like uninsurance rates.

Although state analysts responsible for data collection generally have access to the state survey microdata, there are few public use files made of state survey data (to our knowledge the California Health Interview Survey is the only exception). Making the data public serves two important functions that are not currently being met by most state health insurance surveys. First, it requires good documentation including changes made during survey implementation, data definitions, and sample and weighting strategies. This is required so that an analyst who was not involved in the data collection can reproduce published estimates and conduct appropriate analysis and tabulations from the data. Also, when the data are released in public use files, analysts can find problems with the data and report them to the agency that collected the data. All survey data, whether they are collected by the federal government or by the states, have limitations; knowing the limitations is essential to producing good policy analysis. Producing public use files and making state data available for others to use would help serve this purpose, just as it has for the CPS and other national surveys.

Some Survey Modules Not Well-Developed

Although state surveys do a good job with survey items that determine a respondent's health insurance status, other modules are not as well designed as those of national surveys. Survey modules that include questions related to demographic characteristics, relationships within the household, and families are particularly problematic. State surveys usually select a random person within the household and gather detailed health insurance and demographic data for that one person. They often do not consider the employment status, race, or education level of others in the household, nor do they explore all the relationships within the household. This information is needed in order to accurately simulate a person's health insurance coverage status under various scenarios (e.g., how many family insurance units are there within a household and is anyone in one of these units employed?). Also, the income data on the state surveys tend to be collected with an “all-in-one” income item (e.g., what was your total family income in the last 12 months). This type of “omnibus” income item does not allow for accurate estimates of program eligibility and poverty status, which require more detailed income-related survey questions (Dubay and Kenney 2000; Davern et al. 2005).

State Survey Sample Designs

Sample designs are considered inefficient on the whole if there is a great deal of variation in the probability of selection into the survey that is not highly related to the dependent variable of interest (e.g., some people are selected with a 1 in 20 probability and others are selected with a 1 in 3,000 probability). Sample efficiency is measured by the “design effect,” or the ratio of the variance of a statistic calculated taking into account the complex survey design to the variance of a statistic calculated not taking into account the complex survey design (Kish 1992). The design effect for the CPS is 1.9 for the national estimate of health insurance coverage, meaning that the variance calculated taking the complex sample design into account is 1.9 the size of the variance that is calculated not taking the sampling scheme into account (Davern et al. 2003). Many of the state surveys have design effects of 2–5 for key estimates of health insurance coverage. The advantages of having a larger sample size within a state survey can be offset by an inefficient sample design. For example, a survey with a design effect of 4 will need to have twice as many completed surveys to achieve the same precision (same standard error) as a survey with a design effect of 2.

Sample Coverage and RDD Sampling

Because of the cost and complexity of in-person household surveys, the state surveys, as well as the federal SLAITS and BRFSS, rely heavily on Random Digit Dialing (RDD) telephone surveys. There are two main coverage problems with RDD surveys. First, households that lack telephone service are typically not included in the survey sample. This is problematic because households without a phone are more likely to be uninsured than the general public and the national rate of “phonelessness” is approximately 5 percent (Davern et al. 2004). CPS data indicate that for 1998–2000, persons living in households without phones had an uninsurance rate of 31.2 percent, compared to 15.4 percent for the nation as a whole. Most states do include an adjustment to account for the lack of telephone coverage when producing estimates of health insurance coverage (e.g., Davern et al. 2004).

The second problem with RDD sample coverage is that cell phone-only households are becoming more common. Preliminary analysis of the 2003 NHIS and the February 2004 CPS indicates that roughly 4–6 percent of the population lives in cell phone-only households (Tucker, Brick, and Meekins 2004; Blumberg and Luke 2004). Tucker et al. (2004) looked at the Consumer Expenditure Survey data from 1994 through 2003 and found that the number of households with cell phone bills, but without land lines bills, has been increasing rapidly in recent years. If this trend continues, RDD sample coverage may reach unacceptable levels for general population surveys.

Response Rates

RDD telephone surveys also have two problems related to response rates: (1) there is a lack of standardization for calculating response rates among vendors and (2) response rates for RDD surveys are continuing to decline to the point where 50 percent is considered the best vendors can do in many circumstances. Vendors doing state surveys vary a great deal in how they measure response rates. Even though professional survey research organizations like the American Association of Public Opinion Research (AAPOR) have specific standards regarding how to measure survey response rates, many vendors may not be aware of these efforts, or do not calculate them appropriately. Response rates of 70–80 percent, as reported in some instances, are highly unlikely with general population RDD surveys (if not impossible) using the calculations developed by AAPOR. Many vendors have adopted the AAPOR (or similar) codes for calculating response rates, and it is important that all vendors disclose their calculation methods.

Response rates for RDD surveys have been falling over recent years. One nationwide survey conducted by the Pew Research Center had a response rate of 61 percent in 1997, but 51 percent in 2003 (Pew Research Center 2004). Although the Pew study also found that surveys with response rates as low as 27 percent can be as representative as surveys with 51 percent response rates on opinion and attitude polling, data accuracy is critical when survey estimates are used to inform issues of public policy. Low response rates may lead to biased estimates of state health insurance coverage, even though polling numbers on attitude are unaffected by the wide variation.

Lack of Stable Funding for Follow-up Surveys

States are extremely interested in efforts to increase the availability of relevant information on the uninsured. Several states are responding to the inadequacy of national data on the uninsured by conducting their own surveys. Unfortunately, many states lack the necessary financial resources and technical skills to pursue this option and are left trying to use national survey data to the best of their abilities. The one-time funding provided by the HRSA SPG program dramatically increased state data collection activities. However, to assess the effectiveness of expansion initiatives, the impact of changes in the market, and trends over time, most states cite the need for more than one-time funding for repeated cross-sectional surveys. The challenge in most states will be to find new sources of funding and the expertise to support additional data collection.

OPTIONS AND RECOMMENDATIONS

In order to better meet the need for data on the distribution of health insurance coverage at both the state and national level, we will discuss four possible strategies: (1) leave things as they are with individual states developing, funding, and implementing their own state surveys with no coordination; (2) build on existing federal surveys that currently produce state-level estimates—primarily the CPS; (3) develop a new national survey that would meet the needs of states; and (4) develop a new federal–state partnership that allows states to play a role in data collection and analysis in federal surveys.

Leave Things as They Are

The current situation with federal and state surveys on health insurance coverage can be described as inefficient and perhaps nonsensical. The federal government sponsors at least six national surveys that ask questions about health insurance coverage, each producing different estimates and most are of limited value to states. Policymakers are interested in point estimates and want to know the number of uninsured in their state. States are left with the complex problem of trying to sift through the data and discern what the real rate of coverage is in their state. Table 4 compares rates of uninsurance from different surveys for nine states using state surveys, CPS, and the National Survey of American Families, showing how difficult it is for state analysts to reconcile the different point estimates and then explain the differences to policymakers.

Table 4.

Comparison of Population Estimates of Uninsurance by Survey Source: State Survey versus CPS; NSAF versus CPS^*

	State Survey versus CPS			NSAF versus CPS
	State Survey Year	State Survey Estimate† (%)	CPS Calendar Year Estimate from State Survey Year‡ (%)	NSAF 2002 (Age 0–64)¶	CPS Calendar Year 2002 (Age 0–64)‡
Alabama	2003	11.2	14.2	14.3	14.8
California	2003	15.6§	18.4	18.6	20.0
Colorado	2001	11.7	15.6	16.7	17.8
Florida	2004	19.2§	19.9	20.7	20.6
Massachusetts	2004	7.4	11.7	6.5	11.4
Minnesota	2004	7.4	8.9	7.1	8.8
New Jersey	2001–2002	15.0§	15.8**	13.3	15.8
Washington	2004	9.8	13.0	12.5	15.8
Wisconsin	2003	6.1	10.9	8.3	11.1

Notes:

^*State surveys and NSAF use point in time measurement of uninsurance (unless otherwise noted). The Current Population Survey is used to determine whether respondents were without insurance during the preceding calendar year. So, for example, the 2004 Current Population Survey is used to estimate insurance coverage for calendar year 2003.

^†SPG reports to HRSA can be found at http://www.statecoverage.net/hrsa.htm

^‡The 2001–2004 Current Population Survey. U.S. Census Bureau. Washington, DC. Available at http://www.census.gov/hhes/www/hlthins/historic/index.html

^¶National Survey of America's Families. Urban Institute: Washington, DC.

^§0–64

^**2003 CPS calendar year estimate of 2002 used.

There is very little coordination of state data collection activity funded or required by the federal government. In part because of federal budget processes and short timelines for implementation, the HRSA SPG program did not develop and test a coordinated federal–state data collection effort. The Centers for Medicare and Medicaid Services (CMS) have also chosen a hands-off policy. While the SCHIP legislation required states to monitor changes in rates of uninsured children, CMS did not require states to produce their SCHIP evaluations in any uniform or coordinated way. Only 22 states complied with this requirement in FY 2002 and they used different sources of measurement including state surveys, CPS, and enrollment data (DHHS 2004). We recommend that CMS provide direction for a more coordinated evaluation effort. In a time of resource constraints, effort should be made to better coordinate and consolidate federal support for data collection on health insurance coverage at both the state and national level.

Build on Existing Federal Surveys and Try to Fill the Gaps

One option is to build upon existing federal surveys and redesign them to meet the data needs of states. The logical federal surveys to consider include: the CPS, the BRFSS, and SLAITS. Another related option is to allow states to buy additional sample in either the CPS or another federal survey. A final option will be to support the development of small area estimates currently being pursued by the Census Bureau. Each option will be discussed in turn.

Fix the CPS

Key issues with the CPS include state sample size, health insurance question timeframe, sampling design, and imputation procedures for the health insurance and income questions. While the sample size was increased across states in 2001, with more sample added to states with historically low sample sizes, the way in which the sample is drawn has not changed and many areas within a state are deliberately not included. Additional changes include increasing the sample, modifying the sample design, revising the CPS reference period, and developing imputation strategies for replacing missing values. The CPS reference period for recalling health insurance coverage in the CPS has been a concern for many years and must also be addressed (Holahan, Kenney, and Nichols 2004). Finally, we have concerns about the ability to expand the CPS, which is primarily a labor survey with the health coverage questions added at the end of a fairly long and detailed labor survey. The CPS does include an in-person component, which will likely be required in future surveys to address nonphone coverage and cell phone-only households. The concern is whether the health insurance component would ever get the attention and priority it needs to meet multiple state and national needs for health data.

Add on to the BRFSS

The BRFSS is a good federal–state partnership model. States collect and send data to the Centers for Disease Control and Prevention (CDC) for compilation, comparison, and public release. A logical outgrowth of the BRFSS is to accommodate the need for estimating health insurance coverage for all people in the U.S. The central drawback with the BRFSS is that it is targeted to those 18 years and older, focusing on health status and risk behaviors. Adding children and more detailed questions on insurance, income, and employment would be difficult due to the current focus of the survey, and costly due to the detail needed on key questions. In addition, the BRFSS is an RDD telephone survey and therefore has problems associated with increasing cell phone use and nonphone coverage.

Fund a New SLAITS Module

The SLAITS provides an opportunity for more detailed information at the state and local levels to support state-level policy applications. SLAITS is not a routinely funded survey, but a sampling frame and vehicle for others to support and fund their own surveys. The recent survey module for Children with Special Health Care Needs was representative at the state level but still did not have enough state samples to do subpopulation analysis. Perhaps a more critical concern is that like the state surveys, SLAITS is based on a telephone RDD sample with no in-person interviews. As stated earlier, this is a problem due to the increasing number of cell phone-only households and for those without phones at all. Finally the data release was more than 2 years after the time of data collection. Despite these concerns, the goals of the SLAITS module are laudable, which are to collect data at the federal level that could be used for state and local area estimates.

States Buy Sample in Federal Surveys

Another option some states have pursued is to “buy-in” to existing national surveys to expand a state's sample size. The MEPS-IC has the most experience with state buy-in. This employer survey is administered by the U.S. Agency for Healthcare Research and Quality (AHRQ) and is one of the few federal surveys that provide state-level estimates of employer health insurance offer and take-up rates. Nine states have used their SPG grant funds to increase their sample in the MEPS-IC. These states and their added estimated sample (in parentheses) include: Arkansas (1,000), Delaware (650), Kansas (500), Maryland (800), New Hampshire (500), South Dakota (500), Vermont (650), Virginia (800), and Wisconsin (800). Prior to HRSA, the RWJF also funded additional MEPS-IC sample in Massachusetts (410), Arizona (740), and Washington (1,000). Finally, Massachusetts bought additional sample (800) using state funds also in 1998.

The benefits of buying-in to this survey include the assurance of methodological rigor and data quality, as well as the availability of staff at AHRQ, who provide assistance and additional analysis for participating states. On the other hand, buy-in states are dependent on the data collection and release schedule of the U.S. Census Bureau, which fields this survey for AHRQ. In addition, the microdata for the MEPS-IC are nonpublic and are only available to researchers through the Census Bureau's Research Data Centers (RDC). The reason is that the Census Bureau's sample frame of establishments from which the MEPS-IC sample is drawn, is derived from Internal Revenue Service records, thus making all resulting microdata confidential. Researchers must develop a proposal and buy time at the RDC to conduct on-site analysis. Even AHRQ researchers in Rockville, MD do not have access to the data except through a Census RDC.

Support the Development and Use of Small Area CPS Estimates

Another way the Census Bureau could improve the utility of its data for state analysis is to continue developing its relevant small area estimates of health insurance coverage. These estimates are the result of a small area statistical model that combines survey data with administrative records to develop more precise estimates of particular concepts of interest (e.g., poverty, income, and health insurance coverage) for the smaller areas. These “small” areas include states, counties and school districts. The Census Bureau has produced estimates of the number of children below 200 percent of poverty in each state without health insurance coverage (Campbell and Fisher 2002) and uninsurance rates for counties (Turner and Fisher 2003). Estimates of this type would help fulfill the policy needs of states who need to monitor the number of low income, uninsured children within their states as part of SCHIP and would help them also understand how health insurance coverage varies by county. Estimates from this type of modeling are currently used in the Title I Education funding formula for smaller geographic areas such as school districts and counties, and could also be used in the SCHIP funding formula to improve its precision (National Research Council 2000; Davern et al. 2003).

Small area estimates themselves, however, are not suitable for all policy applications and their limitations should be clearly stated by the producers of such data so consumers know what is and is not an appropriate use of the estimates understood (National Research Council 2000; Davern et al. 2003).

A New National Survey

Most of the recent state health insurance surveys have been funded by the HRSA SPG program and very few state health care surveys will survive without continued federal support. One way the federal government could help is to fund a national survey that produces state-level estimates that are of health policy interest. If a new survey project is to be successfully integrated with state policy needs it must address the state data needs discussed in the first section of this paper.

First, there should be flexibility to adapt the survey on a state-by-state basis to meet the state's current needs for information. The BRFSS survey is instructive as a federal state survey model. The BRFSS mandates a core instrument and methodology but allows states the flexibility to develop and use survey modules to meet their own data needs. Whether or not the state would actually be responsible for collecting their own survey data may not be possible with this survey as with the BRFSS, but allowing states to “buy in” modules and enlarge sample sizes will improve the ability of the survey to meet state's needs.

Next, the new survey should be conducted using area-probability sample design to support subpopulation estimates. As stated earlier, past state surveys and national surveys have thus far relied heavily on RDD sampling methods. The emerging evidence is showing that RDD samples are currently decreasing in population coverage. If the trend continues as it has through 2003 and early 2004, RDD sample coverage will reach unacceptable levels (Blumberg and Luke 2004; Tucker, Brick, and Meekins 2004). Using an area probability design will inflate the cost of doing state surveys, but simple adjustments to RDD surveys for noncoverage of telephone households may not be adequate anymore (Blumberg and Luke 2004).

The area probability design should also use sub-county pieces of geography such as Census block groups or tracts as its primary sampling units (PSUs). Currently the CPS uses counties and groups of contiguous counties as PSUs and does not draw sample from each county within a state. Although such sampling is sound for producing statewide estimates, most states would like to be able to draw sample from the majority of their counties and be able to develop county level estimates for the larger counties in the state. Also, data from the sampled counties could be used by the state to produce small area estimates of their own for those counties without sampled respondents and for those counties with small sample sizes (this is already performed in some states using BRFSS data). Using this strategy will increase the cost of administering the survey, but it will also greatly increase the utility of the survey data for state-level policy analysis.

One way to implement this strategy more cost effectively would be to use the Census Bureau's American Community Survey (ACS) as a vehicle to select the sample cases for the new survey (Davern et al. 2004). The ACS will be sampling 3 million addresses on an annual basis from all states and counties by 2005, and the new survey could be added as supplement to the ACS. This would also allow oversampling of key demographic groups within each of the states. If, for example, the ACS survey determined the household had key characteristics—e.g., children living under 200 percent of poverty—the household could be selected to take the new health supplement survey with a higher probability.

State analysts will also need access to the restricted data files produced by the survey. As survey disclosure review and confidentiality concerns continue to limit the information that the federal government statistical agencies puts on its public use files, the states must be viewed as an equal partner who will protect the confidentiality of the survey respondents and make good use of the restricted data files. This will allow state analysts to develop sound small area estimates using either direct survey estimates or small area estimation models. Not having access to this information reduces the utility of the survey for state purposes.

The survey questionnaire could use the NSAF, MEPS-HC, or the NHIS as the starting point. The NSAF filled significant gaps in the instrument design of the current federal surveys and NSAF innovations have even influenced changes to the CPS including the addition of the recent insurance verification item. The income, employment, demographics, and health care items on this survey are well designed and could provide a useful starting point for the new national survey that provides high-quality state estimates. However, the MEPS-HC and the NHIS also offer advantages in terms of survey design and content. Any of these surveys would be a good starting point for the development of a new national survey that is designed to multiple objectives.

And finally, the survey should be done on an annual basis. The most pragmatic way to make this happen is to follow similar procedures to the ACS or NHIS for interviewing households continuously throughout the year. And like both the ACS and the CPS it will be extremely important to make sure that the survey data are made available to state researchers within 6 months after the end of data collection.

In order for a new survey to be developed, the federal government will need to provide the majority of the funding. Only a handful of states are likely to continue doing surveys when the HRSA SPG grants are finished. In order for a new survey to meet the most critical needs of the states it should meet the criteria mentioned here in addition to meeting the usual high statistical standards and requirements set by the Office of Management and Budget for federal survey data collection.

A New Federal–State Partnership

A new national survey that meets the needs of state analysts would be a great vehicle to build a federal–state partnership for collecting health data to inform policy, but given that funding is unlikely a partnership should be developed even without a new survey. There are several examples of state and federal government cooperation that could be very useful in providing state and national policymakers with the information they need to monitor health care trends and make sound policy. The best examples of such partnerships extend beyond just a survey partnership. Models of federal–state cooperative data collection agreements to provide national and state estimates are well established for agriculture and economic issues in this country (Lepkowski 2001).

While models such as the BRFSS and the Census Bureau's state–federal partnership for producing population estimates in nondecennial Census years are a good start (U.S. Census Bureau 2003), we would like to see something more. Federal funds should be allocated to build relationships with state health analysts that could be expanded to include a reporting system. For example, the partnership should make use of state public health insurance enrollment data to estimate the number of Medicaid and new Medicaid enrollees per year. Information of this sort, along with vital survey information could be reported out to policymakers on an annual basis. Coordinating an annual report of this type would serve state and federal decision-making needs.

Two economic examples are the Workforce Information Council (WIC), a federal–state data collection initiative that facilitates the collaboration between the Bureau of Labor Statistics (BLS) and local state economic analysts to collect and disseminate quality workforce data (Blewett et al. 2004). This initiative uses states as the data collection entities through agreed-upon standardized methods to allow cross-state comparisons and national estimates (Workforce Information Council 2003). This activity is funded by the Department of Labor but does ask for each state to contribute staff time to the various projects. Another example is the Federal Reserve. Each of 12 regions within the Federal Reserve System is to use a variety of survey data, administrative data, as well as idiosyncratic information to inform the Federal Reserve Board about developments in the regional economy. Eight times a year, the regional information is placed into the “Beige Book” and is used to inform economic policy at the national and regional level (Federal Reserve Board 2003).

This type of model would require significantly less investment than a new survey but it is still not likely to become a reality unless state and national funding priorities change. However, at the very minimum national survey expertise should be more readily available to states to enhance survey design, reduce nonsampling errors, and improve the precision of state-level estimates. Finally, joint state–federal collaborative research should be undertaken to improve item nonresponse and noncoverage through the use of telephone surveys to improve survey methods overall (Lepkowski 2001). State surveys provide an opportunity to test different approaches and methods that benefit both state and national survey researchers.

CONCLUSION

We encourage a dialogue between national and state survey researchers and policy analysts on the best use of survey resources to meet state and national data needs (Blewett et al. 2004). Because of the established need for information on health care coverage and access, substantial resources are being devoted to data collection in many venues. The federal government has the resources and expertise and six surveys that could be better coordinated, if not consolidated. The states have a great need for both data and technical assistance and are implementing their own state surveys with limited resources and survey expertise. But both state and national policymakers want better data and information on the changes in the distribution of health insurance coverage and the impact of programs and policies such as SCHIP. Clearly, better coordination between the federal and state efforts is required and we have recommended some strategies to achieve this. It is time to rethink our national data collection strategy on health access and to refocus our efforts on informing policy decisions where they occur: at the state policy level. Now more than ever, good data are needed to make informed policy decisions in an increasingly constrained state budget environment. We envision an active federal–state partnership as a vehicle to achieve these objectives.