Assessing Measurement Error in Medicare Coverage From the National Health Interview Survey

Abstract

Objectives

Using linked administrative data, to validate Medicare coverage estimates among adults aged 65 or older from the National Health Interview Survey (NHIS), and to assess the impact of a recently added Medicare probe question on the validity of these estimates.

Data sources

Linked 2005 NHIS and Master Beneficiary Record and Payment History Update System files from the Social Security Administration (SSA).

Study design

We compared Medicare coverage reported on NHIS with “benchmark” benefit records from SSA.

Principal findings

With the addition of the probe question, more reports of coverage were captured, and the agreement between the NHIS-reported coverage and SSA records increased from 88% to 95%. Few additional overreports were observed.

Conclusions

Increased accuracy of the Medicare coverage status of NHIS participants was achieved with the Medicare probe question. Though some misclassification remains, data users interested in Medicare coverage as an outcome or correlate can use this survey measure with confidence.

Introduction

The National Health Interview Survey (NHIS) is a principal source of information on the health of the civilian noninstitutionalized population of the United States. NHIS provides estimates of multiple types of health insurance coverage, which can be used to examine the relationship between coverage types and sociodemographics, health conditions, and health behaviors. These survey-reported measurements of insurance coverage may be subject to measurement error. Medicare coverage estimates based on the NHIS have been lower than estimates produced by the Centers for Medicare & Medicaid Services. For example, NHIS estimated that in 2004, 87% of U.S. adults aged 65 or older had Medicare (Cohen & Martinez, 2005), compared with 98% based on an enrollment estimate from the Centers for Medicare & Medicaid Services (Centers for Medicare & Medicaid Services, 2011; U.S. Census Bureau, 2011). As most U.S. citizens or permanent residents aged 65 or older are eligible for Medicare coverage (Social Security Administration, 2011), there were concerns that NHIS was underestimating coverage. Therefore, a question was added to the NHIS starting in July 2004 to explicitly probe for Medicare coverage among persons aged 65 or older who did not initially report Medicare coverage. Through the use of this probe question in the 2005 NHIS, Medicare coverage estimates among persons aged 65 or older increased by 8 percentage points (Cohen & Martinez, 2006).

The probe question was intended to capture more persons who truly had Medicare coverage, but this could come at a cost of capturing some persons incorrectly reporting Medicare coverage. We evaluated the performance of the probe question by comparing 2005 NHIS data linked with benefit history data from the Social Security Administration (SSA). We considered how underreporting and overreporting affected NHIS Medicare coverage estimates. We further explored respondent characteristics that contributed to discrepancies in reporting Medicare coverage.

Data Sources

Survey Data

The NHIS is a continuously-administered in-person household interview survey conducted by the CDC's National Center for Health Statistics (NCHS). The 2005 NHIS had four main modules: household, family, sample child, and sample adult. For the household module, a household respondent provided sociodemographic information on all members of the household. The family module was completed by a knowledgeable adult (the “family respondent”) who provided health insurance and other information on each family member in the household. Name, date of birth, Social Security number (SSN) and Medicare health insurance claim number (HIC) were requested from the family respondent and the sample adult to facilitate record linkage to other federal data files, but were not necessary for linkage. Records without name or date of birth, or those where SSN or HIC were explicitly refused when asked, were ineligible for linkage (National Center for Health Statistics, 2009c).

Administrative Data

The NHIS is linked to five SSA data files, two of which are used in this analysis: the Master Beneficiary Record (MBR) and the Payment History Update System (PHUS). The MBR file contains beneficiary data for the Social Security Old Age, Survivors, and Disability Insurance program; a record is created when a person applies for benefits regardless of whether benefits are ultimately received. The start and end dates of Medicare enrollment are collected on this record. The PHUS file contains amounts and dates of benefit payments and information related to any Medicare premiums that were withheld (National Center for Health Statistics, 2009a).

Linked Files

To create the linked files, NCHS sends identifying information, including name, date of birth, sex, state of birth, zip code, father's surname (women only), SSN and Medicare HIC number from survey participants to SSA. Participants are linked to SSA's Social Security Numident file, which contains each SSN ever issued. After linkage to Numident, records can be linked to SSA benefit history records. Details about rates of participant linkage to Numident, the record linkage process, and access to these restricted-use linked files through the NCHS Research Data Center are available on the NCHS Web site (National Center for Health Statistics, 2009b, 2009d). The 2005 NHIS and the 1962–2007 SSA files (the latest available at the time of analysis) were linked in July 2008, with approval granted by the NCHS Ethics Review Board. The resulting analysis file includes NHIS participants who provided identifying information and linked to Numident.

Measures

Medicare Coverage

Since 1997, health insurance coverage has been measured on the NHIS using a choose-all-that-apply question about coverage types. (“What kind of health insurance or health care coverage do you/does this person have?”). The interviewer shows (or reads) a card with several types of health coverage to the family respondent and asks which kinds pertain to each family member. Starting in July 2004, a question was added to probe for Medicare coverage among persons aged 65 or older for whom Medicare coverage was not indicated by the family respondent. The interviewer shows a facsimile of a Medicare card and asks, “People covered by Medicare have a card that looks like this. Are you/is this person covered by Medicare?” (National Center for Health Statistics, 2006b).

We classified a person's Medicare coverage at the time of the interview using two methods. For Method 1, we considered a person to have Medicare coverage if Medicare was reported in the choose-all-that-apply question. For Method 2, we considered a person to have Medicare coverage if Medicare was either reported in choose-all-that-apply or in the Medicare probe question (Cohen & Martinez, 2005, 2006).

Medicare coverage from the SSA record can be captured on either the MBR or PHUS files. The MBR file contains the start and end dates of Medicare Part A and Part B coverage. The PHUS file contains the amount withheld for Medicare Part B premiums in a given calendar month. As actual payment dates varied in 2005, we assumed that withholding occurred on the 15^th of each month (Social Security Administration, 2005). If MBR coverage or PHUS withholding dates were within ±30 days of the NHIS interview date, a person was classified as having Medicare coverage at the time of the interview. Persons who had Medicare coverage outside of this 30-day window were considered not to have Medicare coverage at the time of the interview. Records linked to Numident without a corresponding MBR record or to MBR records with no indication of Medicare coverage were classified as not having Medicare coverage at the time of the interview.

Covariates

Reporting of Medicare coverage may be influenced by health care utilization, which in turn may be affected by such person-level characteristics as gender, age, marital status, race and ethnicity, educational attainment, family income, family size, citizenship status, employment status, and having additional medical coverage (Fiscella, Franks, Doescher, & Saver, 2002; Hartley, Quam, & Lurie, 1994; Huang, Shih, Chang, & Chou, 2007; Leclere, Jensen, & Biddlecom, 1994; Mentnech, Ross, Park, & Benner, 1995; Zuvekas & Taliaferro, 2003). These covariates are reported by the NHIS family respondent. Question text for all items can be found in the survey documentation (National Center for Health Statistics, 2006b). Due to high nonreponse on NHIS income items, imputed family income files were used (National Center for Health Statistics, 2006a). Reporting of coverage may also be influenced by such interview-level characteristics as language of interview and mode of interview (telephone or face-to-face). We further distinguished between the family respondent reporting his/her own coverage (“self-report”) or another family member's coverage (“family respondent report”).

Statistical Analyses

Using linked records, we examined the agreement between Medicare coverage on the NHIS (Methods 1 and 2) and SSA files. Using SSA records as a benchmark, we defined “overreports” as reports of NHIS Medicare coverage with no SSA record of coverage, and “underreport” as no report of Medicare coverage on NHIS when SSA records indicate coverage. We sought to answer the following questions for both methods:

• -What is the probability that the NHIS report correctly detects “true” Medicare coverage in the SSA files (sensitivity) and the absence of Medicare coverage in the SSA files (specificity)?
• -What is the probability that a person who reports Medicare coverage on NHIS will have Medicare coverage in the SSA files (positive predictive value) and that a person who does not report Medicare on NHIS will not have coverage in the SSA files (negative predictive value)?

Using Method 2 only, we calculated the frequency of discrepant reports of Medicare coverage among linked records. We further identified factors associated with discrepant reporting using bivariate and multivariate logistic regression models. Analyses were unweighted as inferences for the total U.S. population were not drawn from this study.

Analysis File

The 2005 NHIS had 38,509 participating households and 98,649 persons, with a total household response rate of 86.5%. There were 11,230 NHIS persons aged 65 or older, 3,572 of whom provided either SSN or HIC number. Of these, 3,246 (91%) were eligible to be linked, and 3,165 (98%) of these were successfully linked to Numident. There were an additional 7,658 participants who provided neither SSN nor HIC number. Of these, 3,073 (40%) were eligible to be linked, and 2,136 (70%) of these were successfully linked to Numident. Of the 5,301 linked to Numident, almost all (98%) had a corresponding record on the MBR. Of those linked, 30 (0.6%) records were excluded because they were missing responses for one or more of the Medicare questions or were later recoded using auxiliary data, leaving 5,271 persons for the remainder of analyses. An additional seven NHIS participants, who had initially not reported Medicare coverage, later reported that Medicare had paid for a private health plan or reported having TRICARE For Life, a medical coverage program available to Medicare-eligible uniformed services retirees aged 65 or older and certain family members. We classified these persons as not reporting Medicare. Three records indicated that the participant's Medicare coverage had ended multiple years before the NHIS interview. We classified these persons as not having Medicare in the SSA files. Analyses excluding each of these groups had similar results to the main analyses and are not shown. All analyses presented started with the analysis file containing data on 5,271 persons.

Results

The Medicare probe question was evaluated among the 5,271 persons aged 65 or older who had been linked to Numident. Medicare coverage was reported for 89% of persons on the NHIS choose-all-that-apply question, increasing to 96% with the addition of the probe question. In comparison, 96% of linked SSA records indicated Medicare benefits. Agreement between the two data sources was high (88% for choose-all-that-apply, increasing to 95% with the probe).

Of those with Medicare on SSA records, 10% did not have Medicare reported on the NHIS choose-all-that-apply question (Exhibit 1). The probe question captured an additional 373 persons with Medicare benefits and reduced this underreporting to less than 3%. Of those without Medicare on SSA records, 67% had reported Medicare coverage on the NHIS choose-all-that-apply question, increasing to 70% with the probe. The probe question contributed only six persons to the total number of overreports. With the addition of the probe, sensitivity increased from 90% to 97%, and specificity decreased from 33% to 30%. Negative predictive value increased from 11% to 30%, and positive predictive value remained unchanged at 97%. (Note that “overreport” and “underreport” are used regardless of whether this was a self-report or a family respondent report.)

Exhibit 1. Percentage of persons aged 65 or older with Medicare benefits reported on the National Health Interview Survey, by benefit indication on Social Security Administration files, and by method used to classify Medicare benefit on the National Health Interview Survey. (n=5,271).

Social Security Administration Records
National Health Interview Survey Reports		Record Indicates Medicare Benefits		Record does not Indicate Medicare Benefits
		n	%	n	%
Method 1	Reported benefits	4571	90.0%	129	66.8%
	Did not report benefits	507	10.0%	64	33.2%
Method 2	Reported benefits	4944	97.4%	135	69.9%
	Did not report benefits	134	2.6%	58	30.1%
Probe Question	Reported benefits	373	73.6%	6	9.4%
	Did not report benefits	134	26.4%	58	90.6%

NOTES. Within each method, cells add to the total number of persons in the analysis file (n=5,271) and percentages in columns sum to 100%.

Method 1 = choose-all-that-apply; Method 2 = choose-all-that-apply plus Medicare probe.

SOURCE: CDC/NCHS, National Health Interview Survey-Social Security Administration linked file, 2005.

NHIS participants with Medicare are asked whether they have Part A, Part B, or both. While only 5% of those initially reporting Medicare indicate only Part A coverage, almost 15% of those reporting coverage on the probe question reported only Part A coverage.

Underreporting was significantly more common among: males; persons age 65–74; those who were not widowed; racial or ethnic minorities; persons living with larger families; non-citizens; those who were working; Medicaid enrollees; those in excellent health; those who did not receive medical services at home or had 10 health care visits in the past year; interviews conducted in a language other than English; those whose information was reported by the family respondent; and interviews conducted over the telephone (Exhibit 2). While the prevalence of underreporting was low, further analysis identified demographic and interview characteristics that were significantly associated with underreporting even after adjustment in a multivariate model, including black or other non-Hispanic race, larger family size, not being a U.S. citizen, being employed (not retired), being enrolled in Medicaid, and conducting at least some part of the interview by telephone (data not shown).

Exhibit 2. Associations between underreporting Medicare and selected demographic, eligibility, health insurance, health care, and interview characteristics, among persons with Medicare on SSA records (n=5,078).

	NHIS = Benefit SSA = Benefit		NHIS = No Benefit SSA = Benefit		p-value
	n	%	n	%
All	4944	97%	134	3%
Demographics
Sex
Male	2263	97%	75	3%	0.02
Female	2681	98%	59	2%
Age
65-74	2588	96%	108	4%	<.0001
75-84	1822	99%	20	1%
85+	534	99%	6	1%
Marital Status
Never married	167	97%	5	3%	0.02
Married	2813	97%	90	3%
Living with unmarried partner	66	94%	4	6%
Divorced/Separated	418	97%	12	3%
Widowed	1473	98%	23	2%
Race/Ethnicity
NH White only	3855	98%	72	2%	<.0001
Hispanic	438	94%	26	6%
NH Black only	514	95%	26	5%
NH Other/Multi race	137	93%	10	7%
Education
< High School	1403	97%	41	3%	0.29
High School or GED	1635	98%	33	2%
Some College, AA	954	97%	27	3%
Bachelor's +	888	97%	28	3%
Poverty level
Poor	551	97%	15	3%	0.41
Near poor	1326	98%	29	2%
Not poor	3067	97%	90	3%
Family Size
1	1489	98%	23	2%	<.0001
2	2670	97%	73	3%
3	474	96%	18	4%
4+	148	93%	12	8%
Eligibility
Citizenship
US Citizen	4852	98%	119	2%	<.0001
Not US Citizen	86	87%	13	13%
Employment status
Not working	4240	98%	78	2%	<.0001
Working	693	93%	55	7%
Reported Insurance status
Private
Yes	2869	97%	78	3%	0.97
No	2075	97%	56	3%
Medicaid
Yes	454	94%	29	6%	<.0001
No	4490	98%	105	2%
Military
Yes	376	97%	10	3%	0.95
No	4568	97%	124	3%
Health Care Status
Health Status
Poor/fair	1463	98%	36	2%	0.02
Good	1679	98%	43	2%
Very good	1199	98%	27	2%
Excellent	597	96%	28	4%
Health Care Utilization
Overnight Hospital Stay
Yes	972	98%	18	2%	0.08
No	3967	97%	115	3%
Home care
Yes	210	100%	0	0%	0.02
No	4731	97%	133	3%
Office visit
Yes	1545	98%	33	2%	0.11
No	3383	97%	100	3%
Received care 10 x / past year
Yes	1172	98%	18	2%	0.006
No	3752	97%	115	3%
Interview Characteristics
Language of Interview
English	4668	98%	117	2%	0.0003
Other language	243	94%	16	6%
Reporting status
Self-report	2959	98%	67	2%	0.02
Family respondent	1985	97%	67	3%
Telephone
In-Person	4326	98%	103	2%	0.0002
Telephone	585	95%	30	5%

NOTES. Rows sum to 100%. Medicare underreporting calculated using Method 2. Chi-squares compare proportion of Medicare underreporting within each characteristic. Poverty level used imputed income data.

SOURCE: CDC/NCHS, National Health Interview Survey—Social Security Administration linked file, 2005.

Overreporting was significantly more common among: females, persons aged ≥75, non-Hispanic Whites, U.S. citizens, those who did not have Medicaid, and interviews conducted in person (data not shown). Analyses of overreporting by subgroup do not meet the NCHS standard of reliability or precision and should be used with caution (National Center for Health Statistics, 2010).

Discussion

The addition of the Medicare probe question increased the overall agreement between NHIS participant reports and linked SSA benefit history data, with little increase in overreports. The Medicare probe question particularly improved reporting among those with limited (Medicare Part A only) coverage.

Positive predictive value was 97%, indicating that when an NHIS participant aged 65 or older reports Medicare coverage, it was likely to be confirmed by SSA records. Negative predictive value was 30% with the probe question, suggesting that NHIS did not reliably identify persons without Medicare coverage. However, a report's ability to predict correctly is intrinsically related to prevalence; only 4% of the final study population was not enrolled in Medicare in the SSA files, so even a perfectly specific NHIS question could only yield a negative predictive value of 59% (Rothman & Greenland, 1998). The bivariate analyses found that persons who utilize the health care system are less likely to underreport Medicare coverage; however, these associations decreased in size and significance when controlling for the demographic and interview variables as confounders in the multivariate analysis.

Demographic factors, like non-White race, have been associated in other studies with measurement error on surveys due to satisficing or social desirability (Krosnick, 1991; Warnecke et al., 1997). Family respondents in larger families may underreport coverage, particularly for family members who are not dependents (Pascale, Beatty, & Woo, 2005). There may be some confusion among non-citizens about their Medicare eligibility, leading to underreporting. Some underreporting may be due to dual enrollment in other insurance programs (especially among those still working and those enrolled in Medicaid). While survey underreporting of Medicaid is well-documented (Davern et al., 2007), further research among older adults who are employed and non- citizens is warranted, as few studies have looked at the impact of these factors on reporting government program participation.

Interview factors, and specifically telephone interviewing, may impact underreporting, because of the inability to show the answers to the choose-all-that-apply item or the picture of the Medicare card on the probe question, or by the decreased rapport between interviewer and participant when the interview is conducted by telephone (de Leeuw & van der Zouwen, 1988). Factors associated with overreporting were not explored further in multivariate modeling due to small sample sizes.

By assessing factors associated with measuring Medicare enrollment, we may be able to move beyond the Medicare probe question to improve the accuracy with which NHIS estimates the number of persons with Medicare and Medicaid. Similar results to the ones found here might be expected of coverage-specific probe questions like the NHIS Medicaid probe question that addresses the “Medicaid undercount” (SNACC, 2008). Improvements could also lead to better estimates of community-dwelling “dual-eligibles,” persons enrolled in both Medicare and Medicaid. This could add to the body of research on this small, but expensive, group; “dual-eligibles” consume a disproportionate share of resources from both programs and are at particular risk for unmet long-term healthcare needs (Komisar, Feder, & Kasper, 2005).

There are several limitations to this work, including characterization of Medicare enrollment data from the SSA files as the benchmark. While administrative files like PHUS used to make payments tend to be accurate (Robst, Boothroyd, & Stiles, 2009), there is the possibility of inaccuracy of Medicare enrollment in the SSA files. Errors in the administrative files would likely classify a person as not being covered by Medicare when they were truly covered. This would inflate the number of “overreport” records and decrease the specificity of the NHIS Medicare questions.

We conducted an additional analysis to explore whether the Medicare data in the SSA files was correct. NHIS interviewers who record Medicare coverage for a participant are asked whether a Medicare card or other documentation was presented. Among interviews conducted in person, 98% of participants recorded as presenting a Medicare card also had SSA confirmation of Medicare enrollment in the linked files. We re-calculated sensitivity, specificity, PPV, and NPV by Methods 1 and 2 assuming that 2 percent of records initially classified by our analysis as not having benefits did in fact have Medicare benefits. We found that the relationship between estimates in Methods 1 and 2 held, and that estimates of sensitivity remained unchanged, PPV increased by 2 percentage points, NPV decreased by 2 percentage points, and specificity approximately doubled. These results imply that in the presence of a small amount of measurement error, the addition of the Medicare probe question to the NHIS still improves the capture of true reports of Medicare coverage without introducing a large number of overreports.

This analysis may also be limited by selection bias, caused by systematic differences between the linked and unlinked records. Only 56 percent of the NHIS records in this analysis file were eligible for linkage and only 47 percent were able to be linked to the Numident file. These linked records are a non-random subsample of the NHIS sample. Selection was more common among some subgroups of the population, including males, non-Hispanic Whites and non-Hispanic Blacks, U.S. citizens, those in 2–3 person families, those who were working, those with other forms of insurance (private, Medicaid, military), those reporting poor health status, and those with interview surveys completed in English, in-person, or by a knowledgeable family member.

One approach to addressing this bias is to re-weight the records in the analysis file to reflect the complete NHIS sample. Based on some of this research, we conducted additional analyses and weighted the linked records to the complete NHIS sample aged 65 or older. Weighted analyses had similar results to those in Exhibits 1 and 2, as well as the multivariate analysis, and are not presented here. The breadth and impact of bias on this and other studies using linked NCHS files are currently being explored (Miller, Gindi, & Parker, 2011; Mirel, 2011).

Finally, the Centers for Medicare & Medicaid Services recently provided the NCHS with Medicare claims data to be linked to the 2005 NHIS. These files were not available at the time of this analysis, but will undoubtedly be an important source of data on NHIS participants' Medicare utilization.

Researchers using NHIS data can be confident that Medicare coverage is collected with a high degree of accuracy. While both overreporting and underreporting of Medicare coverage were found, this work highlights the characteristics of people more likely to be misclassified as to their Medicare status, which may provide insight into the direction of potential misclassification biases. Better understanding of the participant demographics and interview characteristics that contribute to misclassification of Medicare status will help inform future survey design and analysis.