Agreement Between Administrative Data and Patients’ Self-Reports of Race/Ethnicity

Abstract

Objectives. We examined agreement of administrative data with self-reported race/ethnicity and identified correlates of agreement.

Methods. We used Veterans Affairs administrative data and VA 1999 Large Health survey race/ethnicity data.

Results. Relatively low rates of agreement (approximately 60%) between data sources were largely the result of administrative data from patients whose race/ethnicity was unknown, with least agreement for Native American, Asian, and Pacific Islander patients. After exclusion of patients with missing race/ethnicity, agreement improved except for Native Americans. Agreement did not increase substantially after inclusion of data from individuals indicating multiple race/ethnicities. Patients for whom there was better agreement between data sources tended to be less educated, non–solitary living, younger, and White; to have sufficient food; and to use more inpatient Department of Veterans Affairs (VA) care.

Conclusions. Better reporting of race/ethnicity data will improve agreement between data sources. Previous studies using VA administrative data may have underestimated racial disparities.

The growing interest in racial disparities in the provision of health care has fostered an increase in the use of race/ethnicity data derived from administrative data files. Despite the increasing demand for and use of these data, their reliability has been examined in only a few studies. This omission is a significant one, because the reliability of racial designations is crucial for accurate estimation of racial disparities in health care.

A few previous studies have examined the reliability of racial classifications in administrative data from specific states, the federal government, and national insurance programs. Blustein documented that racial classifications for patients with multiple admissions in hospital discharge data in New York state lacked reliability, especially for non–African American and non-White racial categories.1 When California birth certificate race/ethnicity data were compared with race/ethnicity information obtained by interview, Baumeister et al. found that the sensitivity of the birth certificate data was significantly lower for Native Americans.2 In a review of vital statistics on race and ethnicity, Hahn and colleagues noted inconsistencies between birth and death records of infants, especially for Hispanic persons and for races/ethnicities other than White and African American.3 Pan and colleagues compared racial designations in Medicare and Medicaid data, finding significant amounts of contradictory information on race/ethnicity between the programs, with the greatest discrepancies for Hispanic, “other,” and Asian classifications.4 Boehmer and colleagues documented that study outcomes differed markedly depending on whether the source of race/ethnicity information was Department of Veterans Affairs (VA) administrative data or self-report data. Specifically, additional race/ethnicity differences in the use of tooth extraction versus root canal therapy were found when self-report data were used.5 They also noted discrepancies in race/ethnicity classifications between the data sources.

Because a number of recent studies on racial variations in cardiac care have been based on VA databases,6–10 understanding the accuracy of these data is especially important. One study examined the concordance between medical record data on race/ethnicity in the VA and race/ethnicity as recorded in inpatient administrative data files, finding good agreement.11 However, this finding is not surprising, given that medical record data serve as the source for inpatient data on race/ethnicity. The agreement of the administrative files with patient self-report was unknown, as were the sociodemographic and health factors associated with such agreement.

The purpose of this study was to extend previous research by examining the agreement of VA administrative data on race/ethnicity with patient self-reported race/ethnicity, using information obtained from the largest federal survey ever conducted in the Veterans Health Administration.12 Thus, in addition to examining general rates of agreement, we assessed the effect of including or excluding patients with missing race/ethnicity information or with multiple race/ethnicity designations in the survey data. A secondary goal of our study was to identify any sociodemographic and health characteristics of patients associated with agreement between data sources.

METHODS

Data Sources

VA administrative files.

The VA maintains administrative files on inpatient and outpatient care received by veterans for each fiscal year. The Patient Treatment File (PTF) includes patient-level information about short-term discharges from VA inpatient care, including demographic and summary information about each episode of care. The Outpatient Clinic File (OPC) provides information on each outpatient visit in the VA and is organized by visit day. This file contains information about each patient, including sociodemographic characteristics. From the PTF and OPC files for 1996 to 1998, we created a single file with records for every veteran patient who received inpatient or outpatient care provided or paid for by the VA in the 3 years preceding administration of the survey described below.

1999 Large Health Survey of Veteran Enrollees.

In 1999, the largest and most detailed survey of veterans using VA health services was conducted to ascertain their health status and health practices.12 Patients were sampled from the March 1999 enrollment file, which contained 3 760 200 enrollees. After exclusion of 146 323 veterans who had died or who had “bad” addresses, the final mailable sampling frame was 3 613 877. A total of 1 406 049 enrollees were sent surveys by mail using a stratified random sample (those who died or who were ineligible because of bad addresses were excluded), and a total of 887 775 surveys were received, resulting in a response rate of 63%. These surveys included questions about the patient’s race/ethnicity, other basic sociodemographic characteristics, and health. We excluded patients whose race/ethnicity was missing, and for most analyses, we excluded patients who indicated more than 1 race/ethnicity, leaving an analysis sample of 730 149.

Measures

VA administrative data on race/ethnicity.

Race/ethnicity information is recorded in both inpatient and outpatient VA administrative files, which exist for each fiscal year. In the outpatient files, race/ethnicity designations are assigned by the registration clerk on the basis of visual inspection. For inpatient files, the race/ethnicity information is extracted from the medical record documentation provided by the clinician. In the summary file we used, each person-level record contains several variables related to patient birthdate or age, gender, and race/ethnicity. Because in other work with these files we had noted that information on race/ethnicity sometimes varied from file to file, we created a summary race/ethnicity variable based on all the indicators for race/ethnicity in the inpatient and outpatient utilization files for the 3 years represented. For patients with more than 1 value (e.g., both African American and White) across all utilization records, we assigned the value present most often in the patient’s records. This decision affected only a small number of cases; for each of the 3 race/ethnicity categories, the majority (99.8%) of patients had consistent codes. For patients without a majority value or for whom there was no race/ethnicity information, we assigned a value of “unknown.” Thus, each patient’s record indicated whether the patient’s race/ethnicity was White, African American, other (Hispanic, American Indian, Asian and Pacific Islander were grouped together in the file during its preparation), or unknown. Patients in the file who had been seen on an outpatient basis only in 1996, when race/ethnicity information was not yet included in the administrative database, were excluded from our analysis sample.

Large Survey data on race/ethnicity.

Our measurement of race/ethnicity was the patient’s self-reported race/ethnicity provided in the 1999 Large Health Survey of Veteran Enrollees in response to a question developed by the Office of Management and Budget for use in federal surveys.13 Patients were asked to indicate their race/ethnicity in response to the following single question, “What is your race/ethnicity?” Patients were instructed to mark all responses that applied, including “American Indian or Alaskan native,” “Asian,” “Black or African American,” “Spanish, Hispanic, or Latino,” “Native Hawaiian or other Pacific Islander,” or “White.” We excluded patients from our sample whose race/ethnicity was missing from this file (n = 115 349; 13.1% of the original sample). Patients who had indicated more than 1 race/ethnicity (n = 34 113; 3.8% of the original sample) were also excluded from some analyses, leaving a sample of 730 149 veterans.

Sociodemographic characteristics.

Age (in 1998) was taken from the administrative data. Patients were asked to report their educational level by selecting 1 of the following responses: “never attended school or only kindergarten,” “grades 1 through 8,” “grades 9 through 11,” “grades 12 or GED (general equivalency diploma),” “college 1 year to 3 years,” or “college graduate or graduate school.”

As in the Behavioral Risk Factor Surveillance System questionnaire, patients were asked, “In the past 30 days have you been concerned about having enough food for you or your family?” (yes or no) to determine sufficiency of food.14

For health status, we used a single item that assesses general health, drawn from the Veterans Short Form (SF)-36 (“In general, would you say your health is excellent, very good, good, fair or poor?”).15

Patients were asked whether they lived alone (yes or no) and to indicate whether they were married, divorced, separated, widowed, or never married. For the purposes of our analyses, we dichotomized this variable, creating 2 groups: currently married and other.

For employment status, patients were asked whether they were currently employed for wages, self-employed, looking for work and unemployed for more than 1 year, looking for work and unemployed for less than a year, retired, homemaker, student, or unable to work. These responses were grouped as employed, retired, unemployed, or “other.” For some analyses, these categories were further subdivided as employed or not employed.

Under health care utilization, we calculated total number of inpatient stays between 1996 and 1998 and total number of outpatient visits between 1996 and 1998 from the administrative data.

Analyses

We conducted cross-tabulations between self-reported and administrative data, examining percentage agreement between the 2 data sources (administrative vs self-report data on race/ethnicity). Next, we conducted multivariate logistic regression analyses to identify sociodemographic characteristics, health factors, and health care utilization patterns associated with the likelihood of agreement in race/ethnicity between the 2 databases.

RESULTS

Sociodemographic characteristics of the sample are shown in Table 1 ▶. The majority of the sample were White (79.7%); 26% of the total had less than a high school education, 62% were married, nearly 25% lived alone, just over 25% were employed, and more than 10% were concerned about having enough food. On average, patients had 0.5 inpatient stays and 28 outpatient visits during 3 years, and self-reported health was fair to good (3.6 on a 5-point scale).

TABLE 1—

Sociodemographic Characteristics of Sample (N = 730 149)

Mean age in 1998, y	62.6
Race/ethnicity (from self-reported data), %
White	79.7
African American	13.2
Hispanic	5.2
Asian	0.6
Native American	1.1
Pacific Islander	0.2
Education, % with less than high school	26.1
Married, %	62
Employed, %	27.4
Live alone, %	23.1
Concerned about food sufficiency, %	13.7
No. of inpatient stays in past 3 years, mean (SD)	0.51 (1.34)
No. of outpatient visits in past years, mean (SD)	28.0 (43.0) 3
General health status (GH1), mean (SD)	3.5 (1.01)

Note. GH1 = Response to General Health question 1 from the Veterans SF-3615 (“In general, would you say your health is excellent, very good, good, fair or poor?”).

We examined agreement in the racial designations between the 2 files, as shown in Table 2 ▶. The percentage agreement was calculated for each race/ethnicity group reported from the survey. Thus, for Native Americans, Asians, Pacific Islanders, and Hispanics, agreement existed when people who selected these groups in the survey were classified as “other” in the administrative data. The results indicated that whereas rates of agreement were very similar for Hispanics, African Americans, and Whites (approximating 60%), agreement rates were markedly lower for Pacific Islanders (38.4%), Asians (36.6%), and Native Americans (15.8%).

TABLE 2—

Agreement Between Self-Reported Race/Ethnicity and Administrative Data on Race/Ethnicity

	Administrative Data on Race/Ethnicity
	Including Patients With Unknown Race/Ethnicity (n = 730 149)					Including Only Patients With Known Race/Ethnicity (n = 464 683)
Self-Reported Race/Ethnicity	White, %	African American, %	Other, %	Unknown, %	n	White, %	African American, %	Other, %	n
Native American	49.34	4.14	15.81	30.71	8317	71.21	5.97	22.82	5763
Asian	10.73	1.13	36.57	51.57	4605	22.15	2.33	75.52	2230
African American	4.68	60.63	0.58	34.10	96 007	7.11	92.0	0.88	63 265
Hispanic	10.77	1.11	59.63	28.49	37 662	15.06	1.55	83.39	26 931
Pacific Islander	14.88	1.85	38.39	44.88	1727	27.00	3.36	69.64	952
White	61.50	0.44	0.88	37.17	581 831	97.90	.70	1.4	365 542

Note. Bold type indicates agreement between the 2 data sources.

Race/ethnicity was designated as unknown in the administrative files for 36% of patients. We therefore calculated percentage agreement when the veterans for whom race/ethnicity was unknown in the administrative files were excluded. As shown in the right-hand columns of Table 2 ▶, the agreement for most groups increased dramatically, to 97.9% for Whites, 92.0% for African Americans, 75.5% for Asians, 83.4% for Hispanics, and 69.6% for Pacific Islanders. Agreement increased to only 22.8% for Native Americans.

Patients with more than 1 self-reported race/ethnicity presented more opportunities for concordant classification with the administrative data (e.g., a man who considers himself both White and Native American had 2 chances for being administratively classified in a category consistent with 1 of his selfdesignations). Thus, we conducted an additional analysis to examine concordant classifications between all patients with 1 or 2 self-designations of race/ethnicity (99.6% of the sample) and the administrative records of race/ethnicity. As shown in Table 3 ▶, for patients who reported some combinations of race/ethnicities, the concordance between administrative and self-report classifications increased dramatically when the multiple racial classifications were taken into account. For example, patients self-reporting as both Hispanic and White had a total of 70.1% concordance with administrative data when both categories were taken into account, as opposed to 45% or 26% when only 1 category, Hispanic or White, was considered. Similarly, patients reporting combinations of African American and White or African American and Hispanic were approximately twice as likely to have self-designations concordant with the administrative data if 2 racial categories were included. For other combinations, the addition of a second racial category made much less difference, adding 0% to 7% more likelihood of concordance between the 2 data sources.

TABLE 3—

Agreement Between Self-Reported and Administrative Data: Patients Indicating 1 or 2 Races/Ethnicities

	Administrative Data on Race/Ethnicity, %
Self-Reported Race/Ethnicity	White	African American	Other	Unknown	Percentage Agreement
White (n = 581 831)	61.50	0.44	0.88	37.17	61.50
Pacific Islander (n = 1727)	14.88	1.85	38.39	44.88	38.39
Pacific Islander, White (n = 1877)	62.92	0.48	2.34	34.26	65.26
Hispanic (n = 37 662)	10.77	1.11	59.63	28.49	59.63
Hispanic, White (n = 4627)	25.95	0.76	44.64	28.65	70.59
Hispanic, Pacific Islander (n = 207)	12.56	1.45	53.14	32.85	53.14
Black (n = 96 007)	4.68	60.63	0.58	34.10	60.63
Black, White (n = 490)	24.90	36.53	2.24	36.33	61.43
Black, Pacific Islander (n = 35)	17.14	40.00	0.00	42.86	40.00
Black, Hispanic (n = 478)	5.65	29.71	30.54	34.10	60.25
Asian (n = 4605)	10.73	1.13	36.57	51.57	36.57
Asian, White (n = 501)	52.89	0.40	6.59	40.12	59.48
Asian, Pacific Islander (n = 194)	9.28	0.52	38.14	52.06	38.14
Asian, Hispanic (n = 73)	13.70	2.74	34.25	49.32	34.25
Asian, Black (n = 941)	5.21	59.94	1.81	33.05	61.75
Native American (n = 8317)	49.34	4.14	15.81	30.71	15.81
Native American, White (n = 18 950)	66.32	0.47	1.00	32.21	67.32
Native American, Pacific Islander (n = 24)	50.00	4.17	20.83	25.00	20.83
Native American, Hispanic (n = 647)	17.62	1.55	45.13	35.70	45.13
Native American, Black (n = 2081)	6.58	58.29	1.01	34.12	59.30
Native American, Asian (n = 119)	39.50	9.24	18.49	32.77	18.49

Note. Bold type indicates agreement between racial designations in administrative and self-report data sources.

Next, we examined factors associated with agreement between types of records (results not shown in tables). Again focusing on patients with only 1 racial designation, we conducted bivariate comparisons of patients for whom there was agreement between the administrative and survey data with patients for whom there was not on sociodemographic, health, and health care utilization variables. We included patients whose race/ethnicity was unknown in the administrative data. Compared with patients for whom there was no agreement, patients for whom there was agreement regarding race/ethnicity were less likely to have more than a high school education (29% vs 21%), less likely to be employed (22% vs 35%), more likely to indicate that having sufficient food was a problem (15% vs 12%), more likely to live alone (25% vs 20%), and less likely to be married (59% vs 66%; all P < .0001, using χ² tests). Patients with agreement regarding race/ethnicity had worse self-perceived general health (3.68 vs 3.29; a higher score indicates worse health), more inpatient stays (0.80 vs 0.07), and more outpatient visits (37.2 vs 13.6; all P < .0001, using t tests).

Finally, we conducted multivariate logistic regression analyses to examine the unique association of sociodemographic and health care utilization variables with known race in the administrative data, and then to examine the association of these variables with agreement on race/ethnicity between administrative and self-report data (Table 4 ▶). First, among all patients, we examined sociodemographic and health factors associated with known race in the administrative data. Compared with patients for whom race was unknown, patients with complete race information in the administrative data tended to be older, unmarried, less educated, and unemployed; to have insufficient food; to have more inpatient stays and outpatient visits; to be of “other” race/ethnicity, and to report worse health status. Second, when we restricted our analysis to patients for whom race was known in the administrative data, we found that those more likely to show agreement between the 2 data sources on race tended to be younger, White, non–solitary living, and less educated; to have no problems with food sufficiency; and to have more inpatient stays.

TABLE 4—

Results of Multivariate Logistic Regression

	Factors Associated With Agreement on Racial Classificationa			Factors Associated With Known vs Unknown Racial Classification
Independent Variables	Odds Ratio	Wald 95% Confidence Interval	P	Odds Ratio	Wald 95% Confidence Interval	P
Age	1.00	1.00, 1.00	.0448	1.01	1.01, 1.01	< .0001
Education > high school	0.83	0.80, 0.87	< .0001	0.81	0.80, 0.82	< .0001
Employment		.4004			< .0001
Employed	1.14	0.96, 1.35	0.95	0.88, 1.02
Retired	1.16	0.97, 1.40	0.93	0.86, 1.00
Unemployed	1.15	0.97, 1.37	1.14	1.06, 1.22
Other	Reference		Reference
Food sufficiency a problem	0.95	0.91, 0.99	.0098	1.08	1.06, 1.10	< .0001
Live alone	0.94	0.89, 0.99	.0113	1.00	0.98, 1.02	.7979
Married	1.01	0.96, 1.05	.8098	0.88	0.87, 0.90	< .0001
Sum of inpatient stays	1.03	1.02, 1.04	< .0001	8.59	8.32, 8.88	< .0001
Sum of outpatient visits	1.00	1.00, 1.00	.7043	1.04	1.04, 1.04	< .0001
Self-reported race/ethnicity			< .0001			< .0001
White	Reference		Reference
African American	0.25	0.24, 0.26	0.98	0.97, 1.00
Other	0.06	0.06, 0.06	1.34	1.30, 1.37
General health status (GH1)	1.00	0.98, 1.02	.8169	1.15	1.14, 1.16	< .0001

Note. GH1 = Response to General Health question 1 from the Veterans SF-36¹⁵ (“In general, would you say your health is excellent, very good, good, fair or poor?”).

^aIncludes only patients whose race was known in administrative data.

DISCUSSION

The purpose of this study was to examine agreement between administrative data on race/ethnicity in VA data files and patients’ self-reported race/ethnicity and to identify sociodemographic and health characteristics of patients who have agreement on race/ethnicity between the 2 data files. Our results indicated fairly poor overall agreement between administrative data and self-reported race/ethnicity; the best rates of agreement (for Whites and African Americans) were approximately only 60%, and rates of agreement were even lower for other racial groups. We found very high levels of missing or unknown race/ethnicity in the administrative data files, and not surprisingly, when we deleted patients for whom race/ethnicity was unknown, we noted markedly higher rates of agreement between the administrative and self-report data. When VA claimed to know the race/ethnicity, the data agreed with the patient’s self-report over 90% of the time for Whites and African Americans, over 80% of the time for Hispanics, and about 70% of the time for Asians and Pacific Islanders, but still only about 20% of the time for Native Americans. Thus, a strategy the VA may consider to enhance the agreement of the administrative race/ethnicity data with self-report data is to decrease levels of patients with unknown race/ethnicity by either supplementing data from the Large Survey data files or by embarking on new efforts to gather information on race/ethnicity from VA patients for whom race/ethnicity is currently unknown. Indeed, a new policy on the collection of race data implemented by VA in 2003 will improve the quality of newly collected data, because the policy makes the completion of both race and ethnicity data fields mandatory and includes an indicator for specifying how the determination was made (visual or self-identification).

One other possible strategy for improving the concordance between VA and selfreported race/ethnicity data is to consider patients’ multiple races/ethnicities when making administrative classifications (e.g., allowing patients to indicate more than 1 race/ethnicity, as is done on the US Census), thereby allowing more opportunities for administrative designations to match patients’ self-designations. Our analyses showed that this approach would improve agreement when patients considered themselves combinations of Hispanic, White, or African American, but not when other combinations of races/ethnicities are involved. However, given that these 3 combinations of racial classifications accounted for only 0.06% of our total sample, this strategy is unlikely to have a large effect on the overall quality of the administrative race/ethnicity data.

In the bivariate results, the agreement between VA race/ethnicity data and self-report was similar for White, African American, and Hispanic patients and notably lower for Asian, Pacific Islander, and Native American patients. However, the logistic regression results indicated that after control for a variety of sociodemographic factors, African Americans and “others” (including Hispanic, Asian, Pacific Islander, and Native American patients) were less likely than Whites to have agreement between the data sources. This implies that studies examining racial disparities in health care within the VA are particularly likely to have poor agreement with selfreported race/ethnicity data for non-White patients. However, because African Americans were described as White almost 5% of the time, whereas Whites were described as African American only 0.44% of the time, estimates of racial disparities between these 2 groups are likely to be diminished owing to the characteristics of the administrative database. Similarly, Hispanic patients were listed as White almost 11% of the time, whereas Whites were listed as “other” less than 1% of the time; estimates of disparities between these 2 groups are also likely attenuated by the classification as White of a significant proportion of Hispanic patients in the administrative data.

To illustrate this point, consider a prominent VA study on racial disparities in cardiac care. Whittle and colleagues6 observed significantly different crude rates of cardiac catheterization, 19.3% for Whites and 11.8% for African Americans; they observed rates of 1.8% and 0.8%, respectively, for percutaneous transluminal coronary angioplasty; and they observed rates of 5.0% and 1.6%, respectively, for coronary artery bypass grafting. If African Americans were misclassified as White 5% of the time and Whites were misclassified as African American 0.44% of the time, as in our findings, correct identification of race/ethnicity may actually have inflated the procedure rates for Whites and reduced them for African Americans. This assumes that the African Americans misclassified as Whites had rates of invasive procedures similar to those of other African Americans. Consequently, use of self-reports of veterans’ race/ethnicity could actually have magnified the racial differences in procedure use observed by Whittle et al. Our results highlight important implications of the quality of VA data on race/ethnicity for past and present research findings.

These results extend those in the previous literature because of their focus on administrative data from the VA’s national databases. This focus is an important addition to the field, as so many studies of health and health disparities rely on the VA’s data on race/ethnicity.6–8,10 The results also extend those of Boehmer et al.5 by detailing the effects on rates of agreement of excluding patients with unknown race, as well as by elucidating the sociodemographic and health factors associated with available race data and with agreement between self-reported and administrative data on race.

This study was limited by its reliance on a summary file of VA administrative data on race/ethnicity, which grouped individuals of Hispanic, Pacific Islander, Asian, and Native American race/ethnicity together. Because all these groups were included in the “other” category, the level of detail originally available in the VA databases for each racial category was eliminated in this file. Thus, we could not report levels of agreement with self-reported race/ethnicity within each of these 4 categories. However, the benefit of using this file is that the file summarizes 3 years of administrative data on race/ethnicity, as opposed to containing data from a single year.

Our exploration of factors associated with agreement between self-reported race/ethnicity and race/ethnicity in the administrative data files indicated that younger, lesseducated patients who possess some social and material resources and who use VA inpatient care more often are likely to have higher levels of agreement between race/ethnicity information in the 2 databases we studied. In contrast, patients whose administrative data is most likely to include known race have consistently fewer social and economic resources, report worse health, and use more inpatient and outpatient VA care. Other studies have noted that users of VA care are more likely to have a low family income and low labor force participation and are less likely to have a family physician or private health insurance.16 Our results suggest that even within this socioeconomically challenged population, those using the system more often are more likely to have agreement between self-report and administratively classified race/ethnicity. Thus, results indicate that the more opportunities the VA has to record race/ethnicity, the more likely its data are to agree with patient self-reports.