Different Data Systems, Different Conclusions? Comparing Hospital Use Data for the Aged from Four Data Systems

Abstract

Four major national data systems collect data on short-stay hospital use by persons age 65 years and over. This paper examines the extent of agreement on the major statistics reported from the four systems and explains, as far as possible, the reasons for discrepancies among the data. The paper also offers some suggestions to the user of hospital care data.

The study shows that for national trends the four systems agreed on number of admissions, average length of stay, and days of care. Comparisons across census regions revealed agreement on average length of stay, but showed an unanticipated lack of agreement on admissions. When we examined data on admissions and average length of stay by diagnosis and surgical procedure groups there was agreement among the data systems for most groups, but for certain groups wide differences occurred.

The results emphasize the need for data users to understand the nature and limitations of the data they employ. The results also point to a need for data users to consult a number of sources whenever possible.

Introduction

National Hospital Data Systems

Since the mid-1960's four major national data systems have been collecting data on short-stay hospital use by persons age 65 years and older. These data systems are the National Center for Health Statistics (NCHS) National Hospital Discharge Survey (NHDS); the NCHS National Health Interview Survey (NHIS); the American Hospital Association (AHA) National Hospital Panel Survey;¹ and the Health Care Financing Administration's (HCFA) Medicare Statistical System.² From their inception, these data systems have been used extensively in many studies about the health care delivery system.

The NHDS is based on abstracts of hospital stays collected from a sample of hospitals. It focuses on the demographic characteristics of patients and on their diagnoses, surgical operations, and length of stay. The NHIS uses household interviews of the noninstitutionalized population to obtain a broad range of data on demographic characteristics of interviewees, illnesses, chronic conditions, impairments, health service use, and other health topics. The AHA National Hospital Panel Survey gathers summary hospital use and financial information through questionnaires submitted monthly by a sample of hospitals. Medicare's hospital data system is based upon master files containing information on enrollees and participating hospitals which are linked to records of claims submitted by Medicare's fiscal intermediaries containing information on hospital use and reimbursement. The system contains records of all claims, although diagnostic and surgical information is submitted by the fiscal intermediaries and coded in Medicare's central office for only a 20 percent sample. Appendix A, “Description of Data Systems,” gives more detailed information about the four systems.

Each of the systems employs different data gathering methods and definitions. Each reports items not available from the others, yet all contain some variables in common with the other systems. Because data from each system are used to address significant hospital care issues, this paper compares data from these systems for the population age 65 years and over to determine whether or not different conclusions on basic indicators of hospital use would be reached, depending on the data system used. To do this, the paper first measures how closely the selected statistics from the four systems agree with each other. Where substantial discrepancies are found, the paper attempts to explain the reasons for the differences by relating the discrepancies to the nature of the systems under study. Finally, the paper offers some suggestions to the user of hospital care data who must often choose among data from various sources in the course of a study.

In the first part of the paper, we compare national trends from each data source for the period 1967 to 1979 on the three major overall measures of hospital use—admissions (or in the cases of the NHDS and the NHIS, discharges³), average length of stay (ALOS), and days of care (DOC). Reliable information about trends in hospital admissions is of particular importance since analyses of Medicare data indicate a marked upward trend in hospital admissions for the aged population. Next, this paper compares admissions and ALOS by census region for 1972, 1975, and 1977.⁴ We hypothesized that there would be differences in the levels of estimates from each source for any one year, but that time trends and patterns of cross-sectional variation across large geographic areas would be the same.

In the second part of the paper, we compare number of discharges for selected diagnostic groups as reported by the NHDS and Medicare for 1972 and 1976.⁵ In addition, because of the current concern about possible excess surgeries, we compare numbers of operations from the NHDS and Medicare for nine common operations. Two of the systems, the NHIS and the AHA, were not included in the diagnostic and surgical comparisons. The NHIS was excluded because it publishes diagnostic and surgical data by general classes rather than by specific diagnostic and surgical codes, and also because its exclusion of institutionalized persons and deceased persons makes cross-sectional comparisons with NHDS and Medicare data difficult. These exclusions are explained in more detail in Appendix A, “Description of Data Systems.” The AHA was excluded because it does not collect any diagnostic data, and for surgical data it collects only the total number of surgeries for all age groups combined.

Reliability of Diagnostic and Surgical Data

We hypothesized that there might be large differences in the diagnostic and surgical data from Medicare and the NHDS, especially in light of the studies by the Institute of Medicine (IOM 1977, 1980). In the IOM studies, information for a sample of cases—selected from the files of Medicare and the NHDS—was abstracted from the hospital medical records by a trained field team. The abstracted information was compared with the original information in the files to determine the rate of agreement and the causes of discrepancies. The IOM studies found that general information, such as admission and discharge dates, was reported accurately. For diagnostic and surgical information, there were often serious discrepancies between the codes for the principal diagnosis and surgical procedure found in the data systems under study and the codes assigned by the IOM field team. The discrepancy rate varied greatly by diagnostic and surgical category, with some diagnoses (such as cataract) showing a negligible rate of discrepancy, and others (such as chronic ischemic heart disease) having a discrepancy rate as high as 63 percent for codes at the four-digit level. The IOM studies attributed the high rate of discrepancy largely to problems in selecting the principal diagnosis and procedure from the medical record, rather than errors due to coding. The following table summarizes the results of the IOM studies. The data show the percent of records with no discrepancy between the Institute of Medicine field team and the original data source.

	Weighted Percent of Records With No Discrepancy
Data Item	Medicare Record (1974 Data)	NHDS Abstracts (1977 Data)
Admission Date	99.5	99.3
Discharge Date	99.3	97.9
Principal Diagnosis (4 digits)	57.2	63.4
Principal Diagnosis (3 digits)	61.9	73.2
Principal Procedure1	78.9	71.4

Source: From IOM, 1980, pp. 58 and F1 and IOM, 1977, pp. 27 and 40.

¹The percent of records with no discrepancy on principal procedure includes cases without surgery. When the analysis is limited to surgical cases, the percent with no discrepancy on principal procedure drops to 57 percent for Medicare and 60 percent for NHDS.

The IOM studies also included a section in which the work of the IOM field teams was evaluated. A sample of hospital medical records that had been abstracted by the field team was re-abstracted by the expert consultant who trained the field team. This procedure also resulted in rather high discrepancy rates for diagnostic and surgical data, suggesting the complexities involved in medical abstracting, and illustrating that there may be a certain discrepancy rate inherent in the data abstracting process. The following table summarizes the results of the evaluation of the IOM field team and shows the percent of records on which the field team and the consultant agreed.

Data Item	Weighted Percent of Records With No Discrepancy
	Medicare	NHDS
Admission Date	99.6	99.6
Discharge Date	99.2	97.8
Principal Diagnosis (4 digits)	75.8	71.7
Principal Procedure1	88.4	88.1

Source: From IOM, 1980, p. D3; and IOM, 1977, p. 108.

¹The percent of records with no discrepancy on principal procedure includes cases without surgery. If the analysis were restricted to cases with surgery, the percent of cases with no discrepancy would decline.

A different study of quality control in hospital discharge data systems provides further reasons to compare similar data from different data systems (Corn, 1980). The study focused on quality control procedures used in different systems, including the NHDS and Medicare, and found some shortcomings in each system studied. The study found that the NHDS maintained well-documented quality control procedures although it raised questions about whether the NHDS' sole reliance on the face sheet of the medical record—rather than the entire record—might limit the accuracy of its data.⁶ The study noted that the Medicare Statistical System had not established quality control requirements or computer edits at the national level for many data items, including diagnosis and procedure, and that such efforts were left to the discretion of the Medicare fiscal intermediaries.⁷

Results

National Trends from 1967 to 1979 in ALOS, Admissions, and Days of Care

Figure 1 shows the NHDS, NHIS, AHA and Medicare data for ALOS for persons age 65 years and over in short-stay hospitals from 1967 to 1979 in the United States.⁸ The trends shown by the NHDS the AHA and Medicare Were Similar—ALOS increased slightly or stayed the same from 1967 to 1968 and then declined steadily from 1968 to 1979.⁹ ALOS estimates from the NHIS show considerable fluctuation, but they indicate the same basic trend as the other three sources. The NHIS may fluctuate more than data from the other sources because, unlike the other systems which gather information from written records, NHIS data are subject to recall errors. With the exception of the NHIS data, the estimates from the different sources show a consistent relationship from 1967 to 1979, with the NHDS giving the highest values, followed by Medicare, and then by the AHA. We could not determine why the NHDS estimates were consistently the highest and the AHA's the lowest. However, for most years, the difference in ALOS from the NHDS, Medicare and the AHA was less than a day.

Figure 1. Comparison of Short-Stay Hospital Average Length of Stay (ALOS) as Reported by the NHDS, Medicare, the AHA, and the NHIS, for Persons Age 65 Years and Over, U.S., 1967-1979.

Figure 2 shows data on admissions to short-stay hospitals from the four sources. Each data source shows the same trend for admissions—a steady rise in the number of admissions for the aged from 1967 to 1979. The NHIS data again fluctuate more than the other data. The NHIS also gives the lowest estimates due to their survey's exclusions of deceased persons, and persons in institutions. As illustrated in Table 1, which shows the number of admissions from each data source as a percent of the NHDS admissions, the NHIS percentages are substantially lower than the other data sources.

Figure 2. Comparison of Short-Slay Hospital Admissions as Reported by the NHDS, Medicare, the AHA, and the NHIS, for Persons Age 65 Years and Over, U.S., 1967-1979.

Table 1. Number of Short-Stay Hospital Admissions (or Discharges) by Data Source, for Persons Age 65 and Over, U.S., 1967—1979.

Year	NHDS	Medicare1	Medicare As a Percent of NHDS	AHA	AHA As a Percent of NHDS	NHIS	NHIS As a Percent of NHDS
	Discharges (in Thousands)	Admissions (in Thousands)		Admissions (in Thousands)		Discharges (in Thousands)
1967	5,215	5,241	100.5	5,337	102.3	3,543	67.9
1968	5,520	5,927	107.4	5,711	103.5	4,113	74.5
1969	5,694	5,954	104.6	6,058	106.4	4,491	78.9
1970	5,883	6,044	102.7	6,216	105.7	4,443	75.5
1971	5,986	6,227	104.0	6,428	107.4	4,565	76.3
1972	6,634	6,520	98.3	6,823	102.8	5,225	78.8
1973	6,937	6,771	97.6	7,213	104.0	4,816	69.4
1974	7,185	7,067	98.4	7,647	106.4	5,271	73.4
1975	7,654	7,320	95.6	7,992	104.4	5,318	69.5
1976	7,912	7,706	97.4	8,553	108.1	6,039	76.3
1977	8,344	8,044	96.4	8,927	107.0	6,114	73.3
1978	8,708	8,308	95.4	9,362	107.5	6,101	70.1
1979	9,086	8,605	94.7	9,856	108.6	6,301	69.3

¹Medicare admissions are for “All Areas” including Puerto Rico, the Virgin Islands, Guam, and so forth, and thus, are higher than the United States totals in Table 4.

The next lowest estimates in the years from 1967 to 1971 were given by the NHDS. In these years, the NHDS probably understated the numbers of admissions. The estimates for these years came from a sample of hospitals selected in 1965. Since hospitals could leave the sample by closure, merger, and so forth, but no new hospitals could enter the sample, the sample reflected hospitals operating in 1965, but did not reflect the changes occurring since then. With the inclusion in 1972 of a sample of hospitals that came into existence between 1965 and 1969, the NHDS estimate increased markedly from 1971 to 1972. The sample was again updated in 1975, 1977, and 1979 (Moien, 1978). Since 1972, the NHDS has reported about 3 percent more admissions than Medicare. This finding was expected, because an estimated 2 to 5 percent of the population age 65 and over are not covered by Medicare for hospital insurance.

For every year except 1968 the AHA reported more admissions for the aged than the other systems reported. This finding was unexpected since the universe of hospitals for the AHA survey is essentially the same as the universe for the NHDS; therefore, the number of admissions from the two systems should be close. In fact, as shown in Table 2, the estimates of total admissions for all ages are in close agreement, especially in the years after 1971 when the NHDS began to update its sample list of hospitals every 2 or 3 years. Since total NHDS admissions are consistent with Medicare figures, the problem may lie with the proportion of admissions reported as aged by AHA. The table below shows that the percent of admissions accounted for by persons age 65 and over has been consistently higher for the AHA compared to the NHDS.

Table 2. Number of Short-Stay Hospital Admissions (or Discharges) reported by the NHDS and the AHA, for all Ages and for Persons Age 65 and Over, U.S., 1967—1979.

Year	All Ages			65 and Over
	NHDS	AHA	AHA As a Percent of NHDS	NHDS	AHA	AHA As a Percent of NHDS
	Discharges (in Thousands)	Admissions (in Thousands)		Discharges (in Thousands)	Admissions (in Thousands)
1967	27,964	27,202	97.3	5,215	5,337	102.3
1968	28,070	27,768	98.9	5,520	5,711	103.5
1969	28,534	28,353	99.4	5,694	6,058	106.4
1970	29,185	30,127	103.2	5,883	6,216	105.7
1971	29,459	30,260	102.7	5,986	6,428	107.4
1972	31,627	31,036	98.1	6,634	6,823	102.8
1973	32,125	32,122	100.0	6,937	7,213	104.0
1974	33,018	33,309	100.9	7,185	7,647	106.4
1975	34,043	33,409	98.1	7,654	7,992	104.4
1976	34,372	34,532	100.5	7,912	8,553	108.1
1977	35,902	35,404	98.6	8,344	8,927	107.0
1978	35,616	35,561	99.8	8,708	9,362	107.5
1979	36,747	36,508	99.3	9,086	9,856	108.5

Admissions (or Discharges) of Persons Aged 65 and Over as a Percentage of Total Admissions (or Discharges).

	1967	1968	1969	1970	1971	1972	1973	1974	1975	1976	1977	1978	1979
NHDS (Discharges)	18.6	19.7	20.0	20.2	20.3	21.0	21.6	21.8	22.5	23.0	23.2	24,4	24.7
AHA (Admissions)	19.6	20.6	21.4	20.6	21.2	22,0	22.5	23.0	23.9	24.8	25.2	26.3	27.0

None of the explanations examined so far has been able to account for the reasons for the difference in the AHA and NHDS estimates. However, the unnumbered table on page 4 also makes clear that the two sources show the same time trend—the elderly account for an increasing percentage of total admissions.

These data point out two caveats for researchers in using estimates of admissions of the population age 65 and over. First, a researcher should not switch from one source to another when examining trends. For example, if one went from the 1977 AHA estimate of 25.2 percent of admissions accounted for by the elderly to the 1978 NHDS estimate of 24.4 percent, one might conclude that the upward trend in the percentage of admissions accounted for by the elderly had been interrupted. In fact, both sources actually show an increase from 1977 to 1978. Second, and perhaps more important, the data show the difficulty in arriving at precise estimates of the number of hospitalizations of the elderly. Even though both the AHA and the NHDS report on the same universe, their estimates in recent years have differed by about 7 or 8 percent, or from 583 to 770 thousand admissions. Any analyses based on numbers of admissions must give a range of estimates until the differences can be reconciled.

Figure 3 compares each system's figures for total days of care for the aged. The estimates from the NHDS, Medicare and the AHA show the same trend—a rise from 1967 to 1968 or 1969; a slight decline until 1971; and a gradual rise thereafter. The NHIS shows the greatest fluctuations in days of care, as it did for average length of stay and admissions. It also gives the lowest count of days of care. Again, as for admissions, Medicare has given the next lowest estimate since 1972, because it does not cover the entire 65 and over population.

Figure 3. Comparison of Short-Stay Hospital Days of Care (DOC) as Reported by the NHDS, Medicare, the AHA, and the NHIS, for Parsons Aga 65 Years and Over, U.S., 1967-1979.

Regional variations in hospital use have been the subject of a number of studies. The next section examines regional variations in ALOS and admissions reported by the four data systems under study.

Regional Comparisons

The estimates of ALOS from each data system for the four census regions are similar for the selected years 1972,1975 and 1977, as Table 3 illustrates. With the exception of the NHIS for 1977, all data sources show the highest ALOS in the Northeast, followed by the North Central, South and West.

Table 3. Average Length of Stay (ALOS) in Short-Stay Hospitals by Data Source, for Persons Age 65 and Over, by Census Region, 1972, 1975 and 1977.

Region	NHDS		Medicare		AHHA		NHIS
	ALOS (in Days)	Rank	ALOS (in Days)	Rank	ALOS (in Days)	Rank	ALOS (in Days)	Rank
1972
United States	12.2		12.1		11.7		12.9
Northeast	14.8	1	14.6	1	13.9	1	16.9	1
North Central	12.5	2	12.6	2	12.6	2	13.4	2
South	11.3	3	11.0	3	10.6	3	12.1	3
West	10.0	4	9.7	4	9.8	4	9.4	4
1975
United States	11.6		11.2		11.2		12.0
Northeast	14.1	1	13.6	1	13.6	1	14.0	1
North Central	12.1	2	11.6	2	12.1	2	12.5	2
South	10.5	3	10.3	3	10.1	3	11.3	3
West	9.3	4	9.0	4	9.1	4	10.1	4
1977
United States	11.1		10.9		10.7		11.1
Northeast	13.6	1	13.3	1	13.1	1	13.3	1
North Central	11.2	2	11.1	2	11.1	2	11.1	2
South	10.4	3	10.0	3	9.8	3	10.1	4
West	8.9	4	8.9	4	8.8	4	10.4	3

When we compared numbers and rates of admissions by census region among the four data sources, certain inconsistencies appear. Table 4 shows the number and percent of admissions or discharges reported by each data system, and Table 5 gives admission rates for 1972, 1975, and 1977. In 1972, Medicare, the NHDS, and the AHA all reported the highest admissions rate in the South and the lowest in the Northeast. The NHIS reported the South as second highest that year. In 1975, the South had dropped to second place according to the NHDS, although it was ranked highest by the other three Systems. By 1977, the South had dropped to third place as reported by the NHDS, although it was again ranked first by the other three systems. Thus, with the exception of the NHDS, all the data systems were in complete agreement in 1977 on the rankings of the four census regions on admission rates. It is puzzling that NHDS does not agree with the other three sources.¹⁰ Staff members at the NCHS are investigating this question, but have not yet uncovered any definite reasons for the difference.

Table 4. Number and Percent Distribution of Short Stay Hospital Admissions (or Discharges) by Data Source, for Persons Age 65 and Over, by Census Region, 1972, 1975 and 1977.

	NHDS		Medicare		AHA		NHIS
Region	Discharges (in Thousands)	Percent	Admissions (in Thousands)	Percent	Admissions (in Thousands)	Percent	Discharges (in Thousands)	Percent
1972
United States	6,634	100.0	6,486	100.0	6.823	100.0	5,225	100.0
Northeast	1,507	22.7	1,410	21.7	1,512	22.2	1,050	20.1
North Central	1,948	29.4	1,916	29.5	1,888	27.7	1,514	29.0
South	2,204	33.2	2,156	33.2	2,340	34.3	1,695	32.4
West	975	14.7	1,004	15.5	1,083	15.9	966	18.5
1975
United States	7,654	100.0	7,270	100.0	7,992	100.0	5,318	100.0
Northeast	1,685	22.0	1,592	21.9	1,729	21.6	1,192	22.4
North Central	2,340	30.6	2,100	28.,9	2,224	27.8	1,425	26.8
South	2,477	32.4	2,447	33.7	2,708	33.9	1,862	35.0
West	1,152	15.0	1,132	15.6	1,331	16.6	839	15.8
1977
United States	8,344	100.0	7,983	100.0	8,927	100.0	6,114	100.0
Northeast	1,908	22.9	1,737	21.8	1,948	21.8	1,264	20.7
North Central	2,424	29.1	2,261	28.3	2,477	27.7	1,737	28.4
South	2,660	31.9	2,750	34.4	3,044	34.1	2,138	35.0
West	1,352	16.2	1,234	15.5	1,458	16.3	976	160

Table 5. Number of Short Stay Hospital Admissions (or Discharges) per 1,000 Persons, by Data Source, for Persons Age 65 and Over, by Census Region, 1972, 1975 and 1977.

Region	NHDS		Medicare		AHA		NHIS
	Discharges Per 1,000 Persons	Rank	Admissions Per 1,000 Persons	Rank	Admissions Per 1,000 Persons	Rank	Discharges Per 1,000 Persons	Rank
1972
United States	333		313		327		262
Northeast	299	4	266	4	285	4	203	4
North Central	349	2	327	2	322	3	275	3
South	355	1	341	1	362	1	276	2
West	313	3	310	3	331	2	312	1
1975
United States	359		330		356		250
Northeast	321	4	289	4	311	4	221	4
North Central	404	1	344	2	363	3	243	3
South	362	2	354	1	380	1	278	1
West	337	3	321	3	370	2	251	2
1977
United States	374		347		380		275
Northeast	352	4	307	4	339	4	235	4
North Central	404	1	360	2	390	2	290	2
South	363	3	376	1	402	1	292	1
West	371	2	329	3	379	3	273	3

Note: Each data system uses a different source for the population-at-risk for rate calculation. The NHDS uses estimates of the civilian noninstitutionalized population furnished by the Bureau of the Census. Medicare uses counts of Medicare enrollees. The AHA uses Bureau of the Census estimates of the resident population published in Current Population Reports, Series P-25. Estimates of the population-at-risk for the NHIS are developed from the Health Interview Survey itself.

Comparison of Diagnostic and Surgical Data

Estimates of the number of discharges for specific diagnoses and surgical procedures are used for many purposes, including: estimating the cost of various kinds of cases; studying morbidity patterns of hospitalized persons; and developing diagnostic profiles for case-mix based reimbursement schemes. Clearly, if the NHDS and Medicare yield very different estimates of the incidence of common diagnoses and operations, a study would arrive at different conclusions, depending on the data source used.

During the years for which this paper examines diagnostic and surgical data (1972 and 1976), Medicare and NHDS used different guidelines to identify the most important discharge diagnoses and surgical procedures, and neither used the definitions of the Uniform Hospital Discharge Data Set (UHDDS) of the National Committee on Vital and Health Statistics. The UHDDS defines the principal diagnosis as,“… the condition established after study to be chiefly responsible for occasioning the admission of the patient to the hospital for care” (National Committee on Vital and Health Statistics, 1980). The UHDDS guidelines for determining which of several procedures is the principal one are:

• “The principal procedure is one that was performed for definitive treatment rather than one performed for diagnostic or exploratory purposes, or was necessary to take care of a complication.

• The principal procedure is that procedure most related to the principal diagnosis.”

For the 20 percent sample of hospital stays for which Medicare codes the diagnosis and procedure, Medicare instructed hospitals to identify the “primary” discharge diagnosis and all additional diagnoses in the hospital bill, but only the primary diagnosis for each hospital stay was coded. The primary diagnosis was defined as, “… the diagnosis of the illness or condition which was the primary reason for the patient's hospitalization” (Medicare Hospital Manual, 1975). For hospital stays where there was more than one procedure, Medicare asked hospitals to list first the procedure, “… related to the primary diagnosis.” If more than one procedure was listed, Medicare coded only the procedure listed first. In 1980 Medicare adopted the UHDDS definitions of principal diagnosis and procedure. The implementation of the change depends on the extent that hospitals use the UHDDS definition.

The NHDS records diagnoses in the order they appear on the face sheet of the hospital medical record, coding up to five diagnoses. Before 1977, the NHDS assumed that the first-listed diagnosis would usually be the most important one (IOM, 1980). In 1977, the NHDS adopted the UHDDS definition of principal diagnosis. The implementation of the change, however, depends on the extent to which hospitals in the NHDS sample use the UHDDS definition and list the principal diagnosis first on the face sheet. (If a diagnosis is not listed first but is somehow identified on the face sheet as the principal diagnosis, it will be listed first on the NHDS abstract form).

The NHDS records procedures in the order they appear on the face sheet of the medical record and codes up to three procedures. The NHDS does not use the UHDDS definition of principal procedure. However, if a diagnostic procedure is listed first and a non-diagnostic procedure is listed second on the face sheet, the non-diagnostic procedure will be listed first on the NHDS abstract form.

The NHDS publications present data on both “first-listed” and “all-listed” diagnoses and procedures. The “all-listed” category includes not only the diagnosis or procedure listed first, but also all diagnoses or procedures coded for a discharge. In the comparisons that follow, Medicare diagnostic and surgical data are compared with NHDS “first-listed” diagnoses and procedures rather than “all-listed” diagnoses and procedures, because Medicare codes only one diagnosis and procedure per discharge.

The first section (Diagnostic Data) compares counts of discharges by diagnosis from the NHDS and Medicare, and discusses some possible explanations for the differences. The next section (Surgical Data) compares discharges by surgical procedures from the two sources. The third section (Net and Gross Differences) discusses the concept of net and gross differences as they apply to the diagnostic and surgical data from the NHDS and Medicare. Finally, the fourth section (Average Length of Stay [ALOS] by Diagnostic and Surgical Group) compares data on ALOS by diagnosis and surgical procedure from the NHDS and Medicare.

Diagnostic Data

Table 6 shows the number of discharges in 23 diagnostic groups from the NHDS and Medicare for 1972 and 1976.¹¹ The groups are based on those in the NCHS publications and were selected because they are among the most frequent diagnoses for persons age 65 and over. According to the NHDS, these diagnostic groups comprise over 50 percent of all discharges. The table also shows the percentage difference between the number of NHDS and Medicare discharges for each group.

Table 6. Comparison of the Incidence of Short-Stay Hospital Discharges for Selected Diagnostic Groups, as Reported by the NHDS and Medicare, for Persons Age 65 and Over, U.S., 1972 and 1976.

Diagnostic Group and ICDA-8 Code	1972			1976
	Number of Discharges (in Thousands)		Percent Difference	Number of Discharges (in Thousands)		Percent Difference
	NHDS1	Medicare	$\frac{\text{Medicare}‒\text{NHDS}}{\text{NHDS}}\times 100$	NHDS1	Medicare	$\frac{\text{Medicare}‒\text{NHDS}}{\text{NHDS}}\times 100$
All Diagnoses2	6,634	6,320	−4.7	7,912	7,486	−5.4
Diagnoses Listed Below:	3,788	3,610	−4.7	4,368	3,802	−13.0
Malignant Neoplasms, 140-209	585	545	−6.8	753	639	−15.1
Large Intestine and Rectum, 153, 154, 197.5	78	80	2.6	107	97	−9.3
Thoracic Organs, 162, 163, 197.0-197.3	71	63	−11.3	99	85	−14.1
Breast, 174	51	51	0.0	71	58	−18.3
Prostate, 185	61	62	1.6	71	71	0.0
Lymphatic and Hematopoietic Tissues, 196, 200-209	62	53	−14.5	76	62	−18.4
Other and Unspecified Sites, Residual	262	236	−9.9	329	266	−19.1
Diabetes Mellitus, 250	188	171	−9.0	204	174	−14.7
Cataract, 374	179	196	9.5	228	233	2.2
Ischemic Heart Disease, 410-414	887	923	4.1	1,022	921	−9.9
Acute Myocardial Infarction, 410	186	187	0.5	202	188	−6.9
Chronic Ischemic Heart Disease, 412	630	677	7.5	711	660	−7.2
Other Ischemic Disease, 411, 413-414	71	59	−16.9	109	73	−33.0
Congestive Heart Failure, 427.0	146	98	−32.9	195	135	−30.8
Cerebrovascular Disease, 430-438	413	412	−0.2	452	442	−2.2
Arteriosclerosis, 440	98	54	−44.9	70	45	−35.7
Pneumonia, All Forms, 480-486	205	192	−6.3	255	196	−23.1
Ulcer of Stomach, Duodenum, Peptic Ulcer of Unspecified Site and Gastrojejunal Ulcer, 531-534	108	99	−8.3	109	88	−19.3
Inguinal Hernia, 550, 552	96	89	−7.3	99	88	−11.1
Diverticula of Intestine, 562	99	101	2.0	116	98	−15.5
Cholelithiasis, Cholecystitis and Cholangitis, 574, 575	164	141	−14.0	155	132	−14.8
Hyperplasia of Prostate, 600	149	150	0.7	163	145	−11.0
Arthritis and Rheumatism, Except Rheumatic Fever, 710-718	156	137	−12.2	194	153	−21.1
Fracture of Neck of Femur, 820	122	125	2.5	148	133	−10.1
Other Fractures, 800-819, 821-829	193	177	−8.3	205	180	−12.2

¹NHDS “First Listed” rather than “All Listed” diagnoses were used for comparison because Medicare codes only one diagnosis per discharge.

²Includes diagnoses not listed in this table and Medicare discharges with an “unknown” diagnosis.

For both years total discharges reported by the NHDS were about 5 percent more than those reported by Medicare, as shown on the top line of Table 6. As noted earlier, NHDS' totals are expected to be higher because about 4 percent of the population age 65 and over are not covered by Medicare. The table also shows the total number of discharges in the diagnostic groups selected. In 1972, the total number of discharges from the NHDS and from Medicare for these groups differed by 4.7 percent which, as one would expect, was equal to the overall difference of 4.7. In 1976, however, the percentage difference in the numbers of discharges from the NHDS and from Medicare for these groups increased to 13.0 percent, while the overall difference of 5.4 percent was almost unchanged from 1972.

To understand the relationship between the NHDS and Medicare counts of discharges by diagnostic group, it is helpful to examine Figure 4, which shows histograms of the percent differences in discharges by diagnostic group from the two sources for 1972 and 1976. In 1972, in 18 of the 23 diagnostic groups, the percent differences were within plus or minus 10 percent of the average difference of about 5 percent. In other words, for most groups the estimates from NHDS and Medicare were within 10 percent of what they were expected to be, based on the overall difference of about 5 percent between NHDS and Medicare total discharges. In 1976, the whole distribution shifted to the left, with most of the differences between the NHDS and Medicare estimates for individual diagnostic groups clustering around 15 percent. Seventeen of the 23 groups were found within plus or minus 10 percent of the 15 percent line.

Figure 4. Distribution of 23 Selected Diagnostic Groups by the Percent Difference Between the Number of Medicare Cases and NHDS Cases for Persons Age 65 and Over, U.S., 1972 and 1976.

Much of the shift in the distribution of the differences between the numbers of the NHDS and Medicare discharges is probably explained by an increase in the number of Medicare diagnoses coded “unknown.” The “unknown” code is used by Medicare when the information in the Medicare record is inadequate to assign a specific diagnostic code. As the following data show, the number of Medicare discharges with a diagnosis coded “unknown” increased from less than one in a hundred in 1972 to nearly one in ten in 1976.

Percent of Medicare Diagnoses Coded “Unknown”.

1972	1973	1974	1975	1976
0.8	1.2	4.5	9.9	9.6

If the 9.6 percent of discharges coded “unknown” in 1976 could have been assigned a specific code, it is reasonable to assume that the distribution would shift back to the right to a position like that in 1972, clustering around 5 percent. It is also reasonable to assume that the difference of 13.0 percent between the NHDS and Medicare counts of discharges in 1976 in the selected diagnostic groups (Table 6) would drop back close to the overall difference of 5.4 percent. (That is, the 13.0 percent difference for 1976 minus the 9.6 percent of discharges with an unknown diagnosis equals 3.4 percent).

To understand why the number of discharges in the Medicare Statistical System coded “unknown” has increased, we must understand the many recent changes in the way the diagnostic information has been transmitted and coded. The diagnostic and surgical information in the Medicare system comes from bills submitted by hospitals to Medicare fiscal intermediaries, who in turn send the information to HCFA's central office. When the Medicare program began in 1966, hospitals submitted hard copies of bills to the intermediaries, and the intermediaries sent the hard copies to HCFA's central office. The hard copies contained a written narrative description of the primary and secondary diagnoses, and of all surgical procedures, if surgery was performed. For a 20 percent sample, the narrative of the primary diagnosis was given a code from the Eighth Revision, International Classification of Diseases Adapted for Use in the United States (ICDA-8) by a coder working in the central office. One surgical procedure per discharge was assigned a code from the Current Procedural Terminology, First Edition (CPT) of the American Medical Association. If there were any questions about which diagnosis was the primary one, or if the narrative was only partly legible, the coder could consult the hard copy bill for further information about the patient's age, sex, surgical procedure or secondary diagnosis. Often explanatory notes were written on the bill.

In subsequent years, a number of changes occurred in the transmittal and coding of diagnostic and surgical information. The manual coding system was replaced by the Automated Medical Coding System, a computer system developed by HCFA to automatically assign codes by matching the narrative description in the bill record to a code. In addition, more hospitals and intermediaries began transmitting billing information via electronic data processing (tape or direct computer to computer communication). Although these changes decreased the amount of paper transmittals and the costs of coding, they also meant that the central office no longer had the opportunity to consult the whole bill for clues as to the correct diagnostic code.

In addition, some hospitals began to submit codes, rather than narratives, to the intermediaries. Since HCFA required intermediaries to transmit narratives, the intermediaries would convert the code back to narrative for submission to the HCFA's central office where the narrative would once again be coded. All these changes, unaccompanied by the necessary edits and quality controls, created the possibility of new kinds of keypunching, coding and programming errors at the hospital, intermediary, or central office levels, leading to an increase in the number of records with invalid information in the diagnostic field, and thus, to the increased problem with records that had to be assigned a “diagnosis unknown” code.

In addition to the 10 percent downward bias in the Medicare diagnostic data for 1976 caused by discharges with an unknown diagnosis, another kind of systematic difference is revealed by examining the ranks of the diagnostic groups on the percent differences in counts of discharges from the NHDS and Medicare. If the differences in the ranks of the diagnostic groups were due to sampling error, the ranks of the groups on their percent differences in 1972 and in 1976 should not be correlated. That is, groups that showed a relatively large difference between the NHDS and Medicare estimates in 1972 should not show a relatively large difference in 1976. However, the rank correlation coefficient of the ranks of the differences in 1972 and 1976 was r=0.78 (p=0.01) indicating a positive correlation in the ranks of the groups between 1972 and 1976.

The correlation indicates some systematic difference in the NHDS and Medicare estimates above and beyond that resulting from the Medicare “unknown” diagnoses in 1976. Investigation so far has not been able to determine the cause of this difference.

In both 1972 and 1976, the NHDS and Medicare reported very different numbers for a few diagnostic groups. As indicated in Figure 4, in 1972 two diagnostic groups had differences of -25 percent or more between the NHDS and Medicare data, and in 1976 there were three such groups. Two of these groups, congestive heart failure (ICDA-8 Code 427.0) and arteriosclerosis (ICDA-8 Code 440), had large differences in both years.¹² As Table 6 shows, the NHDS reported 32.9 percent more discharges of persons with congestive heart failure in 1972, and 30.8 percent more in 1976 than Medicare reported. The NHDS reported 44.9 percent more discharges with arteriosclerosis in 1972, and 35.7 percent more in 1976, than Medicare reported. The differences are not unexpected, since the IOM studies found low agreement rates for diseases of the circulatory system, but we have not determined the specific reason for the difference.

Surgical Data

Table 7 shows the incidence of 9 surgical operations occurring frequently among persons age 65 and over.¹³ These procedures comprised about 30 percent of all surgical procedures reported by the NHDS in 1976, and include a number of elective operations, such as extraction of lens and hysterectomy, whose appropriateness in some situations is currently being questioned. During the period under study, Medicare coded surgeries using a version of the American Medical Association's Current Procedural Terminology, First Edition, 1966, (that they had adapted for their own use) while the NCHS used the Eighth Revision, International Classification of Diseases Adapted for Use in the United States, 1967 (ICDA-8). Because the NCHS and Medicare used different coding schemes, coding experts were consulted to assure that appropriate codes were compared.

Table 7. Comparison of the Incidence of Short-Stay Hospital Discharges for Selected Surgical Operations as Reported by the NHDS and Medicare, for Persons Age 65 and Over, U.S., 1972 and 1976¹.

Surgical Category	Surgical Code Used By:		1972			1976
			Number of Discharges (in Thousands)		Percent Difference	Number of Discharges (in Thousands)		Percent Difference
	NHDS (ICDA-8)	Medicare (CPT-1)	NHDS1	Medicare	$\frac{\text{Medicare}‒\text{NHDS}}{\text{NHDS}}\times 100$	NHDS1	Medicare	$\frac{\text{Medicare}‒\text{NHDS}}{\text{NHDS}}\times 100$
Discharges with Surgery2			1,907	1,972	3.4	2,412	2,404	−0.3
Extraction of Lens	14.4-14.6	5611, 5612, 5613	180	194	8	227	226	0
Prostatectomy	58.1-58.3	4311,4313,4316, 4321	135	153	13	151	157	4
Repair of Inguinal Hernia	38.2, 38.3	3631, 3635	90	83	−8	95	91	−4
Cholecystectomy	43.5	3515, 3517	79	79	0	82	81	−1
Insertion or Replacement of Electronic Heart Device	30.4-30.5	2306	41	26	−37	73	46	−37
Resection of Small Intestine or Colon	47.4-47.6	3174, 3178, 3180	41	44	7	49	54	10
Mastectomy, Partial. Complete, Radical	65.2,65.3-65.6	0445, 0457, 0458, 0470	34	39	15	43	42	−2
Hysterectomy	69.1-69.5	4617, 4618, 4621, 4625 4626, 4627, 4631	27	31	15	31	30	−3
Arthrosplasty of Hip	87.0 (modified to include 87.1)	1150	22	46	109	28	60	114

¹HDS “First Listed” operations rather than “All Listed” operations were used for comparison because Medicare codes only one operation per discharge.

²Includes operations not listed in this table.

Because of variations in the exact definition of a discharge with surgery, the percentage difference between the NHDS and Medicare counts of discharges with surgery should not be expected to be the same as the percentage difference between the NHDS and Medicare counts of total discharges.¹⁴ These overall percentage differences are shown in the first line of Table 7. In both 1972 and 1976, the NHDS and Medicare reported about the same number of total discharges with surgery. The difference in 1972 was 3.4 percent, and in 1976 it was 0.3 percent.

We would expect that the percentage differences within procedure categories would cluster around the overall percentage difference in each year. For specific surgeries, Table 7 shows that for both 1972 and 1976 the percentage differences between the numbers of operations reported by the NHDS and by Medicare were within plus or minus 20 percent of the overall percentage difference, except in the cases of arthroplasty of the hip and insertion or replacement of electronic heart device (pacemaker). In 1972, the NHDS reported 109 percent fewer hip arthroplasties and 37 percent more pacemaker insertions or repairs than Medicare reported, and in 1976 the NHDS reported 114 percent fewer hip arthroplasties and 37 percent more pacemaker insertions or repairs than Medicare reported.¹⁵ One reason for the large number of Medicare hip arthroplasties may stem from a decision at the beginning of the Medicare Statistical System to code discharges listed only as “arthroplasty” as arthroplasty of the hip. This fact is consistent with the finding that both sources gave similar estimates of total arthroplasties for all joints—in 1977, the NHDS estimated 80,000 and Medicare estimated 79,000 arthroplasties of all types. The sharp differences in the number of operations for arthroplasty of the hip and insertion or replacement of heart pacemaker reported by the NHDS and Medicare mean that before any statements about the incidence of these common and serious operations can be made, data from the two sources must be reconciled.

Net and Gross Differences

The difference between the total number of cases in a diagnostic group reported by the NHDS and by Medicare could be small, even though the two systems may frequently assign different diagnoses to the same cases. This difference between the total number reported is referred to as the net difference. In contrast, the difference between two sources on the basis of a case-by-case comparison is referred to as the gross difference.¹⁶ The case-by-case difference could be large, even when two sources agree closely on totals, because the case-by-case difference includes all disagreements, many of which may be offsetting on a net basis. That is, a small net difference may mask a substantial amount of disagreement at the individual case level.

Actually, a case-by-case comparison between the NHDS and Medicare could be done only for those discharges recorded by both systems. Since both systems sample in different ways, the set of discharges reported by both systems is a small subset of the total discharges covered by each system. Furthermore, even a case-by-case comparison of diagnosis between the NHDS and Medicare could not answer the question of the relative accuracy of the diagnostic information reported by the two systems. A judgment of the accuracy of the two systems could come only through comparison with a third source accepted as a standard because of its superior methodology.

Even though the only way to quantify the case-by-case differences is through examination of the diagnoses assigned to the same cases by the two systems, examination of an accompanying statistic like ALOS can help to indicate whether the systems agree on a case-by-case basis. If both systems report a different ALOS for a diagnostic group, some disagreement must exist on the individual case level, even though the totals may agree.

Average Length of Stay (ALOS) by Diagnostic and Surgical Group

Comparing the ALOS by diagnostic and surgical group, as reported by the NHDS and Medicare, is important for two reasons. First, the ALOS is a commonly used measure of hospital performance, and is employed in the administration and evaluation of the PSRO program. Second, as previously mentioned, such a comparison can help indicate the degree to which diagnostic and surgical information from the NHDS and Medicare for the same diagnostic or surgical groups really reflect the same medical conditions.

Table 8 compares the ALOS for 1972 and 1976, as reported by the NHDS and Medicare for 22 diagnostic groups. For all diagnoses, the NHDS reported a higher ALOS than Medicare reported, as it had done for every year since 1967 (Figure 1). Considering the sampling error involved in the ALOS from both systems and the slightly higher ALOS reported by the NHDS for all discharges, the diagnostic specific ALOS estimates are very comparable. The largest percentage differences were found for arteriosclerosis (17.6) and malignant neoplasms of lymphatic and hematopoietic tissues (11.8 percent) in 1976, but the differences were not statistically significant. For surgical cases, as Table 9 shows, the ALOS for 1972 and 1976 again were very close, except in 1972 for insertion or replacement of electronic heart device, where the percentage difference was 19.7. The difference, however, was not statistically significant.¹⁷ The similar ALOS reported by each system supports the belief that the gross difference rate may be small. However, other statistics may be more sensitive to reporting inaccuracies than the ALOS is. For example, if some records of cases with a relatively high in-hospital mortality, such as emphysema (ICDA-8 Code 492), are incorrectly coded as bronchitis (ICDA-8 Codes 490 and 491), the relative effect on data on the in-hospital mortality of bronchitis patients could be large because of the comparatively low mortality of bronchitis patients.

Table 8. Comparison of Average Length of Stay in Short-Stay Hospitals for Selected Diagnostic Groups as Reported by the NHDS and Medicare, for Persons Age 65 Years and Over, U.S., 1972 and 1976.

Diagnostic Group and ICDA-8 Code	1972			1976
	Average Length of Stay (in Days)		Percent Difference	Average Length of Stay (in Days)		Percent Difference
	NHDS	Medicare	$\frac{\text{Medicare}‒\text{NHDS}}{\text{NHDS}}\times 100$	NHDS	Medicare	$\frac{\text{Medicare}‒\text{NHDS}}{\text{NHDS}}\times 100$
All Diagnoses1	12.2	12.1	−0.8	11.5	11.1	−3.5
Malignant Neoplasms, 140-209	15.0	15.0	0.0	13.5	13.7	1.5
Large Intestine and Rectum, 153, 154, 197.5	19.4	19.5	0.5	17.3	17.6	1.7
Thoracic Organs, 162, 163, 197.0-197.3	14.5	14.9	2.8	13.7	13.9	1.5
Breast, 174	13.3	14.4	8.3	13.1	12.8	−2.3
Prostate, 185	13.3	13.3	0.0	11.5	11.9	3.5
Lymphatic and Hematopoietic Tissues, 196, 200-209	14.7	13.9	−5.4	14.4	12.7	−11.8
Diabetes Mellitus, 250	12.3	12.9	4.9	12.2	12.0	−1.6
Cataract, 374	7.0	6.9	−1.4	5.0	5.0	0.0
Ischemic Heart Disease, 410-414	12.8	12.6	−1.6	11.7	11.1	−5.1
Acute Myocardial Infarction, 410	15.2	15.2	0.0	14.3	13.5	−5.6
Chronic Ischemic Heart Disease, 412	12.4	12.0	−3.2	11.4	10.7	−6.1
Other Ischemic Disease, 411, 413-414	9.9	10.0	1.0	9.0	8.6	−4.4
Congestive Heart Failure, 427.0	12.4	12.2	−1.6	11.3	11.4	0.9
Cerebrovascular Disease, 430-438	13.9	13.9	0.0	13.7	12.8	−6.6
Arteriosclerosis, 440	12.0	12.6	5.0	14.2	11.7	−17.6
Pneumonia, All Forms, 480-486	12.3	12.5	1.6	12.1	11.7	−3.3
Ulcer of Stomach, Duodenum, Peptic Ulcer of Unspecified Site and Gastrojejunal Ulcer, 531-534	12.5	12.0	−4.0	11.9	11.4	−4.2
Inguinal Hernia, 550, 552	8.7	8.9	2.3	8.0	7.9	−1.2
Diverticula of Intestine, 562	9.4	9.7	3.2	9.5	9.2	−3.2
Cholelithiasis, Cholecystitis and Cholangitis, 574, 575	12.9	13.0	0.8	12.4	12.4	0.0
Hyperplasia of Prostate, 600	12.7	12.4	−2.4	11.0	10.8	−1.8
Arthritis and Rheumatism, Except Rheumatic Fever, 710-718	13.3	13.0	−2.3	13.0	12.6	−3.1
Fracture of Neck of Femur, 820	23.2	22.8	−1.7	21.3	20.8	−2.3
Other Fractures, 800-819, 821-829	14.4	14.7	2.1	13.2	13.2	0.0

¹Includes diagnoses not listed in this table.

Table 9. Comparison of Average Length of Stay for 9 Selected Surgical Procedures as Reported by the NHDS and Medicare, for Persons Age 65 Years and Over, U.S., 1972 and 1976.

Surgical Category			1972			1976
	Surgical Code Used By:		Average Length of Stay (in Days)		Percent Difference	Average Length of Stay (in Days)		Percent Difference
	NHDS (ICDA-8)	Medicare (CPT-1)	NHDS1	Medicare	$\frac{\text{Medicare}‒\text{NHDS}}{\text{NHDS}}\times 100$	NHDS1	Medicare	$\frac{\text{Medicare}‒\text{NHDS}}{\text{NHDS}}\times 100$
Discharges with Surgery1			14.2	13.4	−5.6	13.5	12.8	−5.2
Extraction of Lens	14.4-14.6	5611.5612. 5613	7.3	7.0	−4.1	5.3	5.1	−3.8
Prostatectomy	58.1-58.3	4311, 4313,4316,4321	14.6	14.5	−0.7	13.7	12.9	−5.8
Repair of Inguinal Hernia	38.2, 38.3	3631, 3635	9.2	9.2	0.0	8.4	8.3	−1.2
Cholecystectomy	43.5	3515, 3517	16.3	16.7	2.4	16.1	15.9	−1.2
Insertion or Replacement of Electronic Heart Device	30.4-30.5	2306	12.2	14.6	19.7	13.5	13.7	1.5
Resection of Small Intestine or Colon	47.4-47.6	3174, 3178, 3180	21.7	22.6	4.1	22.1	22.1	0.0
Mastectomy, Partial, Complete, Radical	65.2, 65.3-65.6	0445, 0457, 0458, 0470	11.6	10.8	−6.9	9.8	9.9	1.0
Hysterectomy	69.1-69.5	4617, 4618, 4621, 4625 4626, 4627, 4631	11.9	12.7	6.7	11.4	11.9	4.4
Arthroplasty of Hip	87.0 (modified to include 87.1)	1150	27.1	24.0	−11.4	21.7	22.4	3.2

¹Includes operations not listed in this table.

Discussion and Conclusions

This study reveals the worth of comparative analyses of different data systems for validation and for detection of problems. Such comparisons were possible only because the four systems studied collect some common data items. Overlap of certain data items provides each system with a check on its performance. Without such overlap, errors could persist undetected and lead to false conclusions about important issues.¹⁸

The four data systems provide basic data on national patterns and trends of hospital care, and furnish essential information for health care research and policy formulation. This study was concerned that research conclusions might be sensitive to the choice of a data system used to study a variety of common measures of hospital use. For longitudinal trends in the ALOS and admissions, we were reassured to find that all four sources examined in this study showed the same trends, although in recent years the AHA has reported a some-what higher number of admissions than the other sources have reported. On the other hand, analysis by census region indicates that the data sources failed to agree on such a fundamental indicator of hospital use as admission rates.

For many purposes, data for much smaller geographic levels (such as States or counties) than those examined in this paper are needed. Experience with Medicare data has shown that as data are broken down to increasingly smaller geographic levels, the chances for disagreement with data from other sources grow. Errors which offset each other at the national level begin to become apparent as data from smaller areas are analyzed. Thus, the agreement found among different data sources for some variables at the national and regional levels should not cause one to overlook the potential problems with smaller area data. In many situations, however, the researcher is limited to one data source for small area data. For instance, of the four data systems studied, only Medicare can give small area estimates, because of its sample design and large sample size. In these situations, a researcher must look to local data sources or rely on checks of internal consistency for validation.

Comparisons of discharges by diagnostic and surgical groups revealed small net differences between the NHDS and Medicare estimates for most groups at the national level. This finding was surprising in light of the IOM studies that found large discrepancy rates at the individual case level for diagnostic and surgical data in the NHDS and Medicare systems. One can only speculate about why two data systems, each shown by the IOM to be subject to high discrepancy rates, should so often be in agreement. The considerable discrepancy rates resulting from the expert consultant's evaluation of the IOM field team suggest that there may be a high degree of uncertainty in assigning a principal diagnosis. The agreement between the NHDS and Medicare data for most groups may show that the effect of this uncertainty tends to balance out in large national data systems.

The agreement between the NHDS and Medicare on numbers and ALOS of discharges by diagnosis and surgical procedure also shows that two systems that gather diagnostic and surgical data in different ways can arrive at similar results. The diagnostic and surgical information for the NHDS comes from information abstracted from the face sheets of medical records some time after the patient's discharge. The abstracting and coding process are subject to explicit quality controls. The Medicare diagnostic and surgical information is recorded on the Medicare bill shortly after the patient's discharge, and the abstracting and coding process is subject to very few explicit quality checks. In spite of these differences, the IOM studies found similar discrepancy rates for each system's diagnostic and surgical data at the 4-digit level. In the present study, we found that the diagnostic and surgical data for most of the selected conditions examined from the two systems were similar.

Although the degree of similarity observed may be reassuring for some uses of the data, it also points out a limitation of comparisons among aggregate statistics from different data systems for detecting errors. The totals for many diagnostic and surgical groups were similar, even though the IOM studies show that a large percentage of hospital stays were not assigned a correct diagnostic or surgical code. Comparisons of aggregate statistics among data sources can detect errors, but they must be supplemented by validation studies, such as the IOM studies, at the individual case level to study differences that may be masked by aggregate data. Such studies are of great value in identifying discrepancies and in aiding the user to understand the limits of the data used, and should be a regular part of a data system. These validation studies should not only quantify the overall discrepancies, but should also describe the effect of discrepancies on the most important accompanying statistics, such as ALOS.

Sometimes researchers who are aware of the possibility of inaccuracies in the data they are using assume that the errors are “random” or that they are offsetting—that is, overstatement from one reporting unit is balanced by understatement from another. This study reinforces the findings of other studies that many errors are systematic and are related to the nature of the data source.¹⁹ Without actual evidence, it simply cannot be assumed that errors are random or offsetting.

The results of this paper point to an obligation for researchers to consult a number of data sources whenever possible. If the sources agree, they may feel more confident in their conclusions.²⁰ Whenever they do not agree, the researchers must attempt to reconcile the data. If the data cannot be reconciled, the results must be qualified, perhaps by presenting a range of estimates. Further, at present there is very little information to guide a researcher in choosing one data source over another to examine a particular topic. Those managing data systems should give more emphasis to educating the user about their strengths and weaknesses so that a more informed choice of which source to use can be made, depending on how the data will be employed.

Appendix A

Description of Data Systems

National Hospital Discharge Survey (NHDS)

The National Center for Health Statistics (NCHS) National Hospital Discharge Survey (NHDS) has been collecting data annually, since 1965, on patient characteristics from a probability sample of short-stay hospitals (NCHS, 1979b). The universe consists of short-stay (ALOS less than 30 days), non-Federal, general and special U.S. hospitals in the NCHS National Master Facility Inventory (MFI) of Hospitals and Institutions, excluding hospital units of other institutions. From these, a sample of hospitals, stratified by number of beds and region, is drawn. In 1977, 535 hospitals were sampled. Within each hospital, a sample of discharges is selected and information from the face sheet of the medical records is transcribed to abstract forms. About 224,000 abstracts were collected in 1977. Table A summarizes the type of data collected and other characteristics of the NHDS and compares them to the NHIS, the AHA Panel Survey and to the Medicare Statistical System.

National Health Interview Survey

The National Health Interview Survey (NHIS) of NCHS collects data on acute illnesses and injuries, disability days, limitations of activity, chronic conditions, and health care use through household interviews of a sample of the civilian, noninstitutionalized population of the United States, (NCHS, 1979a). The survey was begun in 1957. The households are selected through a multistage probability sample. In 1978, about 41,000 households (containing 110,000 people) were interviewed.

Two exclusions affect hospital data from the NHIS, and make estimates of hospital use from the NHIS lower than from other sources. First, people who died while hospitalized are not covered. For the 65 and over population this is an important exclusion. According to 1977 Medicare data, in 6.6 percent of all hospital stays of the aged the patient died while in the hospital. Second, persons in institutions are excluded. This is another important exclusion for the 65 and over group, because about 5 percent of the aged are in institutions, the great majority of these (about 96 percent) in nursing homes, (Bureau of the Census, 1978). Table A contains more information on the NHIS.

The AHA National Hospital Panel Survey

The National Hospital Panel Survey of the American Hospital Association was begun in 1963 “to provide monthly estimates of community hospital financial and utilization indicators by bed-size group at national and regional levels,” (American Hospital Association, 1979b). The universe consists of short-term (ALOS less than 30 days), non-Federal, general and special U.S. hospitals, excluding hospital units of other institutions. (The AHA term for this set of hospitals is “community” hospitals.) The sampling list is drawn from the latest edition of the AHA Annual Survey of Hospitals. The universe for the AHA Panel Survey is essentially the same as the universe for the NHDS, except that AHA excludes hospitals not registered with the AHA. However, this is not an important difference since there were only 141 non-registered “community” hospitals in 1978 (American Hospital Association, 1979a). In 1975, a probability sample of 2,000 hospitals was drawn. The sample hospitals complete a survey form each month summarizing their utilization and financial experience. Unlike the NHDS, the NHIS, or Medicare, the AHA survey collects aggregate statistics rather than unit records on individual hospital stays. Table A summarizes the principal features of the Panel Survey and the types of data collected.

The Medicare Statistical System

In contrast to the NHDS, the NHIS or the AHA, the HCFA gathers its Medicare hospital use statistics through administrative records rather than a survey. The information comes from two major files. First, information is derived from the Admission Query System, in which hospitals participating in the Medicare program send a notice of admission (query) to the Medicare central office in Baltimore, Maryland when a Medicare beneficiary enters the hospital. This system provides hospital level counts of all admissions very soon after they occur. Second, information is derived from the bills submitted by participating hospitals for services to Medicare beneficiaries. The bills contain the ID numbers of the beneficiary and hospital, and hospital stay information including dates of admission and discharge, charges and reimbursement, and diagnosis and surgical procedure. Information on beneficiary and hospital characteristics are entered into the bill record from independent files maintained on all Medicare beneficiaries and participating hospitals. The principal diagnosis and surgical procedure are coded for a 20 percent sample of hospital stays.

In 1977, 26,093,919 beneficiaries were enrolled for Medicare Hospital Insurance (23,474,546 age 65 and over and 2,619,373 disabled under 65) and 6,131 short-stay hospitals were certified for Medicare. We estimate that about 96 percent of all persons age 65 and over are enrolled for Medicare Hospital Insurance. Table A contains information on characteristics of the Medicare Statistical System.

Table A. Comparison of Selected Characteristics of the NCHS Hospital Discharge Survey, the NCHS Health Interview Survey, the American Hospital Association (AHA) National Hospital Panel Survey and the Health Care Financing Administration (HCFA) Medicare Statistical System (Hospital Portion).

Data System
Characteristic	NCHS National Hospital Discharge Survey	NCHS National Health Interview Survey	AHA National Hospital Panel Survey	HCFA Medicare Statistical System (Hospital Portion)
Universe	Short-stay1 non-Federal general and special U.S. hospitals excluding hospital units of other institutions	Civilian, noninstitutionalized U.S. population	Short-term,1 non-Federal, general and special U.S. hospitals excluding hospital units of other institutions and hospitals not registered with the AHA	Medicare beneficiaries discharged from short-stay1 hospitals participating in Medicare. (Military and Veterans Administration hospitals do not participate in Medicare).
Sample	Stratified probability sample of hospitals and a random or systematic sample of discharges within each hospital	Multistage probabilty sample of households	Probabilty sample of hospitals stratified on number of beds and census region	100 percent data for some information; for a 20 percent probability sample, diagnosis and surgical procedure are coded
Lowest level of aggregation	Individual discharge	Individual discharge	Hospital	Individual discharge
Response Rate	423 of 491 (86 percent) of eligible selected hospitals in 1977	96.2 percent in 1978	Averages about 70 percent each month	Not applicable, administrative records
Methods of Gathering Data	Information on each sample discharge is abstracted from medical record face sheet either by hospital staff and/or Census Bureau representatives	Household interviews conducted continuously through year	Hospitals in sample complete a survey form each month	Data derived from administrative records (Medicare admission notices and Medicare hospital bills)
Principal Data Items Collected	Patient characteristics: age, race, sex, marital status, and beginning in 1977, zip code of residence and expected source of payment Hospital Stay Information: Date of admission and discharge, discharge status. A maximum of 5 diagnoses and 3 surgeries are coded	Demographic data, acute and chronic conditions, limitation of activity, disability days, health care use (physicians, dentists, hospitals). In addition, data on special topics (such as health insurance coverage) may be collected for a year Hospital Stay Information: Dates of Hospitalization(s), name and address of hospital, reason (condition) for entering hospital, operations	Hospital characteristics: size, finances, number of employees Hospital use: number of admissions, patient days and outpatient visits	Hospital Stay Information: Dates of admission and discharge, discharge status, hospital charges and reimbursement, date of surgery. Principal diagnosis and surgical procedure coded for 20-percent sample Patient (beneficiary) and hospital characteristics from other files are linked to hospital stay on basis of beneficiary and hospital ID numbers in hospital bill

¹Average length of stay of less than 30 days.

Appendix B

Sources of the Data

National Hospital Discharge Survey (NHDS)

Data on discharges, ALOS and days of care are contained in the following:

• National Center for Health Statistics, Public Health Service, “Inpatient Utilization of Short-Stay Hospitals in Each Geographic Division, United States—1966-1968,” Vital and Health Statistics, Series 13, No. 10, DHEW Publication No. (HSM) 73-1761, November 1972.

• National Center for Health Statistics, Public Health Service, “Utilization of Short-Stay Hospitals, Summary of Non-medical Statistics, United States—1970,” Vital and Health Statistics, Series 13, No. 14, DHEW Publication No. (HRA) 74-1765, August 1973.

• National Center for Health Statistics, Public Health Service, “Utilization of Short-Stay Hospitals, Summary of Non-medical Statistics, United States—1971,” Vital and Health Statistics, Series 13, No. 17, DHEW Publication No. (HRA) 75-1768, August 1974.

• National Center for Health Statistics, Public Health Service, “Utilization of Short-Stay Hospitals, Summary of Non-medical Statistics, United States—1972,” Vital and Health Statistics, Series 13, No. 19, DHEW Publication No. (HRA) 75-1770, June 1975.

• National Center for Health Statistics, Public Health Service, “Utilization of Short-Stay Hospitals, Summary of Non-medical Statistics, United States—1973,” Vital and Health Statistics, Series 13, No. 23, DHEW Publication No. (HRA) 76-1774, July 1976.

• National Center for Health Statistics, Public Health Service, “Utilization of Short-Stay Hospitals, Annual Summary for the United States—1974,” Vital and Health Statistics, Series 13, No. 26, DHEW Publication No. (HRA) 76-1777, September 1976.

• National Center for Health Statistics, Public Health Service, “Utilization of Short-Stay Hospitals, Annual Summary for the United States—1975,” Vital and Health Statistics, Series 13, No. 31, DHEW Publication No. (HRA) 77-1782, April 1977.

• National Center for Health Statistics, Public Health Service, “Utilization of Short-Stay Hospitals, Annual Summary for the United States—1976,” Vital and Health Statistics, Series 13, No. 37, DHEW Publication No. (PHS) 78-1788, June 1978.

• National Center for Health Statistics, Public Health Service, “Utilization of Short-Stay Hospitals, Annual Summary of the United States—1977,” Vital and Health Statistics, Series 13, No. 41, DHEW Publication No. (PHS) 79-1557, March 1979.

• National Center for Health Statistics, Public Health Service, “Utilization of Short-Stay Hospitals, Annual Summary for the United States—1978,” Vital and Health Statistics, Series 13, No. 46, DHEW Publication No. (PHS) 80-1797, March 1980.

• Preliminary data for 1979 are from unpubished NCHS tabulations.

Data on discharges by diagnosis and surgical procedure are found in the following:

• National Center for Health Statistics, Public Health Service, “Inpatient Utilization of Short-Stay Hospitals by Diagnosis, United States—1972,” Vital and Health Statistics, Series 13, No. 20, DHEW Publication No. (HRA) 76-1771, November 1975.

• Surgical data for 1972 and 1976 and diagnostic data for 1976 are from unpublished NCHS tabulations.

Medicare, Health Care Financing Administration

Data on admissions, ALOS and days of care are available in the following:

• Data on admissions for 1967 to 1973 are from Social Security Administration, “Annual Statistical Supplement, 1976,” Social Security Bulletin, (no date) SSA Publication No. 13-11700.

• Data on admissions for 1974 and later and data on admissions by census regions are from unpublished HCFA tabulations.

• Data on ALOS for 1967 to 1975 are from Ronald Deacon et al., “Analysis of Variations in Hospital Use by Medicare Patients in PSRO Areas, 1974-1977,” Health Care Financing Review, Summer 1979. Data on ALOS for 1976 and 1977 are from “Medicare: Inpatient Use of Short-Stay Hospitals, 1977,” DHEW, Health Care Financing Administration (no date). Data on ALOS for 1978 and 1979 are from unpublished tabulations.

• Data on ALOS by census region are from Charles Helbing, “Ten Years of Short-Stay Hospital Utilization and Costs Under Medicare: 1967-1976,” Health Care Financing Research Report, DHHS, Health Care Financing Administration, August 1980, and from unpublished HCFA tabulations.

Data on discharges by diagnosis and surgical procedure for 1972 and 1976 are from unpublished HCFA tabulations.

(Note: Data on admissions will differ from data on discharges published elsewhere, because they come from separate systems, as explained in Appendix A.)

American Hospital Association (AHA) National Hospital Panel Survey

Nationwide admissions, ALOS and days of care, (1967 to 1977) are from:

• “Hospital Indicators,” Hospitals, Vol. 48, No. 18, September 16, 1974, pp. 19-23.

• “Hospital Indicators,” Hospitals, Vol. 49, No. 8, April 16, 1975, pp. 35-37.

• “Hospital Indicators,” Hospitals, Vol. 50, No. 8, April 16, 1976, pp. 45-51.

• “Hospital Indicators,” Hospitals, Vol. 51, No. 10, May 16,1977, pp. 47-50.

• “Hospital Indicators,” Hospitals, Vol. 52, No. 10, May 16,1978, pp. 71-74.

Admissions, ALOS and days of care for the nation for 1978 and 1979, and admissions and ALOS by census region are from unpublished tabulations from AHA. Rates are calculated using estimates of resident population from: U.S. Census Bureau, “Estimates of the Population of States, by Age: July 1, 1971 to July 1, 1979,” Current Population Reports: Population Estimates and Projections, Series P-25, No. 875, U.S. Government Printing Office, Washington, D.C., January 1980.

National Health Interview Survey (NHIS), National Center for Health Statistics, Public Health Service

Discharges and ALOS for the U.S. from 1967 to 1974, and by census region for 1972, are from: “Hospital Discharges and Length of Stay: Short-Stay Hospitals, United States—1972,” Vital and Health Statistics, Series 10, Number 107, DHEW Publication No. (HRA) 77-1534, September 1976.

Discharges and ALOS for the U.S. for 1975-1978 are from:

• National Center for Health Statistics, Public Health Service, “Current Estimates from the Health Interview Survey: United States—1975,” Vital and Health Statistics, Series 10, No. 115, DHEW Publication No. (HRA) 77-1543, March 1977.

• National Center for Health Statistics, Public Health Service, “Current Estimates from the Health Interview Survey: United States—1976,” Vital and Health Statistics, Series 10, No. 119, DHEW Publication No. (PHS) 78-1547, November 1977.

• National Center for Health Statistics, Public Health Service, “Current Estimates from the Health Interview Survey: United States—1977,” Vital and Health Statistics, Series 10, No. 126, DHEW Publication No. (PHS) 78-1554, September 1978.

• National Center for Health Statistics, Public Health Service, “Current Estimates from the Health Interview Survey: United States—1978,” Vital and Health Statistics, Series 10, No. 130, DHEW Publication No. (PHS) 80-551, November 1979.

Data for 1979 are from unpublished tabulations.

Discharges and ALOS by census region for 1975 and 1977 are from unpublished NCHS tabulations.

Days of care are from “Current Estimates from the Health Interview Survey: United States,” for 1967-1978. Days of care for 1979 are from unpublished NCHS tabulations.