Commentary: Measuring the Quality of the VA Health Care System

The Veterans Administration has a long history of measuring the quality of care it provides. In the 1970s deficiencies in care were documented and the potential for the VA to deliver high quality care was thought to be limited (National Academy of Sciences 1977). However, the VA responded by developing prospective data registries, a state of the art electronic medical record, and patient safety programs (Iglehart 1996; Kizer 1999). By the mid-1990s clear evidence for improvement appeared. In a study of VA and Medicare patients hospitalized in 1994 and 1995 with acute myocardial infarction, Petersen and colleagues found that the VA patients were more likely to receive recommended medications including thrombolytics, beta-blockers, aspirin and ACE inhibitors than were patients treated in the Medicare system (Petersen et al. 2001). Not all MI care was superior in the VA. Recommended angiography was less likely to be performed for VA patients than Medicare (Petersen et al. 2003); however, survival for VA patients was not clearly different from those in Medicare after adjustment for clinical characteristics from detailed chart review (Petersen et al. 2000). The conclusion reached by many was that VA care for acute myocardial infarction was now on par with the community (Fihn 2000). Therefore, when the VA evaluated acute myocardial infarction care for their congressionally mandated comparison with Medicare it was reasonable to expect that outcome would be the same. Thus, the observation that survival in the VA was worse than in Medicare, as reported in this issue of HSR, was surprising (Landrum et al. 2004).

What were the reasons for the disparate findings between Landrum's evaluation and the prior study by Petersen? An obvious difference is the source of data. The current study used administrative data, while the Petersen study used chart review. Whenever the results from an analysis of administrative data are evaluated, we round up the usual suspects of inadequate risk adjustment and poor case ascertainment. Inaccuracy in identification of cases using administrative data is unlikely to explain the discrepancy in results given the reasonable accuracy (96 percent) of coding a primary diagnosis of acute myocardial infarction (Meehan et al. 1995). However, it is plausible that the VA patients were more ill than the Medicare patients and that this was not accounted for in Landrum's administrative data. Indeed, past studies have found that VA patients are poorer, more likely to be homeless, and have more comorbidities than patients in the community (Peabody and Luck 1998; Studenski et al. 2003). Furthermore, the risk adjustment with administrative data is usually inferior to adjustment with clinical data.

A closer look at both studies reveals that risk adjustment with administrative and chart review data gave remarkably similar results. In the chart review study unadjusted 30-day mortality was 17.3 percent in the VA and 18.1 percent in Medicare (Petersen et al. 2000). Landrum's unadjusted 30-day mortality was 16.9 percent for both VA and Medicare. In both studies risk adjustment made the VA outcomes look worse. In the Petersen study adjustment for variables from chart review decreased the risk ratio for 30-day mortality (Medicare vs. VA) from 1.05 (18.1/17.3) to an odds ratio of 0.94 where a value below one favors Medicare. In Landrum's study adjustment using administrative data decreased the risk ratio from 1.0 (16.9/16.9) to 0.88. With the caveat that these ratios are measured slightly differently, it appears that the effects of risk adjustment on 30-day mortality were similar using administrative and clinical data. A comparable finding was observed for one year mortality. Adjustment with detailed clinical data in the Petersen study decreased the risk ratio from 1.01 (31.8/31.5) to 0.94. In Landrum's adjustment with administrative data reduced the risk from 0.90 (30.9/34.5) to 0.84. Thus we cannot blame inadequate risk adjustment with administrative data as the reason for the differences between the two studies.

This does not mean that the risk adjustment was adequate. It is likely that even adjustment with detailed clinical data did not capture important differences between VA and Medicare MI patients. The current risk adjustments conclude that VA MI patients are healthier than Medicare MI patients. Is this likely? Possibly, although comorbidities are higher among VA patients including hypertension, diabetes, stroke, dementia and chronic obstructive pulmonary disease, VA patients were younger (2.1 years younger in the Petersen study, 2.0 years in the Landrum study). However, estimates of the severity of infarction (available in the Petersen study but not in the administrative study) suggested VA patients had more severe infarctions. (Petersen et al. 2000) VA patients likely had other characteristics associated with a poor prognosis such as lower income and education that could not be completely captured by either study and these may have confounded the results. The administrative analysis was able to incorporate differences in income and education at the zip code level but the authors acknowledge that this may have been inadequate. Finally, it seems unlikely that initial treatment differences for acute myocardial infarction will lead to the progressive separation of the mortality curves observed in the current study. Although differences between the VA and Medicare in subsequent care including revascularization cannot be excluded, it is likely that differences in unobserved characteristics were important.

If the use of administrative data was not solely responsible for the difference in results between the Petersen and the present study what is? One important difference is size. If the administrative analysis was substantially smaller (less power), the conclusions may have been: less angiography use in the VA, no significant difference in mortality, prior study confirmed. Another difference is the study samples. While both studies used national samples of VA data, clinical data from Medicare patients in the Petersen study were limited to seven states. The authors note that mortality was higher for Medicare in the seven states used in the Petersen study than in the rest of the country. The Petersen study was conducted earlier (1994–1995 data) and Medicare treatment may have improved relative to VA care by the time Landrum's study was conducted in 1997–1999.

Should we abandon administrative data for the comparison of health care systems because of the inability to perform adequate risk adjustment? As argued above, risk adjustment with administrative data produced a comparable effect to adjustment with detailed chart review. Both risk adjustments concluded that the VA patients were slightly healthier than Medicare patients. When considering a future comparison of health care systems one must consider the alternatives. Do nothing (inexpensive, but unsatisfactory) or do a detailed chart review (expensive, and may still not adequately risk adjust). The benefits of administrative review (relatively inexpensive, national samples, adequate power to examine subgroups of interest) argue for an important role for administrative data in evaluating the care of health systems. The electronic capture of clinical data as part of routine care, currently being tested in VAs cardiac catheterization data capture tool (CART-CL), will ultimately blur the distinction between administrative and clinical data.

A more important question is what should be reviewed. Although outcome is the ultimate product of the health care system's structure and process, any comparison of survival where the patient populations are likely to differ will be a challenge to interpret. The downside of administrative data is their considerable power leads to significant results even when differences in outcome are small. A reasonable approach is to a priori agree to only accept moderate to large differences in outcome as worthy of further evaluation or action. An alternative is to examine process of care measures known to be associated with improved outcome in randomized trials.

Given the results of Landrum's study, should patients at risk for coronary disease leave the VA for Medicare hospitals? This seems unwise given that the VA has recently outperformed Medicare in many process of care measures across many diseases. (Jha et al. 2003) In a comparison of VA and Medicare patients from 2000, process of care measures for acute coronary disease (aspirin and beta-blocker use) and heart failure (measurement of ejection fraction, use of angiotensin converting enzyme inhibitors for left ventricular systolic dysfunction) were an absolute 9 percent–27 percent higher in the VA. The VA has also compared favorably to other managed care organizations in the process of diabetes care with lower HgA1C and LDL cholesterol levels, and more eye examinations and HgA1C screening among diabetics in the VA system (Kerr et al. 2004, Greenfield and Kaplan 2004).

A valid concern raised by Landrum's study is the low rate of angiography and revascularization within the VA. In the earlier chart review study, Peterson found that among patients with a clear indication for angiography, procedure rates were lower for the VA than for Medicare. Data from Landrum's study suggest that the VA-Medicare discrepancy in angiography rates had changed little by the late 1990s. In response to these data the VA is rapidly improving the conditions and number of catheterization laboratories in the system. In addition, VA guidelines now recommend angiography for all patients admitted with a high risk acute coronary syndrome (e.g., elevated troponin).

Landrum's study has had other impacts within the VA system. Since the release of the initial results, each VA hospital has been required to provide a plan to improve the quality of VA care. The VA is now reviewing the charts of every patient discharged with an acute myocardial infarction. The Quality Enhancement Research Initiative (QUERI) for ischemic heart disease (a collaboration of VA Patient Care Services and the Health Services Research Service) is using this data to evaluate the effectiveness of each hospital plan in terms of process of care and outcome. (Demakis et al. 2000, Every et al. 2000)

In summary, Landrum's study is ultimately helpful in that it confirms a discrepancy in angiography use following myocardial infarction between the Medicare and VA systems. The difference in survival between VA and Medicare patients while not inconsistent with an earlier chart review study is difficult to interpret due to the potential for inadequate risk adjustment. Comparing health care systems is an important but extremely challenging exercise regardless of the data source. Hopefully, additional high quality administrative studies will guide us as we focus our efforts to improve quality of care.