Hospital mortality: when failure is not a good measure of success

Mortality for specific conditions and procedures has figured prominently among outcomes-based performance measures.^1–3 However, in recent years there has been renewed interest in overall hospital mortality, specifically the hospital standardized mortality ratio (HSMR) (Box 1).^4–8 Although efforts to measure performance represent welcome developments, the focus on the hospital standardized mortality ratio, especially when the ratio is derived solely from administrative data, goes against the results of decades of investigation.^9,10

Box 1.

Proposed strategies to measure and improve hospital performance efforts have included the establishment of national patient safety agencies,¹¹ mandatory accreditation,¹² financial incentives¹³ and publicly reported performance measures.¹⁴ Performance measures have targeted structure (e.g., patient volumes and recommended staffing patterns for intensive care¹⁵), process (e.g., report cards on adherence to recommended aspects of care for patients with acute myocardial infarction and pneumonia¹⁶) and outcomes.

In this commentary, we apply a previously published framework by Davies and colleagues¹⁷ to analyze the limitations of the hospital standardized mortality ratio as a performance measure. Specifically, we address the validity, precision and potential bias associated with this measure as well as its cost and capacity to promote real improvement in performance (Table 1). We also briefly discuss alternative measures of hospital performance.

Table 1

Validity

For the hospital standardized mortality ratio to represent a valid performance measure, it must correlate with accepted measures of quality. However, the relation between hospital mortality and other measures of quality has been weak and inconsistent.^9,10 One prominent study by Dubois and colleagues¹⁸ reviewed the charts of patients who died from stroke, myocardial infarction or pneumonia in hospitals with substantially higher or lower standardized mortality ratios than expected. High and low outlier hospitals did not differ in terms of adherence to 31 recommended processes of care for patients with these 3 conditions.

In this study, the reviewers of physician charts found no differences among hospitals in terms of the percentages of deaths from stroke or myocardial infarction that were potentially preventable. They did report a higher percentage of preventable deaths from pneumonia in hospitals with high mortality rates; however, agreement for these chart reviews was only 11% better than that expected on the basis of chance alone.¹⁸ More recent studies have confirmed that mortality is a poor predictor of preventable complications and quality problems.^19,20

Precision

A precise measure of hospital performance will yield similar results when applied to hospitals that provide similar quality of care or when applied over time to a hospital where quality has remained constant. Publicly reported Canadian data⁶ show that, in most regions, hospitals experienced an overall decrease in standardized mortality ratios from 2004 to 2007. This decrease was beyond what would be reasonably expected from advances in science or changes in quality of care. In addition, a number of individual hospitals experienced substantial fluctuations from year to year that unlikely represented true cycles of improved or eroded quality.

Simulation studies have also evaluated the precision (and validity) of hospital mortality as a quality measure.¹⁰ Hofer and Hayward²¹ defined hospital quality in terms of the percentage of preventable deaths (25% at poor-quality hospitals and 5% at average-quality hospitals). They performed a Monte Carlo simulation using data from 190 hospitals to generate mortality data. The simulations assumed various prevalence values and showed the resulting false-positive rates from using the hospital standardized mortality ratio to designate poor-quality hospitals. Assuming a true prevalence of poor-quality hospitals of 10%, they found that only 52% of hospitals with significant elevations in the ratio would in fact be classified as poor-quality hospitals. Moreover, 65% of poor-quality hospitals would not have significant elevations in the ratio.

The choice of a risk-adjustment model represents another nonrandom source of variation. Although one cannot expect any specific model to achieve perfect case-mix adjustment, different case-mix models should produce similar hospital rankings. However, several studies revealed striking differences in the classifications of hospitals as either high or low outliers when standardized mortality ratios were calculated using various widely accepted case-mix adjustment models.^22,23

Minimum bias

Measuring hospital performance requires adjustment for important patient factors that predict outcomes. Two problems with administrative data potentially bias the standardized mortality ratio: the omission of important clinical information from routinely collected data and the quality of discharge abstracts (the data source for hospital standardized mortality ratios). If either problem is distributed in a nonrandom manner among hospitals, differences in standardized mortality ratios will reflect bias rather than quality of care.

Patient factors that predict outcomes fall into 3 basic categories: demographics, comorbid conditions and severity of illness (Figure 1). Administrative data capture most demographics (e.g., age, sex) and can approximate socioeconomic status (e.g., by using postal codes). They also capture the number and type of comorbid conditions that patients have (e.g., diabetes, hypertension, dementia), but not the severity of the conditions or the patients' overall functional status. Administrative data provide no information on the severity of acute illness or the complexity of surgical cases. For example, administrative coding of diagnoses does not distinguish mild from severe pneumonia or a stroke that produces moderate deficits that resolve within days from one that causes marked cerebral edema and persistent, dense hemiplegia.

Figure 1: Case-mix adjustment using administrative data to measure hospital performance. Case mix consists of 3 categories of patient factors that predict outcomes of care, including mortality. Administrative data capture demographic factors reasonably well (solid arrow), but they only partially capture comorbidity (dashed arrow) and do not measure severity of illness (broken arrow). *Captured by administrative data. †Partially captured by administrative data. ‡Not captured by administrative data. CABG = coronary artery bypass graft, CT = computed tomography.

Coding accuracy also limits the utility of administrative data for performance measurement.^24,25 Undercoding secondary diagnoses that capture the complexity of patients' conditions will falsely elevate hospital standardized mortality ratios, whereas coding secondary diagnoses that do not affect prognosis will spuriously lower the ratios.^22,26 This explains the paradox observed in several studies that patients who have more chronic illnesses have lower risk-adjusted mortality.^22,26 An additional problem concerns palliative care. Although one might expect that the exclusion of patients receiving palliative care would lower hospital standardized mortality ratios, almost half of Ontario hospitals have been found to have higher ratios after the exclusion of these patients.²⁷ This finding suggests that variation in coding and discharge practices for palliative care patients distort standardized mortality ratios in an unpredictable manner.

Fostering quality improvement

Supporters recommend using the hospital standardized mortality ratio to monitor quality of care over time.^4–6 However, the random fluctuations that most hospitals will exhibit and the low rates of preventable hospital deaths identified in large chart-review studies^18–20 do not bode well for this ratio as a tool for monitoring quality or driving improvement efforts. Nonetheless, 2 published reports describe the use of the hospital standardized mortality ratio to foster and monitor quality-improvement efforts.^4,5

One of these reports⁴ involved an institution that had the highest hospital standardized mortality ratio of all acute care hospitals in England in the year 2000. From 1996 to 2001, this hospital's ratio was more than double the national average on multiple occasions; however, it crossed this “alarm threshold” only twice after 2002. This reduction from multiple crossings of the arbitrarily chosen threshold to only 2 such crossings does not present compelling evidence of real change.

A more noteworthy report by Wright and colleagues⁵ involved a hospital that undertook to lower its standardized mortality ratio despite its rating as an average-quality hospital. The authors attributed the eventual reduction to the following interventions: educational sessions for junior staff on the recognition and treatment of life-threatening emergencies; implementation of medication reconciliation; use of infection-control measures, including a hand-hygiene campaign, staff-awareness sessions, improved ward cleaning, antibiotic guidelines, and surveillance and feedback of infection rates; and changes in the management of patients receiving palliative care, so that they were less frequently admitted to hospital and more frequently discharged to die elsewhere.

Most of these changes would be extremely unlikely to reduce mortality. For example, passive educational efforts generally represent ineffective quality-improvement strategies, and hand-hygiene campaigns tend to produce modest improvements (at best) in hand-washing adherence. No studies performed since the 19th century have demonstrated improvements in hard outcomes such as mortality.²⁸ Similarly, no study of medication discrepancies has shown an effect compatible with reduced mortality.²⁹ The reduction in standardized mortality ratio that Wright and colleagues⁵ observed most likely reflects the changes in palliative care: fewer patients were admitted to die in hospital and more patients were discharged to die elsewhere. Such changes would indeed lower mortality and may even improve care for dying patients, but this hardly counts as saving lives.

Ease of measurement

Hospital standardized mortality ratios derived from routinely collected administrative data cost little to measure.^6,7 However, initiating quality improvement based on these ratios requires more resources than would be needed with other performance measures. Hospital personnel must review numerous charts to identify potentially preventable deaths and then invest further time to find any recurring problems amenable to improvement efforts. Also, a measure that frequently provides misleading information offers low value, even if it is easy to use and costs little.

Other performance measures

Robust performance measurement will likely require multiple measures, including those of outcomes, processes and structures.³⁰ For mortality to provide an adequate measure of performance, either to compare institutions or to track quality of care over time, a focus on single conditions or procedures will be required. Such a focus would facilitate the recognition of explanations for poor performance (e.g., delivery of established processes of care known to reduce mortality among patients with a particular condition) and permit more accurate risk adjustment than would otherwise be possible with overall hospital mortality. In some cases, administrative data alone may be sufficient for risk adjustment.³¹ In others, adequately adjusting for risk may require combining administrative data with select clinical information. In cardiac surgery, for instance, supplementing administrative data with the coronary disease pattern, angina class and left-ventricular function provides an acceptable measurement of risk-adjusted mortality.³² In other cases, more comprehensive collection of clinical data has been considered a worthwhile investment, as with the US Department of Veterans Affairs' National Surgical Quality Improvement Program.³³

The mere avoidance of death conveys little about the quality of care related to most conditions and procedures. Thus, outcomes-based performance measures will have to include important measures of morbidity, such as surgical site infections, key nosocomial infections and functional status (e.g., 30 days after acute stroke), to name a few. Measuring functional status would require substantial investment, but it would serve 2 important goals. First, it would reduce the bias in purely hospital-based measurement owing to variations in lengths of stay (e.g., hospitals would not be rewarded for simply discharging patients to die elsewhere). Second, it would force hospitals to help coordinate follow-up care and decrease the fragmentation that plagues the management of so many patients with serious health problems.

The timely delivery of medicine and other processes of care with a strong relation to important outcomes (e.g., established therapies for acute myocardial infarction) have a clear role to play in performance measurement, but process measures must also be chosen with care. Variations in patient eligibility criteria (or in clinical documentation of such variation) can introduce spurious variations in performance. Overemphasis of process measures can induce “playing for the test” (i.e. hospitals respond to process-based performance measurement by focusing their efforts on the processes of care being measured, regardless of the impact on other aspects of quality), as has likely happened with the focus on time to administration of antibiotics for patients with pneumonia at hospitals in the United States.³⁴

Finally, some structural measures, though relatively blunt, may provide valid performance measures. Patient volume represents a prominent structural measure of quality (since higher volumes tend to correlate with improved outcomes, especially for surgical procedures³⁵), but others might include nurse staffing ratios,³⁶ the organization of intensive care units³⁷ and possibly measures related to hospital governance.³⁸

Advocates of hospital standardized mortality ratios acknowledge that hospitals should adopt a “dashboard” or “scorecard” approach, for which mortality is just one component. However, mortality will always overshadow other measures. The knowledge that a local hospital has exemplary rates of adherence to hand hygiene, for instance, will do little to assuage the public in the face of an apparent excess of deaths. Composite measures have found some application (e.g., a score that combines mortality with performance on several target processes of care). However, just as with composite outcomes in clinical trials, care must be taken to avoid large gradients in importance among components.³⁹ For example, a performance measure for cardiac care that combined mortality after myocardial infarction with various process measures (e.g., rates of smoking-cessation counselling) would raise concerns that differences in performance could be perceived to be associated with mortality but would actually be driven by results for less important components.

Conclusion

The understandable desire to make up for past inattention to health care quality can come at the cost of implementing improvement strategies before they are ready for widespread dissemination.⁴⁰ Hospital standardized mortality ratios can inaccurately label good hospitals as poor performers and fail to identify many hospitals with true quality problems. Moreover, the deceptively low cost of measurement is offset by expenses to institutions that choose to investigate their performance, since, in contrast to other performance measures, hospital mortality ratios provide no indication of the underlying quality problems.

Developing satisfactory performance measures will require substantial investment. Mortality measures will need to focus on specific conditions and procedures with known connections to elements of care that providers are able to control. Adjusting mortality measures for case mix can sometimes be done with the use of administrative data alone, but efforts to obtain clinical information from charts will be required in many situations. Although expensive, the process of obtaining detailed clinical data would support accurate risk adjustment for mortality outcomes. It would also permit capturing other outcomes (e.g., functional outcomes, nosocomial infections, other complications of care) that we will need to highlight if we are truly to take on the task of measuring performance and improving health care quality.

Key points.

• Hospital standardized mortality ratios correlate weakly with other measures of quality of care and have additional limitations when derived solely from administrative data.

• More robust measurement of performance will require a combination of indicators, including mortality for a few specific conditions and procedures, as well as other measures of morbidity and selected processes.

• Developing and implementing valid performance measures will require substantial investment but will avoid the wasted resources.

• Valid and precise measurements will minimize other consequences of mislabelling large numbers of hospitals in terms of the quality of care they deliver.