Public reporting of the hospital standardized mortality ratio (HSMR): implications for the Canadian approach to safety and quality in health care

Things do not get better by being left alone.

– Sir Winston Churchill

The recent provocative report of the Canadian Institute for Health Information (CIHI), entitled HSMR: A New Approach for Measuring Hospital Mortality Trends in Canada, proposes the hospital standardized mortality ratio (HSMR) as a measure that can aid hospitals, regions and provinces in their attempts to improve patient safety.¹ The HSMR compares actual numbers of deaths in a hospital or health region to the number of deaths expected given the types of patients receiving care. This measure has been used in several countries to assess numbers of inpatient deaths, stimulate hospital care improvements, and track success in decreasing inpatient mortality.^2,3 It has also been used in conjunction with other measurements to provide more detailed data on hospital performance.⁴

The HSMR used in the CIHI report compares the actual number of deaths among patients with 65 diagnoses (accounting for 80% of inpatient mortality) with the expected number of deaths using a logistic regression model that controls for age, sex, duration of stay in hospital, reason for admission to hospital, principal diagnosis, comorbidities, and hospital transfers.⁵ Data for the report originate from all acute care hospitals in Canada, excluding Quebec, with an annual number of expected deaths greater than or equal to 20 for the period from April 2004 to March 2007 inclusive.¹

The HSMR report captured Canadians’ attention as multiple local, provincial and national media outlets reported on the results. Although CIHI made repeated assertions that the HSMR should not be used to compare institutions but, rather, should be used as an internal monitor of quality of care, two major Canadian newspapers nonetheless published rankings of hospital performance while other media sources pointed out high- and low-performing hospitals but did not produce a rank order.

So what exactly should hospitals, regions and provinces do with this information? In Calgary, for example, 3 acute care hospitals — the Foothills Medical Centre, the Peter Lougheed Centre and the Rockyview General Hospital — had HSMRs of 84, 88 and 94, respectively, with corresponding Canada-wide media-reported rankings of 9th, 15th and 25th. In such cases, should hospital administrators and providers be satisfied with their performance and do nothing new? Or should they nonetheless seek areas that need improvement? Is it understood what the HSMR measures? What does it fail to measure? Ultimately, does the HSMR address the information needs of hospitals to truly effect change?

While the HSMR report should be applauded for its foray into public reporting in Canada, there are several cautions to consider with the HSMR measure. First, the measure relies on the accurate coding of diagnoses and comorbidities within the CIHI database; the reliability of the coding affects the accuracy of the determination of expected mortality, a critical component of the overall measure (i.e., the denominator of the ratio). Despite this concern, the CIHI administrative database in general has been shown to be fairly accurate in the coding of clinical diagnoses.⁶ To ensure data accuracy, hospitals were given the opportunity to “validate” their data before the release of the HSMR report; as a result of this process, 7 hospitals from 4 hospital corporations declined to have their data appear in the published report. Although this suppression of information for certain hospitals may be on the basis of legitimate data quality issues, it could be argued that such suppression should not be permitted, as it perpetuates (and rewards) poor data practices. Regardless of the reasons for hospitals not publishing their information, observers may suspect that non-participating hospitals actually had poor HSMRs and did not want such information in the public domain.

A second caution is that the HSMR focuses entirely on mortality as an outcome, an incomplete measure of quality of care.⁷ It may be better to measure only the processes of care to determine quality of care, although good performance on process measures does not always decrease in-hospital mortality.⁸ For this reason, it has been suggested that the best hospital performance measures are those that combine outcome indicators, such as risk-adjusted mortality, with process measures such as the use of proven therapies, to provide a more comprehensive picture of quality of care; this approach would be in keeping with the Donabedian framework of quality,^9,10 which incorporates elements of structure, process, and outcomes (Figure 1). For example, the US Institute for Healthcare Improvement uses “whole system measures,” a set of 13 measures (including the HSMR) that addresses structure, process and outcomes of care.⁴ Items include readmission rates, rates of adverse events, functional health outcome scores, patient satisfaction, timeliness in receiving health care, and health care costs. The HSMR as reported by CIHI does not provide this context, nor does the accompanying report suggest that other measures of hospital structure and processes of care should be used in conjunction with the HSMR.

Figure 1.

The Donabedian model of measuring health care system performance

The HSMR measure also overlooks complexities of care within and supporting a hospital (see Box 1). For example, hospitals with greater doctor-to-bed ratios have lower HSMRs, as do teaching hospitals.² Higher discharge rates to patient homes are associated with higher HSMRs, whereas the presence of a greater number of health facilities in the area surrounding a hospital is associated with lower HSMRs, perhaps reflecting a hospital’s ability to move patients out of the acute care setting into more suitable long-term or hospice care. These examples point toward structural elements of quality in Donabedian’s framework¹⁰ that are indirectly measured by the HSMR, and that may not be obvious to administrators and providers as they try to interpret the complex HSMR measure.

Box 1.

The HMSR: strengths, benefits, caveats and limitations

Although CIHI provides hospitals with supplementary analyses (e.g., HSMR for ICU-related cases, excluding transfers), the HSMR remains a composite measure, reflecting an institution’s overall mortality rate with respect to 65 diagnoses (ranging from cancer to various cardiovascular diseases, infections such as pneumonia, and trauma such as hip fracture) that constitute 80% of in-hospital deaths. Given such a broad range of diagnoses that would be cared for under different departments and care units in a hospital, it is difficult to pinpoint where problems with quality of care reside.

CIHI provides working examples of hospitals that have used the HSMR to reduce avoidable deaths¹¹ and a one-page resource¹² for participating hospitals outlining how to understand and interpret the HSMR. This resource also suggests consulting Safer Healthcare Now!, a national campaign to improve patient safety in relation to 6 specific conditions.¹³ However, hospitals are not provided with condition-specific data and are therefore unable to take a targeted approach to adopting the Safer Healthcare Now! interventions for specific conditions such as acute myocardial infarction and bloodstream infections. Provision of condition-specific data would greatly enhance the value of the global HSMR and would be more likely to stimulate quality-of-care improvements targeted to specific conditions.

To provide more detailed reports with condition-specific and process and structure information, data outside of those available to CIHI are required. Provincial ministries of health and health quality councils have access to “meso-” and “micro-level” data, such as medication use and physician claims, and health authorities and hospitals have access to “micro-level” data such as chart reviews and electronic health records (Figure 2). The sharing of such meso- and micro-level data has great potential to improve our understanding of health system quality.

Figure 2.

Levels of health care administration in Canada and data contributions to health care quality reporting

The richer data sources from these 3 levels of administration provide condition-specific data that, in addition to mortality data, speak to other outcomes along with measures of health system process and structure. By utilizing richer data sources, more detail can be provided in performance reports that are simpler and provide more actionable information than does a single, composite measure such as the HSMR. That is not to say that the HSMR should not be used at all, but rather that it should be used in combination with other information so that a hospital with a high HSMR would be able to look to supporting condition-specific data on process, structure and outcomes to determine the source of the problems. For example, a performance report containing a high HSMR (macro-level outcome measure) coupled with a high myocardial infarction death rate (meso-level condition-specific outcome measure), and a low usage of beta-blockers after myocardial infarction (micro-level process measure) would better equip a hospital to launch targeted performance improvement measures to improve its overall score (HSMR) and, ultimately, the quality of care it provides.

“Things do not get better if left alone.” The HSMR report has stimulated discussion regarding quality of health care in Canada. It will have a further short-term impact if it aligns all hospitals with national coding practices and initiates or maintains quality improvement practices in hospitals. However, over the longer term, the report is unlikely to improve the quality of hospital care without additional data to help providers and administrators pinpoint the problems. Acquiring this additional information requires CIHI and other organizations to overcome obstacles relating to decades-old federal-provincial health care barriers created by jurisdictional and funding realities. Canadians deserve the intergovernmental cooperation required for comprehensive reporting on health care quality to effect meaningful change and improvements in their health care system.

Biographies

Dr. Susan E. Brien is a post-doctoral fellow in the Centre for Health and Policy Studies, Department of Community Health Sciences, University of Calgary, Calgary, Alta.

Dr. William A. Ghali is a professor in the Centre for Health and Policy Studies, Department of Community Health Sciences, and a general internist within the Department of Medicine, University of Calgary at the Foothills Medical Centre, Calgary, Alta.