Hospital standardised mortality ratio: a novel method and approach to risk adjustment

Abstract

Background

Hospital standardised mortality ratio (HSMR) is a simple ratio that is plagued by sparsity, dimensionality, overdispersion, exclusions and controversy.

Objective

Describe Hospital Outcome Prediction Equation V.7 (HOPE-7) methodology.

Setting

State of Victoria (Australia), population 6.8 million.

Methods

Multiphase process: (a) principal diagnoses aggregated into 406 clinical diagnosis groups (CDGs); (b) low case fatality rate (CFR<0.02%) CDGs set aside; (c) remaining CDGs ranked according to predicted risk; (d) final generalised linear model fitted to (75%) training dataset; (e) low-risk cases reinserted and allocated zero risk; (e) model performance in validation dataset assessed for calibration (Hosmer-Lemeshow goodness-of-fit (H₁₀), Brier score, calibration plot), discrimination (area under the receiver operator characteristic (AUCROC) and area under the precision recall (AUCPRC) curves) and classification (dispersion value (φ), SD random effect (τ)). Ideal model: Brier score~0, H₁₀ p value>0.05, AUCROC>0.80, AUCPRC>0.30, φ~1 and τ~0. Classification assessed by proportion of outlier CFR reclassified as inlier HSMR.

Results

315 hospitals treated 12.97 million adult separations and 152 (48.3%) reported 63 806 in-hospital deaths, 0.49 (95% CI 0.48 to 0.50) per 100 separations. 10 722 principal diagnoses allocated to 198 non-significant CDGs, 45 low-risk CDGs (5.05 million cases) assigned zero risk and 163 significant CDGs aggregated to 20 risk ranks. Final model (development cohort 9.73 million) included demographic variables (age, birth sex, emergency, aged-care resident, hospital transfer, relationship status), one interaction term (emergency transfer) and 20 diagnosis-risk categories. Validation metrics (cohort 3.24 million): Brier score 0.015; H₁₀ p value 0.09; AUCROC 0.90 (95% CI 0.87 to 0.92); AUCPRC 0.28 (95% CI 0.25 to 0.31); φ=4.31 and τ=0.24. Study hospitals generated 2192 hospital quarters with 2053 (95.7%) outlier CFR values, of which 1975 (96.2%) reclassified as HSMR inliers.

Conclusions

HOPE-7 is a parsimonious and pragmatic HSMR model based on administrative data common to many jurisdictions that displayed satisfactory calibration, classification and discrimination metrics and addressed frequent HSMR limitations.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Monitoring hospital mortality is highly desirable but problematic due to infrequent events (sparsity), case mix (dimensionality), provider diversity, model overdispersion and dependency on administrative data sources and the International Classification of Diseases taxonomy.

WHAT THIS STUDY ADDS

• The Hospital Outcome Prediction Equation V.7 (HOPE-7) addresses sparsity, dimensionality and case mix diversity, providing a parsimonious model which complements other options.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• HOPE-7 supports risk adjustment for monitoring of hospital mortality where results suggest outcome is attributable to patient-related factors far more than hospital factors.

Introduction

The hospital standardised mortality ratio (HSMR) is arguably the simplest and yet the most controversial of all clinical indicators.¹ It is the ratio of observed to expected deaths, where a result above unity is indicative of more deaths than predicted and, possibly, an increased patient safety risk. Death is easy to recognise, difficult to hide and an outcome that most patients and families prefer to avoid. Therefore, comparing HSMRs at the provider level is highly desirable.

The HSMR remains controversial because it has so often failed to deliver, for several reasons.² These include sparsity, diversity, dimensionality, exclusions, overdispersion and misinterpretation. Death in a hospital is a rare event (less than 1%) and frequently unavoidable.²,⁵ Diversity of case mix generates dimensionality, rendering clinically robust statistical analysis problematic. The range of HSMR options in Australia⁶,⁹ and the UK¹⁰,¹³ (online supplemental table S1) attests to its complexity. The absence of a ‘gold standard’ generates scepticism. It is no surprise that a reliable relationship between the HSMR and the quality of hospital care remains elusive.^{1 14} It would therefore seem reasonable to abandon the HSMR as a quality metric if it were not for the equally compelling defence of its retention.

In Australia and the UK, the highest proportion of deaths occurs in the acute hospital setting.^{11 15} Even though the majority are unavoidable, healthcare providers and the communities they serve are, therefore, entitled to be reassured that unexpected deaths remain low. Moreover, a common element of healthcare scandals has been a rise in unexpected deaths¹⁶ together with the absence (or ignorance) of mortality monitoring, and some evidence that continuous monitoring of mortality may have identified the misconduct sooner^{16 17} and, possibly, saved lives. Finally, a robust HSMR also has utility for epidemiology, research, healthcare policy and planning.¹⁸

An ideal HSMR is inclusive of all patients, all diagnoses and all hospitals; is parsimonious yet minimises misspecification²,⁵; and adjusts for important patient-related factors that influence survival but are outside the control of the treating hospital under assessment. Achieving these goals requires reliable and comprehensive data, robust risk adjustment, adequate calibration (risk ranking), simple visualisation tools, expert clinical interpretation and transparent governance.

In this report, we describe the methodology underlying the Hospital Outcome Prediction Equation V.7 (HOPE-7) model, an HSMR based on the above principles and suitable to any jurisdiction that collects administrative data based on the International Classification of Diseases and Health Related Problems (ICD).¹⁹ Our focus is primarily methodological rather than diagnosis, reporting or governance.

Methods

Setting

Administrative datasets in Australia are routinely extracted from medical records and encrypted by qualified clinical coders according to national²⁰ and state²¹ guidelines and (currently) the 20th edition of the Australian modification of ICD, version 10 (ICD-10-AM).¹⁹ Each record describes a unique ‘episode of care’ and includes patient demographics, documented diagnoses, procedures and outcomes.

Population

The Department of Health Victoria (Australia)²² provided a copy of the jurisdictional administrative dataset for all adult (age greater than 17 years) acute care separations from public and private sector hospitals between July 2018 and June 2023. An identification key permitted linkage of sequential episodes of care to reconstruct the entire hospital spell and identify final outcome. We excluded records without a principal diagnosis (reason for admission), paediatric subjects (since they have a different disease profile), hospice services (where death is an expected outcome) and rehabilitation, geriatric evaluation and mental health services (where therapy and risk profiles differ from acute care). Of note, day-procedure and maternity separations were retained. We selected in-hospital death as the outcome for this report, although censoring of survival status (say, 30 days after admission) is feasible with linkage to the jurisdictional death registry.

Statistical considerations

We employed Stata/MP V.18.0 (2023, College Station, Texas, USA) statistical software. Modelling a rare event (death) in a highly dimensional dataset (over 10 000 diagnoses and 2 million records per annum) carries substantial risk of misspecification, overfit and overdispersion. A multiple phase approach (online supplemental figure S1) was selected to mitigate these risks: phases I and III reduced over 10 000 (ICD-10-AM diagnosis codes and) candidate variables to a clinically meaningful and statistically manageable number; phase II addressed the sparsity of outcome (in-hospital death); phase IV generated the final model; and in phase V, we assessed model performance.

Phase I: aggregation of ICD-10-AM codes

Each ICD-10-AM principal diagnosis¹⁹ code was allocated to 1 of 406 predefined clinical diagnosis groups (CDGs). A CDG is defined as a set of ICD-10-AM diagnosis codes that describe a similar clinical condition with similar pathology and are described in more detail elsewhere.²³ For example, community-acquired bacterial pneumonia includes 18, acute stroke includes 39 and chronic kidney disease includes 28 separate ICD-10-AM codes. The CDG classification system is conceptually similar to, but not identical with, the Clinical Classification Software,²⁴ which is frequently employed in HSMR models (online supplemental table S1). Our CDG classification system attempts to address the inherent collinearity, conflation and confounding of the Clinical Classification Software when applied to mortality prediction models.²³

To aid post hoc stratification and sensitivity analyses (not modelling of HSMR), all coded diagnoses were allocated to their relevant CDG to identify comorbidities, chronic disease and complications in each record. Each hospital was allocated to one of six peer group categories: tertiary referral, major metropolitan, major regional, other public and tertiary or other private sector hospitals. Variance at the hospital or patient level was assessed by the intraclass correlation coefficient (ICC) where an ICC (probability) of zero indicates none, and an ICC of unity indicates all, of the mortality variance is attributable to this level.²⁵

Phase II: low-risk separations

Several patient subgroups, notably maternity and day-procedure admissions, have a negligible case fatality rate (CFR) and their inclusion in a prediction model increases the risk of overfit. To address this risk, we identified all CDGs with an observed CFR less than 1 in 5000 (<0.02%) and temporarily excluded them from phases III and IV then reinstated all in the study population at the validation (phase V) with their mortality risk set at zero.

Phase III: ranking of CDG

The purpose of this phase was to quantify the significance of any association between each CDG and the outcome (hospital mortality) and rank each with a weight commensurate with the resultant multivariate risk-adjusted coefficient (ß). Only those CDGs containing a minimum of 50 cases and 30 deaths over 5 years²⁶ were eligible candidate variables. A mixed-effects regression model was fitted to the outcome and adjusted for available patient demographics (age, sex, source of admission, urgency, indigenous, ethnic and domiciliary relationship status) and year of separation to adjust for any temporal shifts. A random intercept for hospital site permitted adjustment for site variations in services and model of care.

Following an iterative process, each candidate variable was interrogated and retained if it (a) was a significant (p<0.157 level²⁷) predictor of outcome and either (b) improved (reduced) the model’s information criteria (consistent AIC and BIC²⁸) or (c) strengthened the coefficient of a dominant covariate. Of note, we did not employ (forward or backward) stepwise selection methods,^{5 6} which may inadvertently discard important covariates.^{27 29}

Statistically significant CDGs were ranked and grouped by risk coefficient (ß) using the Stata command ‘xtile’, with the final number of categories (and the ß range for each) determined, once again, by the effect on the model information criteria. It is important to note that the coefficients generated from this interim (phase III) model were discarded and not employed in the final model.

Phase IV: final model

The HOPE-7 model employed a generalised linear model with a logit link function fitted to the outcome (in-hospital death) while the candidate variables were restricted to significant patient demographics and the ranked diagnosis groups from phase III. The final HOPE-7 model (differed from the interim Phase III model and) excluded variables pertaining to hospital site or year of admission. The form of the estimator was,

logit death age male aged-care emergency transfer single interaction rank-1 rank-2 … rank-i, vce(cluster hospital)

where logit represents the Stata command for a standard logistic regression estimator fitted to the outcome; death is the binary outcome (in-hospital death); age in years transformed to the square root, male (birth) sex, aged-care resident, emergency admission status, hospital transfer to higher level of care, single (relationship) status and interaction term for unplanned interhospital transfer; rank-1 to rank-i represent the ranked categories of CDGs identified in phase III; and the error term adjusted for clustering at the hospital level.

Phase V: model validation

Validation of the HOPE-7 model involved discrimination, calibration, classification and dispersion metrics.²⁹ The large population favoured a standard validation method (random 75:25 split into training and validation cohorts) rather than a more computationally demanding cross-validation or bootstrapping procedure. Coefficients to be generated from the training cohort were applied to the validation cohort for assessment of model fit. The Brier score and Hosmer-Lemeshow goodness-of-fit statistic (H₁₀ with equal sized deciles) were employed to assess calibration.³⁰ The user-written command calibrationbelt³¹ furnished a visual and statistical assessment of calibration. Discrimination was reported as the area under the receiver operator characteristic curve (AUCROC). Since the AUCROC is misleading in an unbalanced dataset (survivors>>deaths), we also report the area under the precision recall curve (AUCPRC³²). An ideal model will produce a Brier score approximating zero, an H₁₀<15.5 (8th df) with p value>0.05, AUCROC >0.80 and AUCPRC >0.30.

Since the primary purpose of an HSMR is assessment of patient outcome at the provider level,⁶,¹⁴ we assessed hospital classification and dispersion characteristics in the following manner. The final model was recalibrated to each fiscal year, generating an annual benchmark and the HSMR in each fiscal quarter for each hospital (hospital quarter) was compared with the benchmark with control limit precision determined by the number of predicted deaths, employing the user-written command funnelinst.³³ The number and proportion of CFR outliers subsequently reclassified as HSMR inliers were thus counted. Overdispersion was quantified by the dispersion value (φ) and the SD of the random effect of the residuals (τ).³⁴

After applying an ideal prediction model to a group of hospitals providing a uniform high standard of care, we would expect φ to approximate unity, τ to approximate zero and fewer than 1% of HSMR values to exceed ±3 SD of the benchmark. Sensitivity analyses of calibration, discrimination and dispersion were undertaken for each hospital peer group and each fiscal year.

Results

In 2023, the State of Victoria had an estimated population of 6.81 million including 5.28 million adults¹⁵ (figure 1) served by 315 acute care hospitals. 152 (48.3%) hospitals reported all 63 806 deaths and 10.78 million (83.0%) adult separations. Observed hospital mortality rates ranged from 0.003 to 15.4 (online supplemental figure S2) with a state mean of 0.49 (95% CI 0.48 to 0.5) deaths 100 separations.

Figure 1. Flow diagram of study population. More than one exclusion criterion possible; low risk=case fatality rate <0.02%.

From the 14 470 ICD-10-AM diagnosis codes available,¹⁹ 12 145 (83.9%) were employed as principal diagnoses by clinical coders over the 5 years and we allocated each to one of 406 CDG sets²³ (phase I). After the exclusion of 45 (11%) low-risk (phase II) and 198 non-significant CDGs, there were 163 CDGs identified as significant predictors of outcome, which were subsequently aggregated into twenty ranked categories (phase III). Rank 1 included all CDGs with ß< −0.9; rank 2 with ß range −0.9 to −0.5; rank 3, ß range 0.5 to 0; rank 4, ß 0; rank 5, ß 0 to 0.4; rank 6, ß 0.4 to 0.7; and so forth, to rank 18, ß 4.1 to 4.4; and rank 19, ß>4.4. All low-risk CDGs were allocated zero risk (rank 0). Thus, phases I and III condensed over 12 000 (ICD-10-AM) diagnoses into 20 risk categories.

For the (phase IV) final model, each record was assigned eight covariates: six demographic variables (age, birth sex, unplanned admission, aged-care resident, interhospital transfer source, single relationship status), one interaction term (unplanned inter-hospital transfer) and one of the twenty ranked risk categories. Details of model covariates and worked examples are available in online supplemental table S3 and online supplemental table S4, respectively.

For the validation phase, the study population was randomly divided into training (9.73 million separations) and validation (3.24 million) cohorts. A calibration plot generated from the validation cohort is displayed in figure 2 and classification (funnel) plot in figure 3. Over the 20 fiscal quarters, the mean Brier score was 0.015 (SD 0.002) and the mean H₁₀ statistic was 14.88 (p=0.094; online supplemental table S4). The mean AUCROC was 0.895 (95% CI 0.874 to 0.915) and AUCPRC was 0.277 (95% CI 0.247 to 0.306). Results from the sensitivity analyses for peer groups, fiscal years and quarters are available in table 1, online supplemental figure S3; table 1, online supplemental figure S4; and online supplemental tables S5, S6, respectively.

Figure 2. Hospital Outcome Prediction Equation V.7 calibration plot in external validation dataset (n=1.09 million), excluding low-risk admissions. Test statistic=99.4 (p<0.001). Bisector (red line) identifies ideal model.

Figure 3. Funnel plot of Hospital Outcome Prediction Equation V.7 (HOPE7) hospital standardised mortality ratio (HSMR) for 100 hospitals (solid circles) in validation dataset reporting 3377 deaths in 611 919 separations for fiscal year ending 30 June 2023 compared with the state benchmark (dashed lines represent ±2 SD and ±3 SD, derived from training dataset); without adjustment for overdispersion (Φ=4.0 and τ=0.29); excluding 75 (42.9%) hospitals reporting no deaths. EHICR = Eastern Health Intensive Care Research team.

Table 1. Outcomes and HOPE-7 model performance for annual and peer group cohorts.

Patient group	LOS, mean (SD) days	Case fatality rate	AUCPRC	AUCROC	Brier score	H10
2018–2019	1.57 (4.40)	0.48%	0.273	0.899	0.014	21.6
2019–2020	1.57 (4.47)	0.49%	0.278	0.894	0.014	27.0
2020–2021	1.56 (4.51)	0.47%	0.264	0.899	0.014	18.4
2021–2022	1.56 (4.57)	0.52%	0.290	0.896	0.016	34.6
2022–2023	1.58 (4.78)	0.50%	0.281	0.888	0.016	42.6
Tertiary referral	1.58 (4.55)	0.79%	0.305	0.937	0.017	17.6
Major metropolitan	1.79 (4.87)	0.71%	0.344	0.949	0.015	22.5
Major regional	1.61 (4.59)	0.47%	0.234	0.950	0.017	259.4
Other public	0.78 (2.91)	0.21%	0.233	0.967	0.017	50.7
Major private	2.11 (4.64)	0.46%	0.263	0.950	0.011	25.9
Other private	1.45 (3.58)	0.34%	0.228	0.970	0.010	174.9

AUCPRC, area under precision recall curve; AUCROC, area under receiver operator characteristic curve; H₁₀, Hosmer-Lemeshow χ2 statistic with equal-sized deciles. An ideal model will produce an AUCPRC>0.30, AUCROC>0.80, Brier score approximating zero, and H₁₀<15.5 (8 df); HOPE-7, Hospital Outcome Prediction Equation V.7; LOS, average length of hospital stay.

Observed (CFR) and predicted (HSMR) mortality rates for 2192 hospital quarters were generated for 152 (48.3%) hospitals reporting in-hospital deaths. Of the 2053 (95.7%) CFR values categorised as outliers (exceeding ±3 SD of state mean) there were 1975 (96.2%) reclassified as HSMR inliers. No CFR inlier was reclassified as an HSMR outlier. The dispersion characteristics, Φ=4.3 and τ=0.24 (online supplemental table S5) indicated the persistence of overdispersion due, in part, to 145 (66.8%) HSMR outlier results from smaller hospitals that reported fewer than five deaths in that quarter. The (unconditional) ICC values for hospital level and patient level were 0.05 (95% CI 0.03 to 0.10) and 0.61 (95% CI 0.60 to 0.62), respectively, indicating a low probability that the observed variance in hospital mortality rates was attributable to hospital factors.

Discussion

Principal findings

This report describes the methodology underlying, and performance characteristics of, the HOPE-7: a parsimonious and inclusive HSMR that complements existing methods and displays acceptable discrimination, calibration and classification metrics (table 1, and online supplemental figures S3–S5) for the purpose of monitoring hospital performance and patient safety.

Like many other HSMR models, HOPE-7 is derived from administrative data and ICD-10 diagnosis codes and it has several features that address some of their limitations (online supplemental table S1). HOPE-7 is inclusive of all hospitals irrespective of size, activity or peer group, and all acute care separations both low risk (day procedure and maternity) and high risk. While the methodology appears complex, the model is not. It requires only eight covariates, one formula, but encompasses all admission diagnoses.^{8 9} Linkage with the jurisdictional death register permits censoring of outcome at a fixed time, although in practice, this may delay analysis and reporting. In-hospital mortality may permit more prompt reporting.

Strengths and limitations

HOPE-7 is parsimonious and applicable to any jurisdiction where administrative data are based on ICD-10, and applicable to hospitals of different size, activity, case mix or service level. The ranking of diagnoses is based on clinically meaningful aggregation of diagnoses (CDG) rather than the ICD-10-AM (highly dimensional) taxonomy or problematic aggregation methods.³⁵ By design, the model excludes (a priori) hospital or treatment-related covariates^{6 13} which the model seeks to assess.

Our methodology has several limitations. Administrative data based on ICD lack granularity (clinical severity) and are dependent on the accuracy of coding and, more importantly, the quality of clinical documentation from which they were derived.²⁰ Although the multiphase methodology appears somewhat cumbersome, this is, arguably, an unavoidable feature of most HSMR models. The diversity of hospital case mix, the dimensionality of ICD-10-AM and the limitations of administrative data all conspire to obfuscate analysis. Moreover, these models are not designed to guide clinical management decisions; they are descriptive rather than prescriptive.

Interpretation

While the primary purpose of this report is methodological, and a thorough review of current options is beyond its scope, certain findings warrant brief comment. Our results suggest a high degree of uniformity in patient outcomes under the current model of care in Victoria, despite the diversity of case mix, acuity and providers. Of note, a substantial period of this investigation straddled the recent SARS-CoV-2 pandemic (table 2), during which case mix shifted and profound changes in the model of care were required. Moreover, hospital survival appears to be determined more by patient-related factors (ICC, 0.61) rather than hospital-related factors (ICC, 0.05) despite the wide range in observed hospital mortality rates (online supplemental figure S2). Our findings suggest that patient demographics and the reason for hospital admission are more significant drivers of hospital outcome, which is consistent with published clinical reviews.²,⁵ These observations do not, however, negate the benefit of continuous monitoring of hospital mortality, unexpected deaths and temporal trends. Yesterday’s reassuring low HSMR provides no guarantee for tomorrow.

Table 2. Population demographics according to fiscal year and hospital peer groups.

Cohort	Records	Death	Age, years IQR	Male	Aged-care resident	Interhospital transfer	Low risk	SARS-CoV-2
2018–2019	2 613 047	12 594	44–74	47.4%	21 097	29 015	993 750	0
2019–2020	2 513 284	12 216	44–74	47.7%	22 392	26 701	972 671	298
2020–2021	2 569 211	11 966	44–74	47.4%	23 284	28 988	1 007 669	3060
2021–2022	2 568 399	13 427	44–75	47.4%	20 201	27 420	1 020 612	42 783
2022–2023	2 710 691	13 603	44–75	47.4%	18 667	26 147	1 052 837	29 977
Tertiary referral	2 371 887	18 672	43–74	53.3%	38 495	68 376	736 628	29 343
Major metropolitan	2 710 035	19 160	40–74	47.2%	30 304	33 281	1 037 037	29 971
Major regional	1 158 755	5398	47–75	48.9%	13 452	13 077	477 644	6833
Other public	3 462 713	7399	44–75	44.6%	22 409	3958	1 841 434	6405
Major private	1 754 036	7986	50–75	47.8%	482	16 016	437 421	2388
Other private	1 517 206	5191	46–74	43.7%	499	3563	517 375	1178

Population demographics according to fiscal year and hospital peer groups.

Transfer=up-transfer from similar or lower service level; low risk=proportion of separations allocated to clinical diagnosis group with case fatality rate<0.002.

Implications for policy, practice and research

The HOPE-7 model complements rather than supercedes other HSMR methods by addressing several of their common limitations (online supplemental table S1). All mortality prediction models employed in Australia⁶,⁹ and the UK¹⁰,¹³ provide important insights despite their limitations. These models are screening rather than diagnostic tools.^{1 2 11} They seek to identify ‘signals of interest’ that warrant further attention, where pejorative terms such as ‘outlier’ are best avoided. Access to more than one monitoring tool facilitates triangulation of such signals. They are formative rather than summative, descriptive rather than prescriptive. They generate probabilities based on groups, not individuals, which inform but do not determine clinical decisions.

In conclusion, the HOPE-7 model is a parsimonious mortality prediction model derived from administrative data, which provides a pragmatic method for epidemiology and monitoring mortality in adult acute health services and seeks to address some of the common pitfalls that pursue current HSMR options.

Data availability statement

Data may be obtained from a third party and are not publicly available.