Separating the wheat from the chaff: How to measure hospital quality in routine data?

Abstract

Objective

To measure hospital quality based on routine data available in many health care systems including the United States, Germany, the United Kingdom, Scandinavia, and Switzerland.

Data Sources and Study Setting

We use the Swiss Medical Statistics of Hospitals, an administrative hospital dataset of all inpatient stays in acute care hospitals in Switzerland for the years 2017–2019.

Study Design

We study hospital quality based on quality indicators used by leading agencies in five countries (the United States, the United Kingdom, Germany, Austria, and Switzerland) for two high‐volume elective procedures: inguinal hernia repair and hip replacement surgery. We assess how least absolute shrinkage and selection operator (LASSO), a supervised machine learning technique for variable selection, and Mundlak corrections that account for unobserved heterogeneity between hospitals can be used to improve risk adjustment and correct for imbalances in patient risks across hospitals.

Data Collection/Extraction Methods

The Swiss Federal Statistical Office collects annual data on all acute care inpatient stays including basic socio‐demographic patient attributes and case‐level diagnosis and procedure codes.

Principal Findings

We find that LASSO‐selected and Mundlak‐corrected hospital random effects logit models outperform common practice logistic regression models used for risk adjustment. Besides the more favorable statistical properties, they have superior in‐ and out‐of‐sample explanatory power. Moreover, we find that Mundlak‐corrected logits and the more complex LASSO‐selected models identify the same hospitals as high or low‐quality offering public health authorities a valuable alternative to standard logistic regression models. Our analysis shows that hospitals vary considerably in the quality they provide to patients.

Conclusion

We find that routine hospital data can be used to measure clinically relevant quality indicators that help patients make informed hospital choices.

What is known on this topic

• To support patients in their hospital choices, the public disclosure of hospital quality measures has become a top priority in many health care systems.

• To account for systematic differences in patient risks across hospitals, public health authorities typically apply simple risk adjustment models.

• Logistic regression models that account for case‐mix differences between hospitals are the most popular approach for risk adjusting patient outcomes.

What this study adds

• We propose two novel approaches that are more suitable to adjust hospital performance for differences in patient risk than the standard logistic regression approach.

• Our approach is tailored to practitioners and public health authorities who intend to compare hospital quality based on routine inpatient data.

• Hospitals vary considerably in the quality they deliver to patients who undergo inguinal hernia repair and hip replacement surgery.

1. INTRODUCTION

Many developed countries including Germany, the Netherlands, Belgium, Switzerland, and the United States (Affordable Care Act marketplaces & Medicare) have adapted elements of “managed competition” in their health care systems. ¹ , ² , ³ In such systems, patients choose the health plan that fits their needs and they typically have some discretion over who treats them. ⁴ Choice, the rationale goes, can have a disciplining effect on providers of care and insurers because “poor” providers will lose patients/clients unless they enhance their quality efforts or lower their costs. ⁵ , ⁶

The public disclosure of quality information has become a top priority in many health care systems. Supporting patients in their provider choices can contribute to a more efficient allocation of patients to the best providers in the system and ultimately enhance the quality of care. ⁷ , ⁸ , ⁹ However, these mechanisms can only unfold if two basic conditions are met. First, quality measures need to account for differences in patient risks between providers. Some hospitals, such as university hospitals, treat more complex cases than other hospitals. Directly comparing hospitals would unfairly penalize hospitals with more complex cases and provide a distorted quality signal. Thus, accounting for differences in patient risk is necessary to ensure a fair provider comparison.

Second, quality measures need to be meaningful and easy‐to‐understand for patients. A growing body of research documents that patients have difficulties understanding quality information ¹⁰ , ¹¹ , ¹² and as a result often base their provider choices on non‐clinical factors such as travel distance or amenities. ⁴ , ¹³ , ¹⁴ , ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ Moreover, the quality indicators that are available might be of little use to inform provider choices. For example, mortality rates are largely meaningless for patients undergoing high‐volume, low‐risk procedures such as hernia repair or hip replacement because patient death is an extreme event that rarely occurs during these procedures.

In this article, we study the question as to how to measure hospital quality indicators based on routine data available in most developed countries. We examine alternatives to the “standard approach” of risk adjusting clinical outcomes based on logistic regression models commonly used by researchers and public health authorities including, among others, the Center for Medicare and Medicaid Services (CMS) in the United States and the Allgemeine Ortskasse (AOK) in Germany. ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³¹ Specifically, we assess how least absolute shrinkage and selection operator (LASSO), a supervised machine learning technique for variable selection, and Mundlak corrections in hospital random effects logits can be applied to account for risk selection and thus make hospitals more comparable. Moreover, we investigate to what degree hospitals vary in the quality they deliver to patients. Our analysis is confined to two high‐volume, low‐risk elective procedures: inguinal hernia repair and primary hip replacement. We analyze clinically meaningful events including complications, infections, and revisions as they provide direct information on patient health based on clinical outcomes instead of indirect measures such as nurse‐to‐patient ratios or the adherence to clinical guidelines. ³² , ³³ Finally, to make the resulting quality indicators more accessible to patients, we translate hospital performance to a simple 3‐star rating.

2. RISK SELECTION

In the Swiss health care system, the federal states (“Cantons”) determine which hospitals can reimburse their services through mandatory health insurance. Coverage is broad so that only a negligible fraction of hospitals is not part of these cantonal “hospital lists.” Patients with mandatory health insurance have free choice among all listed hospitals across Switzerland, that is, access to inpatient care is not restricted to the region of residence or plan choice. ³⁴

Free choice, however, has important implications for the comparison of hospitals. With choice comes selection because certain types of patients select into certain hospitals. As a result, patient risks are unevenly distributed across hospitals—what we refer to as “risk selection” hereafter. Some hospitals might benefit from a desirable risk pool, while others predominantly treat complex cases (e.g., chronically ill, older patients). Such differences in the risk structure likely affect patient outcomes such as complication or infection rates.

Figure 1 illustrates the uneven distributions of risk factors across hospital types in the Swiss inpatient sector in case of inguinal hernia repair surgeries. We classify the 123 acute hospitals that performed inguinal hernia repair in 2019 into one of the five size‐based hospital categories defined by the Federal Office of Statistics: university hospitals, center hospitals, specialized clinics as well as large and small regional hospitals. To give a flavor about the differences in patient risks, we compare average Elixhauser comorbidities per hospital type, which can be constructed based on the International Statistical Classification of Diseases and Related Health Problems 10th Revision (ICD‐10) diagnosis codes. ³⁵ , ³⁶ Figure 1 shows the share of inguinal hernia repair patients suffering from specific comorbidities for each hospital type: brighter colors indicate a higher share of comorbid patients. The figure underlines that patient risks are not evenly distributed across hospitals: university hospitals and central hospitals treat more patients with comorbidities, which is an indication of higher risks of adverse outcomes and thus risk selection.

FIGURE 1.

The share of patients undergoing inguinal hernia repair who suffer from comorbidities by hospital type for the years 2017–2019.

3. METHODS

3.1. Data

We use the Swiss Medical Statistics of Hospitals, which is a comprehensive administrative hospital dataset of all inpatient stays in acute care hospitals in Switzerland. ³⁷ The data are routinely collected and publicly available. Similar datasets are available in other countries such as the United States (e.g., CMS data), the Netherlands (e.g., National Basic Register of Hospital Care), the United Kingdom (e.g., Hospital Episode Statistics), France (e.g., French Hospital Discharge Database), Sweden (e.g., National Patient Register), Finland (e.g., Care Register for Health Care), Denmark (e.g., National Patient Register), and Norway (e.g., Norwegian Patient Registry).

The Medical Statistics of Hospitals includes patient demographics (e.g., age, gender), socioeconomic characteristics (e.g., type of insurance), administrative information (e.g., hospital type), and medical information (e.g., main and secondary diagnoses and procedures). Diagnoses are coded based on the ICD‐10 German Modification (ICD‐10‐GM). Procedures are coded with the Swiss classification of surgical interventions (CHOP). Unique patient identifiers allow us to follow patients over time to capture adverse events during re‐hospitalizations. We enrich the dataset by using secondary diagnoses to create the Charlson ³⁸ and Elixhauser comorbidity index ³⁶ as well as binary indicators for the underlying comorbidities and the number of comorbidities using the R package “comorbidity.” ³⁹

3.2. Study sample

The data analyzed in this study include all hospitalizations and re‐hospitalizations of patients with a primary hip replacement due to coxarthrosis or chronic arthritis or an inguinal hernia repair surgery over period 2017–2019, that is, the pre‐COVID years. Because of long follow‐up periods of certain quality measures (see Section 3.3), we analyze hospital quality for primary hip replacement in 2018. In 2018, 118 hospitals performed 18,028 primary hip replacements due to coxarthrosis or chronic arthritis. While the number of hip replacements remained constant over time (18,086 in 2017, 18,028 in 2018, and 18,175 in 2019), the number of inguinal hernia repairs decreased markedly (17,219 in 2017, 15,634 in 2018, and 12,061 performed by 123 hospitals in 2019). Since 2019, mandatory insurance (with some exceptions) only reimburses inguinal hernia repair if performed in the ambulatory setting. Due to likely changes in the patient risk structure after the reform, we thus confine the analysis to patients treated in 2019.

3.3. Quality measures

We assess hospital quality for the two surgeries based on outcome indicators. For both indications, we collected a set of evidence‐based, disease‐specific indicators that are used by public health agencies in five countries (the United States, the United Kingdom, Germany, Austria, and Switzerland). The final set of indicators results from a two‐step selection procedure. In the first step, we systematically extracted all indicators used by leading agencies in the United States (Center for Medicare and Medicaid Services, CMS; Agency for Healthcare Research and Quality, AHRQ; National Quality Forum, NQF), the United Kingdom (National Health Service, NHS; National Joint Registry, NJR; Getting It Right First Time, GRIFT), Germany (Allgemeine Ortskasse, AOK; Institut für Qualitätssicherung und Transparenz im Gesundheitswesen, IQTIG), Austria (Austrian Inpatient Quality Indicators, A‐IQI), and Switzerland (Bundesamt für Gesundheit, BAG; Nationaler Verein für Qualitätsentwicklung in Spitälern und Kliniken, ANQ; Swissnoso). In the second step, we narrowed down the set of indicators to outcome measures. Such measures are considered the “gold standard” in quality measurement as they directly measure patient outcomes instead of indirect signals such as process quality or hospital endowment (e.g., number of beds). The final set is restricted to the most frequently used indicators, which can be constructed based on routine data available in most developed countries.

We measure quality of inguinal hernia repair surgery based on the following two quality dimensions: general complications during initial stay (e.g., cardiac arrest) and surgical complications within 90 days after the surgery (e.g., vascular complications). The four final quality indicators for primary hip replacement include general complications during initial stay (e.g., myocardial infarction), surgical complications within 90 days after the surgery (e.g., luxation), infections within 365 days after the surgery, and revisions within 365 days after the surgery. Tables 5 and 6 in Appendix S1 provide a detailed account on all adverse events as well as procedure and diagnostic codes used to construct the quality indicators for inguinal hernia repair. Tables 7–10 in Appendix S1 show the same for the quality indicators for the hip replacement surgeries.

To measure hospital quality, we compute quality indices for standard logistic regression models as:

$$\text{inde}{\mathrm{x}}_h=\frac{O_h}{E_h},$$ (1)

whereas $O_h$ is the observed event rate (e.g., complications) in hospital $h$ and $E_h$ is the risk‐adjusted expected rate in the same hospital. The expected rate amounts to the average predicted probability over all cases treated in hospital $h$. $E_h$ can be interpreted as the share of adverse events, which would have occurred if the same patients from hospital $h$ would have been treated in the average hospital instead.

As common in hierarchical models, we compute the quality indices for models with hospital random effects as ⁴⁰ :

$$\text{inde}{\mathrm{x}}_h=\frac{P_h}{E_h},$$ (2)

whereas $P_h$ is the average predicted adverse event rate for patients treated in hospital $h$ (including the hospital random effect ${\alpha }_h$) and $E_h$ is the average predicted event rate if the same patients were treated in the average hospital. The average hospital has a random effect of zero (i.e., ${\alpha }_h=0$) so that $E_h$ is simply estimated by averaging the predicted event rate in hospital $h$ without the hospital random effect. ⁴⁰ , ⁴¹ , ⁴²

3.4. Regression‐based risk adjustment

The section on risk selection above illustrated that the risk profiles of patients differ considerably between hospitals. The main issue with risk selection is that the direct comparison of hospitals not only captures differences in quality but also imbalances in patient risks. Therefore, risk adjustment is necessary. Since individual patient outcomes (e.g., the occurrence of a complication) are binary events, we estimate logistic regression models that account for risk factors that can explain individual patients' risk of experiencing these adverse events (see Table 11 in Appendix S1).

We compare two implementations of logit models: standard logits with patient‐level covariates and Mundlak‐corrected hospital random effects logits with patient‐level covariates and hospital means of covariates. Hospitals may differ in many risk factors from one another—some of which are observable to the researcher such as age, comorbidities, and gender, whereas others are not (e.g., health behaviors). If the unobservable risk factors are randomly distributed across hospitals, hospitals can be compared based on risk‐adjusted patient outcomes and the corresponding variation across hospitals can be interpreted as quality differences. ⁴³ , ⁴⁴ The validity of standard logit models to quality measurement hinges precisely on this assumption. If, however, the unobservable risk factors are systematically concentrated in certain hospitals, then standard logits yield biased estimates of the (relative) hospital quality. ⁴⁵ , ⁴⁶ , ⁴⁷

A possible solution to relax the assumption of uncorrelatedness of hospital random effects with patient‐level covariates is Mundlak‐corrected hospital random effects models that include hospital means of covariates as additional explanatory variables. ⁴⁴ , ⁴⁸ In Mundlak‐corrected models, coefficients of patient‐level covariates capture individual patients' risks, and coefficients of hospital means capture the risk profile of a hospital's entire patient population. The inclusion of the hospital means can correct for omitted variable bias due to unobserved heterogeneity when unobserved hospital‐level risk factors that are captured in the random effects are correlated with patient‐level covariates. Mundlak‐corrected models are consistent in the presence of unobserved heterogeneity under a standard strict exogeneity assumption of explanatory variables (including the hospital means) conditional on unobserved effects. ⁴⁹ Thus, Mundlak‐corrected logit specifications are preferable over the standard logit when observations are grouped at the hospital level, and hospitals vary in unobservable quality.

Besides the statistical properties, researchers must decide which covariates to include in their risk adjustment model. We follow two approaches to specify the set of risk adjusters: First, we estimate a series of hand‐picked specifications for both the standard logits and Mundlak‐corrected logits. Second, we employ LASSO, a supervised machine learning technique designed for “automatic” variable selection. ⁵⁰ , ⁵¹ In essence, the LASSO holds the promise to detect non‐linearities and complex interactions, which could have easily been overlooked in hand‐picked models to explain adverse patient events. Hence, in comparison to hand‐picked specifications, the LASSO automatically selects the best predictors and thus may soften misspecification issues biasing the hospital comparison.

The set of hand‐picked specifications includes 26 models of age, gender, type of insurance plan, nationality, place before admission, the number of Elixhauser or Charlson comorbidities as well as separate indicators for the underlying comorbidities. The Elixhauser and Charlson indices and comorbidities are not used together in any of the model specifications. Table 12 in Appendix S1 summarizes all model specifications in more detail. To tune the LASSO based on 10‐fold cross‐validation, we use the complete pool of candidate variables (in levels), polynomials of continuous variables, discretized categorical variables, and restricted cubic splines of continuous variables. We include all terms in the LASSO specification that received non‐zero coefficients in the final LASSO model from cross‐validation. Once the relevant predictors are selected, we estimate standard logit specifications based on the corresponding selection of variables (“LASSO‐selected logits”). Furthermore, we estimate the combination of the LASSO selection and Mundlak correction to leverage the benefits of both (“LASSO‐selected Mundlak‐corrected logits”).

To discriminate between the different specifications, we select the most suitable model based on the Akaike information criterion (AIC), ⁵² the out of sample performance, and the modified Hosmer–Lemeshow test. ⁵³ All models and the AIC and Hosmer–Lemeshow test are estimated based on the full data spanning over the years 2017–2019. The AIC is a measure of the relative goodness of fit of different models that considers the number of parameters and favors simpler over more complex models with the same fit thus partly safeguarding against overfitting the data. Also model selection based on the AIC has been shown to largely align with models selected based on leave‐one‐out cross validation and maximizing the out‐of‐sample expected log‐likelihood in cross‐validation. ⁵⁴

To assess the out‐of‐sample prediction accuracy of all “hand‐picked” and LASSO specifications, we compute the mean absolute error (MAE) using 10‐fold cross‐validation. The MAE is computed by averaging over all the differences in the average predicted probability per hospital and the actual adverse event rate in the same hospital.

The modified Hosmer–Lemeshow test checks for significant departures of mean residuals in different groups or percentiles of outcomes and explanatory variables. We apply the modified Hosmer–Lemeshow test over intervals of predicted event probabilities as well as age groups and the number of comorbidities to assess how well different models identify patients who vary in event risk. To ensure a “fair” comparison between the standard logits and hospital random effects models, the residuals were based on predictions without hospital random effects.

4. RESULTS

4.1. What is the best risk adjustment model?

The comparison of the different model specifications yields three main findings: First, Mundlak‐corrected hospital random effects logits outperform standard logistic regression models. Besides the more desirable statistical properties, the Mundlak‐corrected logits provide a better in‐sample fit to the data (see Figures 4–9 in Appendix S1). Also, they have a higher out‐of‐sample prediction accuracy. Figures 10–15 in Appendix S1 show that the standard logits are generally outperformed in terms of out‐of‐sample MAE. The modified Hosmer–Lemeshow plots based on predictions without random effects did not show significant differences between the standard logits and the Mundlak‐corrected logits, which confirms that the random effects and the Mundlak terms did not come at the costs of less accurate fixed effects estimates at the individual level.

Second, LASSO‐selected logits tend to have higher overall and group‐specific prediction accuracy than the hand‐picked standard logits. Likewise, the LASSO‐selected Mundlak‐corrected logits outperform the hand‐picked Mundlak‐corrected logits both in terms of overall as well as group‐specific fit to the data.

Third, the LASSO‐selected Mundlak‐corrected logits have the lowest or second lowest AIC of all candidate specifications for both indications (Figures 4–9 in Appendix S1). Furthermore, hand‐picked Mundlak‐corrected logits that adjust for patient demographics (age, gender) and Elixhauser comorbidities (i.e., specifications 21–26) provide an overall prediction accuracy at par with the LASSO‐selected Mundlak‐corrected logits. This might be an especially valuable finding for public health authorities that need to justify the hospital ranking to the public and wish to use interpretable models. The main drawback of the LASSOs is that they typically produce complex specifications as, for example, higher order polynomials are selected into the model. Moreover, the selection of risk factors can change from year to year potentially leading to confusion. In contrast, the hand‐picked Mundlak‐corrected random effects logits are easy to implement, understand, and communicate.

Besides the comparison of the different models, our analysis is informative regarding the risk factors that should be accounted for when comparing hospitals. First, we find that models including Elixhauser comorbidities show lower AIC values indicating better fit than identical models that adjust for Charlson comorbidities. The inclusion of the number of Elixhauser comorbidities and their non‐linear transformations decreases AIC even when being used together with binary indicators for Elixhauser comorbidities. Second, the inclusion of socio‐demographic variables (insurance plan, nationality, and place before admission) does not improve model fit over identical specifications without these variables. Hence, they are not included in our preferred risk adjustment model specification. However, socioeconomic differences can be correlated to unobservable risk factors (e.g., health behaviors) that vary between hospitals so that they should nevertheless be included in the risk adjustment model even when not improving the fit to the data. This might be especially relevant in health care systems with restricted access to care for specific patient groups such as low‐income families and ethnic minorities.

In summary, LASSO‐selected (Mundlak‐corrected) logits and hand‐picked Mundlak‐corrected hospital random effects models outperform standard logistic regression models as they tend to fail to capture relevant differences in the risk structure between hospitals.

4.2. LASSO‐selected versus Mundlak‐corrected logit?

In the last section, we have argued that the gains in prediction accuracy of LASSO‐selected specifications come at the cost of higher complexity. Practitioners might therefore wonder whether it is worthwhile to pay that “price.” To answer this question, we compare the hand‐picked Mundlak‐corrected logits with the LASSO‐selected (Mundlak‐corrected) logits and discuss how the model choice affects the hospital comparison. We find that practitioners tend to pay a small price when opting for the simpler, yet more transparent alternative—they paint an almost identical picture about the question as to which are the “good” and “bad” hospitals in the market.

Figure 2 plots the adjusted rate of surgical complications within 90 days (i.e., the quality index) of the Mundlak‐corrected logit (x‐axis) against the LASSO‐selected Mundlak‐corrected logit (y‐axis) for hospitals performing hip replacement surgery in 2018. The graph shows that the two risk adjustment models widely agree on the overall rating of hospitals as they closely fluctuate around the 45‐degree line (intraclass correlation coefficient [ICC]: 0.896). A similar pattern can be observed when comparing LASSO‐selected Mundlak‐corrected logits with Mundlak‐corrected logits for all the other quality indicators for hip replacement surgery. Hence, the choice between the LASSO‐selected models and hand‐picked Mundlak‐corrected logits does not systematically influence the hospital rating in case of Switzerland.

FIGURE 2.

The quality index for the Mundlak‐corrected hospital random effects logit (x‐axis) versus the LASSO‐selected Mundlak‐corrected logit (y‐axis) for surgical complications within 90 days after hip replacement surgery in 2018. The quality indices are computed as: $\text{inde}{\mathrm{x}}_h=P_h/E_h$, whereas $P_h$ is the average predicted adverse event rate for patients treated in hospital $h$ and $E_h$ is the average predicted event rate if the same patients were treated in the average hospital. ICC, intraclass correlation coefficient.

4.3. Does quality vary between hospitals?

In the last step of the analysis, we approach the question as to whether hospitals differ in the quality they deliver to patients. We analyze the variation in hospital quality based on hand‐picked Mundlak‐corrected hospital random effects logit models that account for age, gender, and comorbidities (i.e., specifications 22 for hip replacement and 23 for hernia repair).

Figure 3 illustrates the variation in hospital quality for the case of general complications during initial stay for primary hip replacement surgeries. Each point in the graph shows the risk‐adjusted quality index of a single hospital, and the size of the points represents the annual case numbers. The solid vertical line corresponds to the index of the average hospital. Hospitals to the left of this line provide above‐average quality (index <1), whereas hospitals with an index above one provide below‐average quality. Hospitals vary considerably in the quality they provide to patients. While we observe hospitals with an index of as little as 0.36, the hospital with the highest index has a value of 3.30, which is equivalent to a 230% higher complication rate than expected in the average hospital.

FIGURE 3.

The risk‐adjusted quality index for general complications during initial stay for primary hip replacement in 2018, using Mundlak‐corrected hospital random effect logit (specification 22). The quality index is computed as: $\text{inde}{\mathrm{x}}_h=P_h/E_h$, whereas $P_h$ is the average predicted adverse event rate for patients treated in hospital $h$ and $E_h$ is the average predicted event rate if the same patients were treated in the average hospital. Each point represents a hospital that treated ≥50 cases (as higher case numbers are associated with learning and better patient outcomes) in 2018. The vertical solid line shows the index for “the average hospital.” Hospitals to the left of this line provide above‐average quality (index <1) and hospitals to the right provide below‐average quality (index >1). The red dashed lines show the 25%‐ and 75%‐quartile of the index.

To make the quality measures more accessible to patients, we translate the index to a simple 3‐star rating. Hospitals with an index value in the lowest 25% of values are classified as 3‐star hospitals. Conversely, hospitals in the top quartile receive a 1‐star rating. Hospitals in between form the 2‐star category (i.e., 50% of hospitals). Although the choice of the cutoff values is arbitrary, we recommend a mapping to a more intuitive scale because patients tend to have difficulties understanding ratios (i.e., the quality index) and thus might altogether ignore quality measures in their hospital choice. The dashed lines in Figure 3 illustrate the 25%‐ and 75%‐quartile and the star rating, respectively. Three‐star hospitals on average have a 47% lower complication rate than the average hospital (average index: 0.53), and the average 1‐star hospital shows a 68% higher complication rate compared to the average hospital (average index: 1.68). Thus, patients treated in 1‐star hospitals face systematically more complications and other adverse events than those treated in 3‐star clinics.

The variation in quality, however, is not confined to this single indicator. We observe pronounced quality differences across all quality indicators and both indications (see Figures 16–20 and Table 13 in Appendix S1). Moreover, we observe a positive volume–outcome relationship for hip replacement surgeries, that is, a higher hospital caseload tends to be associated with being a 3‐star hospital. For hernia repair surgery, on the other hand, we do not observe any benefits from higher volumes.

5. DISCUSSION AND CONCLUSION

Every year, thousands of patients undergo elective surgery. Low quality of care can seriously harm patients and result in unnecessary costs for the health care system. Quality competition between hospitals can only work if patients have access to reliable and easy‐to‐understand quality information. In this paper, we proposed a systematic procedure to measure hospital quality based on routine data and assessed hospital quality for two elective procedures: inguinal hernia repair and primary hip replacement. Our analysis showed that Mundlak‐corrected and LASSO‐selected models were better at accounting for the risk selection between hospitals than standard logistic regression models commonly used in hospital comparisons. Applying the LASSO, a supervised machine learning method to “automatically” select risk adjusters resulted in models with higher explanatory power and identification of high‐risk cases than a wide variety of hand‐picked candidate specifications. While Mundlak‐corrected hospital random effects logits offer more favorable statistical properties than the standard logits, we also found that they generally explained a larger portion of the hospital‐level variation. Finally, our analysis provides evidence for considerable variation in the quality across hospitals for both indications in the Swiss inpatient sector. Hence, hospital choices can have severe clinical implications for patients.

This study comes with different limitations. First, there is no such thing as a perfect risk adjustment based on routine data. ⁷ , ⁵⁵ Although our approach addresses key shortcomings of standard logistic risk adjustment models, hospitals might still vary in unobservable patient risks, which we do not account for. Additional data on, for example, disease severity or patient mobility scores could further improve risk adjustment and de‐bias the hospital comparison.

Second, we selected models both on their in‐ and out‐of‐sample prediction accuracy. However, we had minor convergence issues when cross‐validating the LASSO‐selected Mundlak‐corrected model for one of the quality indicators in case of hernia repair. We thus suggest that future work more systematically focuses on the out‐of‐sample performance of different risk adjustment models.

Third, although hospital routine data have important properties for quality measurement—it is readily available, comprehensive, and results are available in a timely manner—it also comes with certain limitations. Many outcomes are delayed and may only occur after hospital discharge. Specifically, we do not observe adverse events treated in outpatient care (e.g., infections). Thus, the hospital comparison would also be biased if hospitals differ in the management of subsequent adverse events. Another potential drawback is that “coding practices” may differ between hospitals. ⁵⁶ Certain hospitals could code adverse events less stringently than others. Although coding regulations typically leave little space for “creative” coding, this issue deserves further study in future research.

Fourth, mapping quality indices to simple 3‐star ratings can be problematic if quality varies little between hospitals. In that case, the translation to a simpler scale is not informative, and researchers should altogether dismiss the corresponding indicators. Furthermore, hospitals “at the margin” might rightfully complain to be underrated despite providing almost identical quality like hospitals just above the cutoff. Indeed, the choice of the cutoff values (e.g., quantiles) is arbitrary. To alleviate this limitation, the star rating should ideally not entirely be based on point estimates but also incorporate the uncertainty around the estimated hospital quality. For example, stars could only be given to hospitals that significantly differ in their quality from other hospitals across quality segments (e.g., hospitals in the top 25% vs. middle 50% of the index distribution). Note, however, that methods to consistently estimate confidence intervals for the predicted‐to‐expected ratios used in this paper have not been developed yet, and bootstrapping procedures tend to have poor coverage rates in simulation studies. ⁵⁷

Our results have relevance for several stakeholders in the health care system. Public health authorities might be particularly interested in the finding that relatively simple hand‐picked Mundlak‐corrected specifications showed similar prediction accuracy like the more complex LASSO‐selected models. Moreover, since both model types provided an almost identical answer to the question as to which are the low‐ and high‐quality hospitals in the market, the hand‐picked Mundlak‐corrected models have even more practical appeal. For example, this could be relevant for the Agency for Healthcare Research and Quality (AHRQ), which currently applies a LASSO‐selected logit for risk adjustment. ⁵⁸ Complex models are harder to defend, and hospitals might rightfully claim that it is unclear how they are evaluated and thus dismiss the validity of the ranking altogether. Finally, since there is growing interest in composite measures of hospital performance as used by the CMS on its Hospital Compare website, our quality indices can be easily combined into a single measure. Different weighting strategies could be applied to combine the indicators to an overall rating: equal weighting, weights according to expert opinion, or empirical weights. ⁷ , ¹⁰ , ⁵⁹ , ⁶⁰

Patients benefit from access to quality information because it allows them to undergo surgery in high‐quality hospitals with fewer harmful or potentially life‐threatening events. This should both improve patient health and enhance quality competition in the health care sector. In addition, the communication of the quality indicators based on intuitive scales like the 3‐star rating should increase the probability that hospital choices are guided by quality information instead of travel distance and the size of the parking lot. ⁴ , ¹⁰ , ¹³ , ⁶¹ , ⁶² Lastly, while existing quality metrics are often based on mortality, such “extreme” outcomes are of second order importance in high‐volume, low‐risk elective procedures. Patients and/or their referring physicians can thus base their future provider decisions on quality dimensions that are more relevant to them.

Payers benefit from reliable quality measures to allocate resources, and patient streams to high‐quality hospitals. This might be particularly relevant in health care systems where health authorities/states decide which hospitals can reimburse their services via mandatory health insurance. Moreover, quality information could prove valuable in price negotiations in supplementary insurance and value‐based purchasing programs that have gained popularity in many health care systems over the last decade. ⁶³ , ⁶⁴

Supporting information

Appendix S1. Supporting Information.

Bilger J, Pletscher M, Müller T. Separating the wheat from the chaff: How to measure hospital quality in routine data? Health Serv Res. 2024;59(2):e14282. doi: 10.1111/1475-6773.14282