Abstract
Several years ago, the US News and World Report changed their risk-adjustment methodology, now relying almost exclusively on chronic conditions for risk adjustment. The impacts of adding selected acute conditions like pneumonia, sepsis, and electrolyte disorders (“augmented”) to their current risk models (“base”) for 4 specialties—cardiology, neurology, oncology, and pulmonology—on estimates of hospital performance are reported here. In the augmented models, many acute conditions were associated with substantial risks of mortality. Compared to the base models, the discrimination and calibration of the augmented models for all specialties were improved. While estimated hospital performance was highly correlated between the 2 models, the inclusion of acute conditions in risk-adjustment models meaningfully improved the predictive ability of those models and had noticeable effects on hospital performance estimates. Measures or conditions that address disease severity should always be included when risk-adjusting hospitalization outcomes, especially if the goal is provider profiling.
Keywords: risk adjustment, provider profiling, acute conditions, Medicare claims
Background
In 2019, US News and World Report (USNWR) changed their risk-adjustment methodology for the assessment of hospital specialty performance on outcomes like 30-day mortality,1 replacing 3M’s All Patient Refined Diagnosis Related Group (APR-DRG) severity of illness and risk of mortality variables2 with indicators for the Elixhauser conditions.3 This change represented a fundamental shift in risk-adjustment philosophy. While the Elixhauser conditions are common and important conditions to account for as part of risk adjustment, they are almost exclusively chronic, nonacute conditions. Acute conditions such as pneumonia, sepsis, and electrolyte disorders, reflected in 3M’s APR-DRG severity of illness and risk of mortality scoring, are not part of the current Elixhauser conditions list, yet are important predictors of survival among hospitalized patients. Other performance reporting programs, such as the Centers for Medicare & Medicaid Services Care Compare4,5 and the American College of Surgeons National Surgical Quality Improvement Program,6 rely on acute conditions for risk adjustment.
In this study, the impact of adding selected acute conditions, when present-on admission, to USNWR risk models for estimating hospital performance was explored among patients in 4 high-volume specialties: cardiology, neurology, oncology, and pulmonology. This article reports the importance of these acute conditions on risk model performance and changes in estimated hospital performance with the addition of these conditions.
Methods
Data Source
For this analysis, the study team used Medicare data available from the Centers for Medicare and Medicaid Services (CMS), including inpatient fee-for-service claims data for information about hospital admissions with discharge dates between Jan 1, 2019 and Dec 31, 2021, and the Medicare Beneficiary Summary Files (MBSF) for information about patient demographics, enrollment, eligibility, and mortality.
Ethics approval
All aspects of this study were reviewed and approved by the Duke Health Institutional Review Board (Pro00106448) and included a waiver of documentation of consent and Health Insurance Portability and Accountability Act (HIPAA) authorization.
Study Populations
This study examined 4 specialty-based cohorts of hospital admissions—cardiology, neurology, oncology, and pulmonology—consistent with the latest USNWR methodology.7 For each specialty, eligible claims were identified using DRG codes, restricted, as specified, by diagnosis and procedure codes. Included patients were age 65 or older, enrolled in fee-for-service Medicare at admission, and were not transferred from another acute-care hospital. Like USNWR, claims before the inpatient stay were not used to characterize patient comorbid conditions, so there was no requirement for a specified period of prior fee-for-service enrollment.
Outcome
The outcome of interest was mortality within 30 days of hospital admission, based on mortality dates recorded in the Medicare MBSF.
Risk-adjustment covariates
Consistent with USNWR methodology, all models included age at admission, sex, end-stage renal disease status, and Medicaid dual-eligibility, as recorded in the MBSF, along with year of admission and an indicator for whether the patient was admitted from a skilled nursing facility. Indicators for each specialty-specific DRG group, as listed in the inclusion/exclusion criteria, were also included to account for the differential mortality associated with the reason for admission.
Two sets of covariates represented clinical conditions recorded on the inpatient claims. First was the set of 38 chronic conditions defined by ARHQ’s Elixhauser Comorbidity Software Refined for International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM).8 These conditions are listed in Supplemental Digital Content 1, available at http://links.lww.com/AJMQ/A112, and were identified using the diagnosis codes available on each inpatient claim. Second were 15 acute conditions not captured by the Elixhauser software: acute coronary syndrome (ACS); acute renal disease; cardio-respiratory failure and shock; coronary atherosclerosis or angina; disorders of fluid/electrolyte/acid-base balance; heart arrhythmias, specified; hemiplegia, paraplegia, paralysis, functional disability; infection; other respiratory disorders; other significant endocrine and metabolic disorders; pleural effusion/pneumothorax; pneumonia; respirator dependence/respiratory failure; septicemia, sepsis; and severe hematological disorders. These conditions are all included in one or more of the CMS Hospital Compare risk-adjustment models for inpatient mortality or readmission outcomes9,10 and are thus likely to be important for predicting mortality among the chosen specialty hospitalizations. The definitions of these conditions (Supplemental Digital Content 2, available at http://links.lww.com/AJMQ/A112) are based on diagnosis codes that are part of specific predefined categories within the hierarchical condition category coding system developed for CMS.11 These conditions were also identified using the diagnosis codes available on each inpatient claim. Because many of these conditions can develop as complications during an inpatient stay, each of these conditions was required to be present on admission to be used during risk adjustment. To supplement these acute conditions and as a general indicator of admission urgency, an indicator for whether the patient was admitted directly from the emergency department (ED) of the admitting hospital was also included. This was identified by searching for ED-related revenue center codes (0450–0459, 0981) on the inpatient claim.
Statistical analysis
The prevalence of all chronic and acute conditions in each specialty cohort is reported using frequencies and percentages. Within-hospital prevalence the distributions of these conditions are shown as medians and quartiles.
Two risk-adjustment models were estimated within each specialty cohort. Both models incorporated age, sex, end-stage renal disease status, Medicaid dual-eligibility status, year of admission, an indicator for whether or not the patient was admitted from a skilled nursing facility, and reason for admission (DRG groups). Model 1 also included the Elixhauser (chronic) conditions, while Model 2 included the Elixhauser conditions, the additional acute conditions, and the admission-from-ED indicator. Model 1 will be referred to as the “base model” and Model 2 as the “augmented model.” Both models were specified as hierarchical logistic models, with hospital-level random intercepts. These models accounted for intra-hospital correlation among patient outcomes and provided, via the estimated random intercepts, hospital-specific risk-standardized performance estimates.
For each model, odds ratios (ORs) and 95% confidence intervals (CIs) for the estimates associated with the acute conditions are reported, and hospital performance is reported using risk-standardized mortality ratios (RSMR), which are calculated as the exponentiated random effect estimated by each model for each hospital.
As metrics of model performance, the area under the receiver operating curve (AUC) and the Brier score are reported, both of which were calculated using predicted values that did not include the random intercept term. AUC is a measure of discrimination indicating, at the patient level, how well the model is distinguishing those who died from those who did not die within 30 days. Higher AUC values indicate better discrimination. The Brier score12 is a measure of calibration, reflecting the mean square distance between the observed and model-predicted outcomes. Lower Brier scores indicate better calibration. To assess the incremental contribution of acute conditions to the risk-adjustment model, both the difference in AUC and the percent improvement in the Brier score (also known as the Brier skill score13) between the augmented model and the base model are reported. For context, the difference in AUC and the percent improvement in the Brier score between the base model and a model without comorbid conditions are also reported, which reflect the contribution of the Elixhauser conditions to the risk-adjustment model.
While data from all hospitals were included in the estimation of the models, specialty-specific performance was only examined for hospitals with a sufficient number of admissions in that specialty. Supplemental Digital Content 3, available at http://links.lww.com/AJMQ/A112, shows the minimum number of total admissions and, if applicable, the total number of surgical admissions required. The goal was to arrive at a similar number of hospitals as were included in the latest USNWR rankings for each specialty.
Among this smaller group of hospitals, the correlation of RSMRs from the base and augmented models for each specialty is reported. For each hospital, specialty-specific scatterplots are presented to show how RSMRs changed from the base model to the augmented model. Because hospital performance is always relative, 3 reference lines are incorporated on these plots. The first line indicates where relative performance did not change between models. Hospitals on this line fall at the same location in the distribution of random effects (RE) from each model, as determined by Z-scores. The other lines indicate where relative performance was 1 standard deviation (SD) better or worse in the augmented model, compared to the base model. For example, a hospital with an average RE in the base model and an RE that is 1 SD better than average in the augmented model would fall on the −1 SD line, in the improved performance half of the figure.
To explore whether including acute conditions in the risk model improved the performance estimates of sites in particular parts of the country, the specialty-specific average standardized performance scores from each model are presented by census division.
All analyses were performed using SAS v9.4 (SAS Institute, Cary, NC).
Results
The study team identified 3,021,718 cardiology admissions, 1,304,397 neurology admissions, 446,845 oncology admissions, and 3,736,804 pulmonology admissions within the study period. Prevalence of the Elixhauser conditions (Supplemental Digital Content 1, available at http://links.lww.com/AJMQ/A112) across all 4 specialties, ranged from rare—less than 2% of admissions with acquired immune deficiency syndrome, peptic ulcer disease, moderate/severe liver disease, and chronic blood loss anemia, among others—to more common—over 15% of admissions with chronic pulmonary disease, diabetes (with or without chronic complications), and hypertension (uncomplicated or complicated).
Prevalence of the specified acute conditions that were present on admission (Table 1) varied by specialty but also ranged from rare—less than 2% of admissions with respiratory dependence/respiratory failure and severe hematological disorders—to more common—over 20% of admissions with coronary atherosclerosis and disorder of fluid/electrolyte/acid-base balance. Certain conditions were more common in specific populations if the conditions were related to the specialty—eg, ACS, coronary atherosclerosis or angina, and heart arrhythmias were more common among cardiology admissions than admissions for other specialties.
Table 1.
Acute Condition Prevalence by Specialty.
| Condition | Cardiology (N = 3,021,718) |
Neurology (N = 1,304,397) |
Oncology (N = 446,845) |
Pulmonology (N = 3,736,804) |
|---|---|---|---|---|
| Acute coronary syndrome | 693,740 (23.0) | 10,174 (2.3) | 45,515 (3.5) | 329,376 (8.8) |
| Acute renal disease | 612,932 (20.3) | 75,373 (16.9) | 165,851 (12.7) | 1,062,024 (28.4) |
| Cardio-respiratory failure and shock | 671,328 (22.2) | 43,022 (9.6) | 90,036 (6.9) | 1,748,571 (46.8) |
| Coronary atherosclerosis or angina | 1,781,492 (59.0) | 90,934 (20.4) | 373,058 (28.6) | 1,105,213 (29.6) |
| Disorders of fluid/electrolyte/acid-base balance | 797,512 (26.4) | 143,322 (32.1) | 330,683 (25.4) | 1,728,233 (46.2) |
| Heart arrhythmias, specified | 1,431,263 (47.4) | 74,806 (16.7) | 312,646 (24.0) | 1,013,497 (27.1) |
| Hemiplegia, paraplegia, paralysis, functional disability | 22,992 (0.8) | 13,176 (2.9) | 280,266 (21.5) | 92,357 (2.5) |
| Infection | 160,161 (5.3) | 30,947 (6.9) | 94,484 (7.2) | 575,540 (15.4) |
| Other respiratory disorders | 613,361 (20.3) | 57,756 (12.9) | 144,383 (11.1) | 830,216 (22.2) |
| Other significant endocrine and metabolic disorders | 123,165 (4.1) | 22,694 (5.1) | 51,189 (3.9) | 160,350 (4.3) |
| Pleural effusion/pneumothorax | 132,829 (4.4) | 62,718 (14.0) | 16,454 (1.3) | 307,230 (8.2) |
| Pneumonia | 221,679 (7.3) | 31,469 (7.0) | 54,156 (4.2) | 1,494,064 (40.0) |
| Respirator dependence/respiratory failure | 334 (<0.1) | 132 (<0.1) | 187 (<0.1) | 3,848 (0.1) |
| Septicemia, sepsis | 46,134 (1.5) | 11,232 (2.5) | 19,573 (1.5) | 1,592,897 (42.6) |
| Severe hematological disorders | 19,099 (0.6) | 7,979 (1.8) | 5,477 (0.4) | 30,056 (0.8) |
| Admitted from the emergency department | 2,228,656 (73.8) | 266,089 (59.5) | 1,077,774 (82.6) | 3,339,327 (89.4) |
Reported as N (%).
The distribution of acute conditions differed widely within hospitals. Table 2 reports the distribution of acute condition prevalence within hospitals having sufficient volume for reporting. Across all specialties, it was not uncommon for the prevalence of an acute condition to be 40% higher among hospitals in the highest quartile compared to hospitals in the lowest quartile. As an example, the prevalence of ACS at hospitals in the highest quartile for prevalence was twice as high as at hospitals in the lowest prevalence quartile within the neurology (5.1% vs 2.4%), oncology (4.7% vs 1.5%), and pulmonology (11.9% vs 5.9%) cohorts. Even within the cardiology cohort, hospitals in the highest quartile had a prevalence that was 54% higher, relatively, compared to hospitals in the lowest prevalence quartile (26.5% vs 17.2%).
Table 2.
Distribution of Acute Condition Prevalence Among Hospitals with Sufficient Volume, by Specialty.
| Condition | Cardiology (# Sites = 780) |
Neurology (# Sites = 1243) |
Oncology (# Sites = 900) |
Pulmonology (# Sites = 1697) |
|---|---|---|---|---|
| Acute coronary syndrome | 21.6 (17.2–26.5) | 3.6 (2.4–5.1) | 3.1 (1.5–4.7) | 8.5 (5.9–11.9) |
| Acute renal disease | 18.6 (15.6–22.4) | 12.2 (9.7–15.4) | 16.8 (13.8–20.3) | 29.0 (25.2–33.4) |
| Cardio-respiratory failure and shock | 20.5 (16.3–24.9) | 6.8 (5.2–8.7) | 10.4 (7.6–13.2) | 49.0 (43.0–54.9) |
| Coronary atherosclerosis or angina | 61.9 (58.0–65.7) | 28.9 (25.2–33.2) | 20.9 (17.1–24.5) | 30.0 (25.9–33.7) |
| Disorders of fluid/electrolyte/acid-base balance | 24.2 (20.5–28.4) | 24.7 (21.0–29.2) | 32.7 (27.3–38.3) | 47.2 (42.2–52.3) |
| Heart arrhythmias–specified | 47.3 (44.4–50.2) | 23.9 (21.8–26.3) | 16.9 (14.4–19.7) | 27.8 (25.5–30.2) |
| Hemiplegia, paraplegia, paralysis, functional disability | 0.8 (0.6–1.0) | 21.1 (17.6–24.9) | 3.4 (2.6–4.3) | 2.1 (1.6–3.0) |
| Infection | 4.7 (3.9–5.6) | 6.9 (5.6–8.3) | 6.9 (5.7–8.2) | 15.3 (13.2–17.5) |
| Other respiratory disorders | 20.9 (17.2–24.4) | 10.9 (8.3–13.8) | 13.0 (10.7–15.5) | 22.5 (18.9–26.6) |
| Other significant endocrine and metabolic disorders | 3.7 (2.6–5.1) | 4.1 (3.1–5.1) | 5.8 (4.6–6.8) | 4.2 (3.2–5.5) |
| Pleural effusion/pneumothorax | 3.9 (3.0–5.3) | 1.5 (1.2–1.9) | 14.4 (11.4–17.3) | 8.2 (6.8–9.9) |
| Pneumonia | 5.9 (4.1–7.9) | 4.0 (3.1–5.1) | 7.1 (5.0–9.2) | 40.0 (36.0–44.1) |
| Respirator dependence/respiratory failure | <0.1 (<0.1–<0.1) | <0.1 (<0.1–<0.1) | <0.1 (<0.1–<0.1) | <0.1 (<0.1–<0.1) |
| Septicemia, sepsis | 1.3 (1.0–1.8) | 1.9 (1.3–2.6) | 3.4 (2.0–4.7) | 43.5 (36.1–51.2) |
| Severe hematological disorders | 0.7 (0.6–0.9) | <0.1 (<0.1–0.6) | <0.1 (<0.1–3.0) | 1.0 (0.8–1.2) |
| Admitted from the emergency department | 71.4 (60.8–79.5) | 85.7 (79.1–90.8) | 66.0 (52.9–76.5) | 95.7 (92.1–97.7) |
Reported as median (Q1, Q3) site prevalence.
Estimated OR associated with the acute conditions from the augmented models are shown in Table 3. While these ORs varied by specialty, many of these conditions were consistently associated with substantial risks of mortality. For example, ORs across all specialties were 1.34 or higher for ACS, 1.86 or higher for cardio-respiratory failure and shock, and 1.26 or higher for disorders of fluid/electrolyte/acid-base balance. A small number of conditions were associated with consistently reduced risk across specialties—eg, other respiratory disorders. The OR for admission-from-ED varied by specialty, reflecting additional risk among oncology admissions (OR, 1.14; 95% CI, 1.11–1.17), but less risk among pulmonology admissions (OR, 0.85; 95% CI, 0.84–0.86).
Table 3.
Odds Ratios and 95% Confidence Intervals Associated with Acute Conditions in Augmented Model.
| Condition | Cardiology | Neurology | Oncology | Pulmonology |
|---|---|---|---|---|
| Acute coronary syndrome | 1.34 (1.32–1.36) | 1.36 (1.33–1.40) | 1.37 (1.31–1.43) | 1.34 (1.33–1.35) |
| Acute renal disease | 1.66 (1.65–1.68) | 1.16 (1.14–1.18) | 1.54 (1.51–1.58) | 1.59 (1.58–1.61) |
| Cardio-respiratory failure and shock | 2.24 (2.21–2.26) | 4.79 (4.71–4.88) | 1.86 (1.81–1.91) | 2.30 (2.29–2.32) |
| Coronary atherosclerosis or angina | 1.01 (1.00–1.02) | 0.99 (0.97–1.00) | 0.97 (0.95–0.99) | 0.98 (0.98–0.99) |
| Disorders of fluid/electrolyte/acid-base balance | 1.53 (1.51–1.54) | 1.26 (1.24–1.27) | 1.62 (1.59–1.65) | 1.37 (1.36–1.38) |
| Heart arrhythmias, specified | 1.13 (1.12–1.14) | 1.23 (1.21–1.25) | 1.16 (1.13–1.18) | 1.23 (1.23–1.24) |
| Hemiplegia, paraplegia, paralysis, functional disability | 1.43 (1.37–1.50) | 1.22 (1.18–1.25) | 1.14 (1.07–1.22) | 1.40 (1.37–1.44) |
| Infection | 0.96 (0.94–0.98) | 0.90 (0.88–0.92) | 1.04 (1.01–1.07) | 0.73 (0.73–0.74) |
| Other respiratory disorders | 0.81 (0.80–0.82) | 0.84 (0.82–0.86) | 0.90 (0.88–0.92) | 0.78 (0.77–0.78) |
| Other significant endocrine and metabolic disorders | 1.02 (1.00–1.04) | 0.99 (0.97–1.02) | 1.06 (1.03–1.10) | 0.99 (0.97–1.00) |
| Pleural effusion/pneumothorax | 1.32 (1.29–1.34) | 1.59 (1.53–1.65) | 1.42 (1.38–1.45) | 1.50 (1.48–1.51) |
| Pneumonia | 1.34 (1.32–1.36) | 1.63 (1.59–1.67) | 1.38 (1.34–1.42) | 1.34 (1.33–1.34) |
| Respirator dependence/respiratory failure | 3.77 (2.89–4.91) | 7.01 (5.05–9.73) | 1.90 (1.29–2.80) | 0.89 (0.82–0.96) |
| Septicemia, sepsis | 1.89 (1.85–1.94) | 1.84 (1.78–1.91) | 1.70 (1.62–1.77) | 1.22 (1.19–1.24) |
| Severe hematological disorders | 0.94 (0.90–0.99) | 1.11 (1.03, 1.20) | 1.14 (1.07–1.21) | 0.96 (0.93–0.99) |
| Admitted from the emergency department | 0.99 (0.97–1.00) | 1.01 (0.99–1.04) | 1.14 (1.11–1.17) | 0.85 (0.84–0.86) |
Reported as odds ratio (95% confidence interval).
AUC values for the base models, augmented models, and models without any comorbid conditions across specialties are shown in Table 4. Compared to the base models, the AUCs of the augmented models for cardiology, neurology, and pulmonology were 0.03 points higher, which represents a meaningful improvement in the discrimination of the risk model. The augmented model for oncology added slightly less discriminatory power (0.021) to the base model than these other specialties. For context, the AUCs of the base models across all specialties were between 0.05 and 0.06 points higher than models without any comorbid conditions.
Table 4.
Calibration and Discrimination Metrics of Regression Models, by Specialty.
| Metric/model specification | Specialty model | |||
|---|---|---|---|---|
| Cardiology | Neurology | Oncology | Pulmonology | |
| Discrimination | ||||
| AUC | ||||
| No comorbid conditions model | 0.716 | 0.736 | 0.734 | 0.714 |
| Base model | 0.770 | 0.795 | 0.790 | 0.771 |
| Augmented model | 0.800 | 0.825 | 0.811 | 0.801 |
| Δ AUC | ||||
| Base model—No comorbid conditions model (reference) | +0.054 | +0.059 | +0.057 | +0.057 |
| Augmented model—Base model (reference) | +0.030 | +0.030 | +0.021 | +0.030 |
| Calibration | ||||
| Brier score | ||||
| No comorbid conditions model | 0.073 | 0.156 | 0.099 | 0.139 |
| Base model | 0.070 | 0.145 | 0.093 | 0.130 |
| Augmented model | 0.068 | 0.140 | 0.088 | 0.124 |
| Brier skill score (% improvement in Brier score) | ||||
| Base model vs no comorbid conditions model (reference) | +3.6% | +6.8% | +5.9% | +6.1% |
| Augmented model vs base model (reference) | +3.1% | +3.9% | +5.7% | +4.7% |
Higher AUC values indicate better model discrimination. The Δ AUC value is the difference in AUC values between the specified model and the reference model. All Δ AUC values reflect significantly significant differences (P < 0.001). Lower Brier scores indicate better model calibration. The Brier skill score reflects the % improvement in the Brier score of the specified model () compared to a reference model () and is calculated as .
Abbreviation: AUC, Area under the receiver operating curve.
Brier scores for the base models, augmented models, and models without any comorbid conditions across specialties are shown in Table 4. Compared to the base models, the augmented models demonstrated between 3.1% and 5.7% better calibration. For context, the base models demonstrated between 3.6% and 6.8% better calibration than models without any comorbid conditions. For cardiology and oncology, the improvement in calibration due to the addition of acute conditions was very similar in magnitude to the improvement due to the addition of the Elixhauser conditions.
Figure 1 shows scatterplots of hospital RSMRs from the base model compared to RSMRs from the augmented model for hospitals with sufficient volume in each specialty. The correlation between these RSMRs is shown, and the dotted lines indicate differences of ±1 SD in relative performance between models. Across all specialties, RSMRs from the augmented model were highly correlated with RSMRs from the base model (r ≥ 0.93), but there were numerous hospitals demonstrating large differences in performance estimates, both better and worse when accounting for acute conditions. Within cardiology, neurology, and pulmonology, there were half a dozen or more hospitals with a relative improvement of 1 SD within the performance distribution. This movement is equivalent to a hospital being ranked in the bottom half of hospitals by the base model and the top 20% of hospitals by the augmented model.
Figure 1.
Scatterplot of hospital standardized mortality ratios from the base model compared to the augmented model by specialty. Each point represents a hospital, comparing that hospital’s performance estimate from the base model against its performance estimate from the augmented model. Only hospitals with sufficient volume in each specialty are included in the graphs. The number of included hospitals by specialty were: 708 for cardiology; 1243 for neurology; 900 for oncology; and 1697 for pulmonology. The solid lines indicate unchanged relative hospital performance between models. The dotted lines indicate a change in relative hospital performance by 1 standard deviation in either direction. A point below the line indicates a hospital with better relative performance estimated by the augmented model compared to the base model. A point above the line indicates a hospital with worse relative performance estimated by the augmented model compared to the base model.
Supplemental Digital Content 1, available at http://links.lww.com/AJMQ/A112 shows the performance distribution of hospitals, by specialty and by the location of the hospital. Although there are some exceptions to this pattern, hospitals in the New England and Middle Atlantic census divisions tend to perform better, based on estimates from the base model and the augmented model, while hospitals in the East South Central and West South Central census divisions tend to perform worse. In general, the patterns of performance by region do not change markedly with the inclusion of acute conditions.
Discussion
This report describes the effects of adding acute conditions to the USNWR mortality risk model, which currently relies on the Elixhauser chronic conditions to capture a patient’s health status at admission. Among 4 different specialties—cardiology, neurology, oncology, and pulmonology—the prevalence of these acute conditions differed substantially by hospitals. The inclusion of acute conditions meaningfully improved the predictive ability of models at the hospitalization level and had noticeable effects on hospital performance estimates, although the degree of differences varied by specialty.
To ensure the most accurate estimates of provider performance, characterizing each patient’s burden of illness as completely as possible is essential. Relying only on chronic conditions for risk adjustment, as done within the USNWR Best Hospitals Specialty Rankings program, is an incomplete risk-adjustment strategy. Elixhauser et al3 in their initial publication, understood this. They viewed a patient’s burden of illness to be made up of different components, with a clear distinction between the principal diagnosis (ie, the reason for hospitalization), the severity of that diagnosis, and other unrelated conditions present on admission. They were focused on the latter, and the recent decision by the Agency for Healthcare Research and Quality to exclude fluid and electrolyte disorders from the list of Elixhauser conditions makes it clear that this is still the intent of this list. They explained that this exclusion was made because “the diagnoses included in this comorbidity measure are typically considered acute and secondary to an underlying problem or comorbidity.”14
It is worth noting that the Elixhauser list was limited to conditions unrelated to the principal diagnosis because they explicitly presumed that some measure of disease severity would also be included within the risk-adjustment strategy. They referred to the disease staging framework15 as an example, which incorporates information about acute conditions such as cardiogenic shock, septicemia, and infection to differentiate admissions with the same principal diagnosis. Within the Agency for Healthcare Research and Quality’s Healthcare Cost and Utilization Project databases, for example, disease severity is available for each hospitalization record using both the APR-DRG Risk of Mortality and Severity of Illness measures. In clinical research, the use of disease staging, per se, is less common. It is much more likely that acute conditions are simply included within the risk model as covariates. The CMS Care Compare program is an excellent example of a quality program that includes a mix of both acute and chronic conditions in their risk-adjustment strategy.
Another reason that many acute conditions were not included in the original Elixhauser list was because of the concern that they were more likely to be complications of care, rather than preexisting comorbidities. Until relatively recently, administrative health data sources did not allow for this distinction, but now, present-on-admission information exists and is well-populated for each diagnosis on a claim or medical record. For this study, these present-on-admission flags ensured that the acute conditions included in the risk adjustment had not developed during the hospitalization. Even so, this may be a conservative approach since the late development or identification of many acute conditions is unlikely to be due to hospital care. If there is truly interest in identifying acute conditions that are likely complications of care, CMS’ list of hospital-acquired conditions16 may offer a much more targeted list of conditions for exclusion. Most of these conditions are explicitly tied to operative procedures that occurred within the hospitalization.
While the correlation between hospital performance estimates based on models with and without acute conditions was high, some hospitals saw their relative performance increase or decrease substantially. This was evident across all specialties. Hospitals that saw a performance estimate improvement of more than 1 SD when acute conditions were included in the mortality regression model are hospitals that have a higher prevalence of some or many of these acute conditions, compared to the average hospital. To ignore these conditions in risk adjustment may inappropriately disadvantage hospitals that routinely treat higher-risk patients.
Although there was no differential improvement in performance by hospitals in different regions of the country after including acute conditions in the risk model, the generally better performance among hospitals in northeastern states and worse performance among hospitals in southern states persisted. In other words, the concern that regional population-level differences in health status are not completely captured in the existing USNWR risk models17 is not due to the lack of acute conditions in those models. Other markers of population health status, including socioeconomic status, need to continue to be explored.
While the best risk-adjustment strategy is one that includes relevant conditions, acute or chronic, based on clinical knowledge of the specific study population and outcome of interest, it is not surprising that researchers often rely on predefined lists of conditions, especially when using electronic health data where coding algorithms are required. For that reason, it is possible that the USNWR models do not include acute conditions because there is no current, easily incorporated list of such conditions similar to the Elixhauser list for chronic conditions. The acute conditions considered in this study may be a start toward such a list. They were consistently related to mortality among the 4 specialties examined and are likely related to most clinical and utilization outcomes.
Others have raised different legitimate criticisms of the USNWR methodology,18,19 including the inconsistency of performance estimates for low-volume specialties, the reliance on inpatient data in the face of the continued transition of many procedures to the outpatient setting, and the attribution of mortality to a specialty whose providers were not involved in a patient’s care. This last issue arises from the reliance on DRGs for patient identification and grouping, which is problematic since DRG assignment is driven by the ordering of diagnosis codes on a claim to Medicare for payment purposes. In lieu of wholesale changes to the USNWR methodology, the current study points to an easily implemented improvement that addresses unmeasured, but knowable, confounding.
Limitations
This study has some limitations. Only hospitalizations from specific specialties were studied. Even though these cohorts are large and represent many different types of admissions, it is not known how well these findings generalize to other patient populations. Also, the USNWR risk model itself is limited in scope. It only uses information from the inpatient claims, without considering information from claims before hospitalization. While it is possible that some of the risk ascribed to acute conditions in this study would be captured by prior diagnoses, it is likely that these acute conditions on the index hospitalization will still be relevant for risk adjustment.
Additionally, our analysis presumes that any observed differences between hospitals in the prevalence of acute conditions, as coded in claims, truly reflect the patient populations served. While beyond the scope of this study, it is possible that acute condition prevalence was higher at referral hospitals compared to community hospitals for this reason. Within neurology, for example, certified Comprehensive Stroke Centers20 are the ones best able to treat the most complex stroke cases (ie, those with the highest disease severity). It is also possible, however, that some hospitals are just more adept at or attentive to condition coding than others. It is currently difficult or impossible to distinguish between these reasons.
Conclusion
The Elixhauser comorbidity measures are not a complete solution for risk adjustment of in-hospital mortality among hospitalized patients. The inclusion of acute conditions in risk-adjustment models meaningfully improved the predictive ability of those models and had noticeable effects on hospital performance estimates. Measures or conditions that address disease severity should always be included when risk-adjusting hospitalization outcomes, especially if the goal is providing profiling.
Conflicts of Interest
The authors have no conflicts of interest to disclose.
Funding
This study was funded through a collaboration between the Duke University Health System and the Duke Department of Population Health Sciences.
Author Contributions
Dr Hammill conceptualized and designed the study, analyzed and interpreted the data, and drafted and substantively revised the manuscript. Dr Hoffman analyzed and interpreted the data, and substantively revised the manuscript. Dr Clark acquired and interpreted the data and substantively revised the manuscript. Dr. Bae conceptualized and designed the study, interpreted the data, and substantively revised the manuscript. DR Shannon conceptualized the study and substantively revised the manuscript. Dr Curtis conceptualized and designed the study, interpreted the data, and substantively revised the manuscript. All authors have approved the submitted version and agree to be personally accountable for their contributions and ensure that questions related to the accuracy or integrity of any part of the work are appropriately investigated, resolved, and resolution documented.
Supplementary Material
Footnotes
Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s Web site (www.ajmqonline.com).
References
- 1.Olmsted MG, Powell R, Murphy J, et al. Methodology: U.S. News & World Report 2019-20 Best Hospitals: Specialty Rankings.; 2019. Accessed August 4, 2023. https://www.rti.org/publication/methodology-us-news-world-report-2019-20-best-hospitals/fulltext.pdf [Google Scholar]
- 2.3M. 3MTM All Patient Refined Diagnosis Related Groups (APR DRG) Methodology Overview.; 2022. Accessed August 4, 2023. https://apps.3mhis.com/docs/Groupers/All_Patient_Refined_DRG/Methodology_overview_GRP041/grp041_aprdrg_methodology_overview.pdf [Google Scholar]
- 3.Elixhauser A, Steiner C, Harris DR, et al. Comorbidity measures for use with administrative data. Med Care. 1998;36:8–27. [DOI] [PubMed] [Google Scholar]
- 4.Krumholz HM, Lin Z, Drye EE, et al. An administrative claims measure suitable for profiling hospital performance based on 30-day all-cause readmission rates among patients with acute myocardial infarction. Circ Cardiovasc Qual Outcomes. 2011;4:243–252. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Krumholz HM, Wang Y, Mattera JA, et al. An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with an acute myocardial infarction. Circulation. 2006;113:1683–1692. [DOI] [PubMed] [Google Scholar]
- 6.Bilimoria KY, Liu Y, Paruch JL, et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg. 2013;217:833–42.e1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Olmsted MG, Lessem S, Powell R, et al. Methodology: U.S. News & World Report 2023-24 Best Hospitals: Specialty Rankings. RTI International; 2023. Accessed August 4, 2022. https://health.usnews.com/media/best-hospitals/BH_Methodology_2023-2024 [Google Scholar]
- 8.Agency for Healthcare Research and Quality. Elixhauser Comorbidity Software Refined for ICD-10-CM. Accessed August 4, 2023. https://hcup-us.ahrq.gov/toolssoftware/comorbidityicd10/comorbidity_icd10.jsp [Google Scholar]
- 9.Yale New Haven Health Services Corporation. Center for Outcomes Research and Evaluation. Mortality Measures Methodology. Accessed August 4, 2023. https://qualitynet.cms.gov/inpatient/measures/mortality/methodology [Google Scholar]
- 10.Yale New Haven Health Services Corporation. Center for Outcomes Research and Evaluation. Readmission Measures Methodology. Accessed August 4, 2023. https://qualitynet.cms.gov/inpatient/measures/readmission/methodology [Google Scholar]
- 11.Ash AS, Ellis RP, Pope GC, et al. Using diagnoses to describe populations and predict costs. Health Care Financ Rev. 2000;21:7–28. [PMC free article] [PubMed] [Google Scholar]
- 12.Brier GW. Verification of forecasts expressed in terms of probability. Mon Weather Rev. 1950;78:1–3. [Google Scholar]
- 13.Ferro CAT, Fricker TE. A bias-corrected decomposition of the Brier score. Q J R Meteorol Soc. 2012;138:1954–1960. [Google Scholar]
- 14.Agency for Healthcare Research and Quality, Healthcare Cost and Utilization Project. User Guide: Elixhauser Comorbidity Software Refined for ICD-10-CM Diagnoses, V2021.1.; 2020. Accessed August 4, 2023. https://hcup-us.ahrq.gov/toolssoftware/comorbidityicd10/Com-ICD10CM-User-Guide-v2021-1.pdf [Google Scholar]
- 15.Gonnella JS, Hornbrook MC, Louis DZ. Staging of disease: a case-mix measurement. JAMA. 1984;251:637–644. [PubMed] [Google Scholar]
- 16.Centers for Medicare & Medicaid Services. ICD-10 Hospital Acquired Condition List. Accessed August 4, 2023. https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/HospitalAcqCond/icd10_hacs [PubMed] [Google Scholar]
- 17.Bae JA, Curtis LH, Hernandez AF. National hospital quality rankings: improving the value of information in hospital rating systems. JAMA. 2020;324:839–840. [DOI] [PubMed] [Google Scholar]
- 18.Frazier KM, Gourin CG, Stewart CM. Fatally flawed—making sense of US news & world report mortality scores. JAMA Otolaryngol Head Neck Surg. 2021;147:317–319. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Mendu ML, Kachalia A, Eappen S. Revisiting US news & world report’s hospital rankings—moving beyond mortality to metrics that improve care. J Gen Intern Med. 2021;36:209–210. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Gorelick PB. Primary and comprehensive stroke centers: history, value and certification criteria. J Stroke. 2013;15:78–89. [DOI] [PMC free article] [PubMed] [Google Scholar]