Abstract
Background
Between 5% and 10% of patients die or are urgently readmitted within 30 days of discharge from hospital. Readmission risk indexes have either excluded acute diagnoses or modelled them as multiple distinct variables. In this study, we derived and validated a score summarizing the influence of acute hospital diagnoses and procedures on death or urgent readmission within 30 days.
Methods
From population-based hospital abstracts in Ontario, we randomly sampled 200 000 discharges between April 2003 and March 2009 and determined who had been readmitted urgently or died within 30 days of discharge. We used generalized estimating equation modelling, with a sample of 100 000 patients, to measure the adjusted association of various case-mix groups (CMGs—homogenous groups of acute care inpatients with similar clinical and resource-utilization characteristics) with 30-day death or urgent readmission. This final model was transformed into a scoring system that was validated in the remaining 100 000 patients.
Results
Patients in the derivation set belonged to 1 of 506 CMGs and had a 6.8% risk of 30-day death or urgent readmission. Forty-seven CMG codes (more than half of which were directly related to chronic diseases) were independently associated with this outcome, which led to a CMG score that ranged from –6 to 7 points. The CMG score was significantly associated with 30-day death or urgent readmission (unadjusted odds ratio for a 1-point increase in CMG score 1.52, 95% confidence interval [CI] 1.49–1.56). Alone, the CMG score was only moderately discriminative (C statistic 0.650, 95% CI 0.644–0.656). However, when the CMG score was added to a validated risk index for death or readmission, the C statistic increased to 0.759 (95% CI 0.753–0.765). The CMG score was well calibrated for 30-day death or readmission.
Interpretation
In this study, we developed a scoring system for acute hospital diagnoses and procedures that could be used as part of a risk-adjustment methodology for analyses of postdischarge outcomes.
Early death or urgent readmission is common after patients are discharged from hospital. These events are undeniably important to patients and are expensive for the health care system. Being able to accurately predict who is likely to die or be readmitted to hospital can help in directing extra care to those at greatest risk, in performing risk-adjustment analyses for which death or readmission is an outcome and, potentially, in identifying mechanisms for early death or readmission.
We recently published a simple, externally validated scoring system to predict the risk that patients discharged from medical or surgical services will die or be urgently readmitted within 30 days of discharge.1 The LACE index used values for a patient’s length of stay in hospital (L), acuity of admission (A), comorbidity (C) and emergency department utilization before admission (E) to determine the risk of 30-day death or urgent readmission. The LACE index is relatively simple to calculate, works with both primary and administrative data, and was very well calibrated. However, it had only moderate discrimination (C statistic of 0.684).
Hospital diagnoses and procedures have an important influence on postdischarge outcomes. Oddly enough, however, none of the 20 most common diagnoses or 11 most common procedures that we considered in our earlier study1 was independently associated with death or urgent readmission, independent of the other LACE index covariates. Two previously published, validated risk-prediction models for hospital readmission (with substantially larger sample sizes) identified specific diagnoses that significantly predicted readmission.2,3 In those models, diagnoses were expressed in terms of a categorical variable having as many as 71 categories.
Summarizing the influence of the reason for admission to hospital on postdischarge outcomes as a score has several advantages. First, such a score facilitates the comparison or ranking of the influence of various diagnoses on such outcomes. Second, entering n diagnoses in a statistical model requires at least n – 1 binary variables, which necessitates a large number of terms in the model, a requirement that could be problematic when sample sizes are limited. In contrast, a score lets analysts model the effect of admission diagnosis with as few as one term. Third, detecting interactions between admission diagnosis and other terms in the regression model is cumbersome when the diagnosis is expressed using multiple, distinct covariates. A score that summarizes the influence of admission diagnosis on postdischarge outcomes would greatly facilitate the analysis of such interactions.
In this study, we derived and validated a scoring system (or index) to summarize the influence of acute hospital diagnoses and procedures on 30-day death or urgent readmission.
Methods
The study was approved by The Ottawa Hospital Research Ethics Board. Our study methods are outlined briefly in Figure 1.
Figure 1.
Outline of study methods
Data sources
This study used 4 population-based administrative databases that captured data on all Ontarians. The Discharge Abstract Database (DAD) records all nonpsychiatric admissions to hospital. The Ontario Mental Health Reporting System (OMHRS) captures all inpatient mental health encounters after 2006 (before which these admissions were captured in the DAD). The National Ambulatory Care Reporting System (NACRS) records all emergency department visits. The Registered Patient Database (RPDB) records all death dates.
Study cohort
We used the DAD to create the study cohort. We first identified all adult medical or surgical patients discharged to the community between 1 April 2003 and 31 March 2009. We excluded psychiatric and obstetric admissions because the LACE score—the primary covariate in our models—did not apply to these admissions. We chose the study period to ensure that we had NACRS data for at least 6 months before each admission (required to calculate the LACE score needed for our analysis) and 30-day urgent readmission data in the DAD and OMHRS for all patients. If a patient had more than one hospital discharge during the study period, we randomly selected one. From this sample, we randomly selected 200 000 patients (100 000 for model derivation and 100 000 for model validation). We were unable to use the entire sample of discharges in our analysis because of the computationally intensive nature of the statistical methods employed in this study.
LACE index
The LACE index estimates the risk of 30-day death or urgent readmission on the basis of values for 4 covariates from the index hospital stay: length of stay (L), in days; acuity of the admission (A), categorized as urgent or planned; comorbidity of the patient (C), measured with the Charlson score; and emergency department utilization (E), measured as the number of visits to the emergency department in the 6 months before the index admission. A specific number of points are assigned to each covariate value, and these points are summed to determine the total LACE score, which indicates the risk of 30-day death or urgent readmission. The LACE score was moderately discriminative (C statistic 0.684) and well calibrated with 30-day death or urgent readmission.
We measured the LACE score for all patients. The length of stay and urgency of each index admission were noted from the DAD. We calculated each person’s Charlson score using the International Statistical Classification of Diseases and Related Health Problems, 10th revision, Canada (ICD-10-CA) codes cited by Quan and colleagues.4 Finally, we linked to the NACRS to determine the number of visits to an emergency department in the 6 months before the index hospital stay.
Case-mix groups
The Canadian Institute for Health Information (CIHI) has developed methods to categorize acute care inpatients with similar clinical and resource-utilization characteristics. These categories, called “case-mix groups” (CMGs), are based on codes for the most responsible diagnosis and the primary procedure for each admission. In 2007, CIHI updated the CMG algorithm (known as “CMG+”) to group patients on the basis of primary diagnoses according to the ICD-10-CA system and primary procedures according to the Canadian Classification of Intervention codes. Minor revisions are made to this algorithm yearly (that is, patients with the same diagnostic and procedural codes may be assigned to different CMGs in different years). For the primary analysis in this study, we used the 2008 CMG algorithm, which was applied to all hospital stays back to 2003. To represent each patient’s 2008 CMG value, we created a series of binomial covariates (with values of 0 or 1) for each 2008 CMG value in the derivation set.
The derivation set contained more than 500 distinct CMG codes (Appendix 1). This derivation set captured 94% of all CMG codes recorded for 3.3 million eligible hospital admissions identified during the study period.
Outcome
The primary outcome was death or urgent readmission within 30 days of hospital discharge. Death status was determined by linking to the RPDB. Urgent readmission status was determined by linking to the DAD and OMHRS. Readmissions were included regardless of the diagnosis, as long as they were categorized as “urgent” (i.e., unplanned) and were not preceded by an earlier “non-urgent” (i.e., planned) readmission.
Analysis
We used data for 100 000 patients to derive the CMG score model. We used generalized estimating equations (GEE) methods to determine the association of each CMG category with the risk of 30-day death or urgent readmission, independent of each person’s LACE score. The GEE model clustered patients within hospitals and accounted for possible non-independence of such patients when calculating standard errors for the model’s parameter estimates.
For selection of variables, we first excluded all CMGs for which the association with the primary outcome (independent of the LACE score) had a p value exceeding 0.05. Because the model could not generate a parameter estimate for CMGs having no events, we excluded the 79 CMGs that had no events and fewer than 50 admissions (i.e., < 0.05% of all admissions). Each of the 11 CMGs that had no events but more than 50 admissions was combined with another clinically similar CMG (see Appendix 1). We then used forward variable selection (with an inclusion p value of ≤ 0.001) to identify the CMGs that were significantly associated with 30-day death or urgent readmission, independent of the LACE score and other CMGs.
We used the methods described by Sullivan and colleagues5 to modify parameter estimates for the CMG categories in the final model into a risk score. The number of points assigned to each statistically significant CMG covariate was equal to its regression coefficient divided by the CMG parameter estimate having the smallest absolute value, rounded to the nearest whole number. Each patient’s final CMG score was the number of points assigned to his or her CMG. If a patient’s CMG was not included in the model, a CMG score of 0 was assigned.
We validated the CMG score in the remaining 100 000 patients in a GEE logistic regression model with 30-day death or urgent readmission as the outcome and the CMG score as the independent variable. We repeated this model after adding the LACE score as an adjusting covariate.
We measured each model’s discrimination using the C statistic with 95% confidence intervals (CIs).6 We measured calibration by comparing the expected to observed event risk within each CMG score. The expected outcome risk for each patient was calculated as the inverse of 1 + e-(intercept + β[LACE score]*LACE score + β[CMG score]*CMG score) (where β was the coefficient of each covariate in the regression model). This value was summed across all patients with the same CMG score to calculate the expected number of 30-day deaths or urgent readmissions among patients with each CMG score. The expected and observed rates were considered similar if the expected rate was within the exact 95% CI around the observed rate.7 We summarized each model’s overall calibration using the Hosmer–Lemeshow statistic.8 Finally, we assessed discrimination and calibration for the CMG score after stratifying by LACE score quartile.
As a secondary analysis, we refitted these same models using 30-day death and 30-day urgent readmission as separate outcomes. For a sensitivity analysis, we measured the ability of the CMG score to predict 30-day death or urgent readmission when the score was determined using the CMG code based on the discharge year algorithm (rather than the 2008 algorithm). We assessed discrimination and calibration separately for patients discharged between 2003 and 2006 (when the previous CMG grouping methodology was being used), in 2007 and in 2008.
Results
Of approximately 6.5 million hospital admissions in Ontario during the study period, almost 3.3 million were eligible for the study (Figure 2). This sample included more than 1.8 million individuals, of whom 200 000 were randomly selected for the study. Each of these patients had been discharged from 1 of 183 hospitals, with each hospital contributing a median of 320 patients to the study (interquartile range [IQR] 116–1827 patients).
Figure 2.
Outline for creation of study cohort
Study cohort
Patients were generally middle-aged, and one-third had visited the emergency department in the previous 6 months (Table 1). Most hospital stays were less than half a week in duration, and the median LACE score was 5 (i.e., associated with a 5.1% risk of 30-day death or urgent readmission). The derivation and validation groups were essentially identical.
Table 1.
Characteristics of study cohort
Each patient in the derivation group was assigned to 1 of 506 CMGs (Appendix 1), the most common being hysterectomy without malignancy (CMG 502), unilateral knee replacement (CMG 321) and stable angina or chest pain without cardiac catheterization (CMG 208) (Table 1).
The overall risk of 30-day death or urgent readmission was 6.8% (Table 1). Most of the patients had an urgent readmission, and only 0.7% of the patients died in the month after discharge without a prior urgent readmission. The event rate was nearly identical in the derivation (6.8%) and validation (6.7%) groups.
Derivation of CMG score
Ninety-one CMG groups (comprising 96 CMG codes) were individually associated with 30-day death or urgent readmission (Appendix 1). When these variables were applied in multivariable modelling, 47 CMG groups (comprising 50 individual CMG codes) remained in the final model (Table 2). Many of the CMGs associated with an increased risk of 30-day death or readmission were related to neoplasia, important chronic comorbidities or diagnoses indicative of poor overall functional status. More than half of the categories dealt directly with chronic diseases (Table 2). All of the CMGs that were associated with a decreased risk of the outcomes were therapeutic procedures. This model was modified into the CMG score, which ranged from –6 to 7 (Table 2), with negative scores indicating CMGs that were protective against 30-day death or urgent readmission.
Table 2.
Model for case-mix group (CMG) risk score
Validation of CMG score
In the validation set, the CMG score had a modal distribution, with 0 being the most common value (Figure 3). The CMG score was significantly associated with 30-day death or urgent readmission (unadjusted odds ratio [OR] 1.52, 95% CI 1.49–1.56 for each 1-point increase in the CMG score). This association persisted when the LACE score was added to the model (adjusted OR 1.34, 95% CI 1.31–1.36). The CMG score was also significantly associated with each outcome separately, but the association was stronger with 30-day death (adjusted OR 1.55, 95% CI 1.50–1.59) than with urgent readmission (adjusted OR 1.28, 95% CI 1.26–1.31).
Figure 3.
CMG score distribution and its association with 30-day death or urgent readmission in the validation cohort
By itself, the CMG score was only moderately discriminative for predicting 30-day death or urgent readmission (C statistic 0.650, 95% CI 0.644–0.656) (Table 3). However, including the CMG score significantly improved the performance of the LACE index, as indicated by a significant increase in the C statistic, from 0.735 (95% CI 0.729–0.741) to 0.759 (95% CI 0.753–0.765). The CMG score was well calibrated, since the expected event rate was within the 95% CI of the observed rate for 11 (79%) of the 14 CMG scores (Figure 4). When the CMG score was stratified by quartiles of the LACE score, model calibration was even better: the expected event rate fell within the 95% CI of the observed rate for 50 (89%) of 56 strata.
Table 3.
Predictive performance of the LACE index and the case-mix group (CMG) score
Figure 4.
Calibration of CMG score for 30-day death or urgent readmission by LACE index strata
The CMG score alone was only moderately discriminative for 30-day urgent readmission but was very discriminative for 30-day death (Table 3). The discrimination of the CMG score for each outcome separately improved with the addition of the LACE index and was significantly better than that of the LACE index alone (as was observed for the primary outcome).
Sensitivity analysis
When the CMG code was assigned according to the algorithm of the discharge year (rather than the 2008 algorithm), the discrimination of the CMG score (both alone and when combined with the LACE index) for predicting 30-day death or urgent readmission was similar for discharges occurring in 2007 and 2008 (Table 4). However, for discharges occurring before 2007, the discrimination was notably lower for the CMG score alone and slightly lower when the LACE index was added to the model (Table 4). The same findings were observed for each outcome separately (Table 4).
Table 4.
Predictive performance of the CMG score for 30-day death and urgent readmission using the CMG code assigned in the discharge year
Interpretation
In this study of patients discharged from medical or surgical hospital services, we identified the CMGs that were associated with risk of 30-day death or urgent readmission independent of a validated risk index. The CMGs that increased the risk of these outcomes were related to neoplastic processes, other important chronic diseases and poor overall function; only procedure-related CMGs were protective. When expressed as a single index, independent of the LACE index, the CMG score was significantly associated with 30-day death or readmission. When it was combined with the LACE index, the CMG significantly improved the ability of the LACE index to predict the risk of early postdischarge death or urgent readmission.
The CMG score efficiently summarized the effect of a large number of diagnostic and procedural groups on postdischarge outcomes. We identified those CMGs for which there was a significant residual association with 30-day death or urgent readmission after adjustment for factors known to influence this outcome (including length of stay, acuity of the admission, patient comorbidity and preadmission emergency department utilization). We also quantified this residual association with a score.
To validate our use of forward variable selection to identify significant CMGs, we conducted additional analyses to determine if backward selection would have produced different results. We first ran an initial multivariable model containing the LACE score and all eligible CMG categories (n = 91). We then ran a second model containing the LACE score and only those CMG categories with p ≤ 0.01 in the first model (n = 67). In this second model, 49 CMGs had a p value ≤ 0.001, including all 47 CMGs that we had identified using forward selection; the other 2 CMGs had p values of 0.0010 and 0.0009, very close to our cut-off for inclusion. This analysis suggests that our results are stable and would be virtually the same regardless of the selection method used (forward or backward).
The CMG score has several advantages for researchers. It reduces to a single number the independent influence of acute conditions requiring admission on postdischarge outcomes. This in turn simplifies regression modelling by decreasing the number of parameters required to capture important confounders, which is especially important when sample size is limited and model over-fitting is a concern. Reducing the influence of multiple conditions to a single number also simplifies the search for interactions between acute conditions and other covariates. The CMG score can therefore be used to adjust for potential confounding whenever Canadian administrative data are used to examine early death or unplanned hospital readmission.
Our study had several advantages and limitations. Because we used a population-based, random sample to derive our model, it will be applicable to other medical and surgical patients admitted to Ontario hospitals. We believe that the results should also be applicable in other provinces, but this should be established through retrospective population-based analyses before the CMG score is widely applied to data from other Canadian provinces. Second, because our study used CMG codes to cluster acute diagnoses and procedures, the CMG score can be used only with abstract data that have been grouped according to CIHI’s CMG classifier. This means that researchers from other countries will need to derive their own acute diagnosis score. Third, we derived the CMG score using CMGs assigned by the 2008 algorithm. Our sensitivity analysis showed that the CMG score worked just as well when CMGs were assigned according to the 2007 algorithm. However, the CMG score was not as discriminative when CMG algorithms for years before 2007 were used, probably because CIHI used a substantially different CMG algorithm before 2007, one that was based on codes from the ICD 9th Revision, Clinical Modification (ICD-9-CM) translated to ICD-10-CA. These results indicate that our CMG score will likely continue to be valid with future CMG algorithms, so long as there are no major methodological shifts in those algorithms. Annual testing of the CMG score with new CMG algorithms will be required to ensure its applicability.
In this study, we derived and validated an index that quantifies the influence of particular acute diagnoses and procedures on postdischarge outcomes. This index could be used as part of a risk-adjustment methodology for analyses in which early death or urgent readmission is an outcome.
Biographies
Carl van Walraven, MD, MSc, is a Senior Scientist in Clinical Epidemiology at the Ottawa Hospital Research Institute (OHRI), Ottawa, Ontario, Canada; Adjunct Scientist at the Institute for Clinical Evaluative Sciences (ICES), Toronto, Ontario, Canada; and Associate Professor of Medicine at the University of Ottawa, Ottawa, Ontario, Canada.
Jenna Wong, MSc, is a Methodologist in Clinical Epidemiology at OHRI and an Analyst at ICES.
Alan J. Forster, MD, MSc, is a Senior Scientist in Clinical Epidemiology at OHRI, an Adjunct Scientist at ICES and Associate Professor of Medicine at the University of Ottawa.
Appendix
Appendix 1.
Univariate association (independent of the LACE index) of each Case Mix Group (CMG) with 30-day death or urgent readmission
Footnotes
Competing interests: None declared.
Funding source: This study was funded by the Department of Medicine, University of Ottawa.
Carl van Walraven conceived the project idea, wrote the study protocol, directed the study analysis, wrote the first draft of the paper and is the guarantor for the study. Jenna Wong conducted the statistical analysis, produced the tables and figures, and contributed to the writing and editing of the manuscript. Alan Forster helped guide the analysis and interpretation of the study results and also critically reviewed the paper for intellectual content. All authors have read and approved the final version of the manuscript.
References
- 1.van Walraven Carl, Dhalla Irfan A, Bell Chaim, Etchells Edward, Stiell Ian G, Zarnke Kelly, Austin Peter C, Forster Alan J. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ. 2010 Mar 1;182(6):551–557. doi: 10.1503/cmaj.091117. http://www.cmaj.ca/cgi/doi/10.1503/cmaj.091117. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Bottle Alex, Aylin Paul, Majeed Azeem. Identifying patients at high risk of emergency hospital admissions: a logistic regression analysis. J R Soc Med. 2006;99(8):406–414. doi: 10.1258/jrsm.99.8.406. http://ukpmc.ac.uk/abstract/MED/16893941. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Billings John, Dixon Jennifer, Mijanovich Tod, Wennberg David. Case finding for patients at risk of readmission to hospital: development of algorithm to identify high risk patients. BMJ. 2006 Jun 30;333(7563):327. doi: 10.1136/bmj.38870.657917.AE. http://ukpmc.ac.uk/abstract/MED/16815882. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Quan Hude, Sundararajan Vijaya, Halfon Patricia, Fong Andrew, Burnand Bernard, Luthi Jean-Christophe, Saunders L D, Beck Cynthia A, Feasby Thomas E, Ghali William A. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. 2005;43(11):1130–1139. doi: 10.1097/01.mlr.0000182534.19832.83. http://www.scholaruniverse.com/ncbi-linkout?id=16224307. [DOI] [PubMed] [Google Scholar]
- 5.Sullivan Lisa M, Massaro Joseph M, D'Agostino Ralph B., Sr Presentation of multivariate data for clinical use: The Framingham Study risk score functions. Stat Med. 2004 May 30;23(10):1631–1660. doi: 10.1002/sim.1742. http://www.scholaruniverse.com/ncbi-linkout?id=15122742. [DOI] [PubMed] [Google Scholar]
- 6.Gonen M. Analyzing receiver operating characteristic curves with SAS. Cary (NC): SAS Institute Inc; 2007. Single continuous predictor; p. 15. [Google Scholar]
- 7.Daly Leslie. Simple SAS macros for the calculation of exact binomial and Poisson confidence limits. Comput Biol Med. 1992;22(5):351–361. doi: 10.1016/0010-4825(92)90023-G. http://linkinghub.elsevier.com/retrieve/pii/001048259290023G. [DOI] [PubMed] [Google Scholar]
- 8.Hosmer D W, Hosmer T, Le Cessie S, Lemeshow S. A comparison of goodness-of-fit tests for the logistic regression model. Stat Med. 1997 May 15;16(9):965–980. doi: 10.1002/(SICI)1097-0258(19970515)16:9<965::AID-SIM509>3.0.CO;2-O. [DOI] [PubMed] [Google Scholar]