Abstract
Background:
Predicting long-term survival after admission to hospital is helpful for clinical, administrative and research purposes. The Hospital-patient One-year Mortality Risk (HOMR) model was derived and internally validated to predict the risk of death within 1 year after admission. We conducted an external validation of the model in a large multicentre study.
Methods:
We used administrative data for all nonpsychiatric admissions of adult patients to hospitals in the provinces of Ontario (2003–2010) and Alberta (2011–2012), and to the Brigham and Women’s Hospital in Boston (2010–2012) to calculate each patient’s HOMR score at admission. The HOMR score is based on a set of parameters that captures patient demographics, health burden and severity of acute illness. We determined patient status (alive or dead) 1 year after admission using population-based registries.
Results:
The 3 validation cohorts (n = 2 862 996 in Ontario, 210 595 in Alberta and 66 683 in Boston) were distinct from each other and from the derivation cohort. The overall risk of death within 1 year after admission was 8.7% (95% confidence interval [CI] 8.7% to 8.8%). The HOMR score was strongly and significantly associated with risk of death in all populations and was highly discriminative, with a C statistic ranging from 0.89 (95% CI 0.87 to 0.91) to 0.92 (95% CI 0.91 to 0.92). Observed and expected outcome risks were similar (median absolute difference in percent dying in 1 yr 0.3%, interquartile range 0.05%–2.5%).
Interpretation:
The HOMR score, calculated using routinely collected administrative data, accurately predicted the risk of death among adult patients within 1 year after admission to hospital for nonpsychiatric indications. Similar performance was seen when the score was used in geographically and temporally diverse populations. The HOMR model can be used for risk adjustment in analyses of health administrative data to predict long-term survival among hospital patients.
The life expectancy of individual patients can be important for both medical decision-making and research. Patients with a short life expectancy may choose to defer preventive treatments, screening interventions or interventional procedures for conditions that are currently asymptomatic. An accurate assessment of risk of death, particularly if that risk is high, could motivate and inform discussions between patients and physicians regarding goals of care. In addition, accurate prognostications are essential for adjusting statistical models that have death as an outcome (or as a competing risk for other outcomes) in both research and administration.
We recently derived and internally validated a model that predicts the risk of death from any cause at 1 year after admission to hospital.1 The Hospital-patient One-year Mortality Risk (HOMR) model consists of covariates whose values are determined at admission using routinely collected health administrative data (Figure 1). These covariates include patient demographics (age, sex and living status); health burden (measured using the Charlson Comorbidity Index score, home oxygen status and the number of visits to emergency departments and admissions to hospital by ambulance in the previous year); and acuity of illness (admission urgency and hospital service, direct admission to an intensive care unit and whether the admission was an urgent readmission to hospital). The latter category was also gauged using the Diagnostic Risk Score, which quantifies risk of death for particular diagnoses beyond that explained by the other covariates (Appendix 1, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.150209/-/DC1).
Figure 1:
Covariates used to calculate a patient’s Hospital-patient One-year Mortality Risk (HOMR) score at the time of admission to hospital. The Diagnostic Risk Score (Appendix 1) quantifies risk of death for diagnostic groups beyond that explained by the other covariates. Points for the interacting covariates of age and Charlson Comorbidity Index score include the risk of patient age, comorbidity score and their interaction. In contrast, points for living status and admission urgency include the risk of these covariates and their interaction with admissions by ambulance in the previous year; points for the latter covariate are considered separately. See Table 3 for the expected risk of death within 1 year after hospital admission for each HOMR score. ED = emergency department, ICU = intensive care unit. *In the year before admission. †See Appendix 2 for definitions of each service. (Appendices are available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.150209/-/DC1)
Discrete values for each covariate are given specific points, which are summed to create the HOMR score (Figure 1). In an internal validation population, the HOMR score accurately predicted the risk of death from any cause within 1 year after admission, with a C statistic of 0.92 and excellent calibration among adult residents of Ontario admitted to hospital for nonpsychiatric indications in 2011.1
Although these statistics are impressive, external validation is required to determine the true usefulness of any statistical model. External validation is necessary to prove that the model’s performance is not idiosyncratic to the patients, physicians, institutions or data systems used to derive and internally test it.2,3 A prognostic model should remain accurate when retested with different patients (reproducibility), during different periods (historical transportability) and in different locations (geographic transportability).4 We conducted an external validation of the HOMR model in a multicentre study that included Canadian and American hospitals.
Methods
Study settings and data sources
We used health administrative data from 2 Canadian provinces (Ontario and Alberta) and the Brigham and Women’s Hospital in Boston. All data were linked and analyzed anonymously. The study was approved by the research ethics board at each study site (Sunnybrook and Women’s College Health Sciences Centre Research Ethics Board, the University of Alberta Health Research Ethics Board and the Partners Human Research Committee).
All health care activities in Ontario and Alberta are publicly funded and are captured in population-based administrative datasets. These datasets include the Discharge Abstract Database, which captures all hospital admissions and, in the years we sampled, used diagnostic codes from the International Statistical Classification of Diseases and Related Health Problems, 10th revision (ICD-10); the Registered Persons Database, which captures each person’s date of death (including deaths that occur out of province); and the National Ambulatory Care Reporting System, which captures all visits to emergency departments. For the Ontario cohort, we also used the Assistive Devices Program, which captures all patients who receive home oxygen; the Continuing Care Reporting System, which captures all residents of registered nursing homes and hospitals (i.e., inpatients who are unable to live at home or in a nursing home, usually because they have advanced nursing requirements); and the Home Care Database, which captures all publicly funded in-home assistance. Home oxygen status could not be determined for the patients in Alberta. We linked all of the databases deterministically using encrypted health insurance numbers.
The Brigham and Women’s Hospital is a 779-bed not-for-profit teaching affiliate of Harvard Medical School in Boston. It has about 44 000 inpatient admissions, 54 000 emergency department visits and more than 950 000 ambulatory visits each year. Patients come from a variety of referral sources, including practices based in the hospital, community-based private practices and community health centres affiliated with Partners Community Healthcare, Inc., and nonaffiliated practices within and outside of the Boston area. Because the Brigham and Women’s Hospital acts as both a high-level academic tertiary hospital and a primary hospital for the surrounding (mainly low-income) community, we considered it to be similar to other academic American hospitals. All health activities for these patients are captured in health administrative datasets, including the Research Patient Data Registry, which records diagnoses using codes from the International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM), Diagnosis-Related Group codes and dates of death from the Social Security Death Registry; the Longitudinal Medical Record, where clinical data can be extracted from medication or problem lists; the Enterprise Master Patient Index, which coordinates patient identification across the multiple administrative datasets and maintains a database of selected demographic information for patients; and the Brigham Integrated Computing System, which captures all inpatient data. Home nursing and chronic hospital status were unavailable for patients at the Brigham and Women’s Hospital.
Study cohorts
In all 3 cohorts, we included adults (age ≥ 18 yr in Ontario and Alberta; age ≥ 20 in Boston) with a valid identifier who had at least 1 admission to a hospital for an acute, nonpsychiatric indication during the study period. For people who had multiple admissions, we randomly selected 1 admission per person during the study period. The Ontario cohort included eligible admissions to all acute care hospitals in the province between Jan. 1, 2003 (corresponding to the introduction of ICD-10) and Dec. 31, 2010 (final date for which patients not included in the HOMR derivation group were available). The Alberta cohort included eligible admissions to all hospitals in the province between Apr. 1, 2011, and Mar. 31, 2012. The Boston cohort included eligible admissions to the Brigham and Women’s Hospital between Jan. 1, 2010, and Dec. 31, 2012. In all 3 cohorts, we excluded patients who underwent same-day surgery and those who had been transferred from another hospital.
Outcome measure
The primary outcome measure was death from any cause within 1 year after admission to hospital. All deaths, including those that happened during the index admission, were captured. For the Ontario and Alberta cohorts, outcome status was determined by linking to the Registered Persons Database. For the Boston cohort, outcome status was determined by linking to the Research Patient Data Registry.
Other covariates
Definitions and database sources of the covariates in the HOMR model are provided in Appendix 2, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.150209/-/DC1. The HOMR score is calculated using data values at patient admission. The Diagnostic Risk Score (Appendix 1) was previously derived using ICD-10 codes.1 Because ICD-9-CM codes were used for diagnoses in the Boston cohort, we used crosswalk tables (provided by the Canadian Institute for Health Information) to identify the ICD-9-CM code(s) that most accurately corresponded to the ICD-10 codes included in the Diagnostic Risk Score.
Statistical analysis
We calculated each patient’s HOMR score. Within each cohort, we calculated the proportion of patients who died within 1 year after admission for each distinct HOMR score value. To measure discrimination of the HOMR scores, we used a binomial logistic regression model to calculate the C statistic, where death within 1 year after hospital admission was the outcome and the HOMR score (expressed as a linear term) was the sole independent variable. To measure calibration, we calculated 95% confidence intervals (CIs) around the observed risk of death associated with each HOMR score and determined whether this included the expected risk of death calculated from the original study cohort. For each HOMR score, we also calculated the absolute difference between the observed and the expected proportion of patients who died within 1 year after admission and the relative difference (calculated as the absolute difference divided by the expected proportion).
We produced an expected risk of death within 1 year for each HOMR score using data from all 3 validation populations and a random effects model that accounted for patients clustered within each cohort. This model had a single independent covariate — the HOMR score expressed as a linear term — and we used it to generate 95% prediction intervals (not confidence intervals) for the expected outcome risk for each HOMR score.
We performed all statistical analyses using SAS version 9.3 (SAS Institute, Inc.).
Results
Characteristics of the patients in the 3 validation cohorts and in the derivation cohort1 are shown in Table 1. We observed several notable differences between the cohorts. Patients in the Alberta cohort were younger than those in the other cohorts. Patients in the derivation cohort were much more likely to be categorized as living at home with home care services. Patients in the Boston cohort had notably higher Charlson Comorbidity Index scores (with only 36.8% having a score of 0); they also were much more likely to have been admitted to hospital by ambulance in the previous year but were less likely to be admitted to the intensive care unit. General surgery was more prominent within admission services in the Alberta cohort, whereas hematology/oncology, cardiovascular surgery and neurosurgery were more prominent in the Boston cohort.
Table 1:
Characteristics of patients in the derivation cohort and in the 3 cohorts for external validation of the HOMR score
| Characteristic* | Cohort; no. (%) of patients† | |||
|---|---|---|---|---|
| Derivation1 (Ontario 2011) n = 640 022 | Ontario (2003–2010) n = 2 862 996 | Alberta (2011–2012) n = 210 595 | Boston (2010–2012) n = 66 683 | |
| Age, yr, median (IQR) | 59 (37 to 75) | 54 (35 to 72) | 51 (32 to 70) | 57 (45 to 69) |
| Male sex | 244 273 (38.2) | 1 074 241 (37.5) | 77 933 (37.0) | 29 426 (44.1) |
| Living status before admission | ||||
| Home, independent living | 531 017 (83.0) | 2 689 908 (94.0) | 202 949 (96.4) | 66 000 (99.0) |
| Rehabilitation facility | 1 113 (0.2) | 1 029 (0.04) | –§ | 473 (0.7) |
| Home, with home care | 77 589 (12.1) | 109 328 (3.8) | 7 713 (3.7) | –§ |
| Nursing home | 28 628 (4.5) | 60 080 (2.1) | 2 103 (1.0) | 210 (0.3) |
| Chronic care hospital | 1 675 (0.3) | 2 651 (0.1) | 181 (0.1) | –§ |
| Charlson Comorbidity Index score | ||||
| 0 | 369 844 (57.8) | 1 912 524 (66.8) | 129 539 (61.5) | 24 554 (36.8) |
| 1–2 | 138 987 (21.7) | 574 476 (20.1) | 44 023 (20.9) | 20 836 (31.2) |
| ≥ 3 | 131 191 (20.5) | 375 996 (13.1) | 39 384 (18.7) | 21 293 (31.9) |
| Home oxygen | 15 018 (2.3) | 19 235 (0.7) | –§ | 1 702 (2.6) |
| ≥ 1 visit to emergency department in past 12 mo | 287 678 (44.9) | 1 293 810 (45.2) | 71 678 (34.0) | 32 873 (49.3) |
| ≥ 1 admission to hospital by ambulance in past 12 mo | 60 339 (9.4) | 151 452 (5.3) | 22 522 (10.7) | 22 522 (33.8) |
| Admitting service — medicine | ||||
| General | 200 726 (31.4) | 769 804 (26.9) | 76 821 (36.5) | 23 094 (34.6) |
| Cardiology | 40 720 (6.4) | 188 947 (6.6) | 5 387 (2.6) | 6 538 (9.8) |
| Gastroenterology/nephrology/neurology | 31 454 (4.9) | 127 020 (4.4) | 4 589 (2.2) | 2 076 (3.1) |
| Palliative care | 4 999 (0.8) | 6 348 (0.2) | 1 372 (0.7) | –§ |
| Hematology/oncology | 15 031 (2.3) | 62 483 (2.2) | 1 293 (0.6) | 5 730 (8.6) |
| Admitting service — surgery | ||||
| General | 70 486 (11.0) | 344 679 (12.0) | 37 886 (18.0) | 5 490 (8.2) |
| Cardiovascular | 12 191 (1.9) | 54 853 (1.9) | 1 441 (0.7) | 6 766 (10.1) |
| Neurosurgery | 7 667 (1.2) | 36 922 (1.3) | 1 519 (0.7) | 2 405 (3.6) |
| Orthopedic | 53 865 (8.4) | 252 299 (8.8) | 12 850 (6.1) | 5 921 (8.9) |
| Plastic | 13 646 (2.1) | 77 055 (2.7) | 4 355 (2.1) | 1 430 (2.1) |
| Thoracic/transplant | 4 006 (0.6) | 12 808 (0.4) | 713 (0.3) | 258 (0.4) |
| Trauma | 8 573 (1.3) | 43 268 (1.5) | 2 645 (1.3) | 960 (1.4) |
| Urology | 20 997 (3.3) | 99 571 (3.5) | 4 331 (2.1) | 1 959 (2.9) |
| Obstetrics/gynecology | ||||
| Ante-, intra-, postpartum | 131 639 (20.6) | 634 971 (22.2) | 50 045 (23.8) | 29 (0.04) |
| Gynecology | 24 022 (3.8) | 151 879 (5.3) | 5 348 (2.5) | 4 027 (6.0) |
| Admission urgency | ||||
| Elective | 303 145 (47.4) | 1 485 392 (51.9) | 85 232 (40.5) | 32 042 (48.1) |
| Emergency department, no ambulance | 164 619 (25.7) | 787 321 (27.5) | 69 289 (32.9) | 28 068 (42.1) |
| Emergency department, ambulance | 172 258 (26.9) | 590 283 (20.6) | 56 074 (26.6) | 6 555 (9.8) |
| Urgent readmission (≤ 30 d of previous admission) | 29 128 (4.6) | 50 921 (1.8) | 11 979 (5.7) | 5 061 (7.6) |
| Admitted to intensive care unit | 47 637 (7.4) | 209 175 (7.3) | 14 525 (6.9) | 2 771 (4.2) |
| Diagnostic Risk Score,‡ mean ± SD | −1.79 ± 4.42 | −1.99 ± 4.52 | −1.44 ± 3.70 | −1.42 ± 3.72 |
Note: Boston = Brigham and Women’s Hospital, HOMR = Hospital-patient One-year Mortality Risk, IQR = interquartile rage, SD = standard deviation.
Further details about the characteristics and the database sources are given in Appendix 2.
Unless stated otherwise.
Details about the Diagnostic Risk Score are given in Appendix 1. In the validation cohort, the Diagnostic Risk Score ranged from −22 to 12 (median 0, interquartile range −3 to 0).
Data unavailable in the administrative datasets.
Despite their disparate characteristics, the cohorts had similar mean Diagnostic Risk Scores (Table 1) and HOMR scores. The median HOMR score was 22 (interquartile range [IQR] 9 to 32, range −12 to 73) in Ontario, 23 (IQR 12 to 50, range −7 to 69) in Alberta and 27 (IQR 20 to 34, range −8 to 59) in Boston. In the derivation cohort, the median HOMR score was 25 (IQR 12 to 36, range −12 to 76).
In the validation cohorts, the overall all-cause mortality rate within 1 year after admission was 8.7%. It varied slightly between the patient groups: 8.76% (95% CI 8.73% to 8.79%) in the Ontario cohort, 8.45% (95% CI 8.33% to 8.57%) in the Alberta cohort and 8.70% (95% CI 8.49% to 8.91%) in the Boston cohort. In the derivation cohort, the 1-year mortality was 11.73% (95% CI 11.65% to 11.81%).
The HOMR score was strongly associated with 1-year risk of death and was highly discriminative in each validation cohort. When entered as a linear term in a logistic regression model, the score was strongly associated with risk of death in all 3 validation cohorts, with an odds ratio for death per 1-point increase in HOMR score of 1.23 (95% CI 1.23 to 1.23) in the Ontario cohort, 1.24 (95% CI 1.23 to 1.24) in the Alberta cohort and 1.24 (95% CI 1.24 to 1.25) in the Boston cohort. The HOMR score was discriminative, with C statistics of 0.92 (95% CI 0.91 to 0.92) in Ontario, 0.92 (95% CI 0.90 to 0.93) in Alberta and 0.89 (95% CI 0.87 to 0.91) in Boston.
In all 3 validation cohorts, the 1-year risk of death for distinct HOMR scores was similar to the pattern found in the original study1 (Figure 2). For HOMR scores between 30 and 50, the observed risk of death tended to exceed the expected risk, but absolute differences were small. For scores above 50, the observed risk of death tended to be lower than the expected risk in the Boston cohort; there were fewer patients with such scores in this cohort than in the Ontario and Alberta cohorts.
Figure 2:
Observed and expected risks of death within 1 year after admission to hospital in the 3 validation cohorts (Ontario, Alberta and Boston) and the derivation cohort,1 as calculated by the Hospital-patient One-year Mortality Risk (HOMR) model. Error bars indicate 95% confidence intervals. A summary of the calibration of each cohort to the expected risk of death is given in Table 2.
The HOMR score was also well calibrated with the 1-year risk of death. In the validation cohorts, the expected risk of death fell within the 95% CIs of the observed risk 31.0% of the time in Ontario, 57.1% of the time in Alberta and 53.8% of the time in the Boston cohort (Table 2); in the derivation cohort, this happened 66.7% of the time. The relative differences between the observed and expected percentage of patients dead at 1 year were notable, reaching a median value of 0.26 (i.e., 26%) in the Boston cohort. However, the absolute differences between the observed and expected percentages of patients dead at 1 year were small in all of the validation cohorts, with a median value of 0.3% in all 3 patient groups.
Table 2:
Calibration of the HOMR score in the derivation and validation cohorts
| Calibration measure | Cohort | |||
|---|---|---|---|---|
| Derivation n = 87* | Ontario n = 84* | Alberta n = 77* | Boston n = 65* | |
| % of HOMR scores for which 95% CI of observed risk of death included expected risk of death | 66.7 | 31.0 | 57.1 | 53.8 |
| Difference between observed and expected % of patients dead at 1 yr | ||||
| Relative,† % | ||||
| Median (IQR) | 0.05 (0.02 to 0.25) | 0.17 (0.06 to 0.26) | 0.09 (0.03 to 0.54) | 0.26 (0.06 to 1) |
| Range | 0–2.02 | 0–226.86 | 0–1.14 | 0–5.27 |
| Absolute,‡ % | ||||
| Median (IQR) | 0.2 (0.02 to 0.72) | 0.33 (0.09 to 2.78) | 0.34 (0.03 to 1.95) | 0.34 (0.05 to 1.76) |
| Range | 0–4.46 | 0–8.72 | 0–11.1 | 0–39.35 |
Note: Boston = Brigham and Women’s Hospital, CI = confidence interval, HOMR = Hospital-patient One-year Mortality Risk, IQR = interquartile range.
n refers to the number of discrete values of the HOMR score represented by at least 1 patient within the cohort.
Relative difference was calculated as the absolute difference divided by the expected % dead at 1 yr.
Absolute difference was calculated as observed % dead at 1 yr minus expected % dead at 1 yr.
We used data from the validation cohorts to calculate the expected 1-year risk of death for each HOMR score (Table 3). The uncertainty around these predicted risks was small, with relative differences between expected values and the upper 95% prediction interval having a median value of 0.3% (IQR 0.2% to 0.7%).
Table 3:
Expected risk of death within 1 year after admission to hospital, by HOMR score*
| HOMR score | Risk of death (95% predictive interval) |
|---|---|
| < 21 | 0.2 (0.2–0.2) |
| 21 | 1.0 (1.0–1.1) |
| 22 | 1.2 (1.1–1.2) |
| 23 | 1.4 (1.4–1.5) |
| 24 | 1.7 (1.6–1.8) |
| 25 | 2.2 (2.1–2.3) |
| 26 | 2.7 (2.6–2.8) |
| 27 | 3.3 (3.2–3.4) |
| 28 | 4.2 (4.0–4.3) |
| 29 | 4.9 (4.8–5.1) |
| 30 | 6.3 (6.1–6.5) |
| 31 | 7.7 (7.5–7.9) |
| 32 | 9.4 (9.2–9.6) |
| 33 | 11.4 (11.2–11.6) |
| 34 | 13.6 (13.4–13.9) |
| 35 | 16.1 (15.8–16.4) |
| 36 | 19.9 (19.6–20.2) |
| 37 | 23.2 (22.9–23.5) |
| 38 | 27.1 (26.7–27.4) |
| 39 | 32.4 (32.0–32.7) |
| 40 | 37.2 (36.8–37.6) |
| 41 | 42.8 (42.3–43.2) |
| 42 | 46.2 (45.7–46.7) |
| 43 | 50.7 (50.2–51.3) |
| 44 | 53.0 (52.4–53.6) |
| 45 | 57.9 (57.3–58.6) |
| 46 | 63.3 (62.5–64.0) |
| 47 | 69.7 (68.9–70.6) |
| 48 | 74.3 (73.3–75.3) |
| 49 | 78.4 (77.2–79.5) |
| 50 | 80.6 (79.3–81.8) |
| 51 | 82.4 (80.9–83.8) |
| 52 | 86.3 (84.6–87.8) |
| 53 | 89.3 (87.4–90.9) |
| 54 | 91.1 (89.0–92.8) |
| 55 | 90.3 (87.9–92.2) |
| > 55 | 96.9 (96.4–97.3) |
Note: HOMR = Hospital-patient One-year Mortality Risk.
For each HOMR score, the expected risk and 95% predictive interval (not confidence interval) was calculated by means of a random effects model that accounted for the multiple locations of the validation cohort (i.e., Ontario, Alberta and Boston [Brigham and Women’s Hospital]).
Interpretation
This multicentre study included a large and diverse population of patients and showed that the HOMR score readily discriminated between patients who died and those who survived during the year after admission to hospital. The score was well calibrated and was, within this sample, both temporally and geographically transportable. Therefore, the HOMR score can be used with confidence to predict risk of death within 1 year after admission using health administrative data.
We believe that our external validation of the HOMR model has important implications for research and monitoring of health care systems. Researchers and analysts will be able to adjust for risk of death in analyses involving a broad range of hospital patients. Use of the HOMR model will also allow administrators and decision-makers to compare more accurately in-hospital and post-discharge mortality outcomes between institutions, regions or health care systems. Most of the information required to calculate the HOMR score can be determined from fields typically available in hospital discharge abstract databases (Appendix 2). Finally, we believe that the HOMR model can be used to study, at a population-based level, the expected survival of patients undergoing preventive treatments or screening interventions, to assess the appropriateness of their use.
The HOMR model joins several other points-based models that predict risk of death among hospital patients (Table 4), most of which have been discussed in detail by Yourman and colleagues.20 The HOMR model had a much larger derivation population than the other indexes and is applicable to a broader range of patients, since other indexes were limited to particular age groups, admission services or diagnoses. The discrimination of the HOMR model significantly exceeds that of other indexes, although some of this difference may be due to its inclusion of a large population with widely varying risks of death. Finally, in addition to the HOMR model, only 3 other indexes have been externally validated. This finding supports a recently published systematic review, which found that validation of risk-prediction models is uncommon.21
Table 4:
Summary of point-based models for predicting risk of death among hospital patients
| Model/study | N (derivation) | Description of derivation cohort (recruitment period) | Cohort; C statistic | |
|---|---|---|---|---|
| Derivation | External validation | |||
| Silver Code5 | 5 457 | Patients ≥ 75 yr admitted to medical ward from emergency department (2005) | 0.66 | – |
| SAFES6 | 870 | Patients ≥ 75 yr admitted to medical ward from emergency department (2001–2002) | 0.72 | – |
| CARING7 | 435 | All patients admitted to medical service (1999) | 0.82 | – |
| BISEP8 | 525 | Patients ≥ 70 yr admitted to general medical service (1989–1990) | 0.83 | 0.739 |
| SUPPORT10 | 9 105 | Patients ≥ 18 yr with high-risk admission diagnoses (1989–1994) | – | – |
| Levine et al.11 | 6 534 | Patients ≥ 65 yr discharged from general medical service (1997–2001) | 0.70 | – |
| MPI12 | 838 | Patients ≥ 65 yr admitted to geriatric unit (2004) | 0.75 | 0.80–0.8313 |
| 0.8015 | ||||
| 0.7516 | ||||
| 0.6417 | ||||
| 0.7718 | ||||
| HELP14 | 1 266 | Patients ≥ 80 yr admitted ≥ 2 d for nonelective reasons (1993–1994) | 0.74 | – |
| Walter et al.19 | 1 495 | Patients ≥ 70 yr discharged from general medical service (1993–1997) | 0.75 | 0.729 |
| HOMR1 | 319 531 | All adults admitted to nonpsychiatric hospital services (2011) | 0.92 | 0.89–0.92 |
Note: BISEP = Burden of Illness Score for Elderly Persons; CARING = cancer, admissions ≥ 2, residence in a nursing home, intensive care unit admission with multiorgan failure, ≥ 2 noncancer hospice guidelines; HELP = Hospitalized Elderly Longitudinal Project; HOMR = Hospital-patient One-year Mortality Risk; MPI = Multidimensional Prognostic Index; SAFES = Sujet Âgé Fragile – Évaluation Suivi (Frail Elderly Subject – Assessment Follow-up); SUPPORT = Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatments.
Limitations
Several issues about this validation and the HOMR score should be highlighted. First, the validation cohorts were selected from all hospital patients in 3 regions that were chosen on the basis of availability rather than randomly. As such, the generalizability of our results to other hospital populations is uncertain. We did, however, include 3 large and disparate patient populations from distinct periods, health care systems, countries and coding systems. The fact that the HOMR model was both discriminative and well calibrated in these disparate populations is reassuring to future users who plan to apply the model to other patient groups.
Second, further testing of the HOMR model is required to prove its accuracy in other, more geographically and temporally distinct populations.
Third, not all of the data components required to calculate the HOMR score were available in the validation cohorts. The fact that the HOMR model remained discriminative and well calibrated despite these missing data indicates its robustness.
Finally, the HOMR model cannot necessarily be used to predict outcomes with the use of primary data. For most of the HOMR covariates (e.g., patient age, admission service), the difference between administrative data and primary data would likely be small to nonexistent. However, differences could be considerable for the Charlson Comorbidity Index score, because administrative data tend to underestimate comorbid conditions.22–24 As such, the number of points assigned to the Charlson score might be inappropriately high if the HOMR score were calculated using primary data. On the other hand, the predictive capability of the HOMR score may be improved with the use of primary data, because this would improve the accuracy of comorbidity capture. In addition, use of primary data would permit the proper quantification of the severity of comorbid conditions and the functional status of patients, both of which have important prognostic implications.25–27 Thus, evaluation of the performance of the HOMR model in cohorts with primary data capture would be an important direction for future research.
Conclusion
We have shown in this large multicentre study that the HOMR score, calculated using routinely collected administrative data, accurately predicted the risk of death within 1 year after admission to hospital in a diverse population of patients admitted for nonpsychiatric indications. The HOMR model can be used for risk adjustment in analyses of health administrative data to predict long-term survival among hospital patients.
Acknowledgements
Carl van Walraven is supported by a Clinician Scientist Chair Award from the Department of Medicine, University of Ottawa. Finlay McAlister is supported by a salary award from Alberta Innovates – Health Solutions and holds the Chair in Cardiovascular Outcomes Research at the Mazankowski Alberta Heart Institute, University of Alberta.
Footnotes
Competing interests: None declared.
This article has been peer reviewed.
Contributors: All of the authors contributed to the study design. Carl van Walraven, Jacques Donzé, Steven Hawken and Jeffrey Bakal contributed to the data analysis. All of the authors contributed to the writing of the manuscript, approved the final version to be published and agreed to act as guarantors of the work.
References
- 1.van Walraven C. The Hospital-patient One-year Mortality Risk score accurately predicts long term death risk in hospitalized patients. J Clin Epidemiol 2014;67:1025–34. [DOI] [PubMed] [Google Scholar]
- 2.McGinn TG, Guyatt GH, Wyer PC, et al. Users’ guides to the medical literature. XXII: How to use articles about clinical decision rules. Evidence-Based Medicine Working Group. JAMA 2000;284:79–84. [DOI] [PubMed] [Google Scholar]
- 3.Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med 2000;19:453–73. [DOI] [PubMed] [Google Scholar]
- 4.Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med 1999;130:515–24. [DOI] [PubMed] [Google Scholar]
- 5.Di Bari M, Balzi D, Roberts AT, et al. Prognostic stratification of older persons based on simple administrative data: development and validation of the “Silver Code,” to be used in emergency department triage. J Gerontol A Biol Sci Med Sci 2010;65:159–64. [DOI] [PubMed] [Google Scholar]
- 6.Dramé M, Novella JL, Lang PO, et al. Derivation and validation of a mortality-risk index from a cohort of frail elderly patients hospitalised in medical wards via emergencies: the SAFES study. Eur J Epidemiol 2008;23:783–91. [DOI] [PubMed] [Google Scholar]
- 7.Fischer SM, Gozansky WS, Sauaia A, et al. A practical tool to identify patients who may benefit from a palliative approach: the CARING criteria. J Pain Symptom Manage 2006;31:285–92. [DOI] [PubMed] [Google Scholar]
- 8.Inouye SK, Bogardus ST, Jr, Vitagliano G, et al. Burden of Illness Score for Elderly Persons: risk adjustment incorporating the cumulative impact of diseases, physiologic abnormalities, and functional impairments [published erratum in Med Care 2003; 41:446]. Med Care 2003;41:70–83. [DOI] [PubMed] [Google Scholar]
- 9.Martínez-Velilla N, Cambra-Contin K, Ibanez-Beroiz B. Comorbidity and prognostic indices do not improve the 5-year mortality prediction of components of comprehensive geriatric assessment in hospitalized older patients. BMC Geriatr 2014;14:64. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Knaus WA, Harrell FE, Jr, Lynn J, et al. The SUPPORT prognostic model. Objective estimates of survival for seriously ill hospitalized adults. Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatments. Ann Intern Med 1995;122:191–203. [DOI] [PubMed] [Google Scholar]
- 11.Levine SK, Sachs GA, Jin L, et al. A prognostic model for 1-year mortality in older adults after hospital discharge. Am J Med 2007;120:455–60. [DOI] [PubMed] [Google Scholar]
- 12.Pilotto A, Ferrucci L, Franceschi M, et al. Development and validation of a multidimensional prognostic index for one-year mortality from comprehensive geriatric assessment in hospitalized older patients. Rejuvenation Res 2008;11:151–61. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Pilotto A, Addante F, Franceschi M, et al. Multidimensional prognostic index based on a comprehensive geriatric assessment predicts short-term mortality in older patients with heart failure. Circ Heart Fail 2010;3:14–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Teno JM, Harrell FE, Jr, Knaus W, et al. Prediction of survival for older hospitalized patients: the HELP survival model. Hospitalized Elderly Longitudinal Project. J Am Geriatr Soc 2000;48(Suppl 5): S16–24. [DOI] [PubMed] [Google Scholar]
- 15.Pilotto A, Addante F, Ferrucci L, et al. The multidimensional prognostic index predicts short- and long-term mortality in hospitalized geriatric patients with pneumonia. J Gerontol A Biol Sci Med Sci 2009;64:880–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Pilotto A, Rengo F, Marchionni N, et al. Comparing the prognostic accuracy for all-cause mortality of frailty instruments: a multicentre 1-year follow-up in hospitalized older patients. PLoS ONE 2012;7:e29090. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Pilotto A, Sancarlo D, Aucella F, et al. Addition of the Multidimensional Prognostic Index to the estimated glomerular filtration rate improves prediction of long-term all-cause mortality in older patients with chronic kidney disease. Rejuvenation Res 2012;15:82–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Pilotto A, Sancarlo D, Panza F, et al. The Multidimensional Prognostic Index (MPI), based on a comprehensive geriatric assessment, predicts short- and long-term mortality in hospitalized older patients with dementia. J Alzheimers Dis 2009;18:191–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Walter LC, Brand RJ, Counsell SR, et al. Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization. JAMA 2001;285:2987–94. [DOI] [PubMed] [Google Scholar]
- 20.Yourman LC, Lee SJ, Schonberg MA, et al. Prognostic indices for older adults: A systematic review. JAMA 2012;307:182–92. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Siontis GCM, Tzoulaki I, Castaldi PJ, et al. External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol 2015;68:25–34. [DOI] [PubMed] [Google Scholar]
- 22.Kieszak SM, Flanders WD, Kosinski AS, et al. A comparison of the Charlson comorbidity index derived from medical record data and administrative billing data. J Clin Epidemiol 1999;52:137–42. [DOI] [PubMed] [Google Scholar]
- 23.Quan H, Parsons GA, Ghali WA. Validity of information on comorbidity derived from ICD-9-CCM administrative data. Med Care 2002;40:675–85. [DOI] [PubMed] [Google Scholar]
- 24.Hammill BG, Curtis LH, Fonarow GC, et al. Incremental value of clinical data beyond claims data in predicting 30-day outcomes after heart failure hospitalization. Circ Cardiovasc Qual Outcomes 2011;4:60–7. [DOI] [PubMed] [Google Scholar]
- 25.Walter LC, Brand RJ, Counsell SR, et al. Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization. JAMA 2001;285:2987–94. [DOI] [PubMed] [Google Scholar]
- 26.Studenski S, Perera S, Patel K, et al. Gait speed and survival in older adults. JAMA 2011;305:50–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Clegg A, Young J, Iliffe S, et al. Frailty in elderly people. Lancet 2013;381:752–62. [DOI] [PMC free article] [PubMed] [Google Scholar]