Key Points
Question
Do trends in 30-day mortality rates among veterans hospitalized with heart failure (HF) and pneumonia differ according to whether claims-based or clinical variables are used in risk-adjustment models?
Findings
In this observational time-trend study that included 146 924 hospitalizations for HF and 131 325 for pneumonia from 2009 to 2015, the estimated decline in 30-day mortality rates per quarter was substantially smaller when using clinical variables for risk adjustment (HF, −0.017%; pneumonia, −0.026%) than when using claims-based variables (HF, −0.051%; pneumonia, −0.084%).
Meaning
Compared with clinical risk adjustment, claims-based adjustment is likely to overestimate temporal declines in mortality for veterans hospitalized with HF and pneumonia.
This observational time-trend study compares trends in 30-day mortality rates among veterans hospitalized with heart failure and pneumonia when claims-based vs clinical variables are used in risk-adjustment models.
Abstract
Importance
Prior studies have reported declines in mortality for patients admitted to Veterans Health Administration (VA) and non-VA hospitals using claims-based risk adjustment. These apparent mortality reductions may be influenced by changes in coding practices.
Objective
To compare trends in the VA for 30-day mortality following hospitalization for heart failure (HF) and pneumonia using claims-based and clinical risk-adjustment models.
Design, Setting, and Participants
This observational time-trend study analyzed admissions to a VA Medical Center with a principal diagnosis of HF, pneumonia, or sepsis/respiratory failure (RF) with a secondary diagnosis of pneumonia. Exclusion criteria included having less than 12 months of VA enrollment, being discharged alive within 24 hours, leaving against medical advice, and hospice utilization.
Exposures
Admission to a VA hospital from January 2009 through September 2015.
Main Outcomes and Measures
The primary outcome was 30-day, all-cause mortality. All models included age and sex. Claims-based covariates included 22 (30) comorbidities for HF (pneumonia). Clinical covariates included vital signs, laboratory values, and ejection fraction.
Results
Among the 146 924 HF admissions, the mean (SD) age was 71.6 (11.4) years and 144 502 (98.4%) were men; among the 131 325 admissions for pneumonia, the mean (SD) age was 70.8 (12.3) years and 127 491 (97.1%) were men. Unadjusted 30-day mortality rates were 6.45% (HF) and 11.22% (pneumonia). Claims-based models showed an increased predicted risk of 30-day mortality over time (0.019 percentage points per quarter for HF [95% CI, 0.015 to 0.023]; 0.053 percentage points per quarter for pneumonia [95% CI, 0.043 to 0.063]). Clinical models showed declines or no change in predicted risk (−0.014 percentage points per quarter for HF [95% CI, −0.020 to −0.008]; −0.004 percentage points per quarter for pneumonia [95% CI, −0.017 to 0.008]). Claims-based risk adjustment yielded declines in 30-day mortality of 0.051 percentage points per quarter for HF (95% CI, −0.074 to −0.027) and 0.084 percentage points per quarter for pneumonia (95% CI, −0.111 to −0.056). Models adjusting for clinical covariates attenuated or eliminated these changes for HF (−0.017 percentage points per quarter; 95% CI, −0.039 to 0.006) and for pneumonia (−0.026 percentage points per quarter; 95% CI, −0.052 to 0.001). Compared with the claims-based models, the clinical models for HF and pneumonia more accurately differentiated between patients who died after 30 days and those who did not.
Conclusions and Relevance
Among HF and pneumonia hospitalizations, adjusting for clinical covariates attenuated declines in mortality rates identified using claims-based models. Assessments of temporal trends in 30-day mortality using claims-based risk adjustment should be interpreted with caution.
Introduction
Over the past decade, health care systems in the United States have invested substantial resources to measure and reduce mortality rates for hospitalized patients with common and costly medical conditions.1 For example, since 2008 the Centers for Medicare & Medicaid Services (CMS) have publicly reported 30-day mortality rates for admissions for acute myocardial infarction (AMI), heart failure (HF), and pneumonia to virtually all acute-care hospitals in the United States.2 The Centers for Medicare & Medicaid Services include mortality measures in their Hospital Value-based Payment Program, which financially rewards these hospitals for better quality of care. In the Veterans Health Administration (VA), the largest integrated delivery system in the United States, hospital mortality rates (calculated using the CMS approach) inform the Strategic Analytics for Improvement and Learning (SAIL) composite quality score. The VA uses SAIL to monitor and rank performance across VA Medical Centers (VAMCs) on a national level.3
Previous studies have identified significant improvements in hospital and population-level outcomes for AMI, HF, and pneumonia among Medicare beneficiaries and in the VA Healthcare System.2,4,5,6,7,8,9,10 To account for differences in the characteristics of patients across hospitals and/or over time, these studies have adjusted for the presence of comorbid conditions identified in claims or other administrative data. A key concern in the use of claims-based models is that they may be influenced by secular changes in hospitals’ diagnostic coding practices, particularly because comorbid diagnoses often have flexible clinical criteria. Thus, apparent reductions in hospital- and population-level mortality rates may reflect changes in coding practices rather than actual improvements in outcomes.10,11 Prior work has found that the number of comorbidities among hospitalized patients has increased over time.5,6,9,12
Risk adjustment using clinical data (eg, vital signs, laboratory data) provides an alternative to claims-based risk-adjustment. Clinical models often outperform claims-based models in predicting hospital mortality at the individual patient level and have greater face validity among clinicians.13,14,15 Furthermore, whereas claims-based models assess the burden of comorbid conditions, clinical data can provide a measure of the severity of the primary illness for which a patient is admitted.16 However, it is unknown whether the use of clinical models changes VA system–level trends in hospital mortality over time. In this study, we leverage the availability of clinical data from the VA Healthcare System to examine trends in 30-day mortality following hospitalization for HF or pneumonia from 2009 to 2015 using 2 alternative risk-adjustment models: (1) an approach using claims-based comorbidities specified by the CMS model and (2) a model using objective clinical covariates.
Methods
Data Sources/Study Population
We retrieved VA administrative claims data and clinical records from the VA’s Corporate Data Warehouse (CDW). The data included VA hospital admissions, vital signs, laboratory values, outpatient claims, vital status, and fee-basis files of VA-financed health services that occur in non-VA settings.
Analyses were restricted to veterans with at least 1 inpatient admission to a VAMC with a principal discharge diagnosis of HF, pneumonia, and sepsis or respiratory failure (RF) with a secondary diagnosis of pneumonia between January 2009 and September 2015. Although patients with a principal diagnosis of sepsis/RF and secondary diagnosis of pneumonia were excluded from calculations for pneumonia mortality during the years of the study, CMS changed the specifications for the pneumonia mortality measure to include patients with a primary diagnosis of sepsis in 2016. Therefore, we defined the pneumonia cohort by including patients with a primary diagnosis of sepsis or RF and a secondary diagnosis of pneumonia (n = 27 272), consistent with a prior study.10 The specific International Classification of Diseases, Ninth Revision (ICD-9) codes used to establish these diagnoses are in eTable 1 in the Supplement. Consistent with CMS and VA measure specifications, we excluded veterans with less than 12 months of continuous VA enrollment in the year prior to the selected admission, discharged alive within 24 hours of an HF or pneumonia admission, who left against medical advice, and with hospice utilization 1 year prior to admission. Because the unit of analysis was admission, we included more than 1 admission for the same veteran. In sensitivity analyses, we only considered a veteran’s first admission, as well as a random admission within the study period.17,18 The eligible study population consisted of 146 924 admissions for HF and 131 325 admissions for pneumonia across 132 VAMCs (eFigures 1 and 2 in the Supplement).
The Providence VA's institutional review board approved the study protocol and waived the need for informed consent because the research because the research involved no more than minimal tangible or intangible risk to study participants.
Variables
The outcome variable was an indicator for death from any cause in the 30 days after hospital admission that was derived from the VA vital status file, which is comparable to the National Death Index with respect to completeness and accuracy.19 The outcome was attributed to the VAMC to which the veteran was initially admitted, regardless of postadmission transfers to another VAMC or a non-VA hospital.
The primary independent variable was the quarter in which an admission occurred. Each admission corresponded to 1 of the 27 quarters between January 2009 and September 2015.
The adjustment variables were in 3 categories: demographic (age and sex), claims-based, and clinical. The full list of claims-based and clinical variables considered for HF and pneumonia are included in the first column of Table 1.
Table 1. Characteristics of Veterans Hospitalized With Heart Failure and Pneumonia in the First and Last Years of the Studya.
| Characteristic | Heart Failure | Pneumonia | ||
|---|---|---|---|---|
| First Year (n = 20 228)b | Last Year (n = 25 003)c | First Year (n = 18 049)b | Last Year (n = 20 555)c | |
| Age, mean (SD), y | 71.3 (11.5) | 71.9 (11.2) | 69.9 (12.6) | 71.6 (12.0) |
| Male, No. (%) | 19 944 (98.6) | 24 566 (98.3) | 17 546 (97.2) | 19 872 (96.7) |
| Primary diagnosis of sepsis/RF, No. (%) | NA | NA | 1958 (10.8) | 6828 (33.2) |
| Claims-based comorbid conditions, No. (%) | ||||
| Percutaneous transluminal coronary angioplasty | 1718 (8.5) | 3105 (12.4) | 737 (4.1) | 1160 (5.6) |
| Coronary artery bypass graft | 4454 (22.0) | 4826 (19.3) | 1471 (8.2) | 1492 (7.3) |
| Congestive heart failure | 15 986 (79.0) | 20 353 (81.4) | 4951 (27.4) | 6266 (30.5) |
| Acute myocardial infarction | 1552 (7.7) | 2096 (8.4) | 496 (2.7) | 609 (3.0) |
| Ischemic heart disease | 2364 (11.7) | 3622 (14.5) | 749 (4.1) | 1103 (5.4) |
| Chronic atherosclerosis | 13 565 (67.1) | 16 340 (65.4) | 6522 (36.1) | 7506 (36.5) |
| Cardiorespiratory failure and shock | 2948 (14.6) | 5832 (23.3) | 2628 (14.6) | 4752 (23.1) |
| Valvular/rheumatic heart disease | 4612 (22.8) | 5892 (23.6) | NA | NA |
| Hypertension | 17 813 (88.1) | 22 463 (89.8) | 13 302 (73.7) | 15 623 (76.0) |
| Stroke | 2044 (10.1) | 2703 (10.8) | 1622 (9.0) | 1893 (9.2) |
| Cerebrovascular | NA | NA | 1898 (10.5) | 2290 (11.1) |
| Renal failure | 9683 (47.9) | 13 885 (55.5) | 4381 (24.3) | 6678 (32.5) |
| Chronic obstructive pulmonary disease | 8764 (43.3) | 11 363 (45.4) | 8790 (48.7) | 10 550 (51.3) |
| Pneumonia | 4379 (21.6) | 5911 (23.6) | 6536 (36.2) | 8306 (40.4) |
| Malnutrition | NA | NA | 837 (4.6) | 1224 (6.0) |
| Diabetes mellitus with complications | 11 978 (59.2) | 15 529 (62.1) | NA | NA |
| Dementia/senility | 2055 (10.2) | 2943 (11.8) | 2470 (13.7) | 3301 (16.1) |
| Functional disability | 1715 (8.5) | 2251 (9.0) | 1683 (9.3) | 1958 (9.5) |
| Peripheral vascular disease | 5986 (29.6) | 7716 (30.9) | 3864 (21.4) | 5028 (24.5) |
| Cancer | 666 (3.3) | 1039 (4.2) | 1967 (10.9) | 2547 (12.4) |
| Trauma | 5498 (27.2) | 7586 (30.3) | 4992 (27.7) | 6042 (29.4) |
| Major psychiatric disease | 2662 (13.2) | 3900 (15.6) | 3298 (18.3) | 4151 (20.2) |
| Chronic liver disease | 1122 (5.5) | 2057 (8.2) | 1035 (5.7) | 1458 (7.1) |
| Hematological disorder | NA | NA | 498 (2.8) | 471 (2.3) |
| Iron deficiency | NA | NA | 6483 (35.9) | 7900 (38.4) |
| Depression | NA | NA | 4312 (23.9) | 5290 (25.7) |
| Parkinson disease | NA | NA | 444 (2.5) | 546 (2.7) |
| Seizure disorder | NA | NA | 882 (4.9) | 992 (4.8) |
| Lung fibrosis | NA | NA | 1374 (7.6) | 1696 (8.3) |
| Asthma | NA | NA | 995 (5.5) | 1229 (6.0) |
| Vertebral fracture | NA | NA | 205 (1.1) | 391 (1.9) |
| Surgical admission | 735 (3.6) | 1246 (5.0) | 738 (4.1) | 936 (4.6) |
| No. of claims-based comorbidities, mean (SD) | 6.0 (2.8) | 6.5 (3.0) | 5.0 (3.1) | 5.5 (3.3) |
| Clinical variables, mean (SD) | ||||
| Blood pressure, mm Hgd | ||||
| Systolic | 126.7 (31.0) | 130.3 (29.1) | 126.6 (23.3) | 128.6 (22.5) |
| Diastolic | 83.0 (25.2) | 81.4 (21.8) | 70.6 (15.4) | 71.4 (13.7) |
| Heart rated | 80.1 (18.4) | 81.5 (18.8) | 90.2 (19.1) | 89.9 (18.4) |
| Respiratory rated | 26.0 (33.7) | 22.3 (24.8) | 21.8 (15.2) | 20.2 (7.4) |
| Oxygen saturation, %d | 93.2 (15.0) | 94.6 (10.7) | 94.3 (7.1) | 94.9 (4.0) |
| Body mass indexd,e | 30.9 (8.0) | 31.5 (7.9) | 27.0 (7.0) | 27.5 (7.1) |
| Sodium, mEq/Ld | 138.4 (4.2) | 138.3 (4.2) | 136.6 (4.8) | 136.6 (4.8) |
| Potassium, mEq/Ld | 4.2 (0.6) | 4.2 (0.6) | 4.1 (0.6) | 4.1 (0.6) |
| Blood urea nitrogen, mg/dLd | 30.4 (17.2) | 29.8 (16.9) | 23.8 (14.6) | 23.9 (14.5) |
| Creatinine, mg/dLd | 1.7 (0.9) | 1.7 (0.9) | 1.4 (1.1) | 1.5 (1.3) |
| Hematocrit, %d | 36.4 (6.0) | 36.7 (6.3) | 36.4 (6.1) | 36.9 (6.4) |
| Serum glucose, mg/dLd | NA | NA | 136.9 (53.4) | 139.8 (54.9) |
| Inpatient admission 30 d prior, No. (%) | NA | NA | 3342 (18.5) | 4288 (20.9) |
| B-type natriuretic peptide, pg/mLd | 1346.7 (1446.4) | 1359.8 (1592.0) | NA | NA |
| Left ventricular ejection fraction, %d | 37.6 (15.6) | 38.4 (15.4) | NA | NA |
Abbreviations: NA, not applicable for a given clinical condition; RF, respiratory failure.
Mean (SD) reported for continuous variables; count (rate) reported for categorical variables.
First year, January 2009 to December 2009.
Last year, October 2014 to September 2015.
Summary statistics reported among nonmissing values.
Calculated as weight in kilograms divided by height in meters squared.
The claims-based variables were obtained using VA administrative data from the index admission and outpatient and inpatient encounter data for the 12 months preceding admission. The claims-based variables were identical to those included in CMS risk-adjustment models for hospital profiling, which includes different comorbidities for HF and pneumonia. We did not include comorbid conditions from Medicare claims, consistent with VA’s current methodology and a prior study reporting that adding Medicare diagnostic claims for dual users (of VA and Medicare) had little impact on hospital-level risk-standardized mortality rates.19 Furthermore, we excluded such claims to avoid the influence of coding practices in Medicare affecting measured comorbidities in the study population.
The clinical variables were extracted from medical record data in CDW and chosen based on prior literature, clinical face validity, availability, and feasibility of extraction.20,21 They included systolic and diastolic blood pressure, heart rate, respiratory rate, oxygen saturation, body mass index (BMI, calculated as weight in kilograms divided by height in meters squared), sodium, potassium, blood urea nitrogen, creatinine, hematocrit, serum glucose, an indicator for inpatient admission 30 days prior to the index admission, B-type natriuretic peptide, and left ventricular ejection fraction. Natural language processing methods were used to extract ejection fraction for patients with HF from text sources in the VA's computerized personal record system.22 Each clinical covariate was derived during the period of 2 days prior to admission until 1 day after admission using an approach identifying the value closest to the admission time for those with multiple values. The percentage of missing values for each of the clinical variables is presented in eTable 2 in the Supplement; none have a frequency of missing exceeding 10% except for B-type natriuretic peptide (12.7%), BMI (15.0%), and ejection fraction (33.4%) in the HF cohort. For missing data, we used multiple imputation methods to generate 5 completed data sets for analysis.23
We additionally reported the racial/ethnic composition of the HF and pneumonia admissions, as well as the average area deprivation index (ADI), a composite measure of neighborhood disadvantage, but did not include these demographic variables as risk-adjusters, which is consistent with CMS methodology.24,25,26
Statistical Analyses
We computed unadjusted mortality rates for each quarter by condition and conducted time series analyses on unadjusted mortality rates using a Bayesian structural time series model to account for seasonal patterns in mortality prior to examining the trend.27 To account for potential temporal differences in the case-mix of veterans that comprised each of the 27 quarters, we estimated directly standardized mortality rates using 3 sets of adjustment variables (1) age and sex, (2) the claims-based model, which includes age, sex, and the set of claims-based variables used by CMS/VA for hospital profiling, and (3) a model based on age, sex, and the clinical variables.
For all 3 sets of adjustment variables, we first fitted a patient-level logistic regression model that included indicator variables for VAMCs specifying 30-day mortality as the outcome and admission quarter and the relevant set of adjustor variables as predictors.2 We recorded the area under the receiver operating characteristic curve (AUC) statistic and the Hosmer-Lemeshow goodness of fit statistic. The AUC statistic provides a measure of how well the model distinguishes between patients who experienced the outcome of interest vs those who did not, with higher values of the AUC statistic indicating better performance of the model. The DeLong Method was used to determine if differences in the AUCs for claims-based and clinical models were statistically significant. The Hosmer-Lemeshow goodness of fit statistic provides a measure of how well predictions from the claims-based and clinical models match observed mortality. Second, using each model's estimates, the probability of death for each individual in the reference population was estimated using all of their observed values for the relevant adjustor variables fixed at the first quarter. The reference population for HF (pneumonia) was the full set of all 146 924 (131 325) admissions. Finally, the standardized mortality rate for this set of variables in this specific quarter was determined by averaging these probabilities of death. This process was repeated for the other 26 quarters.
These 27 standardized mortality rates can be directly compared across time because they were estimated using a common reference population. We compared quarterly adjusted mortality rates across time by regressing them on time. The slope represents the estimated change in mortality rates per quarter. For models including clinical data, we performed statistical analyses on the 5 imputed data sets and derived combined estimates using multiple imputation rules.23
We also computed the estimated risk per quarter using the 3 sets of adjustment variables. For this, we fit the same patient-level logistic regression model (excluding admission quarter as a predictor) and averaged the estimated probability of death for each quarter.
Statistical analyses were performed using SAS statistical software (version 9.4; SAS Institute, Inc) and R statistical software (version 3.4.3, R Foundation).
Results
Characteristics of the Study Population
Among the 146 924 HF admissions, the mean (SD) age was 71.6 (11.4) years and 144 502 (98.4%) were men. Comparing admissions for HF in the first and last years of the study (Table 1), 20 of the 22 individual claims-based comorbid conditions increased in prevalence between 2009 and 2015. The mean number of claims-based comorbidities increased significantly from 6.0 to 6.5. The largest increases in frequency were for cardiorespiratory failure and shock (8.7 percentage points) and renal failure (7.6 percentage points). In contrast, clinical measures for HF appeared to improve over time: systolic blood pressure, oxygen saturation, hematocrit, and ejection fraction increased, whereas respiratory rates decreased. Moreover, there was no evidence that the renal function of admitted patients, as measured by serum creatinine and blood urea nitrogen, declined over time.
Among the 131 325 admissions for pneumonia, the mean (SD) age was 70.8 (12.3) years and 127 491 (97.1%) were men. Table 1 shows that the proportion of the pneumonia cohort with a primary diagnosis of sepsis/RF and a secondary diagnosis of pneumonia increased from 1958 (10.8%) in the first year to 6828 (33.2%) in the last year (eFigure 3 in the Supplement). Between the first and last years, the prevalence for 27 of 30 comorbid conditions increased, with the largest increases observed for cardiorespiratory failure and shock (8.5 percentage points) and renal failure (8.2 percentage points). The average number of claims-based comorbidities rose from 5.0 to 5.5 in the pneumonia cohort. Signs of lower clinical severity increased over time—systolic blood pressure, oxygen saturation, and hematocrit increased, whereas respiratory rate declined. Serum creatinine and blood urea nitrogen levels remained stable. We observed similar findings in stratified analyses for the pneumonia cohort by primary diagnosis (pneumonia vs sepsis/RF) (eTables 3 and 4 in the Supplement).
The supplementary material presents standardized differences in patient characteristics comparing the first and last years (eTables 5 and 6 in the Supplement) as well as the fraction of patients with abnormal clinical values (eTables 7 and 8 in the Supplement). eTable 9 in the Supplement shows that the racial, ethnic, and socioeconomic characteristics of patients hospitalized for HF and pneumonia were stable over time.
Mortality Trends
The overall 30-day mortality rates were 6.45% for HF and 11.22% for pneumonia. From the first year to the last year, the unadjusted 30-day mortality rate decreased from 6.64% to 6.29% for HF and from 11.09% to 10.26% for pneumonia (eFigure 4 in the Supplement). Mortality for admissions with a primary diagnosis of sepsis/RF and a secondary diagnosis of pneumonia decreased from 33.15% in the first year to 18.29% in the last (eFigure 5 in the Supplement). Bayesian structural time series analyses confirmed that unadjusted mortality rates for all 3 conditions declined (eFigure 6 in the Supplement).
As shown in Table 2, adjusted mortality rates following hospitalization for HF declined from 6.90% in the first year to 6.01% in the last year in the claims-based risk-adjustment model and from 6.48% to 6.39% in the clinical model. For the pneumonia cohort, the claims-based risk-adjustment model showed a decline in adjusted mortality rates from 11.85% to 9.67%; the clinical risk-adjustment model showed a smaller decline from 11.19% to 10.46%. Figure 1 graphs quarterly trends in standardized 30-day mortality rates. The adjusted quarterly change in mortality following hospitalization for HF was −0.051 percentage points (95% CI, −0.074 to −0.027) in the claims-based model and −0.017 percentage points (95% CI, −0.039 to 0.006) in the clinical model. The adjusted quarterly change in the 30-day mortality rate following hospitalization for pneumonia was −0.084 percentage points (95% CI, −0.111 to −0.056) in the claims-based model and −0.026 percentage points (95% CI, −0.052 to 0.001) in the clinical model. In stratified analyses of the pneumonia cohort by primary diagnosis, clinical models attenuated estimated declines in risk-adjusted mortality compared with claims-based models (eTable 10, eFigure 7 in the Supplement).
Table 2. Changes in 30-Day Mortality Among Veterans Admitted With Heart Failure and Pneumoniaa.
| Adjustment Variables | 30-Day Mortality, % | ||
|---|---|---|---|
| Measure | Heart Failure | Pneumonia | |
| Adjusted for age and sex | First yearb | 6.71 | 11.48 |
| Last year | 6.19 | 10.02 | |
| Quarterly change, % | −0.035 (−0.058 to −0.012) | −0.053 (−0.082 to −0.024) | |
| AUC | 0.657 (0.651 to 0.662) | 0.638 (0.633 to 0.642) | |
| Adjusted for age, sex, and claims-based comorbid conditions | First year | 6.90 | 11.85 |
| Last year | 6.01 | 9.67 | |
| Quarterly change, % | −0.051 (−0.074 to −0.027) | −0.084 (−0.111 to −0.056) | |
| AUC | 0.689 (0.684 to 0.694) | 0.709 (0.705 to 0.713) | |
| Adjusted for age, sex, and clinical covariates | First year | 6.48 | 11.19 |
| Last year | 6.39 | 10.46 | |
| Quarterly change, % | −0.017 (−0.039 to 0.006) | −0.026 (−0.052 to 0.001) | |
| AUC | 0.777 (0.772 to 0.781) | 0.750 (0.745 to 0.754) | |
Abbreviation: AUC, area under the receiver operating characteristic curve.
Point estimates (95% CIs) for quarterly changes were calculated using the slope of the linear regression between admission quarter and adjusted rates. Point estimates for clinical models represent average slopes for the linear regression between admission quarter and the adjusted rates among the imputed data sets, whereas standard error estimates used for 95% CIs are the combination of within- and between-imputation variance. We report the AUC (95% CIs) for the first imputed data set for the model with clinical covariates.
First year, January 2009 to December 2009; Last year, October 2014 to September 2015.
Figure 1. Risk-Standardized 30-Day Mortality Rates Among Veterans Hospitalized With Heart Failure and Pneumonia From Claims-Based and Clinical Risk-Adjustment Models.
Each point corresponds to the risk-standardized 30-day mortality rate in each quarter between January to March 2009 and July to September 2015 for the reference population of interest. The claims-based model presented for pneumonia did not include an indicator for primary diagnosis.
Claims-based models had AUC values of 0.689 (HF) and 0.709 (pneumonia); clinical models had AUCs of 0.777 (HF) and 0.750 (pneumonia) (Table 2). Regardless of whether the indicator variables for VAMC were included, adjustment models that included clinical variables had higher AUC statistics than those based on claims-based variables for each condition, demonstrating that clinical models had an improved ability to distinguish between patients who died within 30 days and those who did not (eTable 11 in the Supplement). Hosmer-Lemeshow goodness-of-fit statistics showed that predictions from clinical models more closely matched observed mortality status than claims-based models.
Sensitivity analyses for multiple admissions showed similar results (eTables 12-13 in the Supplement).
Predicted Mortality Risk
Claims-based models showed an increase in the predicted risk of 30-day mortality over time (Figure 2). For HF, the estimated increase in risk per quarter was 0.019 percentage points (95% CI, 0.015-0.023); for pneumonia, the estimated increase in risk per quarter was 0.053 percentage points (95% CI, 0.043-0.063).
Figure 2. Quarterly Trends in Predicted Risk of 30-Day Mortality, as Derived From Clinical and Claims-Based Risk-Adjustment Models.
Each point corresponds to the predicted risk of 30-day mortality in each quarter between January to March 2009 and July to September 2015. The claims-based model presented for pneumonia did not include an indicator for primary diagnosis.
In contrast, models adjusting for clinical variables showed a decline or no significant change in risk over time (Figure 2). The estimated decrease in risk per quarter was 0.014 percentage points for HF (95% CI, −0.020 to −0.008) and 0.004 percentage points for pneumonia (95% CI, −0.017 to 0.008). Similar patterns were identified for the predicted mortality risks for pneumonia (excluding sepsis/RF) and sepsis/RF (eFigure 8 in the Supplement).
Discussion
In this study of temporal trends in 30-day mortality for HF and pneumonia admissions between January 2009 and September 2015, we observed substantial declines in 30-day mortality in the VA using a claims-based risk-adjustment approach. Risk adjustment using clinical variables attenuated or eliminated these apparent decreases in hospital mortality. Compared with claims-based models, those that adjust for clinical variables more accurately distinguished between patients who died within 30 days and those who did not and generated predictions of mortality that more closely matched observed mortality rates.
This study identified 2 mechanisms by which claims-based risk adjustment may overstate temporal declines in hospital mortality. First, hospitals appear to have increased the number and type of coded comorbid conditions over time. For instance, we found that the proportion of patients with a coded diagnosis of renal failure or shock/RF increased temporally across all cohorts, but clinical data measured at or near the time of admission showed no evidence of worsening renal function, decreased oxygen saturation, or increased respiratory rates over time. Whereas the predicted risk of mortality increased over time in claims-based models, clinical variable-based models indicated that the predicted risk of mortality declined or remained constant. A time trend analysis by Heidenreich et al9 for HF in the VA also identified increases in comorbid disease coding and improvements in most laboratory values between 2002 and 2006. The present study preceded the implementation of ICD-10; the adoption of this expanded set of diagnostic codes may further increase opportunities to report comorbid conditions.
Second, hospitals may assign high-risk patients to alternative principal diagnoses, which would remove such patients from calculations of mortality. For instance, we found that the number of patients with coded sepsis/RF and secondary diagnoses of pneumonia more than tripled between the first and last year of the study. Furthermore, the predicted risk of mortality among these patients (as derived from the clinical model) declined markedly over time. Our findings are broadly consistent with analyses of Medicare data that have reported increases in the number of patients with a principal diagnostic code of sepsis/RF and raised concerns about hospitals’ increased labeling of pneumonia patients with a principal diagnosis of sepsis or RF.10,28
Our study focused on hospitalized veterans and was strengthened by the availability of clinical data in the VA’s health information systems. However, it is likely that the findings of this study extend beyond the VA. Studies of hospitalized Medicare beneficiaries have also identified substantial increases in coded comorbidities over time.4,29,30 Most private-sector hospitals in the United States are subject to payment penalties or increased reimbursements on the basis of risk-adjusted outcomes. Thus, private-sector hospitals face strong incentives to comprehensively document comorbid conditions under Medicare’s value-based payments. Our findings therefore suggest that assessments of temporal trends in hospital mortality using claims-based risk adjustment should be interpreted with caution within and beyond the VA. In addition, the higher performance of clinical risk-adjustment models, as measured by AUC and goodness-of-fit statistics, compared with those using claims alone implies that the VA should continue allocating resources toward the collection of clinical data, even as it transitions to an alternative electronic medical record. Other health care systems and payers (including Medicare) should consider collecting and reporting clinical data in a standardized format and using these data for risk adjustment.
Limitations
This study has limitations. Risk adjustment using vital signs and laboratory values may not capture important measures of severity, such as altered mental status, substance abuse, or frailty. Some of the clinical variables in this study were missing. However, most of these had small percentages of missing data, and we used multiple imputation to account for their absence. Our results may not generalize to other conditions beyond HF and pneumonia. Nevertheless, both CMS and the VA have prioritized these conditions in public reporting of hospital mortality rates. We did not examine the effect of including Medicare claims on coded comorbidities among the subset of Veterans eligible for Medicare. Given prior studies, the inclusion of this data would likely produce further increases in the number and type of claims-based conditions in the study population over time.5,6,9 Finally, other observers have noted that the VA likely undercodes comorbid conditions compared with Medicare-financed hospitals; if true, then the discordance between claims-based and clinical models is likely even greater in non-VA hospitals.31
Conclusions
Using the CMS claims-based risk-adjustment methodology, 30-day mortality rates for veterans hospitalized with HF and pneumonia appeared to substantially decline between 2009 and 2015. However, adjustment for objective clinical covariates nearly eliminated these apparent improvements in patient outcomes. Assessments of temporal trends in mortality using claims-based risk adjustment should be interpreted with caution, and health systems should invest in the collection of more objective clinical information for risk adjustment.
eFigure 1. Exclusion Flowchart for the Study Population
eFigure 2. Number of Admissions per Quarter for the Study Population
eFigure 3. Percent of Admissions per Quarter with a Primary Diagnosis of Sepsis/RF [with Pneumonia as Secondary Diagnosis] in Extended Pneumonia Cohort
eFigure 4. Unadjusted 30-day Mortality Rates for Veterans Hospitalized with Heart Failure and Pneumonia
eFigure 5. Unadjusted 30-day Mortality Rates for Veterans Hospitalized with Heart Failure, Pneumonia, Pneumonia [excluding Sepsis/RF] and Sepsis/RF [with Pneumonia as Secondary Diagnosis]
eFigure 6. Bayesian Structural Time Series Analyses
eFigure 7. Risk-Standardized 30-Day Mortality Rates Among Veterans Hospitalized with Pneumonia [excluding Sepsis/RF] and Sepsis/RF [with Pneumonia as Secondary Diagnosis] from Claims-Based and Clinical Risk-Adjustment Models
eFigure 8. Quarterly Trends in Predicted Risk of 30-day Mortality, as Derived from Clinical and Claims-Based Risk-adjustment Models for Pneumonia [excluding Sepsis/RF] and Sepsis/RF [with Pneumonia as Secondary Diagnosis]
eTable 1. ICD-9 Codes for Primary Diagnoses
eTable 2. Summary of Missing Data for Adjustment/Summary Variables
eTable 3. Characteristics of Veterans Hospitalized with Pneumonia [excluding primary diagnosis of Sepsis/RF] Between the First and Last Years of the Study
eTable 4. Characteristics of Veterans Hospitalized with Sepsis/RF [with Pneumonia as Secondary Diagnosis] Between the First and Last Years of the Study
eTable 5. Characteristics of Veterans Hospitalized with Heart Failure Between the First and Last Years of the Study
eTable 6. Characteristics of Veterans Hospitalized with Pneumonia Between the First and Last Years of the Study
eTable 7. Comparison of Clinical Characteristics Between First and Last Years of Study for Patients Admitted with Heart Failure
eTable 8. Comparison of Clinical Characteristics Between First and Last Years of Study for Patients Admitted with Pneumonia
eTable 9. Comparison of Sociodemographic Characteristics Between First and Last Years for Patients Admitted with Heart Failure or Pneumonia
eTable 10. Changes in 30-day Mortality among Veterans Admitted with Pneumonia [excluding Sepsis/RF] and Sepsis/RF [with Pneumonia as Secondary Diagnosis]
eTable 11. Analyses for Area under the ROC curve (AUC)/Goodness of Fit for Claims-based and Clinical Risk-adjustment Models
eTable 12. Sensitivity Analyses for Multiple Admissions – First Admission
eTable 13. Sensitivity Analyses for Multiple Admissions – Random Admission
References
- 1.Kizer KW, Dudley RA. Extreme makeover: Transformation of the veterans health care system. Annu Rev Public Health. 2009;30(1):313-339. doi: 10.1146/annurev.publhealth.29.020907.090940 [DOI] [PubMed] [Google Scholar]
- 2.Joynt KE, Orav EJ, Zheng J, Jha AK. Public Reporting of Mortality Rates for Hospitalized Medicare Patients and Trends in Mortality for Reported Conditions. Ann Intern Med. 2016;165(3):153-160. doi: 10.7326/M15-1462 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.US Department of Veterans Affairs Strategic Analytics for Improvement and Learning (SAIL). https://www.va.gov/QUALITYOFCARE/measure-up/Strategic_Analytics_for_Improvement_and_Learning_SAIL.asp. Accessed November 16, 2019.
- 4.Lee JS, Nsa W, Hausmann LRM, et al. Quality of care for elderly patients hospitalized for pneumonia in the United States, 2006 to 2010. JAMA Intern Med. 2014;174(11):1806-1814. doi: 10.1001/jamainternmed.2014.4501 [DOI] [PubMed] [Google Scholar]
- 5.Chen J, Normand S-LT, Wang Y, Krumholz HM. National and regional trends in heart failure hospitalization and mortality rates for Medicare beneficiaries, 1998-2008. JAMA. 2011;306(15):1669-1678. doi: 10.1001/jama.2011.1474 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Kaboli PJ, Go JT, Hockenberry J, et al. Associations between reduced hospital length of stay and 30-day readmission rate and mortality: 14-year experience in 129 Veterans Affairs hospitals. Ann Intern Med. 2012;157(12):837-845. doi: 10.7326/0003-4819-157-12-201212180-00003 [DOI] [PubMed] [Google Scholar]
- 7.Trivedi AN, Grebla RC. Quality and equity of care in the veterans affairs health-care system and in medicare advantage health plans. Med Care. 2011;49(6):560-568. doi: 10.1097/MLR.0b013e31820fb0f6 [DOI] [PubMed] [Google Scholar]
- 8.Shah RU, Tsai V, Klein L, Heidenreich PA. Characteristics and outcomes of very elderly patients after first hospitalization for heart failure. Circ Heart Fail. 2011;4(3):301-307. doi: 10.1161/CIRCHEARTFAILURE.110.959114 [DOI] [PubMed] [Google Scholar]
- 9.Heidenreich PA, Sahay A, Kapoor JR, Pham MX, Massie B. Divergent trends in survival and readmission following a hospitalization for heart failure in the Veterans Affairs health care system 2002 to 2006. J Am Coll Cardiol. 2010;56(5):362-368. doi: 10.1016/j.jacc.2010.02.053 [DOI] [PubMed] [Google Scholar]
- 10.Lindenauer PK, Lagu T, Shieh MS, Pekow PS, Rothberg MB. Association of diagnostic coding with trends in hospitalizations and mortality of patients with pneumonia, 2003-2009. JAMA. 2012;307(13):1405-1413. doi: 10.1001/jama.2012.384 [DOI] [PubMed] [Google Scholar]
- 11.Ibrahim AM, Dimick JB, Sinha SS, Hollingsworth JM, Nuliyalu U, Ryan AM. Association of coded severity with readmission reduction after the Hospital Readmissions Reduction Program. JAMA Intern Med. 2018;178(2):290-292. doi: 10.1001/jamainternmed.2017.6148 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Sukul D, Hoffman GJ, Nuliyalu U, et al. Association between Medicare policy reforms and changes in hospitalized Medicare beneficiaries’ severity of illness. JAMA Netw Open. 2019;2(5):e193290. doi: 10.1001/jamanetworkopen.2019.3290 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Shahian DM, Silverstein T, Lovett AF, Wolf RE, Normand S-LT. Comparison of clinical and administrative data sources for hospital coronary artery bypass graft surgery report cards. Circulation. 2007;115(12):1518-1527. doi: 10.1161/CIRCULATIONAHA.106.633008 [DOI] [PubMed] [Google Scholar]
- 14.Krumholz HM, Wang Y, Mattera JA, et al. An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with heart failure. Circulation. 2006;113(13):1693-1701. doi: 10.1161/CIRCULATIONAHA.105.611194 [DOI] [PubMed] [Google Scholar]
- 15.Fonarow GC, Pan W, Saver JL, et al. Comparison of 30-day mortality models for profiling hospital performance in acute ischemic stroke with vs without adjustment for stroke severity. JAMA. 2012;308(3):257-264. doi: 10.1001/jama.2012.7870 [DOI] [PubMed] [Google Scholar]
- 16.Iezzoni L. Risk Adjustment for Measuring Healthcare Outcomes, Fourth Edition. Health Administration Press; Chicago, IL. 2012. [Google Scholar]
- 17.Volpp KG, Rosen AK, Rosenbaum PR, et al. Mortality among patients in VA hospitals in the first 2 years following ACGME resident duty hour reform. JAMA. 2007;298(9):984-992. [DOI] [PubMed] [Google Scholar]
- 18.Ross JS, Maynard C, Krumholz HM, et al. Use of administrative claims models to assess 30-day mortality among Veterans Health Administration hospitals. Med Care. 2010;48(7):652-658. doi: 10.1097/MLR.0b013e3181dbe35d [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Sohn M-W, Arnold N, Maynard C, Hynes DM. Accuracy and completeness of mortality data in the Department of Veterans Affairs. Popul Health Metr. 2006;4:2. doi: 10.1186/1478-7954-4-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Fine MJ, Auble TE, Yealy DM, et al. A prediction rule to identify low-risk patients with community-acquired pneumonia. N Engl J Med. 1997;336(4):243-250. doi: 10.1056/NEJM199701233360402 [DOI] [PubMed] [Google Scholar]
- 21.Rahimi K, Bennett D, Conrad N, et al. Risk prediction in patients with heart failure: a systematic review and analysis. JACC Heart Fail. 2014;2(5):440-446. doi: 10.1016/j.jchf.2014.04.008 [DOI] [PubMed] [Google Scholar]
- 22.Garvin JH, DuVall SL, South BR, et al. Automated extraction of ejection fraction for quality measurement using regular expressions in unstructured information management architecture (UIMA) for heart failure. J Am Med Inform Assoc. 2012;19(5):859-866. doi: 10.1136/amiajnl-2011-000535 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Rubin DB, ed. Multiple Imputation for Nonresponse in Surveys. Wiley; Hoboken, New Jersey. 1987. doi: 10.1002/9780470316696 [DOI] [Google Scholar]
- 24.Kind AJH, Buckingham WR. Making neighborhood-disadvantage metrics accessible - the Neighborhood Atlas. N Engl J Med. 2018;378(26):2456-2458. doi: 10.1056/NEJMp1802313 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Kind AJH, Jencks S, Brock J, et al. Neighborhood socioeconomic disadvantage and 30-day rehospitalization: a retrospective cohort study. Ann Intern Med. 2014;161(11):765-774. doi: 10.7326/M13-2946 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Singh GK. Area deprivation and widening inequalities in US mortality, 1969-1998. Am J Public Health. 2003;93(7):1137-1143. doi: 10.2105/AJPH.93.7.1137 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Scott SL, Varian HR. Predicting the present with Bayesian structural time series. IJMMNO. 2014;5(1/2):4. doi: 10.1504/IJMMNO.2014.059942 [DOI] [Google Scholar]
- 28.Sjoding MW, Iwashyna TJ, Dimick JB, Cooke CR. Gaming hospital-level pneumonia 30-day mortality and readmission measures by legitimate changes to diagnostic coding. Crit Care Med. 2015;43(5):989-995. doi: 10.1097/CCM.0000000000000862 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Trivedi AN, Nsa W, Hausmann LRM, et al. Quality and equity of care in U.S. hospitals. N Engl J Med. 2014;371(24):2298-2308. doi: 10.1056/NEJMsa1405003 [DOI] [PubMed] [Google Scholar]
- 30.Ody C, Msall L, Dafny LS, Grabowski DC, Cutler DM. Decreases in readmissions credited to Medicare’s program to reduce hospital readmissions have been overstated. Health Aff (Millwood). 2019;38(1):36-43. doi: 10.1377/hlthaff.2018.05178 [DOI] [PubMed] [Google Scholar]
- 31.Radomski TR, Fine MJ, Gellad WF. Outcome after admission at Veterans Affairs vs non-Veterans Affairs hospitals. JAMA. 2016;316(3):345-346. doi: 10.1001/jama.2016.5391 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
eFigure 1. Exclusion Flowchart for the Study Population
eFigure 2. Number of Admissions per Quarter for the Study Population
eFigure 3. Percent of Admissions per Quarter with a Primary Diagnosis of Sepsis/RF [with Pneumonia as Secondary Diagnosis] in Extended Pneumonia Cohort
eFigure 4. Unadjusted 30-day Mortality Rates for Veterans Hospitalized with Heart Failure and Pneumonia
eFigure 5. Unadjusted 30-day Mortality Rates for Veterans Hospitalized with Heart Failure, Pneumonia, Pneumonia [excluding Sepsis/RF] and Sepsis/RF [with Pneumonia as Secondary Diagnosis]
eFigure 6. Bayesian Structural Time Series Analyses
eFigure 7. Risk-Standardized 30-Day Mortality Rates Among Veterans Hospitalized with Pneumonia [excluding Sepsis/RF] and Sepsis/RF [with Pneumonia as Secondary Diagnosis] from Claims-Based and Clinical Risk-Adjustment Models
eFigure 8. Quarterly Trends in Predicted Risk of 30-day Mortality, as Derived from Clinical and Claims-Based Risk-adjustment Models for Pneumonia [excluding Sepsis/RF] and Sepsis/RF [with Pneumonia as Secondary Diagnosis]
eTable 1. ICD-9 Codes for Primary Diagnoses
eTable 2. Summary of Missing Data for Adjustment/Summary Variables
eTable 3. Characteristics of Veterans Hospitalized with Pneumonia [excluding primary diagnosis of Sepsis/RF] Between the First and Last Years of the Study
eTable 4. Characteristics of Veterans Hospitalized with Sepsis/RF [with Pneumonia as Secondary Diagnosis] Between the First and Last Years of the Study
eTable 5. Characteristics of Veterans Hospitalized with Heart Failure Between the First and Last Years of the Study
eTable 6. Characteristics of Veterans Hospitalized with Pneumonia Between the First and Last Years of the Study
eTable 7. Comparison of Clinical Characteristics Between First and Last Years of Study for Patients Admitted with Heart Failure
eTable 8. Comparison of Clinical Characteristics Between First and Last Years of Study for Patients Admitted with Pneumonia
eTable 9. Comparison of Sociodemographic Characteristics Between First and Last Years for Patients Admitted with Heart Failure or Pneumonia
eTable 10. Changes in 30-day Mortality among Veterans Admitted with Pneumonia [excluding Sepsis/RF] and Sepsis/RF [with Pneumonia as Secondary Diagnosis]
eTable 11. Analyses for Area under the ROC curve (AUC)/Goodness of Fit for Claims-based and Clinical Risk-adjustment Models
eTable 12. Sensitivity Analyses for Multiple Admissions – First Admission
eTable 13. Sensitivity Analyses for Multiple Admissions – Random Admission