Derivation and Validation of a Risk Standardization Model for Benchmarking Hospital Performance for Health-Related Quality of Life Outcomes after Acute Myocardial Infarction

Abstract

Background

Before outcomes-based measures of quality can be used to compare and improve care, they must be risk-standardized to account for variations in patient characteristics. Despite the importance of health-related quality of life (HRQL) outcomes among patients with acute myocardial infarction (AMI), no risk-standardized models have been developed.

Methods and Results

We assessed disease-specific HRQL using the Seattle Angina Questionnaire at baseline and 1 year later in 2693 unselected AMI patients from 24 hospitals enrolled in the TRIUMPH registry. Using 57 candidate sociodemographic, economic, and clinical variables present on admission, we developed a parsimonious, hierarchical linear regression model to predict HRQL. Eleven variables were independently associated with poor HRQL after AMI, including younger age, prior CABG, depressive symptoms, and financial difficulties (R²=20%). The model demonstrated excellent internal calibration and reasonable calibration in an independent sample of 1890 AMI patients in a separate registry, although the model slightly over-predicted HRQL scores in the higher deciles. Among the 24 TRIUMPH hospitals, 1-year unadjusted HRQL scores ranged from 67–89. After risk-standardization, HRQL scores variability narrowed substantially (range=79–83), and the group of hospital performance (bottom 20%/middle 60%/top 20%) changed in 14 of the 24 hospitals (58% reclassification with risk-standardization).

Conclusions

In this predictive model for HRQL after AMI, we identified risk factors, including economic and psychological characteristics, associated with HRQL outcomes. Adjusting for these factors substantially altered the rankings of hospitals as compared with unadjusted comparisons. Using this model to compare risk-standardized HRQL outcomes across hospitals may identify processes of care that maximize this important patient-centered outcome.

While current performance measures evaluate the processes of care, calls are increasing for the use of outcomes to assess healthcare quality.^1–2 Prior to using patients’ outcomes as a means of assessing, comparing and improving quality, it is important to develop models that account for differences in patient characteristics that are present prior to the delivery of care, so that hospitals are judged on the care that they deliver, rather than the types of patients that they treat.³ Risk-standardized outcomes models for quality reporting exist for mortality^4–5 and readmission⁶ in the setting of acute myocardial infarction (AMI), but none have been developed for patients’ health-related quality of life (HRQL)—despite its primary importance to patients.^7–8

In the setting of AMI, evaluating and comparing HRQL may be particularly useful because several interventions improve this outcome, but not necessarily mortality or readmission, such as revascularization,⁹ cardiac rehabilitation,¹⁰ and disease management.¹¹ A model predicting HRQL could be used as a foundation to compare the impact of alternative institutional approaches to AMI management on patients’ HRQL and may support the sharing of best practices and dissemination of more cost-effective care.

To address the need for a model to risk-standardize HRQL outcomes, we used the Translational Research Investigating Underlying disparities in acute Myocardial infarction Patients’ Health status (TRIUMPH) study of over 4000 AMI patients from 24 U.S. hospitals to build and validate a risk-stratification model for 1-year post-AMI HRQL, based on patient demographics, psychosocial factors, comorbidities, and disease severity variables. We then sought to externally validate the model in a unique cohort of AMI patients.

METHODS

Population and Protocol for Model Development

Details regarding the study design, patient selection, site characteristics and follow-up assessments of the TRIUMPH study have been previously published.¹² Briefly, 4340 patients from 24 U.S. hospitals were enrolled into the TRIUMPH study (2005–08). Patients were required to have biomarker evidence of myocardial necrosis and additional clinical evidence supporting the diagnosis of AMI, including prolonged ischemic signs/symptoms or electrocardiographic changes during the initial 24 hours of admission. Patients not presenting to an enrolling institution were eligible if transferred within 24 hours of presentation to ensure that the primary treatment decisions were made by the enrolling hospital. Baseline data were obtained through chart abstraction and a detailed structured interview by trained research staff between 24 and 72 hours of admission.

Follow-up health status data were obtained 1 year after AMI with a standardized telephone interview performed by a central, specialized follow-up center. Only patients who had both baseline and 1-year assessments were included for this study. Each participating hospital obtained Institutional Research Board approval, and all patients provided written informed consent.

Assessment of HRQL

Disease-specific health status was measured with the Seattle Angina Questionnaire (SAQ), a 19-item self-administered questionnaire that measures 5 dimensions of health in patients with coronary artery disease.^13–14 Domain scores range from 0 to 100, with higher scores indicating less disease burden. For this study, we focused on the disease-specific HRQL domain of the SAQ, which quantifies how the patient perceives that coronary artery disease has impacted his or her HRQL over the preceding 4 weeks.¹³ The SAQ has undergone extensive reliability and validity testing,^15–16 and lower scores in the HRQL domain are associated with increased mortality and cardiac hospitalizations.^{15, 17}

Model Development

One-year SAQ HRQL scores were non-normal and left-skewed (eFigure 1). We considered 3 candidate approaches for modeling data with heterogeneous conditional distributions: ordinary least squares, Tobit, and median regression. We compared the absolute prediction errors of the models using a random sample of ¾ of the dataset with the remaining ¼ of the dataset. As each model resulted in similar absolute prediction errors and coefficient estimates, we chose ordinary least squares linear regression due to its ease of analysis and interpretation.

Variables for the HRQL prediction model were selected from 57 candidate variables (Table 1) among 8 larger conceptual domains expected to potentially impact HRQL (eFigure 2): demographics, psychosocial, economics, lifestyle, comorbidities, health status at time of AMI, clinical status at presentation, and admission medications. Because this model was designed to support evaluations of hospital quality, the model was built exclusively with patient-specific characteristics available within 24 hours of admission. Excluding treatments that could influence long-term HRQL outcomes (e.g. revascularization) may reduce the predictive accuracy of the model. However, it is necessary to exclude those processes of care that are specific to particular hospitals in order to be able to use the model for quality assessment and risk-standardization to compare hospitals and their choice of treatment strategies.

Table 1.

Candidate Variables for the Prediction of Long-Term Quality of Life After an Acute Myocardial Infarction

Category	Variables Considered for Prediction Model
Demographics	Age	Married
	Sex	High School Education
	Race
Psychosocial Status	9-Item PHQ Depression Score	ENRICHD Social Support Score
	4-Item Perceived Stress Score	REALM-R Medical Literacy Score
Economic Status	Monthly Financial Reserves	Financial Difficulty Getting Medical Care
	Insurance for Medical Care	Avoided Care Due to Costs in Past Year
	Insurance Coverage for Meds	Avoided Meds Due to Costs in Past Year
Lifestyle	Lives Alone	Activity Level During Leisure Time
	Currently Working	Alcohol Abuse
	Current Smoker
Comorbidities	Prior Myocardial Infarction	Peripheral Vascular Disease
	Prior PCI	Diabetes Mellitus
	Prior Bypass Graft Surgery	Chronic Kidney Disease
	Prior Stroke or TIA	Chronic Dialysis
	Prior Angina	Chronic Lung Disease
	Chronic Heart Failure	History of Atrial Fibrillation
	Dyslipidemia	Body Mass Index
	Hypertension
Baseline Health Status	SAQ Angina Frequency Score	SF-12 Physical Components Score
	SAQ HRQL Score	SF-12 Mental Components Score
	Dyspnea Score
Acute Presentation	Ischemic Symptoms at Arrival	Initial Glucose
	Symptom Onset to Arrival Time	Initial Glomerular Filtration Rate
	ST-Elevations on Arrival	Peak Troponin
	Initial Heart Rate	Left Ventricular Systolic Function
	Initial Mean Arterial Pressure	GRACE 6-Month Mortality Score4
	Initial Hemoglobin
Arrival Medications	Aspirin	ACE Inhibitor/ARB
	Thienopyridine	Statin
	Beta Blocker	Insulin

Abbreviations: PHQ, Patient Health Questionnaire; ENRICHD, Enhancing Recovery in Coronary Heart Disease; REALM-R, Rapid Estimate of Adult Literacy in Medicine, Revised; PCI, percutaneous coronary intervention; TIA, transient ischemic attack; SAQ, Seattle Angina Questionnaire; HRQL, health-related quality of life; GRACE, Global Registry of Acute Coronary Events; ACE, angiotensin converting enzyme; ARB, angiotensin II receptor blocker

Baseline data had a high rate of completion, with an average of 0.76 missing data items per patient. Missing baseline data were imputed using a multiple imputation dataset. Harrell’s backward selection strategy was used to select a parsimonious set of variables for the final model,¹⁸ which supported inclusion of only those variables that provided incremental prognostic value, avoided over-fitting, and maximized the feasibility of future data collection to support quality metrics based upon patients’ HRQL. To accomplish this, the contribution of each significant predictor in the multivariable model was ranked by F-value, and variables with the smallest contribution to the model were sequentially eliminated. This iterative process continued until further variable elimination led to a greater than 10% loss in model prediction, as compared with the initial fully adjusted model. The remaining covariates comprised the reduced model and explained greater than 90% of the variance of the complete model. All models were hierarchical with site included as a random effect to account for the clustering of patients within site, and non-linear spline terms were considered for all continuous variables. Predicted SAQ HRQL scores were plotted against observed scores, and the regression line was compared against the line of equality, which would indicate perfect calibration (intercept=0 and slope=1). Model performance was also evaluated with R².

Model Validation

After the prediction model was created in TRIUMPH, the model was externally validated using the Prospective Registry Evaluating Myocardial Infarction: Event and Recovery (PREMIER) study.¹² PREMIER (2003–04) was a prospective cohort study of 2498 AMI patients from 19 U.S. hospitals. TRIUMPH and PREMIER were similarly designed AMI studies, with identical inclusion and exclusion criteria, identical HRQL outcomes collected at the same follow-up time points, nearly identical baseline covariates, and some overlap in participating sites (12 of the 24 TRIUMPH hospitals also participated in PREMIER). When we used the coefficient estimates from the TRIUMPH prediction model to calculate the predicted SAQ HRQL scores in PREMIER, the intercept for the model was re-centered using the global mean for follow-up SAQ HRQL scores for PREMIER, congruent with approaches used to risk-standardize 30-day outcomes by the Center for Medicare and Medicaid Services.^19–20 Predicted SAQ HRQL scores were plotted against observed scores for PREMIER, and the regression line was compared against the line of equality (intercept=0 and slope=1).

One difference between the 2 studies was the collection of the Perceived Stress Scale (PSS), which was assessed in TRIUMPH at all time points but only assessed in PREMIER at follow-up. Baseline PSS scores for PREMIER patients were imputed by constructing an imputation model in TRIUMPH that included all baseline clinical and health status variables and 1-month PSS scores. The R² of this model for predicting baseline PSS scores in TRIUMPH was 51%. As a sensitivity analysis, we recalibrated the model within TRIUMPH excluding the PSS and then tested the revised model in PREMIER. The model performed well, with little change in the internal or external calibration (Appendix, eFigure 3 and 4).

Risk-Standardized Rates and Hospital Variation

The hierarchical linear regression model was used to estimate hospital-level risk-standardized HRQL within TRIUMPH using a similar approach as has been used to risk-standardize mortality and rehospitalization rates.^{6, 20} Importantly, this approach takes into account the hierarchical structure of the data to account for patient clustering within hospitals. Risk-standardized HRQL was calculated as the difference of the “predicted” HRQL (HRQL predicted on the basis of the hospital’s performance with its observed case mix) and the “expected” HRQL (HRQL expected on the basis of performance of the average hospital with this hospital’s case mix), added to the overall mean HRQL for the whole study population. Finally, we ranked the TRIUMPH hospitals by their unadjusted and risk-standardized HRQL and examined the agreement between these rankings. Based on prior work, we grouped the sites into 3 categories: top 20%, middle 60%, and bottom 20% and compared the groupings before and after risk-standardization of the HRQL scores.²¹

All analyses were conducted using SAS v9.2 (SAS Institute, Inc., Cary, NC) and IVEware (Institute for Social Research, Ann Arbor, MI).

RESULTS

Analytic Population

Among the 4316 patients enrolled in TRIUMPH who were discharged alive, 2693 had both baseline and 1-year follow-up assessments of HRQL and were thus included in our analyses. Exclusion was due to death (n=268; 6.2%), interview refusal (n=258; 5.9%), incomplete interview (n=63; 1.5%), and an inability to contact the participant (n=1058; 24.4%). While unadjusted baseline HRQL scores were lower for those alive but missing follow-up assessments (missing vs. not missing: 62.0 vs. 64.3, p=0.004), HRQL scores were not associated with missing follow-up in a multivariable logistic model (p=0.48). In addition, we performed a sensitivity analysis including all patients with at least 1 follow-up HRQL assessment (n=3612) and used the last known HRQL score for the 1-year HRQL outcome. There was little change in the calibration of the model in the expanded analytic sample, although there was some slight over-prediction of HRQL compared with the original model (eFigure 5), as might be expected if patients’ HRQL continues to improve over time.

Of the 2481 patients in PREMIER who were discharged alive, 1890 had both baseline and 1-year follow-up HRQL assessments and were included in the model validation. Table 2 lists the baseline characteristics of the patients in TRIUMPH and PREMIER who were included in our analyses. Study participants were similar between the two studies, although TRIUMPH study participants were slightly younger than PREMIER study participants (59 vs. 63 years) and were more likely to be of non-White race (36% vs. 29%). SAQ HRQL scores were systematically higher in the PREMIER population than in TRIUMPH (mean: 84.7 vs. 81.7), which may be due to either small differences in the patient populations.

Table 2.

Baseline Characteristics of TRIUMPH and PREMIER Patient Populations

	TRIUMPH (n=2693)	PREMIER (n=1890)
Age (years)	59.9 ± 11.9	61.0 ± 12.4
Male sex	67.0%	67.9%
White race	73.0%	79.1%
Prior myocardial infarction	19.2%	19.3%
Prior PCI	19.0%	17.1%
Prior Bypass Graft Surgery	11.8%	11.9%
Prior Stroke or TIA	6.5%	7.6%
Chronic Heart Failure	6.7%	8.5%
Hypertension	65.3%	61.9%
Peripheral Vascular Disease	4.5%	6.8%
Diabetes	28.9%	26.6%
Chronic Lung Disease	6.8%	11.0%
ST-Elevations on Arrival	44.2%	46.7%
Estimated GFR (mL/min/1.73m2)	76.4 ± 26.4	74.9 ± 29.5
Peak Troponin (ng/mL)	29.3 ± 75.1	80.1 ± 211.8
GRACE 6m Mortality Risk	100.5 ± 28.4	104.8 ± 30.1
Aspirin on Arrival	41.5%	37.1%
Thienopyridine on Arrival	12.0%	6.3%
Beta Blocker on Arrival	33.2%	31.0%
ACE Inhibitor/ARB on Arrival	35.7%	32.6%
Statin on Arrival	33.6%	29.6%

Abbreviations: PCI, percutaneous coronary intervention; TIA, transient ischemic attack; GFR, glomerular filtration rate; GRACE, Global Registry of Acute Coronary Events; ACE, angiotensin converting enzyme; ARB, angiotensin II receptor blocker

Model Creation

The variables associated with post-AMI HRQL in the final reduced model are shown in Figure 1. According to F-values, age had the strongest magnitude of association with post-AMI HRQL, with older patients reporting, on average, 7–8 point higher SAQ HRQL scores 1 year after AMI than patients under 50 years of age (i.e. better HRQL). Patients with a prior bypass surgery reported lower follow-up HRQL than those without prior bypass surgery (mean difference [95% CI] = −6.0 [−7.9 to −3.3]). Depressive symptoms, financial difficulties, and female sex were also strongly associated with poorer follow-up HRQL. Other factors associated with worse 1-year HRQL included current smoking, elevated chronic stress levels, more angina prior to their AMI, and non-white race. The specific formula for calculating expected SAQ HRQL, both in a population of patients similar to TRIUMPH and in an independent dataset with a different patient case-mix, is shown in eTable 1. The R² of the regression model was 20%. The observed versus predicted SAQ HRQL scores in TRIUMPH (Figure 2a) demonstrated excellent calibration, with an intercept of 0.94 (SE 3.44; p-value [for difference from 0] =0.79), a slope of 0.99 (SE 0.04; p-value [for difference from 1] =0.79), and an R² of 98%.

Figure 1.

Predictors of Long-Term Quality of Life After an Acute Myocardial Infarction

Figure 2. Calibration Plot for the HRQL Prediction Model in TRIUMPH (a) and PREMIER (b).

TRIUMPH Internal Validation: intercept of 0.94 (SE 3.44), a slope of 0.99 (SE 0.04), and R² of 98%. PREMIER External Validation: intercept of 14.1 (SE 2.38; p-value [for difference from 0] <0.001), a slope of 0.82 (SE 0.03).

Model Validation

The HRQL prediction model, derived in TRIUMPH, was externally validated in PREMIER. The cohort of 1890 PREMIER patients was divided into deciles according to their predicted SAQ HRQL scores, and the calibration plot of observed vs. predicted 1-year SAQ HRQL scores is shown in Figure 2b. The observed versus predicted SAQ HRQL scores in PREMIER demonstrated reasonable calibration, with an intercept of 14.1 (SE 2.38; p-value [for difference from 0] <0.001), a slope of 0.82 (SE 0.03; p-value [for difference from 1] <0.001), with the model tending to over-predict HRQL scores in the higher deciles.

Hospital-Specific Risk-Standardized HRQL and Variability

The 1-year, unadjusted and risk-standardized HRQL outcomes, by site, are shown in Figures 3a and 3b. The range of unadjusted HRQL outcomes across the 24 hospitals in our sample was 67.4–89.5. After risk-standardization, the median risk-standardized HRQL was 81.7 and the range was much more narrow (79.5–83.3). Comparing the unadjusted vs. risk-standardized rankings of the 24 hospitals in the analysis, the median absolute change in rank position was 4 places (range 1–9). The 20%/60%/20% rankings differed in category for 14 of the 24 hospitals (58.3%).

Figure 3.

Unadjusted (a) and Risk-Standardized (b) 1-Year HRQL Outcomes by Site.

DISCUSSION

To our knowledge, this study represents the first development and validation of a model to risk-standardize 1-year HRQL outcomes after an AMI, which could permit comparisons of hospital performance (e.g., quality assessment) based on patients’ 1-year quality of life and the sharing of best practices to elevate healthcare quality. As we identified many factors associated with HRQL that are not modifiable by the hospitals (e.g., socioeconomic status, age), adjusting for these factors is critically important for fairly comparing HRQL outcomes across hospitals as risk-standardization significantly altered the apparent differences in hospitals in optimizing their patients’ HRQL. Anticipating the value of such assessments, site variation in post-MI mortality has been described²² and used to identify high-level performers (i.e., those with mortality rates better than what was predicted), whose processes of care can be shared with lower-performing facilities. However, the absence of models to appropriately risk-standardize HRQL outcomes, has prevented such comparisons for health status outcomes,²³ despite their importance to patients.^7–8

Better hospital performance in AMI care should be judged not only on mortality but also on other outcomes of importance to patients and society. In so doing, a more complete assessment of a hospital’s care can be acquired from assessing the outcomes of all survivors, and not just the minority who die. As the Centers for Medicare and Medicaid Services attempt to define and implement value-based purchasing, the need to expand our assessment of healthcare quality to include measures of quality of life has become more apparent. In a recent perspective, VanLare and Conway emphasized that patient-centered outcomes must be included as core measures in value-based purchasing, although these measures are currently lacking.²⁴ While having access to instruments that reliably and validly assess patients’ HRQL is necessary to including HRQL as an outcome measure for evaluating the quality of care, being able to risk-standardize these outcomes is also critical prior to their implementation as quality metrics.

Importantly, our model explained ~20% of the variation in HRQL outcomes, which indicates that much of the variation in HRQL outcomes is not explained by patient factors. This leaves open the possibility that the remaining variation could be explained, instead, by the hospitals’ processes of care. For example, one would expect that treatments during and after the AMI, such as early revascularization, smoking cessation, and cardiac rehabilitation, would influence 1-year HRQL.^9–11 As this model includes only patient factors present on admission, it can balance differences in patient populations so that hospitals may be fairly compared, independent of their case mix. While a model based only on patient-level factors that explains 100% of the variation in HRQL outcomes would be ideal for identifying high-risk patients, such a model would not have value in the assessment of healthcare quality. As such, the current model can permit the comparisons of HRQL outcomes across hospitals, independent of patients’ presenting characteristics. Furthermore, using 1-year HRQL as the main outcome measure in our model has the potential to reflect quality across an entire continuum of care, which is particularly relevant given the increased emphasis on care coordination at and after hospital discharge.^24–26

Prior studies

We are unaware of any previously published risk-standardization models for HRQL after an AMI that allow for site-level analyses and identification of better-and worse-performing hospitals. Prior studies that investigated the association of individual factors with worse HRQL after an AMI, such as younger age,²⁷ black race,²⁸ and depressive symptoms,^29–30 provided important information about which covariates would be important for us to include as candidate predictors. We were able to confirm the importance of these factors in our model. A few studies have attempted to identify a set of factors associated with generic HRQL after AMI. ^31–32 However, these studies sought to identify characteristics that would make a patient high-risk for poor HRQL outcomes. While our model could also be adapted for this purpose, it also has the ability to support risk-standardization of HRQL outcomes across hospitals so that one can examine site-level variability, identify better-performing hospitals, and ultimately, determine the processes of care that improve the outcomes of patients after AMI.

HRQL is of clear importance to patients,^7–8 and we present a mechanism for risk-standardizing this outcome so that hospitals can be fairly compared. However, prior to systematically comparing the HRQL outcomes of hospitals after an AMI, novel data collection strategies will be needed to acquire the patient-centered, socioeconomic and psychological characteristics that we identified as important when risk-standardizing HRQL outcomes. If we are committed, as a society, to measuring outcomes that are important to patients, then our current data collection systems need to include novel data elements, such as depression scores, perceived stress scores, and socioeconomic characteristics. In an era of increasing use of electronic medical records—as advanced by the incentives of meaningful use—we can create new ‘core elements’ that are routinely collected on admission so that these important patient-centered outcomes can be used to compare and improve care.

Limitations

There are several potential limitations to the current study to consider when interpreting our results. First, although TRIUMPH included rural, suburban, and urban hospitals across the U.S. and participants represented a broad range of socioeconomic and demographic characteristics, the relatively small number of hospitals in our derivation and validation samples provided a limited number of centers with which to examine variability across centers. Accordingly, some of the narrowing of the range of risk-adjusted HRQL was due to shrinkage correction for hospital sample sizes, which would not be an issue in a larger dataset. However, a unique strength of this study was the ability to externally validate the model in a separate dataset, in which the model showed excellent calibration, increasing our confidence in its generalizability. Second, to be included in this model, patients had to survive at least 1 year. Thus, this model should be used in conjunction with mortality prediction models so that both outcomes can be integrated when evaluating quality and educating patients’ about their prognosis and the benefits of treatments. Finally, we are using the hospital that provided the initial care as the unit of analysis for HRQL 1 year later, after which additional care (both in the inpatient and outpatient settings) will have occurred that could influence HRQL. Although attribution might be challenging outside of integrated systems (e.g., Kaiser Permanente), we believe that this remains a robust model that can form the basis of comparing institutional outcomes and can incentivize poorer performing hospitals to improve their transitions in care after hospital discharge.

Conclusion

Using a multi-site cohort study with a broad spectrum of AMI patients, we developed and validated a prediction model for long-term HRQL. We identified factors associated with HRQL, as compared with mortality or readmissions, such as younger age, psychosocial stress, and socioeconomic status. This model fills an important gap in the use of patient-reported outcomes as markers of healthcare quality by providing a risk-standardization model that can account for patient characteristics prior to hospital presentation. As such, it can facilitate analyses of site-level variation in risk-standardized HRQL outcomes both as a measure of healthcare quality and as a means for identifying hospitals that employ processes of care that maximize patients’ HRQL. Identifying such hospitals can enable their strategies to be disseminated to other hospitals.