The Impact of Pharmacy-Specific Predictors on the Performance of 30-Day Readmission Risk Prediction Models

Abstract

Research Objective

Pharmacists are an expensive and limited resource in the hospital and out-patient setting. A pharmacist can spend up to 25% of their day planning. Time spent planning is time not spent delivering an intervention. A readmission risk adjustment model has potential to be used as a universal outcome-based prioritization tool to help pharmacists plan their interventions more efficiently. Pharmacy-specific predictors have not been used in the constructs of current readmission risk models. We assessed the impact of adding pharmacy-specific predictors on performance of readmission risk prediction models.

Study Design

We used an observational retrospective cohort study design to assess whether pharmacy-specific predictors such as an aggregate pharmacy score and drug classes would improve the prediction of 30-day readmission. A model of age, sex, length of stay and admission category predictors was used as the reference model. We added predictor variables in sequential models to evaluate the incremental effect of additional predictors on the performance of the reference. We used logistic regression to regress the outcomes on predictors in our derivation dataset. We derived and internally validated our models through a 50:50 split validation of our dataset.

Population Studied

Our study population (n= 350,810) was of adult admissions at hospitals in a large integrated healthcare delivery system.

Principal Findings

Individually, the aggregate pharmacy score and drug classes caused a nearly identical but moderate increase in model performance over the reference. As a single predictor, the comorbidity burden score caused the greatest increase in model performance when added to the reference. Adding the severity of illness score, comorbidity burden score and the aggregate pharmacy score to the reference caused a cumulative increase in model performance with good discrimination (C statistic 0.712, Nagelkerke’s R² 0.112). The best performing model included all predictors: severity of illness score, comorbidity burden score, aggregate pharmacy score, diagnosis groupings and drug subgroups.

Conclusions

Adding the aggregate pharmacy score to the reference model significantly increased the C statistic but was out-performed by the comorbidity burden score model in predicting readmission. The need for a universal prioritization tool for pharmacists may therefore be potentially met with the comorbidity burden score model. However, the aggregate pharmacy score and drug class models still out-performed current Medicare readmission risk adjustment models.

Implications for Policy or Practice

Pharmacists have a great role in preventing readmission, and therefore can potentially use one of our models: comorbidity burden score model, aggregate pharmacy score model, drug class model or complex model (a combination of all five major predictors) to prioritize their interventions while exceeding Medicare performance measures on readmission. The choice of model to use should be based on the availability of these predictors in the healthcare system.

Background

Preventable 30-day readmissions put patients at risk of hospital-acquired infections, increase healthcare costs, and put health systems at risk for fiscal penalties by the Centers of Medicare & Medicaid Services (CMS) Hospital Readmissions Reduction Program ¹. Pharmacist-led interventions during transitions of care have shown potential in reducing hospital readmission rates ^2–8 and medication errors ^9,10. These pharmacist-led interventions include medication reconciliation ^8,10, complete medication reviews during admission ^7,8 and medication therapy management programs after discharge ^11,12. Such interventions have become common in health organizations seeking to reduce their readmission rates ¹³. However, limited pharmacist numbers in hospitals, transitions of care programs, and community pharmacies ¹⁴ highlight the need for a risk model that identifies and prioritizes patients at highest risk of readmission and who would be most amenable to intervention.

General readmission risk prediction models already exist but have not included pharmacy-specific data such as pharmacy risk scores and drug classes within their algorithms. Predictive models for non-elective 30-day readmission that do not include pharmacy data have good predictive power ^15–19, but risk-adjustment models in use by CMS have only shown fair discriminative power ²⁰. One study’s findings suggested that use of outpatient pharmacy data as indicators of comorbidity may be a cost-effective approach for a health organization to predict readmission ²¹. While some indicators of outpatient medication-use (i.e. aggregate medication and subclass use score) were readily available within our integrated health system, their value in improving readmission risk prediction had not been evaluated. We therefore hypothesized that these electronically available indicators of medication-use could enhance existing readmission prediction models.

Our objective in this study was to assess the potential value of adding pharmacy-specific predictors (prescription-based diagnostic cost groups- RxDxCG ²² and drug subgroups) to predictive models of post-discharge outcomes. Our goal was also to develop parsimonious models that would (a) be relatively easy to instantiate, (b) be readily comprehensible by clinicians unfamiliar with complex modeling techniques, and (c) cross-validate well.

Methods

Study Design

We used a retrospective cohort study to assess whether adding pharmacy-specific predictors would improve the performance of a 30-day readmission reference model. We derived our models on 50% of our data and validated them on the remaining 50%.

Setting and Study Population

This study was approved by the Kaiser Permanente Northern California (KPNC) Institutional Review Board. KPNC is an integrated health system caring for approximately 3 million adults across 21 hospitals which all use one electronic medical record system. The dataset used in this study was comprised of 350,810 admission episodes for patients. We randomly split the dataset into derivation and validation cohorts. The inclusion criteria were: (1) admission to one of these hospitals during the year 2014 or 2015; and (2) 18 years or older in age. Patients aged less than 18 years old, same day discharges from outpatient or ambulatory surgeries, and maternity (labor and delivery) admissions were excluded.

Outcomes and Predictors

The two outcomes for our primary models were Healthcare Effectiveness Data and Information Set (HEDIS) defined all-cause non-elective readmission within 30 days of discharge ²³ and non-elective readmission or death within 30 days of discharge (a composite variable) ¹⁵.

RxDxCGs are commercially available and proprietary indices that describe aggregate pharmacy-use and have been used by healthcare insurance companies for pooling commercial patients to determine premiums ²². RxDxCG scores are generated monthly for each KPNC member and could be incorporated into predictive models that could run in real time. We used the last recorded RxDxCG score closest to admission for each patient.

We used the following predictors in our models: age at admission, gender, index discharge length of stay in hospital, admission venue, illness severity (Laboratory Acute Physiology Score, version 2- LAPS2 at admission) ²⁴, comorbidity burden (COmorbidity Point Score, version 2-COPS2) ²⁴, aggregate pharmacy score (RxDxCG) ²², admission diagnosis and drug classes. Admission venue was a categorical variable defining the route through which a patient was admitted (surgical or medical) and included four categories: Emergency department (ED) surgical, non-ED surgical, ED medical and non-ED medical. The LAPS2 score is a laboratory-based severity of illness score based on laboratory test results, vital signs, pulse oximetry and neurological status tests ²⁴. The COPS2 score is a diagnosis-based score that transforms a patient’s comorbid conditions within the last 12 months into a single continuous score ²⁴. The drug class variables were grouped into 21 high-level drug groups (e.g. analgesics and anti-inflammatories, anti-infectives, anti-neoplastics, cardiovascular therapy agents) based on the proprietary Verisk software ²². We grouped admission diagnoses into 30 groupings as described previously ^{15, 24}. The reference model predictors (age, sex, length of stay, and admission category) were chosen based on current evidence on readmission risk prediction models ^15–19. Age and sex predictors were the demographic components used in development of the CMS readmission risk prediction model for patients with heart failure and myocardial infarctions ^25,26. Admission category was characterized as an important characteristic in predicting readmission among general medicine patients ²⁷. The length of stay was added to the reference model due to its associations with readmission in multiple studies ^15,28–32.

For each outcome, we tested 8 models in the derivation cohort and validated the results in the validation cohort. Model performance was assessed with the area under the receiver operator characteristic curves (c-statistic ³³), the Nagelkerke’s pseudo-R², and calibration plots. A calibration plot is a method that describes how well a model’s estimated probability of an event’s occurrence matches up with the actual probability of the event ^34,35.

Statistical Analyses

Statistical analyses were completed using R, version 3.2.3. We used logistic regression to model the outcomes. Our modeling strategy was based on adding predictors to a baseline reference model with the following four predictors: age, sex, length of hospital-stay, and admission venue. The 7 models derived from the reference model demonstrated both the individual and additive contributions of the predictors in the order added. We added predictor variables sequentially to a reference model of age, gender, length of stay, and admission venue starting with LAPS2, RxDxCG, diagnosis group, COPS2, and then drug subgroups. Our fully adjusted model therefore included all these predictors. We calculated 95% confidence intervals for the AUCs through bootstrap resampling and averaging methods ³⁶. We also calculated the number needed to evaluate (NNE ³⁷) for the fully adjusted model across all outcomes. In our context, the NNE represents the number of patients that a healthcare provider would have to screen to identify a ‘true positive’ case.

Results

The mean ages of patients in both the derivation and validation cohorts were the same at 65.1 ± 17.4 years. Males accounted for 47.4% of the population in each cohort. Acute illness severity, comorbidity burden, pharmacy risk score, and the Charlson score were similar across the cohorts (Table 1- all p-values>0.2).

Table 1:

Cohort characteristics

Characteristics	Entire cohort	Derivation cohort	Validation cohort
No. of hospitalizations1 (no. of patients)	350,810 (230,764)	180,757 (138,824)	170,235 (132,800)
30-day2 Rehospitalization rate, %	12.1	12.0	12.2
30-day Rehospitalization or death composite rate, %	14.9	14.8	15.0
Male, %	47.4	47.4	47.3
Age (median, mean ± SD3) % ≥ 65 years	67.0, 65.1 ± 17.4 55.9	67.0, 65.1 ± 17.3 55.8	67.0, 65.1 ± 17.4 55.9
LAPS24 (median, mean ± SD)	49.0, 55.6 ± 38.7	49.0, 55.6 ± 38.7	49.0, 55.5 ± 38.7
COPS25 (median, mean ± SD)	27.0, 45.6 ± 46.6	27.0, 45.6 ± 46.6	27.0, 45.6 ± 46.6
RxDxCG (median, mean ± SD)	3.26, 5.21 ± 6.32	3.26, 5.19 ± 6.30	3.26, 5.22 ± 6.35
Charlson comorbidity score (median, interquartile range)	0.00, 1.03, 2.00	0.00, 1.03, 2.00	0.00, 1.03, 2.00

^*There were no significant differences between the derivation and validation cohorts (all p-values were greater than 0.2)

¹The total number of hospitalizations exceeds total number of patients because individual patients could contribute more than one hospitalization to the dataset.

²Ascertained primarily from Kaiser Permanente Medical Care Program hospitalization and patient demographic databases and probably slightly underestimates true rate.

³Standard deviation

⁴Laboratory Acute Physiology Score, version 2; an acute physiology-based score that includes lactate, vital signs, neurological status checks, and pulse oximetry. Increasing degrees of physiologic derangement are reflected in a higher LAPS2, which is a continuous variable that can range between a minimum of zero and a theoretical maximum of 414, although < 0.05% of patients in our cohort had a LAPS2 exceeding 227 and none had a LAPS2 > 282. Increasing values of LAPS2 are associated with increasing mortality ⁴.

⁵COmorbidity Point Score, version 2; a longitudinal, diagnosis-based score assigned monthly that employs all diagnoses incurred by a patient in the preceding 12 months that results in a single continuous variable that can range between a minimum of zero and a theoretical maximum of 1,014, although < 0.05% of patients in our cohort had a COPS2 exceeding 241 and none had a COPS2 > 306. Increasing values of the COPS2 are associated with increasing mortality ⁴.

The reference model’s performance was fair in both discriminating the occurrence of both outcomes- depicted by a c-statistic of 0.637 (readmission) and 0.675 (composite outcome), and predictive power (Nagelkerke’s R² of 0.047 and 0.081 respectively). Our fully adjusted model included: Acute severity of illness (LAPS2), comorbidity burden (COPS2), regimen complexity (RxDxCG), drug subgroups, and diagnostic groupings (Table 2). Of all individual predictors added to the reference model, diagnosis groupings produced the lowest absolute increase (c-statistic 0.661 (readmission) and 0.688 (composite outcome) in model performance while the comorbidity burden (COPS2) was associated with the highest absolute increase in discrimination and explanatory power (c-statistic 0.704, R² 0.102 for readmission and c-statistic 0.730, R² 0.141 for composite outcome).

Table 2:

Changes in model performance with addition of different predictors (validation dataset)

	C-statistics (AUROC) with 95% confidence intervals		Change in AUROC over reference model		Nagelkerke R2
Logistic model covariates	30-day Readmission	Composite outcome (30-day Readmission OR death)	30-day Readmission	Composite outcome (30-day Readmission OR death)	30-day Readmission	Composite outcome (30-day Readmission OR death)
Age, sex, LOS1, admission category2 (Reference)	0.637 [0.634-0.640]	0.675 [0.672-0.677]	Reference	Reference	0.047	0.081
Reference + Diagnosis groupings 3	0.651 [0.648-0.654]	0.687 [0.685-0.690]	0.014	0.012	0.057	0.095
Reference + LAPS24	0.661 [0.658-0.664]	0.704 [0.701-0.706]	0.024	0.029	0.061	0.111
Reference + RxDxCG5	0.683 [0.681-0.686]	0.709 [0.707-0.712]	0.046	0.034	0.079	0.113
Reference + Drug subgroups6	0.684 [0.681-0.687]	0.709 [0.707-0.711]	0.047	0.034	0.085	0.118
Reference + COPS27	0.704 [0.702-0.707]	0.730 [0.728-0.733]	0.067	0.055	0.102	0.141
Reference + LAPS2 + COPS2 + RxDxCG	0.712 [0.710-0.715]	0.744 [0.741-0.746]	0.075	0.069	0.112	0.162
Reference + LAPS2 + COPS2 + RxDxCG + Diagnosis groupings + Drug subgroups	0.716 [0.713-0.718]	0.748 [0.746-0.750]	0.079	0.073	0.120	0.172

^1.LOS: Length of stay in hours

^2.Admission category: A categorical variable differentiating admission through emergency department (ED) and admission to a medical or surgical service- the four categories were therefore: ED surgical, non-ED surgical, ED medical, non-ED medical

^3.Diagnosis groupings: 42 groupings of diagnosis e.g. cancers, trauma, endocrine, neurological, nutritional etc.

^4.LAPS2: Laboratory Acute Physiology Score, version 2; an acute physiology-based score that includes lactate, vital signs, neurological status checks, and pulse oximetry. Increasing degrees of physiologic derangement are reflected in a higher LAPS2, which is a continuous variable that can range between a minimum of zero and a theoretical maximum of 414, although < 0.05% of patients in our cohort had a LAPS2 exceeding 227 and none had a LAPS2 > 282. Increasing values of LAPS2 are associated with increasing mortality ⁴

^5.RxDxCG: Proprietary risk adjustment system by Verisk

^6.Drug subgroups: 21 drug classes as assigned by Verisk software

^7.COPS2: COmorbidity Point Score, version 2; a longitudinal, diagnosis-based score assigned monthly that employs all diagnoses incurred by a patient in the preceding 12 months that results in a single continuous variable that can range between a minimum of zero and a theoretical maximum of 1,014, although < 0.05% of patients in our cohort had a COPS2 exceeding 241 and none had a COPS2 > 306 (Escobar et al, 2015). Increasing values of the COPS2 are associated with increasing mortality ⁴

The aggregate pharmacy score (RxDxCG) and drug subclasses were associated with a nearly identical moderate increase in model performance. Adding severity of illness, comorbidity burden and the aggregate pharmacy score to the model caused a cumulative increase in model discrimination- c-statistic of 0.712 (readmission) and 0.744 (composite outcome), Nagelkerke’s R² 0.112 (readmission) and 0.162 (composite outcome). The best performing model was the fully adjusted model and included all available predictor classes. The c-statistics for both 30-day readmission and the composite outcomes were 0.720 and 0.754 respectively (Table 2). The calibration plots of each model sustained similar breaks between 40 to 50% upon visual inspection. At a specificity level of 95%, the number needed to evaluate (NNE) for the fully adjusted model was 3.8 for both the readmission and composite outcomes.

Discussion

Pharmacists are still in need of a universal pharmacy-specific evidence-based method to identify patients at high risk of readmission. Such a model could be used to prioritize pharmacist interventions. We added pharmacy-specific predictors that were readily available within our health system to a reference model and evaluated their impact on the model performance.

The RxDxCG model had nearly identical performance to the drug subgroup model likely due to the inclusion of drug-class information in the proprietary RxDxCG formula. The RxDxCG and drug subgroups predictor may therefore be interchangeable in readmission risk modeling- a valuable finding for organizations that have not purchased proprietary risk adjustment models such as RxDxCG but have data on drug groupings.

The comorbidity burden index (COPS2) was the only individual predictor to outperform both RxDxCG and drug subgroup models for both outcomes. This was likely due to the inherent advantage of encompassing all comorbidities in a patient’s history that may not be fully captured in medication lists used to formulate the RxDxCG score. As a single predictor, drug subgroups did not improve performance in models already including RxDxCG. This is likely due to the drug classes already being included in the proprietary formula of the RxDxCG score.

Readmission risk models in use by CMS have c-statistic ranges from 0.60–0.63 ²⁰. Our models outperformed the CMS models, however, we included data available at the end of hospitalization (including length of stay). To demonstrate the performance of our models when used for planning at admission (i.e. using information only available at the start of a hospitalization), we removed LOS as a predictor variable in the models (Appendix I). The c-statistics for the most complex models without LOS were nearly identical to a model including LOS at 0.72 for 30-day readmission and 0.74 for the composite outcome (Appendix I).

The isolated performance of the aggregate pharmacy score (RxDxCG) model was better in the composite outcome (c-statistic of 0.709) than in 30-day readmission (c-statistic of 0.684). This is likely because mortality as an outcome is easier to predict than re-admission. Although models in our study had low pseudo-r² values, this range in values was comparable to other readmission studies ¹⁵. The low explanatory value highlights the importance of other factors with respect to readmission, an outcome that is affected by patient, physician, and health system factors. Our study also provided the number needed to evaluate for the fully adjusted model and showed that a clinician using this model would capture one patient likely to experience readmission or the composite outcome for every four they evaluated. We believe this statistic is more comprehensible to clinicians and provides clinical departments with staffing requirements important for planning interventional programs utilizing our models.

Limitations

Some limitations exist in our study. Firstly, our study was conducted in an integrated health system whose patient population may differ from other healthcare settings such as federal or state-run settings. Secondly, our data excluded pediatrics, persons under 18 years old and pregnant women, and therefore those populations are not represented in our findings. Thirdly, we used predictors that were readily available in our health system that may not be accessible in other settings. However, our approach showed that pharmacy data when used in isolation can be useful in building readmission risk models. Also, other pharmacy-specific predictors in use at other institutions may provide an opportunity to improve readmission risk prediction. Lastly, our models do not tell us which patients carry the most pharmacist-relevant risks of readmission- patients identified at high risk do not necessarily have actionable interventions by pharmacists. However, even with these limitations, our models highlight opportunities to include pharmacy-specific data into risk prediction models and with further research and collaboration with pharmacy departments, models specifically predicting pharmacist-modifiable risks can be developed.

Conclusion

Readmission remains a core CMS quality measure and pharmacists could contribute to decreasing readmission rates in settings that employ prediction models. The use of pharmacy-specific predictors in readmission risk prediction has not been widely documented and our models show potential in this area.

While pharmacy-specific predictors such as RxDxCG and drug subgroups individually improve the performance of our reference readmission risk models, other predictors such as the COPS2 score add even more discriminative power. This suggests that pharmacists could use any of the three predictors (COPS2, RxDxCG, drug subgroups) and equal or exceed CMS readmission risk models’ performance and identify patients who are at highest risk of readmission. The choice of which model to use shall be dictated by which of the three predictors is most readily available in that institution or pharmacy department. More research is needed to identify pharmacist-relevant outcomes and predictors that can be used to build predictive models that can be used at most pharmacy settings to help pharmacists plan their interventions by identifying those patients at greatest risk of an outcome.