Choosing Wisely Clinical Decision Support Adherence and Associated Inpatient Outcomes

Abstract

Objectives:

To determine whether utilization of clinical decision support (CDS) is correlated with improved patient clinical and financial outcomes.

Study Design:

Observational study of 26,424 patient encounters. In the treatment group, the provider adhered to all CDS recommendations. In the control group, the provider did not adhere to CDS recommendations.

Methods:

An observational study of provider adherence to a CDS system was conducted using inpatient encounters spanning 3 years. Data comprised alert status (adherence), provider type (resident, attending), patient demographics, clinical outcomes, Medicare status, and diagnosis information. We assessed the association between alert adherence and 4 outcome measures: encounter length of stay, odds of 30-day readmission, odds of complications of care, and total direct costs. The association between alert adherence and the outcome measures was estimated using 4 generalized linear models that adjusted for potential confounders, such as illness severity and case complexity.

Results:

The total encounter cost increased 7.3% (95% CI, 3.5%−11%) for non-adherent encounters versus adherent encounters. We found a 6.2% (95% CI, 3.0%−9.4%) increase in length of stay for non-adherent versus adherent encounters. The odds ratio for readmission within 30 days increased by 1.14 (95% CI, 0.998–1.31) for non-adherent versus adherent encounters. The odds ratio for complications increased by 1.29 (95% CI, 1.04–1.61) for nonadherent versus adherent encounters.

Conclusions:

Consistent improvements in measured outcomes were seen in the treatment group versus the control group. We recommend provider organizations consider the introduction of real time CDS to support adherence to evidence based guidelines, but because we cannot determine the cause of the association between CDS interventions and improved clinical and financial outcomes, further study is required.

Précis:

This analysis examines the association between adherence to Choosing Wisely recommendations embedded into clinical decision support alerts and 4 measures of resource use and quality.

The Health Information Technology for Economic and Clinical Health Act, an important component of the American Recovery and Reinvestment Act, enabled the federal government to subsidize hospitals, health systems, and physicians $40 billion¹ to implement electronic health records (EHRs) and imposed significant penalties on nonadopters. This investment was expected to result in up to $470 billion in inpatient cost savings alone² through reduced patient length of stay,³ reduced utilization of services, and other outcomes.⁴ Today, certified EHRs are operational in 96% of nonfederal acute care hospitals and health systems in the United States,⁵ but the expected cost savings have not yet been realized.⁶ The evidence that EHRs improve quality and patient outcomes has been mixed; studies have found improved quality of care in the ambulatory care setting,⁷ higher guideline adherence, fewer medication errors, and decreased adverse drug effects.⁸ However, other studies have found that EHR use is not associated with decreased readmissions^9,10 or lower rates of mortality.⁸

In 2012, the ABIM Foundation introduced the Choosing Wisely (CW) initiative, a voluntary effort by more than 70 physician subspecialty societies to identify commonly used low-value services.¹¹ The intent of this publicly promoted initiative was to stimulate provider–patient discussions about appropriate care and thereby reduce low-value tests and treatments.⁷ Although the primary aim of CW is not to lower costs, reducing inappropriate care could lead to lower costs for both patients and payers. To date, CW may not have achieved clinically significant changes in reducing low-value care.^12–14 Public promotion alone does not appear to be sufficient to achieve widespread adoption.¹⁰ A 2015 claims-based analysis of 7 CW recommendations found that use of 2 low-value services declined, but the decreases were not clinically significant.¹⁰ In their recommendations, the authors called for innovative methods to disseminate CW recommendations.¹⁰ Provider difficulty interpreting guidelines and evaluating patient risk,^15,16 patient need for reassurance,¹³ and provider fear of malpractice litigation¹⁷ pose additional obstacles.

Ideally, an EHR infrastructure could overcome these obstacles and provide real-time computerized clinical decision support (CDS) to inform healthcare providers when their care deviates from evidence-based guidelines. CDS comprises a variety of tools, including computerized alerts and reminders with information such as diagnostic support, clinical guidelines, relevant patient information, diagnosis-specific order sets, documentation templates, and drug–drug interactions.¹⁸ CDS provides the ability to modify tests and treatments based on context-specific and patient-specific information presented at the point of care. Utilizing CDS can help providers avoid ordering a low-value test or intervention that could lead to additional nontherapeutic interventions or even harm. CDS has been shown to improve a variety of processes, including prescribing practices,¹⁹ appropriate use of diagnostic radiology,²⁰ adherence to quality measures,²¹ and conformance to evidence-based care.¹⁹ Systems that automate CDS, provide tailored recommendations based on patient characteristics, and prompt clinicians to provide a reason for overriding recommendations have been shown to be significantly more likely to succeed than systems that provide only patient assessments.¹⁹

We implemented select CW recommendations in the EHR at a large academic health system in the form of 92 alert-based CDS interventions, both inpatient and ambulatory. Inpatient alerts selected for study were those deemed the most technically feasible to deploy accurately and with a sufficient number of relevant orders that would trigger an alert to fire, thus providing a sufficient volume of alerted encounters to evaluate. When initiating a potentially inappropriate order, a provider received real-time notification of deviation from a CW recommendation. That provider then had the option to cancel, change, or justify the order, if he or she agreed with the alert’s recommendation in the context of the individual patient. The objective of this study was to evaluate the relationship between providers who adhere to CW alerts and measurable outcomes.

METHODS

Study Setting

We conducted an observational study of provider adherence to the 18 highest-volume CW alerts utilizing a commercially available EHR-embedded CDS system at Cedars-Sinai Health System, a nonprofit tertiary 886-bed hospital and multispecialty academic health science center located in Los Angeles, California. The medical staff is pluralistic and includes employed and independent physicians in private practice, physician extenders, and residents.

This study included inpatient encounters from October 22, 2013, to July 31, 2016. The study protocol was approved by the Cedars-Sinai Medical Center (CSMC) Institutional Review Board.

Study Population and Data Sources

Data for the study were collected from 3 sources: data sent from the EHR to the CDS analytics platform, which included the category of the provider triggering the alert (e.g. resident, attending) and clinical data allowing for the assessment of adherence or non-adherence to the alert during the encounter; claims data, which included patient demographics (e.g. age, gender), diagnoses, services provided, admit and discharge dates, Medicare Severity-Diagnosis Related Group codes, and costs; and direct cost data associated with the patient care department, which we describe below. The unit of analysis is the patient encounter; this covers the entire inpatient visit, and there is only 1 encounter per visit. Data were matched using a common encounter identifier. Encounters in which the providers were considered adherent included all encounters that received CW alerts and for which providers adhered to all alerts. Alerts were considered adhered to when the order flagged by the CDS as potentially conflicting with CW was not signed within 1 hour after an alert was shown to a provider. The nonadherent group included encounters where providers received CW alerts and for which they did not adhere to any (complete nonadherence). Approximately 1400 encounters that did not meet either of these criteria were excluded because they included partial adherence to some, but not all of the alerts. The Elixhauser index was computed as an unweighted sum of comorbidities present during all encounters for a given patient to estimate the morbidity burden.²²

Alert Selection and Development

In 2013, we integrated CW recommendations as CDS alerts into the Epic EHR at CSMC. A clinical informatics team enabled the translation of the CW recommendations through a standardized process. First, clinicians reviewed the primary sources cited in each recommendation to define inclusion and exclusion criteria for the CDS rule. Once defined, the clinical logic was deployed in the EHR using standardly available alert tools. Finally, the team reviewed patient charts from encounters in which alerts were triggered and identified opportunities to refine the logic and reduce false positives.

To define the alerts for inclusion in the study, we initially reviewed all inpatient CW alerts that were active in the CSMC EHR at any point during the study period. For this analysis, we eliminated any low-volume alerts that sounded an average of less than once per month. The general definition of when an alert is adhered to is when a provider is advised against taking a particular action and complies with the request. Specifically, because our adherence criteria are used to evaluate EHR data to determine whether a particular order was signed within an hour after seeing an alert, we cannot accurately categorize adherence to alerts that either make recommendations about the appropriateness of individual orders within a series of identical orders (ie, repeat or standing laboratory testing) or that do not flag a particular order as inappropriate and instead are reminders unrelated to avoiding unnecessary care (eg, “Don’t delay palliative care for patients with advanced gynecological cancer.”). All remaining alerts were included in the data set (eAppendix Table 1 [eAppendix available at ajmc.com]).

Outcomes

We assessed the association between alert adherence and 4 outcome measures: encounter length of stay, 30-day readmissions, complications of care, and total direct costs. We defined 30-day readmission as an inpatient readmission to the same facility for any cause occurring within 30 days of discharge that was unplanned and deemed unavoidable. Complications of care are defined using the Agency for Healthcare Research and Quality (AHRQ) Healthcare Cost and Utilization Project (HCUP) classification system for complication codes.²³ Total direct costs are defined as expenses directly associated with patient care, such as labor (wages, salaries, agency, employee benefits), supplies (medical, implant, nonmedical), professional fees, contracted services equipment, and equipment depreciation.²⁴ We selected these 4 outcomes measures due to their relevance to patients, health systems, and payers. As the industry shifts from fee-for-service to value-based contracts, cost containment and quality have become critical priorities for healthcare providers. Length of stay, readmission rates, and complication rates also merit evaluation, given their potential impact on patient outcomes and hospital value-based payment programs.^25,26 Given that many low-value tests and procedures can result in a chain of additional tests and procedures, we also theorized that reducing inappropriate and low-value services may lead to shorter lengths of stay, lower 30-day readmission rates, and lower complication rates.

Statistical Analysis

The adherent/non-adherent encounter groups were compared based on demographic characteristics, number of diagnoses, and case severity. The χ² test was used for categorical variables and the Wilcoxon rank sum test was used for continuous variables.

We estimated the association between alert adherence and the outcome measures using 4 generalized linear models. Alert adherence was measured as a dichotomous predictor. We adjusted for potential confounders, such as illness severity and case complexity, using demographic and clinical variables, such as gender, age, All Patients Refined-Diagnosis Related Groups (APR-DRG) severity level, number of diagnoses, expected length of stay, Elixhauser comorbidity index, Medicare status, and case mix index. A subset of all independent variables was used in each regression model to maximize the quality of fit of the model. Variable selection was performed using a backward stepwise method while minimizing the Akaike information criteria. In addition to alert adherence, variables were included in all models to adjust for the differences between the characteristics of the 2 groups. The continuous covariates generally had skewed distributions and were transformed prior to inclusion in the models. All statistical analyses were performed using R version 3.3.1²⁷ and the following packages: glm2,²⁸ caret,²⁹ and sqldf.³⁰

Multivariate logistic regression was used to estimate the odds of patient outcomes that are dichotomous (ie, 30-day readmissions and complications of care). The 2 continuous outcomes, length of stay and total cost, were estimated using multivariate linear regression models with the dependent variable log-transformed to correct for significant right skew in the distribution of each outcome. The outcome variables also appeared as independent variables in other models. Statistical tests were 2-sided, with P <.05 considered statistically significant. More detailed discussion of the regression models is included in the eAppendix.

RESULTS

A total of 26,424 encounters were included in the analysis out of a total of approximately 100,000 encounters. In 1591 (6%) of these encounters, providers adhered to all alerts (an “adherent encounter”); in the remaining 24,833 (94%) encounters, no alerts were adhered to (a “nonadherent encounter”) (Table 1). Patients in the adherent and nonadherent encounter groups were similar with respect to age (P = .32) and total number of diagnoses (P = .26). Additionally, both encounter groups were comparable with respect to the proportion of patients whose primary payer was Medicare (P = .94). There were significant differences in APR-DRG severity levels (P = .01), with sicker patients in the non-adherent group (a greater proportion of non-adherent patients classified at level 4, extreme). Additionally, there were differences with respect to Elixhauser index scores (P = .04), case mix index values (P = .02), gender (P = .05), and expected length of stay (P <.001) (Table 1).

Table 1.

Characteristics of Patients Discharged From Inpatient Visit

Characteristics	Alert Adherence (n = 1591)a	Alert Nonadherence (n = 24,833)	Pb
Women, n (%)	880 (55.3%)	13,112 (52.8%)	.05
Age, years, mean (SD)	65.9 (18.7)	65.7 (18.3)	.32
APR-DRG severity level			.01
Level 1 (minor), n (%)	254 (15.96%)	3648 (14.69%)
Level 2 (major), n (%)	510 (32.06%)	7574 (30.50%)
Level 3 (severe), n (%)	562 (35.32%)	8945 (36.02%)
Level 4 (extreme), n (%)	280 (17.60%)	4848 (19.52%)
Number of diagnoses, median (IQR)	15 (7.0)	15 (6.0)	.26
Expected length of stay, days, median (IQR)	3.9 (2.2)	4.1 (2.8)	<.001
Elixhauser index, median (IQR)	2.6 (2.7)	2.7 (2.7)	.04
Case mix index, median (IQR)	1.6 (1.4)	1.7 (1.8)	.02
Medicare status	63.0%	63.0%	.94
30-day readmissions rate	17.8%	20.0%	.02
Complications rate	6.7%	10.0%	<.001
Length of stay, days, median (IQR)	4.0 (5.0)	5.0 (6.0)	<.001

APR-DRG indicates All Patients Refined Diagnosis Related Groups; IQR, interquartile range.

^aNumber of encounters.

^bP values are from bivariate analyses; χ² test was used for categorical variables and the Wilcoxon rank sum test was used for continuous variables.

With respect to outcomes, bivariate analyses indicated that patient encounters in the group in which providers did not adhere to CW recommendations had longer unadjusted actual lengths of stay (P <.001), higher complication rates (P <.001), higher 30-day readmission rates (P = .02), and higher direct costs (P <.001).

Overall, adherent encounters had significantly lower total costs, shorter lengths of stay, and lower odds of complications compared with nonadherent encounters. The coefficient of the independent variable used to determine lower odds of 30-day readmissions when the encounter is in the adherent group did not achieve significance. After adjusting for patient characteristics, nonadherent encounters also showed a 7.3% (95% CI, 3.5%−11%; P <.001) increase in total direct costs versus adherent encounters. That represents an increase of $944 for a nonadherent encounter versus an adherent encounter (Table 2).

Table 2.

Association Between CW Alert Adherence and Log Total Costs (in $1000s)^a

	Coefficients (95% CI)	P
CW alert nonadherence	0.07 (0.04–0.11)	<.001
APR-DRG severity level
Level 1 (minor)	Reference
Level 2 (major)	−0.01 (−0.04 to 0.01)	.31
Level 3 (severe)	0.20 (0.17–0.23)	<.001
Level 4 (extreme)	0.94 (0.91–0.97)	<.001
Log length of stay / expected length of stay	0.54 (0.52–0.55)	<.001
Male	0.16 (0.14–0.17)	<.001
Complications, excluding POA	0.89 (0.86–0.92)	<.001

APR-DRG indicates All Patients Refined Diagnosis Related Groups; CW, Choosing Wisely; POA, present on admission.

^aThe adjusted total costs based on alert adherence were estimated using a log-linear model and were fit using ordinary least squares regression and normal error distribution, with the dependent variable log-transformed to correct for significant right skew in the distribution. Adjusted R² = 0.47. The error distribution is assumed to be Gaussian. One variable was log-transformed: the interaction between length of stay and expected length of stay for the given APR-DRG.

We found a 6.2% (95% CI, 3.0%−9.4%; P <.001) increase in length of stay for non-adherent versus adherent encounters. (Table 3). We found that the odds of a patient having a readmission within 30 days were 1.14 (95% CI, 0.998–1.31; P = .0503) times higher in nonadherent encounters (Table 4). The odds of a patient having complications were 1.29 (95% CI, 1.04–1.61; P = .02) times higher in nonadherent encounters (Table 5).

Table 3.

Association Between CW Alert Adherence and Log Length of Stay^a

	Coefficients (95% CI)	P
CW alert nonadherence	0.06 (0.03, 0.09)	.02
APR-DRG severity level
Level 1 (minor)	Reference
Level 2 (major)	0.07 (0.04–0.09)	<.001
Level 3 (severe)	0.25 (0.22–0.28)	<.001
Level 4 (extreme)	0.57 (0.54–0.61)	<.001
Log number of diagnoses	0.13 (0.10–0.16)	<.001
Log expected length of stay based on DRG	0.46 (0.39–0.54)	<.001
Complications, excluding POA	0.22 (0.20–0.25)	<.001
30-day readmission	0.09 (0.07–0.11)	<.001
Interaction: number of diagnoses and log expected length of stay based on DRG	0.05 (0.03–0.07)	<.001

APR-DRG indicates All Patients Refined Diagnosis Related Groups; CW, Choosing Wisely; DRG, Diagnosis Related Group; POA, present on admission.

^aThe adjusted length of stay based on alert adherence was estimated using a log-linear model and was fit using ordinary least squares regression, with the dependent variable log-transformed to correct for significant right skew in the distribution. The error distribution is assumed to be Gaussian. Adjusted R² = 0.50. Three variables were log-transformed: number of diagnoses, expected length of stay based on the given APR-DRG, and length of stay.

Table 4.

Association Between CW Alert Adherence and 30-Day Readmission^a

	Multivariate OR (95% CI)	P
CW alert nonadherence	1.14 (0.998–1.31)	.0503
APR-DRG severity level
Level 1 (minor)	1 (reference)
Level 2 (major)	1.85 (1.61–2.12)	<.001
Level 3 (severe)	3.12 (2.71–3.59)	<.001
Level 4 (extreme)	2.71 (2.32–3.17)	<.001
Log number of diagnoses	1.19 (1.12–1.26)	<.001
Medicare	1.14 (1.06–1.22)	.05
Interaction: length of stay and log number of diagnoses	1.23 (1.18–1.29)	<.001

APR-DRG, All Patients Refined Diagnosis Related Groups; CW, Choosing Wisely; OR, odds ratio.

^aLogistic regression was used to estimate the probability of a readmission within 30 days. The readmissions model had a C statistic of 0.65, and the Hosmer-Lemeshow goodness of fit test using 6 groups was not significant (P = .81), indicating no evidence of poor fit. The link function used was the logit function. The error distribution is assumed to be Bernoulli. One variable was log-transformed: number of diagnoses.

Table 5.

Association Between CW Alert Adherence and Presence of At Least 1 Complication Not Present on Admission^a

	Multivariate OR (95% CI)	P
CW alert nonadherence	1.29 (1.04–1.61)	.021
APR-DRG severity level
Level 1 (minor)	1 (reference)
Level 2 (major)	1.28 (1.06–1.54)	.01
Level 3 (severe)	1.00 (0.83–1.21)	.99
Level 4 (extreme)	0.79 (0.64–0.97)	.03
Log number of diagnoses	1.24 (1.14–1.35)	<.001
Log total costs	1.08 (1.03–1.14)	.003
Log length of stay	1.74 (1.62–1.86)	<.001
Case mix index	2.88 (2.76–3.01)	<.001

APR-DRG indicates All Patients Refined Diagnosis Related Groups; CW, Choosing Wisely; OR, odds ratio.

^aLogistic regression was used to estimate the probability of having at least 1 complication not present on admission. The complications model had a C statistic of 0.64, and the Hosmer-Lemeshow goodness of fit test using 7 groups was not significant (P = .30). The link function used was the logit function. The error distribution is assumed to be Bernoulli. Three variables were log-transformed: number of diagnoses, total costs, and length of stay.

DISCUSSION

To our knowledge, this is the first study to evaluate the association between adherence to multiple CW guidelines delivered via CDS and changes in clinical and financial outcomes. Previous studies have established that effective CDS can impact provider behavior and contribute to improved patient outcomes for specific CDS interventions.³¹ This study contributes to the established body of research indicating that adherence to effective CDS alerts is associated with improved outcomes, such as length of stay,^32,33 complication rates,^34,35 and overall cost.^36,37 Our analysis provides new evidence of the effect that a more comprehensive collection of alerts has on high-level patient and financial outcomes, including shorter length of stay (0.06 days), lower complication rates (odds ratio, 1.29), and reduced cost per adhered patient episode (7.3%).

Our results suggest that the difference in cost savings is both statistically and clinically significant. Adherent encounters resulted in approximately $944 in savings from the median encounter cost of $12,940. A previous study examined the prevalence of 28 low-value services in a large population of commercially insured adults and identified an average potential cost savings of approximately $300 for each patient who received a single low-value service.¹⁰ Our findings surpass this estimate and imply significant cost-savings opportunities through improved and broader utilization of CDS.

Our results also confirm the association between alert adherence and odds of complications as defined by AHRQ’s HCUP.¹⁸ The majority of studies do not specifically analyze the effect of CDS interventions on complication rates for patients; rather, they identify undesired outcomes, such as adverse drug events²⁶ and mortality rates.²⁵ Although it is plausible that the reduction in the utilization of low-value services resulting in lower inpatient lengths of stay may lead to reduced complication rates, we did not evaluate potential causation between specific complications and avoided interventions. Additional research to confirm these findings and, more specifically, to delineate the causal pathway is indicated.

Previous studies’ findings have shown a positive correlation between CDS implementation and patient length of stay.^23,24 However, to our knowledge, no analyses have established a correlation between CDS content targeting unnecessary care and improved lengths of stay. Our findings demonstrate an association between adherence to guideline-based alerts and reduction in unnecessary care with shortened inpatient length of stay.

Limitations

One limitation is our strict definition of “alert compliance”: In order to be in the adherent encounter group, providers had to be adherent to all of the CW CDS-related alerts. Patient episodes in which clinicians followed some but not all of the CW alerts that fired were considered “mixed-adherence” episodes and were excluded from analysis. This strict inclusion criterion limits our understanding of the clinical and financial impact that patients with partiallyadherent episodes may have experienced. Similarly, we were unable to differentiate the impact of specific alerts on our studied outcomes. Although there appear to be some differences between individual alerts, our study did not have enough power to make inferences due to its small sample sizes. Another key limitation of this study is the lack of control for provider effects. The analysis did not include provider characteristics and thus could not examine confounding on the provider level; it is possible that some providers are more likely to trigger alerts or are more likely to be non-adherent to alerts, even though we found no overall correlation between provider acceptance rate and provider outcomes. Additionally, providers who are more likely to adhere to evidence-based guidelines, including CW, may be more likely to ascribe to other system-based approaches and practices consistent with value-based patient care. We need to understand better the differences in characteristics and practice patterns of providers who adhere to CW recommendations compared with providers who do not.

Future analyses should examine the role of specific physician and alert characteristics on adherence to CDS and the effects on outcomes. The Elixhauser index computation was based on all relevant diagnoses made on all encounters for a given patient within our data set. It is possible, however, that the patient could have received additional relevant diagnoses outside of the time frame or hospital system applicable to this study.

Although our regression models adjusted for severity of illness, it is possible that the model did not control for all differences in patient severity or characteristics. Moreover, this study did not seek to establish causation between CW adherence and improved patient and financial outcomes. Many factors determine whether a single alert is adhered to or ignored, including alert fatigue,³⁸ provider familiarity with the guideline presented,³⁹ fear of malpractice,¹³ or need to reassure one’s patient through further diagnostic tests.¹⁶ We were unable to capture some relevant data, including professional billing fees and cost and readmissions data from other facilities, limiting our outcomes analyses. Finally, our demonstrated correlations between adherence and outcomes cannot necessarily be generalized to all CDS interventions, as the alerts evaluated in this study were implemented in the inpatient setting, were deemed the most technically feasible to deploy accurately, and had sufficient volume to evaluate.

Conclusions

We recommend that health systems consider real-time CDS interventions as a method to encourage improved adoption of CW and other evidence-based guidelines. A meta-analysis of CDS systems concluded that by providing context-specific information at the point of care, the odds of providers adopting guideline recommendations are 112 times higher.¹⁹ CDS enables the provision of context-specific information at the point of care and could help to overcome several known barriers to CW guideline adoption.

Our findings contribute to the evidence base surrounding the use of CDS and improvements in patient clinical and financial outcomes. Formal prospective cohort studies and randomized CDS intervention trials, perhaps randomizing providers assigned to receive CDS interventions, should be prioritized to help guide future provider strategies in regard to reducing low-value care.

Takeaway Points.

• Encounters in which providers adhered to all alerts had significantly lower total costs, shorter lengths of stay, a lower probability of 30-day readmissions, and a lower probability of complications compared with nonadherent encounters.

• Full adherence to Choosing Wisely alerts was associated with savings of $944 from a median encounter cost of $12,940.

• Health systems should consider real-time clinical decision support interventions as a method to encourage improved adoption of evidence-based guidelines.