Abstract
Objectives In a randomized controlled trial, we found that applying implementation science (IS) methods and best practices in clinical decision support (CDS) design to create a locally customized, “enhanced” CDS significantly improved evidence-based prescribing of β blockers (BB) for heart failure compared with an unmodified commercially available CDS. At trial conclusion, the enhanced CDS was expanded to all sites. The purpose of this study was to evaluate the real-world sustained effect of the enhanced CDS compared with the commercial CDS.
Methods In this natural experiment of 28 primary care clinics, we compared clinics exposed to the commercial CDS (preperiod) to clinics exposed to the enhanced CDS (both periods). The primary effectiveness outcome was the proportion of alerts resulting in a BB prescription. Secondary outcomes included patient reach and clinician adoption (dismissals).
Results There were 367 alerts for 183 unique patients and 171 unique clinicians (pre: March 2019–August 2019; post: October 2019–March 2020). The enhanced CDS increased prescribing by 26.1% compared with the commercial (95% confidence interval [CI]: 17.0–35.1%), which is consistent with the 24% increase in the previous study. The odds of adopting the enhanced CDS was 81% compared with 29% with the commercial (odds ratio: 4.17, 95% CI: 1.96–8.85). The enhanced CDS adoption and effectiveness rates were 62 and 14% in the preperiod and 92 and 10% in the postperiod.
Conclusion Applying IS methods with CDS best practices was associated with improved and sustained clinician adoption and effectiveness compared with a commercially available CDS tool.
Keywords: clinical decision support systems, heart failure, prescribing, PRISM, RE-AIM, implementation science
Background and Significance
Over the past two decades, there have been numerous advances in drug therapy for the treatment of patients with heart failure and reduced ejection fraction (HFrEF). These medications include β blockers (BB), which have demonstrated improvements in quality of life, rehospitalization, and mortality. 1 2 3 4 5 As such, BBs are recommended for patients with HFrEF. 6 7 8 9 Exceptions to these recommendations are limited to patients with contraindications or who are intolerant of BBs. Unfortunately, prescribing of BBs remains suboptimal, 10 stymying their impact on patient outcomes.
To increase uptake of these medications and their impact, many have tested a variety of different approaches with varying success. These approaches include provider education, direct to patient education and empowerment, 11 new care delivery models, 12 and national efforts such as Get With The Guidelines. 13 However, a rate-limiting step of many approaches is resource availability for initial delivery and ongoing maintenance. In contrast to approaches that require additional personnel to deliver and maintain, automated health IT solutions within the electronic health record (EHR) are alternatives that are receiving increasing attention. 14 15 16
As the technical capacity of EHRs advance and increasingly automate health care workflows, health systems are learning how to effectively implement user-centered health IT solutions. Such solutions include clinical decision support (CDS) tools, which when well designed, can expedite the translation of evidence to practice by providing clinicians with the right information at the right time. 17 Within this evolution of learning how to effectively leverage CDS, there are examples of CDS implemented to improve guideline concordant prescribing for HFrEF. The earliest examples of such CDS demonstrated limited or no change in prescribing behavior, likely because they were constrained by the technical capacity of EHRs at that time. These early examples were also designed with limited consideration of what are now considered best practices in the design of CDS. 18 19 The best practices in CDS design includes a user-centered design process to bolster contextual relevance and careful consideration of implementation issues. 20 21 22 23 However, the best practices provide limited direction on the implementation issues, which can be addressed through the field of implementation science (IS). 24
IS methods aim to improve the systematic uptake of research and evidence-based practices into real-world settings. 25 IS prioritizes the use of pragmatic methods and study designs to develop feasible and sustainable implementation strategies that maximize external relevance and the use of iterative, multilevel partner engagement to promote representativeness and address contextual issues. 25 26 27 28 Central to IS methods is use of a theory, model, or framework to guide systematic assessment of contextual factors and alignment of the intervention with the context to maximize uptake, reproducibility, and sustainability. 29 30 31 Many IS frameworks, including the Practical, Robust Implementation and Sustainability Model (PRISM), 32 33 are broadly applicable across settings and clinical situations; thus, they are used with other criteria or standards specific to the situation at hand. In this case of implementing a CDS to improve HFrEF prescribing, an IS framework helps to integrate the HFrEF clinical practice guidelines and established best practices in CDS design to inform the design of the CDS. 34
In a recent randomized controlled trial, we found that using the PRISM IS framework to inform the design of an “enhanced” CDS resulted in higher rates of BB prescription for patients with HFrEF compared with a traditional or “commercial” CDS (0 vs. 14% led to prescription, p = 0.006). 35 Although these results suggest that a CDS informed by IS principles and methods may be more effective at improving guideline-concordant prescribing, the study had a relatively small sample size (87 patients, 118 clinicians) and analyses did not account for clustering due to clinic or adjust for patient characteristics. The trial was also not able to evaluate the sustained effect of the enhanced CDS over time.
Therefore, we conducted a natural experiment with a larger sample size to better assess the differences between the two CDS tools and to examine the sustained effect of the enhanced CDS. The purpose of this study is to evaluate the real-world effect of applying an IS framework and methods that considers best practices in CDS design compared with a commercially available CDS on the reach, adoption, and sustainment of clinician prescribing for patients with HFrEF.
Methods
Study Design
We conducted a natural experiment that expands the follow-up of our prior cluster randomized trial, 35 where all sites were transitioned to the enhanced CDS tool. Briefly, in the prior trial, 28 primary care clinics were randomized to either a commercial or enhanced CDS using block randomization to allocate each clinic to one of the two CDS. Both CDS were interruptive and designed to alert during a primary care visit and recommend clinicians initiate one of three evidence-based BBs (i.e., metoprolol succinate, carvedilol, bisoprolol) for patients with HFrEF, if not already prescribed. The enhanced CDS was developed using a multilevel user-centered design process informed by the best practices in CDS design and the PRISM IS framework. 34 PRISM guides the systematic assessment and alignment of evidence-based interventions with the multilevel contextual determinants that influence implementation success and sustainability. 32 33 PRISM also includes the widely used Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) evaluation outcomes to inform pragmatic outcome measures. 36 37 The commercial CDS was provided by the EHR vendor (Epic Systems, Verona, Wisconsin, United States), thus to our knowledge was not informed by a user-centered design process or PRISM. A summary of the key differences in design of the two CDS is provided in Table 1 and a detailed summary is described elsewhere. 35
Table 1. Key differences between the commercial and enhanced clinical decision support tools.
| Design feature | Commercial CDS tool | Enhances CDS tool |
|---|---|---|
| Inclusion criteria | Adults with any ICD-9/10 diagnosis for HF and an ejection fraction </= 40% No documented allergy to a BB or β-agonist (relied on vendor supplied knowledge content, which erroneously included β-agonists with BB) |
Adults with an ICD-9/10 diagnosis that explicitly states the EF is </= 40% or an ejection fraction </= 40% No documented BB allergy (was curated by clinician informaticists to ensure accuracy/completeness) |
| Format and timing | Interruptive pop-up at time of opening encounter | |
| Response options (time period that a future alert is delayed for) | Open an order set with many options including order medications, laboratories, referrals, echocardiogram or Indicate “contraindicated” (90 d), “cost concern” (90 d), “patient declines” (90 d) or Dismiss option |
Pend an order for a BB at staring doses or Indicate “never appropriate” (>20 y), “remind me later (1 mo)” (28 d), “provide comment” (28 d) |
| Informational content displayed | Most recent EF, blood pressure, heart rate BB recommended Rationale for recommendation with link to more information Guidance of which BB are evidence-based and that asthma/chronic obstructive pulmonary disease are not contraindications |
Most recent EF BB recommended |
Abbreviations: BB, β blockers; CDS, clinical decision support; EF, ejection fraction; HF, heart failure.
At the conclusion of the randomized controlled trial, the health system decided to discontinue the commercial CDS and switch those 14 clinics to the customized CDS. Details of the prior randomized trial, study population and both CDS interventions are described elsewhere. 35 This study evaluates the real-world effect of the enhanced CDS when expanded to all 28 clinics.
Fig. 1 summarizes the different exposure groups over time.
Fig. 1.
Overview of the exposure groups over time. CDS, clinical decision support.
Data Collection
We manually reviewed EHR records to collect data on patient characteristics, characteristics of health care utilization, and identify potential reasons why a clinician may not have prescribed a BB. We did not rely on audit logs to determine whether a medication was prescribed. To identify potential reasons, unstructured clinical documentation and clinician comments in response to the enhanced CDS alert were evaluated. Clinicians were not permitted to provide comments in response to the commercial CDS. Following a rapid qualitative analysis approach, three investigators (L.W., V.W., and K.E.T.) iteratively and inductively discussed and defined themes of responses, which informed the development of a structured template. This structured template was then used to assignment each unstructured data element into a theme. 38 The manual review of unstructured documentation and comments was conducted by two independent reviewers with clinical training (L.W. and V.W.). A second independent investigator with advanced clinical pharmacy training (K.E.T.) validated a 20% sample of the manually collected data and reviewed the assignment of every unstructured data element into a theme. Data were also collected from automated audit logs that monitor CDS activity, including clinician-stated actions in response to the CDS alerts (buttons clicked). All data from the original randomized trial that were collected from the health system's virtual data warehouse were recollected via manual chart review to facilitate consistency of comparisons.
Outcome Measures
Outcomes were informed by PRISM's RE-AIM outcome measures. 36 RE-AIM outcomes of implementation and maintenance were outside the scope of this study.
The primary effectiveness outcome measure was the proportion of alerts resulting in prescription of an evidence-based BB when indicated. Possible reasons for not prescribing a BB were identified by reviewing patient medical records and comments clinicians made in response to the alerts. Secondary effectiveness outcomes included safety, specifically instances of bradycardia (i.e., heart rate <50 bpm), hypotension (i.e., blood pressure < 90/60), acute heart failure (HF) exacerbation requiring hospitalization of emergency department visits, and unintended consequences such as duplicate therapy during the 1-month period after each alert.
Other secondary outcome measures included patient reach and clinician adoption. Reach was measured as the number of alerts: (1) overall, (2) for unique patients, (3) for unique visits, and (4) for unique clinicians. Adoption was measured as the proportion of clinicians who responded to the alert. For the definition of adoption, clinician response was defined as doing something other than outright dismissing the alert. In the case of the enhanced CDS that did not have a “dismiss” option and would default to pending a BB order if “accept” was selected, adoption included instances when a clinician selected any of the response options or ordered an evidence-based medication. When a clinician initially selected “accept” in response to the enhanced CDS alert and then canceled the pended BB, this was considered nonadoption. More details and rationale for how adoption was measured are described elsewhere. 35 Adoption was also summarized based on the number of unique patients and clinicians for which the CDS alerted. Representativeness of reach, adoption, and effectiveness was evaluated by descriptively comparing outcomes across patient gender, race, and ethnicity. All outcomes were measured at the alert level.
Analytic Plan
Descriptive statistics were produced for each of the four exposure groups (A–D illustrated in Fig. 1 ). The four exposure groups were then collapsed into two treatment groups: commercial (exposure group A) or enhanced (exposure groups B–D). Demographic and baseline characteristics were compared between treatment groups (enhanced vs. commercial) using the chi-square test or Fisher's exact test for categorical variables, and Wilcoxon rank sum test for continuous or discrete variables. A multivariable logistic regression model was used to estimate the odds of adoption comparing enhanced versus commercial. A multivariable linear model was used to estimate the difference in effectiveness between enhanced and commercial alerts. Both models used a generalized estimating equation approach with an exchangeable correlation structure to account for clustering of patients within clinics and adjusted for repeated measures within clinics, time (pre vs. post), diagnosis of chronic obstructive pulmonary disease, and clinician type. All analyses were conducted using SAS software (version 9.4, SAS Institute Inc.).
Results
Patient Reach
The two CDS alerted 367 times for 183 unique patients and for 171 unique clinicians: the commercial CDS alerted 59 times for 26 unique patients and 24 unique clinicians; and the enhanced CDS alerted 308 times for 169 unique patients and 155 unique clinicians. Table 2 describes the baseline characteristics of the study population. Specific baseline characteristics to highlight include: the mean age of patients exposed to one of the two alerts was 73.9 (13) years, 34.4% were female, 13.7% were non-White and 81.4% had Medicare insurance. When considering the distribution of treatment group, there were no significant differences based on patient race, ethnicity, or gender. There were significant differences in treatment group based on patient age (79.4 [15.1] years assigned to commercial vs. 72.9 [12.5] years assigned to enhanced; p = 0.02), diagnosis of chronic obstructive pulmonary disease (26.9% commercial vs. 11.5% enhanced; p = 0.03), and type of clinician (96.2% commercial vs. 69.3% enhanced alerted for an attending physician; p = 0.016). As described in Table 3 , there was a 34% increase in patient reach in the postperiod for clinics that had the enhanced CDS in the preperiod and an 85% increase for clinics that transitioned from commercial CDS to enhanced CDS.
Table 2. Baseline characteristics for unique patients exposed to the alerts.
| Characteristic | Commercial alert ( n = 26) | Enhanced alert ( n = 169) |
p- Value |
|---|---|---|---|
| Age (y), mean (SD) | 79 (15) | 73 (13) | 0.02 |
| Male, n (%) | 19 (73) | 123 (64) | 0.38 |
| White, n (%) | 22 (85) | 168 (88) | 0.58 |
| Hispanic, n (%) | 2 (8) | 12 (6) | 0.79 |
| Medicare, n (%) | 23 (88) | 156 (82) | 0.39 |
| Clinician type: attending physician, n (%) | 25 (96) | 131 (69) | 0.02 |
| Heart rate, mean (SD) | 74 (15) | 79 (14) | 0.12 |
| Heart rate < 50, n (%) | 0 (0) | 3 (2) | 0.52 |
| Last heart rate < 50, n (%) | 1 (4) | 7 (4) | 0.96 |
| Blood pressure < 90/60, n (%) | 1 (4) | 4 (2) | 0.58 |
| Last blood pressure < 90/60, n (%) | 0 (0) | 11 (6) | 0.21 |
| ≥1 visit with cards a in past 1 y, n (%) | 19 (73) | 120 (63) | 0.31 |
| ≥1 visit with cards in past 2 y, n (%) | 21 (81) | 134 (70) | 0.26 |
| Past BB, ever, n (%) | 18 (69) | 145 (76) | 0.46 |
| BB allergy per chart review b , n (%) | 0 (0) | 4 (2) | 0.46 |
| Nonevidence-based BB prescribed, n (%) | 4 (15) | 12 (7) | 0.24 |
| Prescribed metoprolol tartrate, n (%) | 8 (31) | 51 (27) | 0.66 |
| Prescribed ACE, ARB, or ARNI, n (%) | 15 (58) | 88 (46) | 0.27 |
| Prescribed angiotensin receptor-neprilysin inhibitor, n (%) | 0 (0) | 4 (2) | 0.46 |
| Prescribed mineralocorticoid receptor antagonist, n (%) | 5 (19) | 36 (19) | 0.96 |
| Prescribed sodium-glucose cotransporter 2 inhibitor, n (%) | 0 (0) | 1 (1) | 0.71 |
| Prescribed nondihydropyridine calcium channel blocker, n (%) | 1 (4) | 10 (5) | 0.76 |
| Chronic obstructive pulmonary disease, n (%) | 7 (27) | 22 (12) | 0.03 |
| Asthma, n (%) | 1 (4) | 16 (8) | 0.42 |
| Coronary artery disease c , n (%) | 15 (58) | 108 (57) | 0.91 |
| Atrial fibrillation or atrial flutter, n (%) | 16 (62) | 89 (47) | 0.15 |
| Type 2 diabetes mellitus, n (%) | 8 (31) | 52 (27) | 0.71 |
Abbreviations: ACE, angiotensin converting enzyme inhibitor; ARB, angiotensin receptor blocker; ARNI, angiotensin receptor–neprilysin inhibitor; CDS, clinical decision support; SD, standard deviation.
Notes: Of the 183 unique patients, 12 were exposed to both CDS. Significant differences are bolded.
Cards: outpatient cardiology provider.
These patients were incorrectly not excluded from the alert.
Coronary artery disease includes myocardial infarction, percutaneous coronary intervention, bypass, coronary artery disease, and angioplasty.
Table 3. Patient reach, clinician adoption, and effectiveness of changing prescribing across exposure groups.
| Outcome |
Commercial CDS preperiod (Exposure A)
|
Enhanced CDS postperiod (Exposure C) |
Enhanced CDS preperiod (Exposure B)
|
Enhanced CDS postperiod (Exposure D) |
|---|---|---|---|---|
| Patient reach unique patients alerted, n |
26 | 48 | 61 | 82 |
| # of total alerts, n | 59 | 65 | 106 | 137 |
| Adoption alerts not dismissed, n (%) |
17 (29%) | 61 (94%) | 66 (62%) | 124 (91%) |
| Effective alerts resulting in BB prescription, n (%) |
0 (0%) | 5 (8%) | 15 (14%) | 13 (10%) |
Abbreviations: BB, β blockers; CDS, clinical decision support.
Note: Effectiveness rates were calculated based on those alerts that were adopted.
Clinician Adoption
After adjusting for repeated measures within clinics, time (pre vs. post), diagnosis of chronic obstructive pulmonary disease, and clinician type, the odds of adoption were 4.17 (95% confidence interval [CI]: 1.96–8.85) times as high for the enhanced CDS compared with the commercial CDS. Additionally, time (pre vs. post) was significantly associated with the odds of adoption, with alerts occurring in the postperiod at more than 7 times greater odds (odds ratio: 7.07; 95% CI: 4.07–12.26) of adoption compared with alerts in the preperiod.
From Table 3 , the absolute increase in adoption for clinics that received the enhanced CDS in the preperiod was 29%. Of the 13 patients for whom commercial alerts were not adopted, 3 patients were then exposed to the enhanced CDS and all 3 enhanced alerts (100%) were adopted for these patients. Across all 38 patients exposed to one of the two CDS that were not adopted in the preperiod, 26 were later exposed to the enhanced CDS in the postperiod and 10 (38%) of these enhanced alerts were adopted.
Effectiveness of Changing Prescribing
After adjusting for repeated measures within clinics, time (pre vs. post), diagnosis of chronic obstructive pulmonary disease, and clinician type, the increase in effectiveness for the enhanced CDS was 26.1% (95% CI: 17.0–35.1%) higher compared with the commercial CDS. Additionally, clinician type was significantly associated with effectiveness, with physicians (attendings, resident, fellows) having a 10.7% increase in alerts that were effective compared with advanced practice providers. Alerts in the postperiod (exposure groups C and D) were associated with a decrease in effectiveness compared with alerts in the preperiod (exposure groups A and B; −13.9% CI: −3.1 to −24.7%).
From Table 3 , there was an absolute decrease in effectiveness for clinics exposed to the enhanced CDS in the preperiod (14–10%). Of the 26 patients exposed to the commercial CDS in the preperiod (none were effective), there were 6 patients who were later exposed to the enhanced CDS and none (0%) were prescribed a BB. Across all 38 patients exposed to one of the two CDS that was not effective in the preperiod, 26 were later exposed to the enhanced CDS in the postperiod and 1 patient (3.9%) was prescribed a BB with the enhanced alert.
Reasons for Not Prescribing
Although the commercial alert did not allow clinicians to provide comments, clinicians provided 68 comments after receiving the enhanced CDS. Based on the clinician comments in response to the enhanced CDS and chart review of clinical documentation, the most common reasons clinicians appear to have avoided prescription of a BB was a concern of an adverse event or because the patient was either referred to or being managed by a cardiologist. Table 4 describes potential reasons clinicians may have avoided prescribing a BB.
Table 4. Summary of β blockers intolerance or reason β blockers was not appropriate per clinical documentation or clinician comments in response to alerts, n (%) .
| Reason provided | “Exposure A” commercial CDS n = 59 alerts |
“Exposure B, C, or D” enhanced CDS n = 275 a alerts that did not result in prescription of a BB |
|---|---|---|
| Documented reason in clinical notes or in comments in response to CDS (total across alerts) | 33 (55.9) | 161 (58.5) |
| Comments provided in response to CDS (total across alerts) | N/A a | 68 (24.7) |
| Reasons based on comment or documented in clinical notes | ||
| Seeing cards or referred to cards | 0 (0) | 47 (17.1) |
| History of adverse drug event | 7 (21.2) | 35 (12.7) |
| Other reason | 5 (15.15) | 16 (5.8) |
| Ejection fraction >40% | 2 (6.1) | 15 (5.5) |
| Decompensated/HF exacerbation | 5 (15.2) | 7 (2.5) |
| Conduction disorder or device placement planned | 4 (12.1) | 6 (2.2) |
| Defer to primary care provider | 0 (0) | 8 (2.9) |
| On nonevidence-based BB | 0 (0) | 8 (2.9) |
| Concern for hypotension and BP ≥ 90/60 | 2 (6.1) | 6 (2.2) |
| Concern for hypotension and BP < 90/60 | 1 (3) | 6 (2.2) |
| Patient refuses | 3 (9.1) | 4 (1.5) |
| Concern for bradycardia and HR ≥ 50 | 4 (12.1) | 2 (0.7) |
| Concern for bradycardia and HR < 50 | 0 (0) | 1 (0.4) |
Abbreviations: BB, β blockers; BP, blood pressure; CDS, clinical decision support; HF, heart failure; HR, heart rate; N/A, not applicable.
Note: Table represents those CDS alerts that did not result in prescription of an evidence-based BB.
The commercial CDS was not enabled to allow clinicians to provide comments.
Representativeness of Outcomes
Table 5 describes the representativeness of outcomes by gender, race, and ethnicity. The CDS alerted for more patients that were male, White, and non-Hispanic. Clinician adoption and effectiveness rates were incrementally higher for female patients. Although the adoption rate was incrementally higher for non-White patients, effectiveness rates were not higher for non-White patients.
Table 5. Representativeness of reach, adoption and effectiveness outcomes based on patient gender, race, and ethnicity.
| Patient gender | Patient race | Patient ethnicity | ||||
|---|---|---|---|---|---|---|
| Male | Female | White patient | Non-White patients | Hispanic | Non-Hispanic | |
|
Alerts that fired,
n
(reach) |
258 | 114 | 316 | 53 | 17 | 352 |
| Alerts that were adopted, n (%) | 182 (71) | 91 (80) | 229 (72) | 41 (77) | 13 (76) | 257 (73) |
| Alerts that were effective, n (%) | 18 (7) | 15 (13) | 29 (9) | 3 (6) | 2 (12) | 30 (9) |
Safety Outcomes
With respect to safety outcomes, there were no documented instances of bradycardia, hypotension, or acute HF exacerbation requiring hospitalization or emergency department visits in patients prescribed a BB. As previously reported, 35 there were two near misses in the preperiod, in which clinicians inadvertently prescribed a BB in response to the enhanced CDS, but the medication never reached the patient. Manual chart review did not reveal any new instances of near misses in the postperiod.
Discussion
This study provides further empirical support that application of IS principles and methods with CDS design best practices can improve adoption and effectiveness of CDS tools and also suggests this effect is sustained over time. In our study, the enhanced CDS was informed by IS to be contextually relevant to the local health system and significantly improved guideline-concordant prescribing of BBs for patients with HFrEF compared with a commercially available CDS. The current study validates the findings of our earlier randomized controlled trial 35 by addressing some of its potential limitations. Although the original study was randomized, the sample size was relatively small, which limited the ability to adjust for patient characteristics and within-clinic clustering in analyses. In the present study, we extended the follow-up beyond the original randomized controlled trial to conduct comparisons within the same study groups with increased sample size and longer follow-up. Our use of a natural experimental design as a pragmatic follow-up method is a strength and not only validates the findings of the original randomized controlled trial, but also facilitates examination of the long-term sustained effects of the enhanced CDS tool. Thoughtful natural experiments can facilitate a timely and low-cost method to observe the effects of interventions in generalizable, real-world scenarios without threatening internal validity. 39
In the present study, we observed significantly higher rates of clinician adoption with the enhanced CDS compared with the commercial CDS. In fact, the rate of adoption exceeded 90% with the enhanced CDS groups during the postperiod. This finding is especially notable when considering that the number of unique patients reached by an alert and the total number of alerts fired significantly increased during the postperiod as well. Despite receiving more frequent alerts, clinicians were more likely to adopt the enhanced CDS. Some features of the enhanced CDS design that we believe contributed to this high adoption rate include linking to specific medication orders rather than a broad order set and defaulting the ordering of an evidence-based BB and starting dose. We also made it more difficult to ignore the recommendation by removing the “dismiss” button. To bypass the recommendation, the clinician would need to manually deselect the medication order within the alert or deliberately cancel the pended order after clicking “accept.” These features align with current best practices in CDS design and nudge theory, which encourage leveraging small changes in alert presentation to drive clinician behavior and minimize the effort and time required to perform the recommended action. 17 34 40
Historically, medication-related commercial CDS have been associated with high dismissal rates (low adoption) up to 96%. 40 41 In our study, adoption rates of the enhanced CDS were substantially higher (62% in the preperiod and >90% in the postperiod), which suggests a positive return on investment when IS principles and methods are used with CDS design best practices to customize CDS to the local context. Interestingly, the adoption rate of the enhanced CDS increased from 62% in the preperiod to >90% in the postperiod despite no changes to its design. Although the frequency of a given clinician seeing the enhanced CDS more than once was low, one possible explanation may be related to shifts in clinician-perceived usefulness of CDS overall, which may have resulted in clinicians being more likely to adopt instead of outright dismiss the CDS; however, this effect has not yet been observed in other CDS situations at our insitution. The finding of high adoption rates is important, given it is a rate-limiting step for our primary outcome of effectiveness. Unless clinicians pay attention to the CDS (adopt it), effectiveness is not possible. Our application of IS principles and methods prioritized alignment of the enhanced CDS design to meet the needs and preferences of clinicians, which may explain the high rates of adoption that persisted over the study period.
Previous findings of effectiveness were also sustained over time. We found a 26.1% increase in effectiveness with the enhanced CDS, which is consistent with the 24% increase found in the 2021 study. Although the overall effectiveness of the enhanced CDS decreased in the postperiod, this may be explained by saturation of patients eligible for BB therapy for which the CDS recommendation was deemed relevant (e.g., due to patient preference or other reasons). Among the 130 patients in the postperiod, 20% (26 patients) were observed in both time periods and only 1 was prescribed a BB in the postperiod, which may suggest that decisions to not prescribe an evidence-based BB as a result of the CDS during the preperiod were maintained in the postperiod. Given the increase in total number of alerts in the postperiod, we considered the possibility that alert fatigue could explain the change in effectiveness; however, this is contradicted by the fact that clinician adoption increased from the pre- to post time period.
Over time, CDS design best practices, including user-centered approaches are increasingly being employed when developing CDS, 16 19 42 43 but few have formally applied IS principles and methods. 34 35 44 The recent PRagmatic trial Of Messaging to Providers about Treatment of Heart Failure (PROMPT-HF) study applied a user-centered design approach to create CDS for four categories of guideline-recommended HFrEF medications to be used in both cardiology and primary care practices. 16 PROMPT-HF found a significant increase of >40% in guideline-concordant prescribing compared with no CDS. When specifically considering changes in BB prescribing, PROMPT-HF found a 3% improvement with their CDS. In contrast, we found that 20% of patients who received the enhanced alert were prescribed a BB. Differences in prescribing rates could be explained by the differences in the designs of the CDS or our application of IS principles and methods. Interestingly, in our mixed methods evaluation, clinicians generally preferred focused recommendations over more comprehensive order sets that were included in PROMPT-HF's CDS. However, this preference may have been driven by other design features of the CDS that made the order set less appealing; designed differently, the order set may have been more appealing in our study.
Our previous study and this continuation study present an emerging approach to combining IS methods with established best practices in CDS design. This approach enhances the traditional user-centered design process to include additional perspectives beyond the intended end user and to systematically consider how these perspectives dynamically interact with other contextual factors (e.g., guidelines, best practices, resources available). While the initial application of IS methods and CDS best practices may require additional resources compared with the implementation of commercially available CDS tools, our findings support that resource utilization at the design or planning stage can result in significant improvements and sustainment of clinician adoption and guideline-concordant prescribing practices. Further, by applying IS, the reproducibility, scalability, relevance, and representativeness of CDS implementation can be elevated. An important aspect of PRISM and other IS frameworks is the consideration of the sustainability, representativeness, and long-term population health impact of the intervention from multiple perspectives. 26 27 28 45
There are several limitations of this study. First, our study focused on clinics in the primary care setting at a single health system for one clinical situation (HF prescribing) with one commercial CDS and may not be generalizable to all health systems, clinical situations, or all commercially available CDS. Although the natural experimental design increases external validity, challenges to this design are the risk of selection bias or selective exposure to the intervention and inability to control for temporal changes. 46 To address these risks, we controlled for potential confounders that may influence a patient's indication for BB therapy and a clinician's adoption of the CDS tool in our analysis. Further, this study aimed to validate or refute the findings from the original randomized controlled trial, which was designed to control for potential confounders and issues related to study group imbalance. There were also outcomes that we were not able to evaluate robustly such as equity and cost of implementation of the enhanced CDS. Although we assessed the RE-AIM issues of representativeness of reach, adoption, and effectiveness outcomes across different demographic groups, the study was not powered to test for statistical significance and the CDS were not designed to consider differences based on key social determinants. We did find a clinically significant difference in the number of alerts based on sex, which is likely due to a greater proportion of men with HFrEF, but more in-depth analysis of the implications of this difference is warranted. Similarly, additional investigation of the reason for and implications of the customized CDS alerting for more attending physicians is warranted. Given limitations of EHR data, including lack of structured EHR data to accurately identify patients without contraindications to BB prescription, 47 48 we were unable to accurately report the proportion of alerts that fired for patients who are true candidates for BB therapy; thus, we instead described the reach outcome of the PRISM/RE-AIM framework as the number of alerts overall.
Future studies should evaluate the effect of applying IS to promote evidence-based care when designing CDS for other clinical situations and compared with other commercially available CDS. Such studies should also proactively consider equity and unintended consequences in the design and evaluation of CDS tools. In addition, the cost of using this IS approach is an important measure of scalability to other CDS within and external to our health system. In future work, we plan to evaluate the cost of implementing CDS using this IS-based approach using a time-driven activity-based cost approach, 49 given the bulk of the cost is in personnel time. Another important area of future research is exploring ways to increase the specificity of such alerts by leveraging natural language processing systems to transform data from unstructured clinical narratives into structured data elements, which can also be used to more robustly evaluate reach. 50
Future CDS should also consider methods to address primary care hesitance to intensify therapy when patients are comanaged by cardiology. In our study, comanagement with or deferral to cardiology was identified as a common reason for primary care clinicians opting to not follow the CDS recommendation to initiate an evidence-based BB. Given how we identified comanagement as a reason, our estimates of its influence are likely under representative of the true incidence. Others have also identified comanagement as a deterrent to optimizing guideline-concordant prescribing. 51 CDS design features to address this deterrent could include functions that facilitate ease of communication across specialties. Such features could support ease of e-consults or include automated cross-specialty notifications regarding medication changes or reasons a change was not made.
Conclusion
In this natural experiment, applying IS methods with CDS best practices to design a customized CDS tool resulted in improved and sustained clinician adoption and effectiveness compared with a commercially available CDS tool. Future research is needed to replicate and test the generalizability of this approach and to evaluate the value of additional resources needed to develop CDS tools that are aligned with the local context and adhere to CDS design best practices.
Clinical Relevance Statement
Applying IS methods with best practices in CDS may result in improved and sustained clinician adoption and effectiveness compared with commercially available CDS tools. Others seeking to implement CDS solutions may benefit from customizing CDS tools to meet the needs and preferences of the local context while also considering best practices in CDS design versus relying on commercially available CDS tools that are designed for the general or average needs of health systems.
Multiple-Choice Questions
-
Which one of the following is a potential outcome of applying IS principles and methods to the design and implementation of CDS tools?
Increased likelihood of sustaining or maintaining the effect
Increased likelihood of developing a cost-effective tool
Decreased likelihood of scaling the tool to other settings
Decreased likelihood of gaining clinician buy-in
Correct Answer: The correct answer is option a.
-
Which of the following is true regarding the use of IS principles and methods for CDS tools?
Does not provide guidance on outcome measures
Does not require additional resources
Does consider representativeness and equity
Does result in improved outcomes in all settings
Correct Answer: The correct answer is option c.
Funding Statement
Funding K.E.T. was supported in part by the National Heart, Lung, and Blood Institute (grant nos.: K12HL137862 and 1K23HL161352).
Conflict of Interest None declared.
Protection of Human and Animal Subjects
The study was reviewed by the Institutional Review Board and deemed exempt and a full waiver of Health Insurance Portability and Accountability Act authorization was approved.
References
- 1.Carvedilol Or Metoprolol European Trial Investigators Poole-Wilson P A, Swedberg K, Cleland J GFet al. Comparison of carvedilol and metoprolol on clinical outcomes in patients with chronic heart failure in the Carvedilol Or Metoprolol European Trial (COMET): randomised controlled trial Lancet 2003362(9377):7–13. [DOI] [PubMed] [Google Scholar]
- 2.The Cardiac Insufficiency Bisoprolol Study (CIBIS). CIBIS Investigators and Committees . Lechat P. A randomized trial of beta-blockade in heart failure. Circulation. 1994;90(04):1765–1773. doi: 10.1161/01.cir.90.4.1765. [DOI] [PubMed] [Google Scholar]
- 3.MERIT-HF Study Group . Goldstein S, Fagerberg B, Hjalmarson A et al. Metoprolol controlled release/extended release in patients with severe heart failure: analysis of the experience in the MERIT-HF study. J Am Coll Cardiol. 2001;38(04):932–938. doi: 10.1016/s0735-1097(01)01516-9. [DOI] [PubMed] [Google Scholar]
- 4.U.S. Carvedilol Heart Failure Study Group . Packer M, Bristow M R, Cohn J N et al. The effect of carvedilol on morbidity and mortality in patients with chronic heart failure. N Engl J Med. 1996;334(21):1349–1355. doi: 10.1056/NEJM199605233342101. [DOI] [PubMed] [Google Scholar]
- 5.Carvedilol Prospective Randomized Cumulative Survival Study Group . Packer M, Coats A J, Fowler M B et al. Effect of carvedilol on survival in severe chronic heart failure. N Engl J Med. 2001;344(22):1651–1658. doi: 10.1056/NEJM200105313442201. [DOI] [PubMed] [Google Scholar]
- 6.ESC Scientific Document Group . Ponikowski P, Voors A A, Anker S D et al. 2016 ESC Guidelines for the diagnosis and treatment of acute and chronic heart failure: the Task Force for the diagnosis and treatment of acute and chronic heart failure of the European Society of Cardiology (ESC) Developed with the special contribution of the Heart Failure Association (HFA) of the ESC. Eur Heart J. 2016;37(27):2129–2200. doi: 10.1093/eurheartj/ehw128. [DOI] [PubMed] [Google Scholar]
- 7.ESC Scientific Document Group . McDonagh T A, Metra M, Adamo M et al. 2021 ESC guidelines for the diagnosis and treatment of acute and chronic heart failure. Eur Heart J. 2021;42(36):3599–3726. doi: 10.1093/eurheartj/ehab368. [DOI] [PubMed] [Google Scholar]
- 8.Yancy C W, Jessup M, Bozkurt B et al. 2017 ACC/AHA/HFSA focused update of the 2013 ACCF/AHA guideline for the management of heart failure: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines and the Heart Failure Society of America. Circulation. 2017;136(06):e137–e161. doi: 10.1161/CIR.0000000000000509. [DOI] [PubMed] [Google Scholar]
- 9.Heidenreich P A, Bozkurt B, Aguilar D et al. 2022 AHA/ACC/HFSA guideline for the management of heart failure: a report of the American College of Cardiology/American Heart Association Joint Committee on Clinical Practice Guidelines. Circulation. 2022;145(18):e895–e1032. doi: 10.1161/CIR.0000000000001063. [DOI] [PubMed] [Google Scholar]
- 10.Greene S J, Butler J, Albert N M et al. Medical therapy for heart failure with reduced ejection fraction: the CHAMP-HF registry. J Am Coll Cardiol. 2018;72(04):351–366. doi: 10.1016/j.jacc.2018.04.070. [DOI] [PubMed] [Google Scholar]
- 11.Allen L A, Venechuk G, McIlvennan C K et al. An electronically delivered patient-activation tool for intensification of medications for chronic heart failure with reduced ejection fraction: the EPIC-HF trial. Circulation. 2021;143(05):427–437. doi: 10.1161/CIRCULATIONAHA.120.051863. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Bhatt A S, Varshney A S, Nekoui M et al. Virtual optimization of guideline-directed medical therapy in hospitalized patients with heart failure with reduced ejection fraction: the IMPLEMENT-HF pilot study. Eur J Heart Fail. 2021;23(07):1191–1201. doi: 10.1002/ejhf.2163. [DOI] [PubMed] [Google Scholar]
- 13.Get With The Guidelines - Heart Failure | American Heart Association. Accessed April 22, 2022 at:https://www.heart.org/en/professional/quality-improvement/get-with-the-guidelines/get-with-the-guidelines-heart-failure
- 14.Kao D P, Trinkley K E, Lin C T. Heart failure management innovation enabled by electronic health records. JACC Heart Fail. 2020;8(03):223–233. doi: 10.1016/j.jchf.2019.09.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Mukhopadhyay A, Reynolds H R, Phillips L M et al. Cluster-randomized trial comparing ambulatory decision support tools to improve heart failure care. J Am Coll Cardiol. 2023;81(14):1303–1316. doi: 10.1016/j.jacc.2023.02.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Ghazi L, Yamamoto Y, Riello R J et al. Electronic alerts to improve heart failure therapy in outpatient practice: a cluster randomized trial. J Am Coll Cardiol. 2022;79(22):2203–2213. doi: 10.1016/j.jacc.2022.03.338. [DOI] [PubMed] [Google Scholar]
- 17.Kawamoto K, Houlihan C A, Balas E A, Lobach D F.Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success BMJ 2005330(7494):765. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Tierney W M, Overhage J M, Murray M D et al. Effects of computerized guidelines for managing heart disease in primary care. J Gen Intern Med. 2003;18(12):967–976. doi: 10.1111/j.1525-1497.2003.30635.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Blecker S, Austrian J S, Horwitz L I et al. Interrupting providers with clinical decision support to improve care for heart failure. Int J Med Inform. 2019;131:103956. doi: 10.1016/j.ijmedinf.2019.103956. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Bates D W, Kuperman G J, Wang S et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003;10(06):523–530. doi: 10.1197/jamia.M1370. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Osheroff J, Teich J, Levick D, Saldana L, Velasco F, Sittig D. 2nd ed. Healthcare Information Management Systems Society (HIMSS); 2012. Improving Outcomes with Clinical Decision Support: An Implementers Guide. [Google Scholar]
- 22.Horsky J, Schiff G D, Johnston D, Mercincavage L, Bell D, Middleton B. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform. 2012;45(06):1202–1216. doi: 10.1016/j.jbi.2012.09.002. [DOI] [PubMed] [Google Scholar]
- 23.Sittig D F, Wright A, Osheroff J A et al. Grand challenges in clinical decision support. J Biomed Inform. 2008;41(02):387–392. doi: 10.1016/j.jbi.2007.09.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Trinkley K E, Kahn M G, Bennett T D et al. An implementation science approach to clinical decision support within electronic health records: integrating PRISM with CDS design best practices (preprint) J Med Internet Res. 2020;22(10):e19676. doi: 10.2196/19676. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Brownson R, Colditz G, Proctor E. 2nd ed. Oxford University Press; 2018. Dissemination and Implementation Research in Health: Translating Science to Practice. [Google Scholar]
- 26.Kwan B M, Brownson R C, Glasgow R E, Morrato E H, Luke D A. Designing for dissemination and sustainability to promote equitable impacts on health. Annu Rev Public Health. 2022;43(01):331–353. doi: 10.1146/annurev-publhealth-052220-112457. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Shelton R C, Lee M.Sustaining evidence-based interventions and policies: recent innovations and future directions in implementation science Am J Public Health 2019109(S2):S132–S134. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Shelton R C, Chambers D A, Glasgow R E. An extension of RE-AIM to enhance sustainability: addressing dynamic context and promoting health equity over time. Front Public Health. 2020;8:134. doi: 10.3389/fpubh.2020.00134. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Tabak R G, Khoong E C, Chambers D, Brownson R C.Models in dissemination and implementation research: useful tools in public health services and systems research. 2013. Accessed March 18, 2015 at:http://uknowledge.uky.edu/cgi/viewcontent.cgi?article=1012&context=frontiersinphssr&sei-redir=1&referer=http://www.bing.com/search?q=Models+in+dissemination+and+implementationresearch%3A+useful+tools+in+public+health+services+andsystems+research&form=DLRD
- 30.Tabak R G, Khoong E C, Chambers D A, Brownson R C. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012;43(03):337–350. doi: 10.1016/j.amepre.2012.05.024. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53. doi: 10.1186/s13012-015-0242-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Feldstein A C, Glasgow R E. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. 2008;34(04):228–243. doi: 10.1016/s1553-7250(08)34030-6. [DOI] [PubMed] [Google Scholar]
- 33.Rabin B A, Cakici J, Golden C A, Estabrooks P A, Glasgow R E, Gaglio B. A citation analysis and scoping systematic review of the operationalization of the Practical, Robust Implementation and Sustainability Model (PRISM) Implement Sci. 2022;17(01):62. doi: 10.1186/s13012-022-01234-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Trinkley K E, Kahn M G, Bennett T D et al. Integrating the practical robust implementation and sustainability model with best practices in clinical decision support design: implementation science approach. J Med Internet Res. 2020;22(10):e19676. doi: 10.2196/19676. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Trinkley K E, Kroehl M E, Kahn M G et al. Applying clinical decision support design best practices with the practical robust implementation and sustainability model versus reliance on commercially available clinical decision support tools: randomized controlled trial. JMIR Med Inform. 2021;9(03):e24359. doi: 10.2196/24359. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Glasgow R E, Harden S M, Gaglio Bet al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review Front Public Health 20197(MAR):64. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Glasgow R E, Vogt T M, Boles S M. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(09):1322–1327. doi: 10.2105/ajph.89.9.1322. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Lewinski A A, Crowley M J, Miller C et al. Applied rapid qualitative analysis to develop a contextually appropriate intervention and increase the likelihood of uptake. Med Care. 2021;59 03:S242–S251. doi: 10.1097/MLR.0000000000001553. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Desai S, Roberts E. Leveraging natural experiments to evaluate interventions in learning health systems. BMJ Qual Saf. 2021;30(03):183–185. doi: 10.1136/bmjqs-2019-010757. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Nanji K C, Slight S P, Seger D L et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014;21(03):487–491. doi: 10.1136/amiajnl-2013-001813. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Poly T N, Islam M M, Yang H C, Li Y J. Appropriateness of overridden alerts in computerized physician order entry: systematic review. JMIR Med Inform. 2020;8(07):e15653. doi: 10.2196/15653. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Stettner S, Adie S, Hanigan S, Thomas M, Pogue K, Zimmerman C. Effect of replacing vendor QTc alerts with a custom QTc risk alert in inpatients. Appl Clin Inform. 2022;13(01):19–29. doi: 10.1055/s-0041-1740483. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Green L A, Nease D, Jr, Klinkman M S. Clinical reminders designed and implemented using cognitive and organizational science principles decrease reminder fatigue. J Am Board Fam Med. 2015;28(03):351–359. doi: 10.3122/jabfm.2015.03.140243. [DOI] [PubMed] [Google Scholar]
- 44.Shakowski C, Page Ii R L, Wright G et al. Comparative effectiveness of generic commercial versus locally customized clinical decision support tools to reduce prescription of nonsteroidal anti-inflammatory drugs for patients with heart failure. J Am Med Inform Assoc. 2023:ocad109. doi: 10.1093/jamia/ocad109. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Rabin B, Brownson R. 2nd ed. Oxford University Press; 2018. Developing terminology for dissemination and implementation research; pp. 19–45. [Google Scholar]
- 46.Craig P, Cooper C, Gunnell D et al. Using natural experiments to evaluate population health interventions: new Medical Research Council guidance. J Epidemiol Community Health. 2012;66(12):1182–1186. doi: 10.1136/jech-2011-200375. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Nadkarni P M. Drug safety surveillance using de-identified EMR and claims data: issues and challenges. J Am Med Inform Assoc. 2010;17(06):671–674. doi: 10.1136/jamia.2010.008607. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Carrell D S, Schoen R E, Leffler D A et al. Challenges in adapting existing clinical natural language processing systems to multiple, diverse health care settings. J Am Med Inform Assoc. 2017;24(05):986–991. doi: 10.1093/jamia/ocx039. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Huebschmann A G, Trinkley K E, Gritz M, Glasgow R E. Pragmatic considerations and approaches for measuring staff time as an implementation cost in health systems and clinics: key issues and applied examples. Implement Sci Commun. 2022;3(01):44. doi: 10.1186/s43058-022-00292-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Sanyal J, Rubin D, Banerjee I. A weakly supervised model for the automated detection of adverse events using clinical notes. J Biomed Inform. 2022;126:103969. doi: 10.1016/j.jbi.2021.103969. [DOI] [PubMed] [Google Scholar]
- 51.Dev S, Hoffman T K, Kavalieratos D et al. Barriers to adoption of mineralocorticoid receptor antagonists in patients with heart failure: a mixed-methods study. J Am Heart Assoc. 2016;5(03):e002493. doi: 10.1161/JAHA.115.002493. [DOI] [PMC free article] [PubMed] [Google Scholar]