A Framework for Usable and Effective Clinical Decision Support: Experience from the iCPR Randomized Clinical Trial

Abstract

Introduction:

The promise of Clinical Decision Support (CDS) has always been to transform patient care and improve patient outcomes through the delivery of timely and appropriate recommendations that are patient specific and, more often than not, are appropriately actionable. However, the users of CDS—providers—are frequently bombarded with inappropriate and inapplicable CDS that often are not informational, not integrated into the workflow, not patient specific, and that may present out of date and irrelevant recommendations.

Methods:

The integrated clinical prediction rule (iCPR) project was a randomized clinical trial (RCT) conducted to determine if a novel form of CDS, i.e., clinical prediction rules (CPRs), could be efficiently integrated into workflow and result in changes in outcomes (e.g., antibiotic ordering) when embedded within a commercial electronic health record (EHR).

We use the lessons learned from the iCPR project to illustrate a framework for constructing usable, useful, and effective actionable CDS while employing off-the-shelf functionality in a production system. Innovations that make up the framework combine the following: (1) active and actionable decision support, (2) multiple rounds of usability testing with iterative development for user acceptance, (3) numerous context sensitive triggers, (4) dedicated training and support for users of the CDS tool for user adoption, and (5) support from clinical and administrative leadership. We define “context sensitive triggers” as being workflow events (i.e., context) that result in a CDS intervention.

Discussion:

Success of the framework can be measured by CDS adoption (i.e., intervention is being used), acceptance (compliance with recommendations), and clinical outcomes (where appropriate). This framework may have broader implications for the deployment of Health Information Technology (HIT).

Results and Conclusion:

iCPR was well adopted(57.4% of users) and accepted (42.7% of users). Usability testing identified and fixed many issues before the iCPR RCT. The level of leadership support and clinical guidance for iCPR was key in establishing a culture of acceptance for both the tool and its recommendations contributing to adoption and acceptance. The dedicated training and support lead to the majority of the residents reporting a high level of comfort with both iCPR tools strep pharyngitis (64.4 percent) and pneumonia (62.7 percent) as well as a high likelihood of using the tools in the future. A surprising framework addition resulted from usability testing: context sensitive triggers.

Introduction

The promise of Clinical Decision Support (CDS) has always been to transform patient care and improve patient outcomes through the delivery of timely and appropriate recommendations.1–3 CDS is defined as anything that directly aids in clinical decision-making about individual patients. Decision support can include collegial advice, text references, Web sites, and computer systems4 A Clinical Decision Support System (CDSS) is a computerization of CDS, frequently integrated into a clinical information system such as an electronic health record (EHR), and directly aids in clinical decision-making about individual patients. Specifically, CDSS incorporates individual patient data, a rules engine, and a medical knowledge base to produce a patient-specific assessment or recommendation for clinicians.5,6 In a sense CDS is the content and CDSS is the delivery system. For the purposes of this paper, CDS, unless otherwise specified, is delivered through a CDSS.

However, users of CDS, i.e., providers, are frequently bombarded with inappropriate and inapplicable CDS that is often not informational, not integrated into the workflow, not patient specific, and that may present out of date and irrelevant recommendations. Not surprisingly, multiple recent studies and analyses have raised questions about the effectiveness of CDS and ambulatory EHRs and the ability of EHRs to have an impact on care quality.7–11 And a 2012 study determined that while CDS can lead to changes it’s not clear if it leads to changes in clinical outcomes or improved efficiency.12 Dexheimer et al. (2005) examined 23 studies of preventative health care reminders and concluded that paper tools were superior to CDS.13

Successful implementation of Health Information Technology (HIT) is affected by multiple factors, many of which have been studied in regards to implementation of clinical information systems such as inpatient Computerized Provider Order Entry (CPOE), ambulatory EHRs, etc. These factors include the following: leadership, integration with health care and workflow process, value to users, and training and support.14,15 However, these success factors have not been similarly studied in CDS.

The success of CDS interventions can be measured by the adoption rate and the acceptance rate of the CDS intervention (Table 1). Is the CDS intervention being used (adoption rate), and are the CDS-provided recommendations being accepted (acceptance rate).16

Table 1.

Measures of Success for Usable and Effective Clinical Decision Support (CDS)

MEASURE	DEFINITION
Adoption	CDS intervention is used.
Acceptance	Compliance with CDS recommendations.
Changes in Behavior	Changes in process or care(e.g. reviewing medications, ordering more of a medication).
Clinical Outcomes	Demonstrable (statistically significant) changes in care.

The integrated clinical prediction rule (iCPR) project was a novel form of a CDS intervention, i.e., clinical prediction rules (CPRs), which was efficiently integrated into workflow. The CDS intervention resulted in both high adoption- (57.5 percent of intervention users) and acceptance rates (42.4 percent). In contrast the peer reviewed literature cites rates of 10–20 percent for both adoption and acceptance (see Table 2).12,16

Table 2.

Summary of iCPR Results by CDS Measure of Success

	ICPR	PEER REVIEWED LITERATURE
Adoption	57.5%17	10–20%12,16
Acceptance	42.4%17	10–20%12,16
Outcomes	9.2% reduction in ordering antibiotics for strep pharyngitis with a P value =.00817	Wide variability not easily summarized

The framework for usable and effective CDS is derived in part from the peer reviewed literature (see above), which significantly informed integrated iCPR project design and the authors’ experience conducting a randomized clinical trial (RCT) of iCPR. The iCPR project objective was to determine if iCPR—a novel form of CDS in an EHR—could be efficiently integrated into workflow, resulting in changes to patient outcomes (e.g., antibiotic use). Only off-the-shelf EHR functionality was employed to ensure portability. We implemented two well-validated iCPRs, namely, the Walsh rule for streptococcal pharyngitis and the Heckerling rule for pneumonia. The iCPR RCT was conducted at the Internal Medicine Associates at Mount Sinai Medical Center, New York City—with both attending physicians and housestaff providers. At the end of the study, the intervention group completed the iCPR tool in 57.5 percent of visits, and providers in the intervention group were significantly less likely to order antibiotics than was the control group.17

This paper uniquely uses the lessons learned from the iCPR17–21 project relative to successful CDS implementation to illustrate a framework for constructing usable, useful, and effective CDS that is both highly adopted and accepted. The framework was developed in part because multiple studies have questioned the effectiveness and efficacy of CDS delivered by ambulatory EHRs.7–13 Additionally, the framework was developed to address the fact that while leadership, integration with health care and workflow process, value to users, and training and support14,15 have been studied as success factors in implementation for clinical information systems (see above), these same success factors have not been similarly studied in CDS.

This framework combines the following: (1) active and actionable decision support, (2) multiple rounds of usability testing with iterative development for user acceptance, (3) numerous context sensitive triggers, (4) dedicated training and support for CDS tool users to encourage user adoption, and (5) support from clinical and administrative leadership for both successful adoption and acceptance of the CDS intervention.

iCPR Framework for Constructing Usable, Useful, and Effective CDS

A literature review identified four factors that can have a significant impact on CDS adoption and acceptance: active and actionable CDS; usability and clinical workflow integration; training; and clinical leadership. The fifth factor, “context sensitive triggers,” was identified through the iCPR work. The review of the CDS literature was never intended to be a comprehensive or systematic review. Search terms in both PubMed and Google Scholar included clinical decision support; acceptance rates of clinical decision support; clinical decision support and usability; and clinical decision support and implementation. To understand the role of these factors it is necessary to first review the types of CDS and how they are triggered. For the role each of these factors play in the framework please see Table 3.

Table 3.

Relationship of Framework Criteria to Measures of Success for Clinical Decision Support (CDS)

FRAMEWORK CRITERIA	IMPACTED MEASURES OF SUCCESS
Actionable and Active	Adoption and Acceptance, Outcomes
Multiple rounds of usability testing with iterative development	Adoption, Acceptance
Numerous context sensitive triggers	Adoption
Dedicated training and support for users of CDS tool for user adoption,	Adoption, Acceptance
Support from clinical and administrative leadership for both successful adoption and acceptance of the CDS intervention	Adoption, Acceptance

Active and Actionable Clinical Decision Support (CDS)

Background

There are two types of CDS. The first type is passive, in which the user has to input data and then request help from an online source like the National Library of Medicine’s Medline (http://www.pubmed.org). In contrast, the second type—active decision support—is triggered by an event and delivers information to the physician that was not requested by the physician, but that is relevant and of interest.22,23 The physician may or may not have taken a related action to initiate the trigger. For example, a physician may order a nephrotoxic medication and trigger a CDS-generated alert that recommends a dose appropriate to the patient’s kidney function and enables ordering a corollary order for blood levels of the medication. Corollary orders refer to “orders required to detect or ameliorate adverse reactions that may result from the trigger order.”24

To be accepted and adopted, the active type of CDS must be as follows: delivered at the point of care, patient specific, in clinical context (i.e., clinically relevant and logical, timely, delivered to the right providers), automated as much as possible, allow explanation for override, and be both tested and validated.25 However, CDS that results in frequent overrides is far from desirable as overrides can become a source of error. In a study on the effect of overrides on Adverse Drug Events (ADEs), 1 out of 30 overrides resulted in an ADE.26,27

CDS can generate active decision support in several ways, which have been extensively studied: alerts, reminders, corollary orders, and guidelines.5,6,24 An alert is a suggestion requiring immediate response or action. For example, an inappropriately high dose of a medication or a dangerous interaction between a medication and the value from a lab test would trigger an alert (e.g., Digoxin and low potassium). A reminder is a suggestion not requiring immediate response or action. For example, a reminder may caution that the increased risk of heart disease associated with Celebrex must be balanced with the pain relieving benefit of Celebrex.

Guidelines are a series of instructions on how to care for the patient based on information about the patient’s clinical status. In contrast to alerts and reminders, there are several pieces of information required for a guideline to fire. If a physician orders Metformin, a medication used to control blood sugar in diabetes, a guideline might fire that prompts the optional ordering of hemoglobin A1C and calculates the estimated creatinine clearance. However, implementing guidelines for complex chronic disease has proven to be challenging.25,28–31

The success of actionable decision support can be measured by changes in behavior and/or outcome. The term “actionable CDS” is not new,32–37 and implies at the very least that advice is being provided that the user can then take action on and thus influence behavior and outcome. For example, the alert might state “this patient has diabetes and we recommend starting the medication metformin.” In this example, the user then needs to stop whatever he or she was doing at the moment of alert and go to a separate section of the EHR to place an order for metformin. We define actionable CDS as active CDS interventions (e.g., alerts, reminders, etc.) that contain everything (i.e., orders, documentation, patient instructions, prescriptions, etc.) the user needs to take the desired action.25 By our definition using the same example, the alert would still state “this patient has diabetes and we recommend starting the medication metformin” but would also contain orders for metformin. In this example the user simply has to “accept” in order to acknowledge and place the orders.

Successful CDS design as measured by adoption is dependent on triggers.38 For purposes of this paper we introduce a new term “context sensitive triggers” and define these triggers as being workflow events (i.e., context) that result in a CDS intervention. These workflow events are actions taken by the user such as data entry in structured fields like problem lists, billing diagnoses, or migrating to the order entry section to place an order. For example, the user begins seeing the patient (i.e., workflow) and enters a chief complaint. The entry of data in the chief complaint then triggers the alert.

iCPR and CDS Framework

Clinical prediction rules (CPRs) are a form of active and actionable decision support in the EHR. CPRs can be clearly identified as a type of CDS in that these rules aid in clinical decision-making. CPRs possess distinct characteristics in contrast to other forms of CDS integrated into EHRs. Unlike the content of alerts and reminders, the content of CPRs must meet methodological standards that are designed to subject the rules themselves to validation and assessment of clinical outcomes. Specifically, CPRs must include the following: an outcome that is both clearly defined and clinically important; a well-described patient population to account for the effect of the population on rule performance; validation in other patient populations; measurements of clinical use—e.g., is it being used clinically) and outcome; reproducibility; and clinical common sense (i.e., does the rule make sense?).39–41 CPRs, unlike clinical guidelines, are designed to answer one clinical question such as “does this patient have strep pharyngitis or does this patient need a chest x-ray to diagnose pneumonia?” Since iCPR was built as actionable CDS and CPRs are designed to influence outcome, a very applicable measure of success for iCPR would be clinical outcomes. There was a 9.2 percent reduction in ordering antibiotics for strep pharyngitis with a P value =.008 (Table 2).17

As noted above, successful active CDS is delivered at the point of care, is patient specific, is in a clinical context, is automated as much as possible, allows an explanation for override, and is both tested and validated.25 CPRs are delivered at the point of care, are specific to patients with certain complaints and diagnoses, is automated as much as possible, and allows override with explanations.17 The success of active CDS can be measured by adoption, which was 57.5 percent (Table 2).

Usability and workflow integration

Background

Usability is a very significant factor in the successful adoption and acceptance of CDS. Usability can best be defined as both usable and useful. Usefulness along with relevance have been noted as key determinants of CDS success by both Bates and Rousseau.25,42 Poor user-interface design and redundant entry of data lead to poor use of CDS,43 perhaps highlighting the need for iterative design and assessment of usability even more so. The need to get structured user feedback to improve CDS (e.g., usability testing) was also noted with the findings that were observed in multiple other studies (above).43–48 A nursing study indicated the need for redesign emphasizing usability and learnability to improve use of reminders as well as usability and workflow integration.44,45 Shah et al. found that taking into account usability and workflow helped in redesign of alerting in prescribing systems.47 Fung et al. (2008) found that improving integration into workflow as well as a continuous iterative feedback with user feedback were necessary ingredients for well designed and accepted alerting.48 Workflow integration as a determinant of CDS success was stressed in studies by Maviglia28 and Krall.49 This need cannot be underestimated, as failure to integrate with existing processes has led to significant difficulty and user dissatisfaction.50–59

Not only is usability affected by a user population, the different types of users—light, moderate, and heavy users—may require different designs.60 Poor usability and user interface design can contribute to medical errors in the clinical information system.27,61–63 Clinical information systems need to be intuitive and easy to use, and to integrate well in the workflow.64 Ultimately, this is true of CDS as well.45 The conflicting results in the literature on ambulatory CDS effectiveness noted earlier may reflect differences in system design, workflow, usability, and content.65

Ultimately, usability is linked to and affects user satisfaction,66 and user satisfaction is an important predictor of a system’s success.67 The bottom line is that physicians are looking for systems that are easy to use, improve daily efficiency, and provide perceived improvements in patient care.68 Perceived ease of use determines perceived usefulness and has the highest correlation with positive attitudes about EHR use.69 Consistent with this bottom line is that user satisfaction seems to correlate with the ability to perform tasks efficiently.70 This need for task efficiency has been noted for CDS as well,49 in particular for reminders.5,6,71–74 CDS interventions provide some of the perceived quality and efficiency that physicians look for by helping physicians with “their” patient.14,68

iCPR and CDS Framework

Prior to our “go live,” investigators evaluated the usability of iCPR integration at each of the possible trigger points and tested the level of disruption to clinical workflow induced by triggering at that point in workflow. They also evaluated the perceived usefulness of iCPR as a decision aid in each of the two clinical scenarios. Clinical scenarios were constructed for providers, and they were asked—while interacting with the program—to “think aloud” about what they are doing and what they want to achieve, and also to verbalize their experience using the software. The computer screens were recorded and subjects’ verbalizations were audio recorded. A professional transcriptionist for the expert panel’s review transcribed the audio recordings. We paid particular attention to how easily and successfully providers were able to navigate the program and how satisfied they were with the interface and the information. We also recorded the number of errors or problems in understanding different concepts and completing different tasks. We recorded the number of requests for assistance. This first round of usability testing employed eight subjects.18

Over the development period, prototypes were tested in increasingly realistic scenarios, with final versions being tested in our simulation lab. The clinical care sessions reproduced and simulated the issues of time pressures and patient-case complexity. This process allowed the developers to ascertain characteristics of the iCPR experience that are functional, need improvement, fit user expectations, miss expectations, fail to function, or are opportunities for further development.18 This second round of usability testing employed eight subjects.

Refinements based on the results from usability testing were incorporated into subsequent prototypes. For example, the iCPR prototype included a calculator for generating strep and pneumonia risk estimates. This calculator minimized “clicks” and manual data entry because of usability testing and iterative design.18,20

In summary, usability testing assessed usability and usefulness before going live. Time was built in to make iterative changes based on the usability testing. In part by evaluating usability and usefulness, we achieved an adoption rate of 57.5 percent (useable) and an acceptance rate of 42.7 percent (useful).17

Dedicated Training

There is universal agreement on training as a prerequisite for a successful implementation15,53,56,75–78 and thus adoption. Poor training can result in inappropriate or underutilization of functionality79–81 and poor adoption. Users frequently want to learn more or receive advanced training and may need to learn about new features added to the system.14,15

iCPR and CDS Framework

Consenting residents who were randomized to the intervention arm received approximately 45 minutes of training. These training sessions were led by at least one study investigator with the support of a study staff member. The content of these training sessions was divided into three basic sections: (1) background discussion of the definition of CPRs and evidence for their use, specifically for strep pharyngitis and pneumonia; (2) on-screen walk-through of three common clinic patient scenarios employing the iCPR tool; and (3) presentation of a simulated patient-physician encounter demonstrating how the tool is incorporated in the office encounter workflow. Residents who were unable to attend this group session were trained individually in separate sessions. The control group was invited to the training, their consent was obtained, and then the control group was provided with two articles on CPRs. They did not participate in the on-screen walk-throughs or the video presentation.

A clinical champion for the iCPR tool was available on site during most clinical sessions to promptly address questions or problems that arose during the use of the iCPR tool. In addition, technical support was available on site and the pager number for the support personnel was posted throughout the practice area. Recurrent or significant technical problems were promptly communicated to the EHR programmer and subsequently discussed with the study team during weekly meetings.

All 59 residents assigned to the intervention group received the training session and completed the post-training survey.18 The majority of the residents reported a high level of comfort with both the strep pharyngitis (64.4 percent) and pneumonia (62.7 percent) iCPR tools following the training session. In addition, they reported a high likelihood of using the tools in the future, and nearly 95 percent of the residents gave a favorable rating for the overall quality of the presentation. The high level of comfort with the tool and content in part anticipated a high adoption rate 57.5 percent and acceptance rate 42.4 percent. It is difficult to determine whether it is the user-friendly design of the tool and the quality of the training and technical support that have facilitated the widespread adoption and use of the iCPR tool among residents.

Support from Clinical and Administrative Leadership

Leadership15,82 is critical for both clinical adoption and acceptance of CDS. There is some evidence in the peer reviewed literature to suggest that leadership is an independent risk factor that correlates with project success or failure.83 The broad umbrella of leadership has to include clinical, administrative, and IT leadership,84 especially clinicians.64,85–87 Recently the peer reviewed literature has begun to clearly state the importance of clinical leadership in particular as a factor in implementation success or failure.15,88,89 Perhaps most importantly, on-site leadership needs to identify the raison d’etre for doing the intervention (i.e., acceptance).

iCPR and CDS Framework

Prior to the rollout, the team met with all of the clinical leadership for the practice: the medical director for the practice, the residency director, the chief residents for internal medicine, and the attending physicians who precept (i.e., teach and supervise the residents) in ambulatory care. Although all of the clinical leadership became strong advocates for the use of the iCPR tool, we identified the need for two clinical champions who were often present in the clinic. One champion was a frequent precept who could troubleshoot any questions or concerns. The other champion was senior clinical leadership who frequently precepted and provided strong on-site advocacy. This is consistent with the important role clinical leadership can play in project success or failure.15,88,89 Overall, the level of leadership support and clinical guidance for iCPR was key in establishing a culture of acceptance for both the tool and its recommendations contributing to adoption and acceptance.

As noted earlier, the role of people in implementations cannot be underestimated.82,83,90 After unrelated changes in clinical, administrative, and IT roles occurred, the utilization of the tool declined.

Context Sensitive Triggers

iCPR and CDS Framework

Usability testing and workflow integration analysis in iCPR identified a surprising candidate for the framework of successful CDS: context sensitive triggers. Specifically, usability testing and workflow integration identified two distinct workflows: charting first, and orders first. In the orders first workflow, frequently used by housestaff, orders were placed first and charting, which included recording of chief complaint, occurred later. In the charting first workflow, charting was done first—including the recording of chief complaint first—with orders occurring later. These two workflows suggested early on to the research team that CDS triggers need to be thought of in the context of workflow. Context sensitive triggers are workflow events that trigger the CDS intervention when the intervention occurs in the user’s workflow. Initially the placement of triggers was not identified as part of the framework for successful CDS, as the framework was based on analysis of the peer reviewed literature available at the time of study design in 2010. The available peer reviewed literature focused heavily on internally developed systems and a limited number of sites.91,92 What iCPR unearthed was a potential limitation of a commercial EHR system. The triggers in the EHR could be placed at the beginning of the visit when the chief complaint was being placed or at the end of the visit when billing diagnoses and orders were being placed. In short, triggers could be placed at the beginning or the end of the visit. There were no identifiable trigger points in the middle of the workflow such as entering observations in a progress note (i.e., charting). The resulting alerts included noninterrupting alerts in the chief complaint section and interrupting alerts in the diagnoses, order combination, and point-of-care testing sections. Adoption in iCPR was defined as responding to the alert and opening the risk score calculator.

The iCPR tool provided three potential trigger points for iCPR as it was clear from an earlier workflow analysis that three very different workflows were possible. Usability testing confirmed this workflow analysis. One such trigger point was the chief complaint, which accounted for 30 percent of all triggered CDS.19 Another such trigger was the entering of a relevant billing diagnosis, which accounted for 57 percent of triggered events, and a third trigger was placing orders accounting for 13 percent of triggered events. When orders were placed, any existing diagnosis was factored in as well to determine if an event should be triggered. The contribution of each of the context sensitive triggers highlights the importance for flexible triggering in CDS adoption as no one trigger would have presented all suitable candidates for intervention.19 In summary, multiple context sensitive triggers were required for successful adoption (i.e., 57.5 percent adoption rate) of the CDS intervention.

Limitations

The iCPR study was not designed to measure the relative contributions of each of the framework success factors. Further study would be needed to delineate relative contributions of framework factors.

While there was a decline in usage of iCPR due to changes in leadership, there was a decline in iCPR usage prior to the changes in leadership as well.21 The decline in usage prior to leadership change was carefully measured. Data regarding the observed decline after clinical leadership changes were not available.

Finally, while the framework is based on a thorough review of the literature, proof of its applicability and success is provided in the context of only one study at one large practice at one academic medical center. Further study would be needed to examine the applicability and success of the framework itself.

Conclusion

We used the lessons learned from the iCPR project to illustrate and support a framework for constructing usable, useful, and effective actionable CDS as measured by adoption (use), acceptance (compliance) rates, and clinical outcomes (where appropriate). For example, clinical outcomes would be an appropriate measure of CDS for an actionable CDS intervention that suggested additional medications. In contrast, measuring clinical outcomes for a process measure such as percentage of patients with their medication list reviewed, a measure required for Meaningful Use, would not be appropriate. The iCPR was designed to be active and actionable CDS and was both well adopted and accepted when compared to baseline rates in the literature (Table 3). Usability testing with iterative design identified and fixed many correctable issues before the clinical trial of iCPR and had a positive impact on the adoption- and acceptance rates.

A surprising addition to the framework came as a result of the usability testing: context sensitive triggers. Overall we identified three triggers, all of which contributed to identifying suitable intervention subjects, and our data support the need for more than one trigger. Since our EHR implementation provided cursory CDS training, and exposure to CDS can occur long after training, we provided dedicated training and support for the use of the tool. The added bonus of dedicated training and support had a positive impact on adoption and acceptance. The involvement and visibility of clinical leadership resulted in a buy in that was transmitted to housestaff, ensuring adoption and acceptance. The iCPR findings are consistent with prior studies that suggest that usability43–48 user training,14,15,53,56,75–81 and clinical leadership15,64,84–89 are each necessary for successful system implementation if not for CDS implementation alone. However, we are unaware, based on our review of the literature, of any study that combined all of these interventions.

Future work will measure the effectiveness of the CDS framework in similar research studies that employ CDS to change behavior and affect outcome. Additionally, more attention will be paid to measuring the effect of clinical leadership on acceptance and adaptation. While it is beyond the scope of this paper to fully explore and discuss the applicability of this framework to other contexts beyond CDS, there is clear applicability to the broader context of HIT. Three of the framework factors have individually been previously identified as critical to successful implementation and use of clinical information systems in HIT: multiple rounds of usability testing with iterative development for user acceptance,43–48 dedicated training and support for users as critical for adoption,43–48 and support from clinical and administrative leadership.15,64,84–89 Further study might investigate the effect of consistently employing each of these three factors together in clinical implementation, optimization, and overall daily use.

The concepts behind the two remaining factors may be quite important for those working with data. Context sensitivity may apply not only to CDS triggers but to the capture and use of clinical data as well. In other words, at what point in the workflow can the user be prompted to capture high quality and reliable data? Capturing byproducts of clinical data generation such as billing or claims has led to questions about the validity of the data in the past.93–95 Similarly, the concepts of actionable and active may apply not only to decision support but also to presentation of data in clinical information systems. For example, should being actionable (i.e., something that the user can take action on) be a criterion for the presentation of data? For example, should data be displayed actively (i.e., clinically relevant and logical, timely, delivered to the right providers)? Should data display be triggered by an event and deliver information to the physician that was not requested by the physician, but is relevant and of interest?

In conclusion, we have provided a framework for developing usable, useful, and effective actionable CDS as measured by adoption and acceptance rates,1–3 as well as clinical outcomes (where appropriate). We recommend applying this framework to produce highly adopted and accepted CDS interventions.