Measures of User Experience in a Streptococcal Pharyngitis and Pneumonia Clinical Decision Support Tools

Summary

Objective

To understand clinician adoption of CDS tools as this may provide important insights for the implementation and dissemination of future CDS tools.

Materials and Methods

Clinicians (n=168) at a large academic center were randomized into intervention and control arms to assess the impact of strep and pneumonia CDS tools. Intervention arm data were analyzed to examine provider adoption and clinical workflow. Electronic health record data were collected on trigger location, the use of each component and whether an antibiotic, other medication or test was ordered. Frequencies were tabulated and regression analyses were used to determine the association of tool component use and physician orders.

Results

The CDS tool was triggered 586 times over the study period. Diagnosis was the most frequent workflow trigger of the CDS tool (57%) as compared to chief complaint (30%) and diagnosis/antibiotic combinations (13%). Conversely, chief complaint was associated with the highest rate (83%) of triggers leading to an initiation of the CDS tool (opening the risk prediction calculator). Similar patterns were noted for initiation of the CDS bundled ordered set and completion of the entire CDS tool pathway. Completion of risk prediction and bundled order set components were associated with lower rates of antibiotic prescribing (OR 0.5; CI 0.2-1.2 and OR 0.5; CI 0.3-0.9, respectively).

Discussion

Different CDS trigger points in the clinician user workflow lead to substantial variation in downstream use of the CDS tool components. These variations were important as they were associated with significant differences in antibiotic ordering.

Conclusions

These results highlight the importance of workflow integration and flexibility for CDS success.

Introduction

Clinical decision support (CDS) provides relevant clinical content to decision makers at the point-of-care [1]. With the widespread adoption of electronic health records, the potential for CDS to improve decision making is rapidly increasing [2, 3]. CDS can improve quality and effectiveness of care by triggering the delivery of guidelines, reminders and tools to assist clinicians with making appropriate diagnoses and treatment [1, 4]. These tools are designed to reduce the gap between the existence of evidence-based guidelines and their implementation into practice [5, 6].

CDS is an important component of national efforts to improve the adoption of evidence-based practices [3]. They are tools frequently invoked to meet the federal regulations of the Meaningful Use initiative and their effective use is part of the criteria for meeting Meaningful Use Stage 2 [7, 8]. To be successful CDS must deliver accurate information, in the right clinical context, at the point of care, and be integrated into the proper providers workflow [6, 9]. CDS rules should be backed by well-tested, validated evidence based rules [10].

CDS systems can generate several forms of decision support that have been extensively studied, particularly, alerts and reminders [11]. Reminders work best when they require minimal steps and the interventions can be completed rapidly at the point of care [12]. Successful examples of CDS systems have led to: reductions in antibiotic prescribing for bronchitis [13]; improved use of CT pulmonary angiography in the emergency room [5]; improved compliance with preventive health guidelines [11]; and age-specific alerts that reduced inappropriate prescribing in the elderly [14, 15]. However, many implementations of CDS systems have not been effective in altering clinical behavior [16, 17]. A recent systematic review of 148 randomized control studies of electronic CDS systems revealed that 8 out of 12 studies documented low use (<50% of patient visits or providers’ time) [6].

The inconsistent findings on CDS effectiveness likely reflect differences in system design, workflow integration, usability, simplicity and content [18, 19]. Clinicians report that efficiency, perception of usefulness, information content, user interface, and workflow are the keys to effective decision support though these domains are understudied in the CDS literature [1]. System usability and workflow integration in particular are associated with more successful CDS [9, 20, 21]. Accuracy and simplicity are critical to successful CDS [22]. Additionally, over-triggering of CDS tools can create alert fatigue, reducing its usability; this problem can be overcome using narrow, concrete criteria that trigger in a minimally workflow disruptive manner [22].

As the diffusion of EHRs approaches maturation, with 54% of physicians adopting EHR systems, there is a need for new approaches to integration of sophisticated CDS to improve adoption and enhance decision making at the frontlines of care [23]. Successful active clinical decision supports must be timely, delivered at the point of care to the right provider(s), in clinical context, patient specific, automated as much as possible, tested and validated, allow explanation for override and have actionable recommendations [22, 24, 25]. In this context, we recently conducted a successful large scale implementation of a new CDS system encouraging guideline concordant antibiotic prescribing for select upper respiratory conditions. In this case the CDS was testing the impact of electronic health record integrated clinical prediction rules (iCPRs). Clinical prediction rules (CPRs) are a type of evidence-based guideline that uses validated rules for simple sign or symptom-based probability scores to risk stratify patients for specific prognoses and/or diagnostic assessments.

This analysis seeks to identify potential mediators of success, predictors of use and barriers to even wider adoption. We utilized a subset of data from our larger previous study that sought out to develop and integrate a CDS tools based on clinical prediction rules from streptococcal pharyngitis (strep) and pneumonia. Overuse of antibiotics and resulting antimicrobial resistance are well-documented problems of national concern, with over-treatment of acute pharyngitis and presumed pneumonia a major cause of antibiotic overuse. Most upper respiratory infections are viral in nature, but many patients receive presumptive antibiotic therapy for pharyngitis, with rates of prescribing antibiotics for upper respiratory tract infections estimated to be 40–75% [6, 7, 26]. Furthermore, physicians frequently prescribe broad-spectrum antibiotics, which are considered inappropriate as first-line treatment of common infections, exacerbating the spread of antimicrobial resistance [8].

Using user centered design principles a new clinical decision support tool was created to address these challenges. User centered design builds the tool around the user rather than forcing the user to adapt to an idealized tool [27]. It integrates the “end user” (in this case the clinicians) perspective throughout the design process and tests design assumptions with users to guide development [27]. In this case, using the feedback from iterative design and usability testing, the team developed a complex tool based on logic models and included different trigger points, risk calculator, bundle order sets and documentation for discharge notes, all recommendations from end-users. The tool was then integrated at the point of care through a randomized controlled trial to assess the impact of the tool on providers’ antibiotic ordering behaviors. The previous study showed overall high adoption rates (providers used the tool in 58% of visits) and significant impact on antibiotic ordering (37% less likely to order antibiotics in the intervention arm) [28].

Methods

The primary study was designed to assess the overall impact of integrating two clinical prediction rules at the point of care on antibiotic ordering in a primary care setting. This manuscript is a sub-analysis of a larger dataset generated earlier by the RCT, and focuses on functionality and usability of the tool.

Faculty, residents and nurse practitioners at a large academic center in New York City (n=168) were randomized into intervention and control arms to assess the adoption and impact of these two clinical prediction rules over the course of one year, from November 2010 to November 2011. There were 87 providers in the intervention arm and 81 providers in the control arm. Primary care providers were randomized in a one-to-one fashion to a control or intervention arm. The controls received literature on pneumonia and strep pharyngitis clinical prediction rules while the intervention arm attended an in-person training session and had access to the iCPR tool.

This study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed by the Institutional Review Board. None of the authors have conflicts of interest.

Design of the iCPR tool

During the development process, the team conducted two phases of usability testing of the iCPR tool in which “think-aloud” scripted scenarios were conducted followed by “near-live” clinical simulations [29]. A more thorough description of the design of the tool and usability testing methodologies used to evaluate the iCPR tool are described in previous publications [20, 29]. The usability data was used to revise the tool used in the randomized control trial.

The final design of the iCPR prototype included a calculator for generating strep and pneumonia risk estimates that minimized “clicks” and manual data entry. ▶ Figure 1 depicts generic schema of tool workflow, in which the physician accepted or deferred the calculator and then impending triage to order antibiotics (AO) or medications (MO) through either the Smartset order or through usual order entry. The system also used context sensitive trigger points. Alerts launched at one of several specific locations throughout the provider’s interaction with the EHR interface: reason for visit, a relevant and specific diagnosis, a more generic diagnosis in combination with a specific antibiotic order, and a point of care test such as a rapid strep test or throat culture. The alerts included non-interrupting alerts in the chief complaint location and interrupting alerts in the diagnoses, order combination and point of care testing locations.

Fig. 1.

Generic schema of iCPR CDS tool workflow

Upon acceptance of the alert, the tool then launched a risk calculator to produce risk probabilities for strep and pneumonia based on the prediction rules published by Walsh and Heckerling, respectively [29, 30]. Although there are other clinical prediction rules published for strep and pneumonia, they are not as well validated as the aforementioned. Since Heckerling in 1990, there have been several prediction rules for pneumonia. It would be worthwhile to put this into clinical perspective. Most of the active research in prediction rules for pneumonia since then have either focused on predictions for inpatient vs outpatient treatment (PORT score) or adding CRP to the diagnostic evaluation of pneumonia. To date, the CRP additive scores have not been definitely validated and are not yet ready for widespread clinical use.

Specific bundled order sets launched based on the calculator score, which categorized patients as low or high for pneumonia, and low, intermediate or high for strep throat. The bundled medication order sets, or “smartsets”, were considered in the design primarily because they were thought to enhance provider buy-in and usability of the tool. The smartsets connected appropriate suggestions for treatment (e.g. antibiotics, supportive rest) to the calculator risk level. In addition to treatment suggestions, the tool included clinical documentation that would populate progress notes for the user, as well as automatically fill out patient instructions.

Outcomes

We measured the impact of the strep and pneumonia iCPRs on providers’ use of the tool across diseases and providers’ ordering behaviors. Specifically, we measured the specific rates of acceptance of the tool at designated trigger point workflow entry locations (reason for visit, diagnosis, diagnosis and order, point of care test for strep) and the rates of use of the smartsets (opening, completion).

Statistical Analysis

All analyses include iCPR encounters only. Meaning, encounters from physicians in the intervention group that triggered the tool. Frequencies and percentages were used to describe entry (i.e., “trigger”) locations of the tool overall and by diagnosis group, acceptance of the tool by entry location and utilization of the smartset by risk category within each diagnosis group. Generalized Estimating Equations (GEE) with robust standard errors for binary data were used to examine the association between ordering behaviors and the utilization of each tool component. GEE estimation was used to account for the clustering of multiple visits within providers. All analyses were conducted using SAS v. 9.3.

Results

Triggering locations of the iCPR Tools

There were 586 iCPR encounters during the randomized controlled trial, 212 pneumonia and 374 strep (▶ Table 1). The majority of the encounters for pneumonia and strep were categorized by the risk prediction algorithm as low risk (97% and 55%, respectively). The iCPR CDS tools were most commonly triggered by entering a highly relevant diagnosis (56.5%) while chief complaint was used about a third of the time. Less specific diagnoses (such as a cough) combined with a disease specific antibiotic order was a less common trigger for the tool.

Table 1.

Frequency of trigger locations in iCPR encounters

Trigger location N	Combined 586		Strep 374		Pneumonia 212
	n	%1	n	%1	n	%1
Chief Complaint	174	30	143	38	31	15
Diagnosis	331	57	228	61	103	49
Diagnosis and Antibiotic Combination	78	13	0	0.0	78	37
Rapid Strep Test/Throat Culture	3	1	3	1	0	0

¹Percentages represent column percents

Adoption of the Strep and Pneumonia iCPR Tools according to triggering location

Chief complaint triggering of the iCPR tools lead to the highest rates of iCPR component use with over 80% opening the calculator, over 70% opening the smartsets and over half using all tool components (▶ Table 2). Triggering of iCPR by diagnosis lead to more moderate adoption of tool components with about two-thirds opening the first component (calculator) and less than half completing all tool components. Diagnosis-antibiotic combinations were associated with low rates of iCPR component use.

Table 2.

Acceptance of iCPR components according to trigger location

Trigger location	n	Calculator Opened 368		Calculator Completed 337		Smartset Opened 319		Smartset Completed 246
		n	%1,2	n	%1,2	n	%1,2	n	%1,2
Chief Complaint	174	144	83	132	76	123	71	96	55
Diagnosis	331	205	62	188	57	181	55	139	42
Diagnosis and Antibiotic Combination	78	19	24	17	22	15	20	11	14
Rapid Strep Test/Throat Culture	3	0	0	0	0	0	0	0	0

¹Percentages represent row percents

²Percentages were calculated using the total n for each trigger location. They are not conditional.

Acceptance of iCPR components according to risk category

▶ Table 3 examines how acceptance of iCPR components varied by disease risk category. Patients presenting with strep throat symptoms were categorized by the clinical prediction rule into low, medium and high risk for having strep throat. Low risk was the most frequent presentation. All three categories were followed by nearly universal opening of the smartset tool which allowed providers to examine the guideline concordant treatment options. Providers completed the smartset orders in three-quarters of low and medium presentations and two-thirds of high risk presentations. A low risk presentation of pneumonia was far more common than high risk. Like strep throat, both risk categories were associated with nearly universal engagement with the smartset and two-thirds of low risk cases had a completed smartset.

Table 3.

Acceptance of iCPR components according to risk category

Risk Category	Calculator Completed	Smartset Opened		Smartset Completed
	n	n	%1,2	n	%1,2
Strep score
Low (-1 – 0)	138	131	95	106	77
Medium (1–2)	93	87	94	69	74
High (3–4)	18	18	100	12	67
Pneumonia score
Low (-1–3)	85	80	94	57	67
High (4–5)	3	3	100	0	0

¹Percentages represent row percents

²Percentages were calculated using the total n for each risk category. They are not conditional.

Association of iCPR tool components and clinician actions

Completing the smartset was associated with significantly less antibiotic ordering, a small trend towards less medication ordering and significantly more point of care testing (▶ Table 4). Completion of the calculator portion of the iCPR tool was not significantly associated with antibiotic ordering though it was associated with a trend towards increased medication ordering and significantly more point of care testing (over five times more frequently). Opening the smartset on the other hand was associated with a trend towards lower antibiotic ordering and significantly less medication ordering overall.

Table 4.

Medication and Test Ordered by iCPR Component Used

	N*	Antibiotic Ordered				Medication Ordered				Strep	POCT Orders1
				OR (95% CI)**	P =			OR (95% CI)**	P =	N*			OR (95% CI)**	P =
All Intervention encounters		171				247					109
iCPR component used		n	%			n	%				n	%
Calculator Opened	368	86	23			144	39			278	92	33
• Completed the calculator	332	77	23	1.0(0.5,2.1)	0.95	132	40	1.3(0.6,2.9)	0.48	248	89	36	5.2(1.5,17.7)	0.01
• Did not complete the calculator	36	9	25	Reference		12	33	Reference		30	3	10.00	Reference
Calculator Completed	332	77	23			132	40			248	89	36
• Opened the Smartset	314	70	22	0.5(0.2,1.2)	0.12	120	38	0.3(0.1,0.8)	0.02	235	88	38	6.81 (1.0,47.0)	0.05
• Did not open the Smartset	18	7	39	Reference		12	67	Reference		13	1	8	Reference
Smartset Opened	314	70	22			120	38			235	88	38
• Completed the Smartset	240	46	19	0.5 (0.3, 0.9)	0.01	90	38	0.9(0.6,1.4)	0.56	186	78	42	2.78(1.3,5.9)	0.01
• Did not complete the Smartset	74	24	32	Reference		30	41	Reference		49	10	20	Reference

* Numbers presented are conditional on the use of the previous iCPR component

** OR obtained using generalized estimating equations (GEE) to account for nesting (multiple encounters per provider)

1POCT orders is only for strep patients

Discussion

The observed utilization rates of the various tool components guide our understanding of the high rates of engagement with the iCPR CDS tools as well as opportunities for improvement in future versions. Providers who used all components of the CDS tool (calculator and smartset) demonstrated substantially lower rates of antibiotic ordering compared to those who prematurely terminated their use. This process measure supports the face validity of the tool’s original purpose which was to lower antibiotic prescribing rates for possible strep throat and pneumonia in primary care. It also highlights the ability of user centered design to develop CDS tools that are consistent with the perspective and work of the intended user. Similarly, the point of care testing orders which were part of the CDS pathway were also more frequently ordered among providers completing the CDS tool components.

The iCPR tool provides several potential trigger points for the CDS to begin. Entering a CDS relevant diagnosis was a more common entry point than the chief complaint. This suggests that using chief complaint to trigger sophisticated CDS may be premature in primary care workflow where the true presentation of disease is often unclear at the beginning of visits. This conclusion is further supported by the particularly low rate of chief complaint triggering in the pneumonia group. Pneumonia initial complaints may be more nebulous than strep throat as a typical initial complaint of “cough” can lead to more diagnostic possibilities than “sore throat”. This observation highlights the importance for flexible triggering in CDS tools as clinical parameters will likely affect the optimal workflow entry points.

The point of entry to the CDS workflow also appears to affect providers’ perceived usefulness. For example, chief complaint lead to high rates of CDS tool component use while diagnosis triggering lead to modest CDS tool component use. Meanwhile, the diagnosis and antibiotic combinations were associated with lowest rates of CDS component use. One possible explanation for this observation may be that chief complaint triggering are typically very early in a primary care visit workflow and may be less likely to disrupt a provider’s momentum as compared to when they are entering a diagnosis which typically occurs later in the visit. These observations are supported by the even lower rates of CDS tool component use in the diagnosis/antibiotic combination category as this type of trigger would be most likely to occur at the end of a visit workflow. Triggers late in the visit workflow would be too disruptive and lead to a negative “distraction to benefit” ratio for providers and low rates of completion.

Once the tool was opened, the rates of component use were similar across risk categories and diseases. This suggests that providers who engaged with the tool found value at each risk level and in both conditions. This observation supports the success is again of the user centered design and careful usability testing during the tool development process. In this process, typical users, in near-live settings judged the value of the tools. The design team then used rapid iteration cycles to modify the CDS tool to the target users’ preferences. This appears to have added value to the CDS tool, making it more robust to the wide variation in clinical situations.

These observations confirm and extend previous literature examining the factors affecting CDS utilization in real clinical settings. In the classic CDS article by Bates et al., he highlighted the importance of designing CDS tools that fit into the user’s workflow, leverage usability testing, avoid disruptions, seek to influence provider behavior rather than stop it, and are simple [22]. These design principles were critical to the iCPR CDS tool development and the data presented in this study empirically supports the success of that process and its ability to enhance CDS tool adoption and ultimately clinical care. In another CDS intervention to reduce antibiotic use, utilization rates ranged between 40% and 77% across practice sites. Key factors for successful implementation and adoption were the iterative CDS design process and provider perception of usefulness of the tool [30]. Its effect was more pronounced on reducing broad spectrum antibiotic use as compared to overall antibiotic use; a finding consistent with the concept of using CDS to influence provider momentum rather than stop it [22, 31]. A similar study in adult and pediatric settings found a modest improvement in antibiotic prescribing practices from an electronic health record embedded CDS tool. Its success again dependent on involving end users in the design process though in this case perhaps later in the development cycle and without clear usability testing [32]. Conversely, another CDS intervention for reducing antibiotic prescribing observed no impact which was attributed to low rates of provider adoption (used in 6% of eligible visits; 28% eligible providers used it at least once) likely due to poor workflow integration [16, 33]. Strong workflow integration and usability do not guarantee use or effectiveness – the tools must add value to clinical decision making. For example, the lack of effectiveness of a CDS tool for improving chest pain diagnosis and management was attributed to inability of the clinical content of the tool to alter provider decision making [17]. The present study in combination with the prior literature provide empirical support for the guiding principles to effective CDS design and implementation first described by Bates a decade ago.

While the study findings emphasize the importance of incorporating the clinicians’ workflow into the design of a CDS, the experience also provides a uniquely granular and pragmatic view on this critical issue. Workflow is a complicated process that involves multiple branching pathways involving various participants in the delivery of primary care [20, 34]. Our data demonstrate that even in relatively defined clinical scenarios such as strep throat and pneumonia – there are many variations in the decision making and care delivery. Since the inception of CDS, researchers have attempted to develop systems that can anticipate these workflows and insert evidence-based structures to guide decision making [35]. Numerous expert systems were developed in the latter half of the twentieth century to model clinical decisions and guide behavior in conditions such as congenital heart disease or for choosing the appropriate antibiotic in severe infections [36, 37]. These computer aided systems, such as INTERNIST-1, focused on supporting diagnostic and therapeutic clinical decisions [37, 38]. Over time, the limitations in accuracy, adaptability and “real” world usefulness of these systems limited their diffusion but were successful in providing a foundation of the techniques used in the CDS systems of today [38]. Based on these early experiences, modern CDS systems are encouraged to adhere to several key principles such as the CDS Five Rights model which suggests sustainable CDS-supported improvements are more likely if they communicate

1. the right information;

2. to the right person;

3. in the right format;

4. through the right channel; and

5. at the right time [39].

These guidelines are useful but do not provide operational guidance on how to construct a CDS system so that it is successful in aligning with the Five Rights.

Using these general guidelines, our work extends our understanding of how to insert CDS at the right time. Triggering or alerting is a key initial step in many CDS but alert fatigue is a well described phenomenon that limits the potential of CDS to promote evidence based care [40]. By examining how our tools were adopted or ignored in relation to the workflow trigger points we enhance our understanding of how workflow affects their adoption. While triggering CDS at the beginning of a clinical workflow (e.g. chief compliant) is a logical trigger point, our data showed it was the least frequently successful in engaging clinicians. However, these data are more complex; in cases where the chief complaint was the trigger it was the most powerful workflow trigger for changing antibiotic prescribing behavior. Conversely, alternative trigger points such as the diagnosis field engaged more clinicians but were less successful in altering prescribing behavior likely because of its more distal location in the decision making pathway. This suggests that multiple trigger points may be needed to meet the needs of the heterogeneous workflows of primary care even within a single condition. Designers and implementers of CDS, even when constrained within commercial vendor EHR environments, need to be attentive to and creative in their use of triggers and how they interact with clinical workflow. In addition, new CDS tools need to become more flexible in triggering allowing them to adapt to the real-time workflow on a patient level basis [41].

The study findings should be interpreted in the context of several limitations. The results represent the experience at a single large academic health center and describe the results in two very common primary care conditions. They may not be generalizable to other settings and diseases. Similarly, this CDS intervention was built within a single commercial electronic health record system and the ability to replicate its functionality and workflow may vary within other systems. The ability of these data to represent the underlying motivations for provider use or avoidance of various CDS components is to some degree a proxy and would require more qualitative methods to more comprehensively detail these factors.

Conclusion

CDS effectiveness is a product of a user centered design process that leverages usability tools to create a workflow realistic, simple and clinically useful tool. The ability to trigger the tool from multiple workflow entry points was critical as the use of the different triggers was associated with differential use of the tool components. These data support the development of CDS tools that are context dependent in that they are flexible to the unpredictable workflow of clinical care. In addition, it reinforces the importance of simple but clinically relevant tools that providers willingly receive rather than forcing external guidelines into clinical care. More sophisticated CDS tools incorporating increasingly complicated risk algorithms and functionalities are on the horizon. Adhering to fundamental user centered design principles will increase the probability of these tools delivering on the promise of policy tools such as Meaningful Use.