Use of Simulation to Study Nurses Acceptance and Non-Acceptance of Clinical Decision Support Suggestions

Abstract

Our long term goal is to ensure nurse clinical decision support (CDS) works as intended before full deployment in clinical practice. As part of a broader effort, this pilot explores factors influencing acceptance/non-acceptance of 8 CDS suggestions displayed through selecting a blinking red button in an electronic health record (EHR) based nursing plan of care software prototype. A diverse sample of 21 nurses participated in this high fidelity clinical simulation experience and completed a questionnaire to assess reasons for accepting/not accepting the CDS suggestions. Of 168 total suggestions displayed during the experiment (8 for each of the 21 nurses), 123 (73.2%) were accepted and 45 (26.8%) were not accepted. The mode number of acceptances by nurses was 7 of 8 with only 2 of 21 nurses accepting all. The main reason for CDS acceptance was the nurse’s belief that the suggestions were good for the patient (n=100%) with other features being secondarily reinforcing. Reasons for non-acceptance were less clear, with under half of the subjects indicating low confidence in the evidence. This study provides preliminary evidence that high quality simulation and targeted questionnaires about specific CDS selections offers a cost effective means for testing before full deployment in clinical practice.

INTRODUCTION

In 2009, with the adoption of the Health Information Technology for Economic and Clinical Health (HITECH) Act, the use of electronic health records (EHR) became a top priority in the United States.¹ Since that time there has been a dramatic increase in the use of EHRs, with the goal being full adoption for all health care organizations.² As part of the HITECH regulations, systems are required to integrate clinical decision support (CDS) interventions for the purpose of lessening the ever increasing information demands in day to day clinical practice. CDS systems typically provide evidence-based alerts, reminders, guidelines, and best practices delivered to the clinician through an electronic interface during routine workflow^3,4 for the purposes of assisting rather than to replace the practitioners who use them⁵ as well as facilitating evidence based practice.⁶

To be useful, however, CDS systems must clearly align with the clinicians’ needs in practice and demonstrate effectiveness. Based on the literature, it appears that sometimes there is little evidence of efficacy testing, or testing is conducted following deployment in practice.^7–9 The aim of this study is to move toward a general methodology for ensuring nursing CDS works as intended before broad deployment occurs. In this pilot study, we used high fidelity interactive simulation to evaluate subjects’ use of CDS suggestions available in an electronic care planning documentation system through selecting a blinking red button. Participants completed a targeted survey following the test that allowed us to examine the judgement and interface factors associated with nurses’ acceptance and rejection of the evidence based CDS suggestions.

A number of factors are believed to influence nurse acceptance or rejection of CDS suggestions including: nurse education,¹⁰ nurse clinical experience,^7,11,12 nurse experience with a computer,^10,12 fit of technology with local practice,^{7,11,13,14–16} technology and design factors,^7,11,14–16 and organizational factors like training, a supportive environment, and adequate resources.^7,11,13,15 Although important, most of these studies focus primarily on the perceptions and attitudes toward an entire CDS software and not specific to the messages and symbolic features (e.g. specific icons, animation) within the CDS interface.

There are a limited number of studies that focus on nurses' reasons for accepting or refusing specific messages/features of CDS suggestions. For example, in a 12 month study of e-Protocol insulin software conducted in one intensive care unit, Sward et al.¹⁶ examined the reasons nurses declined 5.2% (n=2077) of the total (n=39,640) recommendations generated for 66% (n=548) of patients (n=830) by this CDS system. Two sources of data pertaining to this CDS system, widely accepted by the nurses, were analyzed: survey data gathered from a convenience sample of 14 nurses (28% response rate) and responses entered into the e-Protocol CDS software. Descriptive and thematic analysis of both data sources indicated that the major reasons for declining the CDS recommendations were related to a lack of trust that the algorithm used to calculate the recommendation accurately captured the unique patient factors (49.2%) or other factors (27.3%). Approximately 9.3% of the respondents indicated they declined the recommendations because of CDS system access issues and 4.7% declined due to software or technology issues.

In another study involving two ICUs, using convenience samples, Campion et al.⁷ conducted direct observations of nurses (n=25) and unstructured interviews (n=7) to illuminate the barriers and facilitators to use of an intensive insulin therapy CDS system similar to that examined by Sward et al.¹⁶ Ethnographic methods were employed with the interview data being used to confirm the emerging concepts. The barriers identified included workload tradeoffs, lack of reminders by the system, and poor interface design that was a potential source for errors. The facilitators included the nurse’s trust in the system, nurse resilience, and a paper intermediary at the patient’s bedside.

Dowding et al.¹¹ studied four CDS systems in four different settings for the purpose of discovering how nurses use CDS in practice and the factors that influence use. The CDS cases included systems for: 1) calculating anticoagulant doses, 2) generating quality of life scores from patient entered data, 3) recommending a patient’s disposition from an algorithm based on the answers to a specified set of questions, and 4) monitoring the status of patients with chronic obstructive lung disease in their homes. The researchers observed 115 nurse/patient interactions with the systems and conducted 55 interviews with nurses who worked with the CDS systems at the study sites. The majority of the data were analyzed using thematic analysis and assigned to one of two categories. The researchers found, for category one, that CDS supports decisions by facilitating the recording of information, monitoring patient progress or confirming a decision that had already been made. For category two, factors affecting use included the nurse’s familiarity with the patient, the patient’s condition, and aspects of the technology such as ease of use and appropriateness and usefulness of the content.

Collectively, the studies reviewed underscore the potential value of nursing CDS but also the need to ensure that each feature works as intended. The two most concerning unintended consequences of CDS identified include: 1) declining, ignoring, or over-riding the appropriate recommendations^7,11,15,16 and 2) functionality issues that lead to inappropriate use of the CDS.⁷ The research on nursing CDS conducted to date, however, has predominantly focused on describing the perceptions of nurses about CDS in general. For most studies of specific CDS types, the systems were already implemented in practice and the results were reported as general characteristics of CDS¹¹ or as highly nuanced details about a CDS system.^7,16 A major gap in this literature is the absence of a consistent and feasible methodology for ensuring nursing CDS works as intended before full deployment in practice. We argue that creating a generalizable method that is adaptable for robustly evaluating all types of nurse CDS systems prior to full deployment in practice can reduce many of the unintended and costly consequences that occur without such testing. The study reported here addresses this gap by moving toward a pre-implementation method that will ensure nursing CDS works as intended when it is deployed in clinical practice. In this study, part of a larger effort, we examine the reasons nurses accept or reject specific electronic prototype care plan suggestions for their patients during a high fidelity simulated documentation exercise. A short survey questionnaire was created (see Table 1) to help us target the specific reasons subjects’ accepted or rejected the suggestions generated to quickly assess if our CDS system was working as intended.

Table 1.

Clinical Decision Support Acceptance Report: Reasons for acceptance/refusal of the CDS suggestions.¹

Item	Response	n	%
1. I accepted the suggestions because I believe they were good for the patient situations that were presented.2	Agree	20	100.0
	Disagree	0	0.0
2. I accepted the suggestions because the system displayed information that convinced me they would help achieve the care goals for the patients.	Agree	19	90.5
	Disagree	2	9.5
3. I accepted suggestions because the red button remained present, which told me that I didn’t make enough changes.	Agree	13	61.9
	Disagree	8	38.1
4. I accepted the suggestions because I wanted to make the red button go away.2	Agree	10	58.8
	Disagree	7	41.2
5. I accepted the suggestions because the red button was annoying me (I like a clean screen).2	Agree	7	41.2
	Disagree	10	58.8
6. I accepted the suggestions because I didn’t know what to do.	Agree	7	33.3
	Disagree	14	66.7
7. I did not accept the suggestions because I disagreed with them.	Agree	7	33.3
	Disagree	14	66.7
8. I did not accept the suggestions because I did not have confidence in them.	Agree	6	28.6
	Disagree	15	71.4
9. I did not accept the suggestions because I was uncertain about the strength of the evidence on which they were based.	Agree	4	19.0
	Disagree	17	81.0
10. I did not accept the suggestions because I don’t like others telling me what to do.	Agree	0	0.0
	Disagree	21	100.0
11. I added other diagnoses, interventions, or outcomes not suggested thinking these would take away the red button.2	Agree	6	30.0
	Disagree	14	70.0

¹Items Copyright © 2014 HANDS Research Team, reprinted with permission.

²Some of the statements have a number of total responses less than 21 due to items not rated.

METHODS

Design/Setting

The CDS prototypes were tested in a cross-sectional study using two patient-care scenarios presented in the CAVE2™ immersive environment at the Electronic Visualization Laboratory of the University of Illinois at Chicago (UIC).¹⁷ The study occurred within a life-size simulated nursing station, including sights and sounds typical for a busy hospital unit, and using a computer containing the functional EHR prototypes into which the participants could enter decisions. The UIC Institutional Review Board granted ethical approval for the study.

The CDS suggestions tested were added to a modified version of the Hands-on Automated Nursing Data System (HANDS), an electronic nursing plan of care (POC) software program (Health Team IQ, Chicago, Illinois). The modified version provided POCs with 2–5 nursing diagnoses (NANDA-I), interventions (NIC) and outcomes (NOC), but did not require that nurses build original care plans from scratch. The CDS suggestions that were added to modified HANDS and tested in this study were previously developed through three rounds of usability testing.^18,19 During the usability testing, CDS message wording, appearance and the presence of an alert button were designed and iteratively refined based analysis of nurses’ responses.

Sample

The sample included 21 subjects who were randomly assigned to a CDS group for a clinical trial comprised of 60 total subjects. There were three CDS groups (n=45) and one control group (n=15). The CDS was delivered in three slightly different formats: 1) narrative suggestions, 2) narrative suggestions and a graph forecasting outcomes, and 3) narrative suggestions and a table forecasting outcomes. The registered nurse subjects were recruited from a major Midwest metropolitan area to meet preset quotas for the purpose of ensuring adequate representation of gender, race, education, and nursing experience within the sample.

Experimental stimulus: CDS feature description

In the study CDS suggestions were made available to nurse subjects when key items on the nursing plan of care indicated the current plan was not likely to lead to desired patient outcomes. The subjects were alerted to the CDS suggestions by a red blinking button (red button). When a red button was selected, a list of CDS suggestions associated with the button would appear (see Figure 1). The CDS suggestions included recommendations for changes in the diagnoses, interventions, and outcomes on the POC to meet the patient’s needs and desired outcomes. The evidence backing the CDS suggestions were derived from the analyses of 1,546 end of life (EOL) patients’ HANDS data and known best practices from the literature.^20–22 In summary, when the red button blinked in the patient’s care plan, it indicated that the action was needed to improve patient outcomes. All suggestions were accessed with a blinking red button, but the information and suggestions were presented in one of the three prototypical forms (see Figure 1). When the subject selected items on a CDS screen, the item(s) would move to the care plan. The appearance of the red button changed according to the nurse’s actions as follows: 1) it disappeared when all recommended suggestions had been added to the POC, 2) it stopped blinking, but remained present, when some but not all of the recommended suggestions were added to the POC, and 3) it continued to blink when no recommended suggestions were added to the POC.

Figure 1.

Nursing care plan examples within the three different HANDS prototypes.

Procedures

The study was conducted in five steps: 1) obtaining the participant’s informed consent, 2) orienting the participant on use of HANDS (including post orientation questions to ensure the understanding of the system interface), 3) reporting to the participant the assessment data for two fictitious patients, 4) asking the participant to use HANDS to update patients’ care plans, and 5) asking the participant to complete the post-experimental survey. One of three trained researchers guided each participant during this process using a standard protocol of procedures to ensure that all the participants received the same instructions. Each participant received $100 for time and travel expenses.

Participants were asked to make and enter decisions in the plans of care for the two fictitious patients over the course of three simulated shifts. The red button was displayed on the screen for the first shift. Then, according to the participant’s acceptance or refusal of the CDS suggestions on the subsequent shifts, the red button reappeared, blinked or did not blink per the previous rules. The maximum number of suggestions that each participant could accept was eight across the two fictitious scenarios (three suggestions for patient #1 and five suggestions for patient #2).

Instruments

The Clinical Decision Support Acceptance Report (CDSAR) questionnaire was created for this study to evaluate the reasons affecting the participant’s acceptance or non-acceptance of the suggestions provided and accessed through the blinking red button. The inspiration for the questionnaire occurred midway through our larger clinical trial study, when team members indicated concern that subjects appeared to accept CDS suggestions simply because the “red button” blinked and was annoying. Since our protocol did not include the ability to assess the nurses’ reasons directly, we created the CDSAR (Table 1) and received IRB approval to administer it. The new instrument was then administered to all remaining subjects (n=21) who were assigned to one of the three CDS groups. The CDSAR items were developed, refined, and face validity established by the five nurse experts who had participated in the collection and analysis of data gathered during our earlier formative usability rounds. The instrument consisted of 11 statements that presented possible reasons for nurses to accept or reject the suggestions accessed through the red button (see Table 1). The statements were not mutually exclusive and represents our early efforts that will be refined and psychometrically evaluated in the future. The response option for each statement was on a 4-point Likert scale that included strongly agree (4), agree (3), disagree (2), and strongly disagree (1).

Analysis

Our first step was to pair the reasons for accepting the CDS suggestions to actual CDS suggestions accepted, and pair response items pertaining to rejection with the CDS suggestions not accepted. The acceptance rates of the eight CDS suggestions varied. After completing the descriptive analysis we conducted more in depth analysis of the four CDS suggestions that were not accepted (rejected) by at least 4 (15%) of the 21 subjects and examined the association between a rejection and the responses to the four items of the survey that assessed the reasons related to non-acceptance (see survey items 7 to 10 in Table 1). These four CDS items were: 1) Prioritize Pain, 2) Prioritize Death Anxiety, 3) Remove Impaired Gas Exchange, and 4) Add Respiratory Monitoring.

There were 12 points possible for the three survey items (4 points each according to the 4-point Likert scale). A higher score for those who did not accept a CDS suggestion compared to those who did (i.e., agreement or strong agreement with survey items 7 to 10), may indicate a lack of trust in the CDS suggestion. Descriptive statistics and t-tests were computed using statistical software R version 3.1.1 (R Development Core Team, Vienna, Austria).

RESULTS

In this diverse 21-participant sample, subjects' mean age was 31.3 years (SD 12.2), 15 (71%) were female, 20 (95%) had a degree of BSN or higher, and the mean years of nursing experience was 5.0 years (SD 10.3). Subjects’ demographics can be found in Table 2.

Table 2.

Nurses sample characteristics.

Variable	Category	Frequency1
Gender	Male	6
	Female	15
Age in Years	Mean (sd)	31.3 (12.2)
	Range	22-71
Ethnicity2	Hispanic	1
	Non- hispanic	20
Race2	White	8
	Black	4
	Asian	7
	Other	2
Experience3	Mean (sd)	5.0 (10.3)
	Range	0-44
Education	ADN	1
	BSN or above	20
Familiar with NANDA-I, NOC, and NIC	Yes	21
	NANDA-I only	0
	No	0

¹Values represent frequency unless category indicates otherwise

²Ethnicity and race were self-reported by the subjects.

³Years of professional experience.

From the 168 suggestions displayed during the experiment (8 each per 21 nurse subjects), the acceptance rate was 73.2% (123 out of 168) and the non-acceptance rate was 26.8% (45 out of 168). Nine participants accepted 80–100% (7–8 suggestions) of the total suggestions, eight accepted 60–70% (5–6 suggestions), three accepted 50% (4 suggestions), and one accepted 25% (2 suggestions). The number of accepted suggestions that occurred most often (mode) among the subjects was seven, and only two subjects accepted all of the eight CDS suggestions.

The response frequencies to the CDSAR appear in Table 1, where the 4-level responses were dichotomized to Agree or Disagree. The main reasons for acceptance of the CDS were: 1) believing or being convinced that the suggestions were good for the patients in the presented case scenarios and 2) because the red button remained present, which implied that the subject had not made enough recommended changes to the POC. The most frequent reasons for refusing the CDS suggestions were disagreement and lack of confidence in the displayed information.

When investigating the most often refused CDS suggestions, as is noted in Table 3, we found that the cumulative ratings for survey items 7–9 were significantly lower (p=.04) for those who accepted the suggestion to Remove Impaired Gas Exchange versus those who did not accept this suggestion, indicating that disagreement with the evidence presented for this CDS suggestion likely contributed to its rejection.

Table 3.

Acceptance rate of four least-accepted CDS suggestions and comparison of subjects’ Clinical Decision Support Acceptance Report mean scores (for questions 7, 8, and 9) by subjects who accepted/not accepted each CDS suggestion.

CDS suggestion	Acceptance rate	Accepted Score Mean (sd)	Not Accepted Score Mean (sd)	p Value
Prioritize Death Anxiety	8 (38.1%)	6.9 (2.0)	6.0 (2.6)	.40
Add Respiratory Monitoring	10 (47.6%)	6.7 (2.2)	6.0 (2.6)	.51
Remove Impaired Gas Exchange	12 (57.1%)	5.4 (2.3)	7.6 (2.0)	.04
Prioritize Pain	15 (71.4%)	6.0 (2.2)	7.2 (2.9)	.40

DISCUSSION

In this study, the top reasons subjects’ indicated for accepting the CDS suggestions were agreement that the suggestions were good for the patient care situations presented and displaying of information that convinced the nurses that the suggestions were helpful to achieving the care goals. These results support previous findings^7,23 in which trust in the CDS system combined with clinical judgment and the belief that the CDS suggestion would help the patient were associated with acceptance by the clinician. Collectively, the findings highlight the importance of framing or presenting CDS evidence in ways that help the clinician see the potential benefit to the patient’s care. These findings have implications in an era of “big data” analytics (analysis of complex data sets), in which best practices can be identified, and yet it is often difficult to provide an explanation to the user that is logical because of the large number of variables included in the analysis. To utilize the outputs of big data analysis for evidence based suggestions, therefore, will potentially require additional interactive refinement to help create suggestions that make logical sense to the clinician at the point of care.

Another interesting finding of our study is that the interface design of the red button, at least in part, motivated acceptance of the CDS suggestions as indicated by the subject’s agreement on the red button questions 3, 4, and 5 (see Table 1). This finding also supports a previous case study research finding¹¹ that the level at which an alert was triggered influenced nurses' use of CDS systems. The continued presence of the red button may well serve not only as a reminder of additional nursing diagnoses or interventions to consider adding, but may also prompt nurses to reflect on the completeness of their care plans.

The pattern of responses to the red button questions (3, 4, and 5, see Table 1) suggests that, in some instances, nurses may have accepted suggestions only because the red button was present or blinking, without invoking clinical judgment. Although further study is needed, the absence of the use of clinical judgment when responding to CDS suggestions is a major safety concern. First, it is essential for the clinician to weigh the applicability of a CDS suggestion to the patient’s circumstances, since it is not realistic to expect that decision support algorithms will always include all relevant patient factors. Second, it is possible for a CDS suggestion to be delivered incorrectly or contain incorrect content.²⁴ In light of these concerns, we strongly advocate for comprehensive testing of all nursing CDS systems prior to full deployment in clinical practice. The testing should include thorough evaluation under simulation that mimics real time conditions followed by short validation studies conducted under real-world clinical conditions. In addition, training should include educating the nurse to exercise clinical judgment before accepting suggestions and taking seriously one’s responsibility to report any suspected errors in the CDS suggestions.

The non-acceptance rate for the CDS suggestions was 26.8% (45 out of 168). The most frequent reasons for non-acceptance were disagreement (33%) and lack of confidence (29%) in the displayed information. Lack of subject trust in the CDS suggestions (a composite score of questions 7 to 10) was particularly remarkable in that those who did not accept the CDS suggestion to Remove Impaired Gas Exchange (see Table 3) had a significantly higher lack of trust in the suggestion (p=.04) versus those who accepted it. Regardless, there was generally low percentage of agreement for the four reasons (questions 7 to 10; Table 3) for non-acceptance. For example, we are not clear why 62% of the nurses did not accept the CDS suggestion to Prioritize Death Anxiety, 52% did not accept the suggestion to add Respiratory Monitoring, and 29% did not accept the Prioritize Pain suggestion. If the primary reason was not disagreement with the evidence, which our results seem to indicate, there were likely other factors that interfered with the acceptance of these suggestions. Perhaps the suggestions were misunderstood or somehow not seen. Additional items thus need to be added to the CDSAR to adequately assess subjects’ reasons for rejecting CDS suggestions.

Understanding the reasons why a healthcare provider accepts or rejects a CDS suggestion is also important because decisions made without solid clinical rationale may have an unintended negative impact on patient outcomes. The importance of this study lies in its contribution to understanding the nurses’ rationale for their decisions about CDS acceptance that can be iteratively applied to building high quality and robust CDS for nurses to improve patient outcomes in the future. Besides that, we began the process of developing a quantitative tool that we intend to refine and adapt for use in evaluating all types of nursing CDS systems in the future. Finally, we found our innovative use of simulation in the CAVE to be an excellent format for pre-testing CDS software before deploying to the practice setting. Studying the CDS under these conditions is less costly than deploying poorly evaluated CDS to the clinical setting for refinement through error reporting by the clinicians. As was noted in the literature, the latter results in costly unintended consequences. The study also exposed the need to examine the potential impact on acceptance of CDS suggestions that repeat frequently across time and patients. For this study, the red button at least in part seemed to serve as a reminder that supported acceptance of CDS suggestions. Nonetheless, seven subjects indicated that they “accepted the suggestions because the red button was annoying me.” Since exposure to the red button and suggestions were new for subjects (and occurred for only two patients across three shifts), further study is warranted to determine the long term effectiveness of repeated exposure to the red button and CDS suggestions.

LIMITATIONS AND FUTURE DIRECTIONS

The two main limitations of this study are the small sample size and the use of an instrument, CDSAR, for which the psychometrics had not yet been fully established. As a result, the findings reported here for this pilot can only be considered as preliminary. Nonetheless, the sample size and diversity of subjects helped us identify important deficits in the CDSAR and in the way that it was administered. For example, we found that the CDSAR instrument is clearly missing items that cover the main reasons for rejecting some CDS suggestions. In addition, we learned that by asking subjects to base their survey responses on all CDS items we were unable to clearly discern why a specific CDS item was accepted or rejected. In addition, the instrument in its current form is specific to our care planning software and we would like to make it adaptable for use in evaluating all forms of nurse CDS efficiently. Further studies, with larger sample sizes, are thus planned to enhance the psychometrics and universality of the CDSAR instrument in evaluating all types of nurse specific CDS systems under high quality simulation. This knowledge will contribute substantially to creating an efficient methodology for fully testing nurse CDS systems prior to wide scale deployment in clinical practice.