Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Sep 1.
Published in final edited form as: Nurs Res. 2017 Sep-Oct;66(5):388–398. doi: 10.1097/NNR.0000000000000234

Toward Meaningful Care Plan Clinical Decision Support: Feasibility and Effects of a Simulated Pilot Study

Gail M Keenan 1, Karen Dunn Lopez 2, Yingwei Yao 3, Vanessa E C Sousa 4, Janet Stifter 5, Alessandro Febretti 6, Andrew Johnson 7, Diana J Wilkie 8
PMCID: PMC5657586  NIHMSID: NIHMS883234  PMID: 28858147

Abstract

Background

Clinical decision support (CDS) tools—with easily understood and actionable information, at the point of care—are needed to help registered nurses (RNs) make evidence-based decisions. Not clear are the optimal formats of CDS tools. Thorough, preclinical testing is desirable to avoid costly errors associated with premature implementation in electronic health records.

Objective

To determine feasibility of the protocol for designed to compare multiple CDS formats, and evaluate effects of numeracy and graph literacy on RN adoption of best practices and care-planning time in a simulated environment.

Methods

In this pilot study, 60 RNs were randomly assigned to one of four CDS conditions (control; text; text+graph; text+table) and asked to adjust the plan of care for two patient scenarios over three shifts. A total of 14 best practices were identified for the two patients and sent as suggestions with evidence to the three CDS groups. Best practice adoption rates, care-planning time, and their relationship to the RN’s numeracy and graph literacy scores were assessed.

Results

CDS groups had a higher adoption rate of best practices (p < .001) across all shifts and decreased care-planning time in shifts two (p = .01) and three (p = .02) compared to the control group. Higher numeracy and graph literacy were associated with shorter care-planning times under text+table (p = .05) and text+graph conditions (p = .01). No significant differences were found between the three CDS groups on adoption rate and care-planning time.

Discussion

This pilot study demonstrates the feasibility of our protocol. Findings show preliminary evidence that CDS improves the efficiency and effectiveness of care-planning decisions, and that the optimal format may depend on individual RN characteristics. We recommend a study with sufficient power to compare different CDS formats, and assess the impact of potential covariates on adoption rates and care-planning time.

Keywords: care plans, decision support, standardized nursing terminologies, usability

Clinical decision support (CDS) shows promise for improving healthcare, but many concerns have yet to be adequately addressed. Here, CDS is defined as providing clinicians with computer-generated, clinical knowledge and patient-related information that is intelligently filtered and presented at appropriate times to enhance patient care (Teich, Osheroff, Pifer, Sittig, & Jenders, 2005). One major concern is determining the optimal formats of CDS to enable clinicians to make high-quality decisions in daily practice. Increasing electronic health record (EHR) adoption in recent years has resulted in large volumes of data that contain evidence about the impact of healthcare. To date, there has been a substantial focus on what data to collect and the type of evidence needed by clinicians at the point of care. Lagging behind is the research on effects of display formats (e.g., graphical, symbolic, textual) of CDS on quality and efficiency of decision making and whether RN literacy moderates these effects. This pilot study aimed to determine the feasibility of a larger trial using a simulated environment to compare multiple CDS formats, and examine the effects of numeracy and graph literacy on nurses’ care-planning decisions.

A main goal of CDS is to enable efficient processing of the large quantity of data in the EHR to support high-quality decisions in clinical practice. The CDS systems offer electronic support at the point of care in a variety of forms, including evidence-based alerts, reminders, guidelines, and best practices (Middleton et al., 2013; Osheroff et al., 2007). There is an extensive literature on medical CDS systems (Bright et al., 2012; Eichner & Das, 2010; Jaspers, Smeulers, Vermeulen, & Peute, 2011), but there are far fewer CDS studies focused on registered nurses (RNs) (Dunn Lopez et al., 2017). The CDS nursing studies have focused on: adherence to specific guidelines (Campion, Waitman, Lorenzi, May, & Gadd, 2011; Dumont & Bourguignon, 2012; Sward, Orme, Sorenson, Baumann, & Morris, 2008; Welch et al., 2015); single condition nursing diagnostic decision making (Fick, Steis, Mion, & Walls, 2011; Lee et al., 2009; Sawyer et al., 2011; Welch et al., 2015); medication dosing (Campion et al., 2011; Dumont & Bourguignon, 2012; Sward, Orme, Sorenson, Baumann, & Morris, 2008); supporting situational awareness (Dowding et al., 2009; Dumont & Bourguignon, 2012; Sward et al., 2008; Welch et al., 2015); and triage decision making (Dowding et al., 2009; Ernesäter, Holmström, & Engström, 2009; Lee et al., 2009). Findings provide preliminary evidence that nursing CDS can improve accuracy (Lee et al., 2009; Yeh et al., 2011) and efficiency of nursing care (Effken, Loeb, Kang, & Lin, 2008; Sawyer et al., 2011) and patient outcomes (Ruland et al., 2010; Welch et al., 2015).

Few studies compared the visual formats of CDS delivered to nurses. The format is nonetheless critical to ensuring that the user can quickly understand and apply the information presented. Creating CDS that are meaningful, generalizable, supportive of nurses’ holistic view of the patient, and actionable at the point of care requires iterative building, testing, and refinement. Careful systematic development of CDS is necessary to ensure that it works as intended once implemented in practice. Researchers have found a number of problems in EHR usability testing (Ratwani, Benda, Hettinger, & Fairbanks, 2015) and a paucity of high-quality studies of EHR usability with two thirds performed at prepost implementation without preclinical usability testing reported (Ellsworth et al., 2017). In addition, half of the largest U.S. EHR vendors are not meeting standards for usability testing, with two thirds conducting tests with fewer than the minimum 15 participants, as suggested by the National Institute of Standards and Technology, and one fifth conducting at least half of their tests using subjects with no clinical background (Ratwani et al., 2015). Substandard usability testing may contribute to serious, unintended consequences in the implementation of health information technologies (Graber, Siegal, Riah, Johnston, & Kenyon, 2015; Han et al., 2005; Nebeker, Hoffman, Weir, Bennett, & Hurdle, 2005).

This pilot study is one of a systematic set of studies (Almasalha et al., 2013; Febretti, Lopez et al., 2013; Keenan et al., 2012; Yao et al., 2013) designed to achieve our team’s long-term goal, “to deliver useful and meaningful care-planning CDS to nurses at the point of care.” It builds upon three preceding iterative cycles that generated the content for the CDS prototypes, and preliminary evidence that graph literacy may predict the most efficient CDS format for an individual (Lopez, Febretti, et al., 2016; Lopez, Wilkie, et al., 2016; Febretti, Lopez, et al., 2013). A high fidelity, simulated environment was utilized in this pilot as a safe and cost-effective step toward the eventual deployment of validated CDS in practice (Kushniruk & Borycki, 2014; Wachter et al., 2003).

Purpose

The study aims are to compare three experimental CDS format groups (text, text+graph, and text+table) to control (No CDS) on RNs’ adoption of best practices and care-planning time, and to examine the effects of numeracy and graph literacy on the adoption rates and time. Though the outcomes evaluated at this stage are not patient outcomes, adoption rate of best practices and care-planning time are expected to directly impact the cost and quality of care in clinical practice.

Methods

Design and Sample

In this pilot study, a diverse sample of 60 RNs was recruited in a Midwestern state and randomly assigned to interact with one of four care-planning software conditions in a session consisting of three shifts. It was part of a larger study (R01NR012949) focused on iterative development of CDS and identification of best nursing practices from a large plan of care (POC) database. The four conditions included one with No CDS (control) and three CDS prototypes that included best practice suggestions accompanied by evidence in one of three different formats: text; text+table; and text+graph (Figure 1).

FIGURE 1.

FIGURE 1

Open in a new tab

Nursing care plan examples within the three different S-HANDS CDS prototypes. CDS = clinical decision support; NANDA-I = North American Nursing Diagnosis-International; NIC = Nursing Interventions Classification; NOC = Nursing Outcomes Classification; S-HANDS = Modified Version of the Hands-on Automated Nursing Data System; Copyright 2014 HANDS Research Team. Used with permission.

To recruit a purposeful sample, subjects were recruited via flyers posted at student centers of community colleges and universities. Additionally, recruitment e-mails were distributed via a university announcement system, and to individuals affiliated with nursing programs at community colleges and universities and community-based, academic, and veteran’s hospitals. Snowball methods were also used. Respondents comprised our sampling frame and were stratified by gender, ethnicity, experience, and education to increase the sample diversity as reported elsewhere (Lopez, Wilkie, et al., 2016).

The study was approved by the university Institutional Review Board and data collected in 2014.

Setting

At the electronic visualization laboratory located at a Midwestern state university college of engineering, we tested the CDS formats in the immersive setting (Febretti, Nishimoto, et al., 2013). The simulated environment included a life-size, simulated nursing station with typical hospital unit sounds and visuals. The study computer with orientation materials, the simulated cases, and the documentation software were located in a quiet space within the simulated nurses station.

Procedures

After informed consent, each RN was randomized to one of the four study groups. We utilized block randomization with a block size of eight to maintain group balance. Group assignment was concealed in a sealed, opaque envelope that was opened by the programmer who then activated the CDS version assigned. All other research staff and the subject were blind to the assignment. A standard protocol was executed by a research assistant (RA) to orient the RN to the basic care-planning software features, and validate understanding and present initial assessments (Shift 1) or updates (Shifts 2 and 3) and other contextual information for two exemplar end-of-life (EOL) patients to the RN. Each RN submitted end of shift POCs for the patients for three hypothetical shifts: one each day for three days. A shift was defined as a consecutive 8-hour period during which the RN was responsible for the care of the two patients. A simulated shift lasted up to 20 minutes and was focused on the RN completing and submitting the POCs for that shift. The subjects were left alone in the simulated nurses station while interacting with the software. When the shift POCs were submitted, the RA re-entered the nurses station and the process was repeated for next shift. At the end of Shift 3, the RNs completed the posttest surveys and received compensation of $100 for time and travel expenses (Figure 2).

FIGURE 2.

FIGURE 2

Open in a new tab

Experimental flow.

Experimental Stimulus

A simulated care-planning experience for two EOL patients across three shifts was the experimental stimulus. The Hands-on Automated Nursing Data System (HANDS) was modified to simulate HANDS (S-HANDS) and served as the basic care-planning software for all four experimental conditions. Control group used the basic (No CDS) S-HANDS version, whereas each CDS group used an S-HANDS version containing one of three CDS prototypes. Diagnoses, interventions, and outcomes were coded respectively with the NANDA-International (NANDA-I) (Herdman & Kamitsuru, 2014), Nursing Interventions Classification (NIC) (Bulechek, Butcher, Dochterman, & Wagner, 2013), and Nursing Outcomes Classification (NOC) (Moorhead, Johnson, Maas, & Swanson, 2014). Two EOL patient scenarios based on real cases were used in this study (see Table, Supplemental Digital Content 1).

The HANDS (2016) is a software program designed for documenting the nursing POCs that are created on admission and updated at every formal handoff (shift change). The S-HANDS has similar functionality to HANDS but includes only NANDA-I, NIC, and NOC terms pertaining to EOL care.

Each of the three CDS prototypes delivered the same 14 best practice suggestions with the evidence presented in one of three different formats (Figure 1). Of the 14, three pertained to Patient 1 and appeared on two pop-up screens while 11 pertained to Patient 2 and appeared on three screens. The evidence supporting the CDS suggestions was derived from the literature and our previous data mining studies (Almasalha et al., 2013; Mercadante, 2014; Yao et al., 2013). The text and graph features were iteratively refined through three cycles of usability testing (Febretti, Lopez, et al., 2013). The short text statements accompanying the suggestions and describing the evidence were presented on popup CDS screens, with additional text information about the evidence accessible by clicking the “i” button. The graph feature, available in the text+graph CDS prototype, illustrated the projected effect of suggested action. The CDS table feature, available in the text+table CDS prototype, presented the same information in tabular form.

Pop-up screens presenting suggestions and evidence were accessed by clicking red, blinking, alert buttons placed next to outcomes requiring attention. When the RN clicked to accept suggestions on a CDS screen, the POC was automatically updated to reflect the new items. If all suggestions on a CDS screen were adopted, the corresponding red button disappeared; if some but not all were adopted, the button remained but stopped blinking. Finally, the red button continued to blink if the RN adopted no suggestions.

RNs in the control group did not have access to CDS suggestions and evidence but could, and did, make changes aligned with the best practice suggestions using the S-HANDS basic functionality. The CDS groups also had the option to bypass the CDS in full or part and make changes to the POC using the basic S-HANDS functionality.

For the second and third shifts, patients’ POCs and patient conditions were updated to reflect the changes made by RNs earlier and the effects of those changes. If an RN assigned to a CDS group had not accepted a CDS suggestion on the earlier shift(s), the CDS suggestion reappeared.

Instruments

Independent variable

The independent predictor is a categorical variable representing the four experimental conditions described above.

Covaxriates

In addition to documenting protocol adherence (for study feasibility) via the software and collecting demographic and experience information about the RNs, we assess their numeracy and graph literacy. The Subjective Numeracy Scale (SNS), including eight items with a variety of Likert-type response options ranging from 1–4 or 1–6, was used to measure RNs’ numeracy skills. Cronbach’s alpha ranged from .82 to .84 (Zikmund-Fisher, Smith, Ubel, & Fargelin, 2007). The 13-item Long Graph Literacy Scale (LGLS) was used to assess RNs’ ability to understand health information presented in graphical forms (bar, pie, icon, and line). Reliability and convergent validity with graph comprehension items from existing literacy scales were previously assessed and reported (Cronbach’s α =.85, r = .44) (Galesic & Garcia-Retamero, 2011). For the current sample, the Cronbach’s alpha was .76 for the SNS and .40 for the LGLS.

Dependant variables

The uptake of the best practice items by RNs was measured by the adoption rate, computed as the percentage of the 14 items adopted. Care-planning time was determined using computer timestamps of user actions and included the time an RN spent reviewing the previous POCs, evaluating the suggestions and related evidence, making decisions, and updating and submitting the POC. Previous studies have shown that nurses spend on average 21.5% (Philipsen et al., 2014) of their time in documentation. Reducing documentation time could thus increase direct patient care time. Feasibility of the study was measured by the proportion of subjects completing the study and amount of missing data.

Analysis

Analysis was conducted using R statistical software. Descriptive statistics including mean, standard deviation, frequency, and percentage were generated. ANOVA and independent samples t-tests were used for group comparisons. Linear mixed-effect models with random intercept terms to accommodate between-subject differences were used to examine the effects of CDS, as well as numeracy and graph literacy on adoption rate. Restricted maximum likelihood (REML) method was used to produce estimates of model parameters. Posterior predictive checking, a Bayesian-based diagnostic method, was used to validate model fit (Gelman & Hill, 2006). Statistical significance was set at a two-sided alpha of .05.

Results

Participant Characteristics

Of subjects beginning the pilot study, 100% completed the protocol. The missing data were minimal (0.1%). Table 1 shows participant characteristics. Overall, 60 RNs participated in the study, 80% of whom were female. The participants were between 21 to 71 years old (M = 33.7, SD = 10.8 years), with 42% Caucasian, 22% Black, 27% Asian, and 10% identified as other races; a small minority (8%) were Hispanic. Nearly all (77%) had at least one year of nursing experience (M = 8.1, SD = 9.7 years). Most were college graduates, with 23% having an MSN or higher, 70% having a BSN, and 7% having an ADN.

TABLE 1.

Participant Characteristics and Experimental Outcomes

All
No CDS
Text
Text+Table
Text+Graph
Characteristic n (%) n (%) n (%) n (%) n (%)
Gender
  Male 12 (20) 3 (19) 3 (20) 3 (21) 3 (20)
  Female 48 (80) 13 (81) 12 (80) 11 (79) 12 (80)
Ethnicity
  Hispanic 5 (8) 1 (6) 0 (0) 2 (14) 2 (13)
  non-Hispanic 55 (92) 15 (94) 15 (100) 12 (86) 13 (87)
Race
  White 25 (42) 7 (44) 9 (60) 5 (36) 4 (27)
  Black 13 (22) 3 (19) 3 (20) 3 (21) 4 (27)
  Asian 16 (27) 5 (31) 3 (20) 3 (21) 5 (33)
  Other 6 (10) 1 (6) 0 (0) 3 (21) 2 (13)
Education
  ADN 4 (7) 2 (13) 1 (7) 1 (7) 0 (0)
  BSN 42 (70) 10 (63) 9 (60) 10 (71) 13 (87)
  MSN or more 14 (23) 4 (25) 5 (33) 3 (21) 2 (13)
M (SD) M (SD) M (SD) M (SD) M (SD)
Age 33.7 (10.8) 34.2 (9.6) 32.2 (9.8) 32.1 (11.0) 36.1 (13.3)
Experience 8.1 (9.7) 9.9 (10.3) 7.3 (7.5) 4.6 (6.8) 10.1 (12.8)
Outcome/shift
Adoption rate (%)
  Shift 1 68 (26) 38 (15) 83 (20) 79 (19) 77 (20)
  Shift 2 66 (23) 45 (11) 79 (25) 66 (19) 76 (19)
  Shift 3 66 (20) 49 (13) 76 (22) 67 (16) 74 (18)
Time (minutes)
  Shift 1 7.9 (3.6) 8.1 (3.4) 7.6 (3.2) 8.1 (4.5) 7.7 (3.5)
  Shift 2 3.0 (1.6) 3.8 (1.3) 2.5 (1.4) 3.0 (1.7) 2.7 (1.6)
  Shift 3 2.6 (1.5) 3.3 (1.3) 2.1 (1.4) 2.8 (1.4) 2.2 (1.6)
NANDAs (number)
  Shift 1 8.1 (1.4) 9.1 (1.6) 7.8 (1.3) 7.7 (1.1) 7.9 (1.2)
  Shift 2 8.2 (1.5) 9.1 (1.7) 7.8 (1.2) 7.7 (1.7) 7.9 (1.2)
  Shift 3 8.1 (1.6) 9.0 (1.5) 7.7 (1.3) 7.6 (1.8) 7.8 (1.4)
NOCs (number)
  Shift 1 9.6 (2.0) 10.6 (2.3) 9.3 (2.3) 8.9 (1.2) 9.5 (1.7)
  Shift 2 9.9 (2.3) 11.1 (2.3) 9.5 (2.4) 8.9 (1.9) 9.7 (1.9)
  Shift 3 10.2 (2.6) 11.7 (3.0) 10.1 (2.4) 8.8 (2.0) 10.1 (2.4)
NICs (number)
  Shift 1 28.1 (8.0) 30.1 (9.8) 29.2 (8.5) 25.9 (6.7) 26.9 (6.2)
  Shift 2 31.6 (10.1) 37.4 (11.0) 32.0 (9.7) 26.4 (8.8) 29.7 (8.3)
  Shift 3 33.7 (10.1) 41.1 (9.2) 33.0 (8.9) 28.8 (7.6) 30.9 (10.6)
Open in a new tab

Note. N = 60. CDS = clinical decision support; NANDA-I = North American Nursing Diagnosis-International; NIC = Nursing Interventions Classification; NOC = Nursing Outcomes Classification; SD = standard deviation.

Adoption Rate

The means and standard deviations of the adoption rates over the course of three shifts also appear in Table 1. The adoption rates of the best practice items for CDS group were substantially higher than the control across all shifts: M = 80%, SD = 20% versus M = 38%, SD = 15% for Shift 1; M = 74%, SD = 22% versus M = 45%, SD = 11% for Shift 2; and M = 73%, SD = 19% versus M = 49%, SD = 13% for Shift 3, respectively (p < .001 for all three shifts). We also observed that as time progressed, control group RNs added more of recommended CDS items (though not available to them in a CDS format), leading to higher adoption rates in later shifts; while the adoption rates for the CDS groups decreased slightly over time.

Regression analysis confirmed this observation (Table 2). We set the control (No CDS) as the reference and compared each CDS group against it. All CDS groups had significantly higher adoption rates (p < .001 for all three CDS groups); the adoption rate of every CDS group decreased over time, significantly for the text (p = .03) and text+table groups (p = .001); on the other hand, the adoption rate of the No CDS group increased significantly over time (p = .001).

TABLE 2.

Regression: Adoption Rate on CDS Condition and Shift

Predictor b (SE) p
Groupa
  Text 0.45 (0.065) <.001
  Text+Table 0.38 (0.066) <.001
  Text+Graph 0.39 (0.065) <.001
Shift
  No CDS 0.05 (0.015) .001
  Text −0.04 (0.016) .03
  Text+Table −0.06 (0.016) .001
  Text+Graph −0.01 (0.016) .45
Open in a new tab

Note. N = 60. SE = standard error.

a

Reference group is no CDS.

To determine whether there were significant differences among the three CDS groups, we compared this model with a reduced model treating the three CDS groups as a single group. If the difference among the three CDS formats (text, text+table, and text+graph) was not significant, then merging them into a single group will not reduce model fit. A likelihood ratio test showed no significant difference between the two models indicating that adoption rate difference between the three CDS groups was not statistically significant (p = .20).

We also compared the adoption rates of individual items at Shift 1 for the No CDS and for the CDS groups (see Figure, Supplemental Digital Content 2.) The adoption rates of the CDS groups were higher for every item, and the difference was significant for all but four items. Two items, add NOC: Immobility Consequences: Physiological and add NIC: Pressure Ulcer Prevention, were adopted by 94% of the control group and therefore there was little room for improvement. For two other items, prioritize Pain and add NIC: Respiratory Monitoring, the CDS group had 16%-20% higher adoption rates, but the difference was not statistically significant in this small sample. For the CDS group, the adoption rates for the six items in the respiratory problem mini-care plan as well as the prioritize Death Anxiety item for Patient 2 were relatively low (59%-70%); whereas the remaining items all had very high adoption rates (89%-100%). In the control group, on the other hand, only four items were in the majority (> 50%) of the POCs, while the prioritize Death Anxiety item and five items related to removing treatments were rarely adopted (0%-13%).

The changes in adoption rates from Shift 1 to Shift 3 were also examined. In the CDS groups, the adoption rates decreased across the three shifts with RNs dropping previously adopted items (four items were dropped by 15% or more of the RNs). In contrast, in the control group, the adoption increased across shifts, with five items being added by 19% or more of the RNs in Shifts 2–3. Closer examination revealed that the drop of items; prioritize Pain, add Positioning, and add Respiratory Monitoring could be attributed to removal of the NANDA-I: Acute Pain from a POC once a patients’ pain improved. Similarly, in our simulated scenario, prioritization of the NANDA-I: Death Anxiety resulted in improvement of the related outcome in the subsequent shifts leading some RNs to change the top priority NANDA-I.

Care-Planning Time

The mean and standard deviation of RN care-planning time (in minutes) can also be found in Table 1. There is little difference between CDS groups and the control in the first shift (M = 8.1, SD = 3.4 minutes for control vs. M = 7.8, SD = 3.7 minutes for CDS, p = .80). At Shifts 2 and 3, however, the CDS group, on average, spent only 70% of the time needed by the control. In Shift 2, the control group spent M = 3.8, SD = 1.3 minutes versus M = 2.7, SD = 1.5 minutes for CDS groups (p = .01). In Shift 3, the time spent was M = 3.3, SD = 1.3 minutes versus M = 2.3, SD = 1.5 minutes in favor of CDS groups (p = .02). Comparison among the three CDS groups showed no significant time difference for any of the three shifts.

There was significant, positive correlation between care-planning time and the adoption rate for the control group on the first two shifts (r = .62, p = .01 for Shift 1 and r = .57, p = .02 for Shift 2). For the last shift, this correlation was minimal (r = .11, p = .73). For the CDS groups, the correlations between the care-planning time and the adoption rate were weak and not statistically significant for all shifts.

Numeracy and Graph Literacy with CDS Adoption and POC Entry Time

We did not find any significant association between RN characteristics and adoption rates. Then again, our examination of the relationships between numeracy and graph literacy and time spent care planning revealed two significant findings (Table 3). A higher numeracy score (1 point) was associated with a significant reduction (1.1 minute) in time spent for an RN in the text+table group (p = .05). A higher graph literacy score (1 point) was associated with a reduced (0.8 minute) time in the text+graph group (p = .01). Regression output details for Tables 2 and 3 are presented (see Table, Supplemental Digital Content 3).

TABLE 3.

Regression: Care-Planning Time on Group and Its Interaction With Shift, Numeracy, and Graph Literacy

Predictor/value b (SE) p
Groupa
  Text+Graph −0.66 (0.902) .47
  Text+Table −0.11 (0.891) .90
  Text −0.62 (0.872) .48
Shiftb
  No CDS −2.37 (0.430) <.001
  Text+Graph −2.79 (0.444) <.001
  Text+Table −2.67 (0.460) <.001
  Text −2.76 (0.444) <.001
Numeracyc
  No CDS −0.10 (0.510) .85
  Text+Graph 0.23 (0.657) .73
  Text+Table −1.14 (0.563) .05
  Text −0.25 (0.652) .70
Graph literacyd
  No CDS 0.34 (0.376) .37
  Text+Graph −0.79 (0.299) .01
  Text+Table −0.02 (0.368) .95
  Text 0.28 (0.319) .39
Open in a new tab

Note. N = 60. CDS = clinical decision support.

a

Reference group is No CDS.

b

The coefficients represent rates of change over shifts (1–3) in the four groups.

c

The coefficients indicate the association between care-planning time and numeracy in the four groups.

d

The coefficients indicate the association between care-planning time and graph literacy in the four groups.

Variation in Care

We compared the number of NANDA-I diagnoses, NOC outcomes, and NIC interventions entered by RNs assigned to different groups (Table 1). The number of NANDA-Is on the POCs stayed constant through the shifts, with the control group entering 1.3 more NANDA-Is into the POCs for the two patients and the difference was significant for all three shifts (Shift 1: p = .01; Shift 2: p = .02; Shift 3: p = .01). We also observed a significantly higher number of NOC labels on the POCs of the control relative to CDS groups through all three shifts (Shift 1: p = .04; Shift 2: p = .02; Shift 3: p = .03), with the number increasing over time for both the control and CDS groups. Regression showed significant group difference at Shift 1 (p = .04) and increase over time for CDS groups (p = .03). The control had a larger increase over time, but the increase rate difference with CDS groups was not statistically significant (p = .12). Similarly, the No CDS group entered more NIC labels than the CDS group and the difference was statistically significant for the latter two shifts (Shift 2: p = .02; Shift 3: p = .001). Regression analysis showed a significant increase across time in CDS groups (p < .001), as well as a significantly higher increase rate over time in the control group (p < .001).

Discussion

This pilot study demonstrated feasibility of our innovative protocol and uncovered important trends (some statistically significant)—that justify a larger study powered for differences between CDS groups—instead of just between CDS and No CDS. More specifically, it was feasible to recruit, randomize, orient, and retain a diverse group of 60 RNs through the entire protocol, fully test the automated intervention using a high-fidelity, simulated environment, and collect data with minimal missing data. Surprisingly, we observed statistically significant findings in this small pilot study, but they require confirmation in a larger study. Our analyses of the data revealed findings relevant to planning future studies of CDS in four areas: item adoption rates; time spent care planning; the RNs’ graph literacy and numeracy; and overall size of the POC across time.

With regards to the adoption rates of best practices items, we observed that these can be grouped into three distinct categories based on RNs actions in the No CDS group and CDS groups. The first category included those items for which an overwhelming majority (> 90%) of RNs added on the first shift even for the No CDS group (e.g., monitor NOC: Immobility Consequences; add NIC: Skin Surveillance). The second category included items that very few RNs would adopt without CDS support, with a substantial minority (> 30%) not adopting even with CDS (e.g., remove POC elements addressing the NANDA-I Impaired Gas Exchange). The third contained items adopted on the first shift by a large majority of RNs with CDS support and a substantially smaller portion of RNs without CDS support. The items in this category, nevertheless, were eventually adopted by a substantial portion of the No CDS group over time (e.g., prioritize Pain; add NIC: Positioning to treat Pain; add NIC: Palliative Care Consultation).

These findings provide preliminary evidence that CDS can serve as a reminder for RNs to add items that might otherwise be forgotten or added in a less timely manner without CDS. For items most RNs would enter without prompt, though, CDS suggestions might not be necessary. Excessive CDS messages might create alert fatigue, resulting in RNs ignoring appropriate suggestions. On the other hand, it is less clear why some of the CDS items were never adopted. In a follow-up survey of a subsample of our study, 100% of the RNs (n = 21) receiving the CDS indicated that the main reason for adopting items was their belief that the suggestion(s) were good for the patient (Sousa et al., 2015). Less than half of the RNs, however, indicated that failure to adopt an item was partly due to a lack of confidence in the evidence, indicating the need to probe more thoroughly for these causes (Sousa et al., 2015). Nonetheless, RNs’ unwillingness to adopt CDS suggestions across time indicated that RNs did exercise critical thinking when presented with CDS suggestions.

Our care-planning time analysis also supported the advantages of CDS. Although RNs in CDS groups had to spend time interacting with the CDS user interface and absorbing evidence related to CDS items, they did not spend more time on the POC than RNs in the control group. In fact, they spent significantly less time on updating patients’ POCs on subsequent shifts. Furthermore, there was moderately positive and significant correlation between care-planning time and adoption rate in the control group, while the correlation was weak and insignificant in CDS groups. The potential of a well-designed CDS system to reduce care-planning time has major implications for improving RN efficiency and reducing RN workload.

Our pilot study also provided evidence of the potential effects of RNs’ numeracy and graph literacy, indicating that these may be important factors in designing CDS. Higher numeracy and graph literacy scores were associated with lower care-planning time in text+table and text+graph groups, respectively. Substantiation of these findings with a larger study has the potential to underscore the efficiencies in decision making that can be gained by tailoring the CDS format to fit users’ skills (Lopez, Wilkie, et al., 2016).

Our examination of the POCs’ content indicated another potential benefit of CDS. Specifically, we found that RNs in the control group added significantly more NANDA-I problems, NOC outcomes and NIC interventions to the POC in Shifts 2 and 3 compared to the CDS groups. Since the patient scenarios were consistent across all four groups, the findings suggest RNs may be less sure and, as a result, added unnecessary elements when CDS was not available versus when it was available. In contrast, these findings also raise the question of whether RNs will come to be overly dependent on CDS, adopting suggestions without critical evaluation, and failing to identify problems and treatments not presented in the CDS. Both of these hypothesis warrant further study.

Although we did not find significant differences between the three CDS groups in this pilot study with a very small sample size, we did find trends that warrant a further study comparing them. For example, RNs in the text+table group spent about 0.5 minute more than either text group or text+graph group per shift. Furthermore, significant interactions between numeracy and graph literacy with CDS group assignment indicated a potential need for tailoring the CDS format to individual RNs. With a properly powered study, we will be able to identify the optimal CDS format for RN adoption rate and efficiency tailored to RN characteristics.

Despite our attention to study rigor, as with all research, there are some limitations. Our sample size was too small to be considered fully powered. That being said, the data from this study will inform study design and power analysis for future research by this research team and others. Second, one could posit that since no “inappropriate suggestions” were included, we were unable to assess the impact on critical thinking. This may in part be true, however, several of our prompts were designed to promote critical thinking specifically about EOL care. For example, a patient with impaired gas exchange and labored breathing may benefit from Acid Base Monitoring (often through painful invasive blood gas measurement). However, the patient in our scenario is a “do not resuscitate” with end stage chronic obstructive pulmonary disease and a major goal is to move her towards a more comfortable death. Nurses who critically assess this EOL patient are likely to realize that painful and invasive Acid Base Monitoring procedures are no longer indicated. Furthermore, our finding that nurses did not accept all of the decision support is highly suggestive of the critical thinking of the nurse subjects. Finally, a simulated environment—even the one used in this experiment with visual and auditory similarities to a hospital unit—cannot truly reflect the temporal demands and interruptions of a typical acute-care setting. However, we believe that conducting health information technology research under simulated conditions can play a pivotal role in promoting the appropriate design of technologies that are both safe and effective in clinical practice. Technologies introduced into practice without adequate testing have potential for increased workload, frustration, and patient harm.

Conclusion

This pilot study provided rationale for a larger, randomized, controlled trial of our CDS formats and also generated evidence supporting further evaluation of other factors examined. We demonstrated that different formats of CDS can be successfully studied using high-fidelity, simulated environment (setting and software) as a research approach that allowed us to mimic longitudinal conditions in a condensed time period. Further, our simulated conditions offer a safety benefit by enabling the discovery and fixing of unintended consequences of CDS prior to real-world testing. Since big data science in healthcare is expected to yield an increasing amount of evidence in the near term, it is crucial to ensure high quality CDS is available to deliver actionable evidence in a meaningful and useful format at the point of care. To date, research examining the impact of the CDS format on decision making is lagging seriously behind. This innovative pilot study laid the foundation for a larger more generalizable study that will advance CDS science supportive of clinicians’ decisions that dramatically improve patient outcomes.

Supplementary Material

Supplemental Data File _doc_ pdf_ etc.__1

Supplemental Digital Content 1. Two EOL patient scenarios based on real cases are shown in table. .pdf.

Supplemental Data File _doc_ pdf_ etc.__2

Supplemental Digital Content 2. Figure comparing the adoption rates of individual items at Shift 1 for the No CDS and for the CDS groups. .pdf.

Supplemental Data File _doc_ pdf_ etc.__3

Supplemental Digital Content 3. Table presenting regression output details for Tables 2 and 3. .doc.

Acknowledgments

The research was made possible by Grant Numbers 1R01NR012949, P30NR010680-S1, and R01HS015054 from the National Institutes of Health, National Institute for Nursing Research (NINR), and the Agency for Healthcare Research and Quality (AHRQ). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NINR or AHRQ.

The authors wish to thank the 60 RNs who participated in this study, and Tamara Macieira and Madison Smith (BSN, Graduate Research Assistants) for their help in submitting this manuscript. The authors also thank Dr. Marie Suarez, project director, Veronica Angulo, research specialist for recruitment, David Shuey, research specialist, Kevin McDaniel, research assistant, Brenda Burke, research assistant, and G. W. Douglas, research assistant for assistance with subject recruitment, data collection and processing. Some of the preliminary findings of this study were reported in a conference paper presented at the IEEE VIS 2014 Workshop on Electronic Health Record Data Visualization (EHRVis), Paris, France, November, 2014.

Footnotes

The first author is a principal in the company whose software was modified by the research team specifically for this study and has a conflict management plan in place at the University of Florida. The other authors have no conflicts of interest to report.

Contributor Information

Gail M Keenan, Professor and Annabel Davis Jenks Endowed Chair for Teaching and Research in Clinical Nursing Excellence, Department of Family, Community and Health Systems Science, College of Nursing, University of Florida, Gainesville, FL.

Karen Dunn Lopez, Department of Health Systems Science, College of Nursing, University of Illinois at Chicago, Chicago, IL.

Yingwei Yao, Department of Biobehavioral Nursing Science, College of Nursing, University of Florida, Gainesville, FL, and, Department of Biobehavioral Health Science, College of Nursing, University of Illinois at Chicago, Chicago, IL.

Vanessa E. C. Sousa, Department of Health Systems Science, College of Nursing, University of Illinois at Chicago, Chicago, IL.

Janet Stifter, Department of Health Systems Science, College of Nursing, University of Illinois at Chicago, Chicago, IL.

Alessandro Febretti, Electronic Visualization Laboratory, Department of Computer Science, College of Engineering, University of Illinois at Chicago, Chicago, IL.

Andrew Johnson, Electronic Visualization Laboratory, Department of Computer Science, College of Engineering, University of Illinois at Chicago.

Diana J. Wilkie, Prairieview Trust – Earl and Margo Powers Endowed Professor, and, Director, Academic Center of Excellence in Palliative Care Research and Education, Department of Biobehavioral Nursing Science, College of Nursing, University of Florida, Gainesville, FL, and Professor, Department of Biobehavioral Health Science, College of Nursing, University of Illinois at Chicago.

References

  1. Almasalha F, Xu D, Keenan GM, Khokhar A, Yao Y, Chen YC, Wilkie DJ. Data mining nursing care plans of end-of-life patients: A study to improve healthcare decision making. International Journal of Nursing Knowledge. 2013;24:15–24. doi: 10.1111/j.2047-3095.2012.01217.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, Lobach D. Effect of clinical decision-support systems A systematic review. Annals of Internal Medicine. 2012;157:29–43. doi: 10.7326/0003-4819-157-1-201207030-00450. [DOI] [PubMed] [Google Scholar]
  3. Bulechek GM, Butcher HL, Dochterman JM, Wagner C, editors. Nursing interventions classification (NIC) 6. St. Louis, MO: Elsevier; 2013. [Google Scholar]
  4. Campion TR, Jr, Waitman LR, Lorenzi NM, May AK, Gadd CS. Barriers and facilitators to the use of computer-based intensive insulin therapy. International Journal of Medical Informatics. 2011;80:863–871. doi: 10.1016/j.ijmedinf.2011.10.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Dowding D, Mitchell N, Randell R, Foster R, Lattimer V, Thompson C. Nurses’ use of computerised clinical decision support systems: A case site analysis. Journal of Clinical Nursing. 2009;18:1159–1167. doi: 10.1111/j.1365-2702.2008.02607.x. [DOI] [PubMed] [Google Scholar]
  6. Dumont C, Bourguignon C. Effect of a computerized insulin dose calculator on the process of glycemic control. American Journal of Critical Care. 2012;21:106–115. doi: 10.4037/ajcc2012956. [DOI] [PubMed] [Google Scholar]
  7. Dunn Lopez K, Gephart SM, Raszewski R, Sousa V, Shehorn LE, Abraham J. Integrative review of clinical decision support for registered nurses in acute care settings. Journal of the American Medical Informatics Association. 2017;24:441–450. doi: 10.1093/jamia/ocw084. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Effken JA, Loeb RG, Kang Y, Lin ZC. Clinical information displays to improve ICU outcomes. International Journal of Medical Informatics. 2008;77:765–777. doi: 10.1016/j.ijmedinf.2008.05.004. [DOI] [PubMed] [Google Scholar]
  9. Eichner J, Das M. Challenges and barriers to clinical decision support (CDS) design and implementation experienced in the Agency for Healthcare Research and Quality CDS demonstrations. (AHRQ Publication No. 10-0064-EF) Rockville, MD: Agency for Healthcare Research and Quality; 2010. [Google Scholar]
  10. Ellsworth MA, Dziadzko M, O’Horo JC, Farrell AM, Zhang J, Herasevich V. An appraisal of published usability evaluations of electronic health records via systematic review. Journal of the American Medical Informatics Association. 2017;24:218–226. doi: 10.1093/jamia/ocw046. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Ernesäter A, Holmström I, Engström M. Telenurses’ experiences of working with computerized decision support: Supporting, inhibiting and quality improving. Journal of Advanced Nursing. 2009;65:1074–1083. doi: 10.1111/j.1365-2648.2009.04966.x. [DOI] [PubMed] [Google Scholar]
  12. Febretti A, Lopez K, Stifter J, Johnson AE, Keenan GM, Wilkie DJ. A component-based evaluation protocol for clinical decision support interfaces. In: Bulechek GM, Marcus A, editors. Lecture notes in computer science: Vol. 8012. Design, user experience, and usability Design philosophy methods, and tools. DUXU 2013. 2013. pp. 232–241. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Febretti A, Nishimoto A, Thigpen T, Talandis J, Long L, Pirtle JD, Leigh J. CAVE2: A hybrid reality environment for immersive simulation and information analysis. IS&T/SPIE Electronic Imaging. 2013:864903. International Society for Optics and Photonics. [Google Scholar]
  14. Febretti A, Sousa VEC, Dunn Lopez K, Yao Y, Johnson A, Keenan GM, Wilkie DJ. One size doesn’t fit all: The efficiency of graphical, numerical and textual clinical decision support for nurses. Proceedings of IEEE VIS 2014 Workshop on Electronic Health Record Data Visualization (EHRVis) 2014 Retrieved from http://www.cs.umd.edu/hcil/parisehrvis/papers/clinical_decision_support.pdf.
  15. Fick DM, Steis MR, Mion LC, Walls JL. Computerized decision support for delirium superimposed on dementia in older adults. Journal of Gerontological Nursing. 2011;37:39–47. doi: 10.3928/00989134-20100930-01. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Galesic M, Garcia-Retamero R. Graph literacy: A cross-cultural comparison. Medical Decision Making. 2011;31:444–457. doi: 10.1177/0272989&#x000d7;10373805. [DOI] [PubMed] [Google Scholar]
  17. Gelman A, Hill J. Data analysis using regression and multilevel/hierarchical models. New York, NY: Cambridge University Press; 2006. [Google Scholar]
  18. Graber ML, Siegal D, Riah H, Johnston D, Kenyon K. Electronic health record-related events in medical malpractice claims. Journal of Patient Safety. 2015 doi: 10.1097/PTS.0000000000000240. Advance online publication. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Han YY, Carcillo JA, Venkataraman ST, Clark RSB, Watson RS, Nguyen TC, Orr RA. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics. 2005;116:1506–1512. doi: 10.1542/peds.2005-1287. [DOI] [PubMed] [Google Scholar]
  20. HANDS. Computer software. Chicago, IL: Health Team IQ; 2016. [Google Scholar]
  21. Herdman TH, Kamitsuru S, editors. NANDA international nursing diagnoses: Definitions & classification. Oxford, UK: Wiley Blackwell; 2014. pp. 2015–2017. [Google Scholar]
  22. Jaspers MWM, Smeulers M, Vermeulen H, Peute LW. Effects of clinical decision-support systems on practitioner performance and patient outcomes: A synthesis of high-quality systematic review findings. Journal of the American Medical Informatics Association. 2011;18:327–334. doi: 10.1136/amiajnl-2011-000094. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Keenan GM, Yakel E, Yao Y, Xu D, Szalacha L, Tschannen D, Wilkie DJ. Maintaining a consistent big picture: Meaningful use of a web-based POC EHR system. International Journal of Nursing Knowledge. 2012;23:119–133. doi: 10.1111/j.2047-3095.2012.01215.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Kushniruk AW, Borycki EM. Designing and conducting low-cost in-situ clinical simulations: A methodological approach. Studies in Health Technology and Informatics. 2014;205:890–894. doi: 10.3233/978-1-61499-432-9-890. [DOI] [PubMed] [Google Scholar]
  25. Lee NJ, Chen ES, Currie LM, Donovan M, Hall EK, Jia H, Bakken S. The effect of a mobile clinical decision support system on the diagnosis of obesity and overweight in acute and primary care encounters. Advances in Nursing Science. 2009;32:211–221. doi: 10.1097/ANS.0b013e3181b0d6bf. [DOI] [PubMed] [Google Scholar]
  26. Lopez KD, Febretti A, Stifter J, Johnson A, Wilkie DJ, Keenan G. Toward a more robust and efficient usability testing method of clinical decision support for nurses derived from nursing electronic health record data. International Journal of Nursing Knowledge. 2016 doi: 10.1111/2047-3095.12146. Advance online publication. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Lopez KD, Wilkie DJ, Yao Y, Sousa V, Febretti A, Stifter J, Keenan GM. Nurses’ numeracy and graphical literacy: Informing studies of clinical decision support interfaces. Journal of Nursing Care Quality. 2016;31:124–130. doi: 10.1097/NCQ.0000000000000149. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Mercadante S. Managing difficult pain conditions in the cancer patient. Current Pain and Headache Reports. 2014;18:1–7. doi: 10.1007/s11916-013-0395-y. [DOI] [PubMed] [Google Scholar]
  29. Middleton B, Bloomrosen M, Dente MA, Hashmat B, Koppel R, Overhage JM, Zhang J. Enhancing patient safety and quality of care by improving the usability of electronic health record systems: Recommendations from AMIA. Journal of the American Medical Informatics Association. 2013;20:E2–E8. doi: 10.1136/amiajnl-2012-001458. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Moorhead S, Johnson M, Maas M, Swanson E, editors. Nursing outcomes classification (NOC) 5. St. Louis, MO: Elsevier; 2014. [Google Scholar]
  31. Nebeker JR, Hoffman JM, Weir CR, Bennett CL, Hurdle JF. High rates of adverse drug events in a highly computerized hospital. Archives of Internal Medicine. 2005;165:1111–1116. doi: 10.1001/archinte.165.10.1111. [DOI] [PubMed] [Google Scholar]
  32. Osheroff JA, Teich JM, Middleton B, Steen EB, Wright A, Detmer DE. A roadmap for national action on clinical decision support. Journal of the American Medical Informatics Association. 2007;14:141–145. doi: 10.1197/jamia.M2334. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Philipsen N, Carruthers W, Chi G, Ensey D, Shmorhun A, Valdez R. A mixed-methods assessment of time spent documenting by nurses using an electronic medical records system. Systems and Information Engineering Design Symposium (SIEDS) 2014;2014:118–123. doi: 10.1109/SIEDS.2014.6829925. [DOI] [Google Scholar]
  34. Ratwani RM, Benda NC, Hettinger AZ, Fairbanks RJ. Electronic health record vendor adherence to usability certification requirements and testing standards. JAMA. 2015;314:1070–1071. doi: 10.1001/jama.2015.8372. [DOI] [PubMed] [Google Scholar]
  35. Ruland CM, Holte HH, Røislien J, Heaven C, Hamilton GA, Kristiansen J, Ellison MC. Effects of a computer-supported interactive tailored patient assessment tool on patient care, symptom distress, and patients’ need for symptom management support: A randomized clinical trial. Journal of the American Medical Informatics Association. 2010;17:403–410. doi: 10.1136/jamia.2010.005660. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Sawyer AM, Deal EN, Labelle AJ, Witt C, Thiel SW, Heard K, Kollef MH. Implementation of a real-time computerized sepsis alert in nonintensive care unit patients. Critical Care Medicine. 2011;39:469–473. doi: 10.1097/CCM.0b013e318205df85. [DOI] [PubMed] [Google Scholar]
  37. Sousa VE, Lopez KD, Febretti A, Stifter J, Yao Y, Johnson A, Keenan GM. Use of simulation to study nurses’ acceptance and nonacceptance of clinical decision support suggestions. CIN: Computers, Informatics, Nursing. 2015;33:465–472. doi: 10.1097/CIN.0000000000000185. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Sward K, Orme J, Jr, Sorenson D, Baumann L, Morris AH. Reasons for declining computerized insulin protocol recommendations: Applications of a framework. Journal of Biomedical Informatics. 2008;41:488–497. doi: 10.1016/j.jbi.2008.04.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Teich JM, Osheroff JA, Pifer EA, Sittig DF, Jenders RA. Clinical decision support in electronic prescribing: Recommendations and an action plan. Journal of the American Medical Informatics Association. 2005;12:365–376. doi: 10.1197/jamia.M1822. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Wachter SB, March JA, Noah S, Drews F, Weinger MB, Westenskow D. The employment of an interative design process to develop a pulmonary graphical display. Journal of the American Medical Informatics Association. 2003;10:363–372. doi: 10.1197/jamia.M1207. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Welch G, Zagarins S, Santiago-Kelly P, Rodriguez Z, Bursell SE, Rosal MC, Gabbay RA. An internet-based diabetes management platform improves team care and outcomes in an urban Latino population. Diabetes Care. 2015;38:561–567. doi: 10.2337/dc14-1412. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Yao Y, Keenan G, Al-Masalha F, Lopez KD, Khokar A, Johnson A, Wilkie DJ. Current state of pain care for hospitalized patients at end of life. American Journal of Hospice and Palliative Medicine®. 2013;30:128–136. doi: 10.1177/1049909112444458. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Yeh SP, Chang CW, Chen JC, Yeh WC, Chen PC, Chuang SJ, Peng CJ. A well-designed online transfusion reaction reporting system improves the estimation of transfusion reaction incidence and quality of care in transfusion practice. American Journal of Clinical Pathology. 2011;136:842–847. doi: 10.1309/AJCPOQNBKCDXFWU3. [DOI] [PubMed] [Google Scholar]
  44. Zikmund-Fisher BJ, Smith DM, Ubel PA, Fagerlin A. Validation of the subjective numeracy scale: Effects of low numeracy on comprehension of risk communications and utility elicitations. Medical Decision Making. 2007;27:663–671. doi: 10.1177/0272989&#x000d7;07303824. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplemental Data File _doc_ pdf_ etc.__1

Supplemental Digital Content 1. Two EOL patient scenarios based on real cases are shown in table. .pdf.

NIHMS883234-supplement-Supplemental_Data_File__doc__pdf__etc___1.pdf (118.7KB, pdf)
Supplemental Data File _doc_ pdf_ etc.__2

Supplemental Digital Content 2. Figure comparing the adoption rates of individual items at Shift 1 for the No CDS and for the CDS groups. .pdf.

NIHMS883234-supplement-Supplemental_Data_File__doc__pdf__etc___2.pdf (100.3KB, pdf)
Supplemental Data File _doc_ pdf_ etc.__3

Supplemental Digital Content 3. Table presenting regression output details for Tables 2 and 3. .doc.

NIHMS883234-supplement-Supplemental_Data_File__doc__pdf__etc___3.docx (19.5KB, docx)

RESOURCES