Use of a Novel, Electronic Health Record Centered, Interprofessional Intensive Care Unit Rounding Simulation to Understand Latent Safety Issues

Abstract

Objective

The Electronic Health Record (EHR) is a primary source of information for all professional groups participating in ICU rounds. We previously demonstrated that, individually, all professional groups involved in rounds have significant blind spots in recognition of patient safety issues in the EHR. However, it is unclear how team dynamics impacts identification and verbalization of viewed data. Therefore, we created an ICU rounding simulation to assess how the interprofessional team recognized and reported data and its impact on decision making.

Design

Each member of the ICU team reviewed a simulated ICU chart in the EHR which contained embedded patient safety issues. The team conducted simulated rounds according to the ICU’s existing rounding script and was assessed for recognition of safety issues.

Setting

Academic Medical Center

Subjects

ICU residents, nurses and pharmacists

Intervention

none

Measurements and Main Results

28 teams recognized 68.6% of safety issues with only 50% teams having the primary diagnosis in their differential. Individually, interns, nurses and pharmacists recognized 30.4%, 15.6% and 19.6% of safety items respectively. However, there was a negative correlation between the intern’s performance and the nurse’s or the pharmacist’s performance within a given team. The wide variance in recognition of data resulted in wide variance in orders. Overall, there were 21.8 orders requested and 21.6 orders placed per case resulting in 3.6 order entry mistakes/case. Between the 2 cases there were 145 distinct orders place with 43% being unique to a specific team and only 2% placed by all teams.

Conclusions

While significant blind spots exist in the interprofessional team’s ability to recognize safety issues in the EHR, the inclusion of other professional groups does serve as a partial safety-net to improve recognition. EHR based, ICU rounding simulations can serve as a test-bed for innovations in ICU rounding structure and data collection.

INTRODUCTION

During daily ICU rounds team members gather to review all relevant data, discuss active problems and enact the daily care plan. Over the decades, rounds have shifted from a physician gathering around the bedside to an interprofessional discussion centered around the EHR. With the increase in participants, the addition of technology, the large amount of data generated, and the numerous real-time tasks being accomplished, rounds have exploded in complexity [1, 2]. Yet in spite of this expansion, little is known about how these components interact.

Numerous studies suggest improved outcomes with interprofessional rounds; documenting reduced adverse events, medication errors and mortality [3–5]. This likely reflects the unique lens through which each profession views a patient and the distinctive set of data and expertise they bring to rounds. Additionally interprofessional rounds benefit from distributed cognition, a phenomenon in which collaborative work leads to better performance by taking advantage of differences in individual cognitive properties [6]. The benefits of distributed cognition have been described in the interdisciplinary communication that occurs during cardiac bypass surgery and would seemingly apply to ICU rounds as well [7–9].

Despite widespread adoption, the EHR introduces a social-technical dynamic to rounds that remains poorly understood [10]. We previously published that physicians, nurses and pharmacists when tested individually, all have significant, non-overlapping blind spots in recognizing critical patient safety issues during EHR based simulation [11, 12]. However, little is known about how these differences in data collection interact when the team comes together to collectively care for a patient on rounds. We hypothesize that performance related to data collection, interpretation and verbalization would be improved in the group setting of interprofessional rounds compared to when tested in isolation. The team benefiting not only from the additional shared data, but also taking advantage of distributed cognition and therefore performing greater than the sum of its parts.

To test this hypothesis, we created an EHR-centered interprofessional ICU rounding simulation to study how team members integrated with one another, and the EHR, to perform the crucial tasks of rounds: data extraction, recognition of patient safety items, and execution of plans through order entry. Further, in order to fully capture both the complexity and other potential safety touchpoints in rounds, we incorporated the process of real-time order entry with mobile computers into the simulation. With this multifaceted simulation environment we aim to generate data that improves our understanding of the ubiquitous, but thus far poorly understood, EHR-centered, interprofessional ICU rounds and provide an environment for evaluation of new strategies aimed at improving the overall performance of ICU rounds.

METHODS

The study was conducted at a single academic medical center and the study approved by our institutional’s IRB. High fidelity simulation ICU charts were designed and imported into the simulation instance of our EHR as previously described [12, 13]. We created two cases encompassing the complexity that comes with admission in the MICU. Each case was created with the input of an ICU physician, an informaticist, a nurse, and a pharmacist to ensure they contained the appropriate fidelity for each professional group. Cases contained 5 days of relevant data and were embedded with patient safety events and best practice non-adherence as previously described [13].

We next replicated our MICU’s standardized interprofessional rounds. Our current rounding script begins with an intern (R1) presenting objective data, followed by the nursing report, pharmacist report, the intern discussing the daily plan and finally an order read-back by a resident assigned to order-writing. To reproduce this, we recruited an intern, ICU nurse and ICU pharmacist to collect and present data on rounds and an upper level resident (R2 or R3) as order-writer. All participants were recruited from the existing ICU team and were thus familiar with and had participated in the current rounding structure. Interns were from internal medicine, neurology or emergency medicine categorical programs. The only exclusion criterion was previous participation with the same case. All simulations were conducted in situ to optimize fidelity and participation. Participants were provided with a computer workstation to pre-round in the EHR. A brief, standardized, signout was provided, after which the intern, nurse and pharmacist were given 15, 10 and 5 minutes respectively to collect data in the EHR as previously described [12]. The order-writer did not pre-round on the patient, as their role was simply to input orders requested by the team into the EHR. This role mirrors our real rounds in which the order writer is often not directly involved in the care of the patient being discussed. Workstations were equipped with Tobii studio eye-tracking as previously described to allow observation of participants’ pre-rounding processes [14]. Once pre-rounding was complete, the intern, nurse, pharmacist and resident gathered to round. A member of the research team served as the attending to facilitate rounds but did not contribute in any other way to rounds. Two members of the research team, an ICU fellow and an ICU nurse, scored rounds for recognition of safety items (i.e. performance) as previously described [11–13]. Rounds were audio recorded and recordings were reviewed to resolve discrepancies in scoring and determine length of presentations. Orders placed in the EHR were reviewed and compared to what was requested to determine order integrity.

Differences between groups were analyzed using a two-tailed, Student’s t-test. Correlations were analyzed via Pearson’s correlation. All analyses were performed using Graph Pad Prism (GraphPad Software, La Jolla, CA). A p-value of <0.05 was considered statistically significant.

RESULTS

We conducted 28 simulations (Case 1=18, Case 2=10) involving 112 participants. 50 (44.6%) of the subjects had participated in previous EHR case simulation exercises on different cases and 16 (40%) of the participants in Case 2, participated in Case 1.

We first assessed the validity of the structure of simulated rounds by assessing time spent by the group and individual members of the simulated rounding team compared to actual ICU rounds. For simulated rounds, the mean rounding time was 21.1 ± 1.5min. The objective portion of rounds (data only) averaged 7.9 ± 0.6min, with the intern (3.4 ± 0.2min), nurse (2.2 ± 0.3min) and pharmacist (2.3 ± 0.3min) all presenting. The assessment and plan was the longest portion at 10.1 ± 0.9min (47.7%) and the order read back averaged 2.9 ± 0.5min (13.6%). For comparison, we used data from real rounds previously recorded by our group for a separate study [15]. The overall time for the objective portion was longer in simulated compared to real rounds (7.9 vs. 5.3min; p=0.0008). While, the duration of the intern’s presentation of objective data was similar (3.4 vs. 3.8min), the nurse (2.2 vs. 1.2min; p=0.008) and pharmacist (2.3 vs. 0.3min; p<0.0005) presented significantly longer in simulated rounds (Supplementary Figure 1).

We next evaluated performance in the simulation as determined by verbalization of patient safety items. On average, teams recognized 68.6% of the total possible safety issues (Figure 1A). Further, only 50% teams had the primary diagnosis in their differential. When we looked at individual safety items, there was wide variation in recognition. Certain items were frequently recognized and others nearly universally missed implying systematic blind spots for these issues (Figure 1B). When analyzing performance by professional group, interns recognized 30.4% of safety items, followed by pharmacists (19.6%), nurses (15.6%), and the order-writer (3.1%). There was a statistically significant negative correlation between the intern’s performance and the performance of the nurse and pharmacist within a given team, confirming the ability of other members of the interprofessional rounding team to serve as a safety net (Figure 2).

Figure 1. Recognition of Safety Items by Individual teams.

28 teams (18 Case #1, 10 Case #2) underwent rounding simulations. Panel A- Overall Performance of the teams for each case. Panel B-Description of the 12 safety issues per case and their cumulative percent recognition by all groups, broken down by professional that recognized the issue. RN (Nurse) Rx (Pharmacist). Abbreviations used: DVT-deep venous thrombosis, Bicarb-bicarbonate, CAM-Confusion Assessment Method, Steno- stenotrophomonas, Cipro-ciprofloxacin, TF-tube feed, INR -international normalized ratio, ADE- adverse drug reaction, ARDS- acute respiratory distress syndrome.

Figure 2. Correlation between Intern performance and other team members.

28 teams underwent rounding simulations. The correlation between the percent of safety items recognized by the intern was correlated for the nurses (Panel A) and pharmacists (Panel B) performance within the same simulation.

We next sought to determine factors associated with team performance.. The interns’ recognition of safety items negatively correlated with presentation duration. In contrast, there was a positive correlation between recognition of safety items and report duration for nurses and pharmacists (Figure 3). Teams where the intern accounted for more than 50% of data presentation time had a significantly lower recognition rate of safety items (58.0 vs. 72.5%; p=0.004). Finally, if any member of the team had previously participated in an EHR based simulation, overall team performance was improved (69.9 vs. 61.4%; p=0.032) confirming our prior observations that participation in EHR based simulations improves performance on subsequent simulation testing [11].

Figure 3. Correlation between report time and performance for individual team members.

28 teams underwent rounding simulations and individual team members assessed for the percent of safety items recognized during their report of objective data. The overall recognition rate was correlated to absolute time of their report of objective data for Interns (Panel A), Nurses (Panel B), and Pharmacists (Panel C).

Finally, we analyzed order writing. On average, a rounding team requested 21.8 ± 1.2 orders, resulting in 21.6 ± 1.3 EHR orders submissions per simulation. The number of orders entered positively correlated with team performance (Figure 4A). We assessed how many teams entered a particular order and how many unique orders were entered, grouping orders when possible (e.g. all permutations of pipercillin/tazobactam orders were grouped). For both cases, there was large inter-team variation and low degree of uniformity in entered orders. There were 88 distinct orders entered for Case #1 and 57 for Case #2. Of these, 43% of orders were unique to the team entering them and only 3 orders (2%) were placed by all eligible teams (Figure 4B, Supplementary Figure 2). There were 15 orders entered by 80% of teams or greater, 13 of which addressed major safety items, 9 were orders incorporated in the FASTHUG mnemonic, and 4 were antibiotic-related orders (Supplementary Table 1). In regards to order integrity, teams averaged 1.9 requested but not entered orders and 1.8 entered but not requested orders for a total of 3.6 ordering inaccuracies/team. Analysis of screen navigation videos from the order-entry computer showed that the order-writer spent a significant amount of time navigating the EHR on screens not designed for order entry, similar to previously reported resident navigation pattern (Figure 5) [12]. Overall, the order-writer navigated to more total screens during the presentation than participants preparing for rounds (Supplementary Figure 3). The total number of screens visited by the order-writer was strongly and inversely related to overall team performance (R=−0.58; p=0.001).

Figure 4. Analysis of requested orders.

28 teams (18 Case #1, 10 Case #2) underwent rounding simulations and orders placed into the EHR analyzed. Panel A- correlation between the number of placed orders and overall team performance as assessed by recognition of safety items. Panel B- Distribution of unique orders for each case. Each column represents a unique orders entered into the EHR in Case 1 (N=90-Top Panel) and Case 2 (N=57-Bottom Panel) with the percent of teams that entered that particular order is delineated on the y-axis.

Figure 5. EHR Screen navigation patterns of dedicated order writer during simulated rounds.

28 teams underwent rounding simulation and the screen capture video of the dedicated order entry computer analyzed for navigation pattern. Each screen visited was categorized into 9 “macro” categories. Each row represents an individual participant and each color represents one of the 9 screen macro-categories. The width of a bar represents the time spent on a screen category.

DISCUSSION

We successfully created an ICU rounds simulation, from pre-rounding to order entry, which allowed us to study the complex interchange between health care professionals and the EHR. We previously used EHR simulation to study the gathering and presentation of information by individuals [12]. This study expands our methodology to include multiple professions communicating and processing information together, replicating the workflow of real-life interprofessional ICU rounds. While the use of simulation for studying clinical workflow and systems has been described its use to understand interprofessional rounds is novel [16]. This approach has the advantage of creating a controlled environment in order to generate data and test hypotheses that would be otherwise methodologically difficult. For example, while we demonstrated high rates of deficient communication of lab data in real rounds the intention and importance of these errors is unclear [15]. With simulation, we know which data points are crucial to the case and therefore can easily differentiate between intentional omissions, benign oversight and critical misses.

The duration of time spent on data presentation by each member of the group speaks to the degree of fidelity of the simulation. Our findings on the quantity and quality of the data presented in rounds support the conclusion that each member of the team was able to proceed from EHR data extraction to rounds presentation in a substantive way. This data was then processed as a group and led to clinical decision-making in the form of electronic orders. The variation in safety issues identified and placed orders is representative of an ample level of complexity achieved and serves as face validity of our simulation. We compared time of presentation in simulated versus real rounds as additional form of validation and observed that simulated rounds took significantly longer primarily due to increased nursing and pharmacist contributions. While these differences are likely attributable to experimental factors such as the absence of time pressure felt on real rounds, or differences in complexity of comparator cases, it may also shine light on barriers to optimal interprofessional participation in actual rounds by minimizing dysfunctional, hierarchical power dynamics or feelings of being “undervalued” in the simulation [2, 17]. Determining the factors of this empowered performance so that it can be cultivated in real rounds will be the subject of future investigations.

To assess group performance we evaluated individual and group recognition of the safety items. Evaluating this type of performance is another distinct advantage of the simulation as it is difficult to track these types of errors in actual rounds, as they are often not apparent in real time. In this study, teams averaged nearly 70% recognition of the total possible safety issues. In our previous studies, using similar methodology, individual providers performing independently recognized an average of 32% of safety issues [11, 12]. Based on these results we predicted that an interprofessional team working together would miss approximately 50% of safety issues [12]. The outperformance by the interprofessional group in this study supports the hypothesis that interprofessional rounds improve safety recognition compared to individuals from multiple professions working alone [3–5]. This synergistic effect is supported by the inverse correlation between safety items recognized by the intern and those recognized by the nurse and pharmacist. This indicates that when an antecedent rounder has not been thorough and/or struggled those that follow can fill in the gaps. Analysis of individual safety items demonstrates predictable ways in which each profession protects the blindspots of their rounding colleagues. For example, interns commonly failed to address sedation or feeding status, but nursing additions improved recognition of these items from 6% to 39%, and 33% to 89% respectively. Pharmacists were excellent at catching previously unmentioned medication errors such as unnecessary famotidine or inappropriate DVT prophylaxis, improving these items from 33% to 95% and 10% to 100% respectively. However there were data elements such as low respiratory rate, high calcium level and presence of a port in a bacteremic patient, that were recognized by less than a third of teams, highlighting the system’s vulnerability to ineffective extraction of certain data from the EHR.

We observed several factors associated with performance. Consistent with prior studies, teams which contained members who had participated in prior EHR simulations demonstrated improved data recognition [11]. Further, the number of safety items recognized by nurses and pharmacists correlated directly with the amount of time they spent reporting data. Conversely, the interns’ report time correlated negatively with the performance of the Intern and the team as a whole. There are multiple potential explanations for this. First, is that the longer the presentation, the greater degree of mental fatigue by other team members, consistent with our prior observation that laboratory communication errors in ICU rounds increase once presentations are >20min in length [15]. Another possibility is that longer intern presentations are a marker for confusion and inability to process complex information. This is consistent with a report of longer presentation time correlating inversely with accomplishment of daily goals [18]. In either case, this finding has interesting implications when considered with our previously mentioned finding that nurses and pharmacists speak less on real rounds than in the simulation. The impact of shifting some of the data presentation burden from the intern, commonly the most junior member of the team, to nurses and pharmacists requires further study.

The consequence of the missed safety items was high variability in the differential diagnosis by the team with the primary diagnosis included in the differential in less than half the simulations. This manifested as a wide variance in the orders requested by each team with a large fraction of orders unique to an individual team. This variability in requested orders was associated with a number of “inappropriate” orders for the patient given the actual diagnosis, including inappropriate consults and test ordering. This is consistent with other studies suggesting that selective data gathering and processing are associated with an increase in medical errors and patient morbidity [19]. The increase in morbidity coming not only from delayed or missed diagnosis, the top cause of preventable error, but from unintended consequences associated with inaccurate order entry and unnecessary testing.

Analysis of orders allowed us to identify another potential latent safety risk during ICU rounds. Order entry errors are a known problem despite, and in some reports, as a result of the adoption of computerized physician order entry (CPOE) systems [20, 21]. Our results suggest increased scrutiny on this task is needed. A mean of 3.6 ordering inaccuracies per case were observed, half of which were entering orders not requested on rounds. The role of order writer is a relatively new one, intended to speed up delivery of care and avoid missed orders. However this role introduces a previously unrecognized safety threat into rounds [22]. In the simulation, the order writer did not pre-round and therefore was not familiar with the case, which is typical in real rounds, and consistent with prior reports suggesting a significant disconnect between the role of order writer and clinical decision making [22]. As an unintended consequence of this workflow, they spend a large amount of time browsing the EHR, rather than exclusively listening for and inputting orders, evidenced by their independent recognition of safety items. Further, the degree of screen navigation was inversely related to recognition of safety items, suggesting this behavior is magnified when the order-writer attempted to serve as part of the “safety-net” of rounds. This multitasking in the order-writer’s workflow likely contributes to the number of missed and non-verbalized orders, in spite of a dedicated order readback and highlights the utility of simulation to both understand the function of the order-writer and provide an opportunity for dedicated training in the function in future studies.

Limitations of our study include this was a single center, single ICU study. The simulation diverged from real rounds in a number of ways. First, there was no actual patient, which could have impacted data gathering patterns. We attempted to account for this by limiting safety issues to items readily notable in the EHR and did not require physical examination or inspection of equipment or monitors. Second, the intern rounded without the help of a senior resident. This design was partially due to resource limitations but also in order to understand how the EHR and rounds perform in this vulnerable and not uncommon scenario (e.g. the senior resident’s day off). Third, and perhaps most important, the attending was a confederate meant merely to ensure rounds followed the set script and to ensure adequate fidelity. The reason we chose not to include an actual attending was multifold. For one, the primary goal of the study was to evaluate information gathering and processing by team members assigned to the pre-rounding and presenting roles (which the attending is not), with the premise that effective team decision making is critically dependent upon avoidance of errors in this process [19]. Further, recognition of these items in the EHR should be independent of a physician who is not involved in the data gathering process. ICU rounds, if designed to incorporate each member’s skill set and integrate a high functioning EHR, should overcome the absence of the attending. Finally, the presence of an attending potentially impacts both real-time data gathering and recall bias (e.g. questioning from the attending alters individual perception of what is appropriate selective data reporting/interpretation) and these factors are dependent on the attending’s knowledge base and heuristics, a variable which would be impossible to control for. We recognize there are complicated medical decisions that will always require an attending physician; however, the safety items in these simulations did not fall into that category and teams were credited with recognition simply by mentioning the items (e.g. increased calcium). There was no requirement to address them in the plan, nor were they graded on the quality of this plan. Another limitation is that other professional groups such as Respiratory Therapist, Physician Assistants and Nurse Practioners were not included. This is an important consideration as they likely have significantly different data acquisition patterns [23]. Future studies will aim to determine if their addition adds to team performance. Finally, simulations were done in the afternoon with existing ICU staff. While this ensured that all members of the team were familiar with the current rounding structure, it is possible that ICU acuity contributed to cognitive fatigue and thus underperformance of the teams.

In conclusion, we demonstrated the feasibility of creating a comprehensive ICU rounding simulation which effectively reproduces interprofessional contribution and for analysis of the sociotechnical factors associated with integration of the EHR into the process. By using high complexity, purposively designed simulated EHR charts, we can now use these simulations as a test-bed to assess modification to the rounding structure to improve efficiency, efficacy and safety of ICU rounds.