Novel Representation of Clinical Information in the ICU

Abstract

The introduction of electronic medical records (EMR) and computerized physician order entry (CPOE) into the intensive care unit (ICU) is transforming the way health care providers currently work. The challenge facing developers of EMR’s is to create products which add value to systems of health care delivery. As EMR’s become more prevalent, the potential impact they have on the quality and safety, both negative and positive, will be amplified. In this paper we outline the key barriers to effective use of EMR and describe the methodology, using a worked example of the output. AWARE (Ambient Warning and Response Evaluation), is a physician led, electronic-environment enhancement program in an academic, tertiary care institution’s ICU. The development process is focused on reducing information overload, improving efficiency and eliminating medical error in the ICU.

1. Background

1.1 Systems of health care delivery are imperfect

An estimated 6 million adults are admitted to ICU (intensive care unit) each year and one in five Americans who die, do so using ICU services [1]. The total cost of providing intensive care services in the United States has been estimated at 0.8-1% of gross domestic product, that is, about $US 100 billion, representing 20% of the total budget for hospitals. Patients in the ICU are exposed to multiple potential sources of iatrogenic injury and medical error. An Israeli study identified an average of 178 processes of care delivered to each ICU patient per day of stay and that 1.7 of those were associated with some error [2]. Overall this study identified more than 500 errors including over 200 serious errors in a single 16 bedded ICU over a 4 month period. Another study concluded that human errors were responsible for approximately 15% of all bed utilization days in the form of prolonged ICU stay [3]. These findings suggest that human error in the ICU is common, contributes to patient harm, excess resource utilization and is often preventable. A systems engineering approach has been presented as an ideal starting point in the development of EMRs which address these safety failures [4].

1.2 The problem of information overload

The identification of high value data presents a significant challenge to any health care provider. A study from a Canadian group estimated that the care of critically ill patients generates a median of 1348 individual data points/day [5] and that this quantity has increased 26% over five years. With the advent of a high fidelity digital representation of a patient, there is a potential to organize that data intelligently. Currently patient data elements are scattered across many different platforms and applications. This makes pattern recognition difficult for the physician, and in the context of a fast changing, multi-patient critical care environment, can lead to delays in diagnosis and care delivery. This can have devastating consequences for critically ill patients [6]. Developers intending to create products which add value to a health care delivery system will have to overcome this challenge and devise a method which reliably extracts important data from the electronic soup and presents it clearly to the bedside provider. The definition of those data points which are considered valuable in ICU medical decision making is an essential first step in the development of smart EMRs. Without adequate knowledge of these cues, and the context in which they are utilized, EMRs will be unable to organize the electronic environment and participate in complex care delivery processes.

One of the challenges facing developers is to design new products which enhance rather than disrupts decision making in a working environment. To develop EMRs in isolation, away from the final implementation environment, will decrease the potential for success and potentially result in the opposite of the intended effect [7]. Testing and development should therefore take place in environments which are highly representative of the destination environment, but which protect patients from potential harm. The use of realistic test environments and clinical scenarios will facilitate the conduct of human factors engineering studies, field observation studies and randomized control trials which are necessary to develop and demonstrate the efficacy of new technologies [8]. Such facilities and methodologies are currently lacking and may be one of the reasons there are low rates of uptake of comprehensive EMRs and conflicting reports on their efficacy once deployed [9].

In this paper we describe an electronic environment development research program running in a tertiary care institution. With a worked example, AWARE (Ambient Warning and Response Evaluation), we illustrate how a multidisciplinary team might overcome the challenges of restructuring the electronic environment to deliver a user interface which is effective at reducing provider cognitive load, resource utilization and the occurrence of medical error for defined clinically relevant tasks. The measurement tools and testing environment, which models clinical demands, utilized during development are described.

The objective of the AWARE program was to develop an application which would extract a subset of predefined cues from the existing electronic environment and organize them for presentation to the bedside physician with the aim of reducing cognitive load and medical error associated with specific tasks. The primary outcome measures were NASA Task Load index (NASA-TLX), time to task completion and medical error.

2. Study Context

2.1 Institutional Infrastructure

2.1.1 Electronic infrastructure

Beginning March 21, 2005, the medical records of all new patients coming to Mayo Clinic in Rochester, Minnesota, are in electronic form.

The current standard clinical electronic environment has at its core the Mayo Integrated Clinical System (MICS) LastWord. This interface provides access to demographic and admission information, laboratory data, medications record, nursing flow sheets, clinical notes and procedures reports. MICS incorporates the inpatient Computerized Physician Order Entry (CPOE), outpatient medications prescribing and nursing plan of care.

An additional electronic resource is the Chart+ application which is used in all operating and recovery rooms; all ICUs and many procedural areas providing sedation. Chart+ interfaces with physiologic monitor data and allows the users to manually enter other specific data, including fluids balance. Chart+ data may be viewed in MICS via the Anesthesia and Monitored Care Reviewer (Remote View) available at call computer terminals with clinical information systems access. Radiologic, endoscopic and clinical images and reports are viewed in QREADS, a Mayo developed program.

A number of other applications support clinical document flow including, Digital Dictation, Documents Browser, Image Capture Environment and Scheduling.

Most recently Mayo Clinic developed the Synthesis application in response to physician requests for improved navigation of EMR. Synthesis, offers users a single, integrated view of information from many sources. Information from Last Word, Chart+ Remote View and QREADS are all available from within Synthesis (►Fig. 1).

Fig. 1.

Standard EMR. Synthesis offers a summary view of a single patient’s electronic medical data. This interface offers access to laboratory, radiological, physiological, documents and demographic patient information.

AWARE is a .NET based application which extracts data relevant to the treatment of critically ill patients (as determined through the systematic profiling of provider data utilization patterns) and presents it to the provider in a systems based packages (►Fig. 2). As a single patient viewer, the primary difference between AWARE and Synthesis is that the AWARE content has been selected through the systematic observation and analysis of frontline provider information needs. The user interface has been optimized to the task of treating critically ill patients.

Fig. 2.

The display of data within a systems based information package (in this case the cardiovascular system) is illustrated. The organization of data elements was determined by considering how experts incorporate information into decision making mental models. Reading from left to right the key organizational elements are; clinical context pulled from the patient problem list, procedure, medications and consults lists; system identifying icon with color coded status (red – urgent intervention, orange – abnormal physiology or investigation, green – normal physiology and investigations); physiological status displays current values for key physiological variables – color coding follows organ identifier; Organ Supports, displays the critical care interventions which are supporting the current physiological status; investigations displays the status and, when available, the result of high value system based investigations; provider actions are tracked in this status panel – the content of this panel varies with context e.g. Sepsis v Bleed.

In addition to clinical systems a custom integrative relational research database Multidisciplinary Epidemiology and Translational Research in Intensive Care (METRIC) DataMart contains a near-real time copy of the EMR of ICU patients. METRIC DataMart serves as the main data repository for rules development and validation [10, 11]. Access to the database is accomplished through open database connectivity (ODBC). The METRIC Datamart is already a unique and highly valuable research resource, enabling and stimulating clinical investigations related to critical care [12]. The Mayo Clinic EMR complies with the American National Clinical Document Architecture, which is the widely accepted standard for clinical documentation [13].

2.1.2 Clinical Setting

With almost 15000 new pediatric and adult ICU admissions per year the Mayo Clinic environment is well suited to the development and validation of applications for use in the critical care setting [14]. With an average census of 65 adult critically ill patients in 7 adult ICUs, and a median 36 new ICU admissions during any day, the investigators have ready access to real-time data feeds from a wide variety of critically ill patients. Adult ICUs in both Rochester, MN, hospitals include; medical, cardiac, neurologic, thoracic and vascular surgery, trauma general surgery, cardiac surgery, and a mixed medical surgical ICU (mixed ICU admits critically ill patients from hematology, obstetrics, orthopedics, liver and kidney transplantation). Currently the ICU clinical workflow requires the provider to use multiple EMR interfaces to access relevant diagnostic and treatment data.

2.2 AWARE development program – reducing information overload

►Table 1 provides an overview of the AWARE development process. A similar structured approach has been proposed by others [15, 16]. The initial target for intervention, reducing information overload, was identified in collaboration with a group of ICU experts from within the institution.

Table 1.

AWARE development procedures: Each phase is iterative and cycles until no further improvement accrues. Once in the testing phase, AWARE does not progress to the next stage unless it reduces cognitive load, improves workflow or reduces medical error.

Phase	Description
I. Initial
Focus group	A small multidisciplinary group of experts agreed on the principle areas in which AWARE may have a role (reduction of information overload, oversight, daily rounding, checklist, discharge planning)
Expert opinion	From the focus group discussion, specific tasks were identified and prioritized (1. reduce information overload whilst preserving utility in decision making, 2. Facilitate oversight of distributed cognitive network performance of specific tasks e.g. checklist compliance, sepsis resuscitation bundle, low tidal volume strategy in ARDS)
II. Data Gathering
Prospective observational studies	Tasks were deconstructed and data gathered using a variety of field observation tools (interview, questionnaire, workload measurement). The principle objective of this phase was to identify the sequence of events and key contextual and/or decision making cues which providers utilize to complete the specified task.
Rule development	Observed cues and sequences were mapped to their surrogate markers in the electronic environment. Rules which facilitate successful task completion were written by clinicians and in consultation with the IT experts and programmers these are coded in .NET as plug-ins for the existing electronic environment.
III. Testing
Control environment testing	In order to protect patients and providers from unanticipated adverse effects, a testing environment was developed to assess the reliability of rules and their impact on outcome measures such as medical error, cognitive load and integration within the distributed cognitive network.
Field Testing	Once the application has demonstrated to facilitate safe task completion in the controlled setting, it is released into a clinical area designed to facilitate rapid cycle evaluation of changes to health care delivery mechanisms.
IV. Implementation
Implementation	Once field testing demonstrate the benefit of the application it is released as a supported application in the general work area and an extended observation and evaluation process begins

2.2.1 Multidisciplinary Team

The development of AWARE is beyond the expertise of a single group and requires a multidisciplinary approach. The key team members include individuals with domain expertise in the following areas; clinical ICU; human factors; informatics; science of health care delivery; statistical; information technology and programming.

2.3 Data gathering - Identifying relevant data cues

Our preliminary work in this area introduces a rational basis for the identification of decision making cues within the ICU environment [17]. DUM profiles, (Data Utility in Medical decision making), describe the value of data in terms of it’s frequency of utilization in ICU decision making processes. We prospectively examined over 600 physician-patient interactions which took place at the time the ICU admission. Admitting teams of physicians were asked to identify cues which were useful in their decision making. DUM profiles of data vary significantly with clinical context but are conserved between individual ICUs (►Fig. 3). DUM profiles provided an objective backdrop for the rationalization of the quantity of data needed to make decisions in the critical care setting. Irrelevant data items were identified by an expert group and priority given to relevant data.

Fig. 3.

DUM profiles represent the frequency of data utilization in each admission diagnostic categories. Providers were approached after admitting a patient to the ICU and requested to complete a questionnaire outlining the information they used in the assessment and treatment of the newly admitted patient. The frequency with which each reported data point was used is demonstrated in the figure below. A subset of admission contexts (any admission, sepsis or bleed) are included for comparison purposes. Two distinct classes of provider-determined, valuable data are identified using this approach; one is conserved across admission contexts (HR, RR, MAP) and; one which is relevant only for a particular admission context (Hb in Bleed). The least valuable data elements are those which are infrequently utilized regardless of context (Base deficit). Data which fall into this category are candidates for non-inclusion in the AWARE user interface. (Abbrev.: WCC, white cell count; SpO₂, pulse oximetry oxygen saturation; RR, respiratory rate; PCO₂, arterial CO₂; PaO₂, arterial O₂; MAP, mean arterial blood pressure; Hb, hemoglobin; GCS, Glasgow coma scale; FiO₂, fraction of inspired oxygen; DNR/DNI, do not resuscitate/intubate; CXR, chest X-ray).

2.4 AWARE rule development – Programming pilot application.

Once relevant data were defined, the electronic marker of that data cue was identified and rules written to extract those cues from the EMR, if available. The clinical experts worked closely with the programmers during this phase of development. Examples of rules development are given in Table 2. Data elements were organized in organ based “information packages” (►Fig. 2). The arrangement of that package was determined by an expert focus group and was guided by a need to integrate information into the distributed cognitive network in a manner which would reduce cognitive load and improve the usability of the electronic environment. The AWARE interface was programmed using a .NET framework.

Table 2.

Once identified as a high value data cue, the electronic source for each cue is identified and the rules governing when and how that data should be displayed is determined through a multidisciplinary collaborative approach.

CVS	Display		Description	Display	Value	Source
1	Always	Context	Chronic Health	Text	Text	Documents
2	Always	Physiology	Heart Rate	HR	beats per minute	Vitals
3	Always	Physiology	Mean Arterial Blood Pressure	MAP	mmHg	Vitals
4	Alternate to 2	Physiology	Mean Non-Invasive BP	MBP	mmHg	Vitals
5	Always	Physiology	Rhythm	Rhythm	Text	LW
6	Always	Physiology	ST segment value	ST changes	+/_mm	Vitals
7	Always If Available	Physiology	Central Venous Pressure	CVP	cmH2O	Vitals
8	Always If Available	Physiology	Mixed Venous Saturation	SvO2	%	Vitals
9	Rule ‘Show CVS Support’	Support	Fluid Type	Fluids In [2hrs]	ml	Vitals
10	Rule ‘Show CVS Support’	Support	Vasoactive Drug	Drug Name	Text + mcg/kg/min	Vitals
11	Rule ‘Show CVS Invx/Tx’	Invx/Tx	ECG	ECG	Status + Value	Documents
12	Rule ‘Show CVS Invx/Tx’	Invx/Tx	ECHO	ECHO	Status + Value	Documents
13	Rule ‘Show CVS Invx/Tx’	Invx/Tx	Troponin	Troponin T0	Status + Value	Labs
14	Rule ‘Show CVS Invx/Tx’	Invx/Tx	Troponin	Troponin T3	Status + Value	Labs
15	Rule ‘Show CVS Invx/Tx’	Invx/Tx	Troponin	Troponin T6	Status + Value	Labs
16	Rule ‘Show CVS Invx/Tx’	Invx/Tx	Hemoglobin	Hb	Status + Value	Labs
17	Rule ‘Show CVS Invx/Tx’	Invx/Tx	Lactate	Lactate	Status + Value	Labs
18	Rule ‘Show CVS Invx/Tx’	Invx/Tx	Hematocrit	Hct	Status + Value	Labs
19	Rule ‘Show CVS Invx/Tx’	Invx/Tx	Type and Screen	Active Type	Date of Active Type*	Labs
20	Rule ‘Show CVS Invx/Tx’	Invx/Tx	IV access	IV Access (Type)	Location + Gauge	Assessments
21	Rule ‘Show CVS home meds	Context	Home Medications	Home Meds	Drug Name	Documents
22	Rule ‘Show CVS Active Meds	Context	Active Medications	Active Meds	Drug Name	Medication Profile
23	Always if Available	Context	Catheterization Procedures	Name	Procedure Name	Documents
24	Rule ‘Show CVS Invx/Tx’	Invx/Tx	Blood Loss	Blood Loss [4 hrs]	Value mls	Vitals

2.5 AWARE testing – Measuring safety and demonstrating effective use

The challenge for any new application is to demonstrate safety and meaningful use of the electronic environment. Our working definition of meaningful use, in the context of application development, includes; the measurable reduction in cognitive load; reduction or elimination of medical error; and facilitate efficient completion of defined tasks. As such we measure cognitive load (using NASA TLX), medical error (as defined by experts and guidelines) and efficiency (time and accuracy). Any new application developed in our lab must demonstrate safety and “meaningful use” prior to progressing to the live environment.

By simulating the ICU distributed cognitive network within a closed test facility, we can introduce changes to the electronic environment and examine the impact they have on cognitive load, medical error and procedures without exposing patients or providers to harm. Tasks are defined and deconstructed into sequences of expected actions. In a cross over study, the performance of providers, randomly assigned to complete the structured task with and without AWARE, are observed in cycles of 4 consecutive task completions.

3 Study Design

Following IRB approval, a prospective, randomized, cross over pilot study comparing the impact of AWARE versus standard EMR on cognitive load, time to task completion and number of medical errors was performed

3.1 Setting

Remote testing facility with real-time data feeds from patients currently being treated in the institution’s intensive care units. The testing facility consisted of a single 17 inch monitor with keyboard and mouse for navigation per study subject, with 2-4 study subjects per session. Two electronic environments were compared – AWARE (running the developed user interface) versus Standard (running the institutional EMR).

Live data was simultaneously streamed into each of these stations from a subset of randomly chosen patients currently admitted to one of the institutional ICUs.

3.2 Study Subjects

Off-duty critical care fellows and residents were evaluated.

3.3 Study Procedures

Study subjects were randomized to the AWARE or standard electronic environments (see ►Fig. 4). The providers were then asked to perform a standardized task, “You have been called to evaluate this bleeding patient” on each of 4 consecutive patients. Specifically, they were assigned the task of a team member required to extract a sequence of decision making cues which would impact on the ICU team’s treatment plan (adequacy of IV access; availability of active blood group type and screen; identification of medications which contribute to bleeding; treatment of coagulopathy and thrombocytopenia, blood products given in past 24 hours; intubation history; documented intubation wishes). The providers were timed by a study coordinator performing the task on each patient. As the provider worked through the task, they completed a questionnaire on each patient. At the end of each 4 patient session study subjects were required to complete a NASA-TLX evaluation questionnaire. At the end of the first 4 patient sessions, providers crossed over to the alternative environment and completed the same tasks on a different set of patients.

Fig. 4.

Test Facility and study design: The test facility receives live data feeds from the clinical environment. Providers are randomized to either the Standard EMR or the AWARE electronic environment. Half way through the testing session providers are crossed over to the alternative environment allowing matched pair analysis of outcomes. At all times the patient is isolated from the actions of providers working in the test environment. Once safety and efficacy is demonstrated in the test facility, the developed application is ready for limited field testing in the real ICU environment. (Abbrev.: EMR – Electronic Medical Record, AWARE- ambient warning and response evaluation viewer).

3.4 Measurements

The primary outcome measures were; time to completion of task measured in seconds by the study coordinator by direct observation of provider: the number of medical errors, calculated by comparison of answers provided by study subject on the standardized questionnaire to those extracted by a trained observer from the EMR at the time of task completion; and cognitive load measured using the NASA-TLX instrument administered at the end of each 4 patient/user interface session.

3.5 Statistics

All statistical analysis was performed in JMP v 8.0. The distribution of data was tested for normality using the Shapiro-Wilk test. All descriptive data are presented as median, interquartile range. Matched-pair analysis was used to analyze variance in values collected between AWARE and the standard EMR interface. A p<0.05 was considered significant. The relationship between medical error and NASA TLX was determined using a logistic regression model.

4. Results

Six providers performed the task on 8 patients each (24 with standard EMR and 24 with AWARE).

4.1 Time to task completion

The median (IQR) time(s) to completion of task for task number 1 through 4 were 99 (73-114), 65 (59-73), 58 (55-69) and 53 (45-78) for AWARE and 196 (167-258), 173 (156-187), 132 (119-141) and 137 (122-152) for the standard EMR, n = 6 for all values (►Fig. 5).

Fig. 5.

Time to task completion. The time taken for the provider to compete the standardized task (median, interquartile range) on each of 4 consecutive patients, using either AWARE or the standard EMR. The time taken to complete the task is less as the provider progresses through patient 1 to patient 4. The task is completed more quickly when the provider uses the AWARE electronic environment compared to the standard EMR, p<0.05 matched pair analysis. Significant variability in time to task completion exists across tasks 1 to 4 with the standard EMR (Chi square 13.6, DF 3, p = 0.003) but not with AWARE (Chi square 6.2, DF 3, p = 0.10).

4.2 NASA-TLX – task load

Cognitive load was measured after the fourth task with each interface using NASA-TLX (►Fig. 6). The median, (IQR, n), was measured at 26 (22-33, n = 6) with AWARE and 47 (34-59,n = 6) with standard EMR, p<0.05 by matched pair analysis.

Fig. 6.

Cognitive Load as measured using NASA-TLX. The median, interquartile range is illustrated for each electronic environment. Matched pair analysis is significant at p<0.05 with AWARE imposing a lower cognitive load on the provider compared to the standard EMR.

4.3 Medical Errors

The median (IQR) number of errors for all providers were, 2.5 (1.5-3) and 4 (2-6) for AWARE and EMR respectively, n = 6, p<0.05 by matched pair analysis. The total number of medical errors recorded in 24 patient assessments were 24 with standard EMR and 13 with AWARE.

4.4 Association between NASA-TLX and medical error

The sum of medical errors per provider-interface was calculated and plotted against the corresponding cognitive load as measured using NASA-TLX (►Fig. 7). The linear fit expression Errors = 0.59 + 0.07*cognitive load describes their relationship (R square = 0.42, DF = 11, p<0.05). This suggests that for every 10 point increase in cognitive load there is a corresponding increase 0.7 increase in the number of errors.

Fig. 7.

Linear fit model of cognitive load and medical error. As the cognitive load on the provider increases, a corresponding increase in the occurrence of error is also observed. The relationship is described more fully in the results section.

5. Discussion

In this study we demonstrate that the electronic environment imposes a significant cognitive load on providers and that this has a significant impact on the ability of providers to safely perform a routine ICU task in a laboratory setting. The availability of an interface which organized the most relevant data into systems based information packages, AWARE, significantly improved the performance and efficiency of ICU patient work up. The new interface reduced, time to task completion, provider cognitive load and medical error in the assessment of ICU patients with a hypothesized acute bleed. In addition, an association between cognitive load, (as measured with NASA-TLX), and, medical error exists. These findings suggest that the configuration of the electronic environment has a significant impact on the cognitive load of providers and that high cognitive loads contribute to medical error.

Provider randomization in a cross over trial is a strong study design which effectively isolates the electronic environment as a variable. The numbers included in the study are small, only a single task is tested and the existing EMR is specific to the host institution. This means that it is difficult to generalize these findings beyond the test setting. Another weakness of the study as presented is that the testing environment is only representative of the electronic environment providers work within. The physical cues gathered from the patient, nursing staff, family members as well as the distracters including alarms, pagers, new patient admissions and crises management are entirely absent from the laboratory set up as described in this paper. That these findings might extrapolate outside of the laboratory is supported by case studies of EMR and CPOE deployments which resulted in patient harm [18]. Given the influence medical error in the ICU exerts on patient outcome [2, 3], it is possible that the development of electronic environments focused on reducing cognitive load could deliver system safety improvements which translate into reduced resource utilization and improved patient-centered outcomes. The immediate next step for the group is to conduct a larger trial, incorporating some of those elements, as well as alternative tasks (admission, resuscitation, rounding and discharge) in a high fidelity simulation center. As a proof of concept the findings are consistent with findings from other industries: parsimonious information displays, designed around specific tasks, reduce cognitive load and improve human performance.

A project of the type described in this paper requires a high fidelity EMR, infrastructure to support research and development, access to clinical materials and interested clinicians working in cooperation with non-clinicians to develop and validate rule sets. The success or failure of EMR applications are decided at the point of user interface and workflow integration [19]. The involvement of end users at each phase of the development process greatly increases the chance of a successful application. We are fortunate at our institution to have a group of clinicians, many of whom have extensive experience of using EMRs, committed to improving health care delivery through refinement of the electronic environment. The rules which drive innovations such as AWARE could not be derived without that commitment.

As indicated in the introduction, the release of advanced EMR applications into the live environment may be associated with disruption of established cognitive processes [20]. It is the strong held opinion of the development group that no applications should be released into the real environment without undergoing an “effective use” evaluation. The outcome measures described in this paper might usefully be adapted to compare the ability of applications to deliver “effective use” from the providers’ perspective i.e. reduced task time, reduced cognitive load and reduced medical error. The measurement of cognitive load has long been a standard in other high-tech, high-reliability industries (airline, military and nuclear). A validated tool for this purpose is the NASA task load index (NASA-TLX). Increasingly, this tool has found application in health care [21]. The use of this tool to compare electronic environments in health care is still uncommon. The demonstration that NASA TLX scores correlate with the number of medical errors is one indicator that this tool may be useful for this type of comparison.

An additional assessment which should be performed is that of impact on medical error. The advantage of incorporating this measure into any evaluation process is that it highlights the potential impact an application might have on error rates once released. Clearly the goal is to eliminate error, but the comparative effectiveness of one application over another to reduce error could become a powerful stimulus for the development of electronic environments impact positively on patient safety. The primary barrier to including this as a measure is the need to clearly define error for every application associated task (e.g. checklist, admission, hand-over). Defining error in an objective manner requires consensus from experts. This is a time consuming and imperfect. At the present time a surrogate exists in the form of consensus conference or society guidelines, however, translating these into meaningful measures can be challenging [22]. The compromise our group has arrived at is to combine such guidelines with local expert opinion and develop task specific expected actions. In this case deviation from those actions is counted as an error.

Decision making cues are often scattered throughout the electronic ICU environment, hindering the ability of bedside providers to recognize critical patterns. The impact this has on providers is poorly understood but is likely to be particularly relevant in the ICU. Recently our group identified that failures of information recall at the time of ICU handover were common and are associated with errors of medical judgments [23]. This can lead to errors of cognition and failures to intervene appropriately. By more directly connecting providers to patterns of data, we have demonstrated in our pilot study that significant reductions in errors of cognition result.

6. Conclusions

AWARE technology can dynamically display highly valued data to the physician at the bedside in a timely manner. As tested, AWARE, significantly improved the performance and efficiency of ICU patient work up. The new interface reduced, time to task completion and medical error in the assessment of ICU patients with a hypothesized acute bleed. Providers’ cognitive load by NASA-TLX reduced from 47 (34-59) with standard EMR to 26 (22-33) with AWARE. The development process and outcome measures (cognitive load, medical error and efficiency) outlined in this paper are adaptable to any number of applications and, in the opinion of the authors, illustrates one route towards effective use of the EMR. An expanded program of testing, with alternative tasks and settings, culminating in clinical randomized control trials are needed to determine the generalizability of these findings.

Authors’ contribution

All authors have committed both time and energy to identify the goals of the project, identify appropriate background information, and to accumulate the necessary the scientific data and statistical support. In addition, the specific role of each author is identified below:

Brian Pickering:

• 1.Design of the study; acquisition, analysis, and interpretation of the data
• 2.Design and development of AWARE
• 3.Drafting and revision of the manuscript,
• 4.Preparation final version of the manuscript.

Ahmed, Adil

• 1.Data collection and conduct of pilot study
• 2.Review of the final version of the manuscript.

Gajic, Ognjen

• 1.Design of the study, analysis, and interpretation of the data
• 2.Revision of the manuscript,
• 3.Preparation final version of the manuscript.
• 4.Mentor

Herasevich, Vitaly

• 1.Design of the study; acquisition, analysis, and interpretation of the data
• 2.Review of the final version of the manuscript.
• 3.Design the electronic infrastructure which supports the study.

Competing interests

Dr Pickering, Dr Gajic and Dr Herasevich all have patent application pending on the technology described in this paper (AWARE). They have not received payment of any kind for AWARE or the intellectual property associated with AWARE nor do they hold any consulting positions outside of their host institution.