Orders on file but no labs drawn: investigation of machine and human errors caused by an interface idiosyncrasy

Abstract

In this report, we describe 2 instances in which expert use of an electronic health record (EHR) system interfaced to an external clinical laboratory information system led to unintended consequences wherein 2 patients failed to have laboratory tests drawn in a timely manner. In both events, user actions combined with the lack of an acknowledgment message describing the order cancellation from the external clinical system were the root causes. In 1 case, rapid, near-simultaneous order entry was the culprit; in the second, astute order management by a clinician, unaware of the lack of proper 2-way interface messaging from the external clinical system, led to the confusion. Although testing had shown that the laboratory system would cancel duplicate laboratory orders, it was thought that duplicate alerting in the new order entry system would prevent such events.

Introduction

There is a substantial body of literature documenting many varieties of unintended consequences of computerized provider order entry (CPOE).^1–3 Ash et al.¹ described a taxonomy of CPOE-related errors. Sittig et al.⁴ formulated a larger framework for the evaluation of health care Information Technology (IT) in general and clinical decision support (CDS) in particular to enhance the quality and safety of EHRs by improving design and usability features.

To help organizations prevent unintended consequences, the Office of the National Coordinator published the Safety Assurance Factors for EHR Resilience (SAFER) guides.⁵ To ensure optimal design of an EHR, recommendations 5 and 6 for interfaced systems state, “The intensity and the extent of interface testing is consistent with its complexity and with the importance of the accuracy, timeliness, and reliability of the data that traverses the interface. At the time of any major system change or upgrade that affects an interface, the organization implements procedures to evaluate whether users (clinicians or administrators) on both sides of the interface correctly understand and use information that moves over the interface.”⁶ Furthermore, recommendation 4 regarding CPOE states, “The EHR can facilitate both cancellation and acknowledgment of receipt of orders for laboratory, radiology, and pharmacy.”⁷ This is dependent on the appropriate programming of the interfaces as recommended above. When system integration is not complete, usability suffers and users find workarounds.⁸

The organization that is the focus of this case uses the Eclipsys (later Allscripts) Sunrise Acute Care EHR, version 4.5 in the first case, version 5.5 in the second. Both versions included full CPOE with duplicate order and other alerting functions. The laboratory information system (Meditech version 5.4) interfaces (Figure 1 ) with the EHR via OpenLink for most orders and results; microbiology, pathology, and blood bank orders first pass via eLink and then to OpenLink. There is 2-way messaging for orders and results but not for autocancel functions performed by the laboratory system. Cancellation functions intrinsic to the laboratory information system do not generate an acknowledgement message back to the EHR. Autocancelled orders on the laboratory side of the interface also incorrectly remain visible as “pending collection” on the EHR. The events we describe herein represent a unique combination of circumstances that we believe have not been reported previously. We analyze these events in the context of the recent SAFER guidelines.

Figure 1.

Schematic of interface design between Allscripts EHR and Meditech laboratory system.

Note that orders (down) and results (up) interface bidirectionally, but laboratory system–generated autocancellations (truncated dashed black arrow) do not.

Description of Clinical Events

Emergency department patient

A patient with chest pain entered the emergency department of a hospital that had been using CPOE for ∼2 years. In the emergency department, nurses enter pre-approved protocol orders for patients with chest pain on behalf of the attending physician.⁹ Physicians and nurses were proficient at entering orders. In this case, the nurse began to enter orders, completing the process at 13:32:07.193 (i.e., hours:minutes:seconds:milliseconds). Meanwhile, the physician observed that orders were not on the chart and proceeded to enter the same orders from the same order set on a different workstation. The physician submitted orders at 13:32:20.983 (i.e., 13.8 s later). Neither user received duplicate alerts, although such a CDS was active in the EHR (Eclipsys Sunrise, version 4.5). The physician then reviewed the orders, observed duplication, and cancelled the first set of orders entered by the nurse. All the physician’s orders remained listed as “pending collection,” which normally indicates to the clinician that the order remains in the system. However, unbeknownst to the clinician, the external laboratory system had already cancelled the duplicate without sending an interface message back to the EHR.

Unknown to the physician, the accepting laboratory clinical system automatically cancels submitted duplicate requests for the same laboratory test ordered for the same patient at the same date and time. However, the laboratory system does not send a cancellation message to the ordering EHR via the system-to-system interface, so the second order—appearing to the physician to be “pending”—in fact did not exist on the laboratory system, which is responsible for messaging the phlebotomist to draw the blood. The physician had cancelled the orders that were actually active, leaving the ones that appeared to be “pending” but in fact no longer existed in the laboratory system. Subsequently, concerned clinical staff observed that the blood for the laboratory tests had not been drawn within the expected time frame and reordered the tests. Aside from this minor delay, the patient did not meet with any harm.

Obstetric patient

A physician entered preoperative orders (now using Allscripts Sunrise, version 5.5) on a patient prior to a Caesarean section. The physician entered more orders postoperatively without adjusting the preoperative orders, bypassing (overriding) a duplicate alert (see Figures 2 and 3 ) for laboratory work due the next morning. Later during the night, the physician performing order maintenance noticed the duplicate orders and cancelled those that were part of the preoperative order set.

Figure 2.

Duplicate alert similar to what the obstetrician/gynecologist (Ob/Gyn) physician saw when ordering the postoperative laboratory tests.

By clicking on “Go Back,” the clinician could have cancelled the duplicate order.

Figure 3.

Another method for a clinician to respond to the alert is to click on the “View Actions” button and to either cancel the duplicate order, or retain it and cancel (arrow) the original order.

As in the first case, cancellation of the first set of orders left the patient without any laboratory orders, even though the second set, which remained visible in the EHR as “pending collection,” appeared to be active but had in fact been automatically cancelled by the external laboratory system.

The next morning on rounds, the physician noticed there were no laboratory test results. Inquiry revealed the deletion of the orders. The physician entered new orders, thus averting any patient harm.

Analysis of Events

The lack of a cancellation message through the system interface back to the EHR ordering system was a problem well known to the laboratory staff. The only users of the legacy order entry system (Siemens Invision^®) in use prior to implementation of the Allscripts CPOE system were unit secretaries and some nurses. The laboratory had instructed this small set of users to ignore duplicate order entry alerts “as the laboratory system will take care of it.” Efficacy depended on a combination of allowing the system to fix itself and on the personnel to check the orders, but not attempt to delete duplicates. The group of users was small, so the workaround to the interface issue was easy to manage and appeared successful. There is no way to know if the old workaround resulted in any patients not undergoing tests they should have had. Although encouraging the human to knowingly let a potential error remain and relying on the electronic system to do the right thing may work for exact duplicate testing, it is an unreliable and risky maneuver if there are slight differences between the tests, or if, as in these instances, the wrong test gets cancelled. When the new EHR went into production (with the existing laboratory system still in place), the thought was that incorporating the duplicate order alert would prevent such orders, making the workaround unnecessary. Instructing all users about the vagaries of the existing laboratory system was not feasible. In retrospect, it was a mistake that scenario planning did not include testing of such circumstances.

The current EHR system provides a duplicate order alert prompted by submitted orders or duplicates listed within the same order entry session. In the first case, a duplicate alert did not fire because of the near-simultaneity of the order entry process. The physician entered the orders while the nurse’s orders were still “in the shopping cart” but not yet submitted. In the second case, the physician did receive a duplicate alert but ignored it, thus defeating the purpose of the CDS. Planning for such circumstances requires vendors and configuration experts to program appropriately to deal with clinicians making the unintentional wrong decision.

In both cases, diligent order maintenance led to the inadvertent cancellation of all the laboratory orders due to the vagaries of the system interface between the 2 systems. In retrospect, it would have been prudent to design the interface to avoid such difficulties, but due to cost and complexity, this was not done. Ironically, good human behavior (i.e., order maintenance) in the absence of full user knowledge of the interface messaging complexities led to unintended consequences.

Failure on the part of a computer system to provide users with an accurate picture of its true internal state is a well-known user interface design error.¹⁰ In this case, there were 2 independent computer systems created by different manufacturers at different times. These 2 systems were connected by an interface engine designed and built by a third manufacturer. All the independent computer systems were working as designed. The error occurred as a result of failure to follow a basic principle of communication: Every transmission, or computer-generated action, that changes the internal state of the system should be communicated back to the sending system and clearly displayed to the user.¹¹

Discussion and Recommendations

These 2 system-to-system interface errors provide an excellent example of an EHR and a clinical laboratory system that are both “working as designed, and were configured and used correctly, but interacted with external systems (e.g., via hardware and software interfaces) so that data were lost or incorrectly transmitted or displayed.”¹² With the previous legacy order entry system, users knew to ignore the duplicate entries and allow the laboratory system, as well as the phlebotomists, to ensure that patients would only undergo a single blood draw without duplicate testing. With the new system, the duplicate alert was supposed to substitute for this workflow, but users often override these alerts,¹³ even when the correct course of action is to pay attention to them. In the cases analyzed here, even if the users had received the alert, it is not definitive that the clinician would have cancelled the correct laboratory entry. Given the generally unknown interface limitation, there really was no reliable and consistent way to avoid the duplication that occurred in the first case short of sending and displaying the order cancellation messages.

In the second case, good practice (Centers for Medicare and Medicaid Services (CMS), The Joint Commission) recommends complete review of all existing orders after a patient undergoes a procedure. The event with the obstetric patient occurred before an electronic process for order reconciliation was in place, and the clinician did not follow this practice. Nevertheless, the clinician could have reviewed the preoperative laboratory orders, and either left them in place when alerted that these orders were already on file and not ordered an identical set postoperatively, or cancelled the former and newly ordered the latter. Given the idiosyncratic laboratory process, it is not certain that simultaneously entering a cancel and resubmission order would be successful.

Similar situations, where multiple competing edits to a data element are made, leading to an unexpected result, are common in database systems,¹⁴ and a range of locking and coordinating systems have been proposed to prevent them. However, because the CPOE and laboratory systems were not fully bidirectionally integrated, these techniques would not have applied directly, as both systems had internally consistent models of order status; the inconsistency was between, rather than within, systems.

Options for Remediation

What other mechanisms might have assisted in avoiding these issues?

Timing of alert

One solution might have been to choose a different time in the workflow at which to trigger the alert. If the duplicate alert was engineered to fire later in the process, for example, upon submission of the orders (i.e., storage), rather than upon initiation (i.e., entry), the first instance might not have occurred because by then the nurse’s orders might have been stored. This change would violate a cardinal rule of alerting,¹⁵ which is to interrupt the user only at the correct time in the workflow. By the point in time when the user submits orders, they have already gone through the mental processing¹⁶ and have made decisions that would have to be reconsidered. It is always possible, although not likely, that multiple users enter orders even closer in time, in which case this approach would still fail.

When we tried to time the duplicate alert to fire when the orders were queued in the shopping cart rather than during the order-entry process itself, it became clear that cancelling an individual order within an order set occasionally resulted in cancellation of the entire order set. This risked making the user reenter the entire order set, which could cause errors due to inadvertent changes to other orders, not to speak of the frustration of the user who must perform rework. Such quirky and erroneous system behavior is but 1 of many reasons for clinician complaints about poor EHR usability, and is consistent with the existing evidence¹⁷ that alert fatigue and clinician behaviors contribute to excessive override rates.

Alert during simultaneous order entry

It is possible to advise a user when someone else has the patient’s chart open and is in the process of entering orders. This occurs in the Allscripts system when 2 users have a shared document open, so that 1 user does not inadvertently erase the entries of the other.

Lock charts during simultaneous order entry

Locking the database during order entry may cause delays in order entry and hence care, which would also be a safety concern.

Stop laboratory-generated autocancellation

The cancellation function in the laboratory system occurs automatically. Altering that feature would have required significant and expensive reprogramming of the hard-coded laboratory system and reworking the interface. Clinicians do not see the cancellation function in the laboratory system; alerts are only visible in Allscripts. The duplicate alert in the EHR was thought to be sufficient to overcome the autocancellation feature, and thus the decision was not to go to the expense and expend the effort to fix the laboratory programming and interface messaging.

Eliminate the duplicate alert

One could decide to suppress duplicate lab alerting. This would revert to the prior reliance on the laboratory system to autodelete duplicates, but would give no message to the user that this was happening. This may not be a problem for exact test- and time-matched orders because the ordering clinician’s intent would be followed. However, this would also reinforce negative behaviors for clinicians who do not review current orders; would not prevent similar tests (e.g., a complete blood count (CBC) with or without a differential count), or the same test from being done within a short period of time; or allow for monitoring of undesirable and potentially costly duplicate order entry. Only carefully constructed decision support can fashion alerts according to these nuances of order entry. In contrast, perhaps one could fashion a second duplicate check after orders are entered, but the complexity of building this for such rare events seems unrealistic.

Repair the interface messaging

The right answer is for the receiving system to acknowledge receipt of all orders and send a cancellation message back to the sending system when necessary, which would change the status of the duplicate orders from “pending collection” to “cancelled by receiving system as duplicate” or something similar. Although computer order entry is desirable for speed, accuracy, and avoidance of preventable harms, relying on the computer to perform tasks automatically without proper human notification is fraught with risk. The 2 situations reported here reflect just 1 small “tip of the iceberg.”¹⁸ Clear communication, whether from human to human or computer to human, is critical for safe operations.

As Figure 1 shows, repair of the interface messaging would have required several steps, including messaging via a third party (OpenLink). It was felt that the CDS system (duplicate alerting) adhered to usability principles, and the allure of this automation overrode the cost of an interface repair. The events reported here, in retrospect, reveal the impact of failure to decipher the full ramifications of that choice.

Adopted Workaround

The first case is one of very rapid, identical, and near-simultaneous order entry, which is highly unusual but clearly possible. No changes were made in response to that event. The second occurrence is common and required changes in behavior. When the hospital instituted an electronic order reconciliation process, good practice made it easier for clinicians to perform postprocedure reconciliation of all orders. The Ob/Gyn group, as a department, decided to reengineer their order sets so that a pre- and postprocedure order set could each stand on its own. Some of the orders overlap (e.g., nursing orders) or are actually duplicate (e.g., morning laboratory tests) but may be different enough that the laboratory system does not recognize them as exact duplicates. Also, occasionally obstetric patients require immediate attention without any predelivery orders. When a patient has predelivery orders on file, the postdelivery orders need not contain items such as admission orders, nursing, and other orders that are needed both pre- and postdelivery. However, for those patients who do not have such predelivery orders on file, they must exist in the postdelivery order set. Current workflow is that when preprocedure orders exist, the clinician discontinues all of them before entering the postprocedure orders. This avoids duplicate testing and medications; fulfills CMS and Joint Commission requirements; and promotes rapid clinician workflow with fewer interruptions, no duplicate alerts, and clear orders for the nurses without any confusion between pre- and postprocedure workflows. This was not an imposed workflow or workaround; the department made this choice to resolve other issues without being aware that it would also resolve the laboratory autocancel issue.

Lessons Learned

1. Rapid, near-simultaneous order entry, though unlikely, can have unintended consequences.

2. Automatic handling of duplicates done in parallel with conscientious clinicians performing order maintenance can inadvertently cause unexpected outcomes.

3. Interface connections may require modification to account for new processes during implementation of new systems.

4. Workflow workarounds that work in 1 system may backfire in a new system. Workflow analysis requires exploration of nonstandard as well as standard processes.

5. As new tools come on line, reexamination of workflows and workarounds may uncover other discrepancies, or offer new solutions to old problems.

6. New systems interconnecting with legacy systems present unique problems that may be unresolvable without expensive repairs, or even with such repairs.

7. The case descriptions in this report suggest that there may be underlying idiosyncratic or complex interface issues that are not readily evident without concerted testing.

8. Confirmation with complete message interfacing to ensure accurate order status is critical.

Conclusions

These 2 cases illustrate several of the risks involved in interfacing 2 or more clinical systems. Assessment of all the system interface and usability issues could have prevented these 2 events. In a resource-constrained setting where externally manufactured computer systems cannot be completely redesigned, careful and thorough testing of these individual systems along with the system interface between them is necessary. Therefore, we recommend that all system-to-system interfaces provide a means of transmitting orders, cancellation requests, and cancellation acknowledgements for all orders and that all EHRs provide a means of accepting and displaying these cancellation acknowledgements.

Funding

This work was supported by the National Library of Medicine of the National Institutes of Health, grant number R01LM011966. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Competing Interests

The authors have no competing interests to declare.

Contributors

RS was involved in the events described in the case report, conceived the concept of, and prepared the first draft of the manuscript. All authors made major contributions to the editing of the paper and approved the submission. The vendors mentioned in the report were not involved in any way with the preparation of the manuscript.