Capture and Classification of Problems During CPOE Deployment in an Academic Pediatric Center

Abstract

During a planned, rapid deployment of a modified commercial CPOE product to units at an academic pediatric center, problems from users and staff were collected and entered in text format on a commercially available online problem tracking system. Content analysis of 278 collected text reports collected over 3 weeks after the 24-hour rollout period revealed several themes: center-specific implementation problems (45%), transfer-handoff-collaboration problems (14%), missing product functionalities (11%), inadequate training (11%), hardware problems (5%), password problems (4%) and human error (2%). This analysis may prove helpful in future deployments of CPOE in pediatric clinical environments.

Introduction

Computerized provider order entry (CPOE) and electronic medication administration records (eMARs)) offer functionalities (clinical decision support, standardization, improved legibility, high transmission speed) to prevent and reduce medical errors. Introduction of applications into working clinical environments requires coordination of complex technical, social and adaptive changes. Developers and deployment teams must anticipate and respond to problems on all levels to augur success, in terms of process and outcome error rate reductions and user acceptance.

Pediatric clinical environments have needs and workflows distinct from their adult counterparts, and these pose significant challenges to implementing computerized systems. CPOE adoption within pediatric environments, particularly in multi-specialty institutions, requires adaptation and modification of tools designed and acquired for use in adults. Technical needs include an infrastructure that supports pediatric specification of drug inventories, dosages and indications, universal weight-based calculations and pediatric-specific clinical rules, as well as the resources to create them. Organizational needs include advocacy for the prioritization of pediatric needs, as well as assurance of adequate training of users (particularly those who provide services to children infrequently) to assure safe deployment and operation in the delivery of pediatric care. CPOE implementation in pediatric environment has been associated with increased mortality in at least one report¹, which has been interpreted to be in part the result of failures in implementation² .

A medical informatics research agenda includes development of models that anticipate problems and errors that occur while deploying CPOE within specific environments. To be useful, models must be supported by data from actual deployments (“rollouts”) that characterize: 1) the clinical environment and the application being deployed and 2) the initial and ongoing impacts of changes introduced by the deployment. We describe the capture and classification of problems encountered during a three week period following the deployment of a modified commercial CPOE system on three floors of an academic pediatric center.

Methods

The clinical environment in which the CPOE application was deployed consisted of three separate and diverse units in the Johns Hopkins Children’s Center (JHCC), a multi-specialty academic pediatric hospital: a medical/surgical floor for school-age children (16 beds), a medical/surgical floor for adolescents (24 beds) and a post-anesthesia care unit (7 beds).

The commercial application deployed was the Eclipsys Sunrise Clinical Manager (SCM) with ordering, laboratory result reporting and electronic Medication Administration Record (eMAR) functionality, as the first phase of a multi-phase deployment of this application to all inpatient units within the Children’s Center. The SCM product had been successfully implemented in other departments at the Johns Hopkins Hospital (Medicine, Neurosciences, Psychiatry and several Surgical services). The current deployment was the first pediatric inpatient rollout at the institution.

Inpatient SCM implementations at Johns Hopkins include a bi-directional interface to a pharmacy system used to verify medication orders. Special pediatric modifications to SCM made during the two years prior to rollout included development of pediatric order catalog and dose range checking (more than 13,000 dose ranges built), clinical rules, a customized interface for ordering and documenting pediatric patient controlled anesthesia (PCA) and two external calculators developed at the Children’s Center for clinical decision support for parenteral nutrition³ and continuous infusions⁴ that were seamlessly integrated with the SCM product.

Training in the use of SCM is required for all providers in JHCC, with the duration of training varying with the provider’s role. Physicians are required to attend three hours of classroom training with a one-hour module of online pediatric specific training. Nurses are required to attend eight or more hours and are certified by ‘super users’ as POE competent.

Deployment of SCM to the three floors occurred simultaneously over the course of 24 hours. Prior orders (in paper format) were entered into the electronic system by clinical teams previously trained in the POE workflow and transition. Order entry into POE and use of the eMAR was started immediately. On-floor technical support in the form of ‘Super users’ and dedicated technical support staff was available to round with medical teams, to assist with order entry and eMAR use and to collect and answer questions and address problems for two weeks.

Data collection: Problems, issues, complaints and comments were communicated by nursing, physicians, pharmacists, ‘super users’, clerks, nutritionists and respiratory therapists directly to technical support staff, who transcribed reports into an online issue-tracking and project management tool (JIRA^TM)⁵ according to problem type with a text description (and screenshots if indicated) of issues, follow-ups and resolutions. Responses and outcomes were entered online as they were encountered and listed with a problem title, issue, component involved, and a short text description of the problem and ongoing responses. In the 3 weeks following deployment, a total of 278 submissions were collected.

Data analysis: Problem tracking data (including text comments) were retrieved in spreadsheet format. All text entries reviewed and problems were categorized by two reviewers, who were part of the implementation and deployment of the SCM product (CUL, GRK).

Results

A total 278 reports on 283 separate problems were submitted. Half of all problems had been reported by day 5 of go-live. Most reports covered one issue, with one report relating to 3 issues and 3 reports covering 2 issues each. Forty-seven distinct problem subcategories were identified with 1 to 26 problems per category (mean 6, median 4). For the purpose of this paper, we summarized the 26 sub-categories into 9 categories (Figure 1).

Almost half of the problems (128 – 45%) reported related to errors or problems in the implementation of the SCM product that were specific to JHCC. Examples of categories in this group included: formulary items missing from the CPOE order catalog (Ex: Heparin flush 1:1000), suggestions for design improvements, order set design problems (order set omissions, Ex: oxycodone not present on an order set for a burn patients), missing medication frequencies, problems with print report design and inventory item permissions issues (prescribers not allowed to order items for which they had rights and vice versa).

Forty-one (14%) problems were categorized as Transfer/Hand-Off/Collaboration problems. This category describes all processes where the care of a patient changes from one provider to another (transfer from one unit to another) or where two or more providers must collaborate to achieve a result (verification of a chemotherapy order).

Thirty problems (11%) were categorized as secondary to missing functionality or deficiencies in SCM. Examples included: 1) lack of system functionality to adequately identify and prevent true duplicate orders (the provided functionality was not activated due to an unacceptable noise-signal ratio), 2) lack of CDS for complicated pain medication orders (lack of conditional order discontinuation (“if pain decreases, discontinue morphine”)) and 3) lack of conditional instructions for alternate medication routes at nursing discretion (PO route for a drug, IV or PR if the child is vomiting).

Inadequate coverage of topics in training accounted for 11% of problems. Two individual providers and a group of agency respiratory therapists were not trained at all prior to go-live.

Hardware and computer system interface problems accounted for 16 problems (6%). Failure of wireless devices, mice, and monitors were the leading reported issues.

Fifteen reports (5%) identified problems existing prior to CPOE implementation, but whose impact and effect were modified or became more visible as an effect of the deployment. For example, an existing policy on checking blood glucose prior administration of insulin for snacks had not been followed by all nurses, and the requirement for a glucose value in the eMAR made the deviation from the policy quickly apparent.

In eleven (4%) reports, users had difficulty accessing the application because of password problems (‘Forgot my password’) or because the application had not been rolled out on all computers users wanted to use.

Seven reports (2%) related to human errors (‘clicked the wrong item’) and attempts to bypass or misuse SCM (‘inappropriate use of free text or verbal orders’).

Five reports were specific to issues only relevant to go-live problems.

Discussion

Increasingly, reports on the implementation deployment of IT into the medication delivery process, particularly on CPOE in different clinical environments, are appearing in the clinical literature. As CPOE and other IT are promoted as a major path to patient safety, recent studies suggest that implementation is associated with new and unforeseen errors. Deployment (rollout) is a major socio-technical change in work processes (delivery of medications to patients) undertaken to improve outcomes (reduction of medical errors and subsequent morbidity/mortality) and is the culmination of planning, selection, design and implementation of a system, and can result in success, no change or high profile/high stakes failure⁶ .

We found nine categories of problems reported during the three-week period immediately following the implementation of CPOE in an academic children’s center. The largest number of problems was assessed to be due to institution-specific implementation and design decisions. In consideration of the extent of inventory catalog that was designed for the pediatric rollout (13,000 dose range checks), the number of problems was in our opinion low. More importantly, almost all the problems reported were minor in nature and rapidly resolved usually in a matter of hours or days.

Problems in this category were of low risk with one exception (route for Heparin was iv instead of sc) and none had an impact that reached a patient.

Fourteen percent of problems were related to hand-off, transition of care or the need for providers to collaborate on a CPOE task. This finding was not surprising, and it is anticipated that the number of problems in this group will decrease as more floors incorporate CPOE. This topic has been described in detail by Ash et al⁷ .

Prior to CPOE JHCC had already developed successful IT solutions to three high-risk groups of orders: parenteral nutrition, continuous infusion medications and pain medication order sets. For each of these areas, different solutions were chosen for integration into CPOE. For parenteral nutrition, an online calculator, TPNCalculator³ was left unchanged with CPOE referring to the order generated by it. For continuous infusions, a Web-based CDS tool⁴ developed at the JHCC and modified for CPOE integration, communicates with CPOE to which infusion calculations are returned. These solutions, which had been in place for at least one year, did not generate significant problem reports. For pain medication orders sets, we attempted to replace the functionality of a CDS tool in use through a CPOE order set. Twelve problem reports ensued related to pain medication orders with some having potential for patient harm. This experience reinforced our prior belief that CPOE systems provide only limited clinical decision support in the context of complicated pediatric orders.

Although training was mandatory and every attempt was made to identify all possible users of CPOE, some were missed by the time of deployment, causing some confusion and delay. Several topics were not adequately addressed during training. Fortunately, these problems were low-risk since users were forced to stop and ask the support staff when they did not know what to do. Training classes were added in the weeks following go-live, and we emphasized to clinical team leaders the need to have trained providers.

Hardware problems and lack of access were easily addressed following go-live with the most difficult issue being wireless connectivity requiring exchange of wireless access points in the first week. Neither category caused patient harm.

Conclusion

Deployment of information technology into clinical environments can create stresses and potential for errors due to new information tasks, altered communication pathways, and unforeseen problems. Anticipation of such problems in future deployments may be aided by systematic collection and classification of feedback data from current deployments.