Evaluation of medium-term consequences of implementing commercial computerized physician order entry and clinical decision support prescribing systems in two ‘early adopter’ hospitals

Abstract

Objective

To understand the medium-term consequences of implementing commercially procured computerized physician order entry (CPOE) and clinical decision support (CDS) systems in ‘early adopter’ hospitals.

Materials and methods

In-depth, qualitative case study in two hospitals using a CPOE or a CDS system for at least 2 years. Both hospitals had implemented commercially available systems. Hospital A had implemented a CPOE system (with basic decision support), whereas hospital B invested additional resources in a CDS system that facilitated order entry but which was integrated with electronic health records and offered more advanced CDS. We used a combination of documentary analysis of the implementation plans, audiorecorded semistructured interviews with system users, and observations of strategic meetings and systems usage.

Results

We collected 11 documents, conducted 43 interviews, and conducted a total of 21.5 h of observations. We identified three major themes: (1) impacts on individual users, including greater legibility of prescriptions, but also some accounts of increased workloads; (2) the introduction of perceived new safety risks related to accessibility and usability of hardware and software, with users expressing concerns that some problems such as duplicate prescribing were more likely to occur; and (3) realizing organizational benefits through secondary uses of data.

Conclusions

We identified little difference in the medium-term consequences of a CPOE and a CDS system. It is important that future studies investigate the medium- and longer-term consequences of CPOE and CDS systems in a wider range of hospitals.

Background and significance

Computerized physician order entry (CPOE) and clinical decision support (CDS) systems are increasingly being implemented in high- and middle-income countries with the aim of improving the quality, safety, and efficiency of healthcare.¹ These systems can reduce the substantial disease burden associated with prescribing and medication administration errors and also offer the potential to enhance the quality and efficiency of prescribing decisions.^1–4

In the UK, such systems are commonly considered under a general heading of electronic prescribing systems, which have been defined as: ‘The utilization of electronic systems to facilitate and enhance the communication of a prescription or medicine order, aiding the choice, administration and supply of a medicine through knowledge and decision support and providing a robust audit trail for the entire medicines use process’.² In the USA, basic CPOE functionalities might be defined more narrowly as: ‘computerized ordering of specific medication regimens for individual patients’.⁵

CPOE and CDS systems share common features with respect to medicines’ management in that CPOE systems typically incorporate basic decision support, and CDS systems can impact on order entry. That said, as we are defining them here, they also have important, distinctive features in that CPOE systems are primarily used to facilitate appropriate ordering, including of medications, whereas a CDS system can—in addition to drawing on other sources for decision support—link to a patient's electronic health record and use patient-specific information (eg, allergy history and laboratory test results) to provide more sophisticated, individually tailored decision support.

Much of the published literature reports evaluations of ‘home-grown’ hospital systems, which have been developed over many years.^{1 3 4} The empirical evidence indicates that such locally developed and evaluated systems can result in improvements in care processes and health outcomes.^{3 4 6–8} Locally developed hospital systems are also typically extensively customized. This might reduce the potential transferability of findings to the more generic, commercial applications (which may also be tailored to a specific institution, but in a more limited way) that are increasingly being used internationally.^{9 10}

Commercially available systems, like some home-grown systems, offer the potential for multiple linkages between applications (eg, pathology test results and discharge letters),^{9 10} and may be cheaper to procure and maintain than a locally developed system.¹⁰ There are also significant differences between commercial systems in terms of functionality, and a 10–20-fold variation in their cost. However, knowledge about the impact of different commercial applications on care processes is still limited.^{1 11–13} The available evidence has highlighted major challenges associated with changes to long-established organizational and professional practices over relatively short periods of time and integration with other systems (interoperability).^{2 5 8 10 11} Substantial organizational benefits are, however, expected to accrue once the initial hurdles associated with implementation are overcome and systems are adopted and routinely used across the healthcare organization.¹¹

Current commercially available applications in the UK can be either standalone order entry systems (which is comparable to basic CPOE functionality) or part of an integrated hospital information system offering more sophisticated CDS functionality.^{12 14} In England, the implementation of systems is further complicated by the relative immaturity of the commercial CPOE/CDS market, the limited number of systems tailored to the UK context, variable levels of expertise in implementing information technology (IT) systems in hospitals, and a recent change in the national political context from advocating a centralized, standardized implementation model to an emphasis on local autonomy in systems choice for healthcare providers.^{14 15}

In order to inform ongoing national strategic developments, we conducted a qualitative, in-depth case-study-based evaluation of routinely used, commercially available CPOE and CDS systems linked with medicines’ administration functionalities in two of the first UK hospitals to implement these.¹⁶ We aimed to study the local consequences of the systems once organizations had overcome the initial, now well-recognized, major challenges associated with introducing disruptive technologies. We were interested in answering the question: what are the consequences of introducing CPOE/CDS systems over the medium term in early adopter hospitals? This work is timely, as it has the potential to inform policy and clinical deliberations in relation to the imminent substantial UK and international investments being made in procuring such systems more widely.^{17 18}

Materials and methods

Design

We undertook a qualitative, theoretically informed,^19–21 collective case study^{22 23} of the processes surrounding system optimization in two hospitals. The case study approach is a formal naturalistic research design that involves an in-depth exploration of a complex issue or phenomenon in its everyday real-life context. It typically involves drawing on multiple sources of evidence with the aim to shed light on local processes and extrapolate potentially transferable lessons to other contexts.^{22 23} For the purposes of our study, the ‘case’ was defined as hospitals that had implemented CPOE/CDS systems with a minimum of 2 years’ systems utilization, an approach that thus offered insights into the medium-term consequences of moving to these systems. We investigated similarities and differences between hospitals and systems, which enabled us to theorize about likely medium-term impacts of implementations of commercial systems in a wider range of hospitals.

Ethics and permissions

This work was classed as a service evaluation by a National Health Service (NHS) research ethics committee and gained institutional review board permission from the University of Edinburgh. We obtained all necessary organizational approvals from hospitals prior to starting this work, and participants gave informed consent to taking part. In order to protect anonymity, we removed any potential identifiers of locations and individuals.

Sampling and recruitment

We used our recent surveys of the national landscape and ongoing monitoring of the situation to develop a sampling frame of early adopter hospitals that had implemented commercial CPOE and CDS systems¹⁴; few hospitals in the UK currently have CPOE/CDS systems,²⁴ and very few have a history of established use of commercial systems. We used a purposive sampling strategy to ensure sampling by duration of system usage (measured from the start of implementation) and by system functionality (CPOE and CDS). We purposively specified that hospitals should have a minimum of 2 years’ systems utilization to ensure sufficient time for the system to have become routinely used.^{12 25–28} The ensuing sample of two hospitals allowed assessment of routine use and comparison of CPOE and CDS functionality.

In each of the two included hospitals, we used purposive maximum diversity sampling to identify a range of stakeholders who had been involved in the deployment and use of its CPOE/CDS system.²⁹ We started with local chief pharmacists and project managers, who were asked to recommend other relevant colleagues, including ward managers, lead pharmacists, and IT professionals. Snowball sampling was then used for initial contacts to identify system users, including doctors, pharmacists, and nurses (at varying levels of seniority). Throughout this process, we actively searched for different viewpoints and experiences. No individual approached explicitly refused to participate, but several (8/51) did not reply to our initial invitation.

Data collection

Data collected at the two case study hospitals included a combination of semistructured audiorecorded interviews, observations of strategic meetings and system use by different professions, and documents providing information on implementation plans. This combination of data sources allowed us to investigate: (1) perspectives on the design, uptake, implementation process, and evolution of systems (documents); (2) the nature of everyday use and consequences for practices and processes (observations); and (3) reported experiences and expectations of individuals related to different implementation stages (interviews).

Data were collected by a university-employed research fellow (KMC), between 2 and 3 years after implementation. Interviews were semistructured and informed by topic guides. An initial topic guide was derived from the literature and this was then tailored to individual interviewees’ roles and updated to respond to emerging findings (box 1). Issues explored in interviews included perspectives on the development, implementation, and maintenance of systems, as well as associated lessons learned and suggestions for the future. The average duration of interviews was ∼30 min. Data were digitally audiorecorded and professionally transcribed, and transcripts were checked by the lead researcher (KMC). Non-participant observations were facilitated by a recording sheet (box 1), and accompanied by field notes. This involved the researcher following individual healthcare professionals during their morning work rounds and sitting in on strategic meetings, while taking notes. Specific aspects recorded included: setting, participants, interactions, activities (focusing on those that were medication/computer-related), and the sequence of events. We also recorded emerging thoughts and reflections in a research journal.

Box 1. Indicative interview topic guide and observation recording sheet.

Sample interview topic guide

• Interviewee's background including current position in the organization and specific role in relation to the system

• Details of the system and status

• Views on development of the system: design, uptake, implementation, evolution

• Important local considerations, eg, timelines, resources, infrastructure

• Training and support provided (initial and ongoing)

• Collaboration with the software developer (configurability, management process)

• Data quality and systems’ reliability issues

• How data are produced by the system used

• Lessons learned

• Perceived/anticipated consequences of the system on the quality of care, information flow, patient experience, roles and practices of healthcare professionals, the organization, the local community

• Views on how systems will integrate with existing local and national systems

• Expectations and perceptions (the future, benefits realized or to be realized)

• Perceived changes over time

• Integration with other hospital systems

Observation recording sheet

• Description of the setting—layout: positioning of computers, beds etc

• Description of the participants—the roles of individuals on the ward (names not recorded)

• Activities—focus on healthcare professionals and what they are doing (with a focus on activities surrounding the use of computers such as responses to alerts being generated)

• Events—recording of particular events, eg, speaking to a patient, recording information, speaking to other healthcare professionals

• Time—recording the sequence of events

• Goals—recording what the participants want to accomplish

• Feelings—the researcher's own impressions/feelings in relation to the observation

We continued data collection in each hospital until no new themes were identified and we judged that different data sources did not provide any significant new insights (saturation).³⁰ Documents, observation field notes, and interview transcripts were uploaded to NVivo9 software.

Data analysis

We used a thematic approach to analysis,³¹ using both deductive and inductive approaches.^32–34 In keeping with the case-study-based approach, we analyzed data in each case first.^{22 23} Deductive components involved coding of data into categories developed as part of an analytical framework focusing on system optimization, which was based on a review of the literature and other ongoing work (summarized in box 2).³⁵

Box 2. Coding framework focusing on system optimization³⁵.

Adoption and use

• Changes in work practices

• Usability of hardware and software

Benefits

• Organizational

• Individual

Unintended consequences and risks

• Impact on staff time

• Safety risks

Continuing development and customization

This framework (box 2) helped to: guide the research; facilitate comparison of findings between case sites; identify potentially transferable lessons to other hospitals yet to implement such systems; and relate these findings to the existing literature. Major categories were related to systems’ optimization and included adoption and use, benefits, unintended consequences and risks, and continuing development and customization. We, however, also remained open to additional themes emerging from the data, and the inductive component thus involved identifying these additional new insights. Integration of findings was achieved by combining the framework categories with newly identified themes.

Data collection and analysis were iterative, allowing themes in the data to be explored in depth and contradictory data to be investigated. Initially, data were analyzed within the two cases, allowing triangulation of evidence and exploration of tensions/common themes within hospitals.

Data from interviews in each case were analyzed first, with relevant data extracts being indexed against the framework categories related to the different stages of system implementation (see box 2). This involved exploring particular challenges surrounding implementation and maintenance from different viewpoints (eg, users and implementation teams) by identifying common themes as well as conflicting evidence. We integrated this with data from observation field notes and organizational documents within each case study site before integration across sites. Data were coded along the framework being examined for complementary and contradictory evidence.

To identify overarching themes across the two case studies, and to minimize potential researcher bias, we had frequent discussions among the extended research team. Disagreements about emerging findings were resolved by discussion. This allowed similarities/tensions and negative cases to be explored and interpreted. It also helped to place the findings in a wider context and explore different possible explanations for observed outcomes and the potential transferability of findings.

Results

We identified three key themes in our analysis, which we report in detail here. Overall, differences in medium-term consequences for the two types of commercial systems observed in this work—CPOE and CDS—were negligible. In addition, we identified a range of issues previously reported and discussed elsewhere, and we present those results of our case study in outline only in box 3.

Box 3. Range of other issues identified in this work, which were in line with the existing literature.

Limited evidence on clinical patient outcomes (eg, adverse drug events) and cost effectiveness^36–40

Benefits

• Computerized physician order entry (CPOE) and clinical decision support (CDS) systems can improve practitioner performance by supporting prescribing (eg, helping with inappropriate drug selection, optimizing drug dosage, improving adherence to prescribing guidelines) and reducing medication errors^{38 41–44}

• Reduced length of patients’ hospital stay⁴¹

• CDS and CPOE improve time efficiency and working practices (eg, quicker prescribing, quicker drug turnaround time)^45–47

• Changes in pharmacist work practices—reduced time spent on direct clinical care³⁸

• Improved adherence to guidelines^{36 43 44}

• Less time spent by users looking for drug charts⁴⁸

• Positive effects on resource utilization, provider productivity, and care efficiency^{39 48}

• Reduction in transcribing errors⁴⁹

Risks

• Adverse impact on patients: delaying care and treatment because of issues with computer system^{50 51}

• Implementation process, eg, lack of training^{50 52}

• Medication errors introduced by systems due to information errors associated with failure to integrate with other information systems, and failure of systems to integrate with human work processes (issues in human–system interface)^{53 54}

• Wide variability in the degree of system usage (eg, consultants tend to delegate to juniors)⁴³

• Increase in time spent on patient care and increase in ordering time^{47 48 55 56}

• Use of central station desktops increases time⁵⁷

• Alert fatigue^{58 59}

• Reduction in team-wide discussions⁶⁰

• Users following idiosyncratic practices⁶¹

The three key themes identified in this analysis were

• Understanding the impact on healthcare professionals and support staff

• The introduction of new unanticipated risks to patient safety

• Realizing organizational benefits through secondary uses of data.

We found a high degree of commonality in terms of perceived benefits and risks, and perceived usefulness and usability, between CPOE (hospital A) and CDS (hospital B) solutions. Characteristics of hospitals A and B are given in table 1.

Table 1.

Characteristics of case study hospitals, system description, and data collection progress

Hospital characteristics	System characteristics	System description	Data collected	Data collection period
Hospital A: urban, acute care	CPOE system (not part of a hospital-wide information system)	Have a separate patient administration system which is used for other clinical information including pathology results, clinic letters, discharge summaries, certain scoring and assessments etc Limited decision support for drug–drug interactions and allergies, including order sets Now live in all inpatient wards with the exception of outpatients and critical care System not used for certain types of medications such as infusions and warfarin Have a ‘home-grown’ reporting system that draws on data from CPOE	23 interviews with users (pharmacists, nurses, doctors) and implementers Eight observations (12.5 h) of strategic meetings and system use Notes from recruitment meeting Eight documents relating to anticipated changes (eg, work process maps, implementation plan, business case)	December 2011–August 2012 Planning to return in August 2014
Hospital B: urban, acute care, teaching	CDS modules within an integrated hospital information system	Implemented in all inpatient wards except pediatrics Clinical noting not yet implemented, but ability to see laboratory and pathology results on the system System includes prescribing of all medicines except: variable-dose insulin, patient-controlled analgesia, intravenous hydration fluids. Some other medications such as warfarin are only partially supported They have not switched on drug–drug interactions, duplicates, or contraindications CDS–They use order sentences, order sets, allergy checking, and some locally customized pop-up warnings	20 interviews with users (pharmacists, nurses, doctors) and implementers Four observations (9 h) of strategic meetings and system use Notes from recruitment meeting Three documents relating to anticipated changes (eg, work process maps, implementation plan, business case)	May 2012—June 2013 Planning to return in August 2014

CDS, clinical decision support; CPOE, computerized physician order entry.

Understanding the impact on healthcare professionals and support staff

Benefits for individual users

There were some accounts of systems saving time by, for example: less time spent trying to decipher hand writing as a result of improved legibility; quicker generation of discharge summaries, as forms were created automatically (although the overall discharge process was perceived to be slower as discussed below); faster prescribing—for example, through order sets; the ability to prescribe remotely; less time spent rewriting drug charts by junior doctors; speedier drug-ordering processes; and faster information transfer through improved communication between members of the multidisciplinary team.

…the biggest benefit I would say, is we can actually read things. That has got to be the biggest benefit ever. We can actually read what we're giving now. Hospital B, Nurse, Interview 31

I think once you get used to it, I mean it saves time because you don't have to go and look for the Kardex, you know where it is, it's on the computer system which is good because you've spent many a happy hour looking for Kardexes on wards so it's good from that point of view. Hospital B, Registrar, Interview 42

Negative consequences for individual workloads

However, most users also reported negative consequences for individual workloads, with some tasks perceived as having become more time-consuming for all professions. This contrasted heavily with organizational expectations of time savings for clinical staff.

Increased workloads were reported to be related to poor software usability, with users complaining about the time-consuming nature of searching for and sifting through the increased amount of information held within systems, and the associated activity of scrolling and switching between screens.

…when people have got drug allergies sometimes it can take quite a while to actually find the drug to enter it as an allergy that takes a long time…and you can't sort of type in part of the drug and search for it. And they come under headings of drug, drug classes and so then you're searching in that class and it takes quite a while… Hospital A, Junior Doctor, Interview 10

Yes it's very long-winded…if you want to say add a drug you'd press ‘add’ and then you'd start typing the name of the drug and then hopefully you'll find it. And then in terms of say morphine there must be about 30 different morphine types so they all come up, you've got to find the one you want click on that and then it comes down with a whole list of options…in terms of how you actually give stuff. Hospital B, Consultant, Interview 40

However, software usability problems seemed relatively minor compared with the time-consuming and disruptive consequences of serious shortcomings in the provision of wider technology infrastructure. These included difficulties for users in finding computers, issues with battery life in mobile devices, lengthy log-in times, and slow loading of screens.

…of course you've got to wait for when you go to it and you have to log on all that takes time…Well because the carts seem to break…quite often you can be waiting there and nothing will happen and then it will just go off and you've got to start all over again, then you have to reboot it. All this is just taking time…. Hospital A, Nurse, Interview 18

The most frustrating thing as a consultant when you're…doing the ward round you see maybe 20 patients, you've got a computer that you're working on…and…you go away and then someone else has started to use that computer and I start to get quite angry about that. Hospital B, Consultant, Interview 41

Introduction of new unanticipated risks to patient safety

Although some aspects of prescribing and medicines’ management may have become safer, users also identified a range of potential risks created by the system that had not been anticipated.

Positive impact on safety

In terms of positive consequences, prescribing safety was seen by most to be improved through process-related issues, including

• Fewer transcribing errors resulting from less duplication of information, as it was now ‘all in one place’ and immediately accessible (even more so with CDS systems that included access to laboratory and pathology results)

• Improved accessibility and legibility of information

• Inbuilt order sets and CDS providing information on drugs and doses (which was more sophisticated and customizable in the CDS system).

…we have come up with default values and names…if you're not sure how to spell a medicine it will obviously come up with the one you are likely to want… Hospital A, Consultant, Interview 3

Importantly, nurses reported a reduction in medicines’ administration errors, with systems helping to reduce ambiguity and ensure that medications were administered on time and enabling users to discover if/why they were not administered. This highlights the benefits of scheduling and reporting of where doses are due, delayed, and missed.⁶²

…it's a lot easier to see what you're supposed to be giving and what time you're supposed to be giving it…whereas when you were looking at the old paperwork documentation there was just so much information on it, it would cloud what you…should have been giving at certain times. Hospital A, Nurse, Interview 15

… it's much more clear on [system] what time something should be given and so I guess overall yeah it does improve care because patients are getting their medications at the right time and it flashes up if they're not given at the right time… Hospital B, Nurse, Interview 38

Potential negative consequences for safety

Participants also reported a number of new negative unanticipated consequences for safety associated with the introduction of new systems.

Local adaptations of working practices (workarounds) increased potential for error in clinical practice. For example, some clinicians, particularly doctors, tended to take notes on paper and then enter information into the systems in batches once a computer became available (meaning that electronic records were, as a result, not kept up to date for several hours). Others tended not to look at the electronic charts, as this was viewed as too time-consuming, which could result in them missing potentially important information.

I tend to do it in batches just because of the lack of computers on the ward…ward rounds happen so…you've not got time to in-between patients otherwise the rest of the team will have seen five and you won't know what's going on so it tends to be in batches… I have my clipboard, write what I need to prescribe and then go and prescribe it later. Hospital A, Junior Doctor, Interview 10

…if you're doing a ward round as a consultant in the old school where everyone has got a drug chart at the end of the bed, part of the routine of going round is to take the drug chart and look to see what the patients are on. What becomes sometimes bad practice is because you have to cart the computer round and you have to load it up and it takes time sometimes on some ward rounds…people don't look at the drugs… Hospital B, Consultant, Interview 43

Users also reported that the additional security measures were at odds with the contingent and highly pressured work routines of healthcare professionals. Repeated logging-in and -out of hardware was viewed as too time-consuming, potentially presenting new areas of risk (eg, with users avoiding having to repeatedly log-in).

…it takes you longer than it used to just to look at the card and…people don't look at it as often… Hospital B, Consultant, Interview 43

Participants further referred to particular system properties, related to user interfaces and system functionality, increasing the risk of specific errors. These included a general lack of system flexibility, which was paradoxically often a symptom of systems attempting to improve safety (eg, by making certain tasks or viewing of screens compulsory or sequential and thereby resulting in problems obtaining an overview of past activities).

…your medicines because the times they're due you have to give them at that time otherwise you can't give them, like it won't let you go back. So say if I've missed my eight o'clock medicines and it comes to two o'clock, my list of two o'clock medicines will come up but it won't let me like give the eight o'clock ones now so it's a bit strange but like the old way you could write on your chart what time you give it so it's specifically right… Hospital A, Nurse, Interview 17

…when you look at the paper record you can clearly see what patients are on and what they're not on. Sometimes on our system…it will have an antibiotic which will still appear to be [prescribed] but you have to look at the fine print to see that ‘oh no it was stopped a week ago’, so it doesn't clearly…cross off so there's an error because you might look at that and think ‘oh that's OK he's on antibiotics’ I'm happy with that, so that's a potential error. Hospital B, Consultant, Interview 43

Users suggested the system could also introduce new risks of selection errors (eg, when the use of pull-down menus resulted in users accidentally selecting the wrong drug), automatic allocation of timing and doses (without the user realizing), duplicate prescribing of medications (eg, through different routes), and through free text prescriptions, which were still possible in systems designed with the intention of providing flexibility for users. These meant that users could prescribe doses of medication that were not available from the drop-down menu.

…there is a facility to do a free format prescription. Say for example…a methotrexate dose that is given weekly, you can actually select it weekly but say if the doctor did it in this free format and said methotrexate 10 mg weekly and the system doesn't understand that that's weekly, it doesn't attach it to a frequency so it then comes up every single day for admin… Hospital A, Pharmacist, Interview 4

The prime example, the only thing I've ever seen is if somebody is taking paracetamol orally as prescribed, then the patient becomes sick, nausea, vomiting and you say can you prescribe that paracetamol IV [intravenous] and then they'll add that IV prescription and then you've got two prescriptions of the same. Hospital B, Nurse, Interview 38

Realizing organizational benefits through secondary uses of data

Refining organizational processes

Participants valued opportunities to refine organizational processes surrounding quality improvement. These related to: organizational capacity building around internal implementation expertise; increasing productivity by reducing medical and pharmacy time on certain activities such as prescribing and generating discharge summaries, enabling more rapid turnaround of patients; allowing recorded audit trails of organizational activity; stimulating the implementation of and adherence to guidelines; and exploiting generated data through secondary uses. For example, institutions could run reports, such as on missed doses, and use data within systems to target specific areas of concern.

…in terms of…general improvement in clinical care we can monitor now medications not given, so whereas in the past you might have to…do a one week audit if a ward drug is not given…we've got that data for the whole hospital…hundreds of patients… Hospital B, Implementation Team, Interview 39

Optimizing organizational performance

Senior managers who had implemented the more sophisticated CDS system also gave examples of how systems improved organizational performance through the implementation of new guidelines and by ensuring associated adherence of users:

…there was a manufacturing problem with Tinzaparin and we had to change to a different drug. Previously we would have had to go round every single ward, tell the doctor get them to change it. And we just put a little pop up to prompt them to do that and we knew every single patient who was on the drug so the pharmacist targeted them and within one day the whole of the hospital was changed over and normally that would have taken about a month to kick in. Hospital B, Implementation Team, Interview 14

Quality improvement

The use of data held within systems was seen as fundamental to achieving some key quality improvements for different professional groups. For example, pharmacists could draw on targeted searches for high-risk medications and/or patients and then prioritize these. The following observation notes at hospital A with a CPOE system and a locally developed reporting system using CPOE data illustrate this:

He says that the reports are helpful as they help him to prioritize his workload as they help him to identify high-risk medications and missed doses. The information in the reporting system [which is locally developed] is pulled out from [name of patient administration system] and [name of system]. The reporting system also highlights new patients which are regarded as high risk and which therefore need to be seen first by the pharmacist…. He does the searches for his ward about three to four times a day. In addition to this, centralized searches for high-risk medications are done hospital wide by “someone centrally”. Hospital A, Pharmacist, Observation notes

Discussion

Overview of findings

We found some medium-term, organizational benefits that were related to secondary uses of data, but also reports of some adverse consequences for individual users and patients. Our data, however, also suggest that improvements in system design and integration could improve productivity and workflow (eg, through better user interface design to allow integration of information, eg, by enabling multiple windows to be viewed simultaneously for the same patient). Even greater benefits might result from ensuring adequate availability of computer terminals and underpinning computing and communication facilities. There is also a need to address the log-on time problem, as users tended to develop workarounds, which in turn could create new risks.

Strengths and limitations

Our qualitative study provides important insights into end-user experiences of working with commercial CPOE and CDS systems with different functionalities once the challenging initial implementation phase has been negotiated. Conceptualizing the two hospitals as a case study helped us to explore local processes and consequences of new systems in detail.^{20 21} The in-depth nature of this work allowed us to investigate the complex implications of systems for different organizational stakeholders. The diversity of perspectives consulted—drawing on prior theoretical work—facilitated exploration of the interplay between technical, organizational, and individual dimensions in the ongoing implementation journey.^19–21 The study design was further strengthened by consolidating evidence from a range of data types collected from the two case studies with diverse systems (triangulation).

However, as we interviewed direct users and implementers, we did not necessarily capture all benefits, such as those related to improvements in organizational performance. Some benefits such as time savings may be masked by other staff frustrations arising in complex work processes, presenting issues of attribution. Moreover, directly attributable safety issues observed in this work may be limited, partly because we have focused on medication-related areas only.

Because of the overall very limited number of hospitals in England with systems in routine use, our sampling strategy was designed to identify early adopters. Although early adopters are not necessarily representative of other hospitals in England, studying their experiences provided insights into the processes associated with more established use of commercial systems with different functionalities, which policymakers, hospitals, and clinicians may need to be aware of. Our initial findings reported here may, however, not be generalizable to other sites in the UK or indeed other countries. The two sampled hospitals had some important, potentially confounding, attributes, such as strong organizational leadership, receptive contexts for change, and innovative organizational environments. We plan to address this in follow-on work in a greater number and range of hospitals. That said, we hope that the theoretically driven approach to sampling, analysis and interpretation of data, as well as the richness and depth of data collected, will provide relevant and potentially transferable insights from this work. It is also important to highlight that the hospital that procured the CDS system (hospital B) still has a wide range of options regarding the local continued development/iteration of functionality, whereas this is more difficult with the CPOE system where the functionality is limited. Similarly, hospital B has an implementation roadmap for the future, while hospital A is reliant on any supplier developments of the implemented system.

Furthermore, we did not directly measure safety or efficiency, and perceptions around these domains can differ substantially from objective assessments. Qualitative and quantitative approaches can, however, provide complementary insights. This essential foundational work has allowed us to understand professional/organizational perspectives and experiences, and will serve as a basis for mixed-methods enquiry in a wider array of hospitals. This should include a more detailed exploration of different functionalities, examining a wider range of perspectives over longer periods of time, and investigating the impacts of systems on healthcare professionals, organizations, and patients.¹⁶

Considering our findings in light of the existing literature

In line with current empirical evidence, our findings suggest that the inadvertent introduction of new, often unanticipated, risks with new health technology is likely,^{15 54 55} and the evidence from this investigation into medium-term consequences suggests this may remain the case even when organizations have moved into more established, routine use of these systems. As is repeatedly highlighted, this appears to reflect the challenges of integrating new systems with existing work processes and the difficulties of achieving interoperability between systems.^50–54 Nevertheless, empirical evidence also suggests that CPOE and CDS systems can positively impact on patient outcomes, and commercial systems can result in time savings for healthcare professionals in the medium term.^{1 12 63 64} However, although we observed some system benefits, staff in our study did not report discernible overall time savings. These differences may be due to: (1) international variations in the way healthcare is organized (most existing evidence comes from countries other than the UK); (2) the focus on task automation and lack of emphasis on business process transformations in our case study hospitals; (3) specific issues that might be remedied with improvements in system design and integration/work reorganization; and (4) respondents’ differential reporting of processes that were made easier or slower/harder.

There is also now increasing recognition in the literature that timelines for realizing benefits are often greatly underestimated.⁶⁵ Admittedly, a timeframe of 2 years after implementation is insufficient to make definitive statements about benefits in the long term, but our findings do highlight medium-term consequences for users and organizations, which has hitherto received far less focus than studies investigating short-term consequences. One recent study by the European Commission of several electronic health record and CPOE/CDS implementations over a period of 12 years has reported that it takes at least 4, and more typically up to 9, years before technologies produce returns on investments.⁶⁶ Our work suggests that the long time to realize organizational benefits is likely to reflect the later exploitation of data through secondary and innovative uses.^{1 65}

This study further builds on the existing literature by providing insights into differences related to organizational, user, and safety consequences between CPOE and CDS solutions.^{3 4 12 67} The differences between systems observed in our work were minimal, suggesting that CPOE and CDS solutions pose similar challenges with respect to risks and realizing benefits in the medium term. Any significant differences in post-implementation improvements are more likely to emerge as CDS solutions become more established and more sophisticated functionality is used, or as CPOE systems become more interoperable and/or integrated—but this will take time, and might occur more rapidly if incentives are aligned and/or post-implementation testing of decision support is used.⁶⁸

Implications for policy

Policymakers often fail to appreciate the length of time associated with meaningful secondary uses of data. A recent report by the English Department of Health, for instance, states that estimated financial benefits of CPOE/CDS systems are likely to be in the region of £270 million per annum from year 2 onwards after implementation.⁶⁹ This illustrates the importance of more realistic estimates in terms of timelines, costs, and returns, but also the need to continue tracking emerging benefits through longitudinal evaluations of systems and processes.⁷⁰ Such considerations are particularly important in light of the recent announcement by NHS England of investments of £500 million focusing on the implementation of CPOE/CDS systems and associated functionality in hospitals, and subsequent further funding announcements.^{17 71} The strategy encourages hospitals to move towards increasing system maturity. CDS and secondary uses are viewed as vital to achieving this, but central capital funding must be spent by March 2015, leaving hospitals potentially susceptible to rushing the planning of the complex changes associated with implementation. Policymakers may wish to consider financial incentives for organizations that successfully implement systems, and post-implementation testing to improve the likelihood that key CDS systems will be implemented.

Conclusions

Our work highlights how shortcomings in systems design and inadequate provision of devices and computer infrastructure, and the consequent use of workarounds, can give rise to new errors and a number of unanticipated safety risks. The lack of clarity surrounding benefits and apparent trade-offs between individual workloads and organizational benefits highlights the need for contemporary rather than retrospective study and for quantitative evaluation.

The perceived differences between CPOE systems with and without more advanced CDS were limited. It remains to be seen whether this trend will continue in the long term, with implementation of increasing functionality in CDS solutions. This may involve more complex CDS functionality, inclusion of more complex areas and medications, integration with other existing systems (such as laboratory systems), and more sophisticated exploitation of data through secondary uses. We hypothesize that the greater sophistication of CDS systems and more efficient processing of data will result in more substantial long-term benefits compared with CPOE solutions, but it is as yet unclear whether the same benefits can be realized through incremental implementation and interfacing with CPOE systems. It is important that future studies build on this work by investigating the longer-term consequences of CPOE and CDS systems in a wider range of hospitals.