Qualitative evaluation of an electronic prescribing and administration system

Abstract

Objective

To provide a formative socio‐technical evaluation of a pilot implementation of an integrated electronic prescribing, automated dispensing, barcode patient identification and electronic medication administration record (EMAR) system on one ward.

Design

A qualitative observational approach using discourse analysis within a socio‐technical evaluation framework addressing systems functions, human perspectives and organisational context.

Setting

Surgical ward in a teaching hospital.

Participants

Staff on study ward and in pharmacy.

Intervention

Implementation over time of an integrated electronic prescribing, automated dispensing, barcode patient identification and EMAR system.

Main outcome measures

Assessment of technical performance, developed attitudes to the new system, changes to delivery of care and work practices.

Results

The system was successfully implemented on the ward, and remained in operation for over 2 years. Many of the technical components of the system initially showed problems, but the system evolved, with increased functionality and improved performance. Attitudes to the system in the early stages were mixed. Over time, and with experience of making the system work for them, staff attitudes changed to become more balanced and the potential benefits of the system became clearer to most. The system structured the work of staff, sometimes unexpectedly.

Conclusions

Electronic prescribing systems need to be seen as occasions for change and learning rather than as black‐boxed technical solutions to identified problems. The evaluation framework allows understanding as well as hypothesis testing, and is recommended for future evaluations of electronic prescribing systems.

The introduction of electronic prescribing in hospitals to reduce medical errors is now a major element of health policy in the UK and USA.¹,²,³ These policies follow several studies in the medical literature on a small group of systems that have shown reductions in various types of medication error, and/or improved decision making.⁴,⁵,⁶ Such studies have generally focused on a limited number of process outcomes, such as prescribing or transcription errors, and sometimes resource use. However the value and usefulness of any information and communication technology (ICT)‐based system depends not just on the immediate functionality provided by the hardware and software, but also on how individuals and organisations adapt it to their situation and themselves adapt to it.⁷,⁸ This socio‐technical perspective is often missing from evaluations of medical uses of ICT, which then can only tell half the story. If or when technologies are effective and useful depends on the ‘worked out fit' achieved between them and their institutional and professional setting⁸,⁹,¹⁰ as recent reports of problems and errors caused by electronic prescribing systems illustrate.¹¹,¹²

In the present study, we evaluated a novel integrated (closed loop) electronic prescribing and drug administration system designed to improve patient safety. Our approach was to use an evaluation framework based on socio‐technical theory, something that has been so often missed in past evaluations of electronic prescribing systems. The quantitative part of the framework is presented in a companion paper¹³ and integrated into this qualitative assessment in the Discussion.

Participants and methods

Setting and system

The setting was a 28‐bed general surgery ward in a London teaching hospital. Prior to the introduction of the electronic system, doctors prescribed by writing onto a drug chart, and nurses administered the drugs, circulating the ward at fixed times with a drug trolley containing mainly stock and some individually dispensed drugs. The new computer‐based system established on the ward included electronic prescribing and administration as well as elements of stock control (see box 1). The technical system (hardware and software) came from a small, specialised supplier (ServeRx, MDG Medical, Israel, version 1.13).

Box 1: Description of systems

Principal elements of technology within the system

• Central ward‐based server computer holding patient and pharmacy databases and supporting data back‐up procedures

• Automated drug cabinets to hold ward stock in individual computer controlled drawers

• Central console for drug selection and trolley loading, used principally by nurses via a touch screen interface

• Two workstations, on the nurses' station and in the doctors' office. These are available for all tasks including prescribing and review

• One workstation located in the central pharmacy, available for ward pharmacist

• Two portable tablet computers (handheld touch screen devices carried round the ward), used for prescribing with stylus input. These devices, as with the drug trolleys, did not use wireless networking and only updated the central server computer when returned to their docking cradle

• Two computerised dockable drug trolleys, each with touch screen and barcode readers

• One barcode printer to produce new wrist straps for all patients entering the ward

Work flow of the system

• Prescribing takes place either at the bedside, with doctors using one of the two portable tablet computers, or at one of the two workstations on the ward

• Prescribing through a structured screen with access to patient medication history for the current admission. Uses pull‐down menus for selecting drugs and defaults for doses and timing

• Prescription data held on the database located on the ward‐based server computer

• Drug stock held in large automated drug cabinets (some refrigerated) on the ward

• Administration via two computerised drug trolleys that contain electronically locked individual drawers for each patient

• Patients identified prior to administration by use of barcoded wristbands read by a scanner on the drug trolley. The patient's drawer then unlocks and springs open

• The tablet computers and drug trolleys do not use wireless networking. They update the database only when re‐docked (plugged into base station)

• Prescribing data can be reviewed, checked or changed by pharmacists, using the two workstations on the ward or the one in the pharmacy

With the new system, doctors prescribed using a tablet computer (a handheld touch screen device carried round the ward) or at a fixed workstation computer. The system monitored when patients needed their medicines and prompted nurses in preparing a drug round. To prepare for a round the nurse worked at a central console with the touch screen located at the nursing station and selected patients one at a time. For each drug prescribed, a unique drawer in a wall cabinet opened as did one drawer with the patient's name on it in a computerised drug trolley. Once the dose was transferred from the storage cabinet drawer to the trolley drawer, the cabinet drawer closed and the process was repeated for other required drugs. The patient's drawer would then be closed and automatically locked. Then the nurse selected another patient and the process was repeated. On admission to the ward, patients were given a barcoded wristband. During the drug round this was scanned by a reader attached to the drug trolley. The patient's drawer in the trolley then opened and the drugs were administered with details of administration noted on the trolley's computer.

The project to pilot the system started in January 2003 following initial contacts between the hospital and the technology vendors. It took 6 months to complete preliminary work, including preparing an operational outline, refitting a treatment room on the ward to hold the equipment, installing the equipment and system testing. Brief initial training was carried out using an early version of hardware and software. The system went live on the ward in June 2003 and remained in use until January 2006.

Research design

The research approach was informed by the socio‐technical perspective which considers information systems as having social and technical elements,¹⁴ and technology's characteristics and capabilities as being revealed through use.⁷,¹⁵ This approach is reflected in Cornford and colleagues' evaluation framework,¹⁶ which is used here (table 1), and is similar to that proposed later by the UK Department of Health.¹⁷ The framework is structured as a matrix with Donabedian's¹⁸,¹⁹ model for evaluating quality of care (structure, process and outcome) as one dimension, and system functions, human perspectives and organisational context as the other. The framework addresses technical performance, changes to delivery of care and work practices, as well as the longer‐term prospects of a system and its sustainability within organisational contexts.

Table 1 Evaluation framework.

	System functions	Human perspectives	Organisational context
Structure	Technical detail	Changed work conditions and implied requirements	Sustainability, opportunity costs, management needs, skill requirements
Process	Information processing; correct and valid	Human participation in tasks; social interaction	Altered delivery and practice
Outcome	Relevant, applicable, reliable	Quality of service, and outcomes	Effect in the world

Adapted from Cornford et al.¹⁶

A qualitative approach was adopted for data collection, complementing the quantitative study.¹³ Interviews were held with relevant stakeholders including nursing staff (n = 9), doctors (n = 5), pharmacists (n = 8) and hospital management (n = 4), in a range of circumstances: some on neutral ground, some while they were being observed on the ward or others after project steering group meetings that we had observed. We also held a focus group session 9 months after the system went live, at which doctors (n = 5), nurses (n = 5), pharmacists (n = 3) and hospital managers (n = 1) discussed their experiences and attitudes to the system. We taped and transcribed the formal interviews and a dedicated note taker was present at the focus group. These primary data sources were complemented by findings from the quantitative study and related project documents.

The data for the study was captured by two experienced researchers (TC, EK), who also conducted the analysis using discourse analysis. Discourse was interpreted as embodying all aspects of how people communicate and interrelate.²⁰ Using Foucault's works on the relationship between knowledge, power and discourse²¹ as the theoretical basis, data analysis was undertaken by each researcher reviewing interview transcripts for each professional group, and then in temporal sequence. The researchers sought to unveil different interpretations of the world and belief systems of the interviewees. This resulted in a rich narrative document capturing interviewees' accounts of how the project and the system had evolved through time.

The analysis undertaken individually by EK and TC was then consolidated by discussion and checked against observations. The work was regularly reviewed by other members of the implementation and evaluation teams, and evolved in the light of their comments. Based on this process the researchers developed their analysis into the themes reported here.

Results

Box 2: Prior expectations and initial opinions

There was a spectrum of views.

• Positive examples included:

“It will enable us to capture our practice, eg, asking, ‘Tell me all the changes done in the last 48 hours' this will be a huge benefit. It will make audit possible, eg, listing all the patients on drug X. It will improve patients' care, drug dispensing, reduce risks inherent in paper‐based and human systems.” (Pharmacist)

“Things have to be paper free. Paper is too ‘old'. Computers are the future.” (Junior doctor)

• Some thought there would be benefits but they would take some time to realise:

“The system hasn't reduced medication incidents but in the future it should (now there are hiccups). Still the system highlights that drugs were not given, gives reminders of drugs to be given outside normal drug rounds. I see a future for it but work must be done.” (Sister)

“If and when it is successful then it will be a great thing for everywhere. You will get good responses in 3–4 months. Or am I being too optimistic?” (Charge nurse)

• Others were sceptical or antagonistic to the system:

“I'm a bit sceptical. The electronic prescribing system at hospital X didn't work very well.” (Pharmacist)

“It will cause animosity because the system is based on a presumption that human error is wrong and that it happens often. For example, the promotion video stresses ‘you cock‐up a lot and this is how to stop it'. … Now I am thinking about what I can and cannot do with a drug chart. … I appreciate paper drug chart more.” (Junior doctor)

Box 3: Attitudes and opinions 6–18 months after the system went live

• There were positive views about the benefits of the new system compared with the old:

“I like the system. I am now worried about going to the old system, I prefer this to paper drug charts.” (Sister)

“Coming from different wards using traditional trolley, and trying to find different drugs in the trolley and maybe finding a drug isn't there—something that wouldn't happen with ServeRx—in that respect I find it very beneficial.” (Nurse)

“It is easier to spot problems. Also, it seems that everyone is thinking a bit more about drugs, there is a greater ‘visibility' of prescribing and that's good.” (Pharmacist)

• Others recognised the benefits, but that there were some disadvantages, such as the time taken or the loss of patient contact:

“Once I started to use it ... it was OK, it is quite user friendly and ‘smooth working'. Quite reliable in comparison with the existing system. From a prescribing perspective it works better than drug charts. Drug charts go missing, now we can see the whole record of prescribing history, who made what changes, etc. ... . But now the process is a bit different, need to put passwords in all the time which is good (security) and bad (time).” (Junior doctor)

“Initially I thought it was quite confusing but that was because I wasn't using it all the time. Since I have been doing the ward, I think it is quite good. … But I find it difficult to look at a prescription and know whether it is appropriate without speaking to the patient. Which means I have to go to the ward. It is taking me longer. Overall I would rather it was here and I wouldn't want to go backwards and take it out.” (Ward pharmacist)

“I avoided it because I'm here for the patients. My job is looking after patients. … I spent less time on a drug round but it doesn't help me to do a better job as a nurse. Before nurses could pick up a chart and at a glance see what is there, what happens. In 20 seconds you could see what they've got there (on what drugs a patient is).” [Then she added] “When I had to do a night shift it definitely helped, it was quicker, and as you are so tired there is less chance of errors.” (Staff nurse)

The electronic prescribing and administration system was quickly integrated into the ward's working practices (after a brief period of parallel running alongside the old paper based method) and became relevant to almost all who worked there. The system operated in a live mode for over 2 years, taking 3 months (June–September 2003) from initial use to becoming a stable system. A summary of the findings, structured by the evaluation framework, are presented in table 2 and discussed below. A selection of responses from stakeholders is given in boxes 2 and 3.

Table 2 Principal findings.

	System function	Human perspectives	Organisational context
Structure	ServeRx is a standalone technical system that integrates support for electronic prescribing, scheduling, automated dispensing and electronic administration, as well as elements of stock controlIncludes computerised drug trolley and automated cabinets for storageInstalled on one ward and with no substantial connection (other than basic data) to the hospital's other information systemsExhibited initial technical problems and needed tailoring to local requirements and UK practiceMany problems rectified with subsequent versions but some hardware and software shortcomings remain	Initial training provided to doctors, nurses and pharmacists but ongoing support (including the physical presence of a trainer on the ward) was necessaryDoctors had little involvement in shaping of the system and some considered this as not appropriateNurses hesitant about the system at the outsetSystem driven forward by pharmacists; other professionals felt that it reflected most strongly pharmacists' interestsPharmacists clearer at the outset what the system was for and what they wanted from it	One of a number of ICT‐based projects in pharmacyA pilot project, envisaged as an opportunity to learn from this system and inform future initiativesInitiated and managed by pharmacyEnjoyed extra resourcesSubstantial commitment from many staff members
Process	The system reached stability and became well integrated into the work of the wardOnce stable, the data processing functioned wellInbuilt structuring of core work processes of prescribing, dispensing and administration of drugs performed satisfactorily for most but not all drugsLimited decision support for prescribing	The system influences how, when and where prescribing is done and checked, shaping the work processes of doctors, nurses and pharmacistsExperience of using the system over time and over its versions has meant that the attitudes towards it have evolved and shiftedNurses administering drugs were bound by a sequence of procedures embedded in the systemThe way different professionals communicated with each other changed	Experience emphasised the challenge that comes when the working practices of professional groups and interdisciplinary teams are reshaped by introducing a powerful and structuring technologyTechnology can be explicitly used to enforce a “good” process, but some aspects of practice do not neatly fit, or are incompatible with, the systemThis is apparent on one ward, but across a hospital the effect could be magnified as different specialities are considered
Outcome	A usable technology (hardware and software) that was over time shaped and integrated into ward practiceFacilitates safe or safer prescribing and administration processesProvides data which are of an appropriate quality and available for all participants in the care processContinuing mismatch between system characteristics and the use of certain drugs	A system which pharmacists, and perhaps more reluctantly, nurses came to accept and many would missDoctors' opinions more varied; they identified shortcomings but believed in benefits an ideal system might bringRestructuring effects on the way different professional groups work, varying opportunity to exercise a degree of autonomyGenerally perceived as safer or at least potentially safer, reducing some errors, but also acknowledged as introducing new risks	The pilot has led to valuable lessons: benefits and drawbacks of electronic prescribing; scope of impact; the processes involved in “hosting” such a system; project management and implementation strategiesThe system was not moved with the ward when it was relocatedThe system is now planned to be used on a care of the elderly ward

System functions

The technical elements all started life with some operational troubles. The tablet computers had several problems: limited battery life; occasional data loss when being connected to the main system; and screens that were too small and dim making it hard for two or more people to review data simultaneously. At one point the tablets were taken out of use, later they were replaced with a new version. In practice, most prescribing was undertaken using the two fixed workstation computers.

A number of desired system functions were not available at the start of the period of use, for example the ability to generate and dispatch discharge prescriptions was only available after about 12 months of use. Traditional aspects of decision support, such as drug–drug interaction and allergy checking, were not implemented by the project team as they were not appropriately embedded in the technology. The warnings did not appear at the time of prescribing, only when the nurse was about to administer the drug. However, some features that could be considered as decision support were included by the project team, including default standard doses and frequencies for commonly used drugs.

In the initial period the server computer was often slow and would at times freeze—a problem associated in part with its data backup routines. Other problems were the small size of the trolley drawers and the lack of suitable handles, and initially trolleys broke down. The automated drug cabinet too, while functioning well, raised design issues associated with the ergonomics of its use, particularly when it was being restocked. Finally, in the initial periods, the sheer quantity of demand on the system for all types of functionality meant that staff were often queuing for access, particularly at the workstations.

After the initial period of instability and slow running, and with the release of a new version of the software, the technical system became generally stable from about September 2003. Once stable, the data processing functioned well. The inbuilt structuring of the core work (processes of prescribing, drug selection and administration) performed satisfactorily for most drugs, but there were notable exceptions. Some important aspects of this work were found to be incompatible with the system—that is, warfarin, sliding scale insulin, variable‐dose heparin and intravenous fluids. These drugs could not be safely prescribed through the system because their protocols did not fit easily into the structures embedded in the software, such as regular doses at set times. These remained on the system only as a prompt to consult a subsidiary paper chart. Other problems involved antibiotics. The duration was set to the hospital norm (5 days) and this could lead to doses being omitted thereafter if the prescription was not renewed. It was also some time before it was realised that oxygen had not been put on the system and was being prescribed verbally; this was quickly rectified. Nevertheless, the most important outcome for the technical system is that it worked and remained in continuous use for over 30 months, producing a statistically significant reduction in prescribing and drug administration errors but increasing the time taken for these activities.¹³

Human perspectives

The principal users were doctors, nurses and pharmacists, though almost anybody who came onto the ward to contribute to care was a potential participant (eg, dieticians, physiotherapists). Throughout the period of implementation the ward had substantial support from a specialist pharmacist and nurse/trainer to work with staff as they familiarised themselves with the system, to answer their questions, collect problems and issues for resolution, provide formal and informal training and reassure users. Even so, it was a considerable shock to many staff that a system such as this did not arrive “fully functioning” and that such an effort was needed to support it in use. This was illustrated by staff when, several months after the event, they reflected on the process of implementation:

“It was a living hell for 6 weeks. Not just because of the change process but because of unstable technology. We were completely unprepared for the degree of change although we tried to prepare. It is a complete change of practice.” (Staff nurse)

Another nurse made similar comments and added:

“Once we got to like the system and we were actually doing the drug rounds electronically, it seemed to stabilise a little bit, people's opinion of it was a bit better and as they got used to the system they seemed to have worked with it. They made it work for them. A lot of the staff who have seen it throughout will actually turn around and say no, I don't want it to go.”

Attitudes and opinions developed over time and with experience of the system. The initial period was the most challenging and a certain solidarity emerged, with nurses, pharmacists and doctors helping each other and showing a shared sense of being in the frontline (box 2). As the system stabilised, and by the later period of the study, almost all participants had arrived at more positive or “balanced” views of the system (box 3). These statements by staff are shown by stage of implementation in boxes 2 and 3, and analysed by profession below, showing how the system changed working practices.

Even after extended use some enduring issues remained. For example, the disappearance of paper drug charts was a common theme commented on by pharmacists, doctors and nurses with their use in practice going beyond their “official” purpose and serving as a quick method of assessing the clinical state of a patient while standing at the bedside. Many emphasised the quick access and general overview that the end‐of‐bed chart provided, even when in the same sentence they acknowledged the frustration of physically locating charts, and incomplete, incoherent and illegible entries.

Doctors

Doctors generally understood that electronic prescribing could contribute to improved medical practice and the elimination of error. However, they often maintained a strong distinction between this system with, as they saw it, some major failings and problems, and an ideal system. Although the introduction of the system had been approved by the director of surgery, surgeons had little involvement in the development phases until the system went live. One nurse (a sister) reflected:

“A lot of doctors didn't come to training. Even if they did, they didn't pay much attention.”

Some doctors preferred to learn on the job:

“I haven't had a formal training but I picked it up, it is fairly user friendly. And nurses help you a lot … you can ask them questions.” (Junior doctor)

However, after implementation at least one consultant expressed negative views about involvement:

“There should have been more involvement from clinicians. A single biggest change we had in this ward in 30 years.” (Consultant)

Junior doctors noted the extra work they thought it often created—for example, transcribing drug charts of patients coming on to the ward (a task shared with pharmacists), the need for training as they rotated and queuing for access to terminals. Some doctors also looked for a greater involvement in the design of such systems, and felt that their participation in both design and implementation planning was essential as the only way to achieve really useful, usable and appropriate systems. Nevertheless, most doctors worked with the system, some more wholeheartedly than others, and at times they worked round the system by delegating direct use to nurses or pharmacists.

Nurses

Most nurses were hesitant about the system at the outset, and often feared letting go of familiar aspects of their job such as the bed‐end drug chart and the old‐style drug trolley. They also expressed a lack of knowledge about computers (some had not used a mouse before) as well as fear and resistance to computers becoming a more substantial part of their job and, potentially, coming between them and their patients. At one point during the early period of use the nurses on the ward came close to withdrawing their support for the system, feeling that it placed an intolerable load on them. Following this period, and in response to their protests, extra effort was put into system support.

More senior nurses, and those who had more information during the build up to the implementation by participating on the project team, were generally more positive. They saw the future in this system, with a better, more careful and error‐free regimen of care, with time saved becoming available for more creative nursing activity. They anticipated and kept faith with these promised benefits, even when the system was at its most problematic.

Pharmacists

The system was to a large degree driven by pharmacists. The project was led by the chief pharmacist, and at all stages the work practices and goals of pharmacy were strongly represented. Pharmacists interviewed before the system was in live use could immediately appreciate potential benefits—for example, more comprehensive prescription review, more legible orders with more coherent detail, better adherence to the hospital formulary.

Those who participated in the project directly felt that the implementation of the system had helped to build stronger relationships with other clinical staff, not least through the extended period of providing on‐call help. In contrast with doctors and nurses, pharmacists also had a clearer perception of additional benefits they could deliver to patients—for example, more time for patient contact, taking drug histories, discharge counselling. However, as indicated in the companion paper,¹³ time spent providing a ward pharmacy service increased. But despite this lack of release of time, pharmacists who used the system reported perceiving the system as safer and more convenient, with better access to patient data. Some worried that it could reduce patient contact.

Organisational context

The system was just one of many innovative pharmacy‐related projects being undertaken at the time. For instance, shortly after this system went live a new robot was installed in the central pharmacy. As a pilot project this system was undertaken with the understanding that what was being attempted was new, innovative and worthwhile, but it was demanding of extra resources and effort in technical and organisational design, dedicated space and individual commitment. The project was understood within the broader hospital as an opportunity to learn about this particular electronic prescribing system, and also about the potential of future such systems and issues of their implementation. The project was also seen as a more general opportunity for staff, in particular nurses and pharmacists, to become more knowledgeable about new technologies.

The system remained in use until January 2006 when the surgical ward relocated as part of a surgical modernisation programme. The system did not transfer but remained installed, in the original ward, now medicine for the elderly ward. Since January 2006, while not in use, the system has had a series of software upgrades and is currently scheduled to re‐enter operation soon. Thus the organisational outcome of this system is still being negotiated—judged as not worth moving with the surgical ward in the reorganisation, but to be put back in use in a different clinical context.

Discussion

We present here a structured, theory‐driven evaluation of a novel electronic prescribing and drug administration system on one surgical ward. The system may be seen as successful in that it remained in use for an extended period, that users of the system became increasingly positive about it (box 3), and that it was shown to reduce prescribing and drug administration errors.¹³ Drawing on this qualitative, and the quantitative parts, of the study, the outcome is summarised below:

• Doctors write more coherent and complete prescriptions and they are legible.

• Preparation for and conduct of a drug round is a more structured activity for a nurse with individual patients considered one by one and their medicines systematically collected together.

• The drug round is more structured with specific attention to patient identification through barcodes and a more comprehensive scheme for recording administration.

• Pharmacists review more prescriptions and detect more errors before they reach the patient.

• Overall time spent on medication‐related tasks increased.¹³

These are stated as generally positive outcomes, yet we need to ask how this experience can inform any wider‐scale deployment in the hospital or elsewhere.

Implementation processes

First, these results must be understood as deriving from a pilot project that drew on extra commitment and resources over a long period. If a similar system were more widely implemented this special status would not exist. Managers would need more confidence in the technical components and the functionality they support, and would need to recognise the effort required to resolve the inevitable challenges to existing work practices. Put differently, although the system continued to be developed on the ward, it had not, and we would argue could not, become a “finished product” that could be easily inserted into other hospital or ward environments.

Second, the findings from this study confirm that much of the real work of designing and shaping such systems, understood as socio‐technical ensembles, is undertaken in the local context and often after initial use, a process of “in‐use design”²² which creates a specific “technology in practice”.¹⁵ This is a necessary process that needs to be resourced over a lengthy period to allow for attitudes and opinions to shift, work processes to adapt and, in the case of a system focused on error prevention, formative evaluations to ensure that real existing errors are being reduced and new potential for error is not being created.

In this context it is interesting that doctors chose to have limited engagement with the preimplementation design and training, which may partly reflect the constraints of being a surgeon. However, they may have had fewer concerns about non‐involvement if they had realised that they still had a major role in designing and shaping the system as they used it and responded to its problems.

Technical systems structure work

The effectiveness of electronic prescribing is often seen as a consequence of the technical elements structuring and constraining work tasks, providing relevant forcing functions. For example, in this case, in loading the drug trolley a nurse is bound into one sequence of events: selection of a patient, then of their drugs, then loading trolley drawers and then reading wristbands to ensure administration to the right patient. Similarly a doctor prescribing is prompted to complete the entire medication order and is unable to omit key information. Such enforcement of standard practice, it is often argued, is one of the ways such systems can reduce errors.⁴

Such structuring imposes changes in practice for all those who work with the system. Most, but not all, are seen as beneficial and appropriate, but some aspects of work are then left fragmented and unresolved. Thus the example of loading a drug trolley above also reveals some fragmented work—splitting of the preparation of intravenous fluids, leaving untidy and potentially error‐prone aspects such as referral to paper charts. Similarly, prescribing or administration not undertaken as part of regular drug rounds, as when a nurse gives a “stat” (occasional or elective) dose, now required another work flow to log and administer, one in which the nurse walks to and from the computer twice, once to obtain the drug and then again to record administration. In dealing with the aspects of the system that are not readily structured, new sources of error might be introduced.

Dilemma of structure

The structuring thus poses a dilemma.²³ In addressing the aim of reducing errors in prescribing and administration of drugs, the technology is explicitly used to constrain and enforce a “good” process. But some aspects of practice do not neatly fit, or are mutually incompatible, and take time to work through or around. The structuring may also create wider rigidity, making it harder to introduce other innovations related to medicines management in the future—for example, one‐stop dispensing.²⁴ Such dilemmas are apparent on a single ward, but across a hospital or range of hospitals the effect would be greatly magnified and each setting would reveal its own misfits and fragments.

Consequential change

Reshaped work tasks bring wider changes in how care is delivered and professionals work together. For example, previously pharmacists visited each patient daily and checked their drug chart if available; now they can check through the computer and assess each patient's computer chart for changes, then only visit those whose records indicate a pharmacy‐related problem. The patient is seen through the data, possibly remote from the ward, and there may be fewer opportunities for interaction between staff, and between pharmacist and patient. Is this an appropriate outcome and what will be its consequences? Such emergent changes must be identified over time, assessed and managed when a system is in use.

Another example was the loss of the drug chart as a ready‐to‐hand update of the patient's condition, located at the end of the bed. The information given on the chart, both explicit (such as drugs and doses) and implicit (crossings off, the chart looking battered, etc.), was lost from that location. In a narrow sense, all the explicit data and functions were translated into the new system and even improved on (eg, legibility, completeness), but there remained a feeling across the user groups that an important element of their work practice and of the efficiency and integrity of the prescribing process within wider healthcare had been lost. At the same time the computer system did increase the visibility of some errors, such as missed doses. This increase in information was so noticeable that one consultant thought the computer system had led to a marked increase in missed doses, claimed the computer was unsafe and demanded it be taken out. It was only the data from the quantitative study that convinced him the number of missed doses had actually dropped— they had just been invisible before.

Strengths and limitations of the study

Our use of a wider socio‐technical evaluation framework gives a better understanding of the quantitative data in the companion paper¹³ and the extent to which staff attitudes and behaviours play a part in the final performance of the system studied. It can offer others who are considering adopting similar systems a better understanding of the relevance of our findings to their setting and of issues arising during an implementation. It also helps clarify directions of development for the suppliers of such systems and emphasises the vital need for them to be locally configurable. Finally, this approach ensures that if a system fails to be implemented successfully, the evaluation framework still provides understanding and learning.

This type of qualitative evaluation has limitations. It can offer no guarantees of universal relevance of findings. The attitudes and skills of staff, as well as the existing systems of work, will be different at other sites. Thus the particular socio‐technical accommodations worked out may be different.

Conclusions

This theory‐led evaluation, combining quantitative and qualitative methods, offers valuable insights into a critical contemporary policy area. It reminds policy makers and implementers that acquiring a technology is the start of the process, not the end. Technical systems are never perfect, and they will require time and effort to become embedded into any particular clinical context. It is notable that most of the successful computerised prescribing systems in hospitals are systems developed “in‐house”, a situation in which the relationship between the technology and the users has the greatest potential for mutually beneficial interaction.⁵,⁶

More fundamentally, this study reminds us that, unlike a medicine which has relatively fixed properties, the effectiveness of ICT changes and develops over time, and may have quite different effects in different settings. For this reason a sophisticated evaluation framework, such as the one we have used, is necessary. At a time when there is unprecedented planned expansion of ICT as a method of collecting information and reducing error, it is important that these properties of ICT are recognised and evaluations take them into account.

Abbreviations

ICT - information and communication technology