Implementation and adoption of nationwide electronic health records in secondary care in England: final qualitative results from prospective national evaluation in “early adopter” hospitals

Abstract

Objectives To evaluate the implementation and adoption of the NHS detailed care records service in “early adopter” hospitals in England.

Design Theoretically informed, longitudinal qualitative evaluation based on case studies.

Setting 12 “early adopter” NHS acute hospitals and specialist care settings studied over two and a half years.

Data sources Data were collected through in depth interviews, observations, and relevant documents relating directly to case study sites and to wider national developments that were perceived to impact on the implementation strategy. Data were thematically analysed, initially within and then across cases. The dataset consisted of 431 semistructured interviews with key stakeholders, including hospital staff, developers, and governmental stakeholders; 590 hours of observations of strategic meetings and use of the software in context; 334 sets of notes from observations, researchers’ field notes, and notes from national conferences; 809 NHS documents; and 58 regional and national documents.

Results Implementation has proceeded more slowly, with a narrower scope and substantially less clinical functionality than was originally planned. The national strategy had considerable local consequences (summarised under five key themes), and wider national developments impacted heavily on implementation and adoption. More specifically, delays related to unrealistic expectations about the capabilities of systems; the time needed to build, configure, and customise the software; the work needed to ensure that systems were supporting provision of care; and the needs of end users for training and support. Other factors hampering progress included the changing milieu of NHS policy and priorities; repeatedly renegotiated national contracts; different stages of development of diverse NHS care records service systems; and a complex communication process between different stakeholders, along with contractual arrangements that largely excluded NHS providers. There was early evidence that deploying systems resulted in important learning within and between organisations and the development of relevant competencies within NHS hospitals.

Conclusions Implementation of the NHS Care Records Service in “early adopter” sites proved time consuming and challenging, with as yet limited discernible benefits for clinicians and no clear advantages for patients. Although our results might not be directly transferable to later adopting sites because the functionalities we evaluated were new and untried in the English context, they shed light on the processes involved in implementing major new systems. The move to increased local decision making that we advocated based on our interim analysis has been pursued and welcomed by the NHS, but it is important that policymakers do not lose sight of the overall goal of an integrated interoperable solution.

Introduction

England’s National Programme for IT was, from its outset, an ambitious effort aiming to introduce national electronic health records across NHS care providers throughout the country (see appendix 1 on bmj.com).^{1 2 3 4 5 6} It is distinguished by its scale, unprecedented levels of investment, complexity of systems, centrally driven delivery model, and extremely challenging timelines.^{7 8 9 10} The English endeavour is one of the few sustained attempts to implement electronic health records—described by a recent Minister of Health as “the jewel in the crown of the NPfIT”—on a truly national scale.¹¹ A brief history of the National Programme is given in appendix 2 on bmj.com.

Building on our formative work in five acute and mental health hospitals,¹² we present our summative findings from longitudinal qualitative work conducted over a longer period of time in a broader range of “early adopter” sites. We examined the local consequences of the centralised implementation and adoption strategy in the light of broader national developments to inform the ongoing implementation strategy. Findings from our related quantitative work will be reported in due course.¹³ Our work and the closely related evaluation of the summary care record systems are, we believe, the first attempts to prospectively study the implementation of electronic health record systems on a substantial scale.^{14 15} This report is timely as the future direction of NHS IT—and specifically the NHS Care Records Service—is still highly uncertain. At the time of writing, the public accounts committee has published its updated assessment and the National Audit Office has published its third review of the National Programme for IT concluding that “progress with the delivery of care records systems continues to fall well below expectations.”^{16 17} A public consultation on future NHS IT policy in England has taken place, with an announcement on the ultimate outcome of the National Programme expected in late 2011.¹⁸

Methods

Ethics and governance

Our research was reviewed by an NHS ethics committee and classified as a service evaluation. We obtained informed consent from participating hospitals and individuals, while also complying with local governance requirements. All data were anonymised and care has been taken to ensure that the data presented here are neither attributable to participating sites nor to individuals to protect anonymity.

Design

We conducted a prospective longitudinal and real time evaluation of the introduction of the NHS Care Records Service in hospitals and specialist community care settings (henceforth referred to as hospitals) over a 30 month period from September 2008 to February 2011.¹³ We collected a broad range of qualitative data from “early adopter” hospitals committed to taking one of the three core software systems—Lorenzo Regional Care, Cerner Millennium, and RiO.¹² We use the term “early adopter” in a broad sense to refer to hospitals that were among the first to receive these systems as part of the National Programme. Our evaluation drew on sociotechnical principles.¹⁹ The nature of sociotechnical evaluations has been discussed at length in our interim paper, but this in essence refers to the close and in some instances virtually inseparable relation between organisational/professional and technical considerations. Use of this theoretical approach helped to minimise the risk that we focused unduly (or indeed neglected) either the organisational/professional or technical dimensions of implementation efforts.

We conceptualised participating hospitals as individual case study sites to reflect the importance of local contingencies, using this detailed understanding of local considerations as a platform from which to undertake a cross site synthesis and generate insights that could be transferable to other hospitals.^{20 21 22 23} In doing so, we developed each case study in a way that allowed the specific character of the implementation and adoption of the software in each site to be revealed. This involved the analysis of data obtained from individual hospitals and writing of detailed individual case study reports describing local characteristics and developments over time. Data were collected and analysed without a constraining overarching framework, which enabled us to capture the diversity of experiences and the influence of the local environment on implementation efforts. Case studies were then considered in a cross case comparison to find common and contrasting themes between hospitals but also alternative and distinct experiences. Cross case comparisons were followed by integration with additional data obtained that did not directly relate to hospitals themselves but to the wider environment (such as policy documents, interviews with system developers, etc).

Sampling

Designated lead researchers were assigned to individual sites and were responsible for recruitment of hospitals and individual participants. We selected hospitals using purposive sampling to identify diverse organisations (teaching versus non-teaching; more autonomous versus less autonomous; and acute versus mental health settings) across the three geographical implementation areas (North Midlands and East, London, and Southern England) and the three centrally procured NHS Care Records Service applications (table).^{24 25 26} This was achieved by drawing up a list of hospitals planning to implement different functionality during the course of our evaluation and using maximum variability sampling to identify sites to approach.

Summary of recruited hospitals in study of progress of implementation of nationwide electronic health records in secondary care in England

Site	NHS Care Records Service application	Type of hospital	Teaching status	Foundation status*	Location	No of beds	Annual turnover
A	Cerner Millennium	Acute	Teaching	Non-foundation (less autonomous)	Urban	1000-1499	£800-1000m
B	Lorenzo Regional Care (Release 1 and 1.9)	Acute	Non-teaching	Foundation (more autonomous)	Rural	1000-1499	£200-399m
C	Lorenzo Regional Care (Release 1)	Acute	Teaching	Foundation (more autonomous)	Mixed urban	1000-1499	£200-399m
D	Cerner Millennium	Acute	Non-teaching	Non-foundation (less autonomous)	Urban	500-999	£200-399m
E	Cerner Millennium	Acute	Teaching	Non-foundation (less autonomous)	Urban	500-999	£400-599m
G	RiO (4/5)	Mental health	Teaching	Non-foundation (less autonomous)	Urban	500-999	£200-399m
H	Lorenzo Regional Care (Release 1)	Community	Non-teaching	Non-foundation (less autonomous)	Urban	N/A	<£200m
M	RiO (5.1)	Mental health	Teaching	Foundation (more autonomous)	Urban	Mostly outpatients but <500 inpatient beds	<£200m
P	Originally planned to implement Cerner Millennium	Acute	Teaching	Foundation (more autonomous)	Mixed urban	1000-1499	£400-599m
Q	Lorenzo Regional Care (Release 1)	Mental health	Non-teaching	Foundation (more autonomous)	Rural	Mostly outpatients but <500 inpatient beds	<£200m
R	Implemented Cerner Millennnium R.0 before going back to their patient administration system	Acute	Non-teaching	Non-foundation (less autonomous)	Mixed urban	500-999	No information available, hospital recently merged
X	Lorenzo Regional Care (Release 1)	Acute	Non-teaching	Non-foundation (less autonomous)	Rural	<500	<£200m

*Foundation status indicates level of autonomy of NHS organisation (www.monitor-nhsft.gov.uk/sites/default/files/publications/Guide_for_applicants_Nov2008.pdf).

Within each of the case studies, we purposefully recruited a diverse range of interviewees, actively seeking the broadest range of perspectives.^{27 28} As case studies varied in relation to local arrangements and preferences, approaches to recruitment of participants were flexible, being negotiated with key contacts and gatekeepers at each hospital.

Individual informants were initially recruited through recommendations of managers and, in some cases, also approached directly during site visits, based on recommendations of previous interviewees. Our interviewees included junior and senior hospital managers, implementation team members and IT staff, junior and senior doctors and nurses, allied health professionals, administrative staff, and, when appropriate and possible, patients and carers.

We also purposively sampled key stakeholders outside the immediate environment of implementation to understand the wider contextual landscape as this was found to play a more central role in influencing local implementation than we had originally envisaged.²⁹ These informants were identified through our existing contacts within NHS Connecting for Health, approached at relevant national conferences, or approached directly based on recommendations from case study sites. These were individuals who had an active interest in the implementation and adoption of relevant systems, including civil servants, staff from NHS Connecting for Health, strategic health authorities, local service providers, system developers, and relevant independent sector representatives.

Settings

We initially collected data from 17 different locations, from which we identified 12 sites that satisfied our inclusion criteria for detailed case studies—that is, hospitals that had either begun implementing NHS Care Records Service software or were planning to do so during our evaluation period. These 12 sites therefore represented the focus of our efforts.

Data collection and handling

We conducted semistructured interviews with a range of key stakeholders and, in most cases, these were audio recorded and transcribed. Some interviews were not recorded, in keeping with participants’ wishes. Their concerns about being recorded possibly reflected the politically and commercially sensitive nature of the research environment (in such cases researchers took notes).^{30 31 32 33} Topic guides were tailored to interviewees’ roles and emerging issues but essentially these were designed to explore participants’ attitudes, challenges encountered, and proposed potential solutions to these. Sample topic guides can be viewed in appendix 3 on bmj.com.

Interviews were complemented by researchers’ field notes describing experiences during data collection and noting down early findings, as well as observational and documentary data from hospitals, national documents, meetings with governmental stakeholders, and conferences. Observations were conducted in each case study site. This included researchers shadowing user related activities or strategic meetings, or both. Potentially relevant local and national documents were identified through various routes, including discussions with interviewees, contacts at NHS Connecting for Health, and proactive searches for relevant governmental, media, hospital, and academic reports. These documents were selected on the basis of providing an insight into contexts, planned changes relating to the implementation, and early lessons learnt.

When possible and relevant, to trace changes over time, data collection at each hospital took place in two phases (time 1 and time 2) with a gap of six to nine months in between. In some cases, however, division into two phases was not possible or appropriate. Throughout the study, emerging qualitative findings were fed back to individual participating hospitals, NHS Connecting for Health, and our funders through formative feedback sessions.¹² The discussions that ensued informed subsequent data collection and were taken into account in the final analysis.

Data analysis

Lead researchers (AR, AT, DP, KC, SC, TC, VL, and ZM) collected data in individual hospitals and also led on the thematic analysis of individual case studies.^{20 21 22 23} Regular analysis workshops with the wider team helped us to validate findings from individual case studies and to integrate multiple case studies to draw out transferable findings. Adopting a sequential or iterative analysis enabled us to refine questions, develop and challenge assumptions, and pursue emerging avenues of inquiry in later data gathering.^{21 34} We continued data collection until saturation was reached (that is, no new rich diverse data relevant to the evaluation emerged for the duration we were in the field). This was partly influenced by factors related to the setting, such as the scale of the deployment at each site (for example, limited to a ward or hospital-wide) and type of functionalities being introduced (for example, software modules for ordering of clinical tests or clinical notes).

In analysing data from case studies, we combined deductive thematic coding informed by sociotechnical principles¹² and inductive coding that allowed themes to emerge from the data.^{19 26} This process involved immersion in the data, which was achieved by repeated reading of interview transcripts, discussion among team members, development of provisional analytic categories/themes informed by our theoretical lens and the wider literature, and iterative refinement of these categories using the constant comparative method (comparing our analysis with new data as these emerged). Our overall understanding of the implementation of systems was one that balanced the delivery of the technology (specific software functionality, computers, networks, databases, training manuals, etc) undertaken by the local service providers and software suppliers with the hospitals’ role of integrating this into their operations—what we termed adoption. To understand these adoption issues, analysis drew on notions of “working out” and “changing.”¹⁹ This approach helped with investigating the extended processes of change over time and the ways in which users of new systems and the organisations in which they were based worked out how to accommodate the technology into their work practices and the processes through which they delivered patients’ care.

Results

The table in appendix 4 on bmj.com provides a detailed profile of our case study sites. Our complete dataset comprised 431 semistructured interviews, 590 hours of observations, 234 sets of notes from observations, researcher field notes, and conferences, 809 NHS documents, and 58 national and regional documents.

The interviews were conducted with healthcare professionals (41%); hospital managers and administrative staff (26%); hospital based IT implementation leads (18%); staff from NHS Connecting for Health (8%); local service provider staff (2%); patients and carers (2%); and a range of other relevant stakeholders (3%).

Our longitudinal approach allowed us to differentiate between more isolated transitory challenges and those that were more overarching or persistent, or both. We were therefore able, for example, to understand how local deployments in sites were influenced by wider contextual factors, the impact of which intensified over the period of our evaluation (summarised in box 1). We also developed a detailed understanding of the local challenges of implementation of NHS Care Records Service systems and the range of consequences that followed. Depending on the system in question (described in more detail in box 2), and often heavily influenced by organisational history and developments over time, consequences were identified relating to individual work practices and organisational functioning.

Box 1: Continuing and intensified impact of external context on local deployments—key emerging issues

Contractual tensions

• Perceived to inhibit local choice and decision making, with these arrangements becoming more complex and far reaching as time progressed

• Resulting in an increasingly strained relationship between stakeholders, slowly eroding good will on all parts and inhibiting progress

Media portrayal and impact

• Perceived to have contributed considerably to a negative public perception of the National Programme as a whole by an unremitting focus on negative aspects such as delays, costs, and problems occurring during implementations

• Some implementations were played out in the press as exemplar sites, with their success assumed to either “make or break” the National Programme as a whole and associated software systems, local service providers, and developers

Box 2: Three national electronic health record systems implemented as part of National Programme

Cerner Millennium

• Commercial “off the shelf” software supplied by Cerner, developed in the United States and provided by BT

• Offered to secondary healthcare settings in London and the South of England

• Had to be configured to English healthcare settings

• Initially sites had limited customisability but more choice to adjust functionalities was given in 2010 under the “New Delivery Model”

• Planned to be delivered in four different sequential releases that built on each other with progressively richer functionalities

• Hospitals followed a “big bang” approach to implementations

• By December 2010 Cerner Millennium was live in six out of 32 acute hospitals in London and in nine out of 40 acute hospitals in the South of England

RiO

• Commercial “off the shelf” web based software developed in the United Kingdom by CSE Healthcare (formerly CSE-Servelec)

• Provided by BT to mental health hospitals, community organisations, and primary care settings

• Offered to sites in London and the South of England

• RiO was planned to be deployed in four waves starting with version 4.0 or 5.0 and ending with version 7

• Problems encountered during upgrades from version 4.0 to 5.1 led to temporary suspension of deployments, which resumed by 2010

• Configured to meet the individual needs of mental health settings

• By December 2010, RiO was live in 17/45 community and mental health hospitals in the South, in 8/10 mental health hospitals, and in 30/31 primary care settings in London

Lorenzo Regional Care

• Web based software solution offered to hospitals located in North, Midlands, and East of England

• Supplied by iSOFT, developed in India and provided by CSC Alliance

• Not “off the shelf,” but a growing product intended to be developed in line with requirements of hospitals

• Implementation of Lorenzo Regional Care was stepwise (paper and computer systems were gradually replaced with electronic systems in selected parts of hospitals initially)

• Supplied in four different releases with increasing functionality

• Common characteristic of all implementations was the small scale and scope

• By December 2010, limited Lorenzo Regional Care functionality was implemented in one mental health hospital, two community hospitals, and three acute hospitals

As each implementation was different—different organisations, of different sizes in different geographical areas, and with different legacy software systems, IT infrastructures, skill mixes, employee relations, work processes, histories, visions for change and technology deployed—some findings (unsurprisingly) varied across sites (see appendix 4 on bmj.com). As a result, there were distinct stories of local working out (see box 3 for these in relation to different systems). These detailed stories will be reported in more detail in due course, but in this paper we focus on the more overarching themes identified across sites.

Box 3: Three national electronic health record systems and different ways of working them out

Cerner Millennium

• Hospitals could tailor Cerner Millennium to their respective settings within the constraints of the contract

• Cerner Millennium sites initially required a rewriting of the patient administration system and then made adjustments to the final product

• During configuration Cerner Millennium was tested and any issues that were raised were prioritised by the hospitals before being reported to the local service provider to be fixed

• The New Delivery Model allowed hospitals to configure the product to their needs. Co-location was perceived as an important factor for accomplishing joined up configuration, avoiding bottlenecks in the supply chain and dealing with failures directly. This process also gave sites a sense of control over configuration, though the local service provider would still play an active role by mediating between hospitals and Cerner

• One hospital implemented a version of Cerner Millennium configured by the University of Pittsburgh Medical Centre with a view to anglicising the software so that it represented their clinical pathways and reflected their lines of accountability and hierarchy

RiO

• Hospitals could tailor RiO to their respective settings within the constraints of the contract

• Some argued that RiO did initially not support reporting on performance adequately and thus brought in people to help develop this functionality

• RiO was implemented across all mental health hospitals in London, bringing several benefits

Lorenzo Regional Care

• Lorenzo Regional Care had to be substantially developed and redesigned by sites to meet their needs and support clinical processes

• Sites would get new builds of the system on a regular basis in the testing environment before they went to the live environment. During this process they collected any issues, which were then prioritised, being kept by each site in a log and managed in collaboration with the government and local service providers. These issues would be reported to the Computer Sciences Corporation, which would then report them to iSOFT to be fixed

• To facilitate the process, some project team members travelled to Chennai and worked together with the developers. iSOFT and the local service provider set up regular web-conference meetings with sites during which the software was demonstrated from Chennai and sites were able to comment and provide feedback before any change was made to the code

• Lorenzo Regional Care was, since the point of its adoption, under continuous development, which adversely impacted on hospitals’ ability to plan for anticipated changes

• Despite the fact that Lorenzo Regional Care was one system, its design and functionalities were different in the different settings. While the few healthcare professionals who used Lorenzo Regional Care made it work for them and were happy with the system, other users continued to experience frustrations

Some aspects of our findings confirmed and strengthened the themes reported in our interim paper and are also in line with previous findings from the literature.^{1 14 15 29 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83} Rather than reiterate these confirmatory themes here, we have focused on several novel findings, particularly those of national and international interest. We present these, illustrated by data that have been selected on the basis of representativeness and descriptive power.¹³ Further supporting data are available on request from the corresponding author. Our findings are organised along the two key themes (with other themes and subthemes detailed in box 4): local consequences of the national strategy and the national implementation landscape.

Box 4: New emerging themes and subthemes: local consequences of implementation of NHS Care Records Service systems

Translations of the vision

• The NHS Care Records Service as a multifaceted concept embedding various visions

The arrival of the NHS Care Records Service in institutional settings

• Ways in which hospitals prepared themselves for the NHS Care Records Service

  • Hospitals had different characteristics and varied histories, but reasons for being an “early adopter” were often similar

  • Management and technical infrastructures varied substantially

  • Approaches and methodologies for implementation strategies varied across hospitals

  • Each hospital organised its own training to tailor the service to the local systems and the needs of users

  • Sufficient resources were necessary for the timely implementation of the NHS Care Records Service

• Concerns related to the arrival of the NHS Care Records Service in institutional settings

  • Delays in organisational readiness because of differentiation within each hospital

  • Parallel running of other initiatives and projects

  • Implementation dissociated from actual practice

  • Complex supply and management chain

  • Changing NHS policies

• Implementation as a process

  • Made to work through the mediation of several people and technologies

  • Distinction between implementation and adoption was blurred

  • The software systems were being put into use, used, and adapted and back to being “implemented”—a cyclical process of growth

Standardisation versus localisation

• Different software systems were modified in different ways

• Concerns included: standardisation versus localisation, lack of knowledge about the product, the English NHS, and clinical work

• Complex supply chains

• Involvement of commercial organisations

Some early benefits for staff

• Enhanced availability of data and data management tools

• Multidisciplinary teams sharing a patient’s care on site or across sites, transfers of care especially with referrals/discharges or requests for investigations/reports

• Learning that had taken place

  • Learning about managing and implementing large scale IT led organisational change projects

  • Learning to use IT to support organisational and healthcare goals

Work processes and changes in work practices

• Changes to clinical and administrative work processes for patient care

  • Various users and reasons to use the systems

  • Processes of adaptation, compensating, workarounds

  • Changes to sequencing in recording clinical notes

  • Redistribution of work and time for patient care

  • Flexibility and mobility of work

  • Quality of work life

• Professional identity, peer relations, and engagement with the technology

• Changes to local management processes

• Managing with real time data

  • Making work visible

Overall developments in the face of continuing political, economic, and policy uncertainty

• Implementation progress has been slow

• Shifting political and economic climate and progressive move from the initial “top down” implementation model to an increase in local involvement in decision making

• Lack of sustained leadership and lack of certainty for hospitals yet to implement

• Danger that work invested will not be utilised

• Risk of reverting back to patchy implementations of the main patient administration systems with limited clinical functionality and poor interoperability

Local consequences of the national strategy

Implementing NHS Care Records Service systems on a national scale was an extremely complex activity, with the potential to impact on large numbers of clinical, managerial, and administrative staff with different needs, expectations, and experiences. During this process, staff had to learn and work out the consequences of such systems day by day, and this learning is likely to continue for many years to come as further functionalities are introduced and as the systems become integrated within the often taken for granted practices of the hospital.

Multiple translations of the vision

Within our case study sites, we found that there were many different translations of the overall vision of NHS Care Records Service systems among various stakeholders. These ranged from those that focused on managing patients’ data (such as data capture, storage, and sharing) to changing organisational arrangements and workflows (such as business process change) to more policy related aims (such as modernisation, shift to patient focus). For example, some held a business driven view that emphasised standardised practice at the expense of innovation in workflow, clinical practice, interorganisational arrangements, or management and policy ambitions:

“Ideal is that the NHS will become standardised, so the way in which we interact with computers and the way in which we interact with patients will become standardised” (interview, IT professional, site C).

Such views clashed with those who saw electronic health records as being a way to achieve more patient centred healthcare:

“it’s the patients that hold the record and the patient should control who has access to it . . . Give the patient the record and give the patient the key to unlock it. They are partly responsible for the record themselves” (interview, healthcare professional, site R).

We found little evidence of efforts to align these perspectives to aid the process of working towards agreed goals. The persistence of multiple visions thus possibly reflected limited or ineffective communication channels to maintain or indeed refine or update the original vision.

The arrival of the NHS Care Records Service in institutional settings

Various approaches were taken by local organisations to prepare for implementation, and several internal and external factors shaped the different implementation strategies pursued. These included different levels of maturity of NHS Care Records Service systems, concurrent changes occurring in sites (for example, working to achieve greater autonomy from the Department of Health), changes in the National Programme as a whole (for example, contract renegotiations), and in NHS policies and targets (for example, financial savings and restructuring of commissioning models), all adding uncertainties and delays to the process.

“I think people get a bit, is it worth it? Is it worth me continuing? Should I put the effort in?” (interview, IT professional, site C).

Most of the hospitals we studied were “early adopters.” This meant that all concerned (that is, NHS Connecting for Health, suppliers, local service providers, and hospitals) were encountering problems for the first time. These often involved the challenges inherent in putting software into practice and related problems with connectivity, usability, training, data quality and migration, and system downtime.

The complex supply chains and convoluted communication processes between hospitals, local service providers, software suppliers, and NHS Connecting for Health that we previously described, persisted over time.³⁸ The tensions relating to contracts often led to a rigid focus on a limited set of “deliverables,” thereby hindering any attempts at fostering local ownership or meaningful engagement with NHS staff. There was consequently a lack of attention to more productive deliberations that might have helped to overcome the many challenges that were (inevitably) encountered. Hospitals, as the “client” or “problem owner,” were hampered by a lack of budgetary control, lack of information about contractual arrangements, and lack of ability to configure the software (constrained by contractual clauses). Nor could they effectively engage in direct communication with the software supplier. The primacy of these commercial relationships often meant that participants suggested that contracts were focused on the delivery of the product rather than on its quality, the process of delivery, achieving meaningful use, and wider consequences of its implementation:

“I think it’s always very difficult when you involve a commercial company with a public service, because a commercial company will always be driven by profit and the money that they are making. Maybe as things get critical the quality of what’s delivered becomes a secondary issue” (interview, healthcare professional, site C).

This led at times to a gradual disengagement by hospital staff. Also contributing to disengagement was the communications cycle between user and developer that was often extended and fragmented. The potential for imaginative or pragmatic problem solving furthermore clashed with the “top down” approach governed by software contracts and formal processes of requirement specification.

“The milestones in the plan were set as a contractual milestone so we weren’t allowed to alter those. What was quite difficult was we had to work backwards from those milestones . . . milestones that were set were probably going to be unachievable, but we had to work within the constraints of that contract” (interview, IT professional, site R).

“it takes much longer to do anything than you think it’s going to take and there’s so many people involved, so many committees involved to get anything done at the supply side that it takes a long time to get things sorted and that’s unfortunate” (interview, IT professional, site H).

Standardisation versus localisation

All sites adopting the software systems faced trade-offs between standardisation and localisation. Administrative, technical, and clinical users interviewed were often aware of this tension and of the need to balance the requirements of individual organisations and the NHS more generally. Thus, some were concerned that the more customised the software became the more distant it would get from its centrally defined purposes:

“If you keep giving people the ability to localise things you kind of drive away from a centralised understanding” (interview, healthcare professional, site C).

Assumptions inscribed into NHS Care Records Service systems as to how the English NHS operated (or should operate) were often challenged. The system was often perceived by hospital staff to reflect developers’ lack of understanding of their clinical processes, resulting in systems that were often seen as linear and homogeneous, and unable to fit to the actual complexities of clinical practice:

“What was delivered was a clumsy system that seems to have been designed for one clinician who has clinics booked up in advance that uniquely come in and everybody who comes shows up or maybe they don’t show up. There is nothing more complicated than that” (interview, healthcare professional, site M).

The complex supply chains added tortuous bottlenecks in resolving such issues. In line with this, some participants reported that the software they were implementing was “inflexible,” that they “had to push hard for every single change in the system,” and that “finding solutions seemed to be long winded and difficult” (interview, healthcare professional, site R).

Some early benefits for staff

The challenging experiences described above did not necessarily invalidate the case for electronic health records. Enhanced availability of data and data management tools were perceived as benefits by managers and some clinical users when information was legible, available in “real time,” more easily searchable and retrievable (such as for management purposes), and accessible “any time” and “anywhere” by multiple concurrent users. The team working that exemplifies community mental health provided an example of where sharing of data brought clear benefits.

“The main thing really is that we can read people’s writing. That was a big thing before that you couldn’t actually read what people were writing in the NHS across the board. Now we can read everybody’s writing. That is a major thing. And people I think forget that over time. You quickly forget the bad old days of not being able to read what somebody has written” (interview, healthcare professional, site M).

“It’s good for performance management as well, so you can go back to clinicians and go OK never mind how many patients you saw, this is your risk level of your case load, this is the risk level of somebody else’s caseload, look at the difference? Why do you think that might be, and you can also look at numbers, you know, this is the number of letters that you’ve sent to GPs [general practitioners] how come so and so sends this many letters and you only do this many letters, you know, because you’ve got absolutely everything there” (interview, IT professional, site H).

These positive developments really materialised only after a system had achieved a critical mass of users and data. Initially, there was often seen to be a need to “feed the beast” (interview, healthcare professional, site H), getting little, if anything, in return.

Electronic transmission of information was also reported as making some user workflows faster overall, although individual stages of these workflows could become more time consuming than the previous system—this was, for example, the case if the data input screen required additional or different data from those that were customarily recorded. In such cases, staff often understood that the extra investment of time on their part served an important wider organisational function, but in the context of already heavy clinical or administrative workloads investing this additional time still often rankled.

More generally, our data, drawn from multiple user and managerial communities across sites, suggested that the upheaval associated with implementing these systems had given the opportunity for organisational learning and reflection. Staff were often engaged with the problem of getting the best out of the new system and in exploring its possibilities. As one IT professional stated:

“There are a lot of experienced people now that understand what this type of change means to the NHS and how to help them to make that happen that I think you wouldn’t want to lose that” (interview, IT professional, site D).

Work processes and changes in work practices

We found that the NHS Care Records Service was usually portrayed by planners and implementers as a set of systems designed primarily for clinical users, but that the main users of the software systems, at least in the early phases, were often allied health professionals and administrative staff. Their interests and concerns, however, seemed less likely to be elicited, understood, or acted on as implementations went forward.

“Well you feel like saying if it’s not working well let’s go home shall we, cause you just feel why have I bothered to come into work, you would come in before and you would think well I’ve got three clinics there that day and you’re just thinking well I knew I could get three of them typed you’re lucky now if you can get one typed in a day. I said at one of the meetings well when are you going to start employing some extra staff if you want the work done instead of cutting back on us, which is what they’re doing because now with the system work’s just all piled up and they’re just giving us a system, we don’t have any cover, work piled up for two weeks, and the system is taking two or three times as long” (interview, administrative staff, site B).

Senior clinicians were often (particularly in the early stages of the implementation) less likely to be affected directly by the system on a daily basis than their more junior colleagues (though this to an extent depended on the system in question and the specialty). For instance, a consultant psychiatrist explained that junior doctors would type clinical information into the electronic health record on their behalf:

“I have a team. I probably use [RiO] less than 10% of the time, because if I’m seeing patients in a ward setting, it would be my junior doctor that’s inputting the information” (interview, healthcare professional, site M).

Over time, the software systems were, in most cases, made to work better and more in synchrony with the local practices of delivery of care. Users, despite the problems they faced, were with effort able to “reconfigure” the software or reconfigure their way of working. In this way, their early frustrations were in some cases reduced. Thus a process of adaptation and adoption into the complex settings in which the software systems were implemented occurred. If this adjustment process was not possible, however, staff were obliged to accommodate the idiosyncrasies of the technology (for example, by using other systems, adopting compensating behaviours, or by partial use). Getting to know the limits of the system, users learnt to prepare and compensate. In doing so, they “creatively” devised strategies to overcome usability issues, such as taking screenshots of the just typed notes when the system “froze” and then printing it to add it to the paper file to avoid having to re-do time consuming work. This also, for example, involved leaving identity authentication cards in computer terminals to avoid lengthy log-in processes, as illustrated by the following researcher observation note:

10.45am: Student is now seeing a patient (old lady) in cubicle 1, chatting “How are you today? How are you getting on with your cream? Are you diabetic? Do you have thyroid problems?”; the system is still running on the laptop and the SmartCard [an identity authentication card] is in it while [the healthcare professional] is still out of the room.

The need to “trick the system” to overcome constraints (such as mandated fields or screens unfit for a specific clinical activity) and to “get the job done” or absence of the right clinical code in a drop down menu could result in issues further down the line—for instance, impacting on data quality or management’s ability to monitor activity levels.

Systems also led to a redistribution of work, with in some cases, clinicians doing more of the data entry that would otherwise be done by “typists” or “data entry” clerks. As a result, some participants reported that constant use of computers was “not really what [they] signed up for,” and other interviewees argued that NHS Care Records Service software was undermining their professional standing by forcing them to undertake administrative tasks:

“Especially when it started for the first few months it was very much, we felt like IT people, we felt like admin people instead of actual clinicians because we were spending more time with this system than we were actually with the patient” (interview, healthcare professional, site H).

This was exacerbated when data entry on the computer took (or seemed to take) longer than on paper, which put additional pressure on users:

“All our doctors and nurses are having to work harder now, because we are having to see the same number of patients with less time, because you are spending more time on a computer now and we have got no more doctor or nursing resources to do that” (interview, healthcare professional, site D).

Overall, work practices did not become “paperless” or even “paper light”: note taking while with the patient was still most often done on paper—sometimes on the back of scraps of used paper—with data entry to the computer systems done retrospectively (though this depended again on the maturity of the system in question). A major change in work practices expected by many at the outset, indeed a part of the vision, was concurrent entry of clinical information on the system at the time the activity took place (such as when consulting with a patient). For instance, in one hospital, the intention of management was for:

“Staff to update the record in real-time, so that the NHS Care Records Service became accepted as a normal part of their work” (interview, IT professional, site D).

In most hospitals, however, clinicians did not enter data in the system while they were with the patients (either at bedside, during ward rounds, or during outpatient clinics). This changed to some extent over time and varied between settings, but the example of mental health patients presenting in emergency departments (which had target times within which all patients needed to be seen) was particularly telling:

“I think there is a big issue for junior doctors out of hours and the nightshift in accident and emergency because the psychiatric assessments are quite lengthy and there is quite a lot of notes that go with it. What they usually do while they are in with the patient is, they make the notes as they go along and they are the record. They’ve raised concerns that they will be in with the patient and they are then going to have to come and type those notes up. They are not going to be able to do it while they are with the patient, because of issues like risk. These are patients that are really quite disturbed. You can’t kind of be faffing around getting them by computers. So it’s going to increase the time spent and you are then delayed seeing the next patient, that’s going to impact on the breaches of accident and emergency which is I think the big anxiety” (interview, healthcare professional, site M).

The national implementation landscape

Overall, progress with implementing the NHS Care Records Service has been much slower than anticipated, with little implementation of clinical applications. As of December 2010—by which time the implementation should have been completed¹⁰—8/219 hospitals (4%) were live with limited Lorenzo Regional Care functionality in the North Midlands and Eastern area of England; in the South 17/45 (38%) community and mental health hospitals were live with RiO and 9/40 acute hospitals (23%) were live with Cerner Millennium; and in London 6/32 acute hospitals (19%) were live with Cerner Millennium software, while RiO was being used by 8/10 (80%) mental health hospitals and 30/31 primary care settings (97%). Altogether, of the 377 sites in which implementation should have taken place, 78 (21%) had begun the process of implementing a variant of the NHS Care Records Service. It should, however, be noted that the figures for implementation of the NHS Care Records Service slightly underestimate the level of computerisation of the patient record across England as a few hospitals do have substantial systems outside of the National Programme.

We found that the changing political and economic climate—specifically the change in government and the economic recession—resulted in uncertainty about the future of the National Programme. There was in particular a substantive move away from the initial “top-down” implementation model to an increase in local involvement in decision making, particularly in London and the Southern areas.¹² For example, as one national media report noted:

“BT’s new agreement with the Department of Health may provide a better indication of the Programme’s future. The company, which met the deadline it was set to implement Cerner Millennium at Kingston Hospital last year, agreed to cut £112m (11%) from its £996m local service provider contract to the NHS in London. It will do so by abandoning the idea of uniform software: if London hospital trusts [that is, hospitals] already have fit-for-purpose IT, BT will connect those systems rather than replace them.”⁸⁴

These changes were, at the time of writing, ongoing, with the coalition government still not having published a detailed IT strategy for the NHS. The initial indications were that the future of the National Programme was likely to be characterised by increased local input in decision making and an opening up of the supplier market, thereby allowing greater supplier choice.^{18 85} This should be contrasted with the originally planned NHS Care Records Service, which, as noted above, aimed to deliver an integrated standardised solution.

Locally, we witnessed several major impacts of these wider developments. The resulting lack of sustained leadership from the centre rendered it difficult for the hospitals to develop any coherent long term strategy. There were serious concerns by hospitals that had already begun implementing NHS Care Records Service systems such as Lorenzo Regional Care but with only limited clinical functionality:

“I cannot see how that’s ever going to succeed, I just can’t and as you quite rightly say you’ve got bits of functionality implemented in very small areas. You’ve got bits of functionality implemented in podiatry or somewhere else but you’re not seeing the rollout of that functionality to the rest of an organisation and how on earth are you going to progress if they’re not doing that so, you know, you’re going to have, you know, isolated developments in isolated areas in an organisation and that’s not what the NHS needs, you know, it needs things that are optimising things across organisations” (interview, independent sector).

These concerns extended to questions about the likely “life span” and degree of continuing support for these nationally procured systems (as contracts with local service providers are scheduled to end in 2015) and potential consequences to local morale if efforts to implement were to be “abandoned” in the current austerity climate. Most thought that the considerable work invested in making these systems work and the local expertise gained in the process should not be allowed to wither but were nonetheless unclear as to what direction they should take.

There were no coherent plans for the many hospitals yet to implement these systems, leaving some organisations to consider alternative options. Our work indicated that this was especially apparent in the more autonomous hospitals.

“[name] was pretty plain until quite recently that any [hospital] that broke away from the National Programme would be penalised even if, you know, the guidance says that, you know, [more autonomous hospitals] don’t have to adopt national solutions so I think there was a lot of management pressure to keep [more autonomous hospitals] in the loop” (interview, independent sector).

The over-riding related concern expressed by several mangers was that the move away from the original goals of the National Programme and in particular the NHS Care Records Service—without any parallel focus on ensuring interoperability and shared learning—would risk reverting the NHS back to the same situation that led to the creation of the National Programme—namely, patchy implementations of the main patient administration systems with limited clinical functionality and poor interoperability.

“Well strangely enough of course, you know, going back 20-30 years OK the technology was very different but, you know, there were custom-built NHS solutions so [name of hospital] went through a custom-built approach but what was found was I think it might have worked at [name of hospital] but they could never really get it to work anywhere else. . . So I think decisions were taken in setting up the National Programme which didn’t take enough account of what had happened in the past, there was a view, you know, which was well you guys who’ve been working in this space up till now have failed, you haven’t delivered what was needed so here we are with a fresh new broom and Richard Granger was very much on record as saying, you know, we’re going to clear, we’re going to start afresh and the National Programme is about rip and replace and, you know, in hindsight that was probably the wrong approach” (interview, independent sector).

Discussion

Main findings

Implementation of NHS Care Records Service systems has proceeded much more slowly than expected and with as yet limited benefits for NHS staff or patients. Delays seemed to be caused by the complex and constantly shifting national set-up characterised by complex politically shaped contractual relationships that largely excluded NHS healthcare providers. Many stakeholders had a lack of appreciation of the social and organisational consequences of implementing technology and unrealistic expectations about the capabilities of the software systems; the time needed to build, configure, and customise the software; the work needed to ensure that these systems were supporting rather than hindering provision of care; and the training and support needs of end users (and particularly non-clinical staff). The local consequences for organisations implementing different functionalities were considerable in terms of time and efforts it took to “work out” the new technology and cost implications. Across hospitals, however, we found that some benefits emerged over time, mainly relating to organisational learning and improved local data sharing.

Strengths and limitations of this work

The scale, volume of data obtained, and real time nature of this evaluation of the implementation of electronic health records in English secondary healthcare settings are strengths of our work.¹ We also used a methodologically innovative design,⁸⁶ captured contemporaneous multifaceted real time longitudinal data, and had theoretical grounding.^{19 87} As a result we obtained a rich and nuanced appreciation of the implementation and adoption of the NHS Care Records Service locally, the depth of inquiry offering transferability beyond the immediate context of this evaluation as we sampled from a range of hospitals, software systems, and stakeholders.^{19 88} This depth of inquiry and the richness of the ensuing dataset enabled us to consider a range of explanatory factors. For example, we considered both generational and skills-related factors to explain the problems encountered by healthcare professionals, and, while there was some evidence of these being important in a few instances, they did not emerge as major in explaining the lack of progress with implementation and adoption to date. This diversity of context and data is, we believe, a strength, particularly in the study of electronic health records, where much reported research relates to a single site, often a well resourced centre of excellence.⁴⁸

This work also has several limitations. In working with the “early adopters” we observed problems that were being dealt with for the first time by all stakeholders; these might eventually be resolved through negotiation, and subsequent implementers could learn from these early experiences. Studying “early adopters” could be viewed as a worst case scenario. That said, these sites were selected (sometimes after competition) on the basis of their previous track record of implementing IT and the leadership capacity, or both, within the organisation, and they often received considerable financial and technical support of an order that is unlikely to be offered to later adopters. In some respects these can therefore be seen as best case scenarios. Judging the balance between these two positions comes down in part to the notion of the perfectibility of the technical elements. If “early adopters” “iron out the wrinkles” and refined technology is inherently more acceptable and usable, then things should get better. But on balance we are not convinced by this argument as the more fundamental and challenging issue is organisational change, not technical refinement. The key point is that it is important to extrapolate with caution from our work on “early adopters” to later adopting sites. Our focus was on investigating issues and emergent changes in early implementation processes as opposed to systematically identifying benefits (though we did search extensively for these during our work) such as those outlined in appendix 1 on bmj.com. Benefits of implementing a national IT based health records system might not be seen for several years as there is a natural learning curve in any organisational change initiative, highlighting the need for longer term longitudinal mixed methods work.

Another important issue has been gatekeeper influence at all levels, possibly because of conflicting priorities, but possibly also the, at times politically charged, nature of the implementation of NHS Care Records Service, which has been the subject of major sustained parliamentary, political, and media scrutiny, both in the UK and internationally.^{10 89 90 91 92 93 94 95 96 97} The latest wave of this scrutiny has been triggered by a National Audit Office report, which concluded that “given its past history, the major issues still confronting the care records systems, and with such significant funds still at stake, there is a compelling case for the recently announced Whitehall-wide review to re-evaluate the business case for the Programme to determine what should happen now to safeguard against further loss of public value.”^{16 17} This political environment resulted in restricted access to some stakeholders, including patients, developers, governmental stakeholders, and healthcare professionals. It also meant that some stakeholders initially seemed to hesitate in providing us with information, particularly in relation to what they considered to be sensitive (and sometimes confidential). Nevertheless, the longitudinal nature of the evaluation helped in building trust and facilitating more open discussion, particularly with members of NHS Connecting for Health. Similarly, information relating to the actual “go live” dates of hospitals was difficult to obtain, impacting on our sampling strategy and ability to plan.

Finally, our formative work might have had an impact on the findings from our summative research, and this could therefore be seen by some as a limitation. We, however, view this as a strength as it provided an opportunity to share with hospitals any early lessons learnt.⁹⁸

Policy implications for the NHS

Policymakers have already started to shift the focus to more local efforts to procure and implement electronic health records, these being reinforced by the changes in outlook of the coalition government, the planned changes to the NHS in England (such as a drive to increasing the autonomy of hospitals from the control of central government), and the current economic climate.^{12 18 85} In the light of this evolving policy landscape, and drawing on our research and broader international experiences, we have summarised our main recommendations in box 5. A particularly urgent short term measure is to make the software work well in the NHS organisations that have already started implementation.

Box 5: Summary of emerging key policy recommendations

Overall strategy

• Build on a coherent vision of shared electronic health records to improve the provision of joined up patient centred care

• Devise a strategy that is characterised by flexibility and the ability to respond to evolving needs, while ensuring that there is a clear local rationale aligned with national purposes

• Move away from technology driven models of implementation and refocus attention on adoption as ongoing “working out” between staff and technology; think of technology as an enabler of improved care processes rather than an end in itself

Architecture

• Ensure that software is assessed to be fit for purpose by users in the implementing organisations

Process

• Make software work well in the NHS organisations that have already started implementation

• Ensure that procurement decisions are not based on unrealistic assumptions of achieving cost savings or even short term returns on investment, but rather on introducing clinical functionality early so that these systems are used

• Balance central incentives to implement with large scale interoperability and local input in decision making

• Ensure sustained efforts and appropriate funding characterised by a coherent strategy with realistic timelines to allow local organisations time to “work out” the consequences of the change

• Consider the merits of participating in the development of open source systems as opposed to the purchase of commercially developed systems

• Retain hard won knowledge at both local and national levels and make appropriate use of these skills and expertise both in and across sites

• Facilitate the sharing of experience and learning both nationally and internationally

While welcomed by many NHS hospitals, the move to a more locally driven approach might also mean some major trade offs. These include the risk of potentially conflicting local priorities resulting in insufficient drive and funding for such developments, problems with systems interoperability, and failure to share lessons learnt with consequent entrenchment of local work practices rather than the original vision to “transform” healthcare nationally. Furthermore, not all hospitals will have the expertise to implement such transformative IT initiatives, which in turn could result in unacceptable risks to the safety and quality of care. There is also the danger of double counting potential efficiency gains as those included in the business case for IT implementations overlap with those included in other local cost reduction initiatives.

Implications for the international community

The English experience offers several potentially transferable lessons for ongoing international efforts to implement electronic health records.^{99 100 101 102 103 104} First and foremost, there remain important drivers for the implementation of integrated electronic health records, including the potential for increased accessibility, which is important considering the more fragmented nature of provision of care and the major advantages associated with digitised data in relation to facilitating quality improvement initiatives and research.

As we have shown, the procurement of national systems in England had a range of unanticipated consequences. Large scale procurement was undertaken to save costs, but this meant that implementation timelines were rushed, being driven according to political timeframes in line with procurement arrangements. Hospitals have coped differently with these pressures, often heavily influenced by the particular type of software implemented. Despite the important learning within and between organisations that had taken place across the sites we studied, these national pressures have, in some cases, resulted in software being deployed prematurely with adverse consequences for local organisations, users, and patients’ care. We therefore recommend that procurement decisions should not be based on unrealistic assumptions of achieving cost savings or even short term returns on investment, but rather on introducing clinical and associated decision support functionality early so that these systems are used and deliver demonstrable clinical benefits.^{103 105} Adequate national investments also need to be made to support interoperability, which is fundamental to enabling reuse of data.¹⁰³ Recent announcements in the English strategy, however, indicate that only 2% of the total IT modernisation budget has been allocated to support interoperability.¹⁶

Strategically, it is essential that any health informatics policy is integrated with concurrent policy initiatives and reflects the dynamic environment in which it is taking place. In England, this has to some extent been achieved (such as by gradual movement towards a more localised approach), while on the other hand it was (and still is) hard to adapt nationally set arrangements to evolving needs (such as contracts with local service providers). The consequences of these are often still hard felt on the ground. Admittedly, it is difficult to achieve this balance as healthcare organisations are continually changing. For example, the increasing competition among healthcare organisations might not align with delivering the benefits of shared learning, nor with prioritising efforts to ensure large-scale interoperability.¹⁸ The recent National Audit Office report highlights current uncertainties regarding future responsibilities within the National Programme.¹⁶ This is of particular concern as it is unlikely that organisations that are being encouraged to compete with each other will focus attention on interoperability without specific incentives of the kind that are being promoted in, for example, the US and Canada.¹⁰⁶

Given the challenges inherent in contracting on a national scale, there is a need to consider the merits of participating in the development of open source systems as opposed to the purchase of commercially developed systems.¹⁰⁷ Indeed, the resources devoted to the development of Lorenzo Regional Care already represent a considerable transfer of intellectual property from the NHS.

Implications for future research

How national strategy evolves is one issue, but not the only one. It clearly remains important to investigate the longer term consequences of substantial hospital-wide health records systems, taking into account evolving skill levels of users and evolving organisational capacity.¹⁰³ There is also a need to assess whether once used, NHS Care Records Service software systems improve the quality and safety of care, the more effective management of resources, the way in which care practices are organised and delivered, and professional roles. Finally, we identify a pressing need to develop appropriate models to inform how best to implement national systems and, in particular, to assess whether paying appropriate attention to the sociotechnical dimensions of implementation is beneficial at promoting successful adoption.

Conclusions

The “top down” nature of the national programme has, despite ensuring necessary high level leadership and support,¹² contributed to a lack of organisational and user involvement in decision making. Organisations and users implementing and adopting the NHS Care Records Service have coped with these challenges in different ways, their responses being influenced by the software in question and local arrangements. Despite some early benefits, particularly in relation to infrastructure and internal capacity development, our results show that software systems have as yet had difficulty fulfilling organisational and user needs.

The strategic move towards a more locally autonomous model now needs to be balanced with national requirements in terms of systems interoperability, shared learning, and implementation approaches. Early international experiences suggest that use of such a combination of local and centralised approaches offers the best chances of successful implementation and adoption.¹⁰⁸

England has attempted to pursue a centralised “top down” approach, and there is now a need for international efforts to learn from and build on these early experiences. Although our work has clearly shown that many users, managers, service providers, and implementers have been sorely bruised by the first steps in the journey to implement a truly comprehensive national electronic health record system, history might—particularly if the right calls are now made at this important juncture—be more forgiving.

What is already known on this topic

• Many countries are now actively pursuing the implementation of electronic health records on a national scale

• In 2002, England embarked on implementing electronic health record systems with clinically rich functionality spanning both primary and secondary care settings

• Major challenges associated with the government led “top down” implementation strategy have been identified

What this study adds

• All 377 hospitals in England should have completed the implementation of electronic health record systems by December 2010; only one in five, however, had begun implementation

• There is a need to move away from technology centred models of “implementation” and focus more attention on the process of “adoption,” which needs to be seen as an ongoing “working out” between staff and technology

• Important learning within and between organisations occurred in “early adopter” sites, which needs to be drawn on to support the ongoing implementation efforts

Cite this as: BMJ 2011;343:d6054

Web Extra. Extra material supplied by the author

Appendix 1: Anticipated benefits of electronic health record systems

Appendix 2: History of National Programme for IT, its local service providers, and suppliers for NHS Care Records Service

Appendix 3: Sample topic guides

Appendix 4: Characteristics of case study sites