Acute kidney injury electronic alerts: mixed methods Normalisation Process Theory evaluation of their implementation into secondary care in England

Abstract

Objective

Around one in five emergency hospital admissions are affected by acute kidney injury (AKI). To address poor quality of care in relation to AKI, electronic alerts (e-alerts) are mandated across primary and secondary care in England and Wales. Evidence of the benefit of AKI e-alerts remains conflicting, with at least some uncertainty explained by poor or unclear implementation. The objective of this study was to identify factors relating to implementation, using Normalisation Process Theory (NPT), which promote or inhibit use of AKI e-alerts in secondary care.

Design

Mixed methods combining qualitative (observations, semi-structured interviews) and quantitative (survey) methods.

Setting and participants

Three secondary care hospitals in North East England, representing two distinct AKI e-alerting systems. Observations (>44 hours) were conducted in Emergency Assessment Units (EAUs). Semi-structured interviews were conducted with clinicians (n=29) from EAUs, vascular or general surgery or care of the elderly. Qualitative data were supplemented by Normalization MeAsure Development (NoMAD) surveys (n=101).

Analysis

Qualitative data were analysed using the NPT framework, with quantitative data analysed descriptively and using χ² and Wilcoxon signed-rank test for differences in current and future normalisation.

Results

Participants reported familiarity with the AKI e-alerts but that the e-alerts would become more normalised in the future (p<0.001). No single NPT mechanism led to current (un)successful implementation of the e-alerts, but analysis of the underlying subconstructs identified several mechanisms indicative of successful normalisation (internalisation, legitimation) or unsuccessful normalisation (initiation, differentiation, skill set workability, systematisation).

Conclusions

Clinicians recognised the value and importance of AKI e-alerts in their practice, although this was not sufficient for the e-alerts to be routinely engaged with by clinicians. To further normalise the use of AKI e-alerts, there is a need for tailored training on use of the e-alerts and routine feedback to clinicians on the impact that e-alerts have on patient outcomes.

Strengths and limitations of this study.

• This is the first known mixed methods study to use Normalisation Process Theory to investigate the implementation of acute kidney injury (AKI) e-alerts, providing a unique lens on their implementation.

• The study was conducted in clinical areas where AKI incidence is high; it is unknown whether the e-alert would be more useful (and whether it would be more or less poorly implemented) in clinical areas where AKI incidence is lower.

• The study was also conducted in one region, and so implementation of the AKI e-alert may have been influenced by local networks.

• It is unknown whether the e-alerts had a quantifiable impact on AKI outcomes or staff actions, and so it is not possible to draw conclusions about the effectiveness of the AKI e-alerts studied as a result of implementation.

Introduction

Acute kidney injury (AKI) affects around one in five emergency hospital admissions.¹ AKI is both dangerous, with around 15 000 excess deaths in National Health Service (NHS) England inpatients per year,² and costly, imposing an estimated additional financial burden on this system of £1.02 billion per annum.² Increasing age and comorbidity in the hospital population has increased the number of patients at risk from the condition, which is only likely to rise further with an ageing population. AKI care itself is often poor, with systematic failings in its recognition and management, and frequent omissions of even the basics of care.³

AKI alerting systems are mandated for all NHS England primary and secondary care providers, using a biochemical detection algorithm⁴ and usually implemented electronically. The algorithm, which appears to perform with a high degree of sensitivity (>90%),⁵ has resolved ambiguities in modern diagnostic criteria⁶ around how to interpret baseline serum creatinine (SCr), a historical impediment to the standardisation of automated AKI detection, and as well as outputs for the three stages of disease severity, it also flags out-of-range SCrs in the absence of an historical baseline. AKI electronic alerts (e-alerts) are thought to improve patient outcomes by improving early detection of AKI and triggering earlier intervention by clinicians.⁷ The exact nature of the AKI alerts is not, however, dictated, and may take a number of forms.

The efficacy of AKI e-alerts is limited and has not shown consistent benefit⁸ in terms of reduced mortality or use of renal support, or positive impacts on processes of care,⁹ which may be the result of alert fatigue^{5 10} or disrupted workflow.¹⁰ Inadequate implementation can explain the poor outcomes, particularly as there are some examples of improved care processes¹¹ and treatment outcomes through successful implementation.^12–14 Mandatory incorporation of AKI alerts into all secondary care organisations in England lacked a clear implementation strategy, and recently published systematic reviews recognised large variation in implementation,¹² with an association between poor implementation and poor outcome.⁹ One review specifically identified a paucity of research on the implementation of AKI e-alerts internationally.⁹ To address this paucity of research, the present study aimed to identify factors relating to implementation which promoted or inhibited use of AKI e-alerting systems in secondary care.

Methods

This study incorporated mixed methods (qualitative interviews and observations combined with quantitative surveys) to investigate the implementation of AKI e-alerts from multiple perspectives, including observations, surveys and semi-structured interviews. Normalisation Process Theory (NPT)^{15 16} was chosen as the theoretical basis for the study as it is an internationally recognised theory of implementation that has been used to explain successful and suboptimal implementation in over 100 healthcare initiatives,¹⁷ including through the use of mixed methods.^18–20 NPT therefore provides the explanatory power for understanding how complex interventions, such as AKI e-alerts, become integrated into existing practice through individual and collective implementation. This integration is proposed to occur via four mechanisms: ‘coherence’: how people make sense of what needs to be done, ‘cognitive participation’: how relationships with others influence outcomes, ‘collective action’: how people work together to make practices work and ‘reflexive monitoring’: how people assess the impact of the new intervention. The four constructs are operationalised under 16 subconstructs, which are described in table 1.

Table 1.

Description of Normalisation Process Theory mechanisms and subconstructs

Coherence Differentiation: how participants understand a set of practices and their objects to be different (or not) from each other. Communal specification: extent to which participants have a shared understanding of the aims, objectives, and expected benefits of a set of practices. Individual specification: how participants understand their own specific tasks and responsibilities around a set of practices. Internalisation: how participants perceive the value, benefits and importance of a set of practices.

Cognitive participation Initiation: whether or not key participants are working to drive a new set of practices forward. Enrolment: the extent to which participants organise or reorganise themselves and others in order to collectively contribute to the work involved in new practices. Legitimation: the work of ensuring that other participants believe it is right for them to be involved in the new set of practices, and that they can make a valid contribution to it. Activation: the work that participants do collectively to define the actions and procedures needed to sustain a new practice and to stay involved.

Collective action Interactional workability: the interactional work that people do with each other, with tools/systems, and with other elements of a set of practices, when implementing a new practice. Relational integration: the work that is needed to build accountability and maintain confidence in a set of practices and in each other as they use them. Skill set workability: the allocation of work among participants with different roles and skills in relation to the new set of practices. Contextual integration: the work of managing a set of practices through the allocation of different kinds of resources and the execution of protocols, policies and procedures to support the practices.

Reflexive monitoring Systematisation: the work undertaken by participants to determine how effective and useful the new set of practices is for them and for others, and the information collected to enable this. Communal appraisal: the work undertaken by participants collectively (sometimes in formal collaboratives, sometimes in informal groups) to evaluate the worth of a set of practices. Individual appraisal: individual participants’ own appraisals, based on their experiences, of the effects of a new set of practices on them and the contexts in which they are set. Reconfiguration: the extent to which appraisal work by individuals or groups may lead to respecification or modification of the set of practices.

Sampling and recruitment

Three NHS Trusts in North East England were invited to take part in the study based on being within a single NHS Trust’s renal department catchment area, and the catchment area for referral for complex AKI (see table 2 for a description of Trusts and their AKI e-alerting systems). Three clinical areas were purposively chosen for study at each NHS Trust based on anticipated high levels of AKI incidence: (1) emergency admissions, (2) internal medicine/care of the elderly and (3) general/vascular surgery.

Table 2.

Characteristics of participating NHS Trusts and their AKI electronic alert

NHS Trust	Trust characteristics	AKI electronic alert
1	University-affiliated, 1800-bed, multiple site tertiary referral hospital; contains regional renal unit, transplantation, cardiothoracic and hepatobiliary surgery as well as other major specialisms; the Renal Unit is based at one Trust site and although providing consultative input to the rest of the Trust, has no routine, on-site presence at these venues which include the emergency admissions suite and significant sections of general surgical, internal medical and elderly care services.	Passive electronic alert consisting of a line of black text appearing underneath serum creatinine results in the patient’s electronic medical record (Powerchart, Cerner Millennium, Cerner, Kansas, Missouri , USA), which stated the stage of AKI where present. Additional pop-up electronic alert that appeared when accessing the patient’s electronical medical record, and required dismissal to remove from the screen. A ‘more info’ link on the pop-up window, if clicked, took the end-user to a second window that contained further links to the Trust AKI protocol and specific guidance on aspects of management including essential assessments, key bedside observations and key investigations. The latter linked to quick order test panels including essential blood tests and urgent renal ultrasound requesting.
2	Multisite university-affiliated district general with approximately 900 beds. All acute services on one site with internal medicine, elderly services, general and orthopaedic surgery along with obstetrics and paediatrics. Renal replacement provision from critical care. Renal input, provided from Trust one by remote consultation, no renal consultant presence within the trust.	Passive electronic alert consisting of a line of black text appearing underneath serum creatinine results in the patient’s electronic medical record (TelePath Information Management System, Mill Systems, Belper, UK), which stated the stage of AKI where present. Alert does not link with any other hospital information system, but instructs users to access local AKI guidelines.
3	District general hospital with 300 beds in medicine. Renal input is from Trust 1 through a combination of remote consultation and weekly availability at the time of an outpatient clinic on site.	Passive electronic alert consisting of a line of black text appearing underneath serum creatinine results in the patient’s electronic medical record (Medical Information Technology (MEDITECH), Westwood, Massachusetts, USA), which stated the stage of AKI where present. Alert does not link with any other hospital information system, but instructs users to access local AKI guidelines.

See online additional files 1 and 2 for images of the electronic alerting systems.

AKI, acute kidney injury; NHS, National Health Service.

Participants for semi-structured interviews were purposively sampled based on specialty and clinical experience (determined by grade). Participants were invited through direct contact by JS, or by leaving contact details after completing a survey. Recruitment to survey was conducted through direct contact by JS, or electronically via an internal email by (or on behalf of) the lead consultant for the clinical specialty. Teaching sessions at Trust 3 were also used to invite staff to participate in the survey. Access to observe practice on emergency admission units was facilitated by the lead consultant(s) for the unit. Participants were able to take part in the research activities (interviews, observations and/or survey) in any order, based on what was most convenient. Where possible, the order of activities was balanced to reduce confounding variables.

Data collection

Semi-structured interviews were conducted by a male research associate, JS (PhD), with participants in their place of work or via telephone between May 2017 and September 2017, and lasted an average of 26 min (range 17–41). Interviews were recorded using a digital voice recorder and transcribed verbatim by a professional transcription company. A topic guide (see Additional file 3) was constructed by the research team based on the four mechanisms of NPT (coherence, cognitive participation, collective action, reflexive monitoring; see table 1) and from previous qualitative work on implementation of AKI e-alerts.¹⁰ In addition to questions based on the four NPT mechanisms, the topic guide also included questions about the participants’ clinical experience (job role, length of time in role, experience in other roles) and their experience with AKI e-alerts. Ethnographic data were obtained by JS by observing practice in emergency admission units, guided tours, shadowing of staff and informal conversations and handover meeting attendance. Observational data were documented in fieldnotes.

The emergency admission units function to provide early assessment of adult patients referred via their general practitioner or the emergency department. One of the units (Trust 3) was a hybrid emergency admission unit and acute medicine ward. A total of 44.25 hours of observations were conducted at various times of day (morning, afternoon and evening) during the working week (Monday to Friday).

The Normalization MeAsure Development (NoMAD) survey,^21–23 a validated instrument for measuring implementation,²⁴ was adapted for use with AKI e-alerts (see Additional file 4). Questions were added to identify characteristics of respondents, including:

• Profession;

• Grade;

• Years since obtaining primary medical qualification;

• Years working in the Trust;

• Years working in the department;

• Formal or informal AKI training received in previous 24 months;

• AKI initiatives to improve awareness of AKI other than e-alerts.

In addition, five questions from the Hospital Survey on Patient Safety Culture (SOPS; V.1.0)²⁵ were included. SOPS contains a construct containing four questions titled ‘Overall perceptions of patient safety’. All four questions from this construct were included, along with an overall patient safety grade. Paper and electronic versions of the study survey were made available to potential participants. All data collection was conducted after the AKI e-alerts had been implemented into practice for at least 1 year.

Data analysis

Framework analysis was used for qualitative data,²⁶ with the four NPT mechanisms and their subconstructs forming the framework (table 1). For interview data, one interview transcript was jointly charted by JS and TF, with interpretations of the data discussed until agreement was reached. This discussion familiarised JS with the differential meanings underpinning the 16 subconstructs for subsequent analysis of qualitative data, as TF is an expert in NPT as a co-developer of the theory.^{22 23} JS then charted the remaining interview data into the framework. For observational data, in-depth observer notes were summarised by the observer (JS), then all observation data were charted into the framework jointly with TF.^{15 16} NVivo software (QSR International, V.10) was used to facilitate coding of qualitative data. Once initial analysis was complete, all authors reviewed and discussed the coding in a team meeting before coming to agreement on the final interpretations, which is an established process of qualitative data analysis.²⁷ Participants were not invited to comment on findings.

IBM SPSS Statistics for Windows (IBM, V.24.0) was used for quantitative analysis. Inferential statistics (χ²) were used to compare patient safety culture between NHS Trusts and specialties to identify whether safety culture could influence the subsequent analysis. Wilcoxon signed-rank test was used to analyse differences in current and future normalisation of the e-alerts. Survey items relating to the four NPT mechanisms were then analysed by examining descriptive statistics for each of the four mechanisms. Mechanism scores for each participant were created by taking their average score in each mechanism and dividing by the number of valid responses, which stopped data from being skewed where respondents stated a question was not applicable. Higher scores represent better perceived implementation in relation to each mechanism. Data were then triangulated by exploring (dis)agreements and silences across the qualitative and survey data sets. This was conducted by a single researcher (JS) identifying and listing subconstructs that demonstrated particularly high or low normalisation, comparing these against qualitative themes and then discussed among the research team.

Patient and public involvement

There was no patient and public involvement in the design or planning of the study.

Results

Semi-structured interviews were conducted with 29 staff members. Twenty-eight interviews were with doctors, and one interview was with a pharmacist involved in implementing AKI e-alerts at Trust 1. The survey was distributed to 157 staff, and 102 (65%) responded. Ninety-four (92.2%) completed the paper version, and eight (7.8%), the online version. See table 3 for a summary of interview participants and survey respondent characteristics. Table 3 also acts as a key to participants’ grades, which is used to infer level of experience (grades are competency based) and is also used in the reporting of qualitative data. One survey was excluded as the participant reported on an e-alerting system at an NHS Trust not included in the study, leaving a final sample of 101.

Table 3.

Participant characteristics of interviews and survey

Characteristic	Interview participants	Survey respondents
	N (%)	N (%)
Job grade
Foundation doctor year 1 (F1)	9 (31.0)	16 (15.8)
Foundation doctor year 2 (F2)	4 (13.8)	25 (24.8)
Specialty registrar doctor year 1/2 (ST1/2)	4 (13.8)	23 (22.8)
Specialty registrar doctor year 3/4/5 (ST3/4/5)	3 (10.3)	10 (9.9)
Specialty registrar doctor year 6/7 (ST6/7)	2 (6.9)	4 (4.0)
Staff grade doctor	0 (0)	5 (5.0)
Consultant	6 (20.7)	15 (14.9)
Nurse (band 6)	0 (0)	1 (1.0)
Other	1 (3.4)*	2 (1.0)†
NHS Trust
1	11 (37.9)	30 (29.4)
2	8 (27.6)	60 (58.8)
3	10 (34.5)	11 (10.8)
Department
Internal medicine/care of the elderly	10 (34.5)	46 (45.5)
Emergency admission	8 (27.6)	26 (25.7)
General/Vascular surgery	10 (34.5)	20 (19.8)
Other	1 (3.4)‡	9 (8.9)§

*Pharmacist.

†Medical student=1, locum senior house doctor=1.

‡Pharmacy.

§Palliative care=4, acute medicine=2, cardiology=1, ITU=1, nephrology=1.

Patient safety

Overall patient safety culture, graded on a Likert scale from 1 (very poor) to 5 (excellent), had a mean score of 3.75. A χ² analysis comparing the three NHS Trusts identified no significant difference in patient safety culture (χ²=1.784, df=2, p=0.410). Using the same method, there was also no significant difference between the specialties surveyed (χ²=1.453, df=3, p=0.693). These results indicated that different sites or specialties did not confound the analysis.

Familiarity and perceived normalisation

Participants reported that they were mostly familiar with the e-alerts (mean=7.27, SD=2.562) and that the e-alert was part of their normal work (mean=7.28, SD=2.649). However, it was reported that the e-alerts would become a more normal part of their work (mean=8.32, SD=2.059), with a Wilcoxon signed-rank test confirming the difference was statistically significant (z=−5.049, p<0.001), suggesting that the e-alerts were not yet fully embedded.

NPT mechanisms and subconstructs

Descriptive analysis of the mean scores of the four NPT mechanisms—coherence (x̅=72.3%), cognitive participation (x̅=76.4%), collective action (x̅=66.5%) and reflexive monitoring (x̅=68.8%)—suggested there was no key mechanism that led to (un)successful implementation of the e-alerts. Further analysis of the 16 subconstructs (table 1) identified several subconstructs indicative of (un)successful implementation; mean ratings for the 16 subconstructs are presented in figure 1. More specifically, following triangulation with qualitative data, the NPT subconstructs that were identified to contribute to successful normalisation of the AKI e-alerts were internalisation and legitimation, and those that contributed to unsuccessful normalisation were initiation, differentiation, skill set workability and systematisation. As with the survey data, there were no identified differences in qualitative findings between the two e-alerting systems. Supporting qualitative data (quotes and field notes) for all 16 subconstructs are provided in table 4. The remainder of the results will focus on NPT subconstructs that demonstrate where normalisation was most positive or negative, based on the triangulation of all data sources, representing subconstructs that most promote or inhibit use of AKI e-alerts in secondary care.

Figure 1.

Petal chart showing mean scores for the 16 NPT subconstructs. Likert scale of 1 (strongly disagree) to 5 (strongly agree).

Table 4.

Summary of the qualitative framework analysis for the 16 NPT subconstructs with supportive evidence

NPT mechanisms and subconstructs	Coding summary	Supporting evidence
Coherence
Differentiation Mean survey score: 3.18	Clinicians often did not differentiate between normal practice and use of the AKI e-alert; checking the patient’s renal function was deemed to be routine in the clinical areas studied.	‘A doctor says that it’s a routine part of their job to check renal function and would check it anyway—this is the case even in the emergency department if a patient came in with a broken arm to check for a potential underlying condition’. (Trust 1 observation of emergency admissions, approx. 19:30 hours)
Communal specification Mean survey score: 3.29	Working with the AKI e-alert was perceived to be an individual rather than team action. The e-alert was rarely discussed or used to initiate discussion, with staff often not knowing what others thought about the e-alert.	“I don’t think we’ve had much discussion about the AKI alerts, it’s certainly not something that I’m aware of, that other people have commented on”. (F1 interview, general/vascular surgery, Trust 2) “I haven’t spoken to anyone else(about the AKI e-alert). I know just from being in the doctor’s office, with the other doctors that sometimes, you know, you see people glance at the screen, and go ‘click’ while they are still talking to you”. (ST2 interview, internal medicine/care of the elderly, Trust 2)
Individual specification Mean survey score: 3.8	AKI e-alerts often made staff consider the patient’s AKI and to double check renal function.	“I guess it’s to draw attention to it quickly rather than bloods getting lost in the system for the day, because… especially on a busy ward(…)there’ll be… I don’t know… 20 bloods sent in the morning and then if you’re busy with sick people it could go well into the afternoon before you get to check on bloods”. (F1 interview, emergency admissions, Trust 2)
Internalisation Mean survey score: 4.16	Many staff saw the potential value of the AKI e-alert and understood the need for the e-alerts.	“I think it’s probably the most useful out of the alerts. It generally comes on when the patients genuinely do have an AKI, although, that’s often sometimes not the case. Like we said before, often it does require action, so, yes, they’re pretty useful”. (F1 interview, emergency admissions, Trust 1)
Cognitive participation
Initiation Mean survey score: 2.8	Participants frequently cited a lack of initiation in relation to the AKI e-alerts. This occurred for one of two reasons: 1) the e-alerts just appeared without any training on how to use them or 2) clinicians were newly qualified (or new to the Trust) and the e-alerts were already implemented, but again no training was provided.	“Yes they just sort of bob up. We never had really any induction about them”. (F1 interview, general/vascular surgery, Trust 3)
Legitimation Mean survey score: 4.16	For the more junior doctors, the e-alerts are perceived to be a legitimate part of their role. However, for more senior doctors, particularly in surgical units, the e-alerts were a useful intervention but only for junior doctors. Some clinicians felt that there should be a specialist AKI nurse.	“We have specialist nurses who provide input for absolutely everything. So, the idea that there isn't one for an AKI, is a bit silly, in my opinion. Because, somebody coming… their purpose in my opinion would be to come around, read what they are in for, review their pathology, review the patient, and review reversible factors. Then make a recommendation to the junior and the consultant about reversible factors that hadn’t been looked at yet”. (ST1 interview, internal medicine/care of the elderly, Trust 2)
Enrolment Mean survey score: 4.24	As working with the AKI e-alert was an individual action, it often had no influence on working relationships. For the few clinicians who saw the relational value, it was beneficial by providing the AKI stage that could be easily reported.	“If one of the F1’s came to me and said, ‘this woman’s creatinine has gone up’, then yes, absolutely we would have a chat about meds, and fluid, and have they had an ultrasound scan, and what do you think we should do? But, I don’t think the alert has ever prompted me to do that”. (ST2 interview, internal medicine/care of the elderly, Trust 1) “It's good if you're doing a handover on the phone or something or talking to seniors in critical care or other hospitals. You can say this is Stage 2 AKI and that is sort of a standard term that people do understand even if we don’t use it in general day-to-day discussion in the notes as much as we should do”. (F1 interview, general/vascular surgery, Trust 3)
Activation Mean survey score: 4.06	Prolonged exposure to the AKI e-alerts impacted on clinician’s support for them; the e-alerts had more impact when new, but they became part of the milieu and lost among other e-alerts or working practices.	“Yes, I do actually. I think it's a big component of patient safety and I think it is the direction of travel of where we're going. As time goes on, looking forward, I think we're going to expect more and more of these alerts related to algorithms and severity”. (Consultant interview, emergency admissions, Trust 1)
Collective action
Interactional workability Mean survey score: 3.98	It was generally deemed easy to integrate the AKI e-alerts into normal working practices; they are there as a ‘check’ or ‘backup’ as most clinicians were routinely checking renal function. The e-alerts were perceived to speed up the process of calculating the stage of AKI. E-alerts were seen to be useful where creatinine was within normal range, but with an increase of >1.5 from baseline.	“I think if it’s not someone I already know, then yes, the alerts at least make me glance at the U&Es, which I would do anyway, but you know, just an extra reminder to check back what their previous U&Es were”. (ST7 interview, emergency admissions, Trust 1) “I think in some ways, it probably does speed things up, because you have got that alert there, and I think, when you open up a page of bloods, and it’s quite obvious, and the first thing you see is they have an AKI”. (ST3 interview, emergency admissions, Trust 3)
Relational integration Mean survey score: 3.67	The AKI e-alert did not appear to affect working relationships. Staff mostly do not refer to the e-alert when discussing AKI, and AKI care is often an isolated task. An exception is stage 3 e-alerts, which sometimes trigger discussions with renal services.	“I don’t know about the other staff and how they engage because actually I've not had a lot of feedback from them. I haven’t actually been hearing the juniors saying, ‘Oh there was an AKI alert’ on anyone so I suspect most of them are just clicking and moving on, dismissing and moving on because they probably already know what the creatinine’s doing”. (Consultant interview, general/vascular surgery, Trust 1)
Skill set workability Mean survey score: 3.27	There was a lack of training on how to best use the AKI e-alerts (also reported in initiation), and some clinicians demonstrated a lack of understanding about AKI, particularly the meaning of the different stages.	“When you come to the Trust you get—I don't know how long the sessions are and I don’t know what they cover, and I don’t know whether they cover alerts and things like that. If they do have (AKI e-alert training), I suspect it comes at the end of a very long day of induction where they’ve been told about every single problem under the sun and they’ve probably switched off”. (ST7 interview, emergency admissions, Trust 1) “And there’s ( AKI stages ) 1, 2 and 3, I can't really remember the difference between the three of them but if it flags up something I go, ‘oh, okay, there’s something different here’”. (F1 interview, general/vascular surgery, Trust 3)
Contextual integration Mean survey score: 3.32	Clinicians did not report any specific resource requirements for the AKI e-alert other than training and time. Management support (where considered in the capacity of those responsible for e-alerts; the laboratory) was not identified by participants.	“I’ve no idea(who has responsibility for the AKI e-alerts), no. I assume somebody will do but I don't know, it’s not been communicated”. (F1 interview, general/vascular surgery, Trust 3)
Reflexive monitoring
Systematisation Mean survey score: 2.5	Feedback was never provided to staff on the effect of the AKI e-alert.	“I haven’t had any feedback since the new version (of the AKI e-alert) went in actually(…) I don’t know whether there is a formal mechanism for that getting to anyone”. (Pharmacist interview, Trust 1)
Communal appraisal Mean survey score: 3.39	The e-alert was rarely (if ever) discussed among clinicians, but participants often stated they felt that others would find it worthwhile.	“Most people I'm sure would know it's a good idea having them. That's what I'd say to someone about these alerts”. (Consultant interview, emergency admissions, Trust 1)
Individual appraisal Mean survey score: 3.52	While a small minority of clinicians felt the AKI e-alert had no effect on their work, many did but placed the effect within constraints relating to edge-case scenarios where AKI was most likely to be missed. These included marginal AKI thresholds within ‘normal Cr range’, busy workloads and AKI presenting in patients with chronic kidney disease. The pop-up e-alert was sometimes perceived to be intrusive, while the passive e-alert was often described as being too easy to dismiss.	“Speaking to a doctor, they felt that the AKI alerts are very useful. He says that if he sees an alert then he’ll check the patient’s renal function. He also explains that things at handover will often get missed so doesn’t always know that the patient will have an AKI”. (Trust 1 observation of emergency admissions, approx. 19:45 hours)
Reconfiguration Mean survey score: 3.89	Clinicians often did not know who was responsible for the AKI e-alert. They would never consider providing feedback about the e-alert, and there was no formal mechanism for doing so.	“I’m not sure if there is a feedback mechanism. If there is, I’m not aware of it”. (F1 interview, emergency admissions, Trust 1) “( To provide feedback ) I would ring IT and they would probably be very unhelpful and I would give up at that point”. (ST2 interview, internal medicine/care of the elderly, Trust 1)

AKI, acute kidney injury; e-alert, electronic-alert; NPT, Normalisation Process Theory.

Subconstructs demonstrating positive normalisation

Internalisation

Clinicians often reported that, despite not always utilising the AKI e-alert, they valued the potential of it, which was reflected in the survey score of 4.16. This demonstrated that they had a fundamental understanding of the importance of recognising AKI early, and many clinicians recognised that it was possible to make mistakes and to miss AKI.

in something like renal function, where there’s so much variety, (the AKI e-alert) just helps jolt you to it and especially how severe AKIs can be, it’s even more necessary because hopefully things like that wouldn’t be missed, but there’s always the potential that it could be. And having it say in black and white, this is an AKI, you know, they shouldn’t be missed at all. (F1 interview, emergency admissions, Trust 2)

The times I think it’s probably useful is when it’s one of those slightly sneakier ones, more subtle ones. The creatinine might have only peaked at 120 but, actually, if their creatinine is normally 45, that’s still a big deal but it doesn’t jump out at you as a creatinine of 600 would. (Consultant interview, internal medicine/care of the elderly, Trust 2)

Legitimation

Despite the lack of initiation (as identified in the ‘Initiation’ theme), perceived or otherwise, clinicians still largely understood that responding to the AKI e-alerts was their responsibility, although this perspective was sometimes dependent on the clinician’s seniority. For instance, all clinicians regardless of seniority recognised that the AKI e-alert was important to the work of junior doctors. In particular, some senior staff (consultants and registrars) felt that junior staff did not place sufficient priority on renal function; ‘For (junior staff) it might make a difference because they might not look at all the figures. If it says an AKI e-alert, then they might make the effort to actually do that’ (ST6 interview, general/vascular surgery, Trust 3). However, particularly on surgical wards where foundation-year doctors were mostly responsible for ward-care of patients, the e-alerts were not seen to be part of the senior doctor’s role, even though the AKI e-alerts were still valued.

I think you’ll find that as people progress, their focus of how they manage the patient shifts. They’re more interested in dealing with the active problems and these outcomes of quite secondary issues that solve around the problem. The attitude is a bit like mine: someone more junior will deal with it and you totally lose interest in the other things. (ST3 interview, general/vascular surgery, Trust 3)

Subconstructs demonstrating negative normalisation

Differentiation

How clinicians differentiated the AKI e-alert from what was deemed to be normal practice prior to the implementation of the AKI e-alert, was often based on the length of time that the clinician had been qualified. Clinicians who were newly qualified, particularly foundation-year doctors, consistently reported that they had no experience of working without an e-alerting system, and so using the AKI e-alert by default was deemed to be normal practice.

I suppose I haven’t ever realised it’s actually a new thing. Obviously, I’ve only worked here 11 months, I just assumed it was always there (F1 interview, general/vascular surgery, Trust 3)

Contrary to this, observations identified instances where clinicians were unaware of an e-alert for AKI, or were unaware of how the e-alert should work. In the following extract from observation notes, the clinician initially conceived of an e-alert as always being a pop-up, rather than text embedded into the system.

I chat with (a doctor) and we talk about the AKI alerts. When I explain what it is I’m observing for, he looks a bit confused, says he doesn’t know about the alerts. He opens up a patient record and explains he thinks this patient has AKI, so wants to see if there is an alert there. After I describe what the alert should look like, he says he thought I meant ‘a pop-up rather than a bit of text’; I think he doesn’t see the text as an alert by itself. (Trust 3 observation of emergency admissions, approx. 17:00 hours)

Clinicians also identified that the lack of differentiation was related to the clinical area in which they were working. For instance, it was deemed to be routine to check renal function of all patients entering emergency admission suites. In this setting, clinicians often mentally risk-assessed patients for AKI. For these patients, the clinicians would more regularly check to see if blood test results had been returned.

At the moment, probably not an awful lot else than I would normally do. Normally if I go through people’s bloods specifically for renal function I usually click on each of the numbers and compare it to what it has been previously. I think I interpret renal function quite a lot in the context of what the patient’s renal function IS? Or (sic) usually like. I click on each of the five elements that we get reported here and then have a look at how that varies from the previous. To be honest I would do that irrespective of whether the alert is there or not. (ST1 interview, emergency admissions, Trust 3)

Initiation

Initiation received a mean score of 2.8. This remained consistent across all three Trusts, and was supported by interview participants who consistently reported that either the e-alerts ‘just appeared at some point’ (F1 interview, general/vascular surgery, Trust 3), or that the e-alerts were already implemented when they began working for the Trust, as identified in the differentiation theme. However, there was a key difference; even where alerts were already implemented and thus deemed to be ‘normal’, there was a lack of training provided to clinicians on how to use the e-alerts. This finding was consistent (and is partly duplicated) with the skill set workability subconstruct of NPT.

I think (the AKI e-alerts) just started popping up. So, we didn’t get any training or anything like that on them, or why they were there, or who put them there, or what the purpose was. (ST2 interview, internal medicine/care of the elderly, Trust 1)

In one Trust, the person who contributed to the implementation of the AKI e-alerts acknowledged this suboptimal initiation or training for doctors; “When we first went live we switched the rules on but we didn’t really do a lot of education, and I think (the alerts) were relatively unpopular” (Pharmacist interview, Trust 1). Education consisted of an email with information about the AKI alerts to clinical directors asking them to cascade it to their staff.

Skill set workability

The lack of training provided on how to use the e-alerts, as previously reported in the initiation theme, also contributes to the skill set workability theme. Participants reported that they generally had responsibility for AKI and thus the e-alerts, demonstrating to an extent that that there was appropriate skill set workability among those receiving the e-alerts. However, there were also occasions where participants demonstrated or recognised their own lack of knowledge related to the AKI e-alerts such as incorrectly describing how they thought the e-alerts worked. More specifically, participants regularly did not know how the e-alerts should be incorporated into their own practice.

A teaching session would be really good of explaining, like, how to use the alert, like, the situations when the alert isn’t effective and, then, just, kind of, what to do if you do get an alert. (Consultant interview, internal medicine/care of the elderly, Trust 2)

Systematisation

Across all three of the NHS Trusts, no participants (regardless of seniority) collected information on the effectiveness of the AKI e-alert. While the data collected did not indicate whether anyone in the Trusts collected information regarding the effectiveness of the AKI e-alerts, it was consistently reported by all interview participants that feedback was not given to those using the AKI e-alerts. Furthermore, participants were unaware of whether the AKI e-alert, or more specifically responding to the AKI e-alert, had any effect.

I think maybe a bit of feedback or a bit of education would help staff to engage with the AKI alert. So, feedback as to how things had changed since the alert was introduced. (…) Some sort of outcome measure would be quite interesting. That might, just to show people that it's actually having a benefit. (Consultant interview, internal medicine/care of the elderly, Trust 1)

Discussion

This is the first known mixed methods study to use NPT to investigate the implementation of AKI e-alerts,¹⁷ an area identified as being an international research priority.⁹ The findings of this study suggest that AKI e-alerts are somewhat embedded into routine practice in the English NHS Trusts studied, with several aspects of implementation indicative of positive or negative normalisation. Given AKI e-alerts are now mandated across primary and secondary care in England and Wales,⁴ these findings suggest that more consideration was needed for how the AKI e-alert could be integrated into existing healthcare processes to influence both individual and collective behaviours. Furthermore, the findings highlight how other healthcare systems, where AKI e-alerts are not mandated, could implement AKI e-alerts in the future to improve their use.

The two aspects that particularly promoted normalisation of the AKI e-alert were that the e-alert was seen to be a legitimate part of a clinician’s role, and clinicians within the study mostly recognised the potential benefits of using the AKI e-alert. This demonstrates insight among clinicians that AKI is a significant risk to patient safety,² and consequently clinicians understand the importance of early AKI detection and treatment,²⁸ which have been historically poor.³ This finding also suggests that, when operated raising awareness of AKI and AKI e-alerting is insufficient, when operated as a single strategy, in addressing the problem of poor AKI care. Instead, attention should focus on other aspects of implementation that could be improved.

One such aspect that required improving was initiation to the e-alerts, such as via Trust-specific training, which was lacking or of insufficient quality. This was demonstrated by a lack of knowledge among clinicians about what differentiates the stages of AKI, and how the e-alerts were expected to be used. The definition of AKI has been refined considerably over the past decade, partly in an attempt to reduce variation in practice,^{6 29} but our findings reflect previous studies which have identified gaps in AKI knowledge among medical staff.³⁰ Although education is important in improving AKI care,⁷ there is a gap between the objective volume of delivery of AKI teaching and end-users’ perception of its paucity.³⁰ This dissonance might also be consistent with an alternative interpretation to our findings, which is that the existing definition of AKI lacks intuition and/or clinical credibility. Clinicians in our study reported using the terms minor, moderate or severe, even when they knew the different stages as per the 2012 Kidney Disease: Improving Global Outcomes (KDIGO) guidelines⁶ as they felt it easier to communicate to others. This corresponds closely with our finding, that some clinicians had difficulty in recognising and prioritising AKI e-alerts. Little research has focused on how staff are educated about AKI e-alerts, but some tentative links have been made between effective education and successful implementation.³¹

Another area of implementation identified as needing improvement was the systematisation of the AKI e-alerts through implementing feedback to end-users of the e-alerts. There was no system for providing feedback to clinicians, despite a wide range of safety literature identifying the importance of providing this to people involved in the process.^32–34 NPT proposes that an intervention is normalised through agents’ continuous actions which are enacted over a sustained period of time and space.³⁵ As approval ratings for AKI e-alerts have been reported to reduce over time, giving feedback to those involved in the safety behaviour could slow, pause or even reverse the decline,^{36 37} and can be a transformative process that can lead to improved performance.³⁸

Alert fatigue or disruptions to workflow have been identified as barriers to implementation,^{5 10} and there were examples of these identified in this study. Both e-alert systems produced opposing perspectives on how or whether the e-alerts influenced workflow. It was however common for those receiving the pop-up e-alert to dismiss it instantly and comment on its intrusiveness, while those who received the passive e-alert commented on it not being intrusive enough and being too easy to ignore. This suggests that there is no one-size-fits-all e-alert presentation, and instead they may require tailoring to either the individual or clinical unit. However, the causes of these differing perspectives were unclear and require further research.

It was also notable that collaborative working in response to the AKI e-alert were dismissed or downplayed by participants. Implementation of a complex intervention, or of a simple intervention into a complex environment, requires social activity that results in joint action; agents’ continuous actions are enacted over a sustained period of time and space.³⁵ Using and incorporating the e-alert into practice was often perceived to be an individual action that did not result in or alter discussions among clinicians. Future research should investigate whether the individual nature of an intervention, such as AKI e-alerting, contributes to poorer implementation, and whether such interventions require more collaborative working to be built-in to improve optimality.

Limitations

First, the study was conducted in clinical areas where AKI incidence is high, which may limit the generalisability of the findings; it is unknown whether the e-alert would be more useful (and whether it would be more or less poorly implemented) in clinical areas where AKI incidence is lower and thus clinicians have lower contact time with the AKI e-alert. Second, the study was conducted in one region, and so implementation of the AKI e-alert may have been influenced by local networks. Finally, it is unknown whether the e-alerts had a quantifiable impact on AKI outcomes or staff actions, and so it is not possible to draw conclusions about the effectiveness of the AKI e-alerts studied as a result of implementation. However, the identification of perceived differences between the NPT mechanisms, including subconstructs that were successfully implemented, suggests that a more focused approach, aligned with Safety-II principles, could help to identify successful implementation. Investigating where AKI e-alerts have been successfully implemented on a larger scale would provide valuable lessons for future implementation of both AKI e-alerts and other e-alerts.

Conclusions

Clinicians recognised the value and importance of AKI e-alerts in their clinical practice, although not sufficiently for AKI e-alerts to be routinely engaged with. To further normalise and promote clinician engagement with AKI e-alerting systems, there is a need for tailored training on AKI and how to use e-alerts; feedback should, also, be routinely given to staff about their impact on outcomes. The findings of this study provide a potential explanation for conflicting data on the reported effectiveness of AKI e-alerting systems. The findings have the potential to inform future national improvements to the way in which AKI e-alerts are implemented in the NHS and could be transferred into other countries’ healthcare systems where AKI e-alerts have either not yet been implemented or where this has been suboptimal.