Using simulation to iteratively test and re-design a cognitive aid for use in the management of severe local anaesthetic toxicity

Abstract

Introduction

Cognitive aids, such as a guideline for the management of severe local anaesthetic (LA) toxicity, are tools designed to help users complete a task. Human factors experts recommend the use of simulation to iteratively test and re-design these tools. The purpose of this study was to apply human factors engineering principles to the testing and iterative re-design of three existing cognitive aids used for the management of severe LA toxicity and to use these data to develop a ‘new’ cognitive aid.

Methods

Twenty anaesthetist–anaesthetic assistant pairs were randomised into four groups. Each of the first three groups received one of three different existing cognitive aids during a standardised simulated LA toxicity crisis. Postsimulation semistructured interviews were conducted to identify features beneficial and detrimental to the format and usability of the aid. Synthesis of the interview data with established checklist design recommendations resulted in a prototype aid, which was subjected to further testing and re-design by the fourth group (five more pairs) under the same conditions thus creating the final iteration of the new aid.

Results

Features of the new aid included a single-stream flowchart structure, single-sided, large-font design with colour contrast, simplified instructions and no need for calculations. This simplified tool contains only the information users reported as essential for the immediate crisis management.

Conclusions

Utilisation of formative usability testing and simulation-based user-centred design resulted in a visually very different cognitive aid and reinforces the importance of designing aids in the context in which they are to be used. Simplified tools may be more appropriate for use in emergencies but more detailed guidelines may be necessary for training, education and development of local standard operating procedures. Iterative simulation-based testing and re-design is likely to be of assistance when developing aids for other crises, and to eliminate design failure as a confounder when investigating the relationship between use of cognitive aids and performance.

Introduction

Severe local anaesthetic (LA) toxicity causing cardiovascular collapse is a rare, life-threatening emergency. It occurs when maximum safe doses are exceeded, for example, due to inadvertent intravascular injection, resulting in LA being rapidly absorbed into the circulation causing systemic toxicity. Prolonged resuscitation may be required, including the use of cardiopulmonary bypass in refractory cases. A more recent treatment involves administration of lipid emulsion or ‘intralipid’. Although clinicians will be familiar with this crisis in theory, few will have ever seen an actual case due to its low incidence.¹ To assist in the management of this crisis, various cognitive aids incorporating the use of intravenous lipid emulsion have been developed.^2–4 Cognitive aids are tools (eg, task cards, posters, handbooks, checklists, flowcharts) designed to help users complete a task or series of tasks, and are intended for use during the completion of that task.⁵ Cognitive aids have been developed in a wide range of industries, including healthcare, in an effort to decrease the prevalence of error,^6–8 especially in emergency settings when people are vulnerable to forgetting facts and skipping crucial steps.^9–12

The use of cognitive aids in the management of simulated anaesthetic crises has been shown, in some studies, to correlate with improved performance in the perioperative setting.^{6 7} Anaesthetists have reported high satisfaction with the help provided by cognitive aids during actual management of rare, high-mortality adverse events.¹³ However, the use of cognitive aids in simulated emergency situations has also been associated with inadvertent misuse leading to incorrect or delayed management, which may lead to harmful effects.^{14 15} Similarly, aids that fail to account for practical considerations, such as the availability of equipment, are prone to cause difficulties in implementation, that is, cognitive aids have not always been shown to improve clinical performance.^16–18 Our own observations of participants in simulation-based anaesthesia courses suggested that people struggled, even when using a cognitive aid, with lipid emulsion dosage calculations and made errors, often in the order of magnitude of 10 to 20 times. We also observed participants consciously elect not to use the cognitive aid at all and, subsequently, ‘guess’ the dosage. These observations triggered the local revision of the LA toxicity aids in use at the time. The revised aid removed the need for calculations and was developed using a process of iterative testing and re-design but not including the use of simulation-based testing.

The stimulus for this study was the observation that this revised aid did not appear to help in a simulated crisis even though it had appeared useful in a non-emergency setting. This led us to look more deeply at the design of cognitive aids for use in medical emergencies, specifically whether a simplified tool might be more appropriate for use during an emergency.

While medical devices must undergo appropriate testing to ensure functionality and safety before actual use, cognitive aids are not currently assessed against agreed criteria in terms of functionality or safety before publication.¹⁹ The reasons for this are unclear, but are likely to include a relatively low level of awareness of human factors engineering (HFE) in healthcare, and/or resource constraints associated with iterative testing and re-design. In a review of published anaesthetic cognitive aids, of the 22 aids studied, only one aid was developed iteratively as is the standard for other medical devices.⁵ This review suggests that the majority of published cognitive aids are not tested in the context in which they are to be used. That is, they are not routinely subjected to usability testing, which may include iterative testing in response to simulated emergencies; instead they are often created via expert consensus, or according to ‘work-as-imagined’ (an idealised view of the task environment that does not consider how work is actually done in the ‘real world’).²⁰ Some of these aids may be useful during training sessions to prompt discussion and assist with planning but they may be unsuitable for use in a crisis. Simulation-based testing with a high degree of task, equipment and environmental fidelity will tend to invoke ‘work-as-done’ rather than ‘work-as-imagined’ and thus has the potential to result in the development of tools that are fit for purpose.²⁰

Guidelines for the design of cognitive aids are available based on work in other fields of endeavour that may have applicability for healthcare.^{9 12 19}HFE is a discipline concerned with the ‘design of tools, machines, and systems that take into account human capabilities, limitations, and characteristics’.⁹ In healthcare, HFE is concerned with understanding and improving patient safety including application of a user-centred approach to the design of tools such as cognitive aids. A key characteristic of a user-centred design approach is iterative design and testing.⁹ For example, a cognitive aid presented in paper format may be subjected to iterative testing of legibility and readability of text using appropriate fonts, size, typeface, colours and spacing.¹⁹ Formative usability testing identifies design strengths and opportunities for improvement, whereas summative usability testing determines how well the design performs.²¹

The aim of this study was to conduct formative usability testing of three existing cognitive aids in order to develop, via a user-centred design approach, a new cognitive aid for use in the management of LA toxicity. This new aid is intended to be suitable for all anaesthetist users regardless of clinical experience and prior knowledge of the management of LA toxicity.

Methods

Ethics approval was sought from the Hunter New England Local Health District Human Research Ethics Committee. The committee issued a waiver for the study as they determined it to be a quality improvement activity according to local policy.

All John Hunter Hospital anaesthetic personnel including trainees and anaesthetic assistants were invited to participate in this study via an email containing information about the study. A informed, written consent was obtained from the volunteers on the day of the study. Volunteers were advised that data would be de-identified and results presented in aggregate format only to preserve anonymity.

A random integer generation service (www.random.org) was used to create a randomisation sequence to divide participants into one of four experimental groups differing by the cognitive aid to be supplied. Volunteers were paired into anaesthetist–anaesthetic assistant dyads and then randomised into the groups using sealed envelopes containing the randomised group numbers.

The study had two phases, the first involved selection of and formative usability testing of existing aids and the subsequent generation of design recommendations based on observations, interviews and accepted principles of design for checklists. The second involved user-centred design, specifically the iterative design and re-testing of a prototype in order to develop a new cognitive aid (see figure 1).

Figure 1.

Diagram summarising the methodology of the cognitive aid testing and the new aid development.

The cognitive aids chosen for testing by the first three groups were the following:

1. Association of Anaesthetists of Great Britain and Ireland (AAGBI) guideline ‘Management of Severe Local Anaesthetic Toxicity’, first edition, 2007 (aid 1).²

2. AABGI guideline ‘Management of Severe Local Anaesthetic Toxicity 2’, second edition, 2010 (aid 2).³

3. A flowchart based on the information contained in the 2007 AAGBI guideline (aid 3) (see figure 2). Local clinicians, including one of the authors (CM), developed aid 3 and it had been added to the LA toxicity box kept in the operating theatres in 2008. Some principles of user-centred design were applied to the design of this aid, for example, iterative testing and re-design, but it had not been tested in a simulated crisis.

Figure 2.

Aids tested in phase 1, displaying the front and back pages. Aid 1— Guidelines for the Management of Severe Local Anaesthetic Toxicity, AAGBI, first edition, 2007. Aid 2—Management of Severe Local Anaesthetic Toxicity, AABGI, second edition, 2010. Aid 3—locally developed guideline Management of Severe Local Anaesthetic Toxicity, John Hunter Hospital, 2008.

The aids selected for testing were chosen because they were either published or in local use at the start of the study period. The authors were not aware at the time of the study of any other published LA toxicity aids. Staff had not received formal training in the use of any of the aids. The fourth group of pairs were randomised to test the new aid to be developed in phase 2 of the study.

Setting

The Hunter New England Simulation Centre is a purpose built facility with 10 full-time equivalents of teaching, technical and support staff providing training for 4000 clinicians annually. It is housed on the John Hunter Hospital campus, which contains two large tertiary level facilities serving the Hunter New England Local Health District (catchment population 850 000).

Phase 1: Testing of existing LA toxicity cognitive aids

Participants were told the study was about ‘the role of cognitive aids in the management of anaesthetic crises’ but the nature of the crisis was not disclosed prior to the simulation. Participants were asked not to discuss the study with others in the department to minimise bias in upcoming volunteers.

Each pair of volunteers received a brief orientation to the simulated environment, equipment and mannequin prior to their participation in the scenario. All pairs participated in a standardised scenario using the same simulated case, the same environment and equipment set-up including mannequin (SimMan 3G, Laerdal Medical, Stavanger, Norway) and the same simulation centre staff playing the roles of other theatre personnel. The simulated case was based on a reported case of ropivacaine-induced cardiac arrest due to systemic uptake of LA following a single-shot axillary plexus block.²² In this case report, ropivacaine 1% 40 mL was mistakenly injected intravenously during an axillary plexus block, resulting in the patient suffering a cardiac arrest. Once the crisis was declared, the anaesthetic assistant was prompted by the operating theatre staff (role played by simulation centre staff) to use a LA toxicity box that contained one of the aids. All three aids were tested five times by a different group of five pairs, that is, 15 different pairs tested one aid each so that of these three aids was tested five times. The anaesthetist, once prompted, could choose to use or ignore the aid. Further information about the scenario is provided in online supplementary appendix 1.

During the scenario, one of the researchers (DD) observed and categorised the duration of time spent using the cognitive aid (reading it silently and/or reading it aloud) as follows: -

• Aid looked at but discarded immediately

• Aid used only once briefly

• Aid used repeatedly throughout crisis

After the scenario, participants were first debriefed to defuse any emotional responses to the scenario, and to discuss technical and non-technical issues arising. The model of debriefing used was ‘Debriefing with Good Judgement’ which includes advocacy inquiry.²³

Participants, in their pairs, then took part in a semistructured interview (see online supplementary appendix 2 for list of questions), conducted by one of the researchers (DD). The questions were developed by synthesising data analysed from pilot interviews with a review of the literature. This explored how participants interacted with the cognitive aid and the ways in which it provided assistance or hindered their management. All three cognitive aids (tested five times each) were available in the interviews as prompts for discussion. Prototypes of aid 4, based on analysis of interview data from testing of aid 1, were shown to those testing aid 2, and the subsequent iterations of aid 4 were shown to those testing aid 3. Interview data were analysed and categorised iteratively to identify key concepts and themes. Field notes were taken by the interviewer in the data collection phase and later discussed by the research team in the analysis phase. As the analysis proceeded, notes were taken during meetings and exchanged by email between team members. These notes related to the discussion and identification of themes, subthemes and, eventually, recommendations to be incorporated into the prototype and subsequent iterations of aid 4.

There was no predetermined sample size for the study. Instead, sample size was determined by reaching signals of saturation. These predetermined signals included repetition of information and confirmation of existing concepts identified during interviews, and the failure to uncover new themes and concepts in subsequent interviews. We anticipated that at least five pairs would need to be interviewed before no new discussion themes emerged and so agreed in advance to analyse five pairs at a time.

Phase 2: User-centred design of a new cognitive aid

The data collected in phase 1 were synthesised with information published in design guidelines for cognitive aids to produce aid 4 (See figure 3). Prototypes of aid 4 were shown in the postsimulation interview to participants who tested aids 2 and 3, and these interview data were incorporated into subsequent versions. The final iteration of aid 4 was then subjected to testing in the simulated crisis and iterative re-design cycles based on ongoing analysis of data collected by interview.

Figure 3.

A photo of the entire human factors engineering designed cognitive aid for management of severe local anaesthetic toxicity. To avoid calculations, the aid has a tab with weights in 10–20 kg increments. The actual instructions and dose dosage were individualised on each page of the aid, allowing users to simply flip to the required tab.

Results

Study participants

Forty volunteers comprising 20 anaesthetists (six intermediate level registrars (2–3 years anaesthesia training), seven senior registrars (3–5 years anaesthesia training) and seven consultants) and 20 anaesthetic assistants were recruited and paired into 20 anaesthetist–anaesthetic assistant teams. Sixty per cent had prior experience as a participant in immersive simulation, 48% had prior education (reading/lectures/tutorial) on LA toxicity and 5% (n=2) had previously managed a simulated LA toxicity crisis.

Fifteen anaesthetist–anaesthetic assistant pairs participated in phase 1, with three groups of five pairs testing each of the chosen aids. A further five pairs (ie, 20 pairs in total) were randomised into a fourth group which participated in phase 2.

Signals of saturation appeared in the analysis of the first five pairs for all four groups so no further participants were recruited to the study.

Phase 1: Testing of existing LA toxicity cognitive aids

The three aids initially tested were used (ie, reading silently or reading aloud) only once, or were discarded immediately (table 1).

Table 1.

Summary of phase 1 results

Aid tested	Usage			Dosage	Preference
	Discarded immediately	Used only once	Used repeatedly	Correct bolus and infusion rate	Pairs that preferred the aid they tested	Preferred aid indicated in interview
1	One pair	Four pairs	0	One pair	0	Aid 3 (100%)
2	One pair	Four pairs	0	One pair	0	Aid 4 (100%)
3	One pair	Four pairs	0	0	0	Aid 4 (100%)

All participants eventually gave lipid emulsion but not necessarily in the correct dose.

All three aids were used as prompts for discussion during the postscenario interviews even if they had been discarded in the simulated crisis. Participants were asked which aid they would prefer to use in future; none preferred the aid they had used in the scenario. All those who tested aid 1 stated in the interview they preferred aid 3. These participants did not see the prototypes of aid 4. However, in the interviews for those who tested aids 2 and 3 (ie, those who were shown prototypes of aid 4), all stated they preferred aid 4.

The key themes emerging from the interviews for each of aids 1, 2 and 3 are presented in tables 2, 3 and 4. The themes, subthemes, reported impact and specific recommendations for incorporation into the next iteration of aid 4 are reported in these tables.

Table 2.

Key themes emerging from testing of aid 1

Key themes (aid 1]	Subthemes	Reported impact	Examples	Recommendation to be incorporated into aid 4
Physical characteristics (font size and type, use of colour, contrast)	Colour (poor use of) Only looked at the red	Eye drawn only to red, everything else ignored	"There was writing everywhere…I actually only looked at the red and ignored everything else…"	Sparing use of red to focus attention; use of black and white for maximum contrast and clarity
	Text dense without enough white space Font too small	Aid quickly discarded because too hard to find what was needed (drug dose)	"This was a bit frightening to look at…there was too much text and I couldn’t use it"	Maintain white space; avoid clutter; use larger font
Layout	Narrative text	Information missed; aid discarded Difficult to work out timeline and steps required ‘next’	"I had no idea the resuscitation might take an hour or so" "I knew there were would be steps to follow after the initial bolus but didn’t register what they would be."	Flowchart desirable to communicate sequence of steps
	Two sided	Information missed	"…I didn’t even realise there was information on the other side…"	Single-sided desirable
Content	Volume of information	Took too long to find information needed to treat crisis so discarded aid	"I found all the background useful in an educational sense when I looked at it afterwards but it wasn’t relevant at the time…"	Background information not immediately relevant to the crisis removed and presented elsewhere
	Relevance of content	Overwhelmed with information that did not seem useful at the time	"I read the immediate management aloud by way of an initial checklist which was good but I didn’t use it after that. Information about signs and symptoms wasn’t helpful by the time he’d arrested."	Limit information presented on emergency use aid to that immediately required
	Title not obvious	Purpose of document not immediately apparent	"At first I thought they were just guidelines for information, not for actually using in the crisis."	Title that clearly articulates purpose
	Instructions	Instructions too wordy Dosing information unclear, wrong dose given	"I just gave the dose in red (100 mL)…I didn’t even realise it was supposed to be a per kg dose…I thought it was for all adults."	Provide absolute doses for different weights Use simple instructions (eg, give intralipid 120 mL intravenous bolus)
	Units impractical	Dosing information unclear, wrong dose given	"Our pumps use ml/hour so I had no idea what to make of 0.25 mL/kg/minute."	Use practical units that match infusion pump settings

Table 3.

Key themes emerging from testing of aid 2

Key themes [aid 2]	Subthemes	Reported impact	Examples	Recommendation to be incorporated into aid 4
Physical characteristics (font size and type, use of colour, contrast)	Colour (poor use of) Only looked at the red	Eye drawn only to red, other information ignored	"There was so much red…too much. I couldn’t focus on any one thing or work out what the priorities were."	Sparing use of red to focus attention
	Text dense without enough white space; font too small	Hard to find relevant information; aid discarded	"The front looked too busy and I couldn’t spend time on it."	Larger font, uncluttered appearance
Layout	Flowchart with both left → right and top → bottom streams	Information missed; too many streams resulted in divided attention	"The layout was really confusing. I missed the need for the infusion to be run in parallel because of this."	Single stream desirable
	Two sided	Information missed including dosing information	"There was so much information on the front that I put it straight down. I didn’t see the stuff on the back like the dose… it would have been really useful if I’d seen it because I was totally reliant on the aid to tell me the dose."	Single-sided desirable
	Ordering of steps	Information missed, aid discarded	"The most useful information was right at the bottom on the back – I didn’t notice this until afterwards."	Flowchart desirable to communicate sequence of steps including priorities and timeline
Content	Volume of information	Took too long to find information needed to treat crisis so discarded aid	"This was unusable in the crisis, there was too much information and it was confusing."	Background information not immediately relevant to the crisis removed and presented elsewhere
	Relevance of content	Information not relevant during crisis was distracting; no clear treatment end points; incorrect dose given	"I had to focus on the resuscitation…I couldn’t access any other information (apart from the initial dose)…I didn’t realise I might have needed to continue for 60 min." "I knew I needed to give lipid and all I wanted was the dose. I couldn’t find it." "I completely disregarded the bottom box on follow up."	Limit information presented on aid to that immediately required in emergency Use of clear end points, for example, resuscitation may take 1 hour
	Need for calculations	Incorrect dose given	"I found it impossible to do even simple calculations with everything else going on."	Provide absolute doses for different weights

Table 4.

Key themes emerging from testing of aid 3

Key themes (aid 3)	Subthemes	Reported impact	Examples	Recommendation to be incorporated into aid 4
Physical characteristics (font size and type, use of colour, contrast)	Colour (good use of) Bright colours to focus attention	Eye drawn to things that were important in the brightly coloured boxes; helped prioritise actions	"I knew the things in the orange boxes were the priorities…"	Sparing use of red useful to focus attention
	Colour (poor use of) Too much colour distracting	An abundance of colour (reverse side) was overwhelming; incorrect dose given; aid discarded	"It wasn’t obvious how to access the dose information from the table until afterwards. The colour was distracting." "Afterwards I realised I had read out the dose for a 10 kg patient and the anaesthetist gave it."	Use of black and white for maximum contrast and clarity
	Use of white space, bold and font size	Uncluttered appearance (front side) helped prioritise steps	"The bolding helped make the priorities clear"	Use of bold to focus attention
Layout	Flowchart (good use of)	Provided a timeline for the management including timing between actions (eg, do this then 5 min later do that) and when to escalate treatment; useful information about end points	"There was a clear pathway to follow." "I knew straight away that I might need to continue this resuscitation for 1 hour, that helped my decision making."	Flowchart desirable to communicate sequence of steps
	Flowchart with both left → right and top → bottom streams (poor use of)	Too many streams to flowchart; information missed (eg, need for prolonged resuscitation); actions missed (eg, delay to give epinephrine); priorities unclear to some (eg, prioritised intralipid at expense of continuing ALS in parallel)	"I didn’t get past the middle column once I’d found what I was looking for there. This meant I didn’t notice the bit about prolonged resuscitation." "I neglected the advanced life support management in favour of giving intralipid."	Single stream desirable
	Two sided	Information missed on reverse including dose information; for those who noticed both sides they weren’t clear which was front and which was back	"I didn’t realise the absolute doses were on the back until afterwards so I just guessed the doses based on the front side."	Single-sided desirable Use page numbers if more than one side/page
Content	Need for calculations (when only front side used)	Incorrect dose given		Provide absolute doses by weight
	No need for calculations (when reverse side used)	Correct dose given	"I liked the table with the per kilo doses."	Provide absolute doses by weight
	Impractical units	Incorrect dose given	"The front side (which is all I used) only had the infusions in ml/kg/min"	Use practical units that match infusion pump settings

A consistent theme in all the interviews was that many features of the existing cognitive aids appeared useful in a non-emergency situation but those same features were detrimental in a simulated crisis. For example, the potential need for prolonged resuscitation was readily apparent when people looked at the aid without time pressure, but in the simulated crisis this fact was often overlooked. Similarly, the ability to rapidly identify the relevant information and compute doses was easy in the non-crisis setting but often not possible in the simulated crisis. This led to the aid being discarded and participants relying on memory, which resulted in incorrect doses being given.

At the end of phase 1, several design features were identified as contributors to each cognitive aid’s ability/inability to achieve its desired aims during a simulated anaesthetic crisis (table 5). These recommendations were incorporated into the final version of aid 4, i.e. the ‘new aid’.

Table 5.

Features of the local anaesthetic toxicity cognitive aids tested that were identified as beneficial and detrimental

Beneficial features

Detrimental features

A single stream flowchart structure A single-sided page with large-font design Use of simplified instructions, for example, a single command like ‘bolus 120 mL over 1 min’ Use of colour for contrast and to focus attention Use of black text on white background for maximum contrast and clarity Limiting information to that required during the crisis Absolute doses by weight and avoidance of need to perform calculations

Text dense aids Instructions containing multiple commands at each step Multiple streams in a flowchart (ie, need to go left to right AND top to bottom) Two-sided presentation Excessive use of red font Need for user to perform any calculations (regardless of how simple the calculations might be) Inclusion of information not relevant to treatment of the crisis

Beneficial features were those that led to rapid access to relevant information with minimal distractions, and/or gave participants an indication of the management steps required including a timeline and clear end points.

Detrimental features were those that led to key steps or information being missed, misinterpretation of information (wrong dose) and/or the aid being discarded. For example, one of the anaesthetic assistants looked at aid 2 and could not find the relevant information (ie, dose) on the front and so discarded the aid without realising there was information on the reverse. The anaesthetist subsequently gave a subtherapeutic infusion dose of 0.25 mL/kg/hour instead of 15 mL/kg/hour (0.25 mL/kg/min) because they relied on their memory.

Phase 2: User-centred design of a new cognitive aid

The desirable features identified in phase 1 data analysis (table 5), and in published guidelines for checklist design,^{12 19} were incorporated into aid 4 (see figures 3 and 4). This aid was designed for use in paediatric and adult patients.

Figure 4.

A copy of the page from the human factors engineering designed cognitive aid, aid 4, for the management of local anaesthetic (LA) toxicity, for an 80 kg patient. ALS, advanced life support; CPR, cardiopulmonary resuscitation.

All five pairs tested using prototypes of aid 4, referred to the aid for information repeatedly throughout the crisis. In the interview, participants were shown aids 1, 2 and 3. All participants preferred aid 4 and all participants gave lipid emulsion with four of five pairs giving the correct initial bolus dose followed by the correct starting infusion rate. One of the pairs initially gave the incorrect bolus dose (see below) but this was corrected and followed by the appropriate infusion dosing.

Further changes were made to the aid 4 prototype based on observations during testing and analysis of the postsimulation interview data in phase 2 of testing. The pair who used the first iteration of aid 4 incorrectly gave a paediatric dose (based on a weight of 10 kg instead of 80 kg) to the adult ‘patient’ due to confusion about how to find the dose by weight. The final iteration of aid 4 became the ‘new aid’. The physical format of the aid was amended to incorporate tabs along its edge designed to direct a user to the correct weight-based page. Coloured warnings on pages intended for paediatric patients were included to prevent dose errors.

Discussion

Application of human factors principles, in particular features of user-centred design such as iterative testing and re-design, led to the development of a cognitive aid that was visually very different from the existing aids. Despite each of the three tested existing cognitive aids appearing intuitive and easy to understand in a non-emergency situation, participants were unable to reliably access relevant information in a simulated crisis and the aids were reported to be not useful, and in many instances were discarded immediately. Aid 4 (ie, the new aid), which was used repeatedly by participants, was reported to be easy to use without prior familiarity or training though it should be noted this does not negate the need for education on the use of specific cognitive aids.⁵While an aid should be intuitive to use, the literature suggests that familiarity and/or ongoing education on specific aids may make them more likely to be used appropriately in crises.^{24 25}

Features of the new, preferred aid, identified during testing, such as use of colour, providing a single stream of information (ie, a linear flow rather than a branched flow) and limiting the information to that required during the crisis are consistent with recommendations made by HFE experts and in recent studies.^{12 26} A prototype of aid 4 included information about recognition of LA toxicity but during the iterative testing and re-design process it became clear that participants found this distracting.

The new aid has been introduced into the LA toxicity boxes displayed prominently in the operating theatres of the organisation. The aid has also been incorporated into the organisational guideline on the use of lipid emulsion for the management of severe LA toxicity. Aid 2, developed by an anaesthesia body, contains more detailed background information and has been retained in the guideline for training and education purposes. Staff are familiarised with the aid during their hospital orientation and in ongoing training sessions, for example, operating theatre team training in the simulation centre. Educators and end users have been encouraged to make contact with the authors with any feedback. The guideline is reviewed regularly and it is intended that feedback from these sources will be analysed and incorporated into a future revision of the aid.

This user-centred design study will be followed by simulation-based usability evaluation to collect outcomes data focused on the impact of the aid on task completion (accuracy, patterns of usage) and task efficiency (measures of workload and time to completion) as well as quantitative data on user preference.

Limitations

1. Study participants included junior trainee and senior consultant anaesthetists. This range of experience may have affected how participants managed the crisis. However, given the purpose of the cognitive aid is to support clinicians of all levels of experience it was important the aid was tested by a range of potential users.

2. We did not collect quantitative supporting validity evidence; the primary focus of the study was on the design of the aid. Collecting validity evidence for the effectiveness of the new aid would be an appropriate next step for further research as described above.

3. Using simulation for formative usability testing and user-centred design as described in this study is resource intensive. Further work is required to help determine which type of tools will benefit the most from the application of simulation-based techniques.

Design of cognitive aids

Design guidelines exist in the literature for the development of cognitive aids, particularly for checklists (a type of cognitive aid),5 9 12 17 including a recently developed tool, the ‘Cognitive Aids in Medicine Assessment Tool (CMAT)’.²⁷ This tool may provide a useful quality assurance step for tools already in use as well as offer some guidance for the development of future tools. The CMAT is adapted from the UK Civil Aviation Authority Checklist Assessment Tool. The use of such a tool does not replace the importance of evaluating a cognitive aid in the context in which it is to be used, for example, by testing an aid in the management of simulated crisis. In this study, this is evidenced by the differences in reported usability of the different aids in a simulated emergency, and the differences in visual presentation between the aids tested (ie, those already in use clinically) and the new aid. Aid 3 was iteratively tested and re-designed but not evaluated using simulation; it is unclear how aids 1 and 2 were developed.

A recent review of anaesthesia cognitive aids found little evidence that aids in current widespread usage had been designed for the context in which they are used.⁵ Marshall suggests the ideal process for development of a cognitive aid prior to introduction into clinical usage is one in which content is determined by expert consensus, and design proceeds with the application of user-centred design principles including iterative testing and re-design using simulation.⁵ Outcomes of studies unable to detect improved team performance through the use of cognitive aids in managing crises, or simulated crises, may have been confounded by the use of poorly designed cognitive aids.^{17 28} We recommend that the first step in such research should be a review of the aid design to ensure it is appropriate, for example, by conducting a heuristic evaluation using a tool such as the CMAT.²⁷ If the design is not appropriate, the next step should be to improve the design of the aid before attempting to measure the effect of the aid on team performance.

It is important to note we are not recommending that existing, more detailed aids should necessarily be discarded. They are likely to have a role in education and planning. We suggest that simplified aids for use in a crisis, when our ability to process information can be overwhelmed, are developed alongside more detailed aids and that the two should be viewed as complementary. The notion of two complementary methods of using cognitive aids was described by Reason²⁹ and, more recently, used as an analogy to help understand how to approach a recently revised cognitive aid for use in difficult airway management by Marshall.³⁰

The use of similar sound design principles, as recommended by human factors experts, is likely to be of assistance when developing aids for other crises, and to eliminate design failure as a confounder in studies investigating the relationship between use of cognitive aids and performance.