Helping experts and expert teams perform under duress: an agenda for cognitive aid research

In recent years, there has been a keen interest in how checklists and other cognitive aids can help clinicians during clinical crises. Studies from anaesthesia 1, 2, 3, and emergency medicine 4, 5 have shown that displaying cognitive aids during emergencies reduces omissions, time to perform tasks and improves team skills, communication and performance in most instances 6, 7, 8, 9. In research where no difference was found, the cognitive aids are almost always found to have been introduced without education or have flaws in their physical design 1.

Cognitive aids are ‘implementation tools’ used at the same time that the work is being performed. Consequently, cognitive aids used in emergency situations must be very different from those used in routine settings in both form and function because of the requirement of the content to be physically and cognitively accessible during times of stress 1, 10. Emergency aids should include only those points that are important or commonly omitted rather than provide a comprehensive ‘how‐to’ guide for adherence to local policies and procedures. In short, the function of emergency cognitive aids should be to support trained expert teams to remember and excel, rather than to help novices cope with situations beyond their expertise.

A study in this issue by Everett et al. 11 attempted to prove that cognitive aids can be used to improve the retention and performance of technical aspects of emergency management several months after initial exposure. While on the face of it there appears to be no effect of the aid, the poorer results when the teams did not have access to a cognitive aid in later scenarios could be interpreted quite differently. Education about how to use the checklists invariably includes content about technical performance such that the initial training raised the technical performance in the initial testing. The higher level of performance was only maintained when the cognitive aid was present, suggesting either a lack of ability to cope without the cognitive aid, or that the prompts helped the participants remember their training. This finding is almost identical to Ward's study 12 that showed undergraduate students’ technical performance in basic life support was retained at two months after training only when a cognitive aid was available. Similar problems with finding an effect immediately after education with cognitive aids have been found in other studies where retention was not measured 7. In contrast, the lack of a measurable effect on team performance was hampered by poor inter‐rater reliability and the use of a team measurement tool that was not validated for this context. Nevertheless, the study shows there are still lessons to be learned about the purpose and function of emergency cognitive aids and how we investigate them.

Knowledge gaps in cognitive aid research

Studies consistently show that at least 80% of clinicians report they would use cognitive aids if they were available during an emergency 6, 13, 14, 15, 16. However, the reality is quite different, with as few as 7% of users accessing the aid in some observational studies 13, 17. Barriers to emergency cognitive aid use include: the lack of a supportive clinical and educational culture; insufficient time; forgetting it was available; or poor design of the aid 14, 18, 19. Clearly, we can learn much about the required countermeasures to ensure widespread and effective use of emergency cognitive aids from these data.

Education with and about cognitive aids is an important component of them becoming accepted. A recent study by Goldhaber‐Fiebert et al. demonstrated that use during simulated emergencies and for self‐reflection are powerful methods that promote cognitive aid use 14. Ideally, education should include an orientation to how the cognitive aid is best used in relation to its design. Simulation training then allows the practical implementation in a context that closely resembles the actual clinical context. As in Everett et al.'s study 11, training with the aid can improve aspects of medical and technical management of emergency situations when the aid is used, even several months after the initial training 12. Anaesthetic simulation teaching has long advocated the use of cognitive aids during emergencies 20, however the lack of progress in uptake may reflect that they have rarely been a central component of such education or part of the educational culture. Further research is warranted to determine the most effective way of introducing and training with a cognitive aid to improve uptake and future performance.

Cognitive aids are rarely detailed enough by themselves for novices to learn all of the actions required to manage a crisis. Chrimes 21 distinguishes between ‘foundation’ and ‘implementation’ tools for both low acuity (routine) and high acuity (emergency) situations. Foundation tools are more detailed and lack the simplicity of design that must be present for a tool to be useful in times of stress. Clinicians may consider reading the detailed foundation documents prior to an emergency and crosschecking them with the implementation tools to mentally rehearse their actions in advance of a crisis. An example of this is the ‘DRSABC’ mnemonic as an implementation tool for the detailed Basic Life Support algorithm (Table 1) 22. Although the mnemonic provides a framework of priorities during the early stages of a cardiac arrest, it does not provide sufficient detail in itself of, for example, how to perform the actions such as check for danger, establish if the patient is responsive, or call for help. These specifics are commonly described in a detailed evidence‐based document 23 or a local procedure manual. By cross‐referencing the (foundation) manuals with the (implementation) cognitive aid during learning and then practising during simulation, the prompts provided become vastly more useful in an emergency.

Table 1.

The DRS ABCD algorithm for basic life support from the Australian and New Zealand Committee on Resuscitation (ANZCOR) flowchart 22

D	Dangers?
R	Responsive?
S	Send for help
A	open airway
B	normal breathing?
C	start cardiopulmonary resuscitation 30 compressions: 2 breaths
D	attach defibrillator as soon as possible, follow prompts

Which items to include?

If the number of items listed on the cognitive aid are to be limited to the essentials, how then do we decide what should be included? One early method of cognitive aid design by Runciman and Merry was to produce a comprehensive list of actions based on data from incident reports 24, 25. As a result, a standardised framework in the form of a mnemonic was created, the ‘COVER ABCD’ algorithm with an accompanying handbook (Table 2) 26. Despite initial enthusiasm for this approach, its uptake has ultimately been poor, perhaps because each letter represented multiple actions, making it difficult to implement in a crisis; in addition, it did not fit with the existing cognitive and decision‐making processes of anaesthetists in crises. An alternative method is to determine the items that are commonly omitted or need prompting such as drug doses and only include these in a cognitive aid. Observation of real life emergencies is of course difficult because of their rarity and unpredictability. However, observation of simulated cases has been used to inform design and content of cognitive aids. The Australian and New Zealand Anaesthetic Allergy Group (ANZAAG) and Australian and New Zealand College of Anaesthetists (ANZCA) cognitive aid for management of peri‐operative anaphylaxis was created from feedback and testing of volunteers in a simulation setting 7, 18. The testing revealed that the description of adrenaline doses was confusing, especially for paediatric patients, and fluid volume resuscitation was commonly inadequate. The second version of the ANZAAG/ANZCA cognitive aid addresses these deficiencies in its design by employing the preferred and more effective linear layout including detailed guidelines describing the most effective team structure observed during simulation 27. Another method of identifying the key decisions and omissions is by interviewing clinicians about emergencies they have been involved in using structured cognitive interviews 28. Critical Decision Method is one such technique 29, and can inform broader design and role allocation between team members to strengthen behaviours leading to improved outcomes 30. Again, this method of design is in its infancy in health with few current examples even outside of anaesthesia to learn from 31, 32.

Table 2.

The COVER ABCD (A SWIFTCHECK) algorithm for assessment and treatment of anaesthetic emergencies, after Runciman and Merry (2005) 24

Item	Brief meaning	Examples of actions
C	Circulation	Check pulse
	Colour	Check saturation probe
O	Oxygen	Ensure FIO2 is 100%
	Oxygen analyser	Cross‐check with monitor
V	Ventilation	Hand‐ventilate to check compliance
	Vaporiser	Check settings and for leaks etc.
E	Endotracheal Tube	Check capnography, pass suction catheter
	Elimination	Eliminate anaesthetic machine – use self inflating bag
R	Review monitors	Apply and observe all parameters
	Review equipment	Assess other equipment in contact with patient
A	Airway	Check patency of unintubated airway
B	Breathing	Auscultate chest and assess pattern and adequacy if breathing spontaneously
C	Circulation	Repeat evaluation, consider additional monitoring
D	Drugs	Consider if correct drugs have been given or an unintended consequence has occurred
A SWIFT CHECK	Awareness	Could this be awareness?
	Air embolism	Is the patient at risk of an air embolus?
	Allergy/Anaphylaxis	Are there signs of an allergic reaction?
		Check progress with surgeon, any other items related to patient or procedure

Practical use in clinical contexts

Another unanswered research question is where cognitive aids should be kept so they are accessible and remembered during a crisis 19. It is tempting to decorate the walls of the operating theatre with posters describing the management of every conceivable emergency, but these are still frequently forgotten in stressful situations. To promote their use, cognitive aids should be closely related and displayed near the equipment required to manage the situation. Examples include: taping an envelope with the cognitive aid action cards over the drawer containing the dantrolene on the malignant hyperthermia trolley 33; keeping an anaphylaxis box containing the cognitive aid cards on the cardiac arrest trolley 7; and attaching the local anaesthetic systemic toxicity guidelines to bottles of lipid emulsion on the labour ward epidural trolley 34. When booklets of cognitive aids for the management of a range of emergencies are created, this generates a new question: is it better to have all of the cognitive aids together in one place, or separately and related to the other emergency equipment that will prompt their use?

With the ubiquity of electronic devices and smartphones in the operating theatre, there is now an opportunity to transfer from paper‐based cognitive aids to electronic ones. The advantages would be real‐time updates of the content in keeping with best practice guidelines and perhaps the presentation of dynamic information relevant to the clinical problem such as with aviation style electronic checklists 15, 35. However, the designs of electronic aids need to be sensitive to the context that they are to be used in. Electronic aids could be a hindrance more than a help if additional hands are needed to click through the pages of an app, and may well inadvertently prompt incorrect actions 35, 36. Indeed, the evaluation of a cognitive aid for paediatric anaesthetic emergencies demonstrated that anaesthetists rated the electronic conversion as more cumbersome than the same paper‐based aid 37. Simple conversions from paper to electronic devices may well be flawed and research is needed to determine the ideal design of these electronic aids.

Even with the design of paper‐based cognitive aids, there are still questions to be answered. From early research, it would appear that the most common branched algorithm design seen in cognitive aids such as the ASA difficult airway algorithm 38 are more difficult to interpret than more basic linear designs 7. Best‐practice guidelines for cognitive aids have been applied from aviation by the development of a design specification; the Cognitive Aids in Medicine Assessment Tool (CMAT) 39. The CMAT assesses the contrast, readability, font size, type and structure of the cognitive aid in general terms. However, many seemingly effective cognitive aids including cardiac and neonatal resuscitation algorithms score poorly on this assessment, suggesting that this might not be as simple as applying a set of design rules.

Another question about cognitive aids relates to their integration with other sources of information in the operating theatre. Future monitor designs could conceivably have complex algorithms that interpret and suggest actions during critical times. In effect, the anaesthetic machine could provide not only information, but also decision support during emergencies. Similar monitors have been trialled in trauma settings with promising results 4, but equally automation and prediction poses new challenges for data processing by the anaesthetist 35, 40.

Finally, as Everett et al. have attempted to investigate in their study 11, more research is needed on the effect of cognitive aids on team performance. In their original study describing the separate role of reader of a cognitive aid, Burden et al. 8 describe the role as separate to the leader, making suggestions based on the prompts provided. At present, it is unclear who should undertake this reader role, or even what the ideal team structure should be. A second senior anaesthetist might be considered ideal but may skip over important items or be of more use performing clinical tasks. On the other hand, nurses who are generally considered more used to following procedures may be considered, but only if the hierarchy is flat enough for their advice to be heeded by the anaesthetist leading the emergency. One potential structure for emergency teams involves shared leadership between an operations manager ensuring roles are allocated and coordinated and content expert, prompted by the cognitive aid, that ensures all tasks and options are considered. Team structure is particularly difficult to mandate in a cognitive aid due to the variability of resources available in different locations and times. Nevertheless, attempts have been made to produce some guidance based on observations in simulation 27.

Principles for future cognitive aid research

As has been already noted, simulation is perhaps the ideal mechanism to teach and test cognitive aids 41. Immersive, manikin‐based simulation allows a high degree of reproducibility and control of experimental parameters while simultaneously replicating a realistic situation. Nevertheless, anecdotal reports are important for feedback about accessibility and prompting of cognitive aid use in actual cases. Clear proposals for experimental design can now be generated from existing studies and experience from usability testing 42 (Table 3). In summary, there is still much to learn about how cognitive aids can be designed, implemented and used more effectively. Undoubtedly cognitive aids will eventually lead to better clinical outcomes but only carefully designed simulation based and observational research will show us the way.

Table 3 Experimental design of cognitive aids.

1. Research into cognitive aid use should target experienced practitioners and their performance. The purpose of a cognitive aid is to support experts in delivering the best care possible, not to provide a recipe book of detailed instructions for an inexperienced novice.

2. Teams that closely replicate the normal dynamics and team members should be recruited. If team processes and actions are to be measured, then the teams studied must be representative of the teams that will use them in emergencies similar to Everett et al.'s study design.

3. Testing should be undertaken in the actual environment that they work in or a close replica of it, with similar equipment and layout. This gives the advantage of observing the environmental and organisational factors that may interfere with effective implementation; for example instructions spoken aloud might not be appropriate for circumstances where environmental noise is problematic.

4. Learning effects should be minimised and accounted for by study design. The risk in simulator studies is that exposure to the simulation environment leads to a learning effect on performance. Studies must be designed to minimise this learning effect, and statistical analyses designed to account for it as a confounding factor. Learning effects may be particularly pronounced when scenarios with similar skills or learning points occur in sequence. Sophisticated randomisation or counterbalancing of scenario order may be required to minimise the learning effects.

5. Familiarisation and education with the cognitive aid must be undertaken before testing. Cognitive aids are not intended to be used ‘sight unseen’ and will only be effective if education in their use and familiarisation with them is allowed. Research into a cognitive aid should include a defined, standardised education session that is similar in duration and detail to a comparison group.

6. Control groups are not always necessary. It is now widely accepted that cognitive aids improve technical performance in emergencies. Studies including controls where a group without a cognitive aid is compared with one with a cognitive aid for a technical outcome are unlikely to add to the current fund of knowledge on cognitive aids. Invariably, the cognitive aid group will outperform the control group. More useful comparisons include those between different team structures, leadership styles, cognitive aid designs or methods of use.

7. Blinding is desirable but not always possible. Blinding of observers to groups that do not have a cognitive aid is a significant problem of this type of simulation‐based research and this problem can be solved by the omission of a control group.

8. Use repeated measures designs if possible. Team and individual performance in simulation often has a high inter‐individual variability making comparison of intervention groups difficult. Repeated measures designs such as Everett et al.'s can reduce the sample variance, as multiple comparisons can be made on the same subject or group of subjects.

9. Comparisons between different scenarios might not be valid. Technical performance is often easy to measure by observation of number of actions performed or omitted on a scoring system, or time elapsed to perform a key action. However, the same scoring system cannot necessarily be used for different emergencies, nor comparisons made as a result.

10. Choose team behaviour scoring systems carefully. When team behaviours are measured there is often a degree of confusion. A decision must first be made as to whether the unit of measure is an individual's team skills or the performance of the overall team. Scoring systems for team behaviours and team performance must be relevant and have proven reliability to the context and the type of participant they are to be used with. Furthermore, it is often difficult to achieve adequate inter‐rater reliability for these scoring systems without extensive reliability training of the observers with similar cases to those of the study. As with any observational study attempting to make any interpretation of the results in the face of poor measurement reliability is futile.