Development and impact of an endoscopic non-technical skills (ENTS) behavioural marker system

Abstract

Background

Non-technical skills (NTS) are crucial to effective team working in endoscopy. Training in NTS has been shown to improve team performance and patient outcomes. As such, NTS training and assessment are now considered essential components of the endoscopy quality assurance process. Across the literature, other specialties have achieved this through development of behavioural marker systems (BMS). BMS provide a framework for assessing, training and measuring the NTS relevant to healthcare individuals and team. This article describes the development and impact of a novel BMS for endoscopy: the endoscopic non-technical skills (ENTS) system.

Methods

The initial NTS taxonomy for endoscopy was created through a combination of literature review, staff focus groups and semi-structured interviews, incorporating the critical decision method. Framework analysis was conducted with three individual coders and generated a skills list which formed the preliminary taxonomy. Video observation of Bowel Cancer Screening endoscopists was used to identify exemplar behaviours which were mapped to relevant skills in the NTS taxonomy. Behavioural descriptors, derived from video data, were added to form the basis of the ENTS system.

Results

A taxonomy of 33 skills in 14 separate categories were identified through framework analysis. Following video analysis and behaviour mapping, 4 overarching categories and 13 behavioural elements were identified which formed the ENTS framework. The endoscopy (directly observed procedural skills) 4-point rating scale was added to create the final ENTS system. Since its development in 2010, the ENTS system has been validated in the assessment of endoscopy for trainees nationally. ENTS informs a number of training initiatives, including a national strategy to improve NTS for all endoscopists.

Conclusions

The ENTS system is a clinically relevant tool, validated for use in trainee assessment. The use of ENTS will be important to the future of training and quality assurance in endoscopy.

Key messages.

What is already known on this subject

• The 2004 National Confidential Enquiry into Patient Outcome and Death report highlighted the need to improve the training and assessment of non-technical skills (NTS) in endoscopy.

• A number of studies have demonstrated the use of behavioural marker systems (BMS) in accurately assessing and measuring the impact of NTS in fields such as anaesthesia and surgery.

• No such system had previously existed in the area of endoscopy.

What this study adds

• As a result of this study, the endoscopic non-technical skills (ENTS) BMS was created in 2010.

• The ENTS system has been incorporated into the assessment and training of endoscopists, and is an integral feature of national strategies to help improve the quality and safety of endoscopy.

Background

Non-technical skills (NTS) are cognitive and social skills that are important to quality and safety outcomes in healthcare.¹ Within gastrointestinal endoscopy, the importance of NTS was first highlighted in the 2004 National Confidential Enquiry into Patient Outcome and Death (NCEPOD) report.² Here, NTS were identified as contributory factors to procedure-related mortality and morbidity. NTS are integral to the effective delivery of care by healthcare teams and training has been demonstrated to improve overall team performance which can lead to improved patient outcomes.³ Since the 2004 NCEPOD report, there has been an increasing understanding that training and assessment of NTS should be an essential component in the quality assurance of endoscopy.⁴

Within healthcare, behavioural marker systems (BMS) have been developed in response to training and assessment needs. BMS are behaviour-based constructs that individuals or teams can be compared against. They offer opportunities to accurately assess, train and measure the impact of NTS.⁵ BMS were originally developed in the aviation industry to identify the key NTS required by pilots to perform ‘crew resource management’.⁶ These principles inspired the first healthcare-specific BMS: anaesthetics non-technical skills (ANTS).^{7 8} Following this were BMS designed for individuals in surgery: NOn-Technical Skills for Surgeons (NOTSS),^9–11 Oxford NOn-TECHnical Skills for Surgeons (NOTECHS)^{12 13} and Scrub Practitioners’ Non-Technical Skills (SPLINTS).¹⁴ These systems appear to have similarities in structure but clear differences in content. It is evident that BMS are not created with a ‘one-system-fits-all’ purpose in mind but rather, designed to meet the needs of the specific individuals or teams. This is reflected in the ever-expanding literature base. In the past decade, BMS across a variety of specialties and environments have arisen including: operating theatres,^{12 15 16} acute medicine,¹⁷ critical care,^{18 19} neonatal resuscitation²⁰ and neurosurgery.²¹

In the field of endoscopy, no such BMS had previously been constructed. This article describes the development of the endoscopic non-technical skills (ENTS) system in 2010 and reflects on its introduction and impact. This was the first study of its kind within endoscopy, and only one other endoscopy-based NTS assessment tool has been developed subsequently, which has yet to be applied in a practical setting.²² The ENTS system was developed to identify the NTS relevant to endoscopists and develop a BMS to aid in the assessment and training of these skills.

Methods

A schematic of the overall study design can be seen in figure 1. This study was conducted in two stages:

Figure 1.

Study schematic. ENTS, endoscopic non-technical skills.

1. Development of an endoscopy-specific NTS taxonomy through a combination of focus groups and expert interview.

2. Creation of the ENTS BMS from video analysis of observable behaviours and taxonomy mapping.

Study design was informed by the literature, specifically key concepts in BMS development.²³

Development of the ENTS taxonomy

A combination of comprehensive literature review, focus groups and semi-structured interviews were used to identify relevant skills for development of the ENTS taxonomy. An initial literature review found 24 key NTS relevant to healthcare which informed focus group discussions. A single-site focus group was conducted which included a convenience sample of endoscopists, nurses and support staff. Discussion was focused around three facilitated stages:

1. Knowledge audit: to ask participants what NTS are important in endoscopy.

2. Sorting task: participants split (endoscopists and non-endoscopists) and asked to rank importance of the 24 predefined NTS.

3. Rating task: individuals were asked to rate importance of each predefined NTS between 1 and 5 (1=not important at all, 5=very important).

Following this, a sample of nine consultant gastroenterologists and surgeons from London and South East England were recruited to take part in semi-structured interviews. Recruitment of this sample was through a purposive strategy to identify field experts who would likely have insight into NTS in endoscopy and enough experience to recall a memorable incident.²⁴ Interviews were conducted in line with the critical decision method.²⁵ This has been demonstrated to be an effective method of NTS identification, through use of a structured questioning technique.²⁶ Interviews lasted approximately 30–45 min (full interview schedule can be found in online supplementary file 1).

Focus group and semi-structured interviews were audio-recorded and transcribed by the lead researcher. Framework analysis was chosen to analyse these data. This method allows constant comparison of the data, applying it to a scaffold in order to better understand findings in the context of the research question.²⁷ This suited generation of the taxonomy skills list.

The lead researchers (AH, ST-G) initially had a period of familiarisation with the dataset, helped by being directly involved in the data collection strategies. Data from the knowledge audit, sorting and rating tasks was transcribed onto a database allowing ease of comparison. Interview transcripts were coded by both lead researchers separately. Each code formed part of an evolving thematic framework that was applied to successive transcripts. Once individual frameworks were developed, there was a discussion between researchers to develop the preliminary taxonomy, based on agreement. A third coder, an academic psychologist and medical educationalist (KW), used this new framework to code three transcripts independently to ensure that all skills could be coded by the taxonomy.

Following this, a period of indexing occurred. The taxonomy was applied to transcripts to extract verbatim examples of each identified skill. A further discussion was held between researchers, psychologist, the national endoscopy lead (RV) and the national endoscopy training lead (JA) to refine the taxonomy. The skills and verbatim examples were then arranged by code to create a chart of items and descriptors.

Development of the ENTS behavioural marker system

Videos of observed endoscopic procedures were used to identify behavioural exemplars that could be mapped to the NTS taxonomy. Observations were video-recorded at a single site. The choice to record endoscopic procedures rather than observe them physically was driven by a number of factors: first, use of video reduces observer effect and allows repeated viewing.²⁸ Next, the data acquired from video are comprehensive, rather than a representation or reduction of it that may occur through observation alone. Lastly, fixed cameras were already present in endoscopy rooms allow ease of recording.

To identify behavioural exemplars, a sample of four endoscopists from the Bowel Cancer Screening Programme (BCSP) were chosen to be observed. Screening endoscopists undergo a stringent accreditation process through valid, nationally agreed criteria and therefore one would expect a higher frequency of exemplar behaviours.²⁹ Additionally, BSCP patients generally have a higher likelihood of undergoing therapeutic procedures and therefore cases are more likely to display a range of NTS in each encounter. Written consent was gained from all staff and patients involved in video recording.

Four procedures were analysed, one from each endoscopist chosen at random. Framework analysis was used to analyse each video. Video coding was conducted by the lead investigator (AH) and verified by a second researcher (ST-G). Identified behaviours that were observed or inferred were labelled by video code, time stamp and behaviour description (with accompanying verbatim quote if present). Codes were mapped to the NTS taxonomy and could be linked to more than one skill. Within each taxonomy skill, similar behaviours were grouped together and skills rearranged to refine the mapping process. During the analysis, it became apparent that many behaviours were repeated so behaviours were re-coded based on purpose rather than description. As a result, behavioural elements were formed with the appropriate taxonomy skills assigned to each. Elements were thematically assigned categories which they naturally fell under. This formed the basis for the BMS.

In order to develop behavioural descriptors, performance behaviours were identified directly from video and categorised into ‘good’ or ‘poor’ behaviours. Good performance behaviours were those thought to contribute to positive outcomes as identified by consensus of the lead researchers (AH and ST-G). Poor performance behaviours were developed from the opposite of good behaviours or drawn from statements provided in the interview stage.

Results

ENTS taxonomy

The focus group comprised two consultant gastroenterologists, one nurse consultant, three gastroenterology trainees (specialist registrars), four endoscopy nurses and three administrative support staff. The interviews had nine participants, of which seven were consultant gastroenterologists. Mean total years as a consultant was 7.8 (median 5, range 1–18) and mean total oesophago-gastroduodenoscopy and colonoscopy procedure counts were 4111 (median 3000, range 500–10000) and 3077 (median 2500, range 1200–6000), respectively.

The knowledge audit produced a list of 30 individual NTS (figure 2). Twelve skills were classified into four ‘categorical skills’ and the remainder were maintained as ‘standalone skills’. Sorting tasks were combined between the endoscopist and non-endoscopist groups. These were classified into three overarching themes: organisational skills, individual skills and crisis management skills (table 1). The rating task did not produce any meaningful results as most skills were rated as either being ‘important’ or ‘very important’ by all participants.

Figure 2.

Knowledge audit.

Table 1.

Sorting task

Organisational skills	Individual skills	Crisis management skills
Preparation		Crisis management
Workload management	Situation awareness	Declaring emergency
Teaching	Prioritisation
Maintaining team climate	Communication style
Team building	Confidence/assertion
Planning	Leadership
Information sharing	Recognition
Monitoring	Decision making
Organisational climate	Problem solving
Error reporting	Reviewing situation
Documentation	Professionalism
	Stress/fatigue management

Interviews revealed incidents based around cases including postpolypectomy bleeding, perforation and ‘near misses’. On initial framework analysis, 33 codes were identified in total between the two lead researchers. There was full agreement for 8 codes and partial agreement for 14 codes. Discussion between researchers resulted in the final thematic framework of 16 codes. Repeat coding by a clinical psychologist did not reveal any further codes. Following the indexing and charting processes described previously, the final taxonomy was developed, consisting of 14 skills. Five skills were felt to be generic, required during all parts of the procedure, and the remainder were split into either preprocedure, intraprocedure or postprocedure (table 2).

Table 2.

Final ENTS taxonomy

	Skill		General skills
1	Communication	A	With nursing staff
		B	With trainee
		C	With colleagues
		D	With the patient
		E	With relatives
2	Teamwork	A	Good working relationship/communication with team
		B	Seeks opinion of team
3	Leadership	A	Towards a trainee—appropriate supervision and role model
		B	Of the team
		C	Of the unit
4	Confidence		Has appropriate level of confidence in own abilities, judgement and in the capabilities of the team
5	Emotional control	A	Awareness of own emotions and emotional reaction, and ability to control these appropriately
		B	Awareness of other’s emotions and emotional reaction, and adjusts own behaviour appropriately

	Skill		Preprocedure	Intraprocedure	Postprocedure
6	Planning	A	Makes assessment of preprocedural risk
		B	Ensures appropriate environment at the start of the procedure
		C	Ensures preprocedure organisation and briefing of team
7	Assessing situation		Continually re-assesses situation; collecting data, establishing facts and re-evaluating risk assessment
8	Judgement and decision making	A	Has self-awareness of own abilities and limitations
		B	Recognises when own abilities or the environment are not appropriate for the situation and makes appropriate decisions
9	Focus	A		Has appropriate level of concentration for the situation
		B		Keeps control of the environment appropriate to match the situation
10	Awareness	A		Of patient’s condition and wishes
		B		Of team
		C		Of procedural progression
11	Problem recognition			Ability to recognise when an untoward event has occurred
12	Problem management	A		Ability to ‘switch mode’ to deal with a problem
		B		Ability to institute appropriate management
		C		Knowing when to ask for help
13	Responsibility	A		For own behaviour and actions during procedure
		B			For follow-up postprocedure
14	Reflection	A			Postprocedure of what went right or wrong
		B			Feedback of reflection to other members of the team
		C			Making changes based on reflection to improve practice

ENTS behavioural marker system

In total, four endoscopists were observed: three were consultant gastroenterologists (two males, one female) and one was a nurse consultant (female). Each had a lifetime colonoscopy procedure count of over 2000, all were accredited BCSP screeners and three were BCSP accreditors.

Over 400 behaviours were identified and applied to the initial ENTS taxonomy. Four skills from the taxonomy did not have observed behaviours attached. These are skills that are usually exhibited outside of the endoscopy room, demonstrating some of the limitations of this method of video observation. The unmapped skills were reviewed by lead researchers and a consensus decision was made to the behaviours they most suited. The final result of skills mapping can be seen in table 3. After re-coding behaviours by purpose, 13 different behavioural elements were formed which were classified into four overarching categories: communication and teamwork, situation awareness, leadership and judgement and decision making (table 4). A tri-level hierarchy for the marker system was formed based on category, element and incorporating performance descriptors, as described previously (table 5). The hierarchy was informed by the structure of other BMS identified in initial literature review^{7 9} and through researcher consensus. This appeared to be a useful structure to be replicated given the applicability of other BMS.

Table 3.

Map of non-technical skills to behavioural elements and categories

Category	Element		Skills
Communication and teamwork	Exchanging information	1A	Communication with nurses
		2A/B	Teamwork
		3A	Leadership towards a trainee
		1B	Communication with a trainee
		7	Assessing situation
	Maintaining a shared understanding	1A	Communication with nurses
		2A/B	Teamwork
		3B	Leadership of the team
		10A	Awareness of patient’s condition and wishes
	Maintaining a patient-centred approach	1A	Communication with nurses
		1D	Communication with the patient
		1E	Communication with relatives
		2A/B	Teamwork
		4	Confidence
		5A/B	Emotional control
		7	Assessing situation
		10A	Awareness of patient’s condition and wishes
Situation awareness	Preparation	2A/B	Teamwork
		6A/B/C	Planning
		7	Assessing situation
		8A	Self-awareness of own abilities
		10C	Awareness of procedural progression
	Continuous assessment	7	Assessing situation
		10A	Awareness of patient’s condition and wishes
	Problem recognition	11	Problem recognition
		3B	Leadership of the team
		5A/B	Emotional control
		7	Assessing situation
		8A/B	Judgement
		9B	Control of the environment
		10B	Awareness of team
	Focus	9A	Appropriate level of concentration
		9B	Control of the environment
Leadership	Supporting others	2A/B	Teamwork
		3A	Leadership towards a trainee
		3B	Leadership of the team
		5B	Awareness of other’s emotions
		10B	Awareness of team
	Maintaining standards	3B	Leadership of the unit
		9B	Keeping control of the environment
		13A/B	Responsibility
	Dealing with problems	1A	Communication with nurses
		2A/B	Teamwork
		3B	Leadership of the team
		5A/B	Emotional control
		9B	Control of environment
		11	Problem recognition
		12A	Ability to ‘switch mode’
		12B	Ability to institute appropriate management
Judgement and decision Making	Considering options	7	Assessing situation
		8A	Self-awareness
		10C	Awareness of procedural progression
		12C	Knowing when to ask for help
	Making decisions	1A	Communication with nurses
		3B	Leadership of the team
		4	Confidence
		8A	Self-awareness
		10C	Awareness of procedural progression
		12A	Ability to ‘switch mode’
	Reviewing the situation	7	Assessing situation
		8A	Self-awareness
		14A/B/C	Reflection

Table 4.

Classification of behavioural elements into categories

Communication and teamwork	Situation awareness	Leadership	Judgement and decision making
Exchanging information	Preparation	Supporting others	Considering options
Maintaining a shared understanding	Continuous assessment	Maintaining standards	Making decisions
Maintaining a patient-centred approach	Problem recognition	Dealing with problems	Reviewing the situation
	Focus

Table 5.

Overview of tri-level hierarchy comprising category, element, performance descriptors and rating scale

Category	Element	Example behaviours		Rating (per category)
		Good	Poor
Communication and teamwork	Exchanging information	Gives clear, specific instructions to staff and patient Seeks further information to aid understanding, eg, previous endoscopy reports Listens and responds to team input Confirms team preparation including equipment availability	Fails to give clear instructions Does not seek further information or makes inappropriate assumptions Does not listen to or acknowledge team members Does not check if team ready or if equipment available

Rating	Grade	Descriptor
1	Poor	Performance endangered or potentially endangered patient safety. Serious remediation is required.
2	Marginal	Performance indicated some cause for concern. Considerable improvement is needed.
3	Acceptable	Performance was of a satisfactory standard, but could be improved.
4	Good	Performance was of a consistently high standard, enhancing patient safety. It could be used as a positive example for others.
N/A		Not applicable.

Rating scale

A 4-point rating scale was applied to the ENTS framework to form the final BMS. This scale was replicated from the pre-existing endoscopy directly observed procedural skills (DOPS) scale, which could be mapped directly onto the ENTS framework (table 5).³⁰ The scale emphasises patient safety as the primary outcome and recognises not all behaviours may be observed or relevant to all cases.

Discussion

Summary

NTS are known to be important in teamwork processes and can impact directly on patient outcomes. This study describes the development of the ENTS system to address the deficiencies in NTS training and assessment in endoscopy.

The final ENTS BMS reflects the skills important in routine and non-routine endoscopy. It is evident that this cannot incorporate all conceivable NTS relevant to endoscopy. It is instead intended to provide a framework for identifying NTS through observable behaviours and guiding their assessment in a structured manner. The ENTS BMS provides definitions and examples of good and poor behavioural markers derived from real-life experiences.

Reviewing the literature, there are clear similarities between the assessment categories of ENTS and other BMS within healthcare. Similarities extend beyond content to framework structures. ENTS has a tri-level hierarchy consisting of category, element and behavioural descriptors and a 4-point rating scale. This resembles the structures of the ANTS,⁷ NOTSS⁹ and SPLINTS¹⁴ BMS. It appears that this is a common feature of BMS across the literature, but remains unclear whether these systems outperform those that are structured differently.⁵ Differences arise when you explore each framework in more detail, particularly in the behavioural exemplars that form the elements in each.

BMS can be divided into having ‘low temporal resolution’ or ‘high temporal resolution’ time scales.⁵ BMS with low resolution time scales assess skill over the whole time period, whereas those with high resolution scales assess different phases of performance over time. ENTS fits into the former, which favours the identification of NTS deficiencies. This suits global assessment and correlates with other assessable domains in endoscopy, for example, through DOPS assessment. Assessments of individuals within anaesthetics and surgery appears to occur in a similar fashion.^7–9 BMS that incorporate high temporal resolution time scales may be better suited for longer and more complicated observational periods, for example, whole team skills in surgery.^{31 32}

As far as the authors are aware, only one other NTS-specific assessment tool in endoscopy exists, identified by a recent systematic literature review.³³ Scaffidi et al used focus groups of 40 endoscopy staff (gastroenterologists and nurses) to identify areas for assessment of NTS.²² Thematic analysis of transcripts revealed six ‘NTS dimensions’: teamwork, communication, situational awareness, decision making, leadership and professionalism. A 5-point scoring tool was used to rate skill. There are clear similarities between the dimensions in this tool and the categories within the ENTS system, however it is unclear whether behavioural descriptors were used to help measure skill. Additionally, there is no comment on whether endoscopy teams or individuals are the targets of assessment. Lastly, compared with the ENTS system, there does not appear to be any current real-world application or testing of this tool. Nevertheless, the development of other tools to assess NTS in endoscopy are welcomed, and further research will identify their utility against established systems.

Strengths and limitations

This study draws from others that have been conducted in other healthcare settings, for which there is a broad similarity in study designs.⁵ The critical decision method is advantageous as it gains specific insights compared with the more traditional semi-structured interview.²⁵ Participants’ recollection during interviews could be considered a limitation of the critical decision method as data are dependent on verbal accounts. However, the use of other methods, namely focus groups, aimed to provide a degree rigour to data collection and limit bias. The use of multiple coders also helped to limit the interpretive bias that can be sometimes associated with qualitative analysis methods.²⁴

It should be recognised that the behaviours identified during the observation task may not reflect the full range of NTS that may be encountered in endoscopy, particularly in emergency situations as these were not observed. This limitation of the study could be improved with observation of a greater number of encounters over a longer period of time. However, the goal of video observation in this study was to define exemplar behaviours to map to the original ENTS taxonomy, not generate an exhaustive list of all possible behaviours. This limitation is explicitly conveyed to users of the ENTS system.

Lastly, no reliability or validity measures were conducted during the initial study. However, since the development of the ENTS system almost a decade ago, there have been several studies that have demonstrated validity in a practical setting, as will be described in the following section. An area of further work should focus on reliability measures including internal consistency, inter-rater and test–retest reliability. These parameters have yet to be formally defined for the ENTS system and are an area of ongoing research.

Impact of the ENTS system

The ENTS system was developed almost a decade ago³⁴ and since then there have been numerable applications of its use in research and practice. The Joint Advisory Group on Gastrointestinal Endoscopy (JAG) oversees and supports endoscopy training certification and accreditation of endoscopy services in the UK. JAG published a handbook to promote utility of ENTS in various settings (https://www.thejag.org.uk/Downloads/Training guidance/ENTS handbook v2.pdf). ENTS has now been incorporated into the assessment and training of endoscopists in the UK. The impact of ENTS with respect to these two areas can be seen in table 6, which highlights the literature to date.

Table 6.

Studies related to the impact of ENTS

Topic	Type	Description	Results	Validity evidence	Study
Prospective assessment of ENTS	Assessment	Prospective evaluation of safety practices in endoscopy. Evaluation of ENTS using framework	Varied ENTS scores. Endoscopists scoring higher ENTS scores more likely to perform safety checks (p<0.001).	Relationship to other variables.	Matharoo et al 35
Assessment of ENTS in emergency endoscopy	Assessment	Prospective assessment of ENTS during emergency procedures.	ENTS scores positively correlated with DOPS and safety checklist scores. Patient safety incidents inversely correlated with ENTS scores.	Internal structure. Relationship to other variables.	Matharoo et al 36
ENTS competency during training	Assessment	Collated assessable domains in 8601 DOPS and compared ENTS with other domains.	ENTS competency increased with lifetime procedural count. ENTS competency correlated strongly with other assessable domains.	Internal structure. Relationship to other variables.	Siau et al 37
Development of MARS tool	Assessment	Development of a multiassessor rating scale using ENTS framework. Four domains, 10 items per domain and 7-point rating scale.	MARS tool practical to administer. Good internal consistency and acceptable inter-rater reliability.	Internal structure (for MARS tool).	Kokwara et al 40
Implementation of MARS tool in practice	Assessment	Use of MARS to provide 360 assessment of independent endoscopist ENTS. Nine endoscopists rated by 10 raters (nurses).	Significant differences in domains observed by use of 10 raters. Useful in providing feedback to endoscopists.	Internal structure. Consequences (for MARS tool).	Hawkes et al 41
Validation of paediatric gastroscopy DOPS	Assessment	Prospective national study of paediatric gastroscopy DOPS. Averaged ENTS scores compared with overall procedural scores.	For 157 DOPS, ENTS scores significantly correlated with overall competency scores (p<0.001).	Internal structure. Relationship to other variables.	Siau et al 38
Validation of paediatric colonoscopy DOPS	Assessment	Prospective national study of paediatric colonoscopy DOPS. Averaged ENTS scores compared with overall procedural scores.	For 61 DOPS, ENTS scores significantly correlated with overall competency scores (p<0.001).	Internal structure. Relationship to other variables.	Siau et al 39
ENTS team training	Training	ENTS incorporated into a training day for Bowel Cancer Screening teams to improve knowledge and attitudes around patient safety.	Significant improvement in patient safety knowledge and 29% of safety attitude question items.	N/A	Matharoo et al 44
ENTS simulation training	Training	Description of 5 years of multidisciplinary ENTS simulation.	Significant improvement in three out of eight ENTS self-reported confidence domains. Global acceptability of simulation as a strategy to deliver ENTS training.	N/A	El Menabawey, T et al 45
ISREE strategy	Training and assessment	Multidisciplinary workshop with 35 participants. Theorising how ENTS will be incorporated into training pathways to improve endoscopy safety.	Development of 5-year implementation strategy. One domain is improving ENTS training.	N/A	Thomas-Gibson et al 47
ENTS video learning	Training	Development of simulated case using video media to highlight ENTS. Incorporated into regional endoscopic skills training.	Eight participants improved self-rated confidence in human factors. Resource-low method that can be incorporated into other training courses.	N/A	Macdougall et al 49

Description and results pertain to ENTS only.

DOPS, directly observed procedural skills; ENTS, endoscopic non-technical skills; ERCP, endoscopic retrograde cholangiopancreatography; ISREE, improving safety and reducing error in endoscopy; MARS, Multi-Assistant Rating Scale.

ENTS in assessment

Soon after development, the ENTS system was used in studies to investigate the relationship between ENTS and safety. The authors prospectively assessed endoscopists using the ENTS rating system and demonstrated correlation between ENTS scores and likelihood of safety checks³⁵ and patient safety incidents.³⁶ ENTS was subsequently incorporated into the DOPS and direct observation of polypectomy procedural skills. The validity of ENTS within DOPS was recently established. Siau et al collated findings from over 8000 DOPS and compared scores across all assessable domains, including ENTS.³⁷ They demonstrated that ENTS scores increase with lifetime procedure count and correlate strongly with other measures of competency, inferring a degree of validity. Recently, there has been further validation of ENTS within DOPS for paediatric endoscopy.^{38 39} A summary of all studies related to the validation of ENTS for assessment is summarised in table 6.

Use of the ENTS system in assessment has also moved beyond the trainee. The Multi-Assistant Rating Scale (MARS) has been developed from the original ENTS framework as a tool to assess ENTS in independent endoscopists.^{40 41} Preliminary work has shown this to be a reliable and useful tool, particularly in providing feedback. It is envisaged that tools like this will play a role in the ongoing quality assurance of endoscopists, alongside other established key performance indicators.

One of the areas of recent focus has been assessing endoscopy team function, particularly as this can be directly linked to team performance and patient outcomes.⁴² While ENTS has become engrained into the assessment of endoscopists, it cannot be directly applied to endoscopy teams. Reflecting on progress in other fields,^{12 31 43} it is apparent that development of an endoscopy-specific, team-based BMS may be of use in measuring team performance. This should certainly be an area for further research in the coming years.

ENTS in training

ENTS has been incorporated into several training strategies to date. Utility of the framework was first demonstrated during classroom-based training of endoscopy teams around safety.⁴⁴ Here, patient safety knowledge and attitudes were improved through the training exercise. Since then, the largest application of ENTS has been in simulation-based education. The ENTS framework appears to be well-suited to simulation as a structure to guide training.^{45 46} Acknowledging this, the JAG ‘improving safety and reducing error in endoscopy’ workshop identified the need to implement a nationwide ENTS training strategy, incorporating simulation and other modalities of learning.^{47 48} More recently, video-based learning has been piloted as a means to deliver low-cost, efficacious ENTS training that may complement current training schemes.⁴⁹

Within these strategies, one challenge has been measuring outcomes effectively. These are often defined by participant acceptability and self-rated confidence. In this respect, ENTS may be useful in providing a framework for training and in its utility as a measure of training efficacy. For example, BMS have been used in fields such as anaesthesia and surgery, to demonstrate objective improvement in skill following simulated practice.^{31 43 50} Measuring learning outcomes in this way is a current gap in training strategies and will be an important future step in the use of the ENTS system.

Conclusions

The ENTS system is a novel tool that was developed to aid in the training and assessment of endoscopy. We describe the conception and construction of this tool and the subsequent effects it has had in practice. ENTS has now become integrated into national strategies aimed at improving the quality and safety of endoscopy.