Certification of UK gastrointestinal endoscopists and variations between trainee specialties: results from the JETS e-portfolio

Abstract

Introduction  In the UK, endoscopy certification is administered by the Joint Advisory Group on Gastrointestinal Endoscopy (JAG). Since 2011, certification for upper and lower gastrointestinal endoscopy has been awarded via a national (JETS) e-portfolio to the main training specialties of: gastroenterology, gastrointestinal surgeons (GS) and non-medical endoscopists (NME). Trends in endoscopy certification and differences between trainee specialties were analyzed.

Methods  This prospective UK-wide observational study identified trainees awarded gastroscopy, sigmoidoscopy, colonoscopy (provisional and full) certification between June 2011 – 2017. Trends in certification, procedures and time-to-certification, and key performance indicators (KPIs) in the 3-month pre- and post-certification period were compared between the three main training specialties.

Results  Three thousand one hundred fifty-seven endoscopy-related certifications were awarded to 1928 trainees from gastroenterology (52.3 %), GS (28.4 %) and NME (16.5  %) specialties. During the study period, certification numbers increased for all modalities and specialties, particularly NME trainees. For gastroscopy and colonoscopy, procedures-to-certification were lowest for GS ( P  < 0.001), whereas time-to-certification was consistently shortest in NMEs ( P  < 0.001). A post-certification reduction in mean cecal intubation rate (95.2 % to 93.8 %, P  < 0.001) was observed in colonoscopy, and D2 intubation (97.6 % to 96.2 %, P  < 0.001) and J-maneuver (97.3 % to 95.8 %, P  < 0.001) in gastroscopy. Overall, average pre- and post-certification KPIs still exceeded national minimum standards. There was an increase in PDR for NMEs after provisional colonoscopy certification but a decrease in PDR for GS trainees after sigmoidoscopy and full colonoscopy certification.

Conclusion  Despite variations among trainee specialties, average pre- and post-certification KPIs for certified trainees met national standards, suggesting that JAG certification is a transparent benchmark which adequately safeguards competency in endoscopy training.

Introduction

It is desirable to have a process for determining whether an individual is competent to perform an endoscopic procedure before that individual is allowed to practice independently. This helps reassure and protect both patients and the service. Ideally the process should be identical regardless of the type of health professional seeking certification. Performance following certification should meet current performance standards for each procedure performed and ideally be subjected to ongoing audit and monitoring ¹ .

In the UK, a certification process exists to ensure that trainees in endoscopy have achieved a minimum standard of competence before being allowed to practice independently. This process is applied to all health professionals training in endoscopy and is governed by the Joint Advisory Group on Gastrointestinal Endoscopy (JAG) ² . Since June 2011, the process has been administered through the JETS ( J AG E ndoscopy T raining S ystem) e-portfolio ³ and certification awarded electronically (e-certification) for gastroscopy ⁴ , sigmoidoscopy ⁵ , and colonoscopy ⁶ . This evolved from paper-based portfolios which were cumbersome, time-consuming and inconsistently completed ⁷ . All endoscopy training units are linked to the JETS e-portfolio which allows an individual trainee to enter training data from any JAG-registered training unit during their training rotation. The JETS e-portfolio enables trainees to record details of training episodes to generate procedure-specific key performance indicators (KPIs). Trainers use the same platform to complete competency assessments, i. e. Direct Observation of Procedural Skills (DOPS) and Polypectomy Skills (DOPyS), at regular intervals to monitor skills progression ⁸ . Input of assessments from multiple different trainers increases the reliability of DOPS ⁹ and DOPyS assessments ¹⁰ . The JAG recommends that endoscopy training be delivered at accredited endoscopy units by JAG-certified trainers ^{7 11} . The JETS e-portfolio currently contains more than 2 million trainee procedural records and is unique to the UK.

The JETS e-portfolio has a procedure-specific checklist that informs trainees when they are ready for a summative assessment and can apply for certification ( Appendix Table 1, Table 2, Table 3, Table 4 ) ³ . These include: attainment of minimum procedural numbers and procedure-specific key performance indicators (KPIs), attendance of training courses, and satisfactory completion of DOPS and DOPyS ^{12 13 14} . Once these criteria are satisfied, trainees may apply for summative assessment. For certification in gastroscopy and sigmoidoscopy, the process is entirely formative, until the summative assessment is undertaken. Colonoscopy certification differs and is currently awarded in two phases: provisional and full certification ( Appendix Table 4 ). Once provisional colonoscopy certification status is achieved, endoscopists are allowed to practice diagnostic colonoscopy independently, but are subject to departmental supervision and monitoring of KPIs. ¹⁵ Trainees are able to perform sigmoidoscopy independently at this stage. Full colonoscopy certification is dependent on achieving provisional certification and additional criteria ( Appendix Table 4 ). No further summative assessment of colonoscopy skills is required to progress from provisional to full certification. The final stage of JAG certification requires that a local endoscopy training lead validate the data and sign off on the application for certification in the JETS e-portfolio. The application is then submitted electronically to JAG. Independent JETS assessors then review the application and decide whether certification is to be awarded ^{4 5 6} .

In the UK, the endoscopy workforce consists predominantly of gastroenterologists, gastroenterology surgeons (GS), and non-medical endoscopists (NME), with radiologists and general practitioners forming a small minority (< 2.5 %) ¹⁶ . Irrespective of specialty, all UK endoscopy trainees are required to meet the same procedural standards and go through the same standardized e-certification process. In recent years, demand for endoscopy in the UK has increased, requiring an increase in both endoscopy capacity and workforce ¹⁷ . In response, strategic recruitment initiatives such as the NME training program, backed by the Department of Health, have been successful in increasing numbers of NMEs in the workforce ¹⁸ . It is possible that this may have adversely impacted on the training list availability for other trainees and potentially, the time required to achieve certification.

There is no published literature on endoscopy certification within the UK or elsewhere. Trends in certification numbers have not been studied. Whether the UK certification process ensures that trainees reach acceptable standards for independent practice is unknown. It is important to determine whether the certification process is effective and to identify factors that affect how and when a trainee completes the certification process. This would inform regional and national training programs by allowing them to predict how rapidly a new workforce can be developed as demand for endoscopy changes. Moreover, such data could be used to optimize the existing training pathway, using evidence-based training interventions, to provide maximum efficiency in training an effective workforce.

In this study we aimed to assess whether certification ensures trainees are adequately trained for independent practice by exploring pre- and post-certification KPIs, and explored how specialties compared with regard to trends in certification.

Methods

Study Design

This was a 6-year prospective, UK-wide, observational study. The study cohort consisted of all trainees who were awarded certification for gastroscopy, flexible sigmoidoscopy, and colonoscopy between 1st June 2011 and 1st June 2017 via the JETS e-portfolio. Data extracted included: trainee specialty, trainee level, date of certification, and procedure-specific KPIs.

In addition, lifetime procedure counts prior to certification were also collated for each trainee. For those who commenced training prior to inception of the e-certification system or outside the UK, procedures performed prior to JETS were not electronically recorded. As such, at the point of registration, these trainees recorded the numbers of procedures of each type that they had already performed, which is referred to as the baseline procedural count. These were added to the numbers of procedures recorded in the system, to give the lifetime procedural count.

Outcomes

For each certification modality and trainee specialty the following were reported:

1. Number of trainees awarded certification per year.

2. Lifetime procedure counts prior to successful certification.

3. Time taken from first procedure-specific JETS e-portfolio entry to certification. Trainees with baseline procedures prior to enrollment in the JETS e-portfolio were excluded from analysis of this outcome, as the dates of the first procedure were not recorded electronically.

4. KPIs in the 3 months before and 3 months after certification. These comprised: unassisted cecal intubation rate (CIR) and polyp detection rate (PDR) for colonoscopy, PDR for sigmoidoscopy, and rates of D2 intubation and J-maneuver for gastroscopy (retroflexion in the gastric antrum to visualise the cardia and gastric fundus). The CIR was unadjusted (not adjusted for factors such as poor-quality bowel preparation or impassable stricture). Trainees do not train on colorectal cancer screening procedures and are advised to exclude rectal hyperplastic polyps from the PDR. Trainees with fewer than 10 procedure-specific e-portfolio entries in either of these 3-month periods were excluded, to allow for reliable estimates of rates.

5. Rates and reasons for failed certification applications.

Statistical analyses

The numbers of certifications awarded in each annual period (June to May) were collated, and linear regression models were produced to quantify changes over time for each procedure type, and within each specialty. Lifetime procedure counts and times taken to achieve certification were then compared between specialties using Kruskal-Wallis tests, followed by pairwise Dunn’s post-hoc tests where significant differences were detected. Changes in KPIs between the 3-month pre- and post-certification periods were assessed using Wilcoxon’s signed rank tests, with comparisons across specialties performed using Kruskal-Wallis tests, followed by Dunn’s test. Rates of successful JAG applications were compared between specialties using chi ² tests. All analyses were performed using SPSS (v23, Illinois, United States), with P  < 0.05 indicative of significance throughout.

Baseline characteristics and trends in certification over time

Between June 2011 and 2017, JAG awarded 3157 endoscopy-related certifications across the four endoscopic modalities to 1928 trainees. The number of trainees awarded certification increased over time (  Fig.1 ). Gastroscopy was the modality for which the most certifications were awarded (N = 1312, 41.6 %), followed by provisional (N = 1105, 35.0 %) and full (N = 546, 17.3 %) colonoscopies, and sigmoidoscopy (N = 194, 6.1 %). Numbers of certifications awarded per year increased significantly for all of the modalities by similar absolute rates, ranging from an additional 13 per year in sigmoidoscopy to 32 per year in provisional colonoscopy ( Fig. 1 ). However, when considered as relative increases, sigmoidoscopy and full colonoscopy had the greatest rate of change over the period, with the number of certifications awarded per year increasing 19-fold (N = 75 vs. 4) and 14-fold (N = 152 vs. 11), respectively, between 2011 – 12 and 2016 – 17 (compared to 4-fold in provisional colonoscopy and 3-fold in gastroscopy).

Fig.1.

 Endoscopy certification awarded by year and modality.

Trends in certifications are broken down by specialty in Table 1 . Gastroscopy and both provisional and full colonoscopy certification were predominantly awarded to gastroenterology trainees, followed by GS and NME. While the gastroenterology specialty was the main contributor to the increasing number of colonoscopy certifications awarded per year in absolute terms, trends over time observed in this group were actually the least significant as, due to the large number of gastroenterologists awarded certification early in the period, the relative changes were small. For gastroscopy, the numbers of certifications awarded per year to gastroenterologists was not found to change significantly over time ( P  = 0.403, N = 147 in 2012/13 vs. 134 in 2016/17), suggesting that this has plateaued. Sigmoidoscopy certification was predominantly awarded to NMEs (N = 148, 76.2 %) and, as such, these were the main cause of the observed yearly increase in certification.

Table 1. Changes over time in numbers of certifications awarded by specialty.

Specialty	Gastroscopy			Flexible sigmoidoscopy			Colonoscopy (provisional)			Colonoscopy (full)
	N	Yearly gradient (95 % CI)	P value	N	Yearly Gradient (95 % CI)	P value	N	Yearly gradient (95 % CI)	P value	N	Yearly gradient (95 % CI)	P value
Gastroenterology	770	5 (–10, 20)	0.403	22	1 (0, 2)	0.103	555	14 (–1, 29)	0.059	305	13 (5, 20)	0.008
GS	330	12 (3, 20)	0.018	17	1 (0, 2)	0.117	383	11 (2, 19)	0.027	168	8 (5, 11)	0.002
NME	166	9 (4, 13)	0.007	148	11 (6, 15)	0.003	144	7 (5, 8)	< 0.001	64	5 (2, 7)	0.006
Overall	1312	27 (2, 51)	0.038	194	13 (8, 18)	0.002	1105	32 (9, 54)	0.017	546	26 (18, 34)	0.001

Results are based on linear regression models of the total number of certifications in each 12-month period. N represents the number of trainees awarded certification. Gradients represent the average annual increase in number of certifications awarded. The overall numbers included data from all certified specialties. P values are significant at P  < 0.05. GS, gastrointestinal surgery; NME, non-medical endoscopist

Procedural count and formative assessments before successful certification

The median number (interquartile range [IQR]) of lifetime procedures recorded prior to certification were: gastroscopy: 282 (IQR 237 – 411), sigmoidoscopy: 262 (IQR 210 – 334), provisional colonoscopy certification: 269 (IQR 226 – 342) and full colonoscopy certification: 403 (IQR 339 – 509). The median number required to apply and achieve certification varied significantly by specialty in gastroscopy and provisional colonoscopy ( P  < 0.001) in the order of gastroenterology > NME > GS ( Fig. 2 ). For full colonoscopy certification, the order was NME > gastroenterology > GS ( P  < 0.001), while no significant difference between specialties was observed for sigmoidoscopy ( P  = 0.386).

Fig. 2 .

Variations in precertification procedural counts by specialty trainees and certification modality. GS, general surgeon; NME: non-medical endoscopist. * P  < 0.05.

There was significant variation in the number of precertification DOPS and DOPyS assessments performed between specialties ( P  < 0.001). NMEs had the greatest median number of formative assessments completed by trainers (30 DOPS, 9 DOPyS), followed by GI (29 DOPS, 8 DOPyS) and GS (18 DOPS, 8 DOPyS).

Time-to-certification

For each trainee, time-to-certification was measured using the interval between first procedure-specific e-portfolio entry and certification date. The analysis was restricted to 62.9 % of trainees who reported no baseline procedures outside the JETS e-portfolio. For all specialties, median training times to certification were: gastroscopy 1.9y (IQR 1.2 – 3.0y); sigmoidoscopy: 1.6y (IQR 1.0 – 2.7y); provisional colonoscopy: 3.2y (IQR 2.2 – 4.5y) and full colonoscopy: 4.1y (IQR 2.8 – 5.2y). Training time to certification differed significantly across the three major specialties for each modality ( P  < 0.001) in the order of GS > gastroenterology > NME, except for sigmoidoscopy, which was in the order of gastroenterology > GS > NME ( Fig. 3 ).

Fig. 3.

 Time-to-certification (year) by specialty and modality. Excludes trainees with baseline procedures. GS: general surgeon; NME: mon-medical endoscopist * P  < 0.05.

Key performance Indicators

Gastroscopy

A total of 768 trainees (58.5 %) met the inclusion criteria of ≥ 10 post-certification procedures in the 3 months pre- and post-certification and were included in the analysis ( Table 2 ). Over this period, trainees recorded a median of 47 procedures (IQR 31 – 70) pre- and 34 (IQR 22 – 55) post-certification. Pre-certification, rates of D2 intubation (mean 97.6 %) and J-maneuver (mean 97.3 %) exceeded the JAG certification requirements of ≥ 95 % for both KPIs ( Appendix Table 1 ) ¹² . Following certification, minor but statistically significant reductions in both D2 completion and J-maneuver rates were observed for all specialties. Comparisons between specialties found no significant differences in KPIs, with the exception of precertification J-maneuver rate ( P  = 0.001), which was highest in GS trainees ( P  = 0.001 vs. gastroenterology; P  = 0.019 vs. NME), although this was not evident post-certification ( P  = 0.523).

Table 2. Paired median key performance indicator data for trainees in the 3 months pre- and post-gastroscopy certification period with comparisons performed using Wilcoxon signed rank tests.

Upper gastrointestinal endoscopy certification		Mean D2 intubation Rate			Mean J-maneuver rate
Gastroscopy	N	Pre	Post	P value	Pre	Post	P value
Gastroenterologist	560	97.5 %	96.4 %	< 0.001	97.0 %	95.8 %	< 0.001
GS	101	97.8 %	94.7 %	< 0.001	98.2 %	95.5 %	< 0.001
NME	85	97.6 %	96.7 %	0.016	97.7 %	96.3 %	< 0.001
Overall	768	97.6 %	96.2 %	< 0.001	97.3 %	95.8 %	< 0.001

Trainees with < 10 procedures in the 3 months post-certification period were excluded. The overall numbers include data from all certified specialties. GS: gastroentestinal surgeon; NME: non-medical endoscopist.

Flexible sigmoidoscopy

Paired data were analyzed for 56.7 % of trainees awarded certification in flexible sigmoidoscopy, which primarily comprised of NMEs (92/107, 86.0 %). Median (IQR) procedural numbers were 57 (IQR 32 – 88) pre- and 56 (IQR 30 – 95) post-certification. There was no significant difference in PDR between specialties immediately before (mean 12.6 %; P  = 0.362) and after certification (mean 12.0 %; P  = 0.670). However, on subgroup analysis ( Table 3 ), a significant reduction in PDR was noted for GS (18.8 % to 12.5 %; P  = 0.028), although only seven GS trainees were included in this analysis.

Table 3. Paired mean key performance indicator data for trainees in the 3 months pre- and post-lower GI certification period, with comparisons performed using Wilcoxon signed rank tests.

Lower gastrointestinal endoscopy certification		Mean CIR			Mean PDR
Sigmoidoscopy	N	Pre	Post	P value	Pre	Post	P value
Gastroenterologist	8				20.3 %	20.0 %	0.345
GS	7				18.8 %	12.5 %	0.028
NME	92				12.2 %	12.0 %	0.463
Overall	110				12.6 %	12.0 %	0.674
Colonoscopy (provisional)	N	Pre	Post	P value	Pre	Post	P value
Gastroenterologist	382	95.4 %	94.2 %	0.001	22.9 %	22.7 %	0.871
GS	234	95.1 %	93.4 %	0.008	23.6 %	22.1 %	0.109
NME	115	94.6 %	93.1 %	0.004	20.2 %	22.4 %	0.010
Overall	749	95.2 %	93.8 %	< 0.001	22.7 %	22.5 %	0.979
Colonoscopy (full)	N	Pre	Post	P value	Pre	Post	P value
Gastroenterologist	173	94.9 %	94.4 %	0.492	25.6 %	25.4 %	0.759
GS	92	94.9 %	94.4 %	0.409	24.3 %	20.9 %	0.007
NME	32	93.5 %	94.8 %	0.984	27.9 %	25.9 %	0.421
Overall	301	94.9 %	94.4 %	0.315	25.6 %	24.4 %	0.075

Trainees with < 10 procedures in the 3 months post-certification period were excluded. The overall numbers include data from all certified specialties. CIR, cecal intubation rate – unadjusted, intention to treat; PDR, polyp detection rate; GS: trainee Surgeon, NME: non-medical endoscopists trainee.

Colonoscopy (provisional)

For provisional certification in colonoscopy, paired data were available for 749 certified trainees (67.8 %), each with a median of 31 pre-certification (IQR 21 – 45) and 29 (IQR 19 – 48) post-certification procedures. In the post-certification phase, a small overall reduction in CIR was observed across all groups (95.2 % to 93.8 %, P  < 0.001). There was no overall difference in PDR between the pre- (22.7 %) and post-certification periods (22.5 %, P  = 0.979), although subgroup analysis demonstrated increased PDR was recorded for NMEs (20.2 % to 22.4 %, p = 0.010) post-certification ( Table 3 ).

Colonoscopy (full)

For full colonoscopy certification, 297 endoscopists recorded paired data (55.3 %), each with a median of 43 pre-certification (IQR 26 – 65) and 31 (IQR 21 – 48) post-certification procedures. No significant difference in CIR was detected between the periods pre- and post-certification (mean: 94.9 % vs. 94.4%, P  = 0.315), with subgroup analyses by specialty returning consistent results ( Table 3 ). There was no overall difference in PDR (25.6 % to 24.4 %, P  = 0.075), although a significant reduction was observed within the GS subgroup (24.3 % to 20.9%, P  = 0.007). There were no significant differences between specialty in terms of pre-certification CIR ( P  = 0.283), post-certification CIR ( P  = 0.771) or pre-certification PDR ( P  = 0.469), although there was variation in post-certification PDR ( P  = 0.004), which was lowest in the GS specialty.

Outcomes of applications for certification

Outcomes of summative assessment are summarised in Table 4 . Summative assessment was successful at first attempt in 91.3 %. Failure rates at first attempt varied between specialty for gastroscopy (gastroenterology 11.0 %, GS 10.5 %, NME 3.4%; P  = 0.016) and sigmoidoscopy (gastroenterology 4.6 %, GS 30.0 %, NME 9.3 %; P  = 0.012), but not provisional colonoscopy certification (gastroenterology 6.6 %, GS 5.9 %, NME 10.7 %; P  = 0.130). Analyses combining all of the procedures found no significant differences in summative assessment failure rates at first attempt between specialty (gastroenterology 9.1 %, GS 8.6 %, NME 7.8 %; p = 0.688).

Table 4. Outcomes of summative assessment.

Certification modality	Pass at first attempt	Second attempt if failed	Pass at second attempt
Gastroscopy	1182/1312 (90.1 %)	113/130 (86.9 %)	103/113 (91.2 %)
Sigmoidoscopy	173/194 (89.2 %)	15/21 (71.4 %)	14/15 (93.3 %)
Colonoscopy (provisional)	1029/1105 (93.1 %)	64/76 (84.2 %)	54/64 (84.4 %)
Total	2384/2611 (91.3 %)	192/227 (84.6 %)	171/192 (89.1 %)

A total of 3183 applications for JAG certification fulfilled summative criteria and were reviewed centrally by JETS assessors. Of these, 26 (0.8 %) applications were turned down due to the following reasons: not meeting current criteria (n = 14), insufficient polypectomy experience (n = 3), lack of recent evidence following summative assessment (n = 3) and potential assessment bias, i. e. lack of variation in assessors or spread of formative assessments over time (n = 6).

Discussion

This prospective study provides novel data from the JETS e-portfolio related to endoscopy certification in the UK. Quality criteria laid out in the JAG accreditation standards for endoscopy units ensure that no endoscopist is allowed to perform independently without JAG certification ⁷ , which is reliant on engagement with the e-portfolio. This offers a unique insight into the effectiveness of certification and whether important differences exist in how different specialties approach certification and perform in the independent setting thereafter.

Following national rollout of the JETS e-portfolio, JAG certification numbers have increased in parallel with increasing trainee numbers and rising endoscopy demand. The relative increase in sigmoidoscopy and colonoscopy certification is likely to reflect a response to rollout of the National ‘Bowel Scope’ (sigmoidoscopy screening) programme ¹⁹ and Bowel Cancer Screening Programme ²⁰ , in addition to several public symptom awareness campaigns ^{21 22} . Between 2011 and 2017, NMEs accounted for the largest proportional increase in certification numbers, in part owing to a government initiative to increase the flexible sigmoidoscopy workforce (NHS England NME program) ¹⁸ . Trainees from the NME specialty achieved certification in the shortest overall training time, despite logging significantly more precertification procedures than GS (for gastroscopy and colonoscopy certification) and gastroenterologists (for full colonoscopy certification). The mismatch between the number of procedures and time to certification indicates variation in access to training opportunities, intensity of training and the potential for differences in acquisition of endoscopic skills according to training specialty. UK trainee survey data suggest disparities between specialties ^{23 24} , training grades ²⁵ , and regions ²⁵ in terms of access to endoscopy training. Conflicting commitments with general medical on-call rotas (gastroenterology and GS) and theater-based training (GS), competition for training (with expansion of NME numbers), and a shortage of dedicated training lists are cited as contributory factors ²⁶ . These surveys highlight the imperfections and pressures of real-world endoscopy training systems within the UK, which may be less applicable internationally. It is important to note that, with exception to gastroscopy certification in gastroenterology trainees, JAG certification is not a criterion for completion of gastroenterology and GS specialist training programs, which typically last 5 years.

For each specialty, median procedural numbers at certification exceeded the minimum thresholds set by JAG which were originally determined using procedural completion metrics ^{27 28} . As certification is a composite endpoint requiring the maturation of a range of technical and non-technical competencies which are assessed within DOPS, this study suggests a more realistic requirement of procedural numbers and training times to achieve this competence standard. It is recognized that within a cohort, trainees develop competency at different rates. Setting higher minimum procedural numbers may penalize those who develop competency at a faster pace, whereas lowering the threshold may reduce the breadth of caseload encountered by a trainee. The certification process provides a competency safeguard for those who elect to trigger summative assessment towards the minimum end of the JAG procedural threshold. Formative DOPS assessments enable trainers to monitor development of individual competencies during training ⁸ and to gauge readiness for summative assessment. Trainees may apply for summative DOPS assessment upon fulfilment of JAG criteria, but crucially, also requires support from the trainer and training lead to entrust them for unsupervised practice.

The effect of specialty on performance and skills acquisition during endoscopy training is unclear. Previous JETS e-portfolio analyses ^{27 28} had suggested an association between trainee specialty and unassisted procedural completion rates. After multivariate analysis to account for factors including lifetime procedural count and training intensity, GS trainees were more likely to achieve unassisted procedural completion for gastroscopy ²⁷ and colonoscopy ²⁸ compared to gastroenterology and NME trainees. However, these studies involved trainees at early stages of training, using procedural completion metrics as the outcome. In contrast, our study explored differences in additional KPIs between specialties in a certified cohort who were deemed competent for independent practice. For gastroscopy, although there were post-certification reductions in rates of D2 intubation and J-maneuver, average rates remained above JAG standards. Unadjusted CIR fell after provisional colonoscopy certification but remained above the 90 % national standard, with no significant changes seen after progression to full certification. The PDR in both periods remained above the 20 % aspirational target set in UK guidelines ²⁹ , and comparable to the PDR of 20.3 % for diagnostic colonoscopy demonstrated within the last UK colonoscopy audit ³⁰ . Despite subtle variations, overall performance for all specialty trainees was maintained above the required JAG standards ^{4 5 6} . This suggests that JAG requirements for trainee certification are appropriate in determining subsequent competence for independent practice, defined by national quality standards and guidelines ^{4 5 6 29 31} . Furthermore, the failure rates of 10 % during the summative phase and 0.8 % following central JETS assessor review, attests to the stringency of the certification process.

Several limitations require discussion. First, our study was an exploratory evaluation of the certification cohort. It did not include or assess outcomes of trainees who had commenced endoscopy training or completed gastroenterology/GS specialist training, but not achieved certification. KPIs were limited to those recorded by the JETS e-portfolio and those included in the analysis. They did not assess procedural factors, e. g. age, procedure indication, diagnoses, bowel preparation, discomfort scores, or sedation use, or training factors, e. g. training intensity, which has been explored previously ^{27 28} . It was therefore unclear whether decrements in performance metrics, e. g. PDR in GS trainees after full colonoscopy certification, could be due to changes in caseload after certification. Second, due to the nature of the e-portfolio and e-certification roll-out, a significant number (37.1 %) of trainees had baseline procedures preceding use of the e-portfolio. Exclusion of these cases in the number of procedures needed to achieve certification analyses is a potential source of bias. It was therefore not possible to perform accurate trend analyses for time-to-certification due to the significant proportion who had baseline endoscopy experience prior to the implementation of the e-portfolio. Third, time-to-certification does not necessarily indicate time-to-competency; trainees may defer applying for certification until they feel ready to perform independently. The interval between eligibility for certification and application for certification was not studied. Fourth, although e-portfolio entries have been compared to real-world procedure entries and shown to be reliable ^{27 28} , analyses of a trainee-maintained e-portfolio are not free from selection bias and the unintended consequences of performance monitoring. As such, several measures have been implemented within the certification process to ensure validity of the submitted data. These include use of independent assessors for summative assessments, local review by the training lead, and review of each e-portfolio by JETS assessors. Finally, not all endoscopists continued with e-portfolio participation after certification; 61.2 % submitted the minimum post-certification procedures (≥ 10) to allow for paired KPI comparisons. The issue of entry bias will be addressed through the imminent rollout of the National Endoscopy Database in April 2018 ³² , which intends to auto-populate training records directly from UK endoscopy reporting systems into the JETS e-portfolio. JAG is currently reviewing certification requirements using a performance-based approach with emphasis on formative assessments and performance metrics, particularly when the latter may be assessed more reliably with the National Endoscopy Database. Such data will assist ongoing efforts to develop evidence-based training pathways with continuing review and refinement of specified elements and how these are delivered to greatest effect within the confines of other demands on trainees and trainers.

The value of JAG certification has been accepted by both medical and surgical training committees in the UK and is understood by all practicing endoscopists and trainees. Our data support an assertion that the competency-based elements that comprise the certification standard deliver competent endoscopists who can evidence their performance standard via the e-portfolio. Few other international training systems can provide this level of quality assurance to commissioners (payers) and the populations they serve.

Conclusion

The JAG has developed a national system for certification of trainees in upper and lower gastrointestinal endoscopy. The JETS e-portfolio collects standardized training data to provide real-time monitoring of progress of trainees against certification standards. Our data confirm that the certification process can be delivered for endoscopists from different speciality groups on a national basis, in a transparent and robust manner. Trainees achieving this benchmark generally maintain national quality standards during the early post-certification period of potentially independent endoscopic practice. JAG certification can therefore be considered a mark of quality in diagnostic endoscopy. Data from the JETS e-portfolio provide a firm basis to consider certification in other endoscopic modalities.