A Prospective Multicenter Study Evaluating Learning Curves and Competence in Endoscopic Ultrasound and Endoscopic Retrograde Cholangiopancreatography Among Advanced Endoscopy Trainees: The Rapid Assessment of Trainee Endoscopy Skills Study

Abstract

BACKGROUND & AIMS:

On the basis of the Next Accreditation System, trainee assessment should occur on a continuous basis with individualized feedback. We aimed to validate endoscopic ultrasound (EUS) and endoscopic retrograde cholangiopancreatography (ERCP) learning curves among advanced endoscopy trainees (AETs) by using a large national sample of training programs and to develop a centralized database that allows assessment of performance in relation to peers.

METHODS:

ASGE recognized training programs were invited to participate, and AETs were graded on ERCP and EUS exams by using a validated competency assessment tool that assesses technical and cognitive competence in a continuous fashion. Grading for each skill was done by using a 4-point scoring system, and a comprehensive data collection and reporting system was built to create learning curves by using cumulative sum analysis. Individual results and benchmarking to peers were shared with AETs and trainers quarterly.

RESULTS:

Of the 62 programs invited, 20 programs and 22 AETs participated in this study. At the end of training, median number of EUS and ERCP performed/AET was 300 (range, 155–650) and 350 (125–500), respectively. Overall, 3786 exams were graded (EUS, 1137; ERCP-biliary, 2280; ERCP-pancreatic, 369). Learning curves for individual end points and overall technical/cognitive aspects in EUS and ERCP demonstrated substantial variability and were successfully shared with all programs. The majority of trainees achieved overall technical (EUS, 82%; ERCP, 60%) and cognitive (EUS, 76%; ERCP, 100%) competence at conclusion of training.

CONCLUSIONS:

These results demonstrate the feasibility of establishing a centralized database to report individualized learning curves and confirm the substantial variability in time to achieve competence among AETs in EUS and ERCP. ClinicalTrials.gov: .

In the past decade, training in endoscopic retrograde cholangiopancreatography (ERCP) and endoscopic ultrasound (EUS) has shifted to dedicated advanced endoscopy fellowships at tertiary care centers, occurring in a fourth year of training after a standard gastroenterology fellowship.¹ Data from a recent survey suggest that only 9% and 4.5% of general gastroenterology trainees had anticipated volumes >200 in ERCP and EUS, respectively.² This shift has occurred in part because of the widespread acknowledgement that EUS and ERCP are technically challenging procedures to perform and are associated with a higher rate and wider range of adverse events compared with standard endoscopic procedures.^3,4 Ample evidence demonstrates the operator-dependent nature of these procedures and supports the need for additional training for the development of technical, cognitive, and integrative skills beyond those required for standard endoscopic procedures.⁵

Although advanced endoscopy fellowships are not recognized by the Accreditation Council for Graduate Medical Education (ACGME), there has been a dramatic increase in these programs in the past 15 years.¹ There is no fixed mandatory curriculum, and the necessary intensity and duration of training are highly variable and poorly defined. Advanced endoscopy has traditionally been taught by apprenticeship wherein a trainee is expected to develop skill and expertise with hands-on experience during a fixed duration of training. Competence in EUS and ERCP has historically been assessed by the trainers’ subjective assessment of overall competence and/or meeting an arbitrary volume threshold for procedures completed.⁶ At present, guidelines continue to use an absolute procedure volume to determine competence in EUS and ERCP, with thresholds varying between guidelines.^7–13 It should be noted that these guidelines lack validation with regard to competence, and these thresholds do not account for the variable rates at which trainees learn and acquire endoscopic skills.³

The investigators of this study have previously demonstrated substantial variability in achieving competence in EUS and ERCP and that a specific case load does not ensure trainee competence.^3–5 In addition, we showed that although trainees achieve competence in overall cannulation, there is a consistent need for continued improvement of native papilla cannulation, which is likely the ideal benchmark for competence in cannulation. Finally, these studies also emphasize the need to include all relevant technical and cognitive skills in the assessment of competence in EUS and ERCP. These results require validation in a large cohort of advanced endoscopy training programs. In addition, these studies do not address the impact and feasibility of providing periodic feedback to advanced endoscopy trainees (AETs) during training.

There is an increasing emphasis on standardizing competency assessment and demonstrating readiness for independent practice as medical training in the United States transitions from an apprenticeship model to competency-based medical education (CBME). The ACGME has replaced its reporting system with the Next Accreditation System (NAS), which is a continuous assessment reporting system focused on ensuring that specific milestones are reached throughout training, that competence is achieved by all trainees, and that these assessments are documented by training programs. Thus, it is incumbent on advanced endoscopy training programs and program directors to evolve with these new ACGME/NAS requirements and assess and document competence among all trainees.

By using a standardized competency assessment tool with a comprehensive data collection and reporting system, the primary aim of this prospective multicenter study was to validate learning curves in EUS and ERCP among AETs by using a large sample of advanced endoscopy training programs.

Methods

Study Design

This was a prospective multicenter cohort study that was conducted at 20 tertiary care referral centers (Supplementary Table 1). The study was approved by the Institutional Review Board or the Human Research Protection Office at each participating center (ClinicalTrials.gov: ), and consent to participate was obtained from all AETs. All authors had access to the study data and reviewed and approved the final manuscript.

Study Subjects

Advanced endoscopy fellowship program directors and AETs at all advanced endoscopy programs registered with the American Society for Gastrointestinal Endoscopy (ASGE) (https://www.asgematch.com/) were invited to participate in this study from July 2014 to June 2015. AETs were defined as trainees who had already completed a standard 3-year gastroenterology fellowship and were beginning 1 additional year of advanced endoscopy training. All AETs consented to be evaluated for the study and were introduced to the cognitive and technical aspects of EUS and ERCP procedures at the onset of their training (on the basis of institutional training curriculum). At study onset, AETs completed a questionnaire to determine their baseline characteristics and prior experience with EUS and ERCP (Supplementary Figure 1). AETs also completed a post-study questionnaire that assessed the number of EUS and ERCP exams completed during training, overall comfort level in independently performing EUS and ERCP, as well as comfort level performing individual components of these procedures (on the basis of published quality indicators)^14,15 (Supplementary Figure 2). Responses were recorded by using 5-point balanced Likert items (1, strongly agree; 2, tend to agree; 3, neutral; 4, tend to disagree; and 5, strongly disagree).

Competency-Assessment Tool and Grading Protocol

After the completion of 25 hands-on EUS and ERCP examinations, AETs were graded on every ERCP and every third EUS exam by attending endoscopists (trainers) at each center. This grading interval was based on a fairly homogeneous population of patients under-going EUS compared with ERCP and to reduce the burden of overall evaluations. We used The EUS and ERCP Skills Assessment Tool (TEESAT), a previously validated skills and competency assessment tool, in a continuous fashion throughout the duration of training to grade technical and cognitive skills in EUS and ERCP^3–5 (Supplementary Figure 3). Procedures in which the AETs had no hands-on participation were excluded from grading. Similarly, exams eligible for grading but incomplete for reasons such as medical instability were also excluded. Trainers were asked to complete the assessment immediately after the procedure to reduce recall bias, halo, and recency effect. Although self-explanatory, the process of systematic evaluations was explained, discussed, and clarified by the principal investigator and the program directors at all participating centers individually. The program director then ensured that all trainers and AETs were familiar with TEESAT’s specific assessment parameters and score explanations.

This tool uses a 4-point scoring system: 1 (superior), achieves independently; 2 (advanced), achieves with minimal verbal instruction; 3 (intermediate), achieves with multiple verbal instructions or hands-on assistance; and 4 (novice), unable to complete, requiring trainer to take over. Setting these anchors for specific skills and behaviors was critical to ensure that the data collected were reproducible from one evaluator to the next. Independent grading of individual end points was performed. In addition, a global rating scale was also used to provide an overall assessment of the AET, which used a 10-point scoring system: 1–3 (below average), 4–6 (average), 7–9 (above average), and 10 (attending level).

With regard to ERCP, TEESAT allows for documentation of the indication and the grade of difficulty by using the ASGE ERCP degree of difficulty grading system.¹⁶ The AET was graded for basic maneuvers and all relevant technical and cognitive aspects of ERCP and EUS (Supplementary Text). The time allowed for AET to attempt cannulation was recorded (calculated from the time the cannulation device was out of the duodenoscope to successful cannulation by AET or the duodenoscope taken over by the trainer). A clear distinction for grading was made by this tool on the basis of biliary versus pancreatic indication for ERCP (Supplementary Figure 3). Immediate post-procedure adverse events were documented.

Comprehensive Data Collection and Reporting System

To create a centralized national database, an integrated, comprehensive system was created that supported the data collection and addressed the reporting needs of this project, which included streamlining data collection from all participating centers and applying cumulative sum analysis (CUSUM analysis) (Supplementary Text). All users of the site were provided unique logins, and on the basis of their logins, program directors and AETs were allowed to view individual learning curves and compare results with peers. Learning curves were provided on a quarterly basis (Figure 1).

Figure 1.

Example of graphic representation of learning curves provided to participating center on quarterly basis that includes individual learning curves for the participating AET (green) and in comparison with the national average (orange).

Endoscopic Ultrasound and Endoscopic Retrograde Cholangiopancreatography Procedures

All EUS and ERCPs performed in this study were part of routine clinical care provided at the participating centers. The level of AET participation was at the discretion of the attending endoscopist.

Study Outcomes

The primary study outcome was to validate EUS and ERCP learning curves (overall and individual end points) by using a large national sample of advanced endoscopy programs. The secondary study outcomes were (1) to develop and determine the feasibility of a centralized national database that would allow program directors and trainees to generate reports assessing performance in relation to peers, (2) compare the proportion of AETs achieving competence by using the global rating scale with TEESAT, (3) critically examine and report on the composition of EUS and ERCP training in the United States, and (4) to report practice plans and the number of AETs expressing comfort level in EUS and ERCP after completion of training.

Statistical Analysis

As previously described, CUSUM analysis was applied to create learning curves with regard to overall and individual technical and cognitive end points in EUS and ERCP for each AET (Supplementary Text).^3,4 In the primary analysis, a rating of 1 (no assistance) or 2 (minimal verbal cues) for individual end points was considered a success, whereas a rating of >2 was considered a failure. For the global rating scale using the 10-point scoring system, success was defined as a score of 7–10. The overall scores for the entire ERCP and EUS procedures were calculated as the median performance for all end points. In addition to overall EUS and ERCP performance, comprehensive learning curves were created for individual technical and cognitive end points. The gold standard for this analysis was the impression of the attending physician (trainer). Sensitivity analyses were performed with varying unacceptable failure rates (p1), and competence was also assessed by using a stringent definition of success defined by a score of 1 for individual end points on TEESAT or a score of 10 by using the global rating scale. AETs with <20 overall evaluations or for a specific end point were excluded. Agreement between the results using TEESAT (checklist tool) and the global rating scale was assessed by using kappa (κ) statistics with 95% confidence intervals (CIs). Individual and combined graphs to illustrate the change in cannulation success outcome with increasing ERCP volume during training (proxy measure of the time variable during the 1-year training) were constructed. The Cochran-Armitage trend test was used to assess improvement in success rate (defined as a score of 1 or 2 on TEESAT) by blocks of 10 across time.

Results

Of the 62 advanced endoscopy training programs invited, a total of 20 training programs and 22 AETs participated in this study. On the basis of inclusion criteria, 20 AETs were included in the final analysis. Before starting their advanced endoscopy training, 59.1% and 68.2% of AETs reported formal training on cognitive aspects of EUS and ERCP, respectively. Similarly, a majority of AETs reported at least some hands-on training in EUS (63.6%) and ERCP (86.4%) before their advanced endoscopy training. The median number of EUS and ERCP exams performed before advanced endoscopy training was 26 (range, 1–120) and 50 (range, 4–200), respectively.

Primary Analysis: Learning Curves and Competence in Endoscopic Ultrasound and Endoscopic Retrograde Cholangiopancreatography

Endoscopic ultrasound assessment.

Overall, this study included 1137 graded EUS exams. Using the primary definition of success, defined as a score of 1 or 2 for individual end points on cognitive and technical aspects of EUS on TEESAT, and using an acceptable failure rate (p0) of 0.1 and an unacceptable failure rate (p1) of 0.3, the vast majority of AETs achieved competence in overall cognitive (76.4%) and overall technical (82.3%) aspects of EUS at the end of their training. The variable number of AETs achieving competence for individual technical and cognitive end points is highlighted in Table 1. A graphical representation of learning curves using CUSUM analysis among AETs by using median scores for overall technical and cognitive aspects of EUS is shown in Figure 2. A positive deflection indicates a false (incompetent) result (score of 3 or 4) on an assessment, whereas a negative deflection represents a true (competent) result (score of 1 or 2).

Table 1.

AETs Achieving Competence in EUS

	No. of AETs meeting inclusion criteria	No. of evaluations	No. of AETs achieving competence (%), primary analysisa	No. of AETs achieving competence (%), sensitivity analysisb
Technical aspect
Intubation	17	1063	17 (100)	16 (94.1)
AP window	6	281	6 (100)	4 (66.6)
Body of pancreas	15	908	12 (80)	10 (66.6)
Tail of pancreas	15	887	12 (80)	6 (40)
Head/neck of pancreas	16	911	14 (87.5)	8 (50)
Uncinate process	15	753	11 (73.3)	3 (20)
Ampulla	13	702	9 (69.2)	4 (30.7)
Gallbladder	10	407	9 (90)	6 (60)
Common bile duct/common hepatic duct	15	822	14 (93.3)	5 (33.3)
Portosplenic confluence	13	700	12 (92.3)	7 (53.8)
Celiac axis	14	832	14 (100)	7 (50)
Achieve FNA	10	344	5 (50)	1 (10)
Achieve celiac plexus block/neurolysis	16	960	15 (93.7)	9 (56.2)
Overall technical	17	1070	14 (82.3)	11 (64.7)
Cognitive aspect
Identify lesion of interest of appropriately ruled out	16	970	13 (81.2)	7 (43.7)
Appropriate differential diagnosis	16	868	14 (87.5)	8 (50)
Appropriate management plan	16	960	15 (93.4)	5 (31.3)
Overall cognitive	17	1061	13 (76.4)	8 (47)

AET, advanced endoscopy trainees; AP, aortopulmonary; EUS, endoscopic ultrasonography; FNA, fine-needle aspiration.

^aPrimary analysis: success defined as score of 1 or 2 (no assistance/minimal verbal cues). Acceptable failure rate, p0 = 0.1 and unacceptable failure rate, p1 = 0.3.

^bSensitivity analysis: success defined as score of 1 (stringent definition of success).

Figure 2.

Graphic representation of learning curves among AETs by using CUSUM analysis for overall technical and cognitive aspects of EUS by using acceptable and unacceptable failure rates of 10% and 30%, respectively.

Endoscopic retrograde cholangiopancreatography assessment.

Overall, this study included 2280 biliary ERCP exams and 369 pancreatic ERCP exams. By using the primary definition of success, 60% of AETs achieved overall technical competence in biliary ERCP, and 100% achieved overall cognitive competence. The variable number of AETs achieving competence for individual technical and cognitive end points in biliary ERCP is highlighted in Table 2. Graphical representations of learning curves using CUSUM analysis among AETs by using median scores for overall technical and cognitive aspects and individual end points such as cannulation of the desired duct and sphincterotomy are shown in Figures 3 and 4. Consistent with results from our pilot study,³ although the majority of AETs achieved competence for the end point of overall cannulation, only 17.6% of AETs achieved competence for the end point of cannulation in cases with a native papilla. The limited number of evaluations for pancreatic indications precluded any meaningful learning curve analysis for pancreatic ERCPs. There was a statistically significant improvement in overall cannulation rates and cannulation rates in cases with a native papilla (biliary ERCP, both P < .001; Supplementary Figure 4).

Table 2.

AETs Achieving Competence in Biliary ERCP

	No. of AETs meeting inclusion criteria	No. of evaluations	No. of AETs achieving competence (%), primary analysisa	No. of AETs achieving competence (%), sensitivity analysisb
Technical aspect
Intubation	20	2239	20 (100)	19 (95)
Achieve short position	20	2226	19 (95)	15 (75)
Identify the papilla	20	2223	19 (95)	18 (90)
Overall cannulation	19	2075	13 (68.4)	6 (31.5)
Cannulation, native papilla	17	1041	3 (17.6)	0 (0)
Stent removal	14	737	13 (92.8)	9 (64.2)
Wire placement in biliary duct	18	1815	16 (88.8)	8 (44.4)
Sphincterotomy	15	731	10 (66.6)	0 (0)
Balloon sweep	19	1602	18 (94.7)	10 (52.6)
Stone clearance	14	697	12 (85.7)	6 (42.8)
Stricture dilation	10	432	9 (90)	3 (30)
Stent insertion	17	1029	14 (82.3)	3 (17.6)
Overall technical	20	2259	12 (60)	5 (25)
Cognitive aspect
Demonstrate clear understanding of indication	20	2264	20 (100)	14 (70)
Appropriate use of fluoroscopy	20	2169	18 (90)	7 (35)
Proficient use of real-time cholangiogram	20	2219	19 (95)	9 (45)
Logical plan based on cholangiogram	20	2220	19 (95)	10 (50)
Demonstrate understanding of use of indomethacin	19	1630	19 (100)	16 (84.2)
Overall cognitive	20	2268	20 (100)	17 (85)

AET, advanced endoscopy trainees; ERCP, endoscopic retrograde cholangiopancreatography.

^aPrimary analysis: success defined as score of 1 or 2 (no assistance/minimal verbal cues). Acceptable failure rate, p0 = 0.1 and unacceptable failure rate, p1 = 0.3.

^bSensitivity analysis: success defined as score of 1 (stringent definition of success).

Figure 3.

Graphic representation of learning curves for ERCP (overall technical and cognitive aspects).

Figure 4.

Graphic representation of learning curves for cannulation of bile duct in native papilla cases and sphincterotomy.

Sensitivity analyses.

Sensitivity analyses were performed by using a stringent definition of success (success defined as a score of 1 for individual end points) as highlighted in Tables 1 and 2 (Supplementary Text).

Current Status of Endoscopic Ultrasound and Endoscopic Retrograde Cholangiopancreatography Training

The median number of EUS exams performed per AET was 300 (range, 155–650). In terms of indications, suspected pancreatic mass accounted for 24.5% of the graded procedures, and pancreatic cyst (17.8%), sub-epithelial lesion (7%), and luminal malignancy (6.9%) represented the other major indications. The majority of the graded EUS exams were performed by using the linear echoendoscope (n = 768, 67.5%) and in the ambulatory setting (n = 940, 82.6%).

At the end of training, the median number of ERCP exams performed/AET was 350 (range, 125–500), and the median number of ERCP exams performed/AET in patients with a native papilla was 51 (range, 32–79). The majority of graded ERCPs (86%) were performed for biliary indications, and 59% of all ERCPs were performed as outpatient procedures. For biliary ERCPs, major indications included stricture (34.4%), chol-edocholithiasis (32.1%), stent removal/exchange (28.8%), post-transplant stricture (9.2%), and bile leak(5.9%). The distribution of exams on the basis of the ASGE degree of difficulty grade was as follows: biliary grade 1, 1762 (77%); grade 2, 348 (15%); grade 3, 146 (7%); and missing data, 24 (1%). At a trainee level, the median ASGE degree of difficulty grade was 1, and mean ranged from 1.1 to 1.5. Of all the graded exams, ERCPs were performed in 1371 cases (52%) with a native papilla, and sphincterotomy was performed in 40% of all cases. The overall mean time allowed for AETs to cannulate the duct of interest was 4 minutes (standard deviation [SD], 4.3), and median time was 2 minutes (25%, 75% interquartile range [IQR], 1–5 minutes). The mean time allowed for cannulation in cases with a native papilla was 5.7 minutes (SD, 4.8), and in cases that the AET failed cannulation it was 6.2 minutes (SD, 5), and median time was 8 minutes (25%, 75% IQR, 5–10 minutes). There was no change in the time allowed for native papilla cannulation during the 1-year training period (P = .28) (Supplementary Figure 5). Overall, AETs were exposed to a limited number of graded ERCPs that required advanced cannulation techniques (hands-on or observation) such as placement of pancreatic duct stent to facilitate biliary cannulation, double-wire technique, and precut sphincterotomy (n = 145, 6%). With regard to immediate post-ERCP adverse events, there were 59 patients admitted for abdominal pain, 17 with pancreatitis, 7 with bleeding, and 5 with perforations. After EUS, 8 patients were admitted for abdominal pain and 2 with pancreatitis, and 1 perforation was documented.

Comparison of the Global Rating Scale With The EUS and ERCP Skills Assessment Tool

As highlighted in Supplementary Table 2, a smaller proportion of AETs achieved competence in EUS and ERCP by using the global rating scale, both when success was defined as a score 7–10 and when using a stringent definition of success (score of 10, “attending level”). The overall agreement between results obtained by using the global rating scale and those using TEESAT was fair for competence in EUS (overall technical: κ = 0.38 [95% CI, 0–0.79], overall cognitive: κ = 0.25 [95% CI, 0–0.72]) and slight to fair for competence in ERCP (overall technical: κ = 0.40 [95% CI, 0–0.79], overall cognitive: κ = 0.10 [95% CI, 0–0.29]).

Post-Study Questionnaire: Comfort Level in Endoscopic Ultrasound and Endoscopic Retrograde Cholangiopancreatography and Practice Plans

Of the AETs who completed this questionnaire, 100% strongly agreed/tend to agree regarding their comfort level in independently performing ERCP, and 84.7% were comfortable performing EUS independently. Nearly all AETs were comfortable with deep cannulation of the duct of interest, sphincterotomy, stone clearance (<1 cm), and placement of pancreatic duct stents. Nearly all AETs felt comfortable in performing EUS–fine-needle aspiration (FNA), EUS-guided celiac plexus block/neurolysis, and EUS-guided pseudocyst drainage. However, 50% of AETs were not comfortable placing fiducials and performing biliary/pancreatic EUS-guided rendezvous procedures (Supplementary Table 3). Nearly half planned to practice at an academic center and expected the majority of their practice to be advanced endoscopy (Supplementary Table 4).

Discussion

Because of the increasing emphasis on quality metrics and competency in healthcare, the ACGME replaced their reporting system with the NAS, focusing on CBME. CBME is a concept that is quickly moving from theory to reality for subspecialty fellowship training.¹⁷ In addition, quality measurement and improvement with the help of quality indicators in endoscopy have garnered a great deal of interest in recent times.^14,15 Reimbursement is increasingly being tied to the performance and quality of care as we transition away from a fee-for-service model, although there is little movement in this direction for EUS and ERCP. Within the realm of advanced endoscopy training, current healthcare system (payers) must respond to these needs.

With this foundation, we designed a prospective multicenter study to assess learning curves in EUS and ERCP. By using a standardized evaluation tool and CUSUM analysis, the results of this study demonstrate the substantial variability in the learning curves and number of AETs achieving competence in EUS and ERCP (overall and individual end points) at the end of their advanced endoscopy training. These results validate the findings from our pilot studies and recently published systematic reviews^3–5,18,19 and validate the recommended shift from relying on an absolute number of procedures to determine competence to using performance metrics with well-defined and validated thresholds of performance. This study strengthens the value of selective native papilla deep cannulation as the new benchmark for assessing competence in cannulation during advanced endoscopy training and independent practice.^3,14,20

By using a novel comprehensive data collection and reporting system, this study also demonstrated the feasibility of creating a centralized database that allowed for continuous monitoring and reporting of individualized learning curves provided on a quarterly basis. This study highlights the variability in the training curriculum, the number of procedures performed during training, and limited exposure to advanced ERCP cannulation techniques. Thus, specific training measures and strategies such as ex vivo models to increase exposure to therapeutic EUS and advanced ERCP techniques are warranted. Above all, there is a need not only to establish a standardized advanced endoscopy training curriculum but also to establish the minimum standards for advanced endoscopy training programs. Funding and implementation of a system that supports a national centralized database will warrant the support of gastrointestinal societies and credentialing bodies.

Recent data suggest that evaluations using global rating scales may demonstrate superior or comparable reliability and validity measures and sensitivity to levels of expertise compared with evaluation tools using checklists.²¹ However, there are limited data comparing these 2 approaches in advanced endoscopy training. Discordant results between an objective checklist-based evaluation tool (TEESAT) compared with a global rating scale using a 10-point scoring system was noted in this study. The reasons for these results are not clear. The role of global rating scales in assessing competence in advanced endoscopy training will be further clarified in an ongoing study. Although the use of checklist-based evaluation tools is more time-consuming, it appears unlikely that global rating scales will completely replace checklist evaluation tools because the latter provide granular and actionable feedback to trainees to facilitate ongoing improvement and can allow monitoring competence in key EUS and ERCP quality indicators.^3,14,15

Our post-study questionnaire showed that there is a lack of concordance between the results of competence as assessed by learning curve analysis and comfort level expressed by AETs in independently performing EUS and ERCP after completion of their advanced endoscopy training. This raises several important questions. Do we expect AETs to meet our strict definition of competence when they graduate? Specifically, it is clear that trainees continue to improve during training and after completion of training and may ultimately achieve our predefined measures of competence during independent practice. However, the impact of structured feedback on learning curves, specifically related to quality indicators in EUS and ERCP, during the first year of independent practice for AETs has not been evaluated. This is an important component of construct validity for the proposed evaluation tool and novel Web-based comprehensive data collection and reporting system. Addressing this priority research question along with validation of above described results are the primary aims of our ongoing prospective multicenter trial (RATES 2, ClinicalTrials.gov: ).

There are limitations of this study that merit discussion. This study included about one-third of the advanced endoscopy programs in the country, thus limiting the overall generalizability of these results. However, it should be noted that this is the largest study assessing learning curves and competence in EUS and ERCP in the United States. We compared the basic attributes (number of trainees/year, annual volume of EUS and ERCP offered during training) between participating and non-participating programs, and no differences were noted between the 2 groups, suggesting generalizability of these results (Supplementary Table 5). The limited number of participating AETs precluded stratified analysis that was based on AET background training, type of cases, and number of procedures performed by the AET. Although all advanced endoscopy training programs registered in the ASGE Advanced Endoscopy Matching Program were invited to participate in this study, selection bias cannot be excluded. The subjective opinion of the attending endoscopists is an inherent limitation of any study assessing learning curves and competence by using standardized assessment tools. The interobserver and intraobserver agreement among trainers using TEESAT was not evaluated as a part of this study. This study included trainers with varying cumulative experience and training styles that may have contributed to the variability in trainee performance. However, this was accounted for by the use of a standardized evaluation tool that was discussed and agreed on by the principal investigator and the program directors and by setting anchors for specific end points. The investigators also acknowledge the possibility of spectrum bias because various stages and grades of disease cases were included in the grading process. Self-selection or skipping of cases for evaluation by AETs cannot be excluded. The EUS grading protocol limited evaluation of competence for low-volume EUS exams such as celiac plexus block and fiducial placement. Missing data are also a limitation well-described in previous studies evaluating learning curves in endoscopic procedures and shown not to influence overall outcomes. It may be argued that the time allowed for an AET to cannulate was limited. However, we believe that this is a true representation of current clinical practice and training. The authors acknowledge that the proportion of AETs achieving competence in cannulation may have increased if AETs were allowed more time to cannulate. Balancing efficiency and safety with training continues to be a challenge for trainers in advanced endoscopy. Because of the limited number of cases, this study is unable to assess learning curves involving pancreatic ERCPs and advanced EUS and ERCP techniques, and it remains unclear whether competency in standard EUS and ERCP translates to competency in more advanced techniques. This study only assessed immediate post-procedure adverse events. It would be more meaningful to study the association between AET participation and post-procedure adverse events assessed at a 30-day follow-up period. This question requires further research and is being explored in an ongoing multicenter study (ClinicalTrials.gov: ). The strengths of this study include (1) defining learning curves in EUS and ERCP in one of the largest cohorts of AETs and advanced endoscopy training programs, (2) using a standardized evaluation tool that encompassed all relevant technical and cognitive aspects necessary to perform a high-quality EUS and ERCP, (3) comprehensive data collection and reporting system, and (4) robust statistical methodology for learning curves using CUSUM analysis.

In conclusion, the results of this study have significant implications in this era of CBME. This study confirms the substantial variability in learning curves and competence among AETs in EUS and ERCP and validates the shift away from performing a threshold number of procedures to determine competence. We have demonstrated the feasibility of establishing a centralized database to report “on-demand” individualized EUS and ERCP learning curves. This infrastructure has the potential to help program directors/trainers and trainees identify specific skill deficiencies in training and thus allow for tailored, individualized remediation. Establishing reliable and generalizable standardized learning curves (milestones) and competency benchmarks will facilitate the ability of training programs to evolve with the new ACGME/NAS reporting requirements and demonstrate that AETs have attained the technical and cognitive skills that are required for safe and effective unsupervised practice in advanced endoscopy.

Abbreviations used in this paper:

ACGME

Accreditation Council for Graduate Medical Education

AETs

advanced endoscopy trainees

ASGE

American Society for Gastrointestinal Endoscopy

CBME

competency-based medical education

CI

confidence interval

CUSUM analysis

cumulative sum analysis

ERCP

endoscopic retrograde cholangiopancreatography

EUS

endoscopic ultrasonography

FNA

fine-needle aspiration

IQR

interquartile range

κ

kappa value

NAS

Next Accreditation System

SD

standard deviation

TEESAT

The EUS and ERCP Skills Assessment Tool