Effect of Polypectomy Simulation-Based Mastery Learning on Skill Retention Among Practicing Endoscopists

Abstract

Purpose

Practicing endoscopists frequently perform and teach screening colonoscopies and polypectomies, but there is no standardized method to train and assess physicians who perform polypectomy procedures. The authors created a polypectomy simulation-based mastery learning (SBML) curriculum and hypothesized that completion of the curriculum would lead to immediate improvement in polypectomy skills and skill retention at 6 and 12 months after training.

Method

The authors performed a pretest-posttest cohort study with endoscopists who completed SBML and were randomized to follow-up at 6 or 12 months from May 2021 to August 2022. Participants underwent SBML training, including a pretest, a video lecture, deliberate practice, and a posttest. All learners were required to meet or exceed a minimum passing standard on a 17-item skills checklist before completing training and were randomized to follow-up at 6 or 12 months. The authors compared simulated polypectomy skills performance on the checklist from pretest to posttest and posttest to 6- or 12-month follow-up test.

Results

Twenty-four of 30 eligible participants (80.0%) completed the SBML intervention, and 20 of 24 (83.0%) completed follow-up testing. The minimum passing standard was set at 93% of checklist items correct. The pretest passing rate was 4 of 24 participants (16.7%) compared with 24 of 24 participants (100%) at posttest (P < .001). There were no significant differences in passing rates from posttest to combined 6- and 12-month posttest in which 18 of 20 participants (90.0%) passed.

Conclusions

Before training and despite years of clinical experience, practicing endoscopists demonstrated poor performance of polypectomy skills. SBML was an effective method for practicing endoscopists to acquire and maintain polypectomy skills during a 6- to 12-month period.

In the United States, approximately 6.8 million patients are screened yearly for colorectal cancer (CRC) by colonoscopy.¹ These life-saving screening procedures are performed to detect and remove polyps from the colon (i.e., polypectomy), which, if incompletely removed, could progress to CRC. However, incomplete polypectomy, in which polyps are not fully resected, occurs in up to 10% of polypectomy procedures.^2,3 As a result, up to 30% of CRCs diagnosed within 60 months of a previous colonoscopy are thought to result from incomplete polypectomies.^2–8

Practicing endoscopists frequently perform colonoscopies and polypectomies, but there is no standardized method to train and assess physicians who perform polypectomy procedures.⁹ Training and education for colonoscopy with polypectomy are generally limited to fellowship, often relying on the “see one, do one, teach one” approach.^10–12 Three studies revealed that many gastroenterology fellows lack the skills to perform polypectomy competently,^11–13 suggesting insufficient training in polypectomy procedures during fellowship. Additional studies performed at our institution showed that poor performance also extended to experienced practicing endoscopists who performed and supervised procedures.^2,14 Specifically, only 56% and 64% of video-recorded patient polypectomies were performed competently.

To address this issue, attending physicians who performed and supervised trainees performing screening colonoscopies at our institution were offered an opportunity to participate in a simulation-based intervention in which they practiced polypectomy skills.¹⁵ However, when measuring changes in skills before and after training, there was only a modest improvement and only 71% performed the procedure competently.¹⁵ Therefore, we developed a more intensive curriculum featuring simulation-based mastery learning (SBML) to ensure all physicians would perform the procedure competently.

SBML is a rigorous form of competency-based education in which all learners are required to meet or exceed a minimum passing standard (MPS) at posttest before completing training.^16,17 Learners who do not initially meet this standard complete additional deliberate practice until they reach the MPS. The MPS is set by a panel of experts at a high level that is deemed safe and effective for patient care.¹⁸ Prior studies have shown that training with SBML improves clinical skills and patient outcomes in procedures such as central venous catheter insertion,^19–21 paracentesis,^22,23 thoracentesis,^24,25 and ultrasound-guided intravenous catheter insertion.^26–28 Additionally, SBML results in long-term retention of procedural skills.^29–31

To our knowledge, few studies have evaluated attending physician performance using simulation-based education,^15,32–38 only 2 studies have evaluated attending physician performance after SBML,^32,35 and no studies have assessed skill retention among attending physicians after an SBML intervention. Additional study of SBML for attending physicians has potential implications for credentialing, continuing medical education, and medical education (because many of these attending physicians supervise residents and fellows). We hypothesized that practicing endoscopists who completed polypectomy SBML would, at baseline (during a pretest), have variable polypectomy skills performance, significantly improve their polypectomy skills immediately after training (at posttest), and retain their skills at 6 and 12 months after initial training.

Method

We performed a pretest-posttest cohort study with randomized follow-up at 6 or 12 months of faculty endoscopists’ simulated polypectomy skills at Northwestern University Feinberg School of Medicine from May 2021 to August 2022. We compared simulated polypectomy performance from pretest to posttest and posttest to 6- or 12-month follow-up. The Northwestern University Institutional Review Board approved this study.

Participants

All board-certified practicing endoscopists who performed screening colonoscopy and polypectomy at Northwestern Memorial Hospital, a tertiary care, academic medical center affiliated with Northwestern University Feinberg School of Medicine, were eligible to participate in polypectomy SBML training. Participation was voluntary, and results were not shared with department leadership. Participating endoscopists provided written informed consent before completing SBML and were randomized to the 6- or 12-month follow-up examination group using a computerized “coin flip” with atmospheric noise.

Procedure

Participants provided demographic and clinical experience data, including age, sex, and years in practice. We noted whether participants had participated in prior polypectomy simulation training. We also obtained actual numbers of colonoscopies and polypectomies performed by each participant a year before the study through follow-up testing using an electronic database that captures all colonoscopy and polypectomies performed at Northwestern Memorial Hospital. First, participants took a skills pretest during which they removed 3 sessile and 2 stalked polyps on a bovine colonoscopy simulator (see below). Second, they watched a video lecture on polypectomy best practices (30 minutes) followed by deliberate skills practice on the simulator with an expert instructor (60 minutes). Didactive video and teaching sessions included a review of polyp classification systems, polyp types and locations, polypectomy equipment, and specific techniques for polyp removal. Third, participants completed a posttest (within 2 weeks of the deliberate practice session) during which they again removed 3 sessile and 2 stalked polyps on the simulator. At posttest, all learners were required to meet or exceed a previously determined MPS on all 5 polypectomies. Those who were unable to meet the MPS underwent more deliberate practice until they met the standard at retesting. To assess skill retention, participants were randomized to follow-up testing at 6 or 12 months, when they again removed 3 sessile and 2 stalked polyps on the simulator.

Simulated setting

We replicated our institution’s endoscopy suite at the Northwestern simulation lab with appropriate equipment and personnel (e.g., nurse and/or technician). A high-fidelity ex vivo bovine colon model was used for the simulation experience (EndoSim LLC). Each colon was manipulated to create sessile and stalked lesions that ranged from 8 to 15 mm in size. We purposefully selected these polyp sizes because diminutive polyps are less relevant technically and most larger polyps are referred to advanced endoscopists (interventional gastroenterologists). The model tissue allows for all normal colonoscopy functions to be used, including water irrigation, insufflation, cautery, and polypectomy.

Measurement

Testing sessions were graded live by 1 of 4 authors (R.V.P., R.N.K., A.A.A., and S.K.) using the modified Direct Observation of Polypectomy Skills (m-DOPyS) checklist. The m-DOPyS was previously developed as a shorter version of the DOPyS, both of which produce valid and reliable data.^2,14,39–41 The original DOPyS tool contained 34 items and was used to assess actual patient live and videorecorded polypectomy skills. The DOPyS was created by a working group with expertise in colonoscopy within the British Society of Gastroenterology.³⁹ The group performed polypectomy task deconstruction and achieved consensus on a final structured checklist (DOPyS). Subsequently, 60 videorecorded actual patient polypectomies were scored by raters using this checklist, and 98% agreement was achieved.³⁹ In another study, the data from the DOPyS tool discriminated between polypectomies performed by endoscopists with differing levels of experience.⁴⁰ Subsequently, the DOPyS has been used in multiple evaluations of live and videorecorded polypectomies,^{2,11–14,39–42} but it has never been used for simulation-based interventions. Therefore, the m-DOPyS was developed as a shorter version of the DOPyS because our simulation environment could not fully replicate the entire polypectomy procedure from start to finish and all simulation sessions were videorecorded for later assessment by busy endoscopists.¹⁵ To develop the m-DOPyS, we used the modified Delphi technique with 11 board-certified gastroenterologists with expertise in polypectomy who determined which steps on the original DOPyS tool would be retained or omitted for simulated assessments.¹⁵ This earlier work demonstrated that scoring simulated polypectomies using the m-DOPyS was feasible and achieved moderate interrater reliability (κ = 0.67).¹⁵

The m-DOPyS contains 17 items for each polyp, with different steps for sessile and stalked.¹⁵ The first 16 items were specific steps required to adequately remove the polyp and were scored on a 4-point Likert scale that was dichotomized into not done or done incorrectly (1 or 2) and done correctly (3 or 4). The final step (item 17) was a global overall rating scale scored on a 4-point Likert scale that was also dichotomized into not done or done incorrectly (1 or 2) and done correctly (3 or 4). The rating scale was scored as follows: 1, frequent uncorrected errors; 2, some errors were uncorrected (technique still could be improved); 3, the performer was competent and safe throughout procedure; and 4, the performer was highly skilled.^2,14,39–41 All 4 raters (R.V.P., R.N.K., A.A.A., and S.K.) discussed the rating scale and calibrated their grading on the checklist through discussion of what each score meant. They each then scored 8 archived simulated polypectomy videos from our previous study to assess interrater reliability.¹⁵

The MPS for the m-DOPyS checklist was also established by the 11 board-certified gastroenterologists with expertise in polypectomy using the Mastery Angoff standard setting method.^18,43 The Mastery Angoff method requires expert judges to determine the percentage of “well-prepared” participants who would perform each checklist item correctly after training. Judges considered a “well-prepared” participant as one who would perform the procedure safely and independently on actual patients. The items were then averaged to come up with a mean for each judge’s overall checklist score. Overall checklist scores were averaged to determine the final MPS. The MPS was set at 93% for the posttest, which allowed participants to miss 1 of the first 16 checklist items for each of the 5 polypectomies. Participants were further required to achieve a 3 or 4 on the final item (item 17, rating of overall performance) to reach the MPS. Therefore, if learners missed more than 1 item (got a 1 or 2 score on the Likert scale on the 16 items or a 1 or 2 on the scale for item 17) for any of the 5 polyps at posttest, they would need to return to the simulation laboratory to complete additional deliberate practice and retesting.

Statistical analysis

We evaluated interrater reliability using the Cohen κ coefficient comparing S.K.’s checklist scores with the scores of R.V.P., R.N.K., and A.A.A. We compared pretest to posttest median scores and posttest scores to 6- and 12-month follow-up median scores using the paired Wilcoxon signed-rank test (because data were nonparametric). We compared the percentage of participants reaching the MPS between pretest and posttest scores and posttest and follow-up scores using the Fisher exact test. Spearman ρ coefficients were calculated to evaluate correlations between years in practice, number of colonoscopies, and number of polypectomies performed and pretest and follow-up scores (6- and 12-month scores combined). Finally, we compared pretest performance between those who participated in prior simulation and those who did not for differences in median scores (Mann-Whitney U test) and percentage passing (Fisher exact test). All statistical tests were performed using SPSS software, version 28.0 (IBM Corp).

Power calculations

Our prior work found a mean difference between pretest and posttest procedural skills checklists of approximately 33%; mean (SD) pretest simulated skill checklist performance is 61.5% (24.5%) items correct, whereas mean (SD) posttest performance is 94.9% (6.6%).^{25,33,34,44,45} A sample size of 13 has 90% power to detect a much smaller difference between a pretest mean (SD) score of 85.0% (10.0%) to a posttest mean (SD) score of 95.0% (10.0%), using a paired 2-sided t test (P < .05). A sample size of 10 has 80% power to detect the same 10% effect size. These same calculations hold true for posttest to follow-up test scores.

Results

Twenty-five of 30 eligible physicians (83.3%) consented to participate in the study; 2 of 3 colorectal surgeons and 3 of 27 gastroenterologists did not consent to participate. Of the 25 physicians who started the SBML intervention, 1 dropped out after being unable to achieve the MPS on an initial posttest and voluntarily decided to no longer perform screening colonoscopies. Therefore, 24 of the 30 eligible participants (80.0%) completed training from pretest to mastery posttest. Twenty of 24 endoscopists (83.3%) completed 6- or 12-month follow-up: 2 left the institution and 2 dropped out of the study and voluntarily decided to no longer perform screening colonoscopies before follow-up testing. Figure 1 shows the study flow diagram.

Figure 1.

Study flow diagram. MPS, minimum passing standard; SBML, simulation-based mastery learning.

Study participants’ baseline characteristics are given in Table 1. They had a median (IQR) of 13 (2.5–26.0) years in practice. Participants performed a median (IQR) of 505.5 (183.3–934.3) colonoscopies and 232.0 (67.5–475.3) polypectomies in the year before their pretest. Sixteen of 24 (66.7%) participated in previous simulation training. Interrater reliability between checklist graders was substantial (κ coefficients ranged from 0.68 to 0.76).

Table 1.

Demographic, Prior Simulation Training, and Clinical Characteristics of 24 Participants in the Polypectomy Simulation-based Mastery Learning Training, May 2021 to August 2022

Characteristic	Value
Age, mean (SD), years	46.1 (11.5)
Sex
Female	10 (41.6)
Male	14 (58.3)
Time in practice, years
0–2	6 (25.0)
3–10	4 (16.7)
11–30	10 (41.6)
≥31	4 (16.7)
Prior simulation training at NUFSM
No	8 (33.3)
Yes	16 (66.7)
No. of colonoscopies performed 1 year before pretest
0–100	4 (16.7)
101–500	7 (29.2)
501–1000	7 (29.2)
≥1001	5 (20.8)
Missing	1 (4.2)
No. of polypectomies performed 1 year before pretest
0–50	3 (12.5)
51–100	3 (12.5)
101–500	11 (45.8)
501–1000	6 (25.0)
Missing	1 (4.2)
No. of colonoscopies performed from mastery to follow-up testing b
0–100	5 (25.0)
101–500	8 (40.0)
501–1000	5 (25.0)
≥1001	2 (10.0)
No. of polypectomies performed from mastery to follow-up testing b
0–50	4 (20.0)
51–100	1 (5.0)
101–500	5 (25.0)
501–1000	8 (40.0)
≥1001	2 (10.0)

Abbreviation: NUFSM, Northwestern University Feinberg School of Medicine.

^aData are presented as number (percentage) of participants unless otherwise indicated.

^bOf 20 participants who completed follow-up.

Individual participants’ polypectomy performance (mean of 5 polyps) on the pretest, posttest, and follow-up tests is shown in Figure 2. Median (IQR) scores significantly improved from pretest to initial posttest score (89.4% [77.1%−96.7%] to 97.5% [89.1%−99.7%] items correct; P = .02) and pretest to mastery posttest (89.4% [77.1%−96.7%] to 98.8% [100.0%−100.0%]; P < .001). There were no differences between median (IQR) mastery posttest scores and combined 6- and 12-month (n = 20) follow-up tests (98.8% [100.0%−100.0%]) as well as each of the 6- (n = 10) and 12-month (n = 10) follow-up tests (100.0% [97.2%−100.0%] and 100.0% [98.6%−100.0%], respectively).

Figure 2.

Individual skills checklist performance on the pretest, mastery posttest, and 6- and 12-month follow-up tests. Each circle or star represents an individual attending physician’s mean score on removing 5 polyps. The minimum passing standard (MPS) was set at 93% of checklists items correct on each of 5 polyps during each testing session.

Seven of 24 participants (29.2%) were unable to reach mastery at initial posttest. One dropped out of the study as described above. Of the 6 remaining participants, 3 met mastery with 1 extra deliberate practice session and 2 met mastery after 2 sessions. None of the 7 participants failed to reach mastery solely because of the global performance step of the checklist (item 17). Each extra deliberate practice session lasted 30 to 60 minutes, depending on the participant’s specific learning needs. The pretest passing rate was 4 of 24 (16.7%) compared with the initial posttest passing rate of 17 of 24 (70.8%, P < .001) and the mastery posttest passing rate of 24 of 24 (100.0%, P < .001). There were no significant differences in passing rates from mastery posttest to combined 6- and 12-month posttest (18 of 20 participants [90.0%] passed) and 6-month (8 of 10 participants [80.0%] passed) and 12-month (10 of 10 [100%] passed) follow-up tests.

Years in practice and number of colonoscopies and polypectomies performed did not significantly correlate with pretest and follow-up test performance (6- and 12-month combined). There was a difference in median (IQR) pretest scores between those who completed prior simulation and those who did not, which did not meet statistical significance (90.7% [84.1%−96.7%] vs 77.8% [54.4%−94.2%], respectively; P = .05). Of the 16 who participated in prior simulation training, only 3 (18.8%) passed the pretest compared with 1 of 8 (12.5%) who did not participate in prior simulation (P > .99).

Discussion

In this study, we found 3 important factors that contribute to the medical education literature. First, highly experienced faculty physicians who perform screening colonoscopy demonstrated wide variability in their polypectomy skills at baseline, and only 4 met or exceeded the MPS. This finding is also similar to prior studies showing variable performance of attending physicians’ polypectomy skills.^2,14 This finding is concerning because, in addition to performing procedures independently, these practicing physicians also supervise, teach, and assess trainees’ procedural skills using the Accreditation Committee for Graduate Medical Education (ACGME) Milestones.^46,47 Second, we demonstrated that attending physicians were able to complete training and meet mastery standards despite busy clinical schedules. We know of only 2 prior studies that used SBML to improve attending physicians’ skills in ultrasound-guided intravenous catheter insertion³² and code status discussions.³⁵ Third, faculty participating in our SBML intervention demonstrated a high level of skill that was robust to decay at 6- and 12-month follow-up. SBML has previously been shown to result in skills retention among medical students, residents, and fellows in peripheral intravenous catheter insertion,⁴⁸ lumbar puncture,²⁹ and central venous catheter³⁰ and temporary hemodialysis catheter insertion skills.³¹ However, to our knowledge, this is the first study to show that SBML improves skill retention among experienced faculty physicians. Faculty development using SBML is essential to ensure that best practice procedural skills are developed and retained in supervising faculty who assess trainees.

The number of procedures performed by physicians is often used in hospital privileging and credentialing. The ACGME requires colorectal surgery trainees to perform 140 colonoscopies, including 30 interventional procedures.⁴⁹ However, for gastroenterology trainees, the ACGME and American Board of Internal Medicine require competency in colonoscopy and polypectomy as determined by the fellowship program director. The American Society for Gastrointestinal Endoscopy recommends 275 colonoscopies during gastroenterology fellowship yet provides no requirement regarding number of polypectomies.⁵⁰ Most participants in our study had many years of clinical experience and performed more than 275 colonoscopies in the year before their pretest. However, their baseline (pretest) performance was poor overall because only 16.7% were able to achieve mastery scores. This finding is not surprising because other studies have also demonstrated that experienced practicing physicians have variable skills performance in procedures such as central venous catheter insertion,³³ temporary hemodialysis catheter insertion,³⁴ fundamentals of laparoscopic surgery,³⁸ anesthesia maintenance of certification,³⁷ and laparoscopic bariatric surgery.⁵¹ Multiple other studies also confirm that clinical experience is not a proxy for skill or patient care quality.^{25,45,52–56}

Our results indicate that it is not evidence based to rely on the number of procedures performed to ensure patients undergoing CRC screening undergo high-quality polypectomy procedures. SBML provides a potential solution because this method requires all learners to meet or exceed a rigorous MPS before completion of training. This intervention is more powerful than prior studies using simulation alone (without mastery),^16,48,57,58 including our prior polypectomy training.¹⁵ SBML has also been linked to improved downstream patient care in multiple other clinical skills.^19–28 For these reasons, we have incorporated SBML for attending physician and advanced practice provider training in procedural skills, such as central venous catheter insertion, thoracentesis, paracentesis, and lumbar puncture,⁵⁹ and recommend that SBML is used to document proficiency for procedures in a simulated setting whenever possible.

Our study had several limitations. First, our study was performed at one institution in a small population of endoscopists, possibly limiting generalizability. However, our results are consistent with other studies demonstrating variable skills among endoscopists^2–8 and those showing that SBML results in skills that are largely retained over time.^29–31 Second, reviewers were not blinded to whether the learners were taking a pretest, posttest, or follow-up test. It is possible that this introduced bias because not meeting the MPS initially required further training and testing sessions. However, performance at pretest, posttest, and follow-up tests was similar to prior SBML studies in which raters were blinded to testing occasion.^{23,25,44,45,55,60} Third, we did not assess the reasons why 3 attending physicians decided to voluntarily stop performing screening colonoscopy procedures at Northwestern Memorial Hospital during the study. It is possible that this was related to self-perceptions of their skills but could also have been due to other reasons and is an area for additional study. Of note, 2 of these attending physicians continue to perform upper endoscopy, and 1 no longer performs any endoscopies. Fourth, we may not have been fully powered to detect the differences between the pretest scores of those who participated in the prior simulation intervention (non-SBML) and those who did not. The median baseline (pretest) scores of those who participated in the prior simulation were higher (statistical trend) than the scores of those who did not. However, more importantly, the passing rates on the pretest were not different between the 2 groups. Fifth, we did not measure clinical outcomes resulting from our SBML intervention. Future studies will focus on clinical outcomes resulting from training practicing clinicians using our SBML intervention and correlations with adenoma (polyp) detection rates, histologic outcomes of complete resection, and referral of polyps to advanced endoscopists. We will also evaluate whether improvement on the m-DOPyS checklist after SBML is associated with improved polypectomy skills in actual patient procedures.

Conclusions

Our study found that experienced endoscopists who frequently perform screening colonoscopy have highly variable polypectomy skills at baseline. However, after an SBML intervention, all participants achieved a high level of competency that was largely retained at 6- and 12-month follow-up. The SBML intervention produced superior learning outcomes compared with traditional simulation.¹⁵ Further research is ongoing to determine whether the polypectomy SBML intervention improves patient care.