The impact of the introduction of formalised polypectomy assessment on training in the UK

Abstract

Objective

The aim was to describe the impact on polypectomy experience by the mandatory introduction of the Directly Observed Polypectomy Skills tool (DOPyS) and electronic portfolio as part of the formal colonoscopy certification process.

Design

Applications for colonoscopy certification in the UK in the year prior to the introduction of DOPyS were analysed retrospectively and compared with data collected prospectively for those in the following year.

Setting

UK National Health Service.

Patients

None.

Interventions

None.

Main outcome measures

The outcomes studied included whether evidence of exposure to polypectomy, endoscopic mucosal resection (EMR) and colonoscopy changed over the 2-year period. The nature of the polyps removed by trainees was also studied.

Results

Thirty two per cent of candidates in the first year had evidence of any observed polypectomy with 7% of candidates referring to training in EMR. The median number of formative colonoscopy assessments was 3 (range 0–16). All of these candidates in the second year had evidence of polypectomy assessment, with a median number of DOPyS of 7 (range 3–27). Eighty nine per cent of applicants had evidence of assessed EMR. The median number of formative colonoscopy assessments in this cohort was 32 (range 9–199). There was a significant increase in the number of logged polypectomy assessments (p<0.001), experience of EMR (p<0.001) and formative colonoscopy assessments (p<0.001). There was no significant difference in the total number of colonoscopy procedures performed.

Conclusions

Structured polypectomy assessment improves trainees’ documented exposure to therapeutic endoscopy as well as providing formal evidence of skills acquisition. As polypectomy plays an increasing role globally in colorectal cancer prevention, the DOPyS provides an effective means of assessing and certifying polypectomy.

Background

Polypectomy is regarded as the most hazardous component of colonoscopy,¹ accounting for the majority of procedure-associated morbidity, and yet is a necessary skill for all colonoscopists. Due to the increase in the worldwide uptake of colonoscopy as both a diagnostic tool and the favoured method for colorectal cancer screening, the number of polypectomies performed has continued to increase year-on-year.

Data from national audits^{2 3} and from the English Bowel Cancer Screening Programme¹ have shown that most cases of bleeding and perforation are related to polypectomy. Training in polypectomy has, to date, been variable and poorly structured.⁴ There has been some evidence suggesting poor exposure to polypectomy during training. In one study, only 60% of trainees had ever been assessed in polypectomy.⁴

A novel assessment tool, the Directly Observed Polypectomy Skills (DOPyS), was introduced nationally in the UK in October 2011 with the intention of both improving training and facilitating documentation of competency. This tool contains 34 separate criteria and permits the deconstruction and assessment of polypectomy for both sessile and pedunculated lesions.⁵ At the end of the tool, a global rating is given from 1 (multiple errors) to 4 (highly skilled) to signify the trainer's assessment of the trainee's competency in polypectomy. The methods used to devise and validate this tool have been described elsewhere.⁶

There have always been a set of minimum requirements before an application for certification of competency was permitted. Until October 2011, formal certification of colonoscopy competence in the UK was acquired by trainees collecting paper-based portfolios of their colonoscopy practice. Trainees had to show evidence of competency in colonoscopy by using the Directly Observed Procedural Skills (DOPS) tool for a summative assessment. There was a requirement for a minimum number of 200 completed colonoscopies. Polypectomy assessment itself was not compulsory, although trainers could reflect appropriate generic use of therapy during colonoscopy by marking a single criterion on the generic DOPS tool.

After October 2011, there was a transition to a paperless electronic certification system.⁷ Trainees were required to complete the same summative DOPS assessments, in addition to keeping a log of procedures. However, it was also compulsory for those wishing to be provisionally certified to have completed a minimum of four snare polypectomy assessments using the DOPyS to resect polyps up to 10 mm as well as 10 formative colonoscopy DOPS during the course of their training. Full certification for colonoscopy and polypectomy was awarded to those who had attained provisional certification and then had evidence of at least four satisfactory DOPyS, with a minimum of two assessments tackling both sessile and pedunculated lesions.

Objectives

The primary aim was to describe the impact on polypectomy experience by the mandatory introduction of the DOPyS as part of the formal colonoscopy certification process. The secondary aim was to assess the influence of the release of the electronic certification system on trainees' exposure to colonoscopy.

Setting

UK National Health Service.

Methods

All trainee endoscopists applying to Joint Advisory Group on GI Endoscopy (JAG) for certification of colonoscopy competence for the 2 years from October 2010 to October 2012 were included. The first cohort comprised those using paper portfolios from October 2010 to September 2011; the second cohort included those using the mandatory electronic portfolio from October 2011 to September 2012.

Applications for certification in the year prior to the introduction of DOPyS were analysed retrospectively and compared with data collected prospectively for those in the following year.

Paper portfolios

Endoscopists were not required to submit procedural details for each case. Aggregate data were collected from the logbook on the total lifetime number of colonoscopies performed, the number of assessments submitted for both colonoscopy and polypectomy and whether applicants had any evidence of performing supervised or unsupervised polypectomy before certification of competence in colonoscopy.

If trainees using the paper-based certification system had any DOPS assessment marked demonstrating appropriate use of therapy, this was noted as experience of polypectomy. If trainees had any documented experience of performing endoscopic mucosal resection (EMR), this was also recorded.

Electronic portfolios

The size, location and type of lesions resected by trainees while being assessed using the DOPyS were logged, as well as the overall competency of trainees to perform polypectomy.

Analysis

Data were analysed using SPSS V.20. To assess the differences between the 2 years of data collection, the Mann–Whitney U test was used for non-parametric data. The independent samples t test was used to compare normally distributed data such as polyp size.

Results

Participants

There were 175 applicants for certification in the first year compared with 150 applications in the year after DOPyS was introduced (p=0.99).

Experience of colonoscopy

The median number of colonoscopy procedures prior to seeking certification per candidate was 287 in the first year, compared with 206 in the following year. There was no significant difference in the total number of colonoscopy procedures undertaken (p=0.07). These procedural volumes were both above the requirement to complete at least 200 procedures before applying for certification.

In the first cohort, the median number of formative colonoscopy assessments provided was 3 (range 0–16). This rose to 32 (range 9–199) in the year after the introduction of the electronic portfolio (p<0.001), comprising the 10 mandatory DOPS required by the certification criteria and 22 additional DOPS submitted voluntarily (figure 1).

Figure 1.

Formative colonoscopy Directly Observed Procedural Skill (DOPS), submitted with applications for certification.

Experience of polypectomy

Thirty two per cent of candidates using paper portfolios had evidence of any observed polypectomy, with 7% of candidates referring to training in EMR.

In the year following the introduction of DOPyS, the 150 trainees using the electronic certification process were assessed resecting a total of 1283 polyps. All candidates had evidence of polypectomy assessment throughout that year, with a median number of DOPyS of 7 (range 3–27). This figure included a median of 2 DOPyS for stalked lesions and 5 DOPyS for sessile polyps. Eighty nine per cent of applicants had evidence of assessed EMR (figure 2).

Figure 2.

Trainees' polypectomy experience. DOPyS, Directly Observed Polypectomy Skill; EMR, endoscopic mucosal resection.

As expected, there was a significant increase in the number of recorded polypectomy assessments (p<0.001), given these were mandatory for the second cohort. However, both logged experience of EMR and formative colonoscopy assessments (p<0.001), which were not obligatory, also increased (p<0.001).

Polyp characteristics

The majority of polyps resected by trainees (68.5%) were located in the left colon, but a significant minority were found more proximally (figure 3).

Figure 3.

Location of polyps resected by trainees.

The polyps removed by trainees had a mean size of <10 mm in all colonic segments. However, there was a significant range of polyp sizes seen, with lesions up to 60 mm documented (figure 4). Polyps in the left colon were on average slightly larger (9.1 mm compared with 7.2 mm, p<0.001). The sessile lesions tackled comprised the majority (61%) of polyps, but these were significantly smaller than the pedunculated polyps removed (6.6 mm compared with 10.9 mm, p<0.001).

Figure 4.

Mean and maximum polyp sizes removed by trainees with assessment.

The majority of DOPyS assessments (93%) were scored either 3 or 4, signifying competence at polypectomy (figure 5). There was no difference in polyp size between the DOPyS scored as 1 or 2 compared with those scored 3 or 4 (8.0 vs 8.3 mm, p=0.299).

Figure 5.

Overall polypectomy competency scores using the Directly Observed Polypectomy Skill (DOPyS).

Discussion

These data are the first to describe the implementation of a nationwide certification system on polypectomy training. The introduction of structured polypectomy assessment was both feasible and acceptable, with similar numbers of trainees applying for certification after the introduction of the DOPyS. In addition, these data show clear evidence of an increase in trainees' documented exposure to therapeutic lower gastrointestinal endoscopy as well as providing formal evidence of skills acquisition.

The DOPyS was designed to aid both training and permit certification of competency. However, it appears that it is currently being used predominantly with the latter objective: this may well be related to its inclusion as a mandatory part of the colonoscopy competency certification process. Only a very small proportion (0.4%) of assessments were scored 1 (accepted standards not yet met; frequent errors uncorrected). These low-scoring DOPyS are likely to represent formative assessments for trainees in the early stages of their learning curve.

The reasons for this are likely to be multifactorial. Given the hazards inherent to the technique, it is likely that a trainee performing a supervised polypectomy with frequent errors would either receive instruction from the trainer or pass the endoscope to the more experienced colonoscopist to complete the procedure. In addition, these evaluations are largely instigated at the behest of the trainee, and it is likely that there is an element of selection bias with trainees not requesting trainers' formal documented assessment of procedures that have not been performed well. It is unlikely that the assessments are representative of the full learning curve for polypectomy, given they are so heavily skewed towards exhibiting competency. These results are not unexpected, in that trainees were expected to use DOPyS to demonstrate their polypectomy skills if they wished to apply for certification.

There is a strong argument for the use of the DOPyS tool at an earlier stage of training to standardise the instruction trainees receive when beginning to learn how to perform a polypectomy for the first time. However, it would be difficult to make this a mandatory requirement of certification, and a more pragmatic approach may be to encourage trainers to use this tool to a greater extent than is the case at present.

It is encouraging that all trainees are receiving dedicated separate assessments in both colonoscopy and polypectomy. Just over 10 years ago, the UK national audit showed that only 17.0% of colonoscopists had received supervised training for their first 100 colonoscopies.² This study shows the significant progress made in training in a relatively short space of time.

It is also reassuring that trainees are being taught and assessed in polypectomy on a variety of lesions, including proximal polyps, which may be more risky and technically challenging.^{1 8} Most assessed lesions were <10 mm in size in keeping with the parameters of the mandatory DOPyS assessment and mirroring clinical practice. In the faecal occult blood test (FOBT)-positive Bowel Cancer Screening Programme, 86% of polyps were found to be <10 mm, and it seems that training experience reflects these figures.⁹

The vast majority of polyps found during universal screening with flexible sigmoidoscopy at age 55 are also within the same size parameters, emphasising the importance of all endoscopists possessing the skills required to tackle these common lesions. In this screening setting, complete polypectomy and retrieval are crucial. Although larger sized lesions are associated with higher rates of advanced pathology such as high-grade dysplasia or villous architecture, these features are not confined to bigger polyps and occur in lesions <5 mm.¹⁰

Before the introduction of DOPyS, only a very small minority (7%) of trainees had evidence of EMR skills. The methodology used to assess this figure was generous with even a solitary reference to EMR being counted. These data were obtained from logbooks, and so the quality and amount of training could not be easily gleaned.

However, since DOPyS has found its way into routine clinical practice with trainees performing more assessments than strictly mandated, this would suggest that the majority of trainees seem to be receiving both training and assessment in this invaluable technique, a skill which many would argue is obligatory for any independent colonoscopy practitioner.

Interestingly, with the introduction of the electronic portfolio, most trainees considerably surpassed the minimum requirements for submission of formative DOPS and DOPyS. This suggests that trainees have become familiar with this method of assessment and integrated it into their practice, using it as a training aid as well as for certification of competency. As polypectomy plays an increasing role globally in colorectal cancer prevention, the DOPyS provides an effective means of assessing and certifying polypectomy skills, including lesion assessment as well as the technical proficiencies required for a safe, complete resection, in order to minimise the well-recognised risks associated with this technique. It also has the potential to be a useful tool to help verify endoscopic competence as part of mandatory ongoing clinical revalidation for those practising independently.

The role of assessment in changing academic behaviour is well recognised.¹¹ The closest parallel with practical medical skills assessment comes from the DOPS tool, which is used widely by junior doctors as part of their electronic portfolios. A study looking at doctors who had just qualified from medical school found that most (70%) felt that DOPS helped improve their clinical skills.¹² However, a systematic review of several different types of workplace-based assessments found no available high-quality evidence from this study or any others that DOPS lead to objective performance improvement.¹³

The LAPCO programme, a UK national training programme for laparoscopic colorectal surgery devised to encourage the widespread uptake of laparoscopic surgery, has been extensively evaluated¹⁴ and found to bear some similarity to polypectomy assessment. Similar to the DOPyS, task deconstruction, hands-on training and independent expert rating of cases with feedback are hallmarks of this programme.¹⁵ Analysis of outcome data has shown significant improvement in participating surgeons, with expert levels of proficiency achieved after iterative feedback.¹⁴

The best studied tool with respect to performance improvement is the multisource feedback, in which both senior and junior colleagues anonymously provide ratings of an individual's strengths and weaknesses. The ability of this to change behaviour is reliant on individual characteristics. Studies have shown that some junior doctors¹⁶ and most surgeons¹⁷ were not amenable to change whereas general practitioners were more likely to be receptive to suggestions.¹⁸ Those who consider change to be necessary, react positively, believe that changes are feasible and take suitable actions as a result of this are most likely to benefit from feedback.¹³

It is likely that the DOPyS tool would show similar characteristics, but this has not been evaluated to date. Specifically, the DOPyS tool focuses predominantly on technical skills involved in polypectomy. These are easier to influence than non-technical skills which require behavioural change and insight. A meta-analysis of non-technical skills training revealed that outcome measures used to date in the literature are seldom clinically applicable.¹⁹ However, the data described here do show some correlation with non-technical skills; studies published to date in other fields may not be applicable to polypectomy.

Several questions remain unanswered from this work. Whereas there is growing consensus in the literature as to the number of colonoscopies needed to attain the technical ability to reach the caecum reliably,^20–22 these data are lacking for polypectomy skills. Given that endoscopists who perform low volumes of colonoscopies have been associated with higher complication rates,^{23 24} training in polypectomy is likely to also be a higher risk activity, and the quality of training and assessment is likely to vary considerably between specialties, centres and individual trainers.^{25 26} The relative importance of non-technical skills when undertaking polypectomy at an introductory level is also not clear, but they are likely to be important. Assessment of these other skills is expected to be more difficult than simple technical ability, but may prove to be beneficial in terms of improving the rate of skills acquisition.

Key messages.

What is already known on this topic?

• Training in polypectomy has traditionally been poorly structured.

• Setting standards of competency can drive learning and training.

• The Directly Observed Polypectomy Skills (DOPyS) tool has been validated in the assessment of polypectomy skills.

What this study adds?

• An electronic portfolio helps trainees document proficiency in polypectomy.

• The DOPyS provides an effective means of certifying polypectomy skills nationally.

How might it impact on clinical practice in the foreseeable future?

• Wider uptake of the DOPyS could be used to improve training to polypectomy novices as well as providing a means for independently practising individuals to provide evidence of competency for revalidation.