Individualized feedback on colonoscopy skills improves group colonoscopy quality in providers with lower adenoma detection rates

Abstract

Background and study aims  Colonoscopy inspection quality (CIQ) assesses skills (fold examination, cleaning, and luminal distension) during inspection for polyps and correlates with adenoma detection rate (ADR) and serrated detection rate (SDR). We aimed to determine whether providing individualized CIQ feedback with instructional videos improves quality metrics performance.

Methods  We prospectively studied 16 colonoscopists who already received semiannual benchmarked reports of quality metrics (ADR, SDR, and withdrawal time [WT]). We randomly selected seven colonoscopies/colonoscopist for evaluation. Six gastroenterologists graded CIQ using an established scale. We created instructional videos demonstrating optimal and poor inspection techniques. Colonoscopists received the instructional videos and benchmarked CIQ performance. We compared ADR, SDR, and WT in the 12 months preceding (“baseline”) and following CIQ feedback. Colonoscopists were stratified by baseline ADR into lower (≤ 34 %) and higher-performing (> 34 %) groups.

Results  Baseline ADR was 38.5 % (range 26.8 %–53.8 %) and SDR was 11.2 % (2.8 %–24.3 %). The proportion of colonoscopies performed by lower-performing colonoscopists was unchanged from baseline to post-CIQ feedback. All colonoscopists reviewed their CIQ report cards. Post-feedback, ADR (40.1 % vs 38.5 %, P  = 0.1) and SDR (12.2 % vs. 11.2 %, P  = 0.1) did not significantly improve; WT significantly increased (11.4 vs 12.4 min, P  < 0.01). Among the eight lower-performing colonoscopists, group ADR (31.1 % vs 34.3 %, P  = 0.02) and SDR (7.2 % vs 9.1 %, P  = 0.02) significantly increased post-feedback. In higher-performing colonoscopists, ADR and SDR did not change.

Conclusions  CIQ feedback modestly improves ADR and SDR among colonoscopists with lower baseline ADR but has no effect on higher-performing colonoscopists. Individualized feedback on colonoscopy skills could be used to improve polyp detection by lower-performing colonoscopists.

Introduction

While colonoscopy reduces the risk of developing colorectal cancer (CRC), the magnitude of this risk reduction depends upon the quality of the colonoscopy performed ^{1 2} . Colonoscopy quality is typically measured via the adenoma detection rate (ADR) and has been shown to vary widely between colonoscopists ^{3 4} . Recent work has shown that improvements in colonoscopist ADR reduce the risk of interval CRC ⁵ . Thus, improving colonoscopist ADR is a focus of colonoscopy quality improvement efforts. Several interventions have been shown to improve colonoscopist ADR, including: quality metrics report cards (i. e., providing colonoscopists with their ADRs) ^{3 6} , education ⁷ , training institutional quality champions ⁸ , and mandating minimum standards of practice ³ . However, no single intervention has been shown to fully eliminate the variability in colonoscopy quality.

Videotaping procedures is currently the most feasible way to obtain comprehensive data on technical skill. In the surgical field, assessing technical skill using video recordings has been utilized to identify variations in surgeon quality ⁹ and to provide individualized feedback to drive quality improvement efforts ¹⁰ . In gastrointestinal endoscopy, we and others have shown that assessments of technical skill via a structured review of videotaped colonoscopy procedures highly correlates with existing metrics of colonoscopy quality, including ADR and serrated polyp detection rates (SDR) ^{11 12 13} . However, it is unclear whether providing colonoscopists with structured feedback on technical skills is an effective method to improve colonoscopy quality.

The primary aim of this study was to determine whether providing colonoscopists with individualized technical skills report cards with instructional videos improves quality metrics performance. We hypothesized that providing a colonoscopy inspection quality (“CIQ”) report card to colonoscopists who were already receiving semiannual quality metrics report cards would significantly improve colonoscopy quality.

Methods

Setting

We conducted a prospective quality improvement study to assess the effect of a CIQ report card on high-volume attending screening colonoscopists at a single urban academic medical center. The Northwestern University Institutional Review Board approved the study (IRB #: STU00203769, approval date September 8, 2016). Colonoscopists included in the study provided written informed consent.

Study design

We recruited colonoscopists who had performed 100 or more screening colonoscopies in the year preceding study onset. Prior to this study, all colonoscopists received semiannual reports of benchmarked screening colonoscopy quality metrics performance: ADR, SDR, and withdrawal time (WT). All colonoscopists at the institution were encouraged by the department to participate, so as not to bias the sample by only including self-motivated colonoscopists who might be more likely to improve.

One study investigator (AD) recorded > 28 consecutive de-identified screening or surveillance colonoscopies per colonoscopist (October 3, 2016-November 11, 2016) and randomly selected seven colonoscopies per colonoscopist for CIQ evaluation using a random number generator. CIQ was graded using a scale developed by Rex ¹⁰ and adapted by Lee et al ¹¹ by six U.S. gastroenterologists (RY, MB, AG, CK, TK, RK) with previous experience in colonoscopy quality. The methods for video recording, video selection, and CIQ evaluation have been previously described ¹¹ .

We used the CIQ scores and colonoscopy recordings to create personalized CIQ report cards for each colonoscopist and to create instructional videos demonstrating optimal/poor colon inspection technique as previously described ¹¹ . We then compared ADR, SDR, and WT in the 12 months preceding (October 1, 2016–September 30, 2017) and following report card delivery (November 1, 2017–October 31, 2018).

Personalized CIQ report card and instructional videos

Personalized report cards with links to instructional videos were e-mailed in PDF format to each colonoscopist. The report cards included each endoscopists’ overall CIQ score and their scores on three individual skills important to high quality colonoscopy inspection: fold examination, cleaning and luminal distension ( Fig. 1 ). We also provided mean cohort and 25th/50th/75th percentile scores for overall CIQ and for each skill, so the colonoscopists could benchmark their performance to their peers.

Fig. 1.

 Sample report card given to colonoscopists detailing colonoscopy inspection quality (CIQ) scores and performance relative to peers.

To supplement these scores, we created three instructional videos with narration, each 5 to 8 minutes in length, that demonstrated optimal (receiving high scores) and poor (receiving low scores) technique for fold examination, colon cleaning, and luminal distension ^{11 12 13} . Links to these videos were embedded within the report card document.

Ten days after the report cards were e-mailed to the colonoscopists, we sent a follow-up e-mail asking whether they had reviewed their report card and had viewed all the instructional videos. For those who did not respond, or who responded “no” to either of the above questions, a follow-up email was sent a week later asking the same questions.

Data sources and measurement

ADR, SDR, and WT were calculated using 12-month data for screening colonoscopies performed by each colonoscopist pre- (“baseline”) and post-report card delivery. Data were obtained from our institution’s Enterprise Data Warehouse, a single, integrated database of clinical and research information from all patients receiving treatment through Northwestern University healthcare affiliates.

A screening colonoscopy was defined as a colonoscopy in a patient aged 50 to 75 with an indication for detecting colorectal neoplasia. Patients with a prior history of colon adenomas/serrated polyps, or a colonoscopy performed to evaluate signs or symptoms of gastrointestinal pathology including occult blood loss, anemia, abdominal pain, or rectal bleeding were excluded. ADR was defined as the proportion of screening colonoscopies with ≥ 1 adenoma and SDR was defined as the proportion of screening colonoscopies with ≥ 1 sessile serrated polyp or traditional serrated adenoma; hyperplastic polyps were not included in the SDR. All pathologists were well trained and familiar with the histologic diagnosis of serrated lesions. Withdrawal time was defined as the time spent withdrawing the colonoscope (inspecting for polyps) in screening colonoscopies where no pathology was obtained (i. e., no polyps found, and no biopsies taken).

Study outcomes

The primary study outcomes were colonoscopist and cohort ADR, SDR and WT.

Statistical analysis

Colonoscopists were stratified into two groups: lower (ADR < 34 %) and higher (≥ 34 %) performing colonoscopists by pre-report card ADR. This cut-off was chosen based on colonoscopy quality work that suggests a protective benefit of increasing ADR to at least 34 % ² . We used chi-square analysis to assess change in ADR and SDR (as weighted pooled proportions) for the overall cohort and the pre-report card ADR groups. We compared the lower and higher performing groups directly and also examined the change within each group. Due to the sample size, we employed a Wilcoxon signed rank test to evaluate change in median individual ADR between the lower and higher performing colonoscopists. We used a paired t -test to assess change in WT. We also calculated descriptive statistics for all measures to assess general spread of data.

Results

Sixteen colonoscopists (15 gastroenterologists and 1 colorectal surgeon) met inclusion criteria (at least 100 screening colonoscopies per year over the study period) and provided informed consent. The 16 colonoscopists performed a median of 1547 total (screening, surveillance, and diagnostic) colonoscopies over the 24-month study period.

Baseline ADR, SDR, and WT

During the 12-month baseline (pre-report card) period, the 16 colonoscopists performed a median of 355 (IQR 137–430) screening colonoscopies. The cohort’s baseline mean ADR was 38.5 % (range 26.8 %–53.8 %), SDR was 11.2 % (range 2.8 %–24.3 %) and WT was 11.4 minutes (range 6.1–14.5 minutes) ( Table 1 ). Eight colonoscopists had a baseline ADR ≤ 34 % (range 26.8 %–33.6 %) and were classified as lower-performing. The remaining eight colonoscopists had a baseline ADR > 34 % (range 37.7 %–53.8 %) and were classified as higher-performing. As expected, mean baseline SDR was significantly greater among the higher-performing colonoscopists compared with the lower-performing colonoscopists (15.2 % vs. 7.2 %, P  = 0.01).

Table 1. Effect of a colonoscopy inspection quality skills report card on colonoscopy quality.

	Baseline (pre-report card)	Post-report card	P value
All colonoscopists (n = 16)
ADR	38.5 %	40.1 %	NS
SDR	11.2 %	12.2 %	NS
WT	11.4 min	12.4 min	< 0.0001
Lower-quality colonoscopists (n = 8; baseline ADR ≤ 34 %)
ADR	31.1 %	34.3 %	0.02
SDR	7.2 %	9.1 %	0.02
WT	10.1 min	11.6 min	< 0.0001
Higher-quality colonoscopists (n = 8; baseline ADR > 34 %)
ADR	46 %	45.9 %	NS
SDR	15.2 %	15.3 %	NS
WT	12.6 min	13.5 min	0.04

ADR, adenoma detection rate; SDR, serrated detection rate; WT, withdrawal time; NS, not significant.

Post-report card ADR, SDR, and WT

The colonoscopists performed a median of 309 (IQR 215–462) screening colonoscopies in the 12-months post-report card. All colonoscopists reviewed their CIQ report cards and 69 % watched the videos (all lower-performing and three out of the eight higher-performing colonoscopists). The cohort’s post-report card mean ADR was 40.1 % (range 20 %–55.3 %), SDR was 12.2 % (range 7.5 %–20.7 %) and WT was 12.4 minutes (range 8.3–20.5 minutes) ( Table 1 ).

Baseline vs. post-report card

The proportion of colonoscopies performed by lower-performing colonoscopists was unchanged from baseline to post-report card (46 %). The colonoscopy skills report card did not significantly improve ADR (38.5 % vs 40.1 %, P  = 0.11) or SDR (11.2 % vs 12.2 %, P  = 0.13) among the entire cohort of colonoscopists ( Table 1 and Fig. 2 ). In contrast, WT significantly increased among all colonoscopists (11.4 vs 12.4 minutes, P  < 0.01).

Fig. 2.

 ADR, SDR, and WT were measured in all colonoscopists before and after delivery of colonoscopy inspection quality (CIQ) feedback. WT increased in all colonoscopists after post-CIQ feedback. ADR and SDR both significantly improved in the lower-performing but not higher-performing colonoscopists.

Among the eight lower-performing colonoscopists, group ADR significantly improved by 3.2 % (31.1 % vs 34.3 %, P  = 0.02) from baseline to post-report card. Similarly, SDR significantly improved in the lower-performing cohort post-report card by 1.9 % (7.2 % vs 9.1 %, P  = 0.02). In contrast, among the higher-performing colonoscopists, both ADR (46 % vs 45.9 %, P  = NS) and SDR (15.2 % vs 15.3 %, P  = NS) did not change. However, WT significantly increased post-report card distribution among both the lower-performing (10.1 vs 11.6 minutes, P  < 0.01) and higher-performing colonoscopists (12.6 vs 13.5 minutes, P  < 0.05).

Discussion

In this prospective quality improvement study of 16 colonoscopists, all of whom receive semi-annual colonoscopy quality metrics report cards, we found that providing individualized technical skills report cards with instructional videos did not significantly improve overall colonoscopy quality (as measured by ADR and SDR). However, this intervention did effectively improve the quality of colonoscopy among colonoscopists with a lower baseline ADR. Furthermore, we found that WT significantly increased post-report card for both lower-performing and higher-performing colonoscopists, which we hypothesize could be related to an attempt to correct the deficiencies noted in the report cards.

Given the wide variation in colonoscopy quality and the resultant impact on interval CRC rates, novel interventions to improve ADR (and SDR) are needed. While several studied interventions effectively improve ADR, no single intervention improves the ADR of all colonoscopists. Most prior interventions have centered around audit of quality metrics and feedback ^{3 6} , but there are limitations to this type of intervention. While it is important for a colonoscopist to be aware of his/her ADR, simply knowing one’s performance does not provide sufficient granular feedback to guide improvement. In other words, while being aware of one’s numerical colonoscopy quality metrics is important, information on specific techniques in need of improvement could be of great use, especially to lower-performing colonoscopists. For this reason, we assessed the impact of providing a colonoscopy skills report card – which provided performance feedback on cleaning, distention, and examination behind folds, along with video-based didactics demonstrating optimal and poor performance in each of the skills – upon colonoscopy quality metrics. We found that providing this technical skills report card and video didactics significantly improved colonoscopy quality (as measured by ADR and SDR) in lower-performing colonoscopists. However, this intervention had no impact on colonoscopists with higher baseline (> 34 %) ADR. We also found that WT increased in both groups of colonoscopists.

The use of video-based feedback regarding technical skill has gained prominence, predominantly in the surgical literature, since a landmark study demonstrated a strong association between video-based assessment of surgical skill from a single video and clinical outcomes, including death ⁹ . Subsequent studies have examined the use of video-based feedback derived from multiple sources, including experts, peers, and even novice reviewers. Video-based feedback has been shown to effectively improve surgical and cognitive skills for both independent practitioners and trainees ^{14 15} . However, there are scant data within the gastrointestinal endoscopy community regarding the effectiveness of “video-based coaching” upon subsequent colonoscopy quality. We previously showed that providing video-based coaching regarding technical skill improved the quality of colonoscopy polypectomy ¹⁶ . This is the first study to assess the impact of video-based coaching upon colonoscopist ADR, SDR, and WT.

There are important limitations to consider with this study. At baseline, all included colonoscopists had an ADR which exceeded the current “minimum” ADR threshold of 25 % recommended by joint society guidelines ^{17 18} . However, there are increasing data suggesting that ADR improvements up to 35 % may be beneficial in reducing post-colonoscopy CRC ² . Regardless, it is unclear whether video-based coaching would be more or less effective for those colonoscopists with an ADR < 25 % who are in greatest need of remediation; however, based on this data, we would hypothesize that this intervention would be more effective in lower quality colonoscopists. Second, similar to most quality improvement literature, we utilized a pre-post study design wherein all colonoscopists simultaneously received our intervention after a control period. Thus, it is possible that a portion of the improvement identified in the lower performing colonoscopists is related to ongoing improvements in colonoscopy quality independent of our intervention. However, notably, the colonoscopists did not have significantly different ADRs or SDRs in the 3 years prior to study onset, suggesting this is less likely. Furthermore, due to the study design, we did not conduct an a priori sample size calculation as all high-volume colonoscopists were invited to participate. Third, we paired the colonoscopy skills report card with video didactics consisting of examples of high and low-quality colonoscopy inspection techniques. Therefore, the independent impact of the report card and video didactics cannot be assessed. We were unable to perform a subanalysis comparing the three higher-performing colonoscopists who watched the videos and the five who did, given the small sample size. Finally, expert assessment of colonoscopy skills is an onerous and time-consuming process. We suggest that the impact of peer feedback, which is an intervention which can be scaled more broadly, be similarly assessed. Alternatively, we recently showed that novice raters performed comparably to expert raters and could be utilized to identify lower performing colonoscopists who may be in need of remediation ¹⁹ .

Conclusions

In summary, we have shown that provision of a colonoscopy skills report card detailing performance on core colonoscopy inspection skills, in conjunction with video didactics, significantly improves both ADR and SDR in colonoscopists with lower ADRs. However, this intervention does not significantly improve ADR or SDR in colonoscopists with baseline ADR > 34 %. Because improving ADR is essential to high-quality colonoscopy, we recommend that this intervention be implemented and studied in a broader cohort of colonoscopists with lower baseline quality.