Real-Time Characterization of Diminutive Colorectal Polyp Histology Using Narrow-Band Imaging: Implications for the Resect and Discard Strategy

Abstract

BACKGROUND & AIMS

Narrow-band imaging (NBI) allows real-time histologic classification of colorectal polyps. We investigated whether endoscopists without prior training in NBI can achieve the following thresholds recommended by the American Society for Gastrointestinal Endoscopy: for diminutive colorectal polyps characterized with high confidence, a ≥90% negative predictive value for adenomas in the rectosigmoid and a ≥90% agreement in surveillance intervals.

METHODS

Twenty-six endoscopists from 2 tertiary care centers underwent standardized training in NBI interpretation. Endoscopists made real-time predictions of diminutive colorectal polyp histology and surveillance interval predictions based on NBI. Their performance was evaluated by comparing predicted with actual findings from histologic analysis. Multilevel logistic regression was used to assess predictors of performance. Cumulative summation analysis was used to characterize learning curves.

RESULTS

The endoscopists performed 1451 colonoscopies and made 3012 diminutive polyp predictions (74.3% high confidence) using NBI. They made 898 immediate post-procedure surveillance interval predictions. An additional 505 surveillance intervals were determined with histology input. The overall negative predictive value for high-confidence characterizations in the rectosigmoid was 94.7% (95% confidence interval: 92.6%–96.8%) and the surveillance interval agreement was 91.2% (95% confidence interval: 89.7%–92.7%). Overall, 97.0% of surveillance interval predictions would have brought patients back on time or early. High-confidence characterization was the strongest predictor of accuracy (odds ratio = 3.42; 95% confidence interval: 2.72–4.29; P < .001). Performance improved over time, however, according to cumulative summation analysis, only 7 participants (26.9%) identified adenomas with sufficient sensitivity such that further auditing is not required.

CONCLUSIONS

With standardized training, gastroenterologists without prior expertise in NBI were able to meet the negative predictive value and surveillance interval thresholds set forth by the American Society for Gastrointestinal Endoscopy. The majority of disagreement in surveillance interval brought patients back early. Performance improves with time, but most endoscopists will require ongoing auditing of performance.

Colonoscopy is the dominant colorectal cancer screening modality in the United States¹ and has been associated with a reduction in colorectal cancer incidence and mortality.² Although colonoscopy is cost-effective,³ it does carry a substantial financial burden. Significant cost is associated with the removal and pathologic assessment of diminutive (≤5 mm) colorectal polyps, which are the most commonly found polyps during colonoscopy.⁴ Diminutive polyps harbor a very low risk of advanced adenomatous histology, such as high-grade dysplasia and villous features (1.1%–3.4%)^5,6 or cancer (0%–0.08%),^4,7,8 but are routinely examined by pathologists to guide surveillance interval recommendations. If diminutive polyp histology can be determined optically in real time without the expense of pathologic examination, significant cost reduction can be achieved without compromising clinical decision making or quality.

Narrow-band imaging (NBI) is an optical endoscopic imaging technology that highlights superficial mucosal vasculature, thereby accentuating and highlighting the surface pattern of polyps. These patterns can help characterize polyp histology. NBI can be applied with the touch of a button and is available as a standard feature in the current generation of Olympus endoscopes (Olympus America, Center Valley, PA).⁹ Optical histologic diagnosis of diminutive polyps has led to the proposal of a “resect and discard” strategy for diminutive polyps determined to be adenomatous and a “do not resect” strategy if characterized as hyperplastic. The American Society for Gastrointestinal Endoscopy (ASGE) commissioned a Preservation and Incorporation of Valuable Endoscopic Innovation (PIVI) statement to address the “characterize, resect, and discard” strategy.¹⁰ This statement suggested the following targets before “characterize, resect, and discard” can be adopted in clinical practice: for diminutive polyps characterized with high confidence in combination with histologic confirmation of polyps characterized with low confidence, endoscopists should achieve ≥90% agreement between surveillance intervals predicted by NBI characterization and surveillance intervals based on histology and ≥90% negative predictive value (NPV) for adenomatous polyps in the rectosigmoid. The PIVI statement indicates that an optical diagnosis made with high confidence is defined by clinical judgment to make a diagnosis with sufficient certainty such that histologic confirmation is not necessary. A high-confidence diagnosis constitutes a diagnosis in which an endoscopist is comfortable either resecting and discarding if deemed to be adenomatous, or leaving in place if deemed to be hyperplastic in the rectosigmoid. This strategy has enormous cost-saving potential. Hassan et al¹¹ estimated an annual savings of $33 million if the resect and discard strategy is applied, assuming an 84% high-confidence rate. Kessler et al¹² estimated that, based on the volume of colonoscopy performed in the United States, the annual up-front cost savings of forgoing pathologic assessment for diminutive polyps would exceed $1 billion.

Although multiple studies have shown that experts in advanced endoscopic imaging can achieve these thresholds,^13–16 it is unclear if endoscopists without prior experience with NBI can meet these ASGE benchmarks. Studies including endoscopists without prior training or research interest in NBI have shown that they have not been able to achieve the same level of performance as experts.^17–20 It is unclear if this is due to variability in NBI training because studies were not specifically designed to examine the ASGE PIVI thresholds, because of lack of diligent performance feedback, because studies did not account for histologic input for low-confidence characterizations, or whether the results seen among experts are just not generalizable to all endoscopists. In addition, although learning curves for NBI optical diagnosis have been described in the ex vivo training setting,²¹ it is unclear whether there is a learning phenomenon in real time. Earlier studies have not identified predictors (eg, experience level, demographics, performance in training) of optical diagnostic performance in real time.

The hypothesis of the current study is that with implementation of a novel training method and incorporation of structured real-time feedback, endoscopists with no prior training in NBI can achieve the thresholds set forth by the ASGE for NBI optical diagnosis of diminutive colorectal polyps.

The primary aim of this study was to determine whether academic gastroenterologists without prior training in NBI can achieve a ≥90% NPV for adenomatous polyps characterized with high confidence in the rectosigmoid and a ≥90% agreement in surveillance intervals when combining high-confidence NBI predictions and histologic confirmation of low-confidence NBI predictions after undergoing a standardized NBI training session with structured performance feedback.

Methods

Study Sites, Participants, and Equipment

Gastroenterologists from 2 tertiary academic medical centers (University of Michigan and University of Colorado) and 2 Veterans Affairs (VA) Hospitals (Ann Arbor VA and Denver VA) participated in the study (ClinicalTrials.gov ID NCT02441998). Participants had no prior formal training or experience in NBI and provided informed consent to participate. Participants with low annual colonoscopy volume (<200 procedures annually) were excluded. Each participant completed a questionnaire of demographic characteristics and self-reported level of experience.

Endoscopy suites at 2 sites (University of Michigan and Denver VA) were equipped with Evis Exera II CV-180 processors with CF-H180AL and PCF-H180AL colonoscopes. The other 2 sites (Ann Arbor VA and University of Colorado) were equipped with Evis Exera III CV-190 processors and CF-H190AL and PCF-H190AL colonoscopes (Olympus America). All rooms were equipped with high-definition monitors and displayed standardized posters showing NBI polyp pattern classification (Supplementary Appendix A).

This study was approved by the University of Michigan Quality Improvement Review Board and the Colorado Multiple Institutional Review Board.

Training Session (ex vivo Phase)

All participants viewed a 20-minute audiovisual training tool (designed by AR) describing previously established NBI criteria to distinguish adenomatous from hyperplastic polyps.^14,22,23 Participants subsequently viewed 80 short video clips of diminutive polyps (≤5 mm) displayed under high-definition white light and NBI. Each polyp video had a histologically confirmed diagnosis of adenoma or hyperplastic. After each video was displayed, participants recorded their predicted histology (adenoma or hyperplastic) and their degree of confidence (high vs low as described in the ASGE PIVI statement) in their prediction. Once responses were locked in after each video, the histologic diagnosis was revealed and feedback provided by a moderator (SW, SP) if there was lack of consensus. The training session took approximately 2 hours and all participants completed the training session in October 2013.

Real-Time Assessments (in vivo Phase)

Participants who completed the training session underwent a study orientation during which the concept of “characterize, resect, and discard” strategy and the proposed ASGE PIVI thresholds were presented. The orientation included an over-view of the study protocol and requirements. The definition of high-confidence vs low-confidence characterizations was reviewed and participants were advised to estimate polyp size using standard references (closed jumbo biopsy forceps tip 2.4 mm, snare catheter 2 mm).

After completing orientation, participants were asked to make histology predictions in real-time for diminutive polyps and predict surveillance intervals based on their NBI characterization. All colonoscopies, for any procedure indication, with at least one diminutive polyp between November 2013 and November 2014 were included. Colonoscopies were performed in usual standard of care and all detected polyps were removed in a fashion deemed appropriate by the individual endoscopist.

Polyp Predictions

Participants were asked to photo document all diminutive polyps under white light and NBI. Polyp location, size, NBI patterns seen, predicted histology, and degree of confidence (high vs low) were recorded in real time by research personnel, the endoscopy nurse, or a technician (Supplementary Appendix B). All detected polyps were removed per routine practice and sent in individual jars for pathologic assessment.

Surveillance Interval Predictions

Participants made surveillance interval predictions based on their NBI predictions immediately after completion of the procedure in order to reduce bias. If there were low-confidence characterizations, endoscopists were asked to predict an interval based on all possible histologic outcomes for the low-confidence characterizations (to replicate application of the PIVI statement wherein low-confidence diagnoses are sent for pathologic examination) (see Supplementary Appendix B).

Due to increasing complexity of possible histologic outcomes, participants did not make immediate post-procedure surveillance interval predictions for procedures in which more than 2 low-confidence characterizations were made (because the surveillance interval would be driven by histologic confirmation of low-confidence characterizations). Participants also did not make immediate post-procedure surveillance interval predictions on procedures where the surveillance interval would not be dictated by histology of diminutive polyps (if there were any polyps >5 mm, if there were >10 polyps, if there was any reason to deviate from standard polyp surveillance guidelines [patient with inflammatory bowel disease, history of colorectal cancer or familial cancer syndrome, inadequate bowel preparation, incomplete procedure, stopping screening due to advanced age or comorbidities]) or procedures in which the endoscopist was unable to retrieve all of the polyps removed.

After completion of the study, a single-study investigator systematically reviewed all of the procedures in which an immediate surveillance interval was not predicted by the participating endoscopist. The surveillance interval prediction (based on NBI predictions in combination with histology input for low-confidence and >5 mm polyps) was determined using iterative reasoning for each procedure based on current standard polyp surveillance guidelines (for 0 adenomas, 10 years; 1–2 <10-mm adenomas, 5 years; for ≥3 adenomas and adenomas >10 mm/advanced histology, 3 years).²⁴ For instance, if a procedure was initially excluded because of an 8-mm polyp (adenoma), but 2 diminutive polyps were characterized with NBI as high-confidence adenoma and high-confidence nonadenoma, the predicted surveillance interval taking into account the 8-mm adenoma would have been 5 years. This was then compared with the actual surveillance interval in the pathology letter sent to the patient (also 5 years) to determine agreement. Two separate study investigators audited a random representative sample of these procedures and independently determined the predicted surveillance interval to ensure accuracy of surveillance interval prediction.

Polyp Pathology and Actual Surveillance Intervals

All polyp pathology was reviewed by fellowship-trained gastrointestinal pathologists at participating sites who were blinded to study participation and NBI polyp prediction. A research coordinator entered the actual histology once pathology results were available and the endoscopists' actual surveillance interval recommended to the patient based on letters sent to patients and documentation in the medical record.

Performance Feedback

To promote ongoing learning in real time, endoscopy photos of study polyps under white light and NBI annotated with predicted and actual histology were returned to participants in batches of 10 procedures or every 2 weeks (whichever came first). Formal performance reports summarizing all participants' overall polyp prediction accuracy and agreement in surveillance interval predictions were distributed on a monthly basis throughout the study so that participants could compare their performance with other participants in the study (see Supplementary Appendix C). Feedback reports were generated and distributed using anonymous participant number (only known to individual participant and study coordinators).

Study Aims

Primary aim

The primary aim was to determine whether endoscopists with no prior training in NBI can achieve the ASGE PIVI thresholds of ≥90% NPV for rectosigmoid polyps and ≥90% agreement in surveillance intervals compared with the current gold standard of histology, when NBI optical diagnoses are made with high confidence.

Secondary aims

The secondary aims for the ex vivo (training) phase included evaluating overall performance (accuracy, sensitivity, specificity, predictive values) by degree of confidence; determining predictors of performance during training; and evaluating a learning effect in the setting of ongoing feedback during training. The secondary aims for the in vivo phase included evaluating overall group performance of optical diagnoses using NBI by degree of confidence and by location within the colon; evaluating individual performance on the ASGE PIVI benchmarks; determining predictors of performance; and determining real-time learning effect in the setting of ongoing, structured performance feedback.

Sample Size

Based on 26 participating endoscopists and within endoscopist correlation of .05, sample size was calculated to be able to detect an NPV ≥90%, assuming that the true NPV is 95% for rectosigmoid polyps characterized with high confidence. This required 336 total rectosigmoid nonadenomatous polyps characterized with high confidence. Assuming approximately 2 polyps per colonoscopy, 22% of all diminutive polyps located in the rectosigmoid, 70% with high confidence and 80% nonadenomas, the study required approximately 2727 polyps and 1364 colonoscopies in total.

Data Management and Statistical Analysis

Data from the ex vivo and in vivo portion of the study was collected and managed using REDCap (Research Electronic Data Capture) electronic data capture tools hosted at the University of Colorado.²⁵ REDCap is a secure, web-based application designed to support data capture for research studies.

Diagnostic performance of NBI was evaluated by comparing predicted and actual histology (histology as the gold standard). For sensitivity, specificity, and predictive values, adenoma was considered a positive result and nonadenoma was considered a negative result. Kappa statistic was used to determine inter-observer agreement in polyp video classification during the training session. Sessile serrated polyps were analyzed as “nonadenomas” as reported previously.¹⁸ Surveillance interval predictions based on NBI characterizations and actual surveillance interval recommendations documented/given to the patient were compared with determine agreement in surveillance intervals.

A multilevel logistic regression model was used, adjusting for between-participant variability, to demonstrate predictors of accuracy in both the ex vivo training session as well as the in vivo real-time phase. The dependent variable of the accuracy model was the diagnoses of polyp (correctly diagnosed adenoma vs not), and the potential predictors included institutions, sex of participant, level of experience, endoscope series (180 vs 190), clinical effort (<60% vs ≥60%), confidence level, polyp size, polyp location, and polyp block. The strength of the relationship between a predictor and accuracy was summarized using an odds ratio and its 95% confidence interval (CI) based on the parameter estimate and standard error of the predictor based on the accuracy model, and the associated P value was used to test for the statistical significance of the predictor.

Learning effect was tested for by dividing the sequence of assessment into blocks of 15 colonoscopies and including the dummy categorical indicators for the blocks. In addition to the regression model, real-time learning was demonstrated using cumulative summation (CUSUM) analysis. This method continuously assesses the performance of an individual or process against a predetermined standard in order to detect adverse trends and to allow for early intervention in the form of retraining or continued observation.²⁶ This method has been used extensively in the surgical²⁷ and anesthesia²⁸ literature to assess learning curves for medical procedures and has been applied in endoscopic procedure learning as well.^29,30

The CUSUM scores were calculated from the acceptable probability of failure P₀ and unacceptable probability of failure P₁ as follows:

$$\begin{array}{cc}\hfill & \mathrm{s}=\mathrm{Q}∕(\mathrm{P}+\mathrm{Q})\text{where P}=\ln (P_1∕P_0)\text{and}\hfill \\ \hfill & \mathrm{Q}=\ln \left[\right(1-P_0)∕(1-P_1\left)\right]\hfill \end{array}$$

An acceptable and unacceptable level of incorrect responses was set at 15% and 20%, respectively. By assigning an acceptable failure rate of P₀ = .15 and an unacceptable failure rate of P₁ = .20, s = 0.174, and 1 − s = 0.826. Failure was defined as an incorrect answer for polyp histology. The CUSUM score starts at 0 and with each consecutive reading, the amount is subtracted from the previous CUSUM score and 1 − s is added to the previous CUSUM score. The CUSUM score was plotted against the index of consecutive colonoscopies with acceptable and unacceptable limits defined by type 1 error of 0.1, type II errors of 0.1, P and Q. Performance was judged as follows: if the CUSUM plot fell below the acceptable line (h0), the performance was acceptable; if the CUSUM plot rose above the unacceptable line (h1), the performance was unacceptable; if the plot stayed between the 2 boundary lines, no conclusion could be drawn and further training/observation is recommended. Statistical analysis was performed using STATA software, version 13.1 (StataCorp LP, College Station, TX). All authors had access to the study data and reviewed and approved the final manuscript.

Results

Participant Characteristics

Of the 29 individuals invited to participate, 26 academic endoscopists (65.4% male) with varying years of endoscopy experience and percentage of clinical effort completed the ex vivo and in vivo portion of the study (Table 1). Approximately 57.7% reported performing between 201 and 500 colonoscopies per year and the remaining participants reported performing >500 colonoscopies per year.

Table 1.

Participant Characteristics (n = 26)

Characteristics	n (%)
Sex
Male	17 (65.4)
Female	9 (34.6)
Years' experience
<5	8 (30.8)
5–10	10 (38.5)
11–20	4 (15.4)
>20	4 (15.4)
Annual colonoscopy volume
201–500	15 (57.7)
>500	11 (42.3)
Clinical effort
≥60%	14 (53.9)
<60%	12 (46.2)

Ex Vivo Training Session Performance

The overall performance measures (sensitivity, specificity, predictive values, accuracy) exceeded 90% for polyps in the ex vivo training videos that were diagnosed with high confidence (Supplementary Appendix D). Participant performance improved as they progressed through the training videos, as shown in the Supplementary Appendix E. After adjusting for physician characteristics, participants were more likely to report a correct response in the last quartile of videos viewed compared with the first quartile (odds ratio [OR] = 4.28; 95% CI: 1.97–9.30; P < .001] and more likely to report a correct diagnosis if classified with high-confidence (OR = 5.20; 95% CI: 3.33–8.07; P < .001). There was no difference by institution or colonoscopy volume. However, years of experience were inversely related to correct response. Compared with endoscopists with <5 years of experience, the odds of correct response was 0.32 (P = .04), 0.26 (P = .08), and 0.24 (P = .006) in those with 5–10, 11–20, and >20 years of experience, respectively.

In vivo Real-Time Performance

Colonoscopy and polyp characteristics

The 26 participants performed a total of 1451 colonoscopies where diminutive polyps were found, 898 (61.8%) in which a surveillance interval was predicted immediately after the procedure by the participating endoscopist. An additional 348 procedures required histology input for low-confidence characterizations or polyps >5 mm. These procedures were reviewed by study personnel and a prediction was determined based on NBI predictions plus histology where needed (see Supplementary Appendix F for a summary of the decision trees of iterative reasoning used to determine predicted surveillance interval). The predictions were independently audited by 2 additional investigators and there were no discrepancies found. For an additional 157 (10.8%) procedures, NBI predictions of diminutive polyps did not change the follow-up recommendation (Table 2).

Table 2.

Colonoscopy and Polyp Characteristics

Characteristics
Total study colonoscopies, n	1451
Study colonoscopies per participant, mean (IQR)	56.1 (26.0–71.8)
Immediate post-procedure surveillance interval predictions made	898 (61.8)
Delayed surveillance interval prediction incorporating histologya	348 (24.0)
Polyp >5 mm found	276 (19.0)
>2 Low confidence diagnoses	31 (2.1)
History of colorectal cancer	21 (1.4)
≥10 polyps found	20 (1.4)
Surveillance interval not changed by NBI predictions	157 (10.8)
Inadequate bowel prep	66 (4.5)
Deviation from guidelines (elderly, comorbid)	75 (5.2)
Incomplete procedure	7 (0.5)
Familial syndrome (Lynch, FAP, etc)	6 (0.4)
History of inflammatory bowel disease	3 (0.2)
Procedures excluded from surveillance interval analysis	48 (3.3)
Unable to retrieve study polyp	22 (1.5)
Incomplete research form	26 (1.8)
Study polyps, n	3012
Study polyps per participant, mean (IQR)	114.8 (62.3–167.3)
Diminutive polyp histology, n	3012
Adenoma	1600 (53.1)
Non adenoma	1237 (41.1)
Sessile serrated polyp	45 (1.5)
Otherb	53 (1.8)
High-grade dysplasia/cancer	0 (0)
Not retrieved	8 (0.3)
Missing	69 (2.3)
Diminutive polyp confidence, n	3012
High confidence	2239 (74.3)
Low confidence	731 (24.3)
Missing	42 (1.4)
Diminutive polyp location, n	3012
Cecum	318 (10.6)
Ascending	601 (20.0)
Transverse	615 (20.4)
Descending	343 (11.4)
Sigmoid	566 (18.8)
Rectum	522 (17.3)
Missing	47 (1.6)
Total rectosigmoid diminutive polyp predictions, n	1135
High confidence	844 (74.4)
Low confidence	251 (22.1)
Missing	40 (3.5)

NOTE. Values are n (%) unless otherwise note.

FAP, familial adenomatous polyposis; IQR, interquartile range.

^aPerformed by study personnel.

^bOther: inflammatory, carcinoid, lymphoid.

Of the 3012 diminutive polyps included in the study, 2239 (74.3%) were characterized with high confidence and 1600 (53.1%) were adenomas. A total of 45 (1.5%) were sessile serrated polyps, none with dysplasia, and there were no diminutive polyps with high-grade dysplasia or cancer. One thousand one hundred and thirty-five study polyps were located in the rectum or sigmoid colon. Among rectosigmoid polyps, 844 (74.4%) were characterized with high confidence.

Primary aim

Participants achieved an NPV of 94.7% (95% CI: 92.6%–96.8%) in the rectosigmoid colon for polyps characterized with high confidence. Agreement in surveillance intervals predicted using NBI optical diagnosis for diminutive polyps characterized with high confidence in combination with histology for low-confidence characterizations and polyps >5 mm was 91.2% (95% CI: 89.67%–92.65%; 1279 of 1403). Excluding procedures where NBI did not influence the surveillance interval, agreement was 90.1% (95% CI: 88.39%–91.71%; 1122 of 1246). For procedures where endoscopists were able to make a surveil-lance interval prediction immediately after the procedure, agreement was 88.8% (95% CI: 86.7%–90.8%; 797 of 898). Of the 124 procedures in which there was disagreement in surveillance interval, 31.5% (n = 39) predicted a longer interval and 66.1% (n = 82) predicted a shorter interval than dictated by histology, thus 97.0% ([1279 + 82) / 1403]) of predictions would bring patients back on time or early for their surveillance examination.

Secondary Aims

Overall and individual performance

The NBI optical diagnostic performance for diminutive polyps by confidence level and by location is shown in Table 3. The overall accuracy, sensitivity, and NPV for diminutive polyps characterized with NBI were 76.7% (95% CI: 75.2%–78.3%), 95.2% (95% CI: 92.6%–97.8%), and 94.2% (95% CI: 90.4%–98.0%), respectively. For polyps characterized with high confidence, participants achieved an overall accuracy of 84.8% (95% CI: 82.1%–87.5%), sensitivity of 97.6% (95% CI: 95.3%–99.9%), and an NPV of 98.3% (95% CI: 95.7%–100.0%) for adenomatous polyps. Both overall and by location, performance was significantly better for polyps characterized with high confidence when compared with those characterized with low confidence.

Table 3.

Diagnostic Performance of Optical Diagnosis of Diminutive Colorectal Polyps Using Narrow-Band Imaging

	Overall	High confidence	Low confidence
All, n	2876a	2178	694
Accuracy	76.7 (75.2–78.3)	84.8 (82.1–87.5)	60.2 (55.4–65.1)
Sensitivity	95.2 (92.6–97.8)	97.6 (95.3–99.9)	74.6 (65.9–83.4)
Specificity	61.6 (55.8–67.4)	68.9 (60.5–77.2)	50.6 (45.6–55.7)
Area under ROC	0.78 (0.77–0.80)	0.84 (0.82–0.85)	0.62 (0.58–0.66)
PPV	77.9 (74.2–81.6)	83.1 (79.1–87.2)	55.3 (45.6–64.9)
NPV	94.2 (90.4–98.0)	98.3 (95.7–100.0)	80.8 (67.9–93.7)
Proximal to rectosigmoid, n	1818	1360	456
Accuracy	78.8 (75.5–82.0)	84.7 (80.7–88.6)	61.3 (54.3–68.4)
Sensitivity	91.0 (88.3–94.0)	96.2 (94.1–98.4)	73.7 (65.8–81.5)
Specificity	36.9 (27.7–46.1)	34.9 (22.1–47.7)	44.4 (37.3–51.1)
Area under ROC	0.72 (0.70–0.75)	0.79 (0.76–0.81)	0.59 (0.54–0.65)
PPV	83.5 (79.4–87.6)	85.2 (80.9–89.5)	72.9 (60.2–85.6)
NPV	65.6 (59.2–71.9)	77.1 (67.9–86.2)	54.2 (44.1–64.3)
Rectosigmoid, n	1058	818	238
Accuracy	80.9 (76.7–85.1)	88.1 (83.2–92.9)	60.5 (52.5–68.5)
Sensitivity	88.4 (84.8–92.0)	90.9 (87.4–94.4)	73.9 (61.2–86.6)
Specificity	78.3 (71.8–84.9)	88.6 (81.0–96.1)	57.8 (46.9–68.8)
Area under ROC	0.75 (0.72–0.79)	0.80 (0.77–0.84)	0.60 (0.50–0.69)
PPV	56.8 (51.1–62.4)	65.7 (60.9–70.6)	29.1 (20.8–37.3)
NPV	93.7 (91.8–95.7)	94.7 (92.6–96.8)	90.1 (84.8–95.4)

NOTE. Values are % (95% CI) unless otherwise noted.

NPV, negative predictive value; PPV, positive predictive value; ROC, receiver operating characteristics.

^aPolyps were excluded from analysis if histology was missing, other, or unable to retrieve; and if confidence designation was missing.

Individually, 20 of 26 participants achieved ≥90% NPV in the rectosigmoid. Of the remaining 6, only 1 participant achieved an agreement that did not cross the 90% threshold with a 95% confidence interval (Figure 1). Thirteen of 26 participants achieved an overall agreement in surveillance interval of ≥90%. Of the remaining 13, only 2 had agreement percentages that did not cross the 90% threshold with a 95% CI (Figure 1).

Figure 1.

Individual performance on PIVI thresholds: NPV for high-confidence diagnoses in the rectosigmoid and agreement in predicted and actual surveillance intervals (% with 95% CI); 77% (n = 20) participants met the NPV threshold, 50% (n = 13) met the agreement in surveillance interval threshold and 35% (n = 9) met both, and 8% (n = 2) met neither threshold.

Predictors of Performance

Adjusting for physician characteristics, high-confidence characterization was a significant predictor of optical diagnostic accuracy (OR = 3.42; 95% CI: 2.72–4.29; P < .001). In contrast to the training session, increasing years of experience was associated with higher accuracy; compared with endoscopists with <5 years of experience, the odds of a correct response was 1.43 (95% CI: 1.04–1.36; P = .03), 1.70 (95% CI: 1.22–2.38; P = .002), and 2.06 (95% CI: 1.41–3.00; P < .001) for participants with 5–10, 11–20, and >20 years of experience, respectively. Study polyp size of 5 mm was significantly associated with accuracy (OR = 2.09; 95% CI: 1.35–3.23; P = .001)] when compared with polyp size of 1 mm.

Performance in the training session, institution, participant sex, amount of clinical effort, annual colonoscopy volume, endoscope model (180 vs 190), and study polyp location were not significant predictors of optical diagnostic accuracy in real time.

Learning Effect and Cumulative Summation Analysis

When divided into blocks, accuracy; sensitivity; and agreement in surveillance intervals improved as participants progressed through the study (Figure 2). Based on the likelihood ratio test for the set of indicators of learning blocks, the overall learning effect was significant for agreement in surveillance intervals (P = .02). There was a nonsignificant trend toward learning for accuracy (P = .12) and sensitivity (P = .16) for high-confidence characterizations.

Figure 2.

Learning effect by block of procedures (odds ratio with 95% CI compared with colonoscopy 1 to 15).

CUSUM learning curve analyses are summarized in Figure 3. Twenty-five participants require ongoing observation for accuracy of polyps characterized with high confidence in the rectosigmoid colon, 1 requires formal retraining, and none achieved competence (Figure 3B). Seven participants achieved sufficient sensitivity for characterization of adenomas, such that additional observation is not required, and the remaining participants require ongoing observation before a conclusion on competence in adenoma sensitivity can be drawn (Figure 3C). Seven participants achieved competence in surveillance interval predictions, whereas 18 require ongoing observation and 1 requires retraining (Figure 3D).

Figure 3.

CUSUM learning curve analysis. Graphical representation of learning curve among all 26 participants for (A) overall accuracy for high-confidence diagnoses, (B) accuracy for high-confidence diagnoses in the rectosigmoid colon, (C) overall sensitivity, and (D) agreement in predicted and actual surveillance intervals. Crossing the lower limit threshold (h0) indicates performance within the acceptable rate of 15% (achieved competency), crossing the upper threshold (h1) suggests unacceptable rate of 20% (requires retraining) and results between these 2 thresholds indicate need for ongoing observation.

Discussion

The “characterize, resect, and discard” strategy has great potential for cost savings, while preserving quality of care. Although numerous studies have shown that experts perform at a high level in diminutive polyp optical diagnosis using NBI, studies including those without prior training or academic interest in NBI have not been able to meet the thresholds set forth by the ASGE.¹⁰

This prospective, multicenter observational study included a large number of endoscopists with no prior training in NBI. In contrast to prior studies, participants underwent an innovative training method using real-time NBI video clips and were provided ongoing feedback during training to promote active and corrective learning. This study was designed and powered for the primary aim of evaluating whether these 26 endoscopists could collectively meet the ASGE PIVI targets for the “characterize, resect, and discard” strategy. Histologic input for low-confidence diagnoses were accounted for prospectively and structured feedback was provided throughout the real-time phase of the study to promote active learning/retraining.

Participants achieved a 94.7% NPV for rectosigmoid polyps characterized with high confidence, which exceeds the 90% target proposed by the ASGE. When the histologic outcome of low-confidence characterizations and polyps >5 mm were combined with NBI predictions of high-confidence diminutive polyps, overall agreement in surveillance intervals was 91.2%. Most (66.1%) surveillance interval disagreement would result in performing surveillance colonoscopies earlier than recommended by guidelines, such that 97.0% of the procedures in this study would have brought patients back on time (91.2%) or early (5.8%).

Several prior studies have demonstrated that expert endoscopists can achieve the PIVI thresholds set forth by the ASGE, however, a majority of these studies were performed at academic centers and often included participants with extensive experience in NBI interpretation (Table 4). Most recently, the Veterans Affairs Lesion Interpretation and Diagnosis study¹⁶ included 5 academic gastroenterologists and reported a 95% NPV for left colon polyps characterized with high confidence and a 93% agreement in predicted and actual surveillance intervals. Although these studies demonstrate that endoscopists can meet the ASGE PIVI benchmarks, participants were all academic gastroenterologists, and some contributed to the development and validation of endoscopic advanced imaging descriptions and criteria.^10,31

Table 4.

Summary of Studies of Optical Diagnosis Using Narrow-Band Imaging

First author, year	Setting	Participants, n	NBI experienced participants, n	Training method	Total polyps, n	Diminutive polyps, n	High-confidence diagnosis	Surveillance Predictions, n	Performance	SI agreement
Rex,13 2009	Academic	1	1	Image library	451a	395	All: 368/451 (81.6%) Dim: 314/395 (79.5%)	136	All high confidence: NPV 95% Accuracy 94%	128/136 (94%)
Rastogi,14 2009	Academic	1	1	None	236a	170	NA	NA	All polyps: Sensitivity 95% Specificity 88% Accuracy 92%	NA
Ignjatovic,15 2009	Academic	4	2b	Image library	363c	296	All: 326/363 (89.8%)	82	All polyps: NPV 82% Accuracy 93%	80/82 (98%)
Kaltenbach,16 2015	Academic	5	5	“Standardized training”	1309c	975	All: 774/1309 (59.1%) Dim: 680/975 (69.7%)	558	All high confidence:d NPV 96.4/92 Accuracy 88.5/87 All left colon: NPV 95.3/93.9 Accuracy 88.6/90.2	518/558 (92.8%)
Paggi,17 2012	Community	6	6	Image library with pre/post-test (Rex/Rhagavendra)	511c	399	All: 86.1%e	286	Diminutive polyps: Sensitivity 93.9% Specificity 68.0% Accuracy 84.0% Rectosigmoid polyps: Sensitivity 93.7% Specificity 70.2% Accuracy 84.2%	237/286 (82.9%)
Ladabaum,18 2013	Community	12	0	Image library with pre/post-test (Rex/ Rhagavendra)	2596a	1858	Dim: 1481/1849 (80.1%)	1065	Diminutive polyps: NPV 75.9% Accuracy 78.1% High-confidence diminutive polyps: NPV 78.3% Accuracy 81.1% All rectosigmoid polyps: NPV 87.4% Accuracy 77.4%	851/1065 (80%)
Kuiper,20 2012	Community	3	3? Previous participation in ex vivo and in vivo NBI studies	Image library	281c		All: 231/281 (82%)	54	All polyps: Sensitivity 77% Specificity 78.8% Accuracy 77.9%	44/54 (81%)
Vu,19 2015	Academic + Community	6	4	Image library, training session led by expert	606	606	Dim: 580/606 (95.7%)f	307	All high-confidence polyps:g NPV 69% Accuracy 81%	82.1%
Patel, 2015	Academic	26	0	Video-based training, active feedback, led by expert	3012	3012	Dim: 2239/3012 (74.3%)	1403	All high confidence: NPV 98.3% Accuracy 84.8% High-confidence rectosigmoid: NPV 94.7% Accuracy 88.1%	91.2%

NA, not applicable; SI, surveillance interval.

^aIncluded polyps >10 mm.

^bExperts performed significantly better than nonexperts.

^cIncluded polyps ≤10 mm.

^dFor near focus/standard view.

^eColonoscopies with low-confidence diagnoses were excluded from analyses.

^fHigh confidence defined by visual analog scale ≥7.

^gNo difference in performance between community and academic participants.

In contrast, studies that have included gastroenterologists without prior experience or academic interest in NBI have not met the PIVI thresholds. Paggi et al¹⁷ included 6 community-based endoscopists who achieved 85% agreement in surveillance intervals, and Kuiper et al²⁰ included 3 community-based endoscopists who achieved 81% agreement in surveillance intervals. Ladabaum et al¹⁸ included 12 community gastroenterologists who achieved 91% NPV for the last 20 rectosigmoid polyps characterized with high confidence, but only an 80% agreement in predicted and actual surveillance intervals. Vu et al¹⁹ included 6 endoscopists (4 academic, 2 community-based) who achieved an 82.1% agreement in surveillance intervals. Limitations to these studies have included small number of participants, inconsistent training in the interpretation of NBI, lack of performance feedback during the study period, and lack of prospective accounting for histology informing low-confidence predictions. Furthermore, some of these studies were not specifically designed and powered to address the PIVI thresholds.

Our study attempted to overcome limitations of prior studies by including a large number of gastroenterologists with no prior experience or training in NBI at 2 academic medical centers. Endoscopists underwent a previously described standardized intensive training session^23,30 and had excellent performance during training with a demonstrable learning effect (Supplementary Appendix D). Furthermore, structured feedback was provided throughout the study to promote ongoing learning.

For the majority of cases in which there was a disagreement in surveillance intervals, patients would have been brought back earlier than recommended based on histology. This finding suggests that there is a low likelihood of delaying surveillance (3% in our study) and missing lesions. However, the impact on cost effectiveness and the potential harms of performing procedures early and more frequently (including procedural risks, such as significant bleeding, sedation-related risk, and perforation) would need to be examined. These harms may counter the cost savings associated with forgoing pathologic assessment of diminutive polyps characterized with high confidence, and the issue need to be examined further.

The overall percentage of high-confidence diagnoses in this study was lower than expected at 74%. The proportion of high-confidence characterizations published in the literature is variable and has ranged from 69.7% in the Kaltenbach et al¹⁶ study to as high as 95.7% in the Vu et al¹⁹ study (highlighted in Table 4). It is unclear why there was a lower proportion of high-confidence diagnoses in this study, despite intense training and regular feedback.

Based on the results of this study, there are several potential conclusions about NBI training. First, given that performance in training was not a predictor of performance in real time, one cannot rely on ex vivo performance alone to sign off on real-time application of NBI. Second, although only 34.6% (9 of 26) of endoscopists met both benchmarks by the end of the study, 76.9% (20 of 26) met NPV and 50% (13 of 26) met surveillance interval agreement, our study was not powered to examine individual performance. The learning curve analysis demonstrates that individual participants acquire expertise in NBI at a varying pace, but improvement does occur over time. Structured feedback may need to be continued for longer periods before an endoscopist should apply the “characterize, resect, and discard” strategy. Alternatively, other approaches to educating endoscopists or more targeted retraining/feedback may need to be tested. The wide CIs (Figure 1) that generally cross the 90% threshold suggest that participants might have needed more procedures to achieve adequate performance. The variability in learning curve and performance is not surprising and is akin to other performance measures in colonoscopy, such as adenoma detection rates, which also demonstrate wide variability in performance. Finally, just like other quality metrics, performance should be assessed and audited periodically to ensure that benchmarks are being met and maintained. This study shows that certain individuals consistently perform well and, as such, would be ideal candidates to employ the resect and discard strategy.

Contrary to our expectation, the less experienced endoscopists performed better in the training video set. As we expected, the more experienced endoscopists performed better in real time. It is unclear why this might be, however, it raises the possibility of generational differences in the role of technology in learning. The less-experienced endoscopists are part of a generation dominated by technology and may be better equipped to perform well during a computer-based audiovisual training tool.³² More experienced endoscopists incorporate years of systematic pattern recognition and may rely more on “implicit” memory informed by years of endoscopic experience.³³

Although the NBI classification criteria were designed to differentiate adenomatous from hyperplastic polyps and do not take into account sessile serrated polyps, only 45 (1.5%) of diminutive polyps included in this study were sessile serrated. Of these, none carried cytologic dysplasia and only 5 were located in the rectosigmoid colon. Excluding these polyps from the analysis did not alter the results. As previously advocated, until reliable criteria can be developed for optical differentiation of sessile serrated lesions, most experts recommend resection and pathologic examination of all right colon polyps, regardless of size.³⁴ As reported in numerous previous studies,^13–20 our study confirms a low to nonexistent risk of high-grade dysplasia or cancer in diminutive polyps, adding credence to the safety of the “characterize, resect, and discard strategy.”

There are several limitations to this study. In an attempt to minimize potential gaming bias, endoscopists predicted surveillance intervals immediately after the procedure only, taking into account up to 2 low-confidence predictions. If a surveillance interval prediction required histologic input for ≥3 low-confidence characterizations or a polyp >5 mm, a single-study investigator made a guideline-based prediction based on a combination of NBI and histology. Although this method deviates from real practice where the individual endoscopist would make a prediction once histology was available, we elected this design to eliminate the temptation for gaming. The surveillance intervals determined by the study investigator were independently and blindly audited to ensure accuracy.

Two of the sites used 180 series endoscopes, whereas 2 used 190 series scopes. This is unlikely to significantly impact results, given that the type of endoscope used was not changed for the duration of the study and endoscope model was not found to be a predictor of performance in the multivariate analysis. In addition, earlier studies^35,36 have shown no difference in NBI diagnoses between 180 and 190 series colonoscopes. All sites had pathology reviewed by a fellowship-trained gastrointestinal pathologist, however, pathology review was not centralized among the 4 institutions. The interactive training session, in which feedback was given, was moderated by an expert. This might not be a widely available resource, however, computer-based training modules could be developed to mimic this training method. With the inclusion of 4 different participating centers, there was variability in patient population and procedure indication. As a result, quality metrics, such as adenoma detection rate and polyp detection rate, could not be evaluated as potential predictors of performance.

Despite these limitations, this prospective multicenter study included a large number of academic endoscopists with no prior training in NBI who were able to achieve the NPV and surveillance interval agreement thresholds set forth by the ASGE.

Challenges remain to widely implementing the “characterize, resect, and discard strategy.” In addition to undergoing a standardized training session as used in this study, there must be commitment at the individual, departmental, and institutional level to dedicate time and resources to monitor/audit performance to ensure adequate performance. There is currently no incentive to implement such a program. Even if such a program is put in place, the cost effectiveness and efficiency of the strategy would need to be re-examined, taking into account the time and resources needed to audit performance, the cost of performing procedures early, and the cost of a higher proportion of low-confidence diagnoses in this study (75.7%) compared with that assumed (84%) in the previous cost-effectiveness studies.¹¹ An additional challenge in implementing “characterize, resect, and discard” is how to calculate and monitor quality metrics, such as adenoma detection rate, although there are data that appropriate photo documentation can allow accurate determination of adenoma detection rate.³⁷ If polyps are diagnosed optically, systematic photo documentation and saving high-definition polyp images would be required to support endoscopists' claims of adenoma detection.

In conclusion, our study demonstrates that as a group and select individual endoscopists can reach the thresholds set forth by the ASGE, but that simply performing well in training does not translate into acceptable real-time performance. If the “characterize, resect, and discard” strategy is implemented, a structured and accountable auditing method will be required. In addition, the cost-effectiveness will need to be re-examined.

Abbreviations used in this paper

ASGE

American Society for Gastrointestinal Endoscopy

CI

confidence interval

CUSUM

cumulative summation

NBI

narrow-band imaging

NPV

negative predictive value

OR

odds ratio

PIVI

Preservation and Incorporation of Valuable Endoscopic Innovation

VA

Veterans Affairs.