Development and Validation of an Automated, Real-time Adenoma Detection Rate and Colonoscopy Quality Metrics Calculator

Abstract

BACKGROUND AND AIMS:

High-quality colonoscopy reduces the risk of death from colorectal cancer. The adenoma detection rate (ADR) is the principal measure of colonoscopy quality but is onerous to calculate. We report the development of a fully automated platform for calculation of the ADR and other key colonoscopy quality indicators without the need for manual data entry.

METHODS:

Endoscopy and pathology reports from 6 centers were collected over a 3-month period and collated using a novel data transfer interface. Text-based classification parameters were developed to identify average-risk screening colonoscopies, adenomatous pathology, cecal intubation, and withdrawal time. Automated quality metrics calculators based on these classifications were built into a web-based reporting platform, and the resulting quality metrics were benchmarked against those produced through a manual record review. Confirmation of the calculator’s performance was performed in a validation cohort with data collected over a 1-month period, 6 months after the initial study.

RESULTS:

The study included 3809 colonoscopies (mean age 56.1 ± 6.40 years, 53.7% female, 38 endoscopists). The automated calculator yielded an ADR of 45.1% compared with 44.3% on manual review. Correct classification of ADR-qualifying screening colonoscopies was achieved with high predictive value, with a sensitivity of 0.918 and specificity of 1.0. The cecal intubation rate was 95.8%, and the average withdrawal time was 10:05 minutes.

CONCLUSION:

We demonstrate the feasibility and performance of a colonoscopy quality reporting platform capable of calculating the ADR and other key metrics using novel, fully automated pathology report integration and a text query-based classification accessible in a wide range of practice settings.

Introduction and Background

Colonoscopy is the most used modality for colorectal cancer (CRC) screening in the United States,^1,2 and high-quality colonoscopy has been shown repeatedly to reduce the risk of death from CRC through early tumor detection and excision of precancerous adenomas.^3,4

Numerous metrics are correlated with improved CRC detection by colonoscopy and have been identified by the American Society of Gastrointestinal Endoscopists (ASGE) as quality indicators, including bowel preparation quality, cecal intubation rate, withdrawal time, and adenoma detection rate (ADR).^5–15 The ADR has been proposed as the principal surrogate marker for colonoscopy quality in the setting of CRC screening because of its well-studied, inverse association with the risk of interval CRC, including advanced-stage and fatal malignancies.⁵

Routine widespread use of ADR benchmarking to measure colonoscopy quality, however, has been limited, with as many as 40% of endoscopy facilities failing to calculate this metric.¹⁶ ADR calculation is resource-intensive, often requiring manual collation of endoscopy and pathology data across multiple reporting modalities, making it an impractical tool for frequent quality audits at many centers.

Many attempts have been made to streamline the calculation of the ADR and other colonoscopy quality indicators, but many of these approaches rely on manual pathology data entry, a significant added burden for clinicians. Attempts to automate this process, for instance, using natural language processing to analyze pathology reports, although potentially effective, are costly to implement and difficult to tailor to local institutional needs. Thus, there is a substantial demand for a novel tool to extract and analyze colonoscopy indicators from text-based reports that provides accurate data extraction in a package that is easily implemented and modified by clinicians.

In this study, we report the development of a fully automated colonoscopy quality reporting platform that allows for the high-fidelity extraction and integration of quality data from 2 of the most widely utilized endoscopy and pathology reporting systems using an intuitive and versatile text-matching approach.

Materials and Methods

Study Setting

This study included colonoscopies performed at the Johns Hopkins Medicine Institutions from July 28, 2021, to October 27, 2021. Data were collected from 6 facilities (3 ambulatory surgical centers and 3 hospital-based units) comprising 100 endoscopists. This study was exempted from Johns Hopkins Medicine institutional review board approval as a quality improvement project.

Data Integration and Extraction

Pathology reports generated in the Epic Beaker pathology reporting platform (Epic Systems, Verona, WI) were uploaded and collated in the Provation EndoPro endoscopy reporting platform (Provation Medical, Minneapolis, MN) using a data transfer interface based on the Health Level Seven (HL7) standard for health care information technology (Figure 1). HL7 represents an international standard that facilitates interoperability between electronic health records systems and allows for the automated flow of pathology data from Beaker into EndoPro. Pathology reports were matched to the corresponding endoscopy report by extracting key demographic data points (patient name, medical record number (MRN), procedure code, and specimen collection date) and searching the EndoPro database for a report corresponding to that patient and procedure code, with a procedure date corresponding to the specimen collection date. The collated reports were then queried for Beaker-designated smart phrases corresponding to common benign, pre-malignant, and malignant lesions (Appendix 1). Reports were given binary labels indicating whether an adenomatous lesion was detected based on the presence of the designated smart phrases. Procedural and demographic data were then extracted from the report and uploaded to a central, internet-based analytics suite (Qlik, QlikTech, King of Prussia, PA). This central data repository was automatically updated daily.

Figure 1.

Data integration and quality metric calculation workflow for pathology and endoscopy reporting platforms. After a colonoscopy, procedural data and histopathology results are recorded in the endoscopy (EndoPro) and pathology (Beaker) reporting suites, respectively. Pathology results are transmitted from Beaker to EndoPro via an HL7 data interface and assigned to the individual procedure in EndoPro, allowing for subsequent data extraction and calculation of colonoscopy quality metrics ADR, cecal intubation rate, BBPS, and colonoscope withdrawal time.

Text Criteria Selection for ADR-qualifying Procedures

The ADR is defined by the ASGE as the fraction of asymptomatic average-risk patients undergoing screening colonoscopy who have 1 or more adenomas detected.^5,6,17 Procedures qualifying for inclusion in the ADR were defined as screening colonoscopies in patients at average risk for CRC, using the criteria used by Corley et al (Figure 2).¹⁸ Patients at higher-than-average risk for CRC were excluded, explicitly those with a personal history of CRC or polyps, significant family history of CRC, or inflammatory bowel disease. Patients with an interval between colonoscopies of less than 10 years were also excluded to ensure that no surveillance colonoscopies were captured. Patients with poor bowel preparation, defined as a Boston Bowel Preparation Score (BBPS) less than 6, were also excluded from ADR calculation. To identify patients meeting these criteria, a series of text-based classification rules were developed in an iterative fashion, as described below. The final classification scheme is detailed in Appendix 2.

Figure 2.

Inclusion and exclusion criteria for identification of ADR-qualifying screening colonoscopies. Inclusion criteria were age ≥45 years old, average risk for CRC, and adequate bowel preparation (defined as BBPS ≥6). Patients with inflammatory bowel disease, personal/family history of CRC, or recent surveillance colonoscopies (defined as previous colonoscopy within 10 years) were excluded.

Automated Calculation of Quality Metrics

Formulae for the automated calculation of quality metrics were coded in the Qlik analytics suite. The ADR was defined as the fraction of adenoma-positive ADR-qualifying procedures divided by the total number of ADR-qualifying procedures. ADRs stratified by gender, patient age, facility, and endoscopist were calculated in a similar fashion. For comparison purposes, an ADR inclusive of non-screening procedures was also calculated, as was the polyp detection rate (PDR). The latter was defined as the number of screening colonoscopies in which any polyp was detected divided by the total number of screening colonoscopies. Colonoscope withdrawal time and BBPS⁷ were extracted from corresponding structured data fields in the endoscopy report. Cecal intubation was determined based on the presence of the terms “cecum,” “terminal ileum,” or “ileocecal valve” in the “extent of endoscopic visualization” data field in the endoscopy report. The cecal intubation rate was calculated as the number of procedures in which the cecum was visualized divided by the total number of procedures.

Manual Validation of Quality Metrics

Manual review of the patient’s electronic medical record and data extracted from the endoscopy and pathology reports was performed by a single reviewer. ADR-qualifying screening colonoscopies were identified based on review of the procedural indications stated in the colonoscopy report and referenced against the past medical history to identify patients with higher-than-average risk of CRC. This manual classification of cases facilitated the calculation of sensitivity, specificity, and other test characteristics by establishing a reference standard. The identification of false-positive and false-negative classifications in turn allowed for the identification and inclusion of common variants of key classification terms (ie, “average risk,” “average-risk,” “avg risk”), as well as missing search terms or parameters. This manual review process was applied to the classification criteria for each metric in an iterative fashion, with serial revision of the criteria to minimize the number of incorrect classifications.

Retrospective Validation of the ADR Calculator

To demonstrate the durable performance of this tool over time, we further collected data for all colonoscopies performed at our institution between April 1 and 30, 2022, 6 months after the initial study period. This time point was selected to simulate variability over the course of an academic year, ideally accounting for changes in documentation patterns that occur at academic institutions at which fellows may be the principal authors of many endoscopy reports. Manual validation of the ADR-qualifying classification and adenoma detection tools was performed as above.

Descriptive Statistics

For each quality metric, sensitivity, specificity, and positive predictive value were calculated in Prism 9 (Graphpad, San Diego, CA). Metrics stratified by individual endoscopist, facility, and age group are reported as mean ± standard deviation; correlations between these variables were assessed by R² value.

Results

Study Population Characteristics

A total of 3809 consecutive colonoscopies performed at Johns Hopkins-affiliated facilities were evaluated over a 3-month period. The demographic characteristics of the cohort are summarized in Table 1.

Table 1.

Patient and Endoscopist Demographic Characteristics for the Initial Prospective Study Cohort and the Retrospective Validation Cohort, Respectively

	Initial cohort (n = 3809)	Validation cohort (n = 1384)
Location: Hospital unit (%) Ambulatory surgical center (%)	49.1 50.9	46.2 53.8
Gender (% male)	46.3	45.1
Age	56.1 +/− 6.40	58.1 +/− 0.237
Total # endoscopists	100	74
# Endoscopists with ADR-qualifying procedures	38	36
% Endoscopists with > 5 colonoscopies/month	34	47.0

Automated ADR Calculator Performance

The automated ADR calculator identified 379 ADR-qualifying procedures, with 171 of those procedures yielding adenomatous pathology (Table 2). This resulted in a calculated ADR of 45.12%. Manual review yielded a similar ADR of 44.3%. The discrepancy between manual and automated ADR calculations was exclusively attributable to missed (ie, false negative) identification of ADR-qualifying procedures (sensitivity 0.917, false detection rate 0, positive predictive value 1.0). Of the 43 mislabeled cases, 20 had pathology results that were either pending or represented erroneous pathology sample entries (ie, cases in which a pathology specimen was ordered but never sent due to the absence of polyps for biopsy). The remaining 23 “mislabeled” cases were excluded by the classification system because of variant syntax and/or spelling in the EndoPro report’s documentation of procedural indications and/or documentation of previous colonoscopies. Examples of these variations are shown in Appendix 3. We did not detect any cases of misclassification of the pathology data, likely because Epic Beaker relies on standardized smart phrases, and, to the best of our knowledge, we have included all the Beaker phrases pertaining to adenomatous pathology (Appendix 1). The cecal intubation rate for this cohort was 95.8%, with an average withdrawal time of 10:05 minutes; 77.6% of patients had an adequate bowel preparation (BBPS ≥ 6).

Table 2.

Automated and Manually Calculated ADR for the Prospective Study Cohort and the Retrospective Validation Cohort, Respectively

	Initial study cohort		Validation cohort
	Automated calculation (n, [% total])	Manual validation (n, [% total])	Automated calculation (n, [% total])	Manual validation (n, [% total])
# Colonoscopies	3809	3809	1384	1384
# ADR-qualifying procedures	379 (9.95%)	413 (10.8%)	358 (25.9%)	353 (25.5%)
ADR-qualifying w/adenoma detected	171 (4.49%)	183 (4.80%)	76 (5.49%)	74 (5.35%)
ADR	45.1 %	44.3%	21.2 %	21.0%

To further assess the utility of an automated ADR calculator specific to average-risk, screening colonoscopy procedures, commonly used alternatives to the ADR, such as an ADR agnostic to procedural indication, PDR, and a PDR specific to screening procedures, were also calculated. The screening-agnostic ADR was determined to be 31.8% (1211 adenoma-identifying procedures out of a total of 3809 colonoscopies). The screening-specific PDR was 86.3% (327 polyp-identifying procedures out of 379 average-risk screening colonoscopies), and the screening-agnostic PDR was 69.6% (2652 polyp-identifying procedures out of 3809 total colonoscopies).

Colonoscopy Quality Reporting Platform

Figure 3 shows the web-based colonoscopy quality reporting platform’s central dashboard, which provides the end user with a virtual quality “report card.” This tool displays the ADR, cecal intubation rate, BBPS distribution, and average withdrawal time. The dashboard can also be customized to include institutional targets for each metric, as well as pre-set filters to stratify patients by demographic variables. This platform is updated daily via the HL7 data transfer interface.

Figure 3.

Colonoscopy quality analytics report readout for the validation cohort (n = 3809). The first row presents (left to right) the total number of colonoscopies, ADR-qualifying procedures, ADR-qualifying procedures in which adenomatous pathology is detected, and the ADR as a percentage. The second row contains (left to right) the ADR stratified by patient gender, pie chart representation of BBPS distribution, cecal intubation rate, and mean colonoscope withdrawal time.

Subgroup Analyses

The platform’s quality reporting capabilities for demographic subgroups are demonstrated in Figure 4. Stratification of the cohort by facility allows for calculation of a group ADR for each facility, as well as comparison of facility types. In our cohort of 3 hospital units and 3 ambulatory centers, there was no significant difference between the ADRs of the 2 facility types, with a mean in-hospital ADR of 44.7% ± 13.6% and an ambulatory center ADR of 46.3% ± 5.9% (P = 0.43). Similarly, the group ADR stratified by age group demonstrates an interval increase in the ADR with each successive 10-year age group, consistent with known correlations between age and risk of CRC.

Figure 4.

Grouped ADR stratified by (A) individual facility and (B) patient age group. Each bar represents the average ADR for that demographic category across all endoscopists.

Correlation of the ADR With Other Quality Indicators

We further demonstrate the ability of the platform to examine relationships between quality metrics. Figure 5 shows the group ADR as a function of BBPS (5A), as well as the correlation of the ADR with cecal intubation rate, average withdrawal time, and ADR-qualifying procedural volume, respectively (5B-D). In our cohort, the cecal intubation rate and volume of average-risk screening procedures correlated poorly with an endoscopist’s ADR (R² = 0.014 and R² = 0.0021, respectively), and the average withdrawal time correlated with ADR performance (R² = 0.127).

Figure 5.

Group ADRs stratified by BBPS (A; note that BBPS ≥ 6 was a prerequisite for ADR-qualifying procedures). The ADR stratified by endoscopist is plotted against the per-endoscopist cecal intubation rate for screening procedures (B; R² = 0.014), per-endoscopist withdrawal time for screening procedures (C; R² = 0.1276), and the per-endoscopist number of ADR-qualifying procedures (D; R² = 0.0021). Each circle in B-D represents a single endoscopist; the best-fit linear regression is shown in red.

Retrospective Validation of the ADR Calculator

A total of 1384 colonoscopies were performed at our institution between April 1 and 30, 2022, and analyzed retrospectively using the automated colonoscopy quality metrics platform (Table 1). The automated ADR calculator identified 358 ADR-qualifying colonoscopies with a sensitivity of 0.958 and a specificity of 0.990. The adenomatous pathology detection tool again had a sensitivity of 1.0 and specificity of 1.0. The resultant ADR was 21.2% by automated calculation and 21.0% on manual review.

Discussion

The value of the ADR as a surrogate for screening colonoscopy quality and a predictor of interval CRC has been well established.^5,18,19 However, despite this evidence and a plethora of published methods and semi-automated colonoscopy quality reporting tools, routine ADR calculation remains limited, likely owing to the need for time- and labor-intensive manual data entry.¹⁶

In this pilot study, we demonstrate that real-time data integration across pathology and endoscopy reporting platforms, without the need for manual input of pathology data or artificial intelligence algorithms, is feasible and allows for rapid and accurate calculation of colonoscopy quality metrics, including the ADR, cecal intubation, average withdrawal time, and adequacy of bowel preparation. We believe this tool, in addition to its accuracy, will be easily implemented at other institutions using our roadmap for generalizable implementation in alternative health systems (Figure 6).

Figure 6.

Roadmap for generalizable implementation in other health systems. Practices currently using EndoPro and Epic can develop their own automated colonoscopy quality platforms using these step-by-step instructions (left) and the query language outlined in Appendices 1 and 2. We further outline solutions to anticipated barriers to adoption (right) including the integration of other reporting software and optimization of the classification rules and quality reporting dashboard.

Several attempts have previously been made to automate the calculation of the ADR and related colonoscopy quality metrics. One approach has been to develop dedicated endoscopy reporting systems with structured data fields that allow for easy tabulation of quality metrics, including the ADR; these approaches, however, require manual pathology data entry.^20–23 Alternatively, several attempts have been made to extract quality metrics from free-text pathology and endoscopy reports using an artificial intelligence modality called natural language processing.^24–31 These platforms have the benefit of interoperability with many electronic medical recording platforms, but their widespread implementation has been hampered by high upfront costs and difficulties generalizing the technology to smaller or non-academic practice settings.³² Furthermore, these tools, due to their complexity, would not be optimizable by most end users.

Similar to our approach, Gawron et al used text-based searches to identify adenomatous pathology and calculate quality metrics.³³ This study, however, included all colonoscopies, including diagnostic, surveillance, and advanced endoscopic procedures. Although this streamlines the process of data extraction, the inclusion of non-screening colonoscopies risks distortion of quality metrics and is not in keeping with the ASGE guidelines.⁶ In our cohort, inclusion of non-screening colonoscopies resulted in an ADR of 31.8%, far below the ADR calculated manually. This approach also risks under- or overestimation of the ADR for subspecialists who may perform a large volume of higher-than-average risk or surveillance colonoscopies (ie, inflammatory bowel disease specialists) or perform disproportionate numbers of colonoscopies unrelated to CRC screening (ie, advanced endoscopists).

In contrast to our findings, Kaltenbach et al recently reported minimal change in the ADR with exclusion of non-screening procedures at 2 Veteran Affairs (VA) centers.³⁴ We suspect that this discrepancy between their conclusions and our own stems from 2 factors. One is that our institution is a tertiary referral center with subspecialists likely performing a disproportionate number of non-screening colonoscopies as compared with community practice, where an estimated 43% of colonoscopies are performed for CRC screening.³⁵ The second is that we have opted for a conservative definition of ADR-qualifying screening colonoscopies, for instance by excluding cases in which the patient might have had an interval screening or non-screening colonoscopy in the preceding 10 years or those in whom colonoscopy is performed for minor symptoms without malignancy warning symptoms (eg, gross hematochezia, weight loss). Although this conservative approach does markedly decrease the number of cases that qualify for inclusion in the ADR, it also ensures that patients with higher-than-average risk of CRC are appropriately excluded.

It should also be noted that defining ADR-qualifying procedures based solely on the stated procedure performed, rather than on the stated procedural indication, resulted in a high rate of misclassification in our cohort of cases. At our institution, 33.6% of purported “screening colonoscopies” failed to meet ASGE criteria, whereas 22% labeled as “colonoscopy with polypectomy” or “diagnostic colonoscopy” were determined on manual review to carry a screening indication. Interestingly, Nayor et al reported that in their ADR-qualifying cohort, defined based on the stated procedural indication, only 1.6% were misclassified.³⁰ This raises the possibility that even more accurate classification criteria can be reached through more uniform documentation of average-risk screening procedures, for instance by using a binary, structured data field in the endoscopy reporting suite.

An important novel aspect of our platform is its ability to generate colonoscopy quality metrics on demand, with daily updates of pathology and endoscopy data. By making these data more accessible, we believe this platform offers 2 key advantages over current practice. One is that automated, on-demand reporting of the ADR obviates the need for commonly used surrogate performance metrics, such as the PDR, mean polyp detection, or colonoscope withdrawal time, for which there is less evidence with respect to the prediction of post-colonoscopy outcomes.^5,36,37 As further evidence of this, we demonstrate in our cohort a PDR in the screening population of 86.3%, which, when adjusted with a published adenoma-to-polyp-detection-rate ratio of 0.72,³⁸ represents an estimated ADR of 62.1%, a value that would grossly overestimate the true ADR in our cohort.

A second key advantage of this platform is the opportunity to generate colonoscopy quality reports at far more frequent intervals than is currently possible with manual ADR-reporting methods. There is already substantial evidence that routine quality metrics reporting increases the per-endoscopist ADR,^39–45 and we anticipate that further study will elucidate what kinds of such quality improvement interventions demonstrate an effect.

It should be noted that we have trained the indication-related text queries herein described through iterative analysis of documentation patterns at a single US health care system. We anticipate that application of these quality metrics calculators at other centers, including those without the benefit of standardized pathology reporting syntax, might result in a reduction in their performance due to variations in documentation practices. For example, if institutional practice dictates the reporting of non-adenomatous pathology using negated syntax (such as “no evidence of tubular adenoma”), then text-matching using the phrase “tubular adenoma” to identify adenomatous pathology could result in the incorrect classification of non-adenomatous polyps. Unlike a natural language processing-based approach, however, text-based classification parameters are easily adapted by the end user to local practice and documentation patterns without the need for an engineer or data scientist.³² In the use case described, the end user would simply modify their text-matching criteria to reflect the syntax commonly used in local practice (ie, using specific criteria to differentiate the phrase “no evidence of tubular adenoma” from adenomatous diagnoses such as “fragments of tubular adenoma”). An alternative approach at such an institution, or at institutions in which pathology or endoscopy reporting is heterogenous, text-matching tools such as this could encourage more uniform and comprehensive documentation of procedural details and findings.

Another limitation of this study is that it was constructed within the confines of just 2 of the many electronic medical reporting systems. Current users of both Epic and EndoPro will be able to develop this platform within their own practice by following our roadmap for generalizable implementation in other health systems (Figure 6). By constructing this system using the widely utilized HL7 standard, we are hopeful that users of other electronic record systems will be able to construct similar platforms, and we detail an approach to this process in Figure 6.

It is also worth noting that an inherent challenge for any real-time reporting system is the potential for missing data due to inherent desynchrony in endoscopy and pathology. In our study, however, this rate of data missingness was quite low, with <2% of cases lacking pathology reports at the time of analysis.

Further development of this internet-based colonoscopy quality reporting platform will focus on integrating additional metrics, such as adenomas per colonoscopy, as well as novel metrics, such as a size-stratified ADR, location-stratified ADR, or ADR stratified by polyp histology. The integration of these metrics into a single platform will also facilitate research to assess the prospective predictive capabilities of these metrics, compare them with the ADR, and set appropriate performance benchmarks. We further intend to study the integration of this platform into colonoscopy quality improvement and transparency programs to better characterize the impact of frequent, on-demand ADR feedback on colonoscopy performance.

Conclusion

We have demonstrated the feasibility and performance of an automated colonoscopy quality metrics reporting platform. Relying on customizable, text-based classifiers, we can identify true screening colonoscopies and adenomatous pathology in real time. This allows for the accurate calculation of the ADR and other key ASGE colonoscopy quality indicators. We anticipate that this tool will allow individual endoscopists and institutions to measure screening colonoscopy quality and the impact of quality improvement interventions with greater accuracy, frequency, and transparency.

What You Need to Know.

Background

Routine monitoring of the adenoma detection rate (ADR), cecal intubation rate, colonoscope withdrawal time, and bowel preparation quality is recommended by the American Society of Gastrointestinal Endoscopists. Attempts to streamline the process of ADR calculation have not seen widespread adoption, and many rely on either manual data entry or costly data science infrastructure.

Findings

Automated integration of pathology and endoscopy reporting coupled with text query-based classification rules allows for calculation of the ADR and other key colonoscopy quality metrics with accuracy comparable to a manual chart review.

Implications for Patient Care

This automated system for the calculation of colonoscopy quality data relies on widely available software and skills accessible to most clinicians, hopefully facilitating more widespread adoption of ADR reporting and subsequent research.

Abbreviations used in this paper:

ADR

adenoma detection rate

ASGE

American Society of Gastrointestinal Endoscopists

BBPS

Boston Bowel Preparation Score

CRC

colorectal cancer

HL7

Health Level Seven

PDR

polyp detection rate