The Impact of Exclusion Criteria on a Physician’s Adenoma Detection Rate

Abstract

Background

The adenoma detection rate (ADR) is a validated and widely used measure of colonoscopy quality. There is uncertainty in the published literature on which colonoscopy examinations should be excluded when measuring a physician’s ADR.

Objective

To examine the impact of varying the colonoscopy exclusion criteria on physician ADR.

Design

We applied different exclusion criteria used in 30 prior studies to a dataset of endoscopy and pathology reports. Under each exclusion criterion, we calculated physician ADR.

Setting

A private practice colonoscopy center affiliated with the University of Illinois College of Medicine.

Patients

Data on 20,040 colonoscopy examinations and associated pathology notes performed by 11 gastroenterologists from July 2009 to May 2013.

Main Outcome Measurements

ADR across all colonoscopy exainations, each physician’s ADR, and ADR ranking.

Results

There were 28 different exclusion criteria used when measuring ADR. Each study used a different combination of these exclusion criteria. The fraction of all colonoscopy examinations in the dataset excluded under these combinations of exclusion criteria ranged from 0 to 93.1%. The mean ADR across all colonoscopy examination was 35.9%. The change in mean ADR after applying the 28 exclusion criteria ranged from −4.6 to +3.1 percentage points. However, the exclusion criteria impacted each physician’s ADR relatively equally, and therefore physicians’ rankings via ADR were stable.

Limitations

ADR assessment was limited to a single private endoscopy center.

Conclusions

There is wide variation in the exclusion criteria used when measuring ADR. Although these exclusion criteria can impact overall ADR, the relative rankings of physicians by ADR were stable. A consensus definition on which exclusion criteria are applied when measuring ADR is needed.

In 2014, almost 140,000 Americans will be diagnosed with colorectal cancer (CRC), the second-leading cause of cancer mortality in the United States.¹ Effective screening can prevent a large fraction of CRC cases. Colonoscopy is the most widely used screening modality in the US,² but the effectiveness of colonoscopy is limited by variation in physician quality. Prior research has observed a two- to three-fold discrepancy in the adenoma detection rate (ADR) across physicians and an inverse relationship between ADR and the incidence of subsequent, interval CRC.^3,4

ADR has become the primary measure of colonoscopy quality. Clinical experts⁵ and the American Society of Gastrointestinal Endoscopy (ASGE)⁶ have recommended that physicians regularly measure ADR. The Centers for Medicare and Medicaid Services⁷ incorporated ADR, as measured by Gastrointestinal Quality Improvement Consortium (GIQuIC),⁸ as a quality measure for the 2014 Physician Quality Reporting System. Based on expert opinion, the ASGE/American College of Gastroenterology (ACG) Taskforce on Quality in Endoscopy proposed that the minimum acceptable ADR is ≥ 25% in men and 15% in women healthy asymptomatic patients undergoing screening colonoscopy examinations.⁶

When measuring a physician’s ADR, there is notable variation in which colonoscopy examinations are included and excluded. Because the proposed minimum standards for ADR focus on first-time screening ADR, some studies exclude non-screening colonoscopy examinations such as surveillance studies, diagnostic colonoscopy examinations (for example, those done for gastrointestinal bleeding), or colonoscopy examinations performed on patients whose age is outside the typical range for CRC screening.^6,9,10 Other studies omit incomplete cases (for example, where the preparation is inadequate).^10–12 Exclusion criteria have also been employed to address differences in patient populations across physicians. For example, some physicians specialize in care for inflammatory bowel disease and some studies exclude colonoscopy examinations of patients with inflammatory bowel disease to allow for a more homogenous comparison across physicians.^13–15

The aim of this study is to explore the effect of various exclusion criteria on physician ADR. We first surveyed the literature to identify previously used exclusion criteria. We then applied each of these exclusion criteria to a dataset of approximately twenty thousand colonoscopy reports from 11 gastroenterologists in a private endoscopy center and assessed (1) what fraction of colonoscopy examinations were excluded, (2) the change in overall ADR across all physicians, and (3) the relative physician ranking by ADR.

METHODS

Identifying Different Definitions of Denominator for ADR

Prior work has used various combinations or sets of exclusion criteria to evaluate physician ADR. To identify commonly used exclusion criteria, we identified a convenience sample of prior studies; we did not feel it was critical to identify every study that uses ADR because our goal was to illustrate the impact of common exclusion criteria on ADR. Given their importance for quality measurement, we did include the ADR definitions used by GIQuIC¹⁶ and the American Gastroenterological Association.¹⁷ In total, we examined the exclusion criteria used in 30 prior studies.

We categorized the exclusion criteria by age, prior colonoscopy, incomplete colonoscopy, and indication. Indications were categorized into routine screening, high-risk screening (defined as family history, history of polyposis), surveillance procedures, and diagnostic procedures (defined as cases where patient had any symptoms reported, including cases where screening or surveillance was another indication).

Setting

We applied the various exclusion criteria employed in the literature to data from the Central Illinois Endoscopy Center (CIE). CIE is a private endoscopy center with 11 gastroenterologists in Peoria, Illinois and is affiliated with the University of Illinois College of Medicine at Peoria. All 11 gastroenterologists are generalists who do not subspecialize. We obtained all 20,040 colonoscopy and associated pathology reports from colonoscopy examinations performed between July 2009 and May 2013 at the endoscopy center. July 2009 was the earliest date available because this was when a new electronic health record was introduced. The reports were a combination of structured data and free text. Inpatient colonoscopy examinations were not included because they were not captured in the electronic health record.

Abstracting Relevant Information from Colonoscopy and Pathology Reports

Relevant data from the reports were abstracted using a previously developed Natural Language Processing (NLP) software application.¹⁸ Details of the development and testing of this tool are reported elsewhere.^18,19 In brief, NLP is a field of computer science in which a computer “reads” unstructured text to extract relevant data. The accuracy of the NLP program was confirmed by comparing ADR in 453 colonoscopy and associated pathology reports which were analyzed both by the NLP program and manually abstracted by physicians.¹⁸

The NLP program extracted the following variables from each colonoscopy report: patient age, family history of colon cancer, documentation of whether the cecum was reached, documentation of whether there was a prior colonoscopy and the timing of any prior colonoscopy, and indication for procedure (up to three). From the pathology reports, the NLP program identified whether an adenoma was reported.

Analyses

For each exclusion criterion, we calculated the fraction of cases among the CIE reports that would be excluded and the average ADR among the excluded cases and the remaining cases. We assessed what fraction of colonoscopy examinations were excluded because it is important from a statistical perspective to have a sufficient quantity of colonoscopy examinations performed in order to accurately assess a physician’s ADR. We also calculated each physician’s ADR under each exclusion criterion. Finally, we ranked the 11 participating gastroenterologists according to their ADRs before and after applying each criterion.

RESULTS

Across the 30 studies, 28 different exclusion criteria were applied with some of the most common being surveillance procedures (63% of studies used this exclusion criteria), high-risk screening (57%), age less than 50 (50%), prior colonoscopy (47%), inflammatory bowel disease (40%), and incomplete colonoscopy (40%), (Table 1). The fraction of colonoscopy examinations excluded under each of the 28 criteria varied widely – surveillance (22% of colonoscopy examinations excluded), any prior colonoscopy (18%), colonoscopy examinations in patients younger than 50 years (13%), and colonoscopy exainations in patients older than 75 years (10%), (Table 2).

Table 1.

Exclusion criteria employed in Previous Studies

Excluded subjects based on this criteria	Number of studies that use this exclusion criteria n (%)”	GIQuIC	AGA Registry	Raju	DenisA	Kahi (2013a)	Lee (2011)	SchoenfeldD	Boroff	Imperiale (2000)	Barclay	Jiang	Leffler	Gohel	Betes	Coe	Imperiale (2009)	Williams	Robertson	Lieberman	Nelson	Munson	Anderson	Chen	Sanchez	Millan	Kahi (2013b)	Corley	LeeE	Shaukat	KaminskiF	Hernandez
Age	19 (63.33)	X	X			X	X	X		X		X		X	X		X	X		X	X		X				X	X	X	X	X
<18H	1 (3.33)						x
<40	3 (10.00)							x							x																x
<50	15 (50.00)	x	x			x				x		x		x			x	x		x	x		x				x	x		x	x
>75B	7 (23.33)		x					x				x		x						x	x								x
Prior colonoscopy	14 (46.67)		X	X		X		X	X	X	X		X	X	X		X			X	X									X
Any prior colonoscopy	9 (30.00)			x					x	x	x		x	x			x			x										x
If prior colonoscopyK
<5yrs	2 (6.67)					x									x
<10yrsC	4 (13.33)		x					x												x	x
Incomplete colonoscopy	12 (40.00)	X					X							X	X	X		X	X	X		X	X		X						X
Did not reach cecum	10 (33.33)						x							x	x	x		x	x	x		x	x		x
Anatomy was not intactI	1 (3.33)															x
Prep was inadequate	7 (23.33)	x												x		x						x	x		x						x
Indication
High-risk screening	17 (56.67)	X		X		X	X	X	X	X				X	X	X		X	X			X	X				X			X		X
Family hist of CRC/polyps	5 (16.67)													x	x							x	x				x
Familial syndrome/ polyposisI	10 (33.33)			x		x	x	x						x		x		x	x				x				x
Surveillance				X		X	X	X		X	X			X	X		X	X	X	X	X	X	X	X	X		X					X
Personal historyJ	19 (63.33)			x		x	x	x		x	x			x	x		x	x	x	x	x	x	x	x	x		x					x
Diagnostic	18 (60.00)							X		X	X			X	X	X	X	X		X	X	X	X		X	X	X		X	X		X
Inflammatory bowel disease	12 (40.00)			x		x	x	x		x				x		x		x				x	x	x			x
BleedingG	10 (33.33)							x		x				x	x	x				x	x	x	x						x
Pain	6 (20.00)									x				x	x					x	x	x
Anemia	4 (13.33)							x							x							x	x
Change in bowel habits	5 (16.67)									x					x					x	x	x
Symptoms of colon disorders (ex. colitis)	4 (13.33)										x					x				x	x
Weight loss	3 (10.00)							x							x							x
Abdominal radiationI	1 (3.33)														x

^ADenis et al did not include any exclusion criteria in this study as it was the goal of the study to identify better indicators for colonoscopy quality than their institution was currently using.

^BSchoenfeld et al established the upper age limit at 79 years

^CSchoenfeld et al excluded patients that had a positive colonoscopy study within the prior 10 from the starting point of their study

^DThe prior colonoscopy exclusion criteria applies to screening colonoscopies, not to surveillance colonoscopies included in this study

^EThe age limit in the study was between 60–74 years of age.

^FUpper limit for this study is 66 years of age

^GExclusion criteria of bleeding for diagnostic colonoscopies includes rectal bleeding, hematochezia, and a positive FOBT

^HWe did not use this criterion in our analyses because there was only one colonoscopy in our dataset performed on a patient <18

^IWe could not identify and exclude these cases with sufficient accuracy using the data we abstract from each colonoscopy report. Therefore we did not include them in the rest of the analyses

^JPersonal history of polyposis/polyps/prior polypectomy/previous colon resection/CRC

^KSome studies excluded patients with prior colonoscopy, but did not designate a cutoff on how long ago that colonoscopy was performed

Table 2.

Effect of applying exclusion criteria on the ADR in the Central Illinois Data set

Excluded subjects based on this criteria	Colonoscopy examinations in dataset excluded N (%)	ADR among colonoscopy examinations excluded (95% confidence interval)	ADR among remaining colonoscopy exainations (difference in ADR compared to ADR when no exclusion criteria applied, 95% CI of the difference in ADR)
No Exclusion Criteria	0 (0)	N/A	35.9 (0, −0.9 to +0.9)
Age (excluded patients who were)
<40	1158 (5.7)	10.9 (9.1–12.8)	37.4 (+1.5, +0.5 to +2.5)
<50	2668 (13.3)	15.4 (14.0–16.8)	39.0 (+3.1, +2.1 to +4.1)
>75	2004 (10)	70.8 (68.7–72.8)	32.0 (−3.9, −3.1 to −5.0)
Prior colonoscopyA
Any prior colonoscopy	3518 (17.6)	50.7 (49.0–52.4)	32.7 (−3.2, −2.2 to −4.2)
Incomplete colonoscopy
Did not reach cecum	532 (2.7)	21.2 (17.8–25.0)	36.3 (+0.4, −0.6 to +1.4)
Prep was inadequate	852 (4.3)	28.2 (25.2–31.3)	36.2 (+0.3, +0.2 to 2.2)
Indication
high-risk screening
family hist of CRC/polyps	1725 (8.6)	37.8 (35.5–40.1)	35.7 (−0.2, −1.2 to +0.8)
surveillanceB	4352 (21.7)	52.6 (51.1–54.1)	31.3 (−4.6, −3.6 to −5.6)
diagnostic
inflammatory bowel disease	184 (0.9)	13.0 (8.5–18.8)	36.1 (+0.2, −0.7 to +1.1)
bleedingC	1798 (9.0)	27.0 (24.9–29.1)	36.8 (+0.9, −0.6 to +1.9)
pain	1026 (5.1)	20.5 (18.3–23.1)	36.7 (+0.8, −0.2 to +1.8)
anemia	637 (3.2)	30.9 (27.4–34.7)	36.1 (+0.2, −0.8 to +1.2)
change in bowel habits	735 (3.7)	23.8 (20.8–27.1)	36.4 (+0.5, −0.5 to 1.5)
weight loss	176 (0.9)	29.0 (22.4–37.3)	36.0 (+0.1, −0.8 to +1.0)

^APrior colonoscopy includes any prior colonoscopy, colonoscopy greater than 5 years in past, colonoscopy greater than 10 years in past

^BSurveillance includes personal history of polyposis, prior polypectomy, previous colon resection

^CExclusion criteria of bleeding includes rectal bleeding, hematochezia, and a positive FOBT

Across the 30 studies, there were 30 unique combinations of the 28 different exclusion criteria (Table 1). In one study, no exclusion criteria were applied.²⁰ If we applied the most stringent combination of exclusion criteria used in a prior study, 93.1% (95% confidence interval (CI), 92.7%–93.4%) of colonoscopy examinations in our data were excluded.²¹ In this study colonoscopies were excluded for persons below 50 and above 75 years of age, any prior colonoscopy; incomplete colonoscopies (i.e., did not reach cecum or had inadequate preparation), any family history of CRC/polyps; familial syndrome/polyposis, patients with a personal history of polyps or CRC, or patients with symptoms (e.g., GI pain, abdominal bleeding) or inflammatory bowel disease.

The overall ADR when no exclusion criteria were applied was 35.9% (95% CI, 35.2%–36.7%), and the ADR among cases excluded under each criterion varied (Table 2). The ADR for the 1,158 colonoscopy examinations excluded due to an age <40 was 10.9% (95% CI, 9.1%–12.8%). The ADR for colonoscopy examinations excluded for an age <50 and age >75 was 15.4% (95% CI, 14.0%–16.8%), and 70.8% (95% CI, 68.7%–72.8%), respectively. The ADR among surveillance colonoscopy examinations was 52.6% (95% CI, 51.1%–54.1%).

We compared the overall ADR when there were no exclusion criteria vs. the ADR of the colonoscopy examinations remaining when we applied various individual exclusion criteria (Table 2). The change in ADR varied: if we excluded procedures in which the physician did not reach the cecum, the overall ADR was +0.4 percentage points (95% CI, −0.6 to +1.4 percentage points) higher among remaining colonoscopies than when no exclusion criteria were applied; age < 50 [+3.1 (+2.1 to +4.1)], > 75 [−3.9 (−3.1 to −5.0)], prior colonoscopy [−3.2 (−2.2 to −4.2)], family history [−0.2 (−1.2 to +0.8)], surveillance colonoscopy examinations [−4.6 (−3.6 to −5.6)] and inflammatory bowel disease [+0.2 (−0.7 to +1.1)]. Variation across providers, as measured by coefficient of variation, did not change appreciably across the different exclusion criteria.

We calculated each physician’s ADR and ranking under each of the exclusion criteria. Although the exclusion criteria did impact a physician’s ADR, the physician ADRs moved in concert (Figure 1) and therefore the physician ranking by ADR was not substantially affected (Figure 2). The rankings across the 11 physicians were generally constant regardless of ADR denominator definition. Only two exclusion criteria, family history and inflammatory bowel disease, caused a single physician’s ADR ranking to change by more than one ranking, from fourth to sixth place, and from third to fifth place, respectively.

Figure 1.

Figure 2.

Discussion

Using data from over 20,000 colonoscopy examinations performed by 11 gastroenterologists, we evaluated the effect of applying various combinations of exclusion criteria on the overall ADR, the physician’s individual ADR, and a physician’s relative ADR. We found the individual exclusion criteria excluded up to approximately 20% of colonoscopy examinations. If we applied the combination of exclusion criteria used in the most stringent study, more than 90% of colonoscopy examinations in our data were excluded. Applying exclusion criteria did result in notable shifts in mean ADR. However, our analysis shows the exclusion criteria affected physicians relatively equally, and thus, the relative ranking in ADR was largely unchanged.

Given the growing use of ADR in the gastroenterology community for quality monitoring, credentialing, and reimbursement, and the need for generalizability across research studies, our results highlight the necessity of a consensus definition of ADR with a consistent set of exclusion criteria. This would facilitate comparison of the performance of gastroenterologists to others. To increase acceptance in the gastroenterology community, this consensus ADR definition should be developed with endorsement from specialty organizations and quality measurement organizations such as the National Quality Forum. Our results can help inform what combination of exclusion criteria might be used in this consensus standard.

Because physicians care for different patient populations, an ADR definition should attempt to address these differences to ensure an “apples to apples” comparison. Exclusion criteria have been used to create a more homogenous patient population to facilitate comparison. Although this goal is important, we recommend using relatively few exclusion criteria and instead using other methods to address differences in patient population. First, our data suggest that varying the exclusion criteria does not appreciably change physician ADR ranking. Second, any differences in patient population can be addressed via statistical risk adjustment instead of exclusion criteria. Risk adjustment is the norm in quality measurement in other areas of medicine.^22–27 Third, using extensive exclusion criteria substantially reduces the number of colonoscopy reports used to generate a physician’s ADR. This results in wider confidence intervals around each physician’s ADR estimate, thus sacrificing precision. Fourth, limiting exclusion criteria reduces the potential for gaming, a problem that has existed in prior physician quality efforts.²⁸ For example, when the New York State Coronary Artery Bypass Graft Surgery mortality public reporting initiative was implemented, physicians changed their coding of comorbidities.^29,30 Regarding colonoscopy, a number of assessment criteria are subjective, such as the assessment of preparation quality²⁸ and even procedure indication, where symptoms can be emphasized as a reason for the procedure or not. The use of fewer exclusion criteria reduces the chance for subjective manipulation of which colonoscopy examinations are included or excluded.

Although we advocate for relatively few exclusion criteria, there may be a role for a small number of exclusion criteria. The goal of the ADR is to capture the effectiveness of physicians in identifying and removing cancer precursors. Thus, on theoretical grounds, it may be appropriate to exclude colonoscopy examinations that are clearly not related to cancer prevention or are conducted for distinctly different reasons, such those performed in young patients who are being evaluated for evidence of inflammatory bowel disease.

There are several limitations of this study. Our analysis is limited to 11 general physicians from a private endoscopy group. At an academic medical center with gastroenterologists who subspecialize (for example, focus on inflammatory bowel disease) the relative ranking of physicians by ADR could change more under some exclusion criteria. Additional exclusion criteria which we did not identify may need to be evaluated. Despite prior research validating its performance, the use of NLP software to identify the characteristics of a colonoscopy may have created some bias in our findings, though it is reassuring that other groups have developed and tested NLP programs for the purposes of colonoscopy quality.^31,32 We did not use a standardized reporting form for the colonoscopies and instead depended on the data provided in the colonoscopy report. Therefore key elements (e.g., adequacy of colon preparation) may be missing from some reports.

In summary, our data show that in a general GI practice, exclusion criteria can substantially reduce the number of colonoscopy reports included in the analysis of ADR. However, applying these criteria does not impact relative physician ranking. As ADR becomes a commonly used quality metric, our findings emphasize the need for a common set of exclusion criteria.

Acronyms

ADR

Adenoma Detection Rate

CRC

Colorectal Cancer

GIQuIC

Gastrointestinal Quality Improvement Consortium

CIE

Central Illinois Endoscopy Center

NLP

Natural Language Processing