Variable Reliability of Endoscopic Findings with White-Light and Narrow-Band Imaging for Patients with Suspected Eosinophilic Esophagitis

Abstract

Background & Aims

Endoscopic findings have been used to support a diagnosis of eosinophilic esophagitis (EoE) and to assess response to therapy, but their reliability is unknown. The aim of the study was to assess inter- and intra-observer reliability of endoscopic findings with white-light endoscopy and to assess changes in inter-observer reliability when narrow band imaging (NBI) was added to white light.

Methods

We collected data from 35 academic and 42 community adult gastroenterologists using 2 self-administered, online assessments of endoscopic images in patients with suspected EoE. First, gastroenterologists evaluated 35-single white light images. Next, they examined 35-paired images of the initial white-light image and its NBI counterpart. To assess intra-observer reliability, a second survey, to re-examine the single white light images, was performed ≥2 weeks later. Agreement was determined by calculating κ values for multiple observers.

Results

Among all gastroenterologists, inter-observer agreement was fair to good when white light was used to identify rings (κ = 0.56) and furrows (κ = 0.48). Inter-observer agreement was poor for identification of plaques (κ = 0.29) and for images with no findings (κ = 0.34). Levels of agreement did not change in an analysis stratified by practice setting or patient volume. Agreement did not improve when NBI images were added to white light images. Levels of intra-observer agreement varied greatly and in some cases were not greater than those expected by chance.

Conclusions

Using white-light endoscopy and NBI to analyze EoE, gastroenterologists identified rings and furrows with fair to good reliability, but did not reliably identify plaques or normal images. Intra-observer agreement varied. Endoscopic findings might not be reliable for supporting a diagnosis of EoE or for making treatment decisions.

Introduction

Eosinophilic esophagitis (EoE) is a poorly understood disease of the esophagus characterized by dysphagia and food impaction in adults.¹ The diagnosis of EoE has become increasingly common as a result of growing recognition and increased prevalence.^2-7 Guidelines recommend that a diagnosis of EoE is made when a patient presents with symptoms of esophageal dysfunction and esophageal biopsy demonstrates 15 or more eosinophils in a high power field (eos/hpf) in the absence of competing causes such as gastroesophageal reflux disease (GERD).¹

While not pathognomonic, EoE may present with rings, linear furrows, or white plaques on endoscopy.¹ The presence or absence of these endoscopic findings is used by a large proportion of gastroenterologists, in part, to make a diagnosis of EoE, to guide biopsy decisions and to assess a patient's response to therapy.^8-11 There are no studies evaluating whether endoscopists can reliably and accurately identify these findings. Additionally, in our clinical experience we have observed that narrow band imaging (NBI), a noninvasive optical technique that uses spectral filters to restrict transmitted wavelengths of light to 415 and 540 nm, often makes subtle findings in EoE more prominent.¹² The impact of NBI on an endoscopist's reliability and accuracy in detecting typical endoscopic finding in EoE has also not been described.

The objective of this study was to assess inter- and intra-observer reliability in the identification of three common endoscopic esophageal findings (rings, furrows, plaques) in patients with suspected eosinophilic esophagitis who were examined with standard white light endoscopy. We further sought to assess inter-observer reliability in the identification of endoscopic findings when NBI was used in addition to white light endoscopy, as well as to examine whether inter-observer reliability improved with the addition of NBI compared to white light alone.

Methods

This was a prospective study of academic and community gastroenterologists using two self-administered web-based online assessments. During the initial assessment, gastroenterologists evaluated endoscopic images under two conditions. First, they evaluated 35 single images obtained with standard white light endoscopy (Figure 1a). Next, they examined 35 paired images (from the same patients, but in a random order) of the initial white light image and its NBI counterpart, respectively (Figure 1b). During the second survey (completed at least two weeks later) they again examined the single white light images but in a different (randomly determined) order. The study was conducted between March 2010 and May 2010. The survey was piloted prior to the primary study to assess comprehensibility and comprehensiveness. This study was approved by the University of North Carolina Institutional Review Board. All participants consented to study participation.

Figure 1.

(A) Endoscopic image in white light showing linear furrows, white plaques, and subtle rings. (B) The corresponding narrowing band image of the same endoscopic findings.

Image Selection

We used endoscopic images previously obtained during the routine care of patients evaluated at University of North Carolina Hospitals with a clinical presentation concerning for eosinophilic esophagitis. Patients with food impaction, dysphagia, and heartburn refractory to proton-pump inhibition are typically evaluated in the University of North Carolina Center for Esophageal Diseases and Swallowing, and often undergo upper endoscopy under both white light and NBI (Olympus GIF-Q180, Olympus America Inc, Center Valley, PA). Because we were interested in the endoscopic signs of EoE, and because these are typically encountered prior to biopsy results being available, we felt it was most appropriate to assess reliability of endoscopic findings in patients suspected of having EoE. We reviewed all endoscopy reports of such patients from January 2009 through October 2009 and identified 60 patients who underwent upper endoscopy for suspicion of eosinophilic esophagitis and had images captured with both white light and NBI. Of these, images from 35 patients were felt to be of high quality with well-matched white light and NBI frames and were included in the survey (images were reviewed and selected by consensus of three of the co-authors: AFP, NJS, and ESD). All images were stripped of patient identifiers.

Image Evaluation

For each single white light endoscopic image displayed we asked: “Which of the following can you identify in the image above?” The respondent could select one or more of the following findings: rings, furrows, plaques. They could also select “none of the above”. The respondent could not advance to the next image until the question was answered, and once a question had been answered the respondent could not return to prior questions.

Two questions were asked when paired images were presented. The first was: “Which of the following can you identify in the images above?” The respondent could select one or more of the following findings: rings, furrows, plaques. They could also select “none of the above”. The second question was: “The findings in the two images above are: more prominent with white light, more prominent with blue light, or equivocal.” There was no time limit.

For all images, no specific clinical information was given about the patients from whom the images were obtained. At the beginning of the survey, there was a general statement of introduction that the responder would “be presented with a series of images taken from patients who underwent upper endoscopy for suspected EoE”.

Study Population and Assessment Administration

In order to assess agreement across a spectrum of practice, we surveyed two provider groups. The first was a sample of academic gastroenterologists who concentrate on esophagology primarily in adult patients from referral centers across the United States or international EoE experts. They were identified by their peer-reviewed publication record and/or national presentations related to research in esophageal disease, including EoE. The second group was a random sample of practicing North Carolina adult gastroenterologists identified through activity in state university-run continuing medical education programs.

Potential subjects were emailed an IRB approved invitation to participate and a link to the survey. The survey could only be accessed via the email link and could only be completed once. At least two weeks after completing the first survey, an invitation to complete the second survey was sent to all respondents who completed the first survey. All responses were anonymous.

Analysis

Respondent characteristics

Means and standard deviations are reported for continuous variables. Proportions are reported for categorical data. To compare responder characteristics between groups of interest (e.g., those in academic versus community practice), we used a 2-sample t-test or the Pearson's chi-square test, as appropriate. All tests of significance were two-tailed and p-values <0.05 were considered significant.

Inter-observer agreement

We estimated overall inter-observer agreement using kappa for multiple ratings per subject.¹³ Agreement was estimated for each of the possible endoscopic findings: rings, furrows, plaques and no findings. This analysis was performed first with the white light images alone, and then repeated with the paired white light and NBI images. We then determined the difference in these kappa estimates to examine whether inter-rater agreement improved with the addition of NBI compared to white light alone. Since the same subjects were evaluated under two different conditions (white light alone vs white light plus NBI), a jackknife analysis was used to estimate these standard errors.¹⁴ We also repeated the inter-rater agreement analysis for subgroups defined by practice setting (academic or community) and monthly volume of EoE patients (≥4 or <4). We also performed a post hoc analysis to estimate kappa for the first 17 white light with NBI images viewed, to explore the potential impact of rater fatigue on our findings.

To assess the relative strength of agreement, we used thresholds defined by Fleiss and colleagues.¹³ A kappa of 1.0 is perfect agreement, a kappa of greater than 0.75 is defined as excellent agreement, between 0.40 and 0.75 is fair to good agreement, less than 0.40 is poor agreement, and a kappa of 0 is agreement expected by chance alone.¹³

Intra-observer agreement

We estimated each individual gastroenterologist's agreement for each of the possible endoscopic findings on the first and second assessment for standard white light imaging using Cohen's kappa. We summarized the distributions of the observed kappas using histograms.

Sample size

We derived a novel method to approximate the power to detect a difference in inter-observer kappas under different assumptions about the true difference in kappas, the proportion of images with findings, the number of raters, and the number of images.¹⁵ In brief, we standardized the difference of two dependent kappa statistics and used Gaussian approximation for sample size calculation. The variance of the difference is overestimated so that the sample size estimate is conservative. We assumed that the providers' assignments to different findings (rings, furrows, plaques, none) were not skewed under the two conditions of white light and white light plus NBI. With this method, we approximated that there would be at least 90% power to detect a true 0.2 difference between kappas if at least 50 gastroenterologists evaluated 30 images.

Results

We distributed 190 assessments. A total of 61% (35 of 57) of academic and 32% (42 of 133) of community gastroenterologists participated in the study (Table 1). Of the academic gastroenterologists, 97% (34 of 35) were adult gastroenterologists. As expected, all of the academic gastroenterologists identified themselves as sub-specialized in esophageal disease or therapeutic endoscopy compared with 37% of the community gastroenterologists (p≤0.001). Academic gastroenterologists reported caring for a greater volume of EoE patients per month (mean 6 ± 8 vs. 2 ± 2; p=0.005) and reported greater familiarity with the EoE consensus guidelines (47% vs. 19%; p=0.01) compared with community gastroenterologists. In an analysis stratified by EoE patient volume, 35% (27 of 77) of gastroenterologists reported caring for 4 or more EoE patients per month. The higher volume group cared for a mean of 9 EoE patients per month compared with 1 EoE patient per month in the lower volume group (mean 9 ± 8 vs. 1 ± 1; p≤0.001). The higher volume group compared with the lower volume groups was more likely to be familiar with the EoE consensus guidelines (52% vs. 21%; p=0.007) and was more likely to have identified themselves as sub-specialized in esophageal disease or therapeutic endoscopy but not to a degree that reached statistical significance (78% vs. 60%; NS).

Table 1. Characteristics of survey respondents.

	All Gastroenterologists (n= 77)	Academic Gastroenterologists (n= 35)	Community Gastroenterologists (n=42)	p-value*
Years in practice (mean ± s.d.)§	16 ± 10	14 ± 10	17 ± 10	NS
Sub-specialized in esophageal disease and/or therapeutic endoscopy (%)§	67%	100%	37%	≤ 0.001
General GI practice (%)§	29%	3%	53%	≤ 0.001
“Very familiar” with Eosinophilic Esophagitis consensus guidelines (%)§	33%	47%	19%	0.01
EoE patients per month (mean ± s.d.)	4 ± 6	6 ± 8	2 ± 2	0.005

EoE, eosinophilic esophagitis; NS, not significant

^*Comparisons performed with 2 sample t-test or Pearson's chi-square test

^§Data missing for 5 participants

^**Guideline familiarity was self-assessed with the question “How familiar are you with the consensus guidelines for the diagnosis and treatment of EoE?” The respondents could chose one of three responses: 1) Very familiar, 2) Somewhat familiar or 3) Not familiar

Inter-observer agreement

In patients with suspected EoE, inter-observer agreement was fair to good under traditional white light for rings (k = 0.56) and furrows (k = 0.48) among all gastroenterologists (Table 2). Inter-observer agreement was poor for plaques (k = 0.29) and for the absence of endoscopic findings (k = 0.34). Agreement for these four endoscopic findings in white light did not change substantially in an analysis stratified by practice setting or patient volume (Table 2). Figure 2a is an example of an image from the survey with excellent agreement; figure 2b is an example of poor agreement.

Table 2. Inter-observer agreement for rings, furrows, plaques and no findings.

	All Gastroenterologists (n= 77)		Academic Gastroenterologists (n= 35)		Community Gastroenterologist (n=42)		Treat & Diagnosis ≥ 4 EoE Pts/Month (n= 27)		Treat & Diagnosis < 4 EoE Pts/Month (n=50)
	kappa	95% CI	kappa	95% CI	kappa	95% CI	kappa	95% CI	kappa	95% CI
Rings (white light only)	0.56	0.53, 0.58	0.54	0.51, 0.57	0.57	0.54, 0.59	0.53	0.50, 0.55	0.57	0.54, 0.60
Rings (white + NBI)	0.50	0.47, 0.52	0.50	0.47, 0.53	0.49	0.47, 0.51	0.52	0.49, 0.55	0.48	0.46, 0.51
Difference	-0.06	-0.08, -0.04	-0.04	-0.05, -0.02	-0.08	-0.10, -0.06	-0.01	-0.02, 0.01	-0.09	-0.11, -0.07
Furrows (white light only)	0.48	0.47, 0.50	0.53	0.51, 0.55	0.43	0.41, 0.45	0.47	0.44, 0.49	0.50	0.48, 0.51
Furrows (white + NBI)	0.49	0.47, 0.51	0.55	0.53, 0.57	0.45	0.43, 0.47	0.54	0.52, 0.56	0.47	0.45, 0.49
Difference	0.01	-0.01, 0.03	0.02	0.00, 0.04	0.01	-0.01, 0.03	0.07	0.05, 0.09	-0.01	-0.03, -0.01
Plaques (white light only)	0.29	0.28, 0.30	0.31	0.30, 0.33	0.29	0.27, 0.30	0.29	0.28, 0.31	0.29	0.27, 0.30
Plaques (white + NBI)	0.24	0.23, 0.26	0.23	0.22, 0.24	0.28	0.26, 0.30	0.24	0.23, 0.25	0.25	0.23, 0.27
Difference	-0.05	-0.06, -0.03	-0.08	-0.09, -0.07	-0.01	-0.02, 0.01	-0.05	-0.06, -0.04	-0.04	-0.05, -0.02
None (white light only)	0.34	0.29, 0.38	0.35	0.30, 0.40	0.33	0.29, 0.37	0.31	0.27, 0.35	0.35	0.30, 0.40
None (white + NBI)	0.23	0.21, 0.26	0.27	0.23, 0.30	0.21	0.19, 0.23	0.32	0.27, 0.36	0.19	0.17, 0.20
Difference	-0.10	-0.13, -.08	-0.08	-0.11, -0.06	-0.12	-0.14, -0.09	0.01	-0.01, 0.02	-0.17	-0.20, -0.13

Figure 2.

(A) Example of images for which there was good inter-observer agreement, with 88% of respondents identifying furrows, 9% no findings, 3% rings, and 1% plaques (endoscopic images in white light and NBI, respectively). (B) Example of images for which there was poor inter-observer agreement with 10% of respondents indentifying rings, 27% furrows, 43% plaques and 43% no findings (endoscopic images in white light and NBI, respectively).

Inter-observer agreement did not improve with the addition of NBI to white light. Instead, agreement for rings, plaques and no findings was significantly worse under NBI and white light compared with white light alone (Table 2). For example, the kappa for rings decreased from 0.56 to 0.50 when NBI was added, but remained about the same for furrows (k = 0.48 for white light and k=0.49 for white plus NBI). In a sensitivity analysis to assess observer fatigue, inter-observer agreement for rings (white light k = 0.54, NBI plus white light k = 0.45, difference = -0.09, 95% CI -0.19, 0.02) and plaques (white light k = 0.23, NBI plus white light k = 0.25, difference = 0.02, 95% CI -0.01, 0.04) was no different for white light alone compared with NBI and white light when the analysis was restricted to the initial 17 images. Agreement for furrows (white light k = 0.47, NBI plus white light k = 0.51, difference = 0.04, 95% CI 0.00, 0.08) was slightly better for NBI with white light compared with white light alone. Agreement for the absence of endoscopic findings (white light k = 0.23, NBI plus white light k = 0.17, difference = -0.06, 95% CI -0.11, -0.01) remained significantly worse.

Subjectively, gastroenterologists reported that endoscopic findings were more prominent with NBI in 50% of images, more prominent with white light only in 7% of images, and equivocal in 43% of the images.

Intra-observer agreement

Of the 77 gastroenterologists who responded to the first survey, 72 completed the entire survey and were sent the second survey. A total of 46% (33 of 72) of gastroenterologists completed the second survey. Intra-observer agreement for the majority of gastroenterologists was fair to good for all four endoscopic findings (furrows, 69%; rings, 53%; plaques, 50%; no findings, 34%). There was, however, wide variation in intra-observer agreement (Figure 3). For example, the range of kappas for the finding of esophageal rings was 0.2 – 0.9. This distribution did not change significantly in an analysis stratified by practice setting or patient volume.

Figure 3.

Histograms displaying the ranges of kappas for intra-observer reliability of EoE findings in white light for rings, furrows, plaques, and no findings.

Discussion

Both in clinical practice and research, findings of endoscopic mucosal abnormalities are used to support a diagnosis of EoE and to assess a response to treatment.^8-11 We performed a study to assess whether adult gastroenterologists can reliably identify endoscopic findings in suspected EoE. Our results were unexpected. We hypothesized that there would be excellent inter-observer and intra-observer reliability for identification of endoscopic findings of rings, linear furrows, and white plaques, and that the addition of NBI imaging would provide an added benefit. Instead, we found that our population of adult gastroenterologists identified rings and furrows with only fair to good reliability, and did not reliably identify plaques or the absence of findings. NBI did not improve endoscopic recognition of findings in EoE. Individual gastroenterologist's observations appeared to be largely consistent over time but demonstrated a range of values, from chance-alone to excellent intra-observer agreement.

While endoscopic findings of EoE are not essential for the diagnosis to be made, we have recently found that a large proportion of gastroenterologists consider the presence of these findings necessary to support the diagnosis.¹¹ Given their use in the diagnosis and management of EoE, many have assumed that “classic” findings, such as rings, furrows, and plaques, are obvious to endoscopists. This supposition is not supported by our data. Therefore, the practice of using these stigmata in clinical management deserves re-examination.

Our results have important implications. Several recent clinical trials of therapy in patients with EoE have followed endoscopic findings as a secondary outcome to monitor response to treatment.^8-10 The endoscopic abnormalities assessed included: white exudates, red furrows, corrugated rings, solitary rings, crêpe paper sign and severe stenosis. Prior investigators have suggested that exudates and furrows may “represent reliable endoscopic indicators of active eosinophilic inflammation”.⁸ We would caution, based on the results of our study, that the detection of endoscopic findings in EoE is subjective and prone to error. Even the use of a single endoscopist performing the majority of assessments of endoscopic findings may not provide adequate intra-observer reliability. Based on our results, it appears that these findings may not have adequate operating characteristics on which to base treatment decisions.

We also hypothesized that the addition of NBI would improve inter-observer agreement for all endoscopic findings of EoE based on our clinic experience and a small case series,¹⁶ as well as a prior report assessing the value of chromoendoscopy in EoE.¹⁷ Instead, we found that NBI actually decreased agreement regarding recognition of endoscopic findings in patients with suspected in EoE. We considered the impact of responder fatigue given that a total of 70 images or image sets were included in the survey. However, fatigue does not appear to be the explanation. There is conflicting evidence that NBI in addition to white light improves the reliability of detecting endoscopic findings in esophageal diseases. In a study of mucosal morphology in Barrett's esophagus, Curvers and colleagues found that NBI in addition to high-resolution white light endoscopy did not improve inter-observer agreement.¹⁸ In contrast, in a study of erosive esophagitis, Lee and colleagues found that NBI in addition to white light did improve inter-observer agreement, from an overall kappa of 0.45 to 0.62.¹⁹

Several strengths of our study deserve mention. This was a prospectively conducted study with extensive planning both for image selection and data analysis, as well as a large sample size of images and gastroenterologists. The results were consistent across strata of practice setting, GI subspecialty, and EoE case volume. The sample size and analysis were appropriate to the question, and considered issues of inter- and intra-subject variability and rater fatigue.

Several limitations also exist. Unlike real life endoscopy, we presented gastroenterologists with still images, and reliability may have been significantly different had we used video footage or live endoscopy. Plaques compared to rings and furrows might possibly be harder to assess on still images where the examiner does not have the ability to wash debris or bubbles, or maneuver more closely to get a better look. However, we made every effort to pick clear, high resolution, and illustrative images that were well matched to an NBI counterpart. Because our images were selected retrospectively, we could not include video but future study designs could readdress this issue by using video clips or live endoscopic procedures. Also, all gastroenterologists who participated in the study were informed about the objective of the study. Awareness of this objective is known to impact performance for the better and as a result we may have over-estimated reliability.²⁰ If this is the case, then our findings of relatively poor reliability may actually over estimate ‘real world’ agreement.

Finally, this study was not designed to assess the sensitivity and specificity of endoscopic findings of EoE. Prior studies have addressed the validity of endoscopic findings in EoE and found that the classic endoscopic findings of EoE are not necessarily specific.²¹ We felt that it would be difficult to create a reproducible endoscopic definition for each of the endoscopic findings of EoE, and our data supported this contention. Even among experts in esophageal diseases and EoE, there was no clear consensus (as measured by a kappa > 0.75 indicating excellent agreement) about which images had rings, furrows, or plaques, and which images were normal. Instead, our study highlights the global subjectivity of identifying endoscopic mucosal abnormalities in EoE, even among experts in the field. Our question was fundamental – are we all seeing the same things, and does advanced imaging help us better see the same things? The answers to both questions are appear to be no.

In conclusion, adult gastroenterologists identified rings and furrows with fair to good inter-observer reliability, but did not reliably identify plaques or no findings. NBI did not improve endoscopic recognition. Intra-observer agreement was highly variable. Given these results, endoscopic findings in suspected EoE may not be reliable markers on which to base diagnostic or treatment decisions. Instead, the entire clinical and pathologic picture should be considered to make a diagnosis of EoE, as recommended by the current guidelines.

Abbreviations

EoE

eosinophilic esophagitis

GERD

gastroesophageal reflux disease

eos/hpf

high power field

NBI

narrow band imaging