Interobserver Agreement for the Detection of Barrett's Esophagus with Optical Frequency Domain Imaging

Abstract

Background

Optical frequency domain imaging (OFDI) is a second-generation form of optical coherence tomography (OCT) providing comprehensive cross-sectional views of the distal esophagus at a resolution of ∼7 μm.

Aims

Using validated OCT criteria for squamous mucosa, gastric cardia mucosa, and Barrett's Esophagus (BE), the objective of this study was to determine the inter- and intra-observer agreements by a large number of OFDI readers for differentiating these tissues.

Methods

OFDI images were obtained from 9 subjects undergoing screening and surveillance for BE. 64 OFDI image regions of interest were randomly selected for review. A training set of 19 images was compiled distinguishing squamous mucosa from gastric cardia and BE using previously validated OCT criteria. The 10 readers then interpreted images in a test set of 45 different images of squamous mucosa (n=15), gastric cardia (n=15), or BE (n=15). Interobserver agreement differentiating the three tissue types and BE vs. non-BE mucosa was determined using multi-rater Fleiss's κ value. The images were later randomized and four readers repeated the test three weeks later to assess intraobserver reliability.

Results

All 10 readers showed excellent agreement for the differentiation of BE vs. non-BE mucosa (κ= 0.811 p<0.0001) and for differentiating BE vs. gastric cardia vs. squamous mucosa (κ=0.866, p<0.0001). For the 4 readers who repeated the test, the median intraobserver agreement (BE vs. non-BE) was high (κ=0.975, IQR: 0.94, 1.0).

Conclusions

Trained readers have a high interobserver agreement for differentiating BE, squamous, and gastric cardia mucosa using OFDI.

Introduction

Barrett's esophagus (BE) is a premalignant condition of the distal esophagus associated with long-standing gastroesophageal reflux disease (GERD) [1]. The presence of Barrett's esophagus or intestinal metaplasia in the esophagus is associated with a 30-fold increased risk over the general population of developing esophageal adenocarcinoma (EAC), a cancer with a rising incidence in Western countries and an overall 5-year survival rate of 19% [2-6]. As the only known precursor lesion for EAC, Barrett's esophagus may be detected and eradicated to prevent progression to esophageal adenocarcinoma [7].

While screening to detect Barrett's metaplasia can decrease morbidity and mortality, current strategies for screening and surveillance of Barrett's esophagus are limited. Studies suggest that most cases of EAC still arise de novo with a recent Danish cohort study finding that BE-associated EAC accounted for only 7.6% of all incident EAC between 1992 and 2009 [8-11]. Furthermore, the reported annual incidence of EAC in BE has varied considerably from 0.19% per year in a large population-based cohort study to a pooled incidence rate of 0.65% per year in a 2010 meta-analysis [8,11]. These data suggest that our current screening practices are not identifying all patients at risk for developing BE, and thus EAC.

The diagnosis of BE relies upon endoscopic and histopathologic findings. The International Working Group for the Classification of Esophagitis created the Prague classification to guide endoscopic recognition of BE and to improve diagnostic yield of biopsies for potential BE [12]. However, this method is not only costly and invasive, but also subject to sampling error, as biopsies are required to confirm the diagnosis. According to a study by Harrison et al., if eight biopsies were obtained per procedure, the likelihood of identifying intestinal metaplasia would be 67.9%. If only four biopsies were obtained per procedure, that number would be much lower at 34.7% [13]. Furthermore, biopsies from areas with esophagitis have a lower likelihood of identifying Barrett's esophagus by 12% [14-15].

OCT is an optical imaging modality that can produce high-resolution (7 μm axial resolution) cross-sectional images of the GI mucosa using near-infrared light. A novel form of OCT called optical frequency domain imaging (OFDI) uses a balloon catheter which centers helically scanning optics to acquire three-dimensional volumetric, microscopic views of the esophagus faster (100 fold faster) than standard OCT with comparable image quality [16-17]. With OFDI, a 6 cm segment of esophagus can be imaged in 2 minutes [18-19]. Based on the scattering of light from different structures within the esophageal wall, an image can be constructed by OFDI, revealing microscopic features such as crypt and glands and mucosal layers [20-21]. This technology can overcome some of the limitations of biopsy sampling for BE by providing high resolution, volumetric imaging of the entire distal esophagus.

Prior studies have shown that OCT can distinguish between intestinal metaplasia and non-metaplastic tissue at the gastroesophageal junction with 85% sensitivity and 95% specificity [22]. Diagnostic criteria for squamous mucosa, gastric cardia mucosa, and BE have been described in prior publications [22]. However, the interobserver agreement for diagnosing BE with these criteria has not been evaluated. The goal of this study was to assess whether gastroenterologists with limited OFDI experience could be trained to diagnose intestinal metaplasia on OFDI-acquired images of the distal esophagus and to determine inter- and intra-observer agreement between a large number of OFDI readers for the detection of BE.

Patients/materials and methods

We evaluated the interobserver agreement of ten readers in diagnosing BE from OFDI images obtained from the distal esophagus. The ten readers, with a full range of OFDI experience, reviewed the training set and interpreted images subsequently in a test set in a blinded fashion. None of the ten readers had prior formal training. However, 4 readers have had significant experience with OFDI technology. “Significant experience” was defined as having seen more than 20 esophageal OFDI images. Three of the four experienced readers were not clinicians. 6 readers included in the study were gastroenterologists (5) and a pathologist (1) with no formal training in esophageal OFDI image interpretation. The 4 expert readers were included in the study to serve as a reference for comparison with the inexperienced readers who had limited OFDI image exposure.

OFDI imaging system

OFDI uses interferometry to produce depth-resolved images of tissue microstructure by measuring the amplitude and echo time delay of light backscattered from a sample [23]. The OFDI system consists of a wavelength swept source centered at 1310 nm with a tuning range of 143 nm resulting in a 7 μm axial resolution in tissue [18]. Cross-sectional esophageal images were acquired at a frame rate of 10 frames/second with 4096 A-lines per frame. A 25 mm diameter, balloon-centering catheter was used in this study with the imaging parameters described, resulting in a longitudinal image-image spacing of 50 um. Three-dimensional images of a 6 cm segment of distal esophagus were captured in 2 minutes.

Endoscopy and study subject recruitment

The protocol was reviewed and approved by the institutional review board at Massachusetts General Hospital. Recruited subjects included patients who were undergoing routine upper endoscopy. Data for this study were collected between 2007 and 2009. Subjects underwent routine outpatient upper endoscopy with conscious sedation. A standard gastroscope (model EG 3470K; Pentax Medical, Tokyo, Japan) with a 3.8 mm instrument channel was used.

OFDI Imaging

Written informed consent was obtained prior to the procedure. After adequate sedation, an upper endoscopy was performed and the squamocolumnar junction was identified with standard high definition white light imaging. Then, a guidewire was placed through the auxillary channel of the endoscope. The endoscope was removed and the OFDI balloon was inserted to the required distance over the guidewire to the GE junction. The balloon was then inflated prior to imaging. Once the imaging probe was in place, spiral OFDI imaging data were obtained throughout the distal 6 cm of the esophagus. The total added time from insertion and inflation of the OFDI balloon was approximately 10 minutes. After the OFDI image acquisition, biopsies of the EG junction with standard EGD were performed.

OFDI image analysis

Images from 9 subjects were chosen for inclusion in the study. Characteristics of these subjects are included in Table 1. These image sets were considered to be of highest quality for image interpretation to determine intestinal metaplasia, gastric cardia and squamous mucosa. 19 image frames were used to assemble a training set of images and 45 images were used in a test set. Figure 1 shows the criteria that were described in the training set to distinguish between Barrett's esophagus, gastric cardia and squamous mucosa. In a previous paper by Evans et al., these criteria for intestinal metaplasia were validated using histopathology from corresponding biopsy sites as the gold standard [22]. Ten readers reviewed the training set of images and performed the test within one week. Four readers repeated the test three weeks later to assess intraobserver reliability. Three of the four readers did have significant experience with the technology but were not clinicians.

Table 1.

Subject Characteristics

Subject Characteristics (n=9)
Median Age	62.9 (IQR: 60,76.6)
Sex (% male)	78%
BMI (kg/m2)	30.6 (27.3,31)
Indication for Procedure:
Reflux symptom	2/9 (23%)
Barrett's Surveillance	7/9 (79%)
Dysphagia	1/9 (11%)
Endoscopic suspicion of BE	6/9 (67%)
Histopathology confirms BE	5/9 (56%)
Reflux Esophagitis	3/9 (33%)

Figure 1. Squamous, Gastric Cardia and Barrett's Metaplasia via OFDI.

1. Squamous Mucosa Layered structure No glands in the epithelial layer Squamous epithelium is homogeneous
2. Gastric Cardia Mucosa (at least 2) Vertical pit architecture Highly reflective surface (red arrow) Relatively poor image penetration Broad regular foveolar region Rugae Blue arrow is the balloon wall
3. Barrett's Metaplasia (at least 2) Loss of layered or vertical pit and crypt architecture Irregular mucosal surface Heterogeneous scattering

Each reader was asked to make a diagnosis of intestinal metaplasia, gastric cardia or squamous mucosa for the 45 images in the test set, which was compared against the gold standard of an expert OFDI reader's interpretation (G.T) for each image. As 6 cm three-dimensional data from the distal esophagus were obtained and biopsies were randomly performed, precise histopathologic correlation per image could not be performed.

Statistical Analysis

Interobserver agreement for differentiating squamous mucosa vs. cardia mucosa vs. intestinal metaplasia, and BE vs. non-BE mucosa was determined using multi-rater Fleiss'sκvalue in SPSS (Version 20.0.0) The kappa statistic was used to assess the average agreement across all pairs of readers. A two-sided p-value of < 0.05 was used to determine a statistically significant difference.

Results

A total of 45 images were included in the test set from nine subjects. 15 squamous mucosa, 15 gastric cardia and 15 intestinal metaplasia were included in the test set, confirmed by the interpretation of an expert OFDI reader. No adverse events were reported upon acquisition of the images.

Overall, all ten readers had excellent agreement for the OFDI differentiation of intestinal metaplasia versus non-Barrett's esophagus (gastric cardia and squamous mucosa) with a kappa value of 0.811 (95% CI: 0.73-0.89; p<0.0001). When non-Barrett's esophagus was evaluated distinctly as gastric cardia and squamous mucosa, there also was excellent agreement for the OFDI diagnosis of intestinal metaplasia versus gastric cardia and squamous mucosa with a kappa value of 0.866, p<0.001. Six readers with limited OFDI exposure and no formal training demonstrated good agreement with a kappa value of 0.765, (95% CI: 0.62-0.87; p<0.001). The four readers with prior OFDI experience but no formal training demonstrated excellent agreement (κ=0.872, 95% CI: 0.76-0.98; p<0.001). Four readers repeated the test and showed excellent intraobserver agreement of diagnosing BE vs. non-BE (median κ=0.975, IQR 0.94, 1.0). Compared to the expert OFDI reader's interpretation, the median percent correct diagnosis for all ten readers was 96.7% (IQR: 93.3%, 100%). The features of Barrett's esophagus that most frequently drove the correct diagnosis were heterogeneous backscatter and irregular surface morphology. The median time to completion of the test for all ten readers was 27.5 minutes (IQR: 25, 31.5 minutes).

Discussion

This preliminary, single-center study demonstrated that a large group of endoscopists and non-endoscopists could interpret OFDI images after viewing a training set of images with excellent interobserver agreement. The group of readers included one pathologist, three engineers and technicians and six gastroenterologists. The ability to learn the technology did not vary amongst gastroenterologists and the other readers. Only one of the gastroenterologists (E.C) had significant experience with OFDI image interpretation. Our results demonstrate a high inter-observer agreement of previously validated OCT criteria for Barrett's esophagus, gastric cardia and squamous mucosa [22] and show that these criteria can be learned by a large group of readers with limited exposure to the technology.

Currently, diagnosing Barrett's esophagus requires endoscopic biopsies to confirm the presence of intestinal metaplasia. However, the accuracy of this method is questionable as biopsies can only sample a small portion of the distal esophagus. One study suggests that there are five unrecognized cases of Barrett's for every case diagnosed [24]. Furthermore, current screening methods are expensive. New imaging technologies, such as narrow band imaging and confocal endomicroscopy, can enhance surface patterns associated with Barrett's esophagus. However, these imaging technologies ultimately require the use of endoscopy with or without topical or intravenous contrast agents [25, 26]. Targeted screening in high risk populations, such as those with chronic reflux who are over the age of 50, with a low cost screening device not requiring sedation would be ideal. OFDI provides circumferential microscopic views of the distal esophagus in 2 minutes, which is not currently possible with standard of care endoscopy with biopsies. As a standalone device, OFDI would not require the added cost of endoscopy or sedation and could potentially be a low-cost, viable screening tool to assess the presence or absence of intestinal metaplasia [27].

In this paper, we did not use pathologic diagnosis as our gold standard because of our inability to accurately register biopsy sites with the balloon in place. Therefore, the results reported here cannot be used to assess the diagnostic accuracy of OFDI with respect to histology. Previously, a first generation optical coherence tomography (OCT) technology has been shown to have good accuracy for grading dysplasia in Barrett's esophagus[28]. Since this second generation OFDI provides equivalent or better image quality, it is likely that OFDI will also be capable of diagnosing dysplasia. We have developed a marking laser that allows the operator to target regions of interest in an OFDI dataset and studies to validate this marking laser in Barrett's esophagus are ongoing [23]. The diagnostic accuracy of OFDI for dysplasia and Barrett's esophagus will be validated in future histopathologic correlative studies using the laser marking technology.

Additionally, we only included high-quality images from a selected group of patients to demonstrate that key OFDI features of Barrett's esophagus could be learned and distinguished from squamous mucosa and gastric cardia. In this preliminary study, we evaluated these images under ideal circumstances. Future, larger studies in a non-selected group of patients with varying imaging quality will need to confirm the ability of readers to interpret three-dimensional OFDI data and determine the ability of OFDI to detect Barrett's esophagus in real time.

This is the first study to demonstrate that OFDI criteria can be learned by a large group of readers with limited exposure to the technology. These initial findings require confirmation with larger studies but suggest that OFDI shows promise as a potential tool in clinical practice for low-cost screening and surveillance of the distal esophagus.