Diagnosing Barrett’s esophagus: reliability of clinical and pathologic diagnoses

Abstract

Background

The accuracy of a Barrett’s esophagus diagnosis is not well studied.

Objective

Our purpose was to evaluate the accuracy of a clinical Barrett’s esophagus diagnosis and the reproducibility of an esophageal intestinal metaplasia diagnosis.

Methods

All patients with a Barrett’s esophagus diagnosis between 1994 and 2005 were identified by use of International Classification of Disease (ICD) and Systematized Nomenclature of Medicine (SNOMED) coding. Subsets received manual record review (endoscopy/pathology reports), slide review by a referral pathologist (interrater reliability), and 2 blinded reviews by the same pathologist (intrarater reliability).

Setting

An integrated health services delivery system.

Main Outcome Measurements

Accuracy of electronic clinical diagnosis and reproducibility of esophageal intestinal metaplasia diagnosis.

Results

A total of 2470 patients coded with Barrett’s esophagus underwent record review; a subgroup (616) received manual pathology slide review. Review confirmed a Barrett’s esophagus diagnosis for 1533 (61.9%) patients: 437 of 798 subjects (54.8%) with a SNOMED diagnosis alone, 153 of 671 subjects (26.8%) with an ICD diagnosis alone, and 940 of 1101 subjects (85%) who had both a SNOMED and an ICD diagnosis. The same metaplasia diagnosis occurred with 88.3% of subjects (original vs referral pathologist, interrater reliability; κ =.42, 95% CI, 0.34–0.48). The referral pathologist made the same metaplasia diagnosis twice for a given patient for 88.6% of subjects (intrarater reliability, 2 reviews by same pathologist; κ = 0.65, 95% CI, 0.35–0.93).

Limitations

The accuracy of a Barrett’s esophagus diagnosis likely represents the minimum number, given the strict criteria.

Conclusions

A community pathologist’s diagnosis of esophageal intestinal metaplasia is likely to be confirmed by a referral pathologist. Electronic diagnoses of Barrett’s esophagus overestimate the prevalence, although they are usually confirmed in patients with both a SNOMED and ICD diagnosis of Barrett’s esophagus.

The importance of accurate methods for the assignment of clinical diagnoses cannot be overemphasized; the management of patient conditions, the identification of patients for clinical research, health care financial compensation, and the assignment of human resources all depend at least partially on recorded diagnoses. Pathology classifications are required for many clinical diagnoses, yet few studies examine whether these assignments are reproducible for many GI diseases. Similarly, electronic diagnoses, such as those found in large administrative data sets (eg, health plans and Veterans Affairs hospitals), the U.S. Medicare program, and endoscopic databases, provide abundant opportunities for identifying patients for clinical care (eg, recalling patients who need cancer screening or surveillance for high-risk conditions) and for research studies, but little is known about the overall accuracy of many common GI diagnoses, including Barrett’s esophagus. The validation of pathologic and clinical diagnoses for this condition would inform clinicians, researchers, and policy makers whether these codes can be used alone for decision making or whether additional verification is required.

Prior studies have evaluated interobserver variation for the diagnosis of dysplasia in Barrett’s esophagus^1–3; however, a literature search by our group did not identify any studies that directly evaluated the accuracy of a coded diagnosis of Barrett’s esophagus itself. Similarly, another search identified only a single study of 5 patients that evaluated the reproducibility of a histologic diagnosis of esophageal intestinal metaplasia (using search terms for Barrett’s esophagus combined with the terms classification, interobserver, or intraobserver),⁴ although the presence of intestinal metaplasia is required for a Barrett’s esophagus diagnosis by most criteria.^4–6

We thus evaluated the accuracy of diagnostic codes for Barrett’s esophagus by contrasting codes from electronic databases with diagnoses from a detailed medical record review. We also evaluated the reproducibility of a pathologic diagnosis of Barrett’s esophagus (defined here as the presence of esophageal intestinal metaplasia) between 2 pathologists and between a single pathologist on 2 different occasions.

METHODS

We conducted a study within the Kaiser Permanente, Northern California (KPNC) population, an integrated health services delivery organization. KPNC contains approximately 3.3 million members (approximately one third of the insured population in the region). Research within this setting encompasses practice patterns across a broad geographic area that includes 17 medical centers plus additional free-standing offices and endoscopy units; its membership demographics closely approximate the underlying census population of Northern California.⁷ We identified all persons who received a Barrett’s esophagus diagnosis between 1994 and 2005 according to the International Classification of Disease, 9th revision (ICD-9), codes 530.2 and 530.85, which at KPNC were uniquely coded on reporting sheets as “Barrett’s esophagitis” at the time of an outpatient visit, and the Systematized Nomenclature of Medicine (SNOMED) code M73330 (Barrett’s esophagus). SNOMED codes are commonly used by pathology departments for assigning specific diagnoses. This search identified 5953 persons with an electronic diagnosis of Barrett’s esophagus: 1803 (30.3%) with only a SNOMED diagnosis, 1630 (27.4%) with only an ICD-9 diagnosis, and 2520 (42.3%) with both a SNOMED and an ICD-9 diagnosis. From the written and electronic medical records, we retrieved EGD and relevant pathology reports from a subset of 2470 subjects (not the entire group due to resource limitations) for manual verification of the Barrett’s esophagus diagnosis. These included all subjects with a new electronic diagnosis of Barrett’s esophagus between October 2002 and September 2005 (these patients were then used as part of a case-control study) and serial subjects (both new and prevalent diagnoses) extending before and after these dates within funding limitations. Reviews were performed by a board-certified gastroenterologist (D. A. C.) for 1221 subjects and by professional medical record data abstractors (trained by the gastroenterologist and approximately a 10% subset reviewed by the gastroenterologist) for 1249 subjects; the verification rates for both groups were comparable and are presented together. The reviewer recorded whether each subject met the criteria for diagnosis, and if they did not meet the diagnosis why they were excluded or whether there was insufficient information to make an assignment. Subjects were confirmed to have a diagnosis of Barrett’s esophagus if the endoscopist clearly described a visible length of columnar-type epithelium proximal to the gastroesophageal junction/gastric folds, this area was biopsied, and the pathologist reported specialized intestinal epithelium.⁵ A diagnosis was not confirmed if the endoscopy did not clearly describe the above findings, no biopsy was taken, the pathology reports did not describe intestinal metaplasia, or if, to minimize misclassification, the report described biopsy specimens only from an irregular squamocolumnar junction (ie, an “irregular z-line”).

Capsule Summary

What is already known on this topic

• Validation of both the pathologic and clinical diagnosis of Barrett’s esophagus (BE) would inform clinicians, researchers, and policy makers whether electronic diagnostic codes can be used alone for decision making.

What this study adds to our knowledge

• In a record review of 2470 patients coded for BE, electronic diagnosis overestimated the prevalence of the disease.

We evaluated the reproducibility of pathologic interpretations between 2 pathologists (interrater reliability) by retrieving the pathology slides for a subset of 616 subjects (approximately 91% of those attempted). The esophageal biopsy slides were from serially diagnosed persons with a new electronic diagnosis of Barrett’s esophagus between October 2002 and September 2005. Among persons with endoscopic findings consistent with Barrett’s esophagus, all persons with a community pathologist’s written diagnosis of intestinal metaplasia and a subset of patients with an initial diagnosis of nonintestinal metaplasia were included. Selection of the latter was at regular time intervals but not truly random given the effort to balance subjects with and without intestinal metaplasia. A single referral pathologist (G. J. R.) reviewed the slides blinded to the first pathologic interpretation and to the balance between patients with versus without intestinal metaplasia.

We evaluated the reproducibility of pathologic interpretations for a single pathologist (intrarater reliability) by having the referral pathologist rereview 44 slides he had previously reviewed during the duration of the 3 year study. Approximately every 4 weeks during the review period, we selected a patient from 2 months prior for rereview; this process was consistent over time but not truly random given the effort to balance subjects with and without intestinal metaplasia. Slides for rereview were mixed in with slides awaiting initial review and were not separately identified from regular slides, and the second review was conducted several weeks after the first review (over the course of a 3-year study) to decrease the possibility of recall. The pathologist was not aware of which slides were rereviewed and was thus blinded to the results of the first review.

Statistical analysis

Standard descriptive statistics were calculated. The κ statistic was calculated for agreement between the first and second pathology reviews. The κ statistic is the proportion of agreement achieved beyond that expected to occur by chance. The κ statistic performance was rated according to standard nomenclature: < 0, poor; 0–0.2, slight; 0.21–0.4, fair; 0.41–0.6, moderate; 0.61–0.8, substantial; 0.81–1, almost perfect.⁸ The κ statistic and its SE were calculated by use of the STATA statistical package (version 8, STATA, College Station, Tex); CIs were estimated assuming a normal distribution. The study was approved by the institutional review board.

RESULTS

Full or partial records were retrieved for 2470 subjects, for whom a Barrett’s esophagus assignment was completed for 2378 (96.3%) (Table 1). Among the 92 subjects lacking a final assignment, the pathology reports could not be retrieved for 3 (3.3%), endoscopy data could not be retrieved for 10 (10.8%), neither pathology nor endoscopy reports were available for 12 (13.0%), and the data available were insufficiently detailed to establish a final assignment for 67 (72.8%) patients.

TABLE 1.

Final Barrett’s esophagus assignment and reasons for exclusion on the basis of medical record review

	All, no. (%)	SNOMED	ICD	Both SNOMED and ICD
Total subjects	2470 (100.0)	798	571	1101
Barrett’s esophagus confirmed	1530 (61.9)	437 (54.8)	153 (26.8)	940 (85.4)
Barrett’s esophagus diagnosis not confirmed	848 (34.3)	330 (41.4)	390 (68.3)	128 (11.6)
Insufficient data available to classify	92 (3.7)	31 (3.9)	28 (4.9)	33 (3.0)
Reasons for exclusion*
Irregular z-line only†	88 (3.6)	35 (1.2)	10 (1.8)	43 (5.4)
No endoscopic findings‡	228 (9.2)	119 (14.9)	39 (6.8)	70 (6.4)
No intestinal metaplasia on biopsy	214 (8.7)	63 (7.9)	140 (24.5)	11 (1.0)
No endoscopic findings and no intestinal metaplasia	240 (9.7)	87 (10.9)	147 (25.7)	6 (0.5)
Possible Barrett’s esophagus, no biopsies clearly from area of interest	37 (1.5)	1 (0.1)	34 (5.6)	2 (0.2)
No pathology record available§	4 (0.2)	1 (0.1)	2 (0.4)	1 (0.1)
No endoscopy record available§	12 (0.5)	7 (0.9)	2 (0.4)	3 (0.3)
Neither endoscopy nor pathology records available§	18 (0.7)	4 (0.5)	10 (1.8)	4 (0.4)
Insufficient detail§	70 (2.8)	19 (2.4)	24 (4.2)	27 (2.5)
Other	29 (1.2)	17 (2.1)	10 (1.8)	2 (0.2)

^*Among the 848 persons in whom a diagnosis was not confirmed.

^†The squamocolumnar junction (the “z-line”) was described as irregular and the report did not clearly describe substantial tongues of columnar mucosa extending proximally into the body of the esophagus.

^‡No endoscopic findings reported that were clearly diagnostic of Barrett’s esophagus.

^§The reviewer assigned these reviews as partially incomplete but stated that the available data were sufficient for classification. These included persons, for example, with no record of an endoscopy being performed to support the diagnosis (and no outside records), a note of a biopsy specimen being taken but no pathologic interpretation or specimen recorded in the pathology department, clear coding errors, etc.

Record review

After medical record review, an assignment of “Barrett’s esophagus” was confirmed in 1530 (61.9%) and rejected in 848 (34.3%), and there were insufficient data in 92 (3.7%) of all subjects (Table 1). A diagnosis was confirmed among 437 of 798 persons (54.8%) with a SNOMED diagnosis alone, 153 of 571 patients (26.8%) with an ICD diagnosis alone, and 940 of 1101 persons (85.4%) who had both a SNOMED and an ICD diagnosis of Barrett’s esophagus. If any ICD diagnosis was used (regardless of whether a SNOMED diagnosis was assigned), a diagnosis was confirmed among 1093 of 1672 persons (65.4%).

The reasons for exclusion are outlined in Table 1. These included only an irregular z-line for 88 subjects (3.6%), no clearly described endoscopic findings consistent with Barrett’s esophagus for 228 (9.2%), no intestinal metaplasia on biopsy for 214 (8.7%), and neither endoscopic nor pathologic findings consistent with Barrett’s esophagus for 240 (9.7%) patients.

The endoscopic findings among persons excluded are described further in Table 2. Among the persons excluded because of no definitive recorded endoscopic findings, the endoscopist frequently reported a hiatal hernia or esophagitis but did not clearly describe esophageal columnar metaplasia suspicious for Barrett’s esophagus.

TABLE 2.

Other endoscopic findings among persons not meeting a Barrett’s esophagus definition, by main reasons for exclusion

		Other endoscopic findings within each exclusion category*
Reason for exclusion	Total with exclusion	Esophagitis	Hiatal hernia	Esophageal ring	Stricture	Mass
Irregular z-line only*	88	6 (6.8)	28 (31.8)	4 (4.5)	3 (3.4)	0 (0)
No endoscopic findings clearly consistent with Barrett’s esophagus	228	62 (27.2)	38 (16.7)	12 (5.3)	24 (10.5)	4 (1.8)
No intestinal metaplasia	214	58 (27.0)	79 (36.7)	4 (1.9)	4 (1.9)	0 (0)
No endoscopic findings and no intestinal metaplasia	240	69 (28.8)	37 (15.4)	11 (4.6)	12 (5.0)	5 (2.1)
Possible Barrett’s esophagus, no biopsies	37	16 (43.2)	16 (43.2)	0 (0)	5 (13.5)	0 (0)

^*Number and percent. The percent is the percent of all persons with that reason for exclusion with the listed finding. Totals may add up to > 100% because more than one finding may have been reported for a single patient.

Pathology review

We evaluated interrater reliability by having a separate pathologist (blinded to the results of the first pathologist) review pathology slides retrieved from 616 subjects (Table 3). These included 580 patients with an initial diagnosis of intestinal metaplasia and 36 patients in whom the initial diagnosis was gastric metaplasia or columnar metaplasia.

TABLE 3.

Agreement between 2 pathologists for the diagnosis of intestinal metaplasia (interrater agreement)

	Pathology slide review assignment
Original pathology report assignment	Columnar or gastric metaplasia	Intestinal metaplasia
Columnar or gastric metaplasia	31	5
Intestinal metaplasia	67	513

Reviews were conducted blinded to the assignment of the other pathologist.

The overall agreement between the pathologists was 88.3% (Table 3). Among the 580 patients with an initial diagnosis of intestinal metaplasia, an intestinal metaplasia diagnosis was also made by the referral pathologist for 513 subjects (88.4%). Among 36 patients with an initial diagnosis of gastric or columnar metaplasia, the referral pathologist similarly did not describe intestinal metaplasia in 31 (86.1%). The κ statistic for interobserver agreement was 0.41 (95% CI, 0.34–0.48), indicating “moderate” agreement beyond that expected by chance alone.

We evaluated intrarater reliability by having the referral pathologist conduct a blinded rereview of 44 slides he had previously reviewed during the 3-year duration of the study (see Methods) (Table 4). The overall intrarater accuracy was 88.6%, with κ = 0.64 (95% CI 0.35–0.93), indicating “substantial agreement” beyond that expected from chance alone.

TABLE 4.

Agreement between 2 reviews of the same patient by the same pathologist for the diagnosis of intestinal metaplasia (intrarater agreement)

	Review pathologist’s second assignment
Review pathologist’s first assignment	Columnar or gastric metaplasia	Intestinal metaplasia
Columnar or gastric metaplasia	6	1
Intestinal metaplasia	4	33

The 2 reviews were conducted blinded to each other (see Methods).

Among all patients receiving both written medical record review and manual review of their pathology slides, the pathologist’s slide review changed the classification (on the basis of the presence or absence of intestinal metaplasia) for 74 (12%) patients: from “include” to “exclude” in 58 (9.4%), from “exclude” to “include” in 5 (1%); among 11 persons with “uncertain” assignments from the medical record review, 4 were included and 7 were excluded. Among the 339 persons with a Barrett’s esophagus diagnosis by both SNOMED and ICD coding, the pathology review changed the assignments of 28 (8.3%) patients.

DISCUSSION

The accurate identification of patients with Barrett’s esophagus for either clinical care (eg, call-backs for surveillance examinations) or clinical research requires valid pathologic and clinical diagnoses. The purpose of this study was to evaluate the reproducibility of a pathologic diagnosis of intestinal metaplasia and the accuracy of electronic diagnoses of Barrett’s esophagus compared with manual record review. We found that a pathologic diagnosis of esophageal intestinal metaplasia is highly likely to be reproduced by a separate review of the slides. In addition, the modest intraobserver variation observed for a single pathologist suggests that a proportion of the discordance for pathology reviews between different pathologists may result from somewhat random misclassification rather than from an incorrect reading by the original pathologist. In contrast, a coded diagnosis of Barrett’s esophagus was confirmed by record review only 61.9% of the time––a number that is likely too low by itself for either clinical or research uses without supplemental manual verification. However, among the substantial proportion of persons who had both a SNOMED and an ICD diagnosis, record review confirmed a diagnosis in 85.4%. It should be emphasized these numbers likely represent the minimum proportion of persons who had Barrett’s esophagus, given the strict criteria used. Persons excluded may have had endoscopic findings not adequately recorded by the physician that supported the diagnosis of Barrett’s esophagus, or the diagnosis may have been based on knowledge not discernible from available reports (for example, a remote examination that showed Barrett’s esophagus).

This study expands the existing literature on the diagnosis of Barrett’s esophagus. There are minimal data on the reproducibility of the pathologic diagnosis of esophageal intestinal metaplasia, although pathologic examination is the “gold standard” against which other techniques are compared, and the presence of intestinal metaplasia is a critical component for establishing a Barrett’s esophagus diagnosis by most criteria.^5,9 Data exist on observer variation for diagnosing dysplasia in Barrett’s esophagus,^1–3 the use of specialized techniques for identifying areas of columnar metaplasia,^10,11 optimal biopsy methods for detecting metaplasia,¹² cell types in Barrett’s esophagus,¹³ and endoscopic criteria for assigning the extent of columnar metaplasia.¹⁴ However, as noted, a literature search by our group found only one article that specifically evaluated the reliability of the actual pathologic diagnosis of intestinal metaplasia; that study had several pathologists review 5 slides with different types of metaplasia/dysplasia.⁴ The recent publication of additional endoscopic standards may help standardize the diagnosis of Barrett’s esophagus, but the final diagnosis (particularly for persons with conditions that may complicate the diagnosis such as hiatal hernias or esophagitis) also depends on the accuracy of a histologic finding of intestinal metaplasia.¹⁴ This can be challenging because a seminal research study of Barrett’s esophagus diagnosis found that, even among patients with two examinations within 6 weeks of each other, 20% of persons with intestinal metaplasia and endoscopic changes of Barrett’s esophagus on one examination did not have intestinal metaplasia identified on the other examination.¹⁵ That finding has been attributed largely to sampling error for biopsy location; however, the current study suggests that a single pathologist may rate intestinal metaplasia as gastric metaplasia approximately 10.8% of the time during a blinded rereview. Thus, a substantial portion of the difference noted in the prior study may have been solely due to disparate classifications of intestinal metaplasia (either between pathologists or between 2 examinations by the same pathologist) rather than solely to sampling differences.

There are several strengths to the current study. First, the study included a large number of patients, which increases the precision of the estimates. Second, the study was community based, which enhances the applicability of the findings to similar large populations and the “real world” use of clinical diagnosis codes and pathologic evaluations. Third, the design enabled the evaluation of both interobserver and intraobserver variation in the pathology readings, which provides insights as to whether differences between pathologists are due, in part, from random differences between readings rather than only disagreements between pathologists.

There are limitations to this analysis. First, it should be emphasized again that the proportion of persons confirmed to have Barrett’s esophagus likely represents the minimum number, given the strict criteria used. Second, ICD coding was modified in the year 2003 from 530.2 (listed as esophageal ulcer in the ICD coding manual) to a more specific 530.85 code for Barrett’s esophagus. Our results should reflect the specificity of this newer coding because the prior 530.2 coding was always specifically designated for Barrett’s esophagus within physician coding sheets and programs at KPNC, and the 3 were seamlessly overlapped on coding sheets at our centers during the years of this study. Third, some patients with extensive esophageal columnar metaplasia may have little or no intestinal metaplasia; these patients did not meet the current study definitions of Barrett’s esophagus in the United States, although the natural history of such patients is not known and some physicians may consider such patients as having “Barrett’s esophagus.”^16,17 Fourth, there is no true “gold standard” correlated with clinical outcomes, for the diagnosis of esophageal intestinal metaplasia. Methods include hematoxylin and eosin (H&E) staining, special stains (such as alcian blue), and cytokeratin markers; however, it is unclear which of these has the best performance characteristics for identifying persons at risk for esophageal adenocarcinoma.^12,18,19 Most slides at our facilities used routine H&E staining and did not use special stains. The use of specialized stains provided only 5.4% additional sensitivity in a recent study, and H&E alone represents the standard at the majority of community and academic centers surveyed.^12,18 Finally, additional strategies such as identifying persons with repeated ICD diagnoses would likely improve the predictive value of a Barrett’s esophagus diagnosis, but this would decrease the sensitivity.

In conclusion, an initial pathologic diagnosis of esophageal intestinal metaplasia in our population was highly likely to be confirmed with a slide review from a second referral pathologist. Thus, a second slide review provided relatively little additional value, particularly because a substantial proportion of any difference in classification between pathologists was likely from random variation between slide reviews (as seen in the intraobserver study) rather than from true “errors” in classification. Second, a strategy using persons identified with either an ICD or a SNOMED code for Barrett’s esophagus provided the greatest sensitivity for detecting persons with a Barrett’s esophagus diagnosis within a population; however, this method was only moderately accurate. Identifying persons with both ICD and SNOMED codes correctly classified approximately 85.4% of subjects compared with medical record review, but this method only detected 61% of all patients with Barrett’s esophagus. Electronic coding thus overestimates the prevalence of Barrett’s esophagus, and most clinical and research uses will require a manual verification of disease status. These results can help inform diagnoses of Barrett’s esophagus for patient care, health policy, and clinical research.

Abbreviations

H&E

hematoxylin and eosin

ICD

International Classification of Disease

KPNC

Kaiser Permanente, Northern California

SNOMED

systematized nomenclature of medicine coding