Validity of code based algorithms to identify primary open angle glaucoma (POAG) in Veterans Affairs (VA) administrative databases

Abstract

Purpose:

The validity of the International Classification of Diseases, 9th revision, Clinical Modification (ICD-9) code for primary open angle glaucoma (POAG) in the Department of Veterans Affairs (VA) electronic medical record has not been examined. We determined the accuracy of the ICD-9 code for POAG and developed diagnostic algorithms for the detection of POAG.

Methods:

We conducted a retrospective study of abstracted data from the Michael E. DeBakey VA Medical Center’s medical records of 334 unique patients with at least one visit to the Eye Clinic between 1999 and 2013. Algorithms were developed to validly identify POAG using ICD-9 codes and pharmacy data. The positive predictive value (PPV), negative predictive value (NPV), sensitivity, specificity and percent agreement of the various algorithms were calculated.

Results:

For the ICD-9 code 365.1x, the PPV was 65.9%, NPV was 95.2%, sensitivity was 100%, specificity was 82.6%, and percent agreement was 87.8%. The algorithm with the highest PPV was 76.3%, using pharmacy data in conjunction with two or more ICD-9 codes for POAG, but this algorithm also had the lowest NPV at 88.2%.

Conclusions:

Various algorithms for identifying POAG in the VA administrative databases have variable validity. Depending on the type of research being done, the ICD-9 code 365.1x can be used for epidemiologic or health services database research.

Introduction

Primary open angle glaucoma (POAG) is a potentially blinding disease with a prevalence of 1.86% in Americans over the age of forty.¹ This has been estimated to increase to 3.36 million people by the year. 2020^1,2 Visual impairment produced by glaucoma has been associated with decreased quality of life, increased dependence in activities of daily living, increased risk for depression and increased morbidity due to fall-related injuries.^3,4

The use of administrative health care databases is appealing in order to study the risk factors and outcomes of chronic and debilitating diseases such as POAG in a large, diverse population of patients. Many of these databases use International Classification of Disease, 9th revision, Clinical Modification (ICD-9) codes; but the validity of these codes in correctly identifying the presence of conditions is a potentially limiting factor for such studies.^5–8 Therefore, establishing the validity of these codes and developing algorithms to optimize the valid identification of POAG in administrative databases is important to obtain valid study results.

The Department of Veterans Affairs (VA) is the largest integrated healthcare provider in the United States, serving over 8 million enrolled veterans at 154 medical centers and 875 ambulatory care and community-based outpatient clinics. The VA provides care to a disproportionate number of elderly patients at risk for developing POAG. Only a few small studies have examined POAG risk and outcomes in veterans using VA databases and none validated the use of the ICD-9 code necessary to support large scale population-based research using the VA national healthcare databases.

We performed a validation study in veteran VA users to calculate the test performance characteristics of the ICD-9 code for glaucoma (365.1x) against the gold-standard of POAG determined from medical record review. The secondary aim was to combine ICD-9 data with data on use of medications used to treat POAG to create several different algorithms for detecting POAG in the VA database, and compare the accuracy of these algorithms with each other.

METHODS

Study design and source population

Approval for the study was obtained from the institutional review board of Baylor College of Medicine and the MEDVAMC. We conducted a validation study of 334 veterans seen in ophthalmology or optometry clinics using a combination of administrative and electronic medical records at the Michael E. DeBakey Veterans Affairs Medical Center (MEDVAMC) in Houston, Texas. The MEDVAMC is one of the largest facilities in the national Veterans Health Administration; it serves 28 counties in Southeast Texas and is the primary healthcare provider for more than 116,000 veterans as well as regional tertiary referrals.

Glaucoma definition and case ascertainment

The VA nationally has a long standing history of use of electronic medical records and has utilized an electronic medical record (EMR) since 1997; however, the MEDVAMC eye clinic completed the transition from paper to electronic records in 2002. The VA auto- extracts clinical data including all diagnostic and procedural codes and pharmacy use associated with inpatient, outpatient and pharmacy encounters from these EMRs daily and compiles this data into structured relational databases maintained in the Corporate Data Warehouse (CDW) for visits from 1st October 1999 and onwards.

We searched the CDW to identify veterans seen at the MEDVAMC outpatient clinic by optometrists or ophthalmologists, including resident and attending physicians or optometrists, with at least one ICD-9 code of 365.1x (POAG) between 1st October 1999 and 1st October 2013. For all patients abstracted, we also recorded demographic data, pharmacy data (any dispensed prescription for acetazolamide, betaxolol, carbachol, dorzolamide, dorzolamide-timolol, latanoprost, travoprost, levobunolol, methazolamide, or pilocarpine), and surgical data (tube shunt or trabeculectomy; CPT code 66170, 66172, or 66180).

We randomly selected a sample of patients with at least one ICD-9 code for POAG for our EMR review based on feasibility and a sample size calculation. We determined that with 229 patients we would be able to calculate a positive predictive value of approximately 65% with precision of 0.6–0.7. A glaucoma fellowship-trained board-certified faculty ophthalmologist (KSB) reviewed the charts to determine if there was sufficient evidence of POAG documented. The POAG “gold standard” was defined by a panel of ophthalmologists (KSB, BJF, SON) as the presence of one or more of the following: 1) Cup to disc ratio ≥ 0.6 in either eye, 2) Cup to disc ratio difference ≥ 0.2 between right and left eyes, 3) Humphrey visual field (HVF) consistent with glaucomatous defects, or 4) History of prior surgery for glaucoma (trabeculectomy or tube shunt implantation). Patients meeting one or more of these criteria were classified as having confirmed POAG (true positives). The absence of POAG was defined as the absence of all of the above criteria, and these patients were classified as false positives. Gonioscopy records were reviewed to determine if the angle was open or closed. A second glaucoma fellow- ship-trained ophthalmologist (SON) reviewed a random sampling of ten charts and there was 100% concordance.

We also randomly selected a sample of patients seen at the MEDVAMC Eye Clinic during the same study period who did not have an ICD-9 code for POAG for medical review. These were to be classified as true negatives if they met no criteria for POAG and false negatives if they did meet one or more of the criteria for POAG.

There were some patients coded incorrectly as having POAG according to the gold standard for this study, but on medical record review had another type of glaucoma (i.e. neovascular glaucoma, secondary glaucoma) or had risk factors making them susceptible to the development of glaucoma (i.e. pseudoexfolation syndrome, ocular hyper-tension): we defined these patients as “glaucoma spectrum” patients. Charts were determined to be incomplete if not enough information was recorded to make a determination regarding if the patient truly had POAG or not.

Data analysis

We compared demographic and clinical characteristics of patients with the ICD-9 code vs. those without the code, as well as between those with chart review con-firmed POAG vs. no POAG using the chi-square test for categorical variables and student’s t-test for the continuous variable. We used sampling weights and then calculated the positive predictive value (PPV), negative predictive value (NPV), sensitivity, specificity and percent agreement for the ICD-9 code 365.1x using the gold standard from the medical record. PPV indicates the probability that a patient with the ICD-9 code actually has that condition as defined by the medical record, or how well the code predicts the presence of confirmed disease. NPV is the probability that a patient without the code actually is without the disease as defined by the medical record, or how well the absence of the code predicts the absence of the disease. Percent agreement is the total of the two concordant cells (i.e. the proportion without both the code and condition plus the proportion with the code and the condition).

We also created algorithms that used a combination of ICD-9 codes and pharmacologic information to identify POAG. Algorithm 1 used one or more codes for 365.1x. Algorithm 2 used two or more codes for 365.1x. Algorithm 3 used pharmacy data to identify patients who had ever dispensed glaucoma medication at any point during the study period. Algorithm 4 included one or more codes for 365.1x and glaucoma medication dispensed. Algorithm 5 used two or more codes for 365.1x and glaucoma medication dispensed. All data analyses were performed using SPSS version 23 (SPSS Inc., Chicago IL).

Results

We identified 19,015 patients who had at least one ICD-9 code for POAG, and of those we randomly selected 229 patients for review. We identified 55,496 patients who had never received an ICD-9 code for POAG, and of those we randomly selected 105 patients for review. Of the 334 EMRs reviewed, 19 had insufficient data to make a POAG determination. Of the total charts reviewed, 311 (93%) were male and 23 (7%) were female. The mean age was 63.5 years, with 45% (149) Blacks, 47% (158) Whites, 2% (8) other, and 6% (19) unknown. The sex between patients with the POAG code vs. patients without the POAG code was similar; however, patients with the POAG code were slightly older (64.8 years vs. 60.5 years) and more likely to be Black (52.4% vs. 27.6%) than those without it (Table 1). Also, the true positive POAG patients were more likely to be Black (57.6% vs. 27.6%) and slightly older (65.3 years vs. 60.5 years) than the true negative patients.

Table 1.

Demographics of patients with and without ICD-9 code 365.1x.

		ICD-9 code 365.1x
		Positive		Negativea
	POAG present “True positives” (n = 151)	POAG absent “False positives” (n = 78)	Total ICD-9 code 365.1 x positive	POAG absent “True negatives” (n = 105)	Total POAG absent	P valueb	P valuec
Sex (n,%): Male	141 (93.4%)	71 (91.0%)	212 (92.5%)	99 (94.3%)	170 (92.9%)	0.57	0.86
Female	10 (6.6%)	7 (9.0%)	17 (7.4%)	6 (5.4%)	13 (7.1%)
Race (n,%): Black	87 (57.6%)	33 (42.3%)	120 (52.4%)	29 (27.6%)	62 (33.9%)	<0.01	<0.01
White	57 (37.7%)	35 (44.9%)	92 (40.2%)	66 (62.9%)	101 (55.2%)
Other	3 (2.0%)	3 (3.8%)	6 (2.6%)	2 (1.9%)	5 (2.7%)
Unknown	4 (2.6%)	7 (9.0%)	11 (4.8%)	8 (7.6%)	15 (8.2%)
Mean age ± SD (years)	65.3 ± 12.9	63.9 ± 13.3	64.8 ± 13.0	60.5 ± 15.0	61.9 ± 14.4	<0.01	0.01

^aThere were no “false negative” patients. Abbreviations: SD, standard deviation

^bComparing ICD-9 positives with negatives

^cComparing POAG present with POAG absent

There were 187 (66%) patients who had at least one POAG code who also received glaucoma medication during the study period, while no patients without the POAG code received glaucoma medication during the study period (Table 2). Of the patients who received glaucoma medications, 16% were dispensed medication on the same day as their index date. The median amount of time between glaucoma medications dispensed and the index date was 67 days, with only 26% of patients having medications dispensed greater than one year from their index date. There were 18 (8%) patients with the POAG ICD-9 code who underwent glaucoma surgery (either trabeculectomy or tube shunt) during the study period, while no patients with- out the POAG code underwent glaucoma surgery. Gonioscopy was also reviewed and the angle was found to be open in all patients for whom it was recorded. However, due to the fact that many patients had paper charts before the MEDVAMC eye clinic converted to electronic records in 2002, there was no available gonioscopy data for approximately 50% of the patients. Our review of all charts with complete data on practitioner type indicate that 71% were coded by an ophthalmology resident, 18% by an optometrist, 3% by an attending ophthalmologist, and 0.5% by an ophthalmology fellow. The remainder (7%) of the notes were coded by technicians, nurses, or primary care physicians. Also 90% of the patients had a 5^th digit code of primary open angle glaucoma (365.11) or low tension open angle glaucoma (365.12).

Table 2.

Clinical characteristics of patients with and without ICD-9 code 365.1x.

		ICD-9 code 365.1x
	Positive		Negative
	POAG present “True positives” (n = 151)	POAG absent “False positives” (n = 78)	POAG absent “True negatives” (n = 105)
Dispensed glaucoma medicationsa during study period (n,%)	141 (93.4%)	46 (59.0%)	0 (0%)
Glaucoma surgery at VA during study period (n,%)	16 (10.6%)	2 (2.6%)	0 (0%)

^aacetazolamide, betaxolol, carbachol, dorzolamide, dorzolamide-timolol, latanoprost, travoprost, levobunolol, methazolamide, or pilocarpine btrabeculectomy or tube shunt

Compared to the gold standard, 151 (66%) of the 229 patients with an ICD-9 code of 365.1x were coded correctly, 64 (28%) were coded incorrectly, and 14 (6%) had incomplete data to make a determination. In order to calculate test performance, the most negative assumption was used, which is that the incomplete charts were coded incorrectly. After applying sampling weights, the PPV was 66% and the NPV was 95% (Table 3). The sensitivity was 100%, specificity was 83% and percent agreement 88%. When the incomplete charts were excluded, the PPV increased to 70%, NPV increased to 100%, sensitivity remained at 100%, specificity increased to 91%, and percent agreement increased to 92%. When the most positive assumption was used (that the incomplete charts were coded correctly), the PPV increased to 72%, the NPV remained at 100%, sensitivity remained at 100%, specificity increased to 91% and percent agreement increased to 93%.

Table 3.

Test characteristics of various algorithms examined to determine true POAG.

Algorithm	Description	PPV (%)	NPV (%)	Sensitivity (%)	Specificity (%)	Percent Agreement (%)
1a.	1+ code (most negative assumptiona)	65.9	95.2	100.0	82.6	87.8
1b.	1+ code (excluding incomplete charts)	70.2	100.0	100.0	90.7	92.4
1c.	1+ code (most positive assumptionb)	72.1	100.0	100.0	91.3	92.9
2	2+ codes	68.3	92.9	69.9	92.3	88.1
3	Meds dispensed	75.4	93.2	76.0	93.0	89.2
4	Meds dispensed and 1+ codes	75.4	93.2	76.0	93.0	89.2
5	Meds dispensed and 2+ codes	76.3	88.2	48.0	96.3	86.7

^aIncomplete charts were coded incorrectly

^bIncomplete charts were coded correctly

For algorithms 2–5, the most negative assumption was used. Algorithm 2 which was defined as the use of two or more codes of 365.1x, decreased the PPV to 68% and NPV to 93%. Algorithm 3 which was defined as a history of dispensed glaucoma medications had a PPV of 75% and NPV of 93%. Algorithm 4 which was defined as dispensed glaucoma medications plus one or more POAG code had a PPV of 75% and NPV of 93%. Of note, the addition of one POAG code to the history of glaucoma medications did not change the test statistics between algorithms 3 and 4. Algorithm 5 which was defined as dispensed glaucoma medications plus two or more POAG codes improved the PPV to 75% and decreased the NPV to 88%.

In Table 4, the test statistics were computed again, classifying the “glaucoma spectrum” patients as true positives instead of false positives. This improved the PPV for all algorithms, but also decreased the NPV for algorithms 2–5.

Table 4.

Test characteristics of various algorithms examined to determine true POAG (with “glaucoma spectrum” included as true POAG).

Algorithm	Description	PPV (%)	NPV (%)	Sensitivity (%)	Specificity (%)	Percent Agreement (%)
1a.	1+ code (most negative assumptiona)	89.5	95.2	86.6	96.4	93.8
1b.	1+ code (excluding incomplete charts)	95.3	100.0	100.0	98.4	98.8
1c.	1+ code (most positive assumptionb)	95.6	100.0	100.0	98.5	98.9
2	2+ codes	92.8	90.5	70.3	98.1	90.9
3	Meds dispensed	92.0	77.6	54.0	97.1	80.8
4	Meds dispensed and 1+ codes	92.0	77.6	54.0	97.1	80.8
5	Meds dispensed and 2+ codes	93.1	72.7	32.7	98.7	75.2

^aIncomplete charts were coded incorrectly

^bIncomplete charts were coded correctly

Discussion

To our knowledge, this is the first study to systematically validate ICD-9 code 365.1x to identify the POAG using outpatient data from the Veterans’ Affairs administrative databases. We found that the code for POAG has a PPV of 65.9% and a sensitivity of 100%. The PPV was improved by increasing the number of times, the code was used; however, sensitivity was significantly diminished in the process. Also of interest were algorithms 3–5, which all used pharmacy data and had the highest positive predictive values (ranging 75–76%); however, they also had much lower sensitivities (ranging 48–76%).

A prior study conducted at two academic, non-VA medical centers and reviewing 250 charts reported a PPV of 98% for the ICD-9 code 365.11⁹ This is very accurate, and may reflect the level of coding accuracy expected in an academic institution, where presumably attending physicians or optometrists are entering medical codes as opposed to trainees. However the study did not specify how the gold standard was defined, so there is no way to know if the criteria were as stringent as our study’s criteria. There have also been two studies of the validity of the glaucoma code in the UK Clinical Practice Research Datalink (CPRD) that reported PPVs of 89% and 84%; however, both studies validated the code using general practitioners instead of ophthalmologists.^10,11 Also the criterion used for defining POAG was patient self-report of POAG, which is not as stringent as our criteria which were developed by a team of glaucoma specialists.

A panel of three glaucoma trained ophthalmologists chose to define the gold standard definition of glaucoma in this study as any eye with a cup to disc ratio ≥ 0.6, as indicated in the literature that when this criteria is met the probability of abnormality increases dramatically and is indicative of glaucomatous risk across all racial group in the United States.^12,13 Our gold standard definition also included a cup to disc ratio difference of ≥ 0.2 between right and left eyes because 88% of normal subjects have a cup to disc ratio difference equal or less than 0.1¹⁴ A recent meta-analysis showed that the likelihood ratio of detecting primary open angle glaucoma by using a cup to disc ratio ≥ 0.6 was 7.0–7.5, and for using asymmetry ≥ 0.2 the likelihood ratio was 3.9–4.1¹⁵ These likelihood ratio levels thus demonstrate that their inclusion in diagnostic test criteria is associated with a moderate to large increase in post-test probability that a patient had clinically confirmed glaucoma. There have been other published studies in peer reviewed journals, which have used similar criteria to this.^16,17 Even in patients with ocular hypertension, without glaucoma, increased cup to disc ratio and cup to disc asymmetry has been shown to correlate with increased risk for development of glaucoma.^18,19

The moderately low PPV of the code in our study is likely due to error by the coding resident, optometrist, or physician. Many times patients with the diagnosis of ocular hypertension, glaucoma suspect, pseudoexfoliation syndrome without evidence of glaucoma and other types of glaucoma (i.e. neovascular glaucoma) as noted in the chart were coded incorrectly as POAG. The majority of care providers coding for visits at the MEDVAMC are residents who are relatively inexperienced at coding, and during busy clinics may select the easiest available ICD-9 code when closing the patient encounter. However, although the code for POAG is used for many patients who do not meet strict criteria for POAG, many do have other types of glaucoma (i.e. neovascular glaucoma, pseudoexfoliation glaucoma) or have risk factors for the development of glaucoma (i.e. ocular hypertension, pseudoexfoliation syndrome, etc.).

Results from this study are useful to the clarify uncertainty, regarding the validity of the POAG ICD- 9 code which will help researchers interested in using administrative databases to examine the questions regarding patients with POAG. Also, the absence of a POAG code was highly predictive of absence of the disease. Various algorithms described here can be used depending on the type of research being done. For example, in a prevalence study, it is advisable to use the most sensitive algorithm to ensure capture of all patients; whereas, in doing a study of risk factors it would be important to use the algorithm with the highest PPV to ensure all subjects have the disease of interest. Of note, in research including patients in the “POAG spectrum” the code performs very well.

Strengths of this study include a large number of charts which were individually reviewed by a glaucoma trained ophthalmologist and demonstrated high concordance with a second glaucoma trained ophthalmologist in a subset of patients. Additionally, the patients included in our study were racially diverse. We also developed and compared multiple algorithms to detect POAG. To our knowledge, this is the first study to validate the use of the POAG ICD-9 code 365.1x in VA administrative databases.

Finally, there are several limitations to this study. It was conducted in a single VA center and thus may not reflect the coding practices of other VA sites or non-VA health care systems. However, MEDVAMC is one of the largest medical centers in the VA system, and many other VA hospitals use residents as well as optometrists and attending physicians to examine and code patient encounters. Secondly, the MEDVAMC Eye Care Line did not fully transition from paper charts to a wholly electronic medical record until the year 2002 with many enrolled patients also using the VA in years prior to this transition. It is thus possible that some individuals may have fulfilled the criteria for POAG if the paper charts were able to be reviewed, in which case the PPV would be closer to that of the most positive assumption. Also our results would be more accurate if gonioscopy results were available for every patient, but presumably many of these results were in the paper charts. Also, this study validated how well the assigned ICD-9 codes matched the diagnosis indicated by the medical record, but we did not assess the reliability of the information provided in the medical record itself, a bias inherent to any study using medical records. We did not seek to validate the individual fifth digit codes 365.10–365.15; however, we found that of the cases included in our study with 365.1x, 90% had a 5^th digit code of 365.11 (POAG) or 365.12 (low tension open angle glaucoma) thus supporting that our results based on use of 365.1x reflect POAG or POAG spectrum patients. Additionally, the glaucoma specialist reviewing the medical record was not blinded to whether the patient had ever received the diagnostic code for POAG. However the high concordance with the random selection of medical records reviewed by a second ophthalmologist is reassuring. Finally, we do not know how the recent change from ICD-9 to ICD-10 will affect the test characteristics for use of ICD codes to identify POAG, so the code needs to be validated within ICD-10 for research involving records after October 2015.

In conclusion, we found one ICD-9 code of 365.1x was moderately valid in predicting POAG in the VA EMR. PPV increased by adding more criteria (such as multiple ICD-9 codes plus glaucoma medications), but sensitivity decreased. Also, the POAG ICD-9 code includes a sizeable proportion of patients who have other types of glaucoma or have characteristics suspicious for the development of glaucoma, but do not meet strict criteria for POAG. The various algorithms we developed for POAG can be reliably used depending on the research question of interest to identify veterans with POAG seen in the VA system. They will be useful in encouraging additional system-wide epidemiologic and health services research on POAG, an understudied area among the 78 million veterans enrolled in VA healthcare.