Techniques for improving ophthalmic studies performed on administrative databases

Over the last several years, there has been a surge in the amount of data at our fingertips with the increased popularity of large electronic health datasets. In a pessimistic response to this, many have touted the adage “garbage in, garbage out,” when referring to the pitfalls of using “big data,” citing inaccuracies in the coding and billing process leading to erroneous conclusions. However, as our experience with using these administrative datasets created from EHR data and insurance claims has grown, successful efforts have been made to critically evaluate and validate the data available.

One way of assessing insurance claims data is to perform studies that validate the use of billing codes for the identification of ophthalmic diagnoses. In general, International Classification of Diseases (ICD) codes for most common diagnoses have been shown to be highly accurate.^1–6 In one study evaluating the accuracy of ICD-9 codes for cataract, primary open angle glaucoma, neovascular age-related macular degeneration, and proliferative diabetic retinopathy, the medical record supported the diagnosis code in 97% of cases.¹ Another recent study of Current Procedural Terminology (CPT) and drug codes (J codes) used in the treatment of diabetic retinopathy also found high positive and negative predictive values.³ Thus, with the proper safeguards big data has become a powerful tool to study these ocular conditions and identify both practice patterns and clinical outcomes on a much larger scale than previously was possible.

As the scope of big data has expanded, we also have been able to broaden the spectrum of diseases studied. Multiple recent studies have aimed to validate the use of ICD coding for identifying cases of uveitis.^7–9 While some studies suggest that coding for uveitis has variable reliability based on the specific EHR used and the type of provider^8,9, others have shown that select codes may be useful for research purposes. For example, a study of 893 uveitis cases evaluated by individual chart review found quite variable positive predictive values of specific ICD9 codes, ranging from 0 to 100 percent. Despite this, 11 of these codes had a positive predictive value exceeding 80%, the typical cut-off used for suitability in scientific research.⁷

Although the results of validation studies for rare diseases, such as uveitis, have been mixed, more sophisticated epidemiological methodology may overcome many of these shortcomings. For example, uveitis is often a chronic or recurrent process; thus, using a second confirmatory diagnosis within a prespecified time interval after the first diagnosis can help improve accurate identification within claims data. This strategy already has been demonstrated to be effective in numerous other chronic diseases including hypertension and diabetes mellitus.^10,11

Similarly, in designing most studies, it is important to be able to differentiate a pre-existing condition from a new diagnosis. Using a look back period or disease-free interval prior to the incident date a code first appeared in the medical record can help improve the accuracy of estimates of incident cases. For example, one study showed that using a look back period of two years allowed for considerably more accurate estimates of disease incidence as opposed to a one year look back period for cataract, primary open angle glaucoma, age-related macular degeneration, and nonproliferative diabetic retinopathy.¹²

In addition to diagnosis codes, using supplemental billing codes for both therapeutic and diagnostic procedures can enhance the likelihood of a correct diagnosis in large electronic health datasets. For example, a patient newly diagnosed with proliferative diabetic retinopathy who also received panretinal photocoagulation on the same day reasonably can be concluded to truly have proliferative diabetic retinopathy. The procedural codes for diabetic retinopathy have been shown to have both high positive and negative predictive values.³ Outside of ophthalmology, the diagnostic accuracy of both rheumatoid arthritis and diabetes have been shown to be greatly improved when ICD coding is combined with medication use.¹³ Although yet to be proven in ophthalmology, it is reasonable to believe applying similar concepts to patients with eye disease should apply. Again, taking uveitis as an example, supplementing a diagnosis code with medication fills of oral corticosteroids or immunosuppressants, or with procedure codes for corticosteroid injections, can be expected to improve the accuracy of uveitis case definitions in administrative datasets.

While it is unlikely that any approach to identifying diagnoses in administrative data will achieve perfect accuracy, a variety of statistical methods are available to account for any residual error in outcome ascertainment. For example, if the sensitivity and specificity of an algorithm are known, they can be incorporated into subsequent statistical analyses to account for the probability that some cases will actually be misclassified controls and vice versa. Methods to account for imperfect sensitivity and specificity have been developed for both binary¹⁴ and survival outcomes¹⁵ and have been demonstrated to produce valid estimates of the association between exposures and outcomes. An alternative approach is to use quantitative bias analysis methods to construct an interval within which the true association is expected to lie under a variety of assumptions about the magnitude of the misclassification.¹⁶

The steps outlined above can increase validity of analyses using diagnoses or outcomes derived from administrative data, but they are only part of the epidemiologic armamentarium used to reduce bias within administrative database studies. Major progress has been made using analytic methods to better address unmeasured confounding. One example of this is the use of propensity scores.¹⁷ While propensity scores can be used in numerous ways, their primary goal is to create comparison groups with balanced covariates, and this methodology already has been frequently used in ophthalmic studies.^18–20

One of the best examples of the benefits of propensity scores can be seen in the recent publication by Rim et al. who examined the effects of low dose aspirin on neovascular age-related macular degeneration (nAMD).¹⁹ They found that while using typical methods to control for covariates, aspirin was found to be associated with an increased risk for nAMD. However, once a propensity score was used to create comparison groups with balanced covariates, no difference was seen.¹⁹ Intuitively, this makes sense since patients taking low dose aspirin typically have cardiovascular disease and related comorbidities, which also are associated with AMD. Once these factors were properly accounted for via the propensity score, no difference was seen.

When performing clinical research of any type, we always must consider the advantages and disadvantages of any database or analytic technique used. As such, no single methodology should ever be considered above reproach. The usual considerations regarding the positive and negative aspects of any research methodology hold true for observational studies, including those conducted in administrative datasets. For instance, it is not surprising when a study finds that a disease without a specific ICD9 or ICD10 code cannot be accurately ascertained in a database based on ICD9/10 coding.⁸ However, the strategies reviewed here provide examples of strategies to enhance the validity of observational research using administrative databases. In appropriate applications, such methods should overcome the major difficulties of using administrative databases for epidemiological research. Administrative databases offer a powerful tool for answering previously unanswerable questions, but their limitations must be acknowledged, just as with other study designs.

Extensive work has been done to improve the quality of research performed on administrative databases across all of medicine offering better methods to validate exposures, diagnoses and outcomes studied. Over the past 15–20 years, several new analytic techniques have been developed specifically to better utilize “big data;” however, these methods are just starting to be implemented in ophthalmic studies. As the epidemiologic community continues to develop these tools in other fields of medicine, ophthalmology has an opportunity to take advantage of these advanced analytic techniques to address some of the important clinical questions in vision research.