Identifying the mechanism of missingness for unspecified diabetic retinopathy disease severity in the electronic health record: an IRIS^® Registry analysis

Abstract

Observational studies of diabetic retinopathy (DR) using electronic health record data often determine disease severity using International Classification of Disease (ICD) codes. We investigated the mechanism of missingness for DR severity based on ICD coding using the American Academy of Ophthalmology IRIS^® Registry. We included all patient encounters in the registry with a DR ICD-9 or ICD-10 code between January 1, 2014 and June 30, 2021. Demographic, clinical, and practice-level characteristics were compared between encounters with specified and unspecified disease severity. Practices were divided into quartiles based on the proportion of clinical encounters with unspecified DR severity. Encounters with unspecified disease severity were associated with significantly older patient age, better visual acuity, and lower utilization of ophthalmic procedures. Higher volume practices and retina specialist practices had lower proportions of clinical encounters with unspecified disease severity. Results strongly suggest that DR disease severity related to ICD coding is missing not at random.

INTRODUCTION

Diabetic retinopathy (DR) is a chronic disease that represents the leading cause of blindness in the working age population.^1,2 Observational studies utilizing electronic health record (EHR) data have demonstrated a high level of agreement between International Classification of Diseases (ICD) coding and the documented clinical assessment of DR severity.^3,4 This enables researchers to use ICD codes to determine disease severity longitudinally throughout the patient journey in EHR registry studies.

In ophthalmology, the transition from ICD-9 to ICD-10 coding in October 2015 provided a significant increase in the number of codes, which increased granularity. However, ICD-10 still allows for coding unspecified DR severity. Often, clinical encounters associated with unspecified ICD codes need to be excluded in longitudinal analyses.

To facilitate more accurate interpretation of studies, it is imperative to understand the mechanism of missingness when excluding a subset of clinical encounters in observational research. The overall objective of this study was to understand whether DR severity was missing completely at random (MCAR), missing at random (MAR), missing not at random (MNAR),⁵ or some combination of these mechanisms in the EHR using data from the American Academy of Ophthalmology (Academy) IRIS^® Registry (Intelligent Research in Sight), the world’s largest ophthalmology EHR registry. We also investigated whether there were specific demographic, clinical, or practice-level characteristics associated with unspecified DR severity.

METHODS

For this analysis, we used deidentified data from the IRIS Registry, which contains over 75 million patient encounters.⁶ This study was reviewed and approved by the WCG IRB.

All clinical encounters in the IRIS Registry between January 1, 2014, and June 30, 2021, associated with a DR ICD-9 or ICD-10 code were included. Encounters were categorized as having either specified or unspecified disease severity based on the ICD code. Specifically, encounters with an ICD-9 code of 362.01 or 362.03 or an ICD-10 code of E08.31x, E08.37x, E09.31x, E09.37x, E10.31x, E10.37x, E11.31x, E11.37x, E13.31x, or E13.37x were considered unspecified encounters for all analyses.

Since DR is a progressive disease process, we established a hierarchy to choose the most severe form of DR documented at the clinical encounter to address the potential for diagnosis codes to be carried forward from prior encounters. For DR, this logic followed the order of proliferative DR, severe nonproliferative DR, moderate nonproliferative DR, mild nonproliferative DR, and unspecified DR. Thus, those encounters classified as unspecified DR had no other known stages of DR associated with that encounter. For patients with multiple clinical encounters, all qualifying encounters during the study period were included in this encounter-level analysis.

Demographic information at the time of the clinical encounter was obtained. Patient-reported sex, race, age, and ethnicity at the time of encounter were used. For statistical deidentification purposes, age at the time of encounter was defined as age from June 1 of the birth year to the encounter date. Age greater than 90 years at the time of encounter was provided as a range for deidentification purposes. Additional information on insurance type at the time of encounter was also included, if available. If multiple insurance types were reported, a hierarchy was used to choose one insurance type, which prioritized Medicare Advantage, followed by Medicare, Medicaid, commercial, military, government, miscellaneous, no insurance, and unknown. Data on prior encounters were also compared between encounters with specified and unspecified disease severity, including time (in months) since a prior encounter, if any, and whether disease severity was known at the time of prior encounter.

We evaluated clinical characteristics associated with each encounter. The visual acuity (VA) in the worse-seeing eye at the time of each encounter was included. Additionally, diagnoses of ocular comorbidities, such as glaucoma and age-related macular degeneration, were assessed based on ICD coding. Performance of specific diagnostic and therapeutic procedures (ie, fluorescein angiography, panretinal photocoagulation, and intravitreal anti-VEGF injections) that are commonly done for more severe stages of DR was determined using Current Procedural Terminology codes.

Practice- and clinician-level characteristics were also ascertained. Specifically, provider subspecialty was evaluated based on the NPI number for the clinician who billed for the encounter. Practice location was classified into geographic regions based on US census categorizations. Additionally, practices were classified into quartiles based on the proportion of DR clinical encounters with unspecified DR severity in each practice during the study period. Comparisons were done by quartile evaluating practice region, patient volume, payer mix, and duration of time contributing to the IRIS Registry.

Demographic and clinical characteristics at the time of the clinical encounter, as well as practice- and provider-level features, were compared between the specified and unspecified disease severity encounters for both ICD-9 and ICD-10 clinical encounters. The objective was to understand if DR severity is MNAR (systematic differences between observed and missing values are at least partly related to the missing values themselves), MAR (systematic differences between observed and missing values can be explained entirely by the observed values), or MCAR (no systematic difference between observed and missing values) in the EHR. Categorical variables were compared using a chi-square test, and the means of continuous variables were compared using a t-test. Analyses were performed using PySpark version 2.4.7 (Apache Spark) and RStudio version 1.4.1717.

RESULTS

Overall, we identified 10 456 243 clinical encounters with ICD-9 codes for DR and 24 077 735 clinical encounters with ICD-10 codes for DR in the IRIS Registry. This corresponded to 3 566 138 unique patients. Of these, 1 897 962 patients had ICD-9 encounters and 3 197 208 patients had ICD-10 encounters. Of the total encounters, 1 950 442 (18.65%) ICD-9 and 1 052 344 (4.37%) ICD-10 encounters had unspecified disease severity. A comparison of various demographic-, clinical-, and practice-level characteristics between unspecified and specified disease severity encounters were evaluated (Tables 1 and 2).

Table 1.

Baseline demographic characteristics of IRIS encounters with unspecified and specified diabetic retinopathy (DR) disease severity for ICD9 and ICD10 encounters

	ICD-9					ICD-10
	ICD-9 DR encounters with unspecified disease severity		ICD-9 DR encounters with specified disease severity			ICD-10 DR encounters with unspecified disease severity		ICD-10 DR encounters with specified disease severity
	N (or mean)	% (or SD)	N (or mean)	% (or SD)	P-values	N (or mean)	% (or SD)	N (or mean)	% (or SD)	P-values
N, % of total DR encounters	1 950 442	18.7%	8 505 801	81.3%		1 052 344	4.61%	23 025 391	95.4%
Age					<.005					<.005
Mean, SD	68.05	11.85	63.88	13.04		67.97	12.02	64.74	12.42
Age categories, n (%)					<.005					<.005
≤50	145 469	7.46%	1 212 972	14.26%		81 078	7.70%	2 849 121	12.37%
51–60	319 182	16.36%	1 883 250	22.14%		171 554	16.30%	4 926 939	21.40%
61–70	626 928	32.14%	2 750 095	32.33%		336 350	31.96%	7 586 053	32.95%
71–80	584 360	29.96%	1 901 374	22.35%		315 177	29.95%	5 550 838	24.11%
81+	273 403	14.02%	755 378	8.88%		147 455	14.01%	2 106 875	9.15%
Sex, n (%)					<.005					<.005
Female	1 000 308	51.29%	4 139 436	48.67%		536 817	51.01%	11 209 471	48.68%
Male	941 394	48.27%	4 321 349	50.80%		510 573	48.52%	11 700 680	50.82%
Race, n (%)					<.005					<.005
White	1 168 855	59.93%	5 214 163	61.30%		612 966	58.25%	13 737 105	59.66%
Black or African American	303 968	15.58%	1 312 661	15.43%		160 510	15.25%	3 228 836	14.02%
Asian	91 784	4.71%	291 851	3.43%		48 309	4.59%	735 839	3.20%
Native American and Alaska Native	9475	0.49%	49 130	0.58%		5789	0.55%	155 140	0.67%
Native Hawaiian and Other Pacific Islander	6513	0.33%	33 648	0.40%		3921	0.37%	78 390	0.34%
Other	62 547	3.21%	270 147	3.18%		38 246	3.63%	819 395	3.56%
Unknown	307 300	15.76%	1 334 201	15.69%		182 603	17.35%	4 270 686	18.55%
Ethnicity, n (%)					<.005					<.005
Hispanic	247 978	12.71%	1 320 465	15.52%		129 234	12.28%	3 502 601	15.21%
Non-Hispanic	1 368 278	70.15%	1 335 236	15.70%		725 674	68.96%	14 752 693	64.07%
Unknown	334 186	17.13%	5 850 100	68.78%		197 436	18.76%	4 770 097	20.72%
Payer type, n (%)					<.005					<.005
Medicare	1 093 946	56.1%	4 458 892	52.4%		586 828	55.7%	11 268 664	48.9%
Medicaid	100 922	5.17%	568 235	6.68%		53 771	5.11%	1 758 175	7.64%
Commercial	518 597	26.59%	2 417 851	28.43%		273 479	25.99%	7 005 937	30.43%
Other	67 264	3.45%	326 408	3.84%		36 908	3.51%	1 076 048	4.67%
Unknown	161 721	8.29%	693 707	8.16%		98 413	9.35%	1 793 969	7.79%
No insurance	7992	0.41%	40 708	0.48%		2945	0.28%	122 598	0.53%

Table 2.

Clinical-level characteristics of IRIS encounters with unspecified and specified DR disease severity for ICD9 and ICD10 encounters

	ICD-9					ICD-10
	ICD-9 DR encounters with unspecified disease severity		ICD-9 DR encounters with specified disease severity			ICD-10 DR encounters with unspecified disease severity		ICD-10 DR encounters with specified disease severity
	N (or mean)	% (or SD)	N (or mean)	% (or SD)	P-values	N (or mean)	% (or SD)	N (or mean)	% (or SD)	P-values
N, % of total DR encounters	1 950 442	18.7%	8 505 801	81.3%		1 052 344	4.6%	23 025 391	95.4%
Visual acuity a					<.005					<.005
Mean, SD	64.2	20.5	59.8	21.8		63.8	20.8	61.0	21.5
Visual acuity categories a					<.005					<.005
Better than 20/40	759 084	38.9%	2 963 019	34.8%		419 353	39.6%	8 402 059	36.5%
20/40–Better than 20/80	318 042	16.3%	1 751 495	20.6%		173 819	16.5%	4 459 151	19.4%
20/80–Better than 20/200	71 441	3.7%	465 308	5.5%		40 176	3.8%	1 218 823	5.3%
20/200 or worse	167 339	8.6%	1 130 748	13.3%		98 695	9.4%	2 712 857	11.8%
Unknown	634 536	32.5%	2 195 231	25.8%		320 301	30.4%	6 232 501	27.1%
Procedures a
Anti-VEGF injections	474 771	24.3%	4 229 722	49.7%	<.005	251 636	23.9%	12 789 566	55.6%	<.005
Panretinal photocoagulation	81 449	4.2%	2 145 907	25.2%	<.005	69 535	6.6%	5 530 963	24.0%	<.005
Pars plana vitrectomy	28 293	1.5%	779 134	9.2%	<.005	19 598	1.9%	1 524 677	6.6%	<.005
Optical coherence tomography	1 136 355	58.3%	6 481 352	76.2%	<.005	626 718	59.6%	18 831 674	81.8%	<.005
B scan ultrasonography	1366	0.1%	11 889	0.1%	<.005	1291	0.1%	29 142	0.1%	<.005
Fluorescein angiography	520 948	26.7%	4 040 143	47.5%	<.005	235 618	22.4%	10 661 885	46.3%	<.005
Comorbidities a
Glaucoma	625 147	32.1%	2 657 991	31.3%	<.005	422 994	40.2%	7 999 943	34.7%	<.005
Cataract	1 239 476	63.6%	5 698 303	66.9%	<.005	725 842	68.9%	16 620 837	72.2%	<.005
Retinal vein occlusion	104 097	5.3%	455 301	5.4%	.38	60 817	5.8%	1 314 363	5.7%	<.005
Exudative age-related macular degeneration	111 299	5.7%	440 761	5.2%	<.005	57 267	5.4%	1 174 008	5.1%	<.005
Nonexudative age-related macular degeneration	212 233	10.9%	687 521	8.1%	<.005	126 105	11.9%	2 049 208	8.9%	<.005
Diabetic macular edema	791 612	40.6%	5 450 188	64.1%	<.005	460 805	43.8%	16 278 357	70.7%	<.005
Encounter characteristics
Evaluated by retina specialist	820 958	42.1%	6 447 495	75.8%	<.005	347 563	33.0%	15 574 829	67.64%	<.005
Months since last encounter (mean, SD)	4.23	7.3	2.80	4.9	<.005	4.09	6.65	3.11	5.4	<.005
Disease severity known at prior encounter (n, %)	115 864	5.9%	7 053 442	82.9%	<.005	113 333	10.8%	19 942 578	86.6%	<.005
First encounter (n, %)	616 808	31.6%	1 262 723	14.9%	<.005	308 577	29.3%	2 841 244	12.3%	<.005

^aClinical characteristic determined at the time of clinical encounter.

On average, patients with clinical encounters with specified disease severity were significantly younger at the time of the encounter than those with unspecified disease severity (ICD-9: 64 [SD: 13 years] vs 68 years [SD: 12 years], P < .005; ICD-10: 65 [SD: 12 years] vs 68 years [SD: 12 years], P < .005). Additionally, VA at the time of encounter was significantly worse for encounters with specified disease severity (ICD-9: 59.8 vs 64.2 ETDRS letters, P < .005; ICD-10: 61.0 vs 63.8 ETDRS letters, P < .005). Since a 5-letter difference is considered clinically significant, this was a clinically significant difference for the ICD-9 encounters. Diabetic macular edema was much more commonly associated with specified encounters compared to unspecified encounters (ICD-9: 64.1% vs 40.6%, P < .005; ICD-10: 70.7% vs 43.8%, P < .005).

All diagnostic and therapeutic procedures evaluated were performed at higher rates during clinical encounters with specified disease severity. Retina specialists were found to be the evaluating physician at a significantly higher percentage of specified encounters compared to unspecified encounters (ICD-9: 75.8% vs 42.1%, P < .005; ICD-10: 67.6% vs 33.0%, P < .005).

We did additional analyses to understand differences in encounter history between specified and unspecified disease severity encounters. A significantly lower percentage of specified disease severity encounters was the first clinical encounter for a patient compared to unspecified encounters (ICD-9: 14.9% vs 31.6%, P < .005; ICD-10: 12.3% vs 29.3%, P < .005). Similar trends were seen for both ICD-9 and ICD-10 encounters.

We did a practice-level analysis on all active 2923 ophthalmology practices with DR encounters for ICD-10 between 2016 and 2021. Practices were divided into quartiles based on the percentages of overall encounters for the practice during the study period that had unspecified DR severity. Quartile 1 had 731 practices, quartile 2 had 732 practices, and quartiles 3 and 4 each had 730 practices. The median proportion of encounters with unspecified disease severity in quartiles 1, 2, 3, and 4 were 0.25%, 1.33%, 6.88%, and 20.47%, respectively.

There was a strong association at the practice level between the proportion of unspecified encounters and the average number of DR patients seen per year. Practices with higher annual practice volumes tended to have lower proportions of encounters with unspecified disease severity (Figure 1). We did additional analyses to evaluate the proportion of clinicians who were retina specialists in each quartile. The quartile of practices with the lowest proportion of unspecified encounters consisted of 77.3% retina specialists, while the quartile with the highest proportion of unspecified encounters consisted of 31.6% retina specialists.

Figure 1.

The relationship between the proportion of unspecified diabetic retinopathy (DR) encounters and practice volumes in the IRIS Registry between 2016 and 2021. Scatterplot with overlying LOWESS line displaying the practice-level relationship between proportion of unspecified diabetic retinopathy (DR) encounters over the study period and average number of annual DR encounters for the practice over the study period. Because the study period ended on June 30, 2021, averages were calculated assuming a 5.5-year study period. Practices with lower proportions of unspecified DR encounters were associated with higher annual volume of DR encounters.

DISCUSSION

We evaluated differences between clinical encounters with specified and unspecified DR severity using over 35 million clinical encounters from the IRIS Registry with the goal of understanding whether disease severity was MCAR, MAR, or MNAR. We found differences between encounters with specified and unspecified disease severity. These differences are clinically meaningful because several clinical characteristics that occurred in a higher proportion of clinical encounters with specified disease severity are associated with more advanced DR severity and suggest that DR severity is MNAR in the EHR.

Specifically, our results demonstrate that a higher proportion of clinical encounters with specified disease severity are associated with anti-VEGF injections and panretinal photocoagulation compared to encounters with unspecified disease severity. These procedures are performed for more severe stages of DR,⁷ and in some cases, accurate coding of disease stage may be required for reimbursement for the procedure. Additionally, the progression of retinopathy is associated with vision loss.⁸ The mean VA for encounters with specified disease severity was significantly lower than the mean VA for encounters with unspecified disease severity. This supports the assessment that clinical encounters with unspecified disease severity are associated with less severe DR severity.

The Academy Preferred Practice Pattern^® for DR recommends that patients with vision-threatening DR, defined as diabetic macular edema, severe nonproliferative DR, or proliferative DR, be referred to a retina specialist.⁷ Our results demonstrated a significantly higher proportion of clinical encounters with specified disease severity were completed by a retina specialist compared to clinical encounters with unspecified disease severity. This also suggests that specified DR severity is associated with more severe disease.

On a practice level, we also found a significant association between practice volume and the proportion of clinical encounters with unspecified disease severity. Specifically, higher volume practices had lower proportions of clinical encounters with missing disease severity. There may be several reasons for this. Higher volume practices may have scribes or experienced coders to help with billing. Additionally, many of the practices with higher proportion of specified disease severity were retina specialist practices. Because retina specialists are more likely to see patients with more advanced DR that necessitate therapeutic procedures, they may be more likely to specify disease severity for reimbursement purposes. Further, we noted that practices in the quartiles with higher proportions of specified disease severity clinical encounters saw each patient with DR on average more times each year. Our encounter-level analysis demonstrated that initial encounters were more likely to be unspecified, also suggesting that practices that see patients more frequently may have a higher proportion of clinical encounters with specified disease severity.

In the ophthalmology literature, prior studies have focused on the accuracy of coding for clinical encounters when disease severity is specified.^3,4 However, there is limited understanding of DR clinical encounters with unspecified disease severity. This study shows that they account for almost 20% of ICD-9 clinical encounters, making the findings useful for future observational research using EHR registry data. While coding validation studies are also common in other areas of ophthalmology, these studies do not evaluate patterns in missing data for these other disease processes.^9–11

Although this study employed rigorous methodology, there are some limitations. For example, differences in practice workflow or EHR user interfaces may impact coding practices that we were not able to ascertain in our analyses. Given the tremendous size of the dataset, it is likely that even small differences in demographic and clinical characteristics between clinical encounters with unspecified and specified disease severity would be statistically significant. However, our analysis and interpretation of results focused on differences with a large effect size that are considered clinically significant. Additionally, our analyses were focused on DR, and these results may not be generalizable to other ophthalmic conditions, particularly those without therapeutic interventions that necessitate coding of a specific disease severity. Nonetheless, DR is the leading cause of blindness in the working-age population and a common topic for real-world evidence studies, making these results helpful for understanding missing disease severity in many studies.^1,2,12–15

This study also has several strengths. The size of the IRIS Registry allowed for meaningful comparisons across multiple structured data elements including demographic, procedure, and ocular comorbidity data. Similarly, the opportunity to use data from various EHRs and practice environments allowed for the analysis of practice-level characteristics that may be associated with coding unspecified disease severity. Because the IRIS Registry captures over 70% of practicing ophthalmologists in the United States,⁶ these results are generalizable to studies using various EHR registries in ophthalmology. This has important implications for other DR real-world evidence studies, particularly because clinical encounters with unspecified disease severity are often excluded from observational research studies.

Furthermore, the results of this study provide important insights to guide future investigations aimed at imputing missing DR disease severity. Understanding the associations between DR severity and other clinical and demographic characteristics will help investigators build more accurate predictive models. The current results identifying the mechanism of missingness for DR severity as MNAR will ground the evaluation of any postimputation distribution in a more nuanced context.

In summary, this study found that DR severity based on ICD-9 and ICD-10 billing diagnoses is likely MNAR in ophthalmology clinical encounters. Our analysis of over 35 million clinical encounters demonstrated that clinical encounters associated with better VA and fewer procedures had a higher proportion of unspecified DR severity. These findings suggest that unspecified disease severity is likely associated with less severe DR. These clinical encounters remain an important part of observational research studies as they capture a key part of the patient journey. Future studies could investigate the ability to impute DR severity using other structured data associated with the clinical encounter, such as demographic and clinical procedure information.

FUNDING

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

AUTHOR CONTRIBUTIONS

All authors contributed to the conception and design of the study. The acquisition and analysis of data was done by MH, THN, and RC. The interpretation of data was done by MH, THN, RC, and DB. Authors DB, THN, and MH substantially contributed to the drafting of the article, and all authors contributed to revising the manuscript critically. All authors gave final approval of the version of the article to be published. All authors agree to be accountable for all aspects of the work.

CONFLICT OF INTEREST STATEMENT

TL has grant funding from Astellas; is on the Advisory Board of Apellis and Regeneron; and receives consulting fees from Roche/Genentech, Protagonist Therapeutics, Nanoscope Therapeutics, Verana Health, and Graybug. DB receives consulting fees from Allergan/AbbVie, Glaukos, Iveric Bio, and Verana Health. The other authors are employees of Verana Health.

DATA AVAILABILITY

This was a retrospective study using data from the American Academy of Ophthalmology's IRIS Registry. Interested parties can find more information about data access here: https://www.aao.org/iris-registry/data-analysis/requirements.