Abstract
Objective
To develop a novel methodology to identify lapses in diabetic retinopathy care in electronic health records (EHRs) and evaluate health disparities by race and ethnicity.
Design
Retrospective cohort study.
Subjects
Adult patients with diabetes mellitus who were evaluated at the Wilmer Eye Institute from January 1, 2013 to April 2, 2022.
Methods
The methodology to identify lapses in care first identified diabetic retinopathy screening or treatment visits and then compared the providers’ recommended follow-up timeframe with the patient’s actual time to next encounter. The association of race and ethnicity with odds of lapses in care was evaluated using a mixed-effects logistic regression model controlling for age, sex, insurance, severity of diabetic retinopathy, presence of other retinal disorders, and glaucoma.
Main Outcome Measures
Lapses in diabetic retinopathy care.
Results
The methodology to identify diabetic retinopathy-related visits had a 95.0% (95% confidence interval, 93.0–96.6) sensitivity and 98.8% (98.1–99.3) specificity as compared with a gold standard grader. The methodology resulted in a 97.3% (96.2–98.4) sensitivity and 98.1% (97.3–98.9) specificity for detecting a follow-up recommendation, with an average error of −0.05 (−0.31 to 0.21) weeks in extracting the precise timeframe. A total of 39 561 patients with 91 104 office visits were included in the analysis. The average age was 61.4 years. More than 3 (77.6%) in 4 patients had a lapse in care. In multivariable analysis, non-Hispanic Black patients had 1.24 (1.19–1.30) odds and Hispanic patients had 1.26 (1.13–1.40) odds of ever having a lapse in care compared with non-Hispanic White patients (P < 0.001, respectively).
Conclusions
We have developed a reliable methodology for identifying lapses in diabetic retinopathy care that is tailored to a provider’s recommended follow-up. Using this approach, we find that 3 in 4 patients experience a lapse in diabetic retinopathy care and that these rates are higher among non-Hispanic Black and Hispanic patients. Deploying this methodology in the EHR is one potential means by which to identify and mitigate lapses in critical ophthalmic care in patients with diabetes.
Financial Disclosure(s)
Proprietary or commercial disclosure may be found after the references.
Keywords: Diabetic retinopathy, Health disparities, Lapses in care
Vision loss from diabetes mellitus can result from lapses in diabetic retinopathy care, which can lead to delays or interruptions in ophthalmic treatment.1, 2, 3, 4, 5, 6, 7, 8, 9 Racial and ethnic minoritized groups, including Black and Hispanic populations, have higher rates of vision loss from diabetes when compared with White patients.10 However, studies have found conflicting results on the association among race, ethnicity, and risk of lapses in diabetic retinopathy care.1,5,9 Obeid et al5 found higher rates of lapses in diabetic retinopathy care among Black and Hispanic populations as compared with White patients in a large retrospective cohort study. However, no differences were identified by race and ethnicity in another study at a similarly large urban institution.9 In the context of a clinical trial, non-White individuals, including Black, Hispanic, and other race populations, had a higher proportion of lapses in care compared with White participants; however, race and ethnicity was no longer statistically significantly associated with care lapses after adjusting for other baseline demographic and ophthalmic characteristics.1
Differences in conclusions between these studies are likely because of variations in the context of the analysis, whether a clinical trial or observational study, and the specific definition of lapse in care. Most observational studies in clinical settings apply a singular definition for lapses in care for diabetic retinopathy (e.g., not returning in 6 months or 1 year).3, 4, 5, 6,8,9 However, the typical clinical trial uses an adaptive definition that accommodates the providers’ recommended follow-up (e.g., not returning in 10 weeks if the provider recommended an 8-week follow-up, or 20 weeks for a 16-week follow-up).1 Tailoring the definition of a lapse in care to the providers’ recommendation has several advantages, including that it considers the underlying urgency of follow-up and potential risk for vision loss.1,7 Because the risk of vision loss can be proportional to the number of missed days, lapses in care should be identified earlier in those with more urgent follow-up needs.7
Methods are needed to study lapses in care in the clinical practice setting that incorporate the providers’ recommended follow-up. Therefore, there were 2 purposes of this study. The first was to develop a methodology for identifying lapses in diabetic retinopathy care in the electronic health record (EHR) that incorporates providers’ recommendations and is scalable across the range of diabetic eye disease complications. The second was to use this methodology to evaluate racial and ethnic disparities in lapses in diabetic retinopathy care in the practice of ophthalmology.
Methods
Study Design/Patient Selection
This was a retrospective cohort study of adult patients aged > 18 years with diabetes mellitus seen at the Wilmer Eye Institute at Johns Hopkins Hospital from January 1, 2013 to April 2, 2022. Data were collected and patients were screened for eligibility. Diabetes mellitus was defined as having a qualifying International Classification of Diseases (ICD) code (Table S1, available at www.ophthalmologyscience.org) at any hospital encounter or a hemoglobin A1c value of ≥ 6.5%. This sensitive definition of diabetes mellitus was used to include as many patients as possible in this first step. Patients were included if they had ≥ 1 completed office visit with a Wilmer provider either for diabetic retinopathy screening or treatment with the first one occurring between January 1, 2013 and April 2, 2020. Patients were excluded if they had missing sociodemographic data or ophthalmic diagnoses (N = 6) (Fig 1). Office visits from January 1, 2013 to April 2, 2022 in the first 2 years after patient’s entry into the cohort were analyzed to standardize the observation period and opportunity for lapse in care. Office visits were excluded if providers’ recommended follow-up timeframe was missing or occurred after 52 weeks (see step B of the pipeline for identifying lapses in diabetic retinopathy care). This study was approved by the institutional review board of the Johns Hopkins University School of Medicine and adhered to the tenets of the Declaration of Helsinki.
Figure 1.
Flow diagram for identification of adult patients with diabetes and the diabetes-related office visits in the electronic health record.
Methodology for Identifying Lapses in Diabetic Retinopathy Care in the EHR
Defining lapses in diabetic retinopathy care in the EHR involved the following 3 major components (Fig 2): (1) office visits related to the screening or treatment of diabetic retinopathy were identified and isolated; (2) the providers’ recommended follow-up timeframe was extracted; and (3) the providers’ recommended follow-up timeframe was compared with the patient’s actual time to the next office visit. A patient was defined as having a lapse in care after an office visit if they returned beyond the preset threshold (Table S2, available at www.ophthalmologyscience.org). This multistage approach was adopted because of the complexities of utilizing EHR data and the specialization of eye care in our ophthalmology department. Some providers only address and follow specific ophthalmic issues (e.g., glaucoma and cataracts). Thus, to define a lapse in diabetic retinopathy care in a multidisciplinary ophthalmology department, we had to isolate the relevant visits first and identify the follow-up specific to diabetic retinopathy care.
-
1.
Office visits related to the screening or treatment of diabetic retinopathy were identified using a combination of ICD codes and a rule-based natural language processing (NLP) algorithm. This hybrid approach was taken because the sensitivity of ICD codes to identify clinical phenotypes (such as diabetic retinopathy) can be limited; 29.0% of visits analyzed in our study were identified using only the NLP algorithm.11 Office visits were included if there was an encounter ICD code related to diabetes mellitus (Table S1). The rule-based NLP algorithm was applied on the clinical progress notes and used to identify additional office visits (Supplemental NLP Algorithm Material, available at www.ophthalmologyscience.org). The re Python package was used to search for a list of target terms in the provider notes to produce a binary output regarding the presence of diabetic retinopathy screening or treatment. The list of target terms was fine-tuned through an iterative process, qualitatively reviewing outputs on a development sample of data and then updating the target term list until a sufficient level of performance was achieved. Quantitative validation of the NLP pipeline was performed on a separate held-out sample of test data when the fine-tuning process was complete.
-
2.
The providers’ recommended follow-up timeframe was also extracted using a hybrid approach combining structured data available in the check-out section of the EHR and a custom rule-based NLP algorithm applied to the unstructured provider notes. Similar to the aforementioned extraction task, this hybrid approach was needed because most office visits (77.6%) did not contain the necessary follow-up in the structured check-out section, and this information was only available in unstructured provider notes. The algorithm leverages a dictionary of target terms and character patterns to flag the portion of the note that is likely to be immediately preceded or followed by the recommended return time interval. Additional logic was introduced to search through the provider notes in a specific sequence based on prior clinical knowledge regarding the location of follow-up recommendations in a provider note. When available, the recommended follow-up timeframe was extracted and numerically standardized. The list of target terms, character patterns, and processing logic were iteratively fine-tuned by qualitative review and evaluation of the algorithm’s performance on the same development set as above (Supplemental NLP Algorithm Material). The NLP pipeline was quantitatively validated using a held-out sample of test data.
-
3.
The providers’ recommended follow-up timeframe was compared with the patients’ time to the next diabetic retinopathy screening or treatment office visit. A lapse in care was defined as not returning within certain timeframes that varied based on provider recommendations (Table S2). These thresholds generally correspond to ∼ 25% of the provider’s recommended timeframe (e.g., threshold of 2 weeks for an 8-week recommended follow-up, 4 weeks for a 20-week follow-up, 8 weeks for a 32-week follow-up, and 12 weeks for a 52-week follow-up) and were developed based on the existing literature and expert guidance (C.X.C.).1 If the threshold to return fell beyond the 2-year observation window, then that encounter was categorized as being unknown for lapse in care (Fig 2).
Figure 2.
Methodology for identifying lapses in diabetic retinopathy care in the electronic health record (EHR). After the initial cohort selection, the ophthalmology office visits related to diabetic retinopathy screening and treatment were isolated (step A), the providers’ recommended follow-up timeframe was identified (step B), the time to the patient’s next office visit was calculated (step C), and the thresholds for defining lapses in diabetic retinopathy care were applied (final). A sample patient’s timeline of different office visits in the EHR is shown on the right with examples of which visits were considered to have lapses in diabetic retinopathy care.
Validation of Rule-Based NLP Algorithms
Task 1: automated detection of diabetic retinopathy screening or treatment from provider notes (step A of the pipeline)
The NLP algorithm was evaluated using a held-out test set of 2000 randomly selected notes sampled from the initial set of patients with diabetes mellitus. The notes were annotated by 2 expert graders with extensive clinical experience (C.X.C., a fully trained vitreoretinal specialist; and T.T., a postgraduate year 4 ophthalmology resident). The graders assessed whether the note text and ICD codes addressed diabetic retinopathy screening or treatment and annotated the document with a binary label. Performance of the NLP algorithm was compared with the gold standard grader (C.X.C.), and sensitivity and specificity were calculated. The 95% confidence interval (CI) for both metrics were estimated via the bootstrap algorithm in which the test set was resampled 1000 times with replacement, and the sensitivity and specificity calculated and cached for each sample.12 Agreement between the 2 graders was assessed using the Cohen’s κ statistic. Kappa statistic < 0.00 indicates poor agreement, 0.00 to 0.20 slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, and 0.81 to 1.00 almost perfect agreement.13
Task 2: automated detection of providers’ recommended follow-up timeframe (step B of the pipeline)
The NLP algorithm for this task was evaluated on the same test set of 2000 notes annotated by the same graders. The graders first annotated the document with a binary label indicating whether a provider recommended follow-up. When present, the graders extracted the timeframe of the providers’ recommended follow-up from the note text. The timeframe was normalized into week-based units for ease of comparison (e.g., 1 day = 0.14 weeks, 1 week = 7 days, 1 month = 4 weeks, and 1 year = 52 weeks). If a range was provided in the note (e.g., the provider wanted the patient to return in 4–6 weeks), then the larger value was used in the analysis (e.g., 6 weeks).
The algorithm was evaluated based on coverage (i.e., correctly identifying the presence of a recommended follow-up) and precision (i.e., correctly identifying the follow-up timeframe when present). For coverage, the sensitivity and specificity of the NLP algorithm was compared with the gold standard grader and the 95% CI calculated using bootstrapping with 1000 resamples.12 The agreement between the 2 graders was evaluated using Cohen’s κ.13 The precision of the algorithm was evaluated using the average error (e.g., gold standard grader minus the NLP algorithm or second grader) and average absolute error. Agreement on precision between the 2 graders and the NLP algorithm was also assessed qualitatively using the Bland-Altman plot.
Statistical Analysis
Differences in sociodemographic characteristics were assessed between patients who had ever had a lapse in diabetic retinopathy care and those who did not. These sociodemographic characteristics included the following: age, sex, race and ethnicity, and insurance. Continuous variables were compared using Student t test, and categorical variables using the Pearson’s chi-square test. A mixed-effects logistic regression model with random intercepts at the person level was constructed to evaluate the association of race and ethnicity on lapses in diabetic retinopathy care, controlling for the other sociodemographic characteristics, severity of diabetic retinopathy (divided into no diabetic retinopathy, nonproliferative diabetic retinopathy, and proliferative diabetic retinopathy on the basis of ICD codes), other retinal disorders (including age-related macular degeneration, vein occlusion, and artery occlusion), and glaucoma14 (Table S3, available at www.ophthalmologyscience.org). The mixed-effects model with a random intercept for person effect was chosen to account for multiple office visits per patient and differences in the number of available office visits per patient. Conditional odds ratios were transformed into marginal odds ratios for reporting.15 Statistical significance was set at P value of ≤ 0.05. All analyses were conducted in Python (Python Software Foundation, Python Language Reference, version 3.8.9) and Stata statistical software (Version 17.0 for Windows; StataCorp LLC).
Sensitivity Analyses
A sensitivity analysis was performed on the subcohort of patients with diabetes mellitus initially evaluated in the ophthalmology department for diabetic retinopathy screening and treatment from January 1, 2016 to December 31, 2017, with the end of the observation window occurring on December 31, 2019. This analysis was conducted to avoid any potential effects because of the transition from ICD9 to the more complex ICD10 which occurred in October 2015 and the significant interruptions in ophthalmic care that resulted from the coronavirus disease 2019 pandemic.11,16
Results
Validation of Rule-Based NLP Algorithms
Task 1: automated detection of diabetic retinopathy screening or treatment from provider notes: the sensitivity of the NLP algorithm for detecting diabetic retinopathy screening or treatment was estimated to be 95.0% (95% CI, 93.0–96.6), and specificity was estimated to be 98.8% (95% CI, 98.1–99.3). The κ statistic for agreement between the 2 graders was 0.973 (standard error 0.022, P < 0.001).
Task 2: automated detection of providers’ recommended follow-up timeframe: a total of 810 out of 2000 notes contained a follow-up timeframe identified by either the NLP algorithm or gold standard grader. The estimated sensitivity of the algorithm was 97.3% (95% CI, 96.2–98.4) and specificity 98.1 (95% CI, 97.3–98.9) for detecting the presence of a follow-up timeframe. Eight hundred seven notes were identified as containing a follow-up timeframe by either of the 2 graders. The κ statistic for agreement between the 2 graders was 0.97 (standard error 0.02, P < 0.001). In the 767 notes in which both the NLP algorithm and gold standard grader detected a follow-up timeframe, the average error was −0.05 (95% CI, −0.31 to 0.21) weeks, whereas the average absolute error was 0.42 (95% CI, 0.20–0.72) weeks. In the set of 782 notes in which the 2 graders detected a follow-up timeframe, the average error was −0.09 (95% CI, −0.23 to 0.08) weeks and average absolute error 0.32 (95% CI, 0.17–0.52) weeks. Bland-Altman plots comparing the NLP algorithm and gold standard grader and between the 2 graders are shown in Figure S3 (available at www.ophthalmologyscience.org). Overall, most differences fell between the 95% limits of agreement, suggesting that there were no systematic biases in the NLP algorithm.
Lapses in Care and Racial and Ethnic Disparities Results
A total of 39 561 patients with 91 104 office visits related to diabetic retinopathy screening and treatment were included in this study (Fig 1). Overall, 46.9% of patients were non-Hispanic White, 36.7% non-Hispanic Black, 4.8% Hispanic, and 11.6% other race and ethnicity. The majority of patients were between ages of 45 and 65 (average, 61.4, range, 18–100 years) (Table 4). Most patients were women, had private or Medicare insurance, and did not have diabetic retinopathy, other retinal diseases, or glaucoma.
Table 4.
Demographic Characteristics of Patients with Diabetes Mellitus Seen in the Ophthalmology Department for Diabetic Retinopathy Screening or Treatment Stratified by Race and Ethnicity (N = 39 561)
| Total N = 39 561 (%) |
Non-Hispanic White N = 18 565 (%) |
Non-Hispanic Black N = 14 502 (%) |
Hispanic N = 1912 (%) |
Other N = 4582 (%) |
P | |
|---|---|---|---|---|---|---|
| Age (yrs) | < 0.001 | |||||
| ≤ 20 | 104 (0.3) | 53 (0.3) | 34 (0.2) | 8 (0.4) | 9 (0.2) | |
| > 20 to ≤ 45 | 4697 (11.9) | 1516 (8.2) | 2020 (13.9) | 552 (28.9) | 609 (13.3) | |
| > 45 to ≤ 65 | 18 880 (47.7) | 7863 (42.4) | 7838 (54.0) | 950 (49.7) | 2229 (48.6) | |
| > 65 | 15 880 (40.1) | 9133 (49.2) | 4610 (31.8) | 402 (21.0) | 1735 (37.9) | |
| Sex | < 0.001 | |||||
| Female | 20 635 (52.2) | 8608 (46.4) | 8754 (60.4) | 962 (50.3) | 2311 (50.4) | |
| Male | 18 926 (47.8) | 9957 (53.6) | 5748 (39.6) | 950 (49.7) | 2271 (49.6) | |
| Insurance | < 0.001 | |||||
| Private | 14 838 (38.1) | 7530 (40.8) | 4914 (34.4) | 502 (29.0) | 1892 (42.3) | |
| Medicare | 15 694 (40.3) | 8352 (45.3) | 5554 (38.8) | 366 (21.1) | 1422 (31.8) | |
| Medicaid | 4214 (10.8) | 895 (4.9) | 2613 (18.3) | 212 (12.2) | 494 (11.0) | |
| Other | 3164 (8.1) | 1561 (8.5) | 936 (6.5) | 111 (6.4) | 556 (12.4) | |
| None | 1045 (2.7) | 111 (0.6) | 283 (2.0) | 543 (31.3) | 108 (2.4) | |
| Diabetic retinopathy (DR) | < 0.001 | |||||
| No DR | 31 331 (79.2) | 15 006 (80.8) | 11 238 (77.5) | 1493 (78.1) | 3594 (78.4) | |
| Nonproliferative diabetic retinopathy (NPDR) | 5957 (15.1) | 2679 (14.4) | 2251 (15.5) | 298 (15.6) | 729 (15.9) | |
| Proliferative diabetic retinopathy (PDR) | 2273 (5.7) | 880 (4.7) | 1013 (7.0) | 121 (6.3) | 259 (5.7) | |
| Other retinal disorders | < 0.001 | |||||
| Absent | 37 880 (95.8) | 17 399 (93.7) | 14 195 (97.9) | 1881 (98.4) | 4405 (96.1) | |
| Present | 1681 (4.2) | 1166 (6.3) | 307 (2.1) | 31 (1.6) | 177 (3.9) | |
| Glaucoma | < 0.001 | |||||
| Absent | 33 551 (84.8) | 16 474 (88.7) | 11 485 (79.2) | 1694 (88.6) | 3898 (85.1) | |
| Present | 6010 (15.2) | 2091 (11.3) | 3017 (20.8) | 218 (11.4) | 684 (14.9) | |
| Total office visits, mean (SD) | 2.3 (2.6) | 2.3 (2.7) | 2.3 (2.5) | 2.2 (2.6) | 2.1 (2.6) | < 0.001 |
| 1 office visit | 17 883 (45.2) | 7794 (42.0) | 6842 (47.2) | 940 (49.2) | 2307 (50.3) | < 0.001 |
| 2 office visits | 13 294 (33.6) | 6752 (36.4) | 4488 (30.9) | 622 (32.5) | 1432 (31.3) | |
| 3+ office visits | 8384 (21.2) | 4019 (21.6) | 3172 (21.9) | 350 (18.3) | 843 (18.4) | |
| Ever have a lapse in care | 30 707 (77.6) | 13 492 (72.7) | 11 905 (82.1) | 1577 (82.5) | 3733 (81.5) | < 0.001 |
SD = standard deviation.
The proportion of office visits with subsequent lapses in care varied by provider recommended follow-up timeframe (Table 5). When the recommended follow-up was < 8 weeks, 29.9% of office visits were followed by a lapse in care; with 32 to 52 weeks recommended, the lapse rate was 42.7%.
Table 5.
Proportion of Office Visits Followed by Lapses in Diabetic Retinopathy Care
| Providers’ Recommended Follow-up Timeframe | Total Number of Office Visits | Proportion of Office Visits with Lapses in Care |
|---|---|---|
| ≤ 8 wks | 27 231 | 29.9% |
| > 8 to ≤ 20 wks | 8799 | 37.9% |
| > 20 to ≤ 32 wks | 10 636 | 37.1% |
| > 32 to ≤ 52 wks | 44 348 | 42.7% |
| Total | 91 104 | 37.7% |
On average, patients had 2.3 office visits in the 2-year observation window. More than 3 (77.6%) in 4 patients had a lapse in their diabetic retinopathy care at least once during this time (Table 6). When stratified by the providers’ recommended follow-up, the rate of any lapse in care decreased with the scheduled time until the next visit. For example, when the recommendation was ≥ 8 weeks, we estimate a lapse in care rate of 96.0%; when between 32 and 52 weeks to the recommended visit, 71.5% had a lapse in care (Table 6).
Table 6.
Proportion of Patients with Diabetes Mellitus Who Have Ever Had a Lapse in Diabetic Retinopathy Care and Proportion of Office Visits per Patient Categorized as a Lapse in Care. Both Outcomes Are Stratified by Median Provider Recommended Follow-up Timeframe and Total Number of Visits in the 2-year Observation Window
| Recommended Follow-up Timeframe by the Provider | Number of Office Visits | Proportion of Patients Who Ever Had a Lapse in Care | Average Proportion of Office Visits with Lapses in Care |
|---|---|---|---|
| ≤ 8 wks | 1 | 100.0% | 100.0% |
| 2 | 100.0% | 67.6% | |
| 3+ | 92.2% | 28.1% | |
| Total | 96.0% | 59.4% | |
| > 8 to ≤ 20 wks | 1 | 100.0% | 100.0% |
| 2 | 100.0% | 69.7% | |
| 3+ | 83.2% | 29.0% | |
| Total | 92.8% | 61.9% | |
| > 20 to ≤ 32 wks | 1 | 100.0% | 100.0% |
| 2 | 100.0% | 59.9% | |
| 3+ | 56.9% | 19.7% | |
| Total | 84.1% | 58.9% | |
| > 32 to ≤ 52 wks | 1 | 100.0% | 100.0% |
| 2 | 48.2% | 24.2% | |
| 3+ | 24.6% | 8.1% | |
| Total | 71.5% | 60.2% | |
| Total | 1 | 100.0% | 100.0% |
| 2 | 58.0% | 31.8% | |
| 3+ | 61.0% | 19.8% | |
| Total | 77.6% | 60.1% |
On the patient-level, the proportion of office visits for each patient categorized as a lapse in care averaged 60.1% (Table 6). This value did not vary much when stratified by the providers’ follow-up timeframe but differed significantly depending on the total number of office visits in the 2-year observation window. By definition, 100.0% of office visits among patients who had only the index visit during the 2-year window were categorized as lapses in care. The lapses in care decreased to 31.8% among those with 2 visits and 19.8% among those with ≥ 3 visits (Table 6). Patients who only had 1 office visit in the 2-year observation period, compared with those who had ≥ 2 office visits, were younger (mean, 60.1 years compared with 62.5 years; P < 0.001), more frequently non-Hispanic Black or Hispanic persons (43.6% compared with 39.8%; P < 0.001), and had Medicaid or no insurance (15.9% compared with 11.5%; P < 0.001). (Table S7, available at www.ophthalmologyscience.org).
In multivariable analysis, after controlling for age, sex, severity of diabetic retinopathy, other retinal disorders, and glaucoma, non-Hispanic Black patients were estimated to have 24% increased odds of ever having a lapse in care compared with non-Hispanic White patients. Hispanic patients had an estimated 26% increased odds, and patients of other race/ethnicity had an estimated 38% increased odds (P < 0.001, respectively) as compared with non-Hispanic White patients (Table 8). Non-Hispanic Black patients had similar odds for having a lapse in care compared with Hispanic patients, but decreased odds compared with those of other race/ethnicity (P < 0.05). Hispanic patients had similar odds of having a lapse in care compared with patients with other race/ethnicity.
Table 8.
Adjusted Odds Ratio of Having a Lapse in Diabetic Retinopathy Care, Results from the Multivariable Mixed-Effects Logistic Regression Model with a Random Person Effect (N = 39 561)
| Ever Have a Lapse in Care N (%) |
Adjusted Odds Ratio∗ (95% Confidence Interval) | P | |
|---|---|---|---|
| Age (yrs) | |||
| ≤ 20 | 87 (83.7) | (reference) | |
| > 20 to ≤ 45 | 3960 (84.3) | 0.68 (0.42–1.08) | 0.103 |
| > 45 to ≤ 65 | 14 971 (79.3) | 0.50 (0.31– 0.8) | 0.004 |
| > 65 | 11 689 (73.6) | 0.38 (0.24–0.61) | < 0.001 |
| Sex | |||
| Female | 16 169 (78.4) | (reference) | |
| Male | 14 538 (76.8) | 0.97 (0.93–1.01) | 0.103 |
| Race and ethnicity | |||
| Non-Hispanic White | 13 492 (72.7) | (reference) | |
| Non-Hispanic Black | 11 905 (82.1) | 1.24 (1.19–1.30) | < 0.001 |
| Hispanic | 1577 (82.5) | 1.26 (1.13–1.40) | < 0.001 |
| Other | 3733 (81.5) | 1.38 (1.29–1.47) | < 0.001 |
| Insurance | |||
| Private | 11 478 (77.4) | (reference) | |
| Medicare | 11 888 (75.7) | 1.05 (0.99–1.10) | 0.096 |
| Medicaid | 3693 (87.6) | 1.26 (1.18–1.35) | < 0.001 |
| Other | 2208 (69.8) | 0.81 (0.75–0.87) | < 0.001 |
| None | 893 (85.5) | 1.46 (1.27–1.68) | < 0.001 |
| Diabetic retinopathy (DR) | |||
| No DR | 23 895 (76.3) | (reference) | |
| Nonproliferative diabetic retinopathy (NPDR) | 4879 (81.9) | 0.47 (0.45–0.49) | < 0.001 |
| Proliferative diabetic retinopathy (PDR) | 1933 (85) | 0.31 (0.29–0.33) | < 0.001 |
| Other retinal disorders | |||
| Absent | 29 342 (77.5) | (reference) | |
| Present | 1365 (81.2) | 0.76 (0.70–0.82) | < 0.001 |
| Glaucoma | |||
| Absent | 25 691 (76.6) | (reference) | |
| Present | 5016 (83.5) | 1.18 (1.12–1.24) | < 0.001 |
Marginal odds ratios are presented
Older patients, those aged > 45 to ≤ 65 and > 65 years, had lower odds of ever having a lapse in care compared with those aged ≤ 20 years. Sex was not associated with lapses in care. Having Medicaid or no insurance was associated with higher odds of lapses in care, and having other insurance was associated with lower odds. Patients with diagnosed diabetic retinopathy had lower odds of having a lapse in care; those with nonproliferative diabetic retinopathy were estimated to have a 53% decreased odds, and those with proliferative diabetic retinopathy had an estimated 69% decreased odds (P < 0.001, respectively) compared with patients without diabetic retinopathy. A diagnosis of other retinal disorders, including age-related macular degeneration, vein occlusion, and artery occlusion, was also associated with an estimated 24% lower odds of ever having a lapse in care. Finally, having glaucoma was associated with 18% increased odds for a lapse in diabetic retinopathy care.
Sensitivity Analysis
Similar results were seen in the cohort of patients (N = 9560) who were seen after the ICD9 to ICD10 transition and before the coronavirus disease 2019 pandemic from January 1, 2016 to December 30, 2019 (Tables S9, S10). Non-Hispanic Black patients and patients with other race/ethnicity had increased odds for ever having a lapse in care compared with non-Hispanic White patients. Hispanic patients also had increased odds for a lapse in care that was not statistically significant (Table S10, available at www.ophthalmologyscience.org).
Discussion
In a large clinical sample, we found that rates of lapses in diabetic retinopathy care were very high. Overall, we estimated that > 3 in 4 patients had ≥ 1 office visit followed by a lapse in care in the 2-year observation window. The overwhelming majority of patients who were recommended to frequent follow-up (≤ 8 weeks or > 8 to ≤ 20 weeks) had a lapse in care. Those who were recommended less frequent follow-up (> 20 to ≤ 32 weeks or > 32 to ≤ 52 weeks) had fewer lapses. Patients who had more overall office visits (2 or 3+) in the 2-year period had lower rates of lapses in care compared with those who only had 1 office visit. More than half (60.1%) of any given patient’s office visits were followed by a lapse in care. Non-Hispanic Black patients, Hispanic patients, and those of other race/ethnicity were more likely to have lapses in diabetic retinopathy care than otherwise similar non-Hispanic White patients.
The rates of lapses in diabetic retinopathy care reported in this study are largely consistent with those in the published literature. When using a fixed interval of time to define a lapse in care, other observational studies in a clinical setting have found rates ranging from 16% to 61%.2, 3, 4, 5, 6,8,9 In this study, we found 44.6% had a lapse in diabetic retinopathy care when applying a similar definition of having an interval of ≥ 6 months between visits (data not shown). Rates of lapses in diabetic retinopathy care in routine clinical practice are higher than those found in clinical trials. Using our novel methodology for identifying lapses in care, we were able to apply an adaptive definition of lapses like the clinical trial studies.1 In the context of the clinical trial, 55.3% of patients had ≥ 1 long lapse in care, but in our study, rates were much higher at 77.6%. Not surprisingly, most patients recommended for frequent follow-up had a lapse in care because of more opportunity to do so. The higher rates of lapses in care found in this study are also driven by a large proportion (45.2%) of patients who only had 1 office visit for diabetic retinopathy screening or treatment in the 2-year observation window. Most prior studies using retrospective data excluded this group of patients by restricting the analysis to patients who have presented at least twice.2, 3, 4, 5, 6,8,9 In our cohort, patients who only had 1 office visit had similar characteristics as those with lapses in care—they were younger, more frequently of non-Hispanic Black or Hispanic race and ethnicity, and had Medicaid or no insurance. Future interventions to decrease lapses in care should identify and assist this population.
The finding that non-Hispanic Black and Hispanic patients had higher odds of a lapse in diabetic retinopathy care is consistent with other reports of disparities in diabetic eye care. These same populations bear a disproportionate burden of diabetes, present with more severe diabetic retinopathy and diabetic macular edema, and sustain more vision loss from diabetic eye disease.10,17, 18, 19, 20 They also have lower rates of eye care utilization.21 The root causes underlying racial and ethnic disparities in diabetic retinopathy care likely can be traced to structural inequities that have led to the maldistribution of social determinants of health—the social, economic, and physical conditions in which people live, work, and play that affect health outcomes.22 Future work to better understand the social determinants of health that underlie higher rates of lapses in diabetic retinopathy care can be a means by which to address and eliminate this health disparity.
Our findings of other variables associated with lapses in care are in line with prior reports. Older patients are less likely to have a lapse in diabetic retinopathy care, consistent with other studies finding that the older population is more likely to have high eye care utilization.23 We did not find a difference in lapses in care by sex, but other studies have.23 Insurance is a major predictor of lapses in care and eye care utilization.24 Consistent with other studies, we found that having Medicaid or no insurance was associated with higher odds of having lapses in care. Having other insurance types was associated with lower odds of care lapses in our study. This is likely because this category included insurances, such as Tricare, worker’s compensation, and international coverage. Patients with worker’s compensation or international coverage could be specifically seeking care in ophthalmology because of an acute issue in addition to their more long-term diabetic retinopathy care. Having a diagnosed retinal disorder, whether diabetic retinopathy or another retinal disorder, decreased the odds for a lapse in care, which is consistent with other studies.24 However, having another major eye disease, glaucoma, increased the odds for a lapse in care. It could be that patients were busy with glaucoma treatment and thus delayed care for their diabetic retinopathy.
This study has several limitations. First, the proposed pipeline relies on NLP algorithms that do not have perfect sensitivity and specificity; however, the performance is in line with other algorithms targeted toward follow-up.25,26 The NLP algorithm to identify office visits related to diabetic retinopathy screening and treatment behaves conservatively with higher specificity than sensitivity. As such, we are more likely to overestimate than underestimate lapses in diabetic retinopathy care. The NLP algorithm to identify a providers’ recommended follow-up timeframe tended to identify longer follow-up as compared with the gold standard grader (on average 0.05 weeks). Identifying a longer follow-up timeframe would cause the algorithm to underestimate lapses in diabetic retinopathy care. It is possible that these 2 components of the pipeline (one that overestimates and one that underestimates lapses in care) balance each other out. Second, because we are relying on data from 1 institution, we do not know whether the patients sought diabetic retinopathy care elsewhere. Clinical experience and other studies suggest that most patients tend to stay at the same practice for longitudinal care.27, 28, 29 However, the degree to which this is true needs to be studied more directly in a future study. Third, we were unable to evaluate the presence of other medical comorbidities that might impact a patient’s ability to follow-up for eye care. Most of this population only had office visits at the Wilmer Eye Institute and no available diagnosis codes for other systemic medical conditions (e.g., hypertension). Future NLP-based studies are needed to extract systemic medical conditions from ophthalmology progress notes to further refine risk factors for lapses in care. Fourth, we also do not account for patient death in this analysis because the exact date of death is not reliably captured in the EHR. We presume this affects a small proportion of our cohort because the average age of this population is 61.4 years, we observe them for only 2 years, the average life expectancy in the United States is 77 years, and only 14% of our patients were aged > 75 years.30 Last, because we are relying on ICD codes for diagnoses, we do not know the extent to which diagnosis codes represent reality. There could be patients with vision-threatening diabetic retinopathy who are not coded as such in the EHR. Future studies should be aimed at understanding the clinical significance of lapses in care and should offer additional resources and support for the population(s) at highest risk of vision loss and lapses in care to offer additional resources and support.
Our novel methodology of adaptively defining lapses in diabetic retinopathy care based on the providers’ recommended follow-up timeframe in the EHR confers numerous advantages. The more sensitive definition allows the detection of a lapse in care before a preset timeframe, such as 6 months, has elapsed. This definition also allows for identification of patients who only have 1 office visit followed by a lapse in care. This methodology could be deployed in real-time in the EHR to identify lapses in diabetic retinopathy care and create clinical decision support systems.31 For example, once a lapse has been identified, the system could alert administrative staff to call and follow-up with patients. The clinical decision support system could also be deployed at the point of care and assist clinicians. For example, the system could show the treating clinician a patient’s history of lapses in care and future risk for lapses. Knowing the patient’s risk for a lapse in care can help guide choice of ophthalmic treatment, for example, choosing between anti-VEGF or panretinal photocoagulation for proliferative diabetic retinopathy, or offering social work support to address social determinants of health. This methodology can also be expanded to monitoring other eye diseases that require routine follow-up to prevent blindness, for example, glaucoma.
The approach presented in this study can be implemented at other institutions. The NLP tools used to identify diabetic retinopathy screening or treatment visits and the ones to extract a providers’ recommended follow-up timeframe are available on GitHub.32 Deploying these tools on unstructured provider notes in combination with structured data elements in the EHR will allow researchers at other institutions to extract the 2 components needed to identify lapses in diabetic retinopathy care. However, the performance of these tools at other institutions, or external validity, is currently unknown. Rule-based NLP algorithms can have limited generalizability outside the text used to develop it, thus, modifications of the search terms and rules will likely be needed. Validation studies should be performed, and the NLP tools should be tailored to optimize the output for the unique documentation styles at other institutions. Furthermore, the NLP algorithms developed were specific to the task of identifying lapses in diabetic retinopathy care. Researchers who wish to adapt the code to identify lapses in other types of ophthalmic care will also likely need to modify the rules to include target terms specific to a subspecialty (e.g., intraocular pressure for glaucoma care).
In conclusion, we have developed a novel methodology for identifying lapses in diabetic retinopathy care in the EHR and demonstrate high rates of lapses in diabetic retinopathy care in the clinical practice setting. Furthermore, we have identified health disparities whereby non-Hispanic Black, Hispanic, and other race/ethnicity populations have higher odds for lapses in diabetic retinopathy care compared with non-Hispanic White patients. This methodology is a major step towards EHR-based solutions, such as clinical decision support systems, that could be leveraged to eliminate disparities in diabetic retinopathy care.
Manuscript no. XOPS-D-22-00210R1.
Footnotes
Supplemental material available atwww.ophthalmologyscience.org.
Disclosure(s):
All authors have completed and submitted the ICMJE disclosures form.
The authors made the following disclosures:
C.X.C.: Funding – Regeneron Pharmaceuticals, Inc.
D.C.C.: Advisory board – Bayer HealthCare Pharmaceuticals, Inc.
Supported by a Career Development Award from the Research to Prevent Blindness (to C.X.C.), National Center for Advancing Translational Sciences funded Clinical and Translational Science Award (grant no.: KL2TR003099 [to C.X.C.]), K23 award from the NIH/NEI (award no.: K23EY033440 [to C.X.C.]), and an unrestricted grant from Research to Prevent Blindness (Wilmer Eye Institute). Dr. Cai is the Jonathan and Marcia Javitt Rising Professor of Ophthalmology. The sponsor or funding organizations had no role in the design or conduct of this research.
HUMAN SUBJECTS: Human subjects were included in this study. Johns Hopkins University School of Medicine institutional review board approved the study. All research adhered to the tenets of the Declaration of Helsinki. Consent was waived by the institutional review board given the retrospective nature of the study.
No animal subjects were used in this study.
Author Contributions:
Conception and design: Cai, Nagy, Kharrazi, Crews, Zeger
Analysis and interpretation: Cai, Tran, Scott, Zeger
Data collection: Cai, Tran, Tang, Liou, Harrigan
Obtained funding: Cai
Overall responsibility: Cai
Supplementary Data
References
- 1.Maguire M.G., Liu D., Bressler S.B., et al. Lapses in care among patients assigned to ranibizumab for proliferative diabetic retinopathy: a post hoc analysis of a randomized clinical trial. JAMA Ophthalmol. 2021;139:1266–1273. doi: 10.1001/jamaophthalmol.2021.4103. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Abdelmotaal H., Ibrahim W., Sharaf M., Abdelazeem K. Causes and clinical impact of loss to follow-up in patients with proliferative diabetic retinopathy. J Ophthalmol. 2020;2020 doi: 10.1155/2020/7691724. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Angermann R., Rauchegger T., Nowosielski Y., et al. Treatment compliance and adherence among patients with diabetic retinopathy and age-related macular degeneration treated by anti-vascular endothelial growth factor under universal health coverage. Graefes Arch Clin Exp Ophthalmol. 2019;257:2119–2125. doi: 10.1007/s00417-019-04414-y. [DOI] [PubMed] [Google Scholar]
- 4.Gao X., Obeid A., Aderman C.M., et al. Loss to follow-up after intravitreal anti-vascular endothelial growth factor injections in patients with diabetic macular edema. Ophthalmol Retina. 2019;3:230–236. doi: 10.1016/j.oret.2018.11.002. [DOI] [PubMed] [Google Scholar]
- 5.Obeid A., Gao X., Ali F.S., et al. Loss to follow-up in patients with proliferative diabetic retinopathy after panretinal photocoagulation or intravitreal anti-VEGF injections. Ophthalmology. 2018;125:1386–1392. doi: 10.1016/j.ophtha.2018.02.034. [DOI] [PubMed] [Google Scholar]
- 6.Obeid A., Su D., Patel S.N., et al. Outcomes of eyes lost to follow-up with proliferative diabetic retinopathy that received panretinal photocoagulation versus intravitreal anti-vascular endothelial growth factor. Ophthalmology. 2019;126:407–413. doi: 10.1016/j.ophtha.2018.07.027. [DOI] [PubMed] [Google Scholar]
- 7.Ramakrishnan M.S., Yu Y., VanderBeek B.L. Visit adherence and visual acuity outcomes in patients with diabetic macular edema: a secondary analysis of DRCRnet Protocol T. Graefes Arch Clin Exp Ophthalmol. 2021;259:1419–1425. doi: 10.1007/s00417-020-04944-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Suresh R., Yu H.J., Thoveson A., et al. Loss to follow-up among patients with proliferative diabetic retinopathy in clinical practice. Am J Ophthalmol. 2020;215:66–71. doi: 10.1016/j.ajo.2020.03.011. [DOI] [PubMed] [Google Scholar]
- 9.Green M., Tien T., Ness S. Predictors of lost to follow-up in patients being treated for proliferative diabetic retinopathy. Am J Ophthalmol. 2020;216:18–27. doi: 10.1016/j.ajo.2020.03.023. [DOI] [PubMed] [Google Scholar]
- 10.Kahn H.A., Hiller R. Blindness caused by diabetic retinopathy. Am J Ophthalmol. 1974;78:58–67. doi: 10.1016/0002-9394(74)90010-5. [DOI] [PubMed] [Google Scholar]
- 11.Cai C.X., Michalak S.M., Stinnett S.S., et al. Effect of ICD-9 to ICD-10 transition on accuracy of codes for stage of diabetic retinopathy and related complications: results from the CODER study. Ophthalmol Retina. 2021;5:374–380. doi: 10.1016/j.oret.2020.08.004. [DOI] [PubMed] [Google Scholar]
- 12.Efron B., Tibshirani R. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statist Sci. 1986;1:54–75. [Google Scholar]
- 13.Landis J.R., Koch G.G. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. [PubMed] [Google Scholar]
- 14.Lee P.P., Feldman Z.W., Ostermann J., et al. Longitudinal prevalence of major eye diseases. Arch Ophthalmol. 2003;121:1303–1310. doi: 10.1001/archopht.121.9.1303. [DOI] [PubMed] [Google Scholar]
- 15.Zeger S.L., Liang K.Y., Albert P.S. Models for longitudinal data: a generalized estimating equation approach. Biometrics. 1988;44:1049–1060. [PubMed] [Google Scholar]
- 16.Brant A.R., Pershing S., Hess O., et al. The impact of COVID-19 on missed ophthalmology clinic visits. Clin Ophthalmol. 2021;15:4645–4657. doi: 10.2147/OPTH.S341739. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Hill-Briggs F., Adler N.E., Berkowitz S.A., et al. Social determinants of health and diabetes: a scientific review. Diabetes Care. 2020;44:258–279. doi: 10.2337/dci20-0053. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Harris M.I., Klein R., Cowie C.C., et al. Is the risk of diabetic retinopathy greater in non-Hispanic blacks and Mexican Americans than in non-Hispanic whites with type 2 diabetes? A U.S. population study. Diabetes Care. 1998;21:1230–1235. doi: 10.2337/diacare.21.8.1230. [DOI] [PubMed] [Google Scholar]
- 19.Varma R., Bressler N.M., Doan Q.V., et al. Prevalence of and risk factors for diabetic macular edema in the United States. JAMA Ophthalmol. 2014;132:1334–1340. doi: 10.1001/jamaophthalmol.2014.2854. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Spanakis E.K., Golden S.H. Race/ethnic difference in diabetes and diabetic complications. Curr Diab Rep. 2013;13:814–823. doi: 10.1007/s11892-013-0421-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Zhang X., Cotch M.F., Ryskulova A., et al. Vision health disparities in the United States by race/ethnicity, education, and economic status: findings from two nationally representative surveys. Am J Ophthalmol. 2012;154:S53–S62.e1. doi: 10.1016/j.ajo.2011.08.045. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.World Health Organization A conceptual framework for action on the social determinants of health. https://www.who.int/publications/i/item/9789241500852
- 23.Cai C.X., Li Y., Zeger S.L., McCarthy M.L. Social determinants of health impacting adherence to diabetic retinopathy examinations. BMJ Open Diabetes Res Care. 2021;9 doi: 10.1136/bmjdrc-2021-002374. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Murchison A.P., Hark L., Pizzi L.T., et al. Non-adherence to eye care in people with diabetes. BMJ Open Diabetes Res Care. 2017;5 doi: 10.1136/bmjdrc-2016-000333. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Were M.C., Gorbachev S., Cadwallader J., et al. Natural language processing to extract follow-up provider information from hospital discharge summaries. AMIA Annu Symp Proc. 2010;2010:872–876. [PMC free article] [PubMed] [Google Scholar]
- 26.Lau W., Payne T.H., Uzuner O., Yetisgen M. Extraction and analysis of clinically important follow-up recommendations in a large radiology dataset. AMIA Jt Summits Transl Sci Proc. 2020;2020:335–344. [PMC free article] [PubMed] [Google Scholar]
- 27.Mold J.W., Fryer G.E., Roberts A.M. When do older patients change primary care physicians? J Am Board Fam Pract. 2004;17:453–460. doi: 10.3122/jabfm.17.6.453. [DOI] [PubMed] [Google Scholar]
- 28.Weiss L.J., Blustein J. Faithful patients: the effect of long-term physician-patient relationships on the costs and use of health care by older Americans. Am J Public Health. 1996;86:1742–1747. doi: 10.2105/ajph.86.12.1742. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Safran D.G., Montgomery J.E., Chang H., et al. Switching doctors: predictors of voluntary disenrollment from a primary physician’s practice. J Fam Pract. 2001;50:130–136. [PubMed] [Google Scholar]
- 30.Arias E., Tejada-Vera B., Ahmad F., Kochanek K.D. National Center for Health Statistics; Hyattsville, MD: 2021. Provisional Life Expectancy Estimates for 2020. [Google Scholar]
- 31.Sim I., Gorman P., Greenes R.A., et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc. 2001;8:527–534. doi: 10.1136/jamia.2001.0080527. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.NLP for Patient Notes https://github.com/wilson3090/NLP-for-Patient-Notes
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.