Skip to main content
NCBI home page
Ophthalmology Science logoLink to Ophthalmology Science
. 2025 Oct 10;6(1):100967. doi: 10.1016/j.xops.2025.100967

Identifying Patients at High Risk of Lapses in Diabetic Retinopathy Care: A Machine Learning Study

Jizhou Tian 1, Diep Tran 2, Zainab Rustam 2, Gina Zhu 2, Paul Nagy 3, Hadi Kharrazi 4, Deidra C Crews 5, Zitong Wang 1, Scott L Zeger 1, Cindy X Cai 2,3,∗
PMCID: PMC12902138  PMID: 41693809

Abstract

Objective

To predict lapses in diabetic retinopathy (DR) care.

Design

Retrospective cohort study.

Subjects

Adults ≥18 years with diabetes seen at the Wilmer Eye Institute for DR screening or treatment between 2012 and 2022.

Main Outcome Measures

Whether an office visit for DR screening or treatment was followed by a lapse in care.

Methods

Three versions of prediction algorithms were constructed using random forests (RFs). XGBoost (XGB) was used as a confirmatory analysis. Random forest-A and XGB-A included electronic health record (EHR) variables alone (e.g., sociodemographic, insurance, ophthalmic diagnoses, lead time, and recommended follow-up time). Random forest-B and XGB-B added location-based social determinants of health (SDoH) variables (e.g., Area Deprivation Index). Random forest-C and XGB-C added history of lapses in care (e.g., whether the patient has ever had lapses in care before). The area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC) were calculated for each algorithm.

Results

A total of 36 995 patients (mean age 62 years, 53% female, 47% non-Hispanic White, 38% non-Hispanic Black, and 4% Hispanic) and 141 930 office visits were included. The best performing model was RF-C with an AUROC of 0.774 (0.772–0.776) and AUPRC of 0.707 (0.704–0.711), outperforming RF-A and RF-B in AUROC and AUPRC (P < 0.001 for each comparison). XGB-C similarly outperformed XGB-A and XGB-B (P < 0.001 for each comparison).

Conclusions

We developed RF algorithms, as well as XGB confirmatory models, to predict whether patients with diabetes will experience a lapse in DR care. The best prediction was achieved using EHR variables, location-based SDoH variables, and history of lapses in care. These models offer the opportunity to identify high-risk patients and offer additional resources to reduce lapses in care and potentially vision loss from DR.

Financial Disclosure(s)

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Keywords: Diabetic retinopathy, Electronic health record, Lapses in care, Prediction, Social determinants of health

Diabetic retinopathy (DR) remains a major cause of vision loss among working age adults.1,2 Existing treatments are largely effective in reversing or preventing vision loss but require screening and diligent follow-up with eye care.3,4 Lapses in care, or not returning as recommended by the provider, are major risk factors for vision impairment.5, 6, 7, 8, 9 For example, patients undergoing intravitreal anti-VEGF treatment for diabetic macular edema had worse vision outcomes if they missed visits and treatments.6 For patients with proliferative DR (PDR), having long lapses in care was associated with decrements in visual acuity and increased risk for neovascularization.7

Given the importance of lapses in DR care for vision outcomes, it is critical to predict and identify patients who are at highest risk. Existing prediction models in ophthalmology focus primarily on clinic no-shows.10,11 Although similar in concept, clinic no-shows are distinct from lapses in care. Having a clinic no-show requires an appointment, but we do not know whether that appointment was what the provider had recommended. Our previously published lapse in care algorithm allows the identification of patients who do not return as scheduled, regardless of whether or not an appointment was made.5 Identifying and predicting lapses in care will allow us to intervene before the lapses become sustained and patients are lost to follow-up. The aim of this study was to develop a machine learning model to predict lapses in DR care at a single academic institution.

Methods

Study Design/Patient Selection

This was a retrospective study of adults ≥18 years with diabetes mellitus who sought DR screening or treatment at the Wilmer Eye Institute between April 4, 2013 to April 1, 2022.5 Diabetes was defined as having a qualifying International Classification of Diseases code at any hospital encounter or a hemoglobin A1c value ≥6.5%.5 Qualifying patients and all associated DR screening or treatment office visits were included. This study followed the Prediction Model Risk of Bias Assessment Tool reporting guidelines.12 The study was approved by the Johns Hopkins Institutional Review Board and adhered to the Declarations of Helsinki. The Institutional Review Board waived the need for consent.

Outcome

The primary outcome was lapse in care, as previously defined,5 after a DR screening or treatment visit. Patients were defined as having a lapse in care if they returned in ≥10 weeks when recommended to return in ≤8 weeks, ≥24 weeks when recommended to return between 8 to 20 weeks, ≥40 weeks when recommended to return between 20 to 32 weeks, and ≥64 weeks when recommended to return between 32 to 52 weeks.5

History of Lapses in Care

We created multiple metrics to reflect patients’ histories of lapses in care. We calculated the number of prior DR screening or treatment visits, number of prior office visits followed by lapses in care, and whether the patient ever had a lapse in care (binary). Office visits in which lapses in care could not be defined were censored. We also created 2 indicators for fraction of prior lapses in care. The first indicator is the total fraction of prior lapses defined as the number of prior lapses divided by the total number of prior visits across all office visits for that patient. The second indicator is the cumulative fraction of prior lapses defined as the number of prior lapses divided by the number of prior visits up to that point. Calculations for a sample patient are shown in Supplemental Figure 1, available at www.ophthalmologyscience.org.

Electronic Health Record Predictor Variables

Patient sociodemographic and other baseline characteristics were extracted from the electronic health record (EHR) including age, sex, race or ethnicity, and insurance (i.e., private, Medicare, Medicaid, other, and none). Ophthalmic characteristics included the severity of the DR, the presence of arterial or venous occlusions, and the presence of glaucoma.5 The type of provider (e.g., physician, optometrist, resident, and fellow) was extracted. The physician’s recommended follow-up time was identified as previously described.5 The wait time was calculated as the time between check-in and recording of the visual acuity or intraocular pressure, whichever one came first. The lead time for the office visit was calculated as the difference from when the appointment was last scheduled to the date of the encounter. Office visits with missing covariates were excluded.

Location-Based Social Determinants of Health Variables

The 2018 census tract of each patient’s residential address was obtained from the Johns Hopkins Precision Medicine Analytics Platform.13 The census tract of the residential address was matched to the 2018 Area Deprivation Index, which is a factor-based composite measure of neighborhood socioeconomic disadvantage with higher percentiles indicating more socioeconomic disadvantage.14,15 Patients with null or non-numeric Area Deprivation Index were excluded. The census tract of the residential address was also linked to measures from the 2018 US American Community Survey 5-year data including: the percent living in poverty, the per capita income, the percent unemployed in the labor force, the percent of ≥25 year olds with at least a high school diploma, the percent of renters, the percent using food assistance programs, the percent of households without vehicles, and the percent uninsured.16 The clinic office location where the patient received care was geocoded to a latitude and longitude, and the Euclidean distance of the patient's residential address and the clinic location was calculated in miles using the GeoPy package in Python 3.8 (Python Software Foundation).17

Random Forest and XGBoost Models

Three random forest (RF) models using the randomForest package in R (version 4.4, R Foundation for Statistical Computing) were constructed to predict whether any given office visit would be followed by a lapse in DR care. Random forest-A was fitted using EHR variables alone, version B (RF-B) used the EHR and location-based social determinants of health (SDoH) variables, and RF-C had the history of lapses in care in addition to the predictors in RF-B. As a validation check on the performance of the RF predictions, we also fit 3 XGBoost (XGB) models using the xgboost package in R (version 4.4), with the same predictor sets and outcome definitions as the RF models.

Data Splitting

To prepare for hyperparameter tuning and model evaluation, we randomly partitioned the dataset into 5 mutually exclusive groups by randomly assigning patients, such that each patient's entire clinic visit records were kept in a single group or fold. This yielded 5 unique combinations of training and test sets, each consisting of approximately 80% training data and 20% test data. Every patient appeared exactly once in a test set across the 5 folds, ensuring full population coverage.

Hyperparameter Tuning

To optimize model performance, hyperparameter tuning was performed separately within each of the 5 folds, using the training data only. No information from the corresponding test sets was used during tuning. Details of hyperparameter tuning for the RF and XGB models are located in Supplemental Tables 1-2, available at www.ophthalmologyscience.org.

Model Performance

For each of the 5 folds, the model with the optimal hyperparameter combination (as selected from the training set) was retrained on the entire training set and evaluated on the corresponding test set. Predicted probabilities on all 5 test sets and the corresponding observed outcomes were pooled to evaluate overall model performance.

The primary performance metric was the area under the receiver operating characteristic curve (AUROC), calculated using the pROC package in R (version 4.4).18 Confidence intervals and differences in model performance were evaluated using the DeLong method.19 The area under the precision-recall curve (AUPRC) was also calculated for each model using the PRROC package in R (version 4.4).20,21 We reported the AUPRC based on continuous interpolation.20,22 The confidence interval for the performance was calculated based on a logistic transformation method.22 The difference in AUPRC between models was calculated using the usefun package in R (version 4.4), which applies Davi’s interpolation and tests for statistical significance via the bootstrap method.23,24

Single Regression Tree

We fitted a single regression tree to approximate and visualize the results obtained from the RF. The predicted probabilities generated by RF models on all 5 test sets were used as the response variable, and the same covariates from the RF were employed to fit the regression tree. To assess how well the regression tree approximates the RF, we calculated the Pearson correlation between the predicted probabilities from the RF and those from the single regression tree.

Site-Based Cross-Validation

Patients were assigned to 1 of the 10 possible clinic locations based on their visit history. For patients who visited multiple clinics, they were assigned to the most frequently visited clinic. If there were multiple frequently visited clinics, then patients were assigned to the clinic from the most recent encounter. These 10 clinics were then geographically divided into 4 groups based on proximity. A cross-validation approach was implemented in which 3 of the 4 groups were used as the training set and the remaining group as the test group. This process was repeated so that each group served as the test group once. For each iteration, the 2 models RF-C and XGB-C were trained separately on the training set, with hyperparameters tuned using training data only (Supplemental Tables 3-4, available at www.ophthalmologyscience.org). Both AUROC and AUPRC for the test group were reported in each iteration.

Results

A total of 36 995 patients and 141 930 office visits were included in the study (Supplemental Figure 2, Supplemental Table 5, available at www.ophthalmologyscience.org). Baseline characteristics are shown in Table 6.

Table 6.

Baseline Characteristics of the Patients Included in the Study

Characteristic No./Total (%) 36 995
 Age∗ 62.1 (SD: 13.5)
 Sex
 Male 17 488 (47.3)
 Female 19 507 (52.7)
 Race and ethnicity
 Non-Hispanic White 17 196 (46.5)
 Non-Hispanic Black 14 135 (38.2)
 Hispanic 1558 (4.2)
 Other 4106 (11.1)
 Insurance
 Private 14 226 (38.5)
 Medicare 15 110 (40.8)
 Medicaid 4139 (11.2)
 None 731 (2.0)
 Other 2789 (7.5)
 Diabetic retinopathy (DR)
 No DR 29 843 (80.7)
 Nonproliferative diabetic retinopathy (NPDR) 5307 (14.3)
 Proliferative diabetic retinopathy (PDR) 1845 (5.0)
 Other retinal disorders∗
 Absent 35 466 (95.9)
 Present 1529 (4.1)
 Other ophthalmic disorders
 Absent 31 028 (83.9)
 Present 5967 (16.1)
 Distance to clinics in miles∗ 10.7 (11.9)
 History of prior lapses in care (yes) 16 362 (44.2)
Neighborhood-level characteristics
 Area Deprivation Index, national percentile rank∗ 38.8 (25.6)
 Per capita income ($)∗ 38 451 (16 197)
 Percent poverty income in 12 months (%)∗ 12.0 (11.0)
 Percent unemployment in the labor force (%)∗ 4.1 (2.6)
 Percent education in high school or greater 85.8 (9.9)
 Percentage using food assistance program (%)∗ 14.4 (14.1)
 Percent without vehicle owner (%)∗ 5.9 (8.7)
 Percentage uninsured (%)∗ 5.0 (3.8)
 Percentage renter (%)∗ 32.9 (22.8)
Open in a new tab
∗

Continuous variables presented as mean (standard deviation).

Random forest-A included EHR predictor variables alone. The AUROC of the model was 0.703 (0.700–0.705) and AUPRC 0.610 (0.606–0.614) (Table 7 and Supplemental Table 8, available at www.ophthalmologyscience.org). The correlation between the single regression tree and the RF-A model was 0.86 and demonstrated that severity of DR, lead time ≥5.5 weeks, and recommended follow-up <6.4 weeks were important predictor variables (Supplemental Figure 3, available at www.ophthalmologyscience.org). Not having DR, having a shorter lead time (<5.5 weeks) in those without DR, and longer provider recommended follow-up (≥6.4 weeks) in those with non-PDR or PDR all increased the probability of lapsing.

Table 7.

AUROC of the Random Forest Models

Model Variables Included AUROC (95% CI) Comparisons (95% CI) Comparisons (95% CI)
A EHR only 0.703 (0.700–0.705) (Reference)
B EHR + location-based SDoH 0.694 (0.691,0.697) –0.009 (–0.01, –0.007) P < 0.001 (Reference)
C EHR + location-based SDoH variables + history of lapses in care 0.774 (0.772,0.776) 0.071 (0.069,0.073) P < 0.001 0.080 (0.078,0.082) P < 0.001
Open in a new tab

AUROC = area under the receiver operating characteristic curve; CI = confidence interval; EHR = electronic health record; SDoH = social determinants of health.

Random forest-B included EHR and location-based SDoH variables. The AUROC was 0.694 (0.691–0.697) and AUPRC 0.605 (0.601–0.609) (Table 7 and Supplemental Table 8, available at www.ophthalmologyscience.org). The correlation between the single regression tree and the RF-B model was 0.80. The single regression tree demonstrated that the severity of DR, lead time ≥5.5 weeks, and recommended follow-up <6.9 weeks were the most important predictor variables (Supplemental Figure 4, available at www.ophthalmologyscience.org). Not having DR, having a shorter lead time (<5.5 weeks) in those without DR, and longer provider recommended follow-up (>6.9 weeks) in those with non-PDR or DR increased the probability of having a lapse in care.

Random forest-C included EHR, location-based SDoH variables, as well as history of lapses in care. The AUROC for model C was 0.774 (0.772–0.776) and AUPRC 0.707 (0.704–0.711) (Table 7 and Supplemental Table 8, available at www.ophthalmologyscience.org). The correlation between the single regression tree and the RF-C model was 0.88 with the most important predictors being the number of prior visits ≥1.5, total fraction of prior lapses <0.25, and DR severity (Supplemental Figure 5, available at www.ophthalmologyscience.org). Having fewer prior visits (<1.5), no DR in those with fewer prior visits (<1.5), and higher fraction of prior lapse (≥0.25) in those with more prior visits (≥1.5) increased the probability of the office visit being followed by a lapse in care.

Between models, RF-C had higher AUROC compared with both RFs B and A (P < 0.001 for each comparison) (Table 7). In the site-based cross-validation, RF-C resulted in AUROCs ranging from 0.717 to 0.778 (Table 9).

Table 9.

Random Forest Site-Based Cross-Validation

Group Clinic No. Patients, No. Office Visits AUROC AUPRC
1 Frederick 925, 5646 0.717 (0.702–0.732) 0.527 (0.503,0.551)
2 Lutherville 3230, 11 178 0.759 (0.755,0.763) 0.721 (0.715,0.726)
Bel Air 9852, 38 237
White Marsh 3229, 7755
3 East Baltimore 7582, 37 060 0.778 (0.773,0.782) 0.683 (0.676,0.690)
Bayview 2285, 8537
Wyman Park 161, 250
4 Columbia 5048, 16 869 0.764 (0.759,0.769) 0.704 (0.697, 0.712)
Odenton 3101, 10 919
Bethesda 1582, 5479
Open in a new tab

AUROC = area under the receiver operating characteristic curve; AUPRC = area under the precision-recall curve.

Internal validation using model C that includes electronic health record, location-based social determinants of health, as well as history of lapses in care variables. The ten clinics where patients received diabetic retinopathy screening or treatment were grouped into 4 based on proximity.

Qualitatively, similar results were obtained using the XGB method. XGBoost-C had a higher AUROC compared with both XGBs B and A (P < 0.001 for each comparison) (Table 10, Supplemental Tables 11-12, available at www.ophthalmologyscience.org).

Table 10.

AUROC of the XGBoost Models

Model Variables Included AUROC (95% CI) Comparisons Comparisons
A EHR only 0.709 (0.706,0.712) (Reference)
B EHR + location-based SDoH 0.709 (0.707,0.712) 0.0002 (–0.0004,0.0008) P < 0.001 (Reference)
C EHR + location-based SDoH variables + history of lapses in care 0.785 (0.783,0.787) 0.076 (0.074,0.078) P < 0.001 0.076 (0.074, 0.078) P < 0.001
Open in a new tab

AUROC = area under the receiver operating characteristic curve; CI = confidence interval; EHR = electronic health record; SDoH = social determinants of health.

Discussion

We developed machine learning models to predict lapses in DR care. We found that the inclusion of history of lapses in care improved the prediction of whether or not a DR screening or treatment office visit will be followed by a lapse in care compared to EHR variables and location-based SDoH variables. Of the EHR variables, the most important predictors appeared to be the severity of DR, lead time, and recommended follow-up time. Our prediction models can help identify high risk individuals and guide deployment of future interventions aimed at eliminating lapses in care and potentially reducing vision loss from DR.

The EHR variables most predictive of lapses in DR care were the severity of DR, lead time, and recommended follow-up time. Not having DR, as identified by International Classification of Diseases codes, increased the probability of having a lapse in care by onefold to threefold, as estimated by the single regression trees for RFs A and B. In line with this, being recommended a longer duration of follow-up (beyond 6.4 weeks in RF-A and 6.9 weeks in RF-B) also increased the probability of having a lapse in care by onefold to threefold. Early stages of DR require less frequent monitoring and are often asymptomatic.3 Multiple prior studies have found that patients with less severe DR are more likely to have lapses in care.5,25 Having a lead time <6 weeks (i.e., the appointment was made within 6 weeks of the office visit) also increased the probability that the visit was followed by a lapse in care. This increased probability was onefold to twofold, as estimated by the single regression trees. This finding is in contrast to existing literature in which longer lead times were more likely associated with missed appointments.26, 27, 28, 29, 30 There could be nuances in local scheduling workflows that underlie this contrast. It is also possible that patients that schedule appointments with shorter lead times, that is, more last minute, compared with patients who schedule appointments with longer lead times, are less engaged with their health care or have treatment fatigue and are thus more likely to have a lapse in care.

Including location-based SDoH variables in addition to the EHR variables did not improve the prediction of lapses in care. Despite the importance of individual-level SDoH and neighborhood-level SDoH impacts on health outcomes, their added value in predictive modeling is unclear. Areas with socioeconomic disadvantage and concentrated poverty often lack resources, like public transportation or health care facilities, that make following up with DR care challenging.31, 32, 33 However, in our case, the inclusion of neighborhood-level SDoH did not improve the prediction of lapses in care beyond EHR variables alone (i.e., comparing RF-B to RF-A and XGB-B to XGB-A). This is consistent with other studies that show the inclusion of neighborhood-level SDoH does not improve the prediction of various health outcomes.34, 35, 36, 37 It is possible that the inclusion of individual-level SDoH would have more predictive value.38 However, like most EHRs, individual-level SDoH data are not yet routinely collected.39,40

The inclusion of race and ethnicity in predictive modeling is controversial.41 Although we included race and ethnicity, it was one of the least predictive variables across all models. Algorithms trained on data that reflect racial biases may yield racially biased output, but could also be more efficient.41 In the Accuracy-Fairness tradeoff, one must choose whether to preserve the efficiency offered by using machine learning tools that include race and ethnicity data or discard that efficiency to avoid perpetuating health disparities.41 One solution, as outlined by some authors, is to include an “intent behind the design” by choosing appropriate questions and settings for machine learning use and to ensure that these algorithms have beneficial effects.41, 42, 43 In our case, we developed this algorithm as a way to identify patients at high risk for having lapses in care so that we can offer additional social resources such as transportation vouchers. In this way, we hope to use the results of our predictive algorithm to advance health equity rather than exacerbate health disparities.

The best-performing model was one that included the patient’s history of lapses in care, in addition to EHR and location-based SDoH variables. Many prior studies have shown that the best predictors of an outcome is the patient’s history of that outcome and interaction with the health care system. For example, in predicting clinic no-shows, whether in ophthalmology or other specialties, the most predictive feature is often a history of no-shows.26, 27, 28,44, 45, 46, 47 This is why some studies have shown more accurate prediction for follow-up patients, where we have documented prior behavior, as compared with new patients.29 However, at the Wilmer Eye Institute, coding for follow-up versus new patients in the structured data is not rigorously differentiated; thus, we decided not to include it in our prediction model.

The performance of our prediction model is in line with other studies that predict clinic no-shows. Published prediction models have reported AUROCs ranging from 0.7s to 0.9s and AUPRCs from 0.1s to 0.7s.10,11,26, 27, 28,44, 45, 46, 47 Although some models had better performance than ours, it is important to keep in mind that the prediction task is different. No-shows are typically defined as appointments that were made where patients canceled the same day or did not attend. We do not know if this appointment was what the provider had recommended. Our definition of lapses in care takes into consideration what the provider had recommended and is more nuanced than a no-show.5 For example, patients who were recommended to follow-up in 6 weeks but never made an appointment would not be identified as a “no-show.” By capturing lapses in care, our prediction model will allow us to identify individuals who potentially never made follow-up appointments and allow us to conduct interventions earlier.

Further studies are necessary before our prediction model can be integrated into routine clinical practice. Because this model was developed and tested at a single academic institution, the generalizability of our findings may be limited. External validation is still needed. Furthermore, appropriate cutoff probabilities must be established to guide practical actions, such as initiating screening for social needs or referral to social work based on model predictions. The selection of these thresholds will likely depend not only on the model’s performance but also on the health care system’s capacity and resource availability.

Our study has a few limitations. First, again, because our prediction models were built on data from a single academic institution, we do not know the generalizability of our findings to other data sources. Second, we chose to build our model to identify lapses in DR care because more specific models can outperform general models.48 However, it is unknown if this model can be expanded to lapses in care for other ophthalmic care, such as glaucoma, for example. Third, although we did not separately evaluate the performance of our algorithm on patients who were new as compared with return patients, the performance of RF-C that includes history of lapses in care suggests that our algorithm performs better among patients who have more longitudinal data. Fourth, we excluded patients with missing predictor variables, including sociodemographic data, other EHR variables, and the Area Deprivation Index. The Area Deprivation Index can be suppressed for various reasons, including low population or housing units, or poor data integrity.49 Thus, it is possible that our algorithm will not perform as well for patients from certain neighborhoods. The exclusion of patients with incomplete sociodemographic and EHR data could introduce bias, but this only affected a small number of patients. Fifth, we used International Classification of Diseases codes for DR severity, which has known limitations. However, evidence suggests that the broad categories of non-PDR and PDR, as used in this study, are accurate.50 Future studies could examine whether more nuanced DR severity staging improves prediction.51 Finally, although we included all patients who sought DR care at our institution, it is possible that our algorithm will have bias and perform differently for subgroups within the population (e.g., certain race and ethnicity groups, patients seen by providers with a certain specialty or training).42 Future implementation of the model should include studies to thoroughly evaluate its potential biases and potential to exacerbate rather than ameliorate health disparities, as was the intended use.

In conclusion, we have created robust machine learning algorithms that can predict whether patients who seek DR screening or treatment at a single academic institution will experience a lapse in care after their office visit. Our motivation for creating the model is to help identify patients who are at high risk for lapses in care so that we can screen for relevant social needs and offer additional support (e.g., transportation and food vouchers). This work represents an important step toward leveraging population-level EHR data to address disparities in DR care.

Manuscript no. XOPS-D-25-00419.

Footnotes

Disclosure(s):

All authors have completed and submitted the ICMJE disclosures form.

The authors made the following disclosures:

C.X.C.: Grants – Regeneron; Travel expenses – Boehringer Ingelheim, 4D Molecular Therapeutics; Receipt of equipment, materials, drugs, medical writing, gifts or other services – Optomed USA, Inc.

This project was supported by a Career Development Award from the Research to Prevent Blindness (C.X.C.), K23 award from the NIH/NEI (award number K23EY033440) (C.X.C.), and an unrestricted grant from Research to Prevent Blindness (Wilmer Eye Institute). C.X.C. is the Jonathan and Marcia Javitt Rising Professor of Ophthalmology.

Support for Open Access publication was provided by Wilmer Eye Institute, Johns Hopkins School of Medicine.

HUMAN SUBJECTS: Human subjects were included in this study. The study was approved by the Johns Hopkins Institutional Review Board (IRB) and adhered to the Declarations of Helsinki. The IRB waived the need for consent.

No animal subjects were used in this study.

Author Contributions:

Conception and design: Zeger, Cai

Data collection: Tian, Tran, Rustam, Zhu, Cai

Analysis and interpretation: Tian, Tran, Rustam, Zhu, Nagy, Kharrazi, Crews, Wang, Zeger, Cai

Obtained funding: Cai

Overall responsibility: Tian, Zeger, Cai

Supplemental material available atwww.ophthalmologyscience.org.

Supplementary Data

Supplementary Material
mmc1.pdf (1.1MB, pdf)

References

  • 1.Saaddine J.B., Honeycutt A.A., Narayan K.M.V., et al. Projection of diabetic retinopathy and other major eye diseases among people with diabetes mellitus: United States, 2005-2050. Arch Ophthalmol. 2008;126:1740–1747. doi: 10.1001/archopht.126.12.1740. [DOI] [PubMed] [Google Scholar]
  • 2.Cheung N., Mitchell P., Wong T.Y. Diabetic retinopathy. Lancet. 2010;376:124–136. doi: 10.1016/S0140-6736(09)62124-3. [DOI] [PubMed] [Google Scholar]
  • 3.Flaxel C.J., Adelman R.A., Bailey S.T., et al. Diabetic retinopathy preferred practice pattern. Ophthalmology. 2020;127:P66–P145. doi: 10.1016/j.ophtha.2019.09.025. [DOI] [PubMed] [Google Scholar]
  • 4.Ferris F.L., 3rd Results of 20 years of research on the treatment of diabetic retinopathy. Prev Med. 1994;23:740–742. doi: 10.1006/pmed.1994.1127. [DOI] [PubMed] [Google Scholar]
  • 5.Cai C.X., Tran D., Tang T., et al. Health disparities in lapses in diabetic retinopathy care. Ophthalmol Sci. 2023;3 doi: 10.1016/j.xops.2023.100295. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Ramakrishnan M.S., Yu Y., VanderBeek B.L. Visit adherence and visual acuity outcomes in patients with diabetic macular edema: a secondary analysis of DRCRnet protocol T. Graefes Arch Clin Exp Ophthalmol. 2021;259:1419–1425. doi: 10.1007/s00417-020-04944-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Maguire M.G., Liu D., Bressler S.B., et al. Lapses in care among patients assigned to ranibizumab for proliferative diabetic retinopathy: a post hoc analysis of a randomized clinical trial. JAMA Ophthalmol. 2021;139:1266–1273. doi: 10.1001/jamaophthalmol.2021.4103. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Obeid A., Su D., Patel S.N., et al. Outcomes of eyes lost to Follow-up with proliferative diabetic retinopathy that received panretinal photocoagulation versus intravitreal anti-vascular endothelial growth factor. Ophthalmology. 2019;126:407–413. doi: 10.1016/j.ophtha.2018.07.027. [DOI] [PubMed] [Google Scholar]
  • 9.Abdelmotaal H., Ibrahim W., Sharaf M., Abdelazeem K. Causes and clinical impact of loss to Follow-Up in patients with proliferative diabetic retinopathy. J Ophthalmol. 2020;2020 doi: 10.1155/2020/7691724. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Breeze F., Hossain R.R., Mayo M., McKelvie J. Predicting ophthalmic clinic non-attendance using machine learning: development and validation of models using nationwide data. Clin Exp Ophthalmol. 2023;51:764–774. doi: 10.1111/ceo.14310. [DOI] [PubMed] [Google Scholar]
  • 11.Chen J., Goldstein I.H., Lin W.C., et al. Application of machine learning to predict patient No-Shows in an academic pediatric ophthalmology clinic. AMIA Annu Symp Proc. 2020;2020:293–302. [PMC free article] [PubMed] [Google Scholar]
  • 12.Wolff R.F., Moons K.G.M., Riley R.D., et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med. 2019;170:51–58. doi: 10.7326/M18-1376. [DOI] [PubMed] [Google Scholar]
  • 13.https://pm.jh.edu Website.
  • 14.Kind A.J.H., Buckingham W.R. Making neighborhood-disadvantage metrics accessible - the neighborhood Atlas. N Engl J Med. 2018;378:2456–2458. doi: 10.1056/NEJMp1802313. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Kind A.J.H., Jencks S., Brock J., et al. Neighborhood socioeconomic disadvantage and 30-day rehospitalization: a retrospective cohort study. Ann Intern Med. 2014;161:765–774. doi: 10.7326/M13-2946. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Ong S.S., Tran D., Westlund E., et al. Neighborhood-level social determinants of health and presenting characteristics for rhegmatogenous retinal detachments. JAMA Ophthalmol. 2024;142:845–854. doi: 10.1001/jamaophthalmol.2024.2889. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Cai C.X., Klawe J., Ahmad S., et al. Geographic variations in gender differences in cataract surgery volume among a national cohort of ophthalmologists. J Cataract Refract Surg. 2022;48:1023–1030. doi: 10.1097/j.jcrs.0000000000000938. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Robin X., Turck N., Hainard A., et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 2011;12:77. doi: 10.1186/1471-2105-12-77. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.DeLong E.R., DeLong D.M., Clarke-Pearson D.L. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837–845. [PubMed] [Google Scholar]
  • 20.Keilwagen J., Grosse I., Grau J. Area under precision-recall curves for weighted and unweighted data. PLoS One. 2014;9 doi: 10.1371/journal.pone.0092209. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Grau J., Grosse I., Keilwagen J. PRROC: computing and visualizing precision-recall and receiver operating characteristic curves in R. Bioinformatics. 2015;31:2595–2597. doi: 10.1093/bioinformatics/btv153. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Boyd K., Eng K.H., Page C.D. Advanced Information Systems Engineering. Lecture notes in computer science. Springer; Berlin Heidelberg: 2013. Area under the precision-recall curve: point estimates and confidence intervals; pp. 451–466. [Google Scholar]
  • 23.Davis J., Goadrich M. Proceedings of the 23rd International Conference on Machine Learning - ICML ’06. Computing Machinery; New York, NY: 2006. The relationship between precision-recall and ROC curves. [DOI] [Google Scholar]
  • 24.https://cran.r-project.org/web/packages/usefun/usefun.pdf [No title]
  • 25.Murchison A.P., Hark L., Pizzi L.T., et al. Non-adherence to eye care in people with diabetes. BMJ Open Diabetes Res Care. 2017;5 doi: 10.1136/bmjdrc-2016-000333. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Goffman R.M., Harris S.L., May J.H., et al. Modeling patient No-Show history and predicting future outpatient appointment behavior in the veterans health administration. Mil Med. 2017;182:e1708–e1714. doi: 10.7205/MILMED-D-16-00345. [DOI] [PubMed] [Google Scholar]
  • 27.Mohammadi I., Wu H., Turkcan A., et al. Data analytics and modeling for appointment No-show in community health centers. J Prim Care Community Health. 2018;9 doi: 10.1177/2150132718811692. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Nelson A., Herron D., Rees G., Nachev P. Predicting scheduled hospital attendance with artificial intelligence. NPJ Digit Med. 2019;2:26. doi: 10.1038/s41746-019-0103-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Drewek R., Mirea L., Adelson P.D. Lead time to appointment and No-Show rates for new and Follow-up patients in an ambulatory clinic. Health Care Manag (Frederick) 2017;36:4–9. doi: 10.1097/HCM.0000000000000148. [DOI] [PubMed] [Google Scholar]
  • 30.McMullen M.J., Netland P.A. Lead time for appointment and the no-show rate in an ophthalmology clinic. Clin Ophthalmol. 2015;9:513–516. doi: 10.2147/OPTH.S82151. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Tang T., Tran D., Han D., et al. Place, race, and lapses in diabetic retinopathy care. JAMA Ophthalmol. 2024;142:581–583. doi: 10.1001/jamaophthalmol.2024.0974. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Yusuf R., Chen E.M., Nwanyanwu K., Richards B. Neighborhood deprivation and adherence to initial diabetic retinopathy screening. Ophthalmol Retina. 2020;4:550–552. doi: 10.1016/j.oret.2020.01.016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Gaskin D.J., Thorpe R.J., Jr., McGinty E.E., et al. Disparities in diabetes: the nexus of race, poverty, and place. Am J Public Health. 2014;104:2147–2155. doi: 10.2105/AJPH.2013.301420. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Bhavsar N.A., Gao A., Phelan M., et al. Value of neighborhood socioeconomic status in predicting risk of outcomes in studies that use electronic health record data. JAMA Netw Open. 2018;1 doi: 10.1001/jamanetworkopen.2018.2716. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Tan M., Hatef E., Taghipour D., et al. Including social and behavioral determinants in predictive models: trends, challenges, and opportunities. JMIR Med Inform. 2020;8 doi: 10.2196/18084. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Hatef E., Ma X., Rouhizadeh M., et al. Assessing the impact of social needs and social determinants of health on health care utilization: using Patient- and community-level data. Popul Health Manag. 2021;24:222–230. doi: 10.1089/pop.2020.0043. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Hatef E., Kitchen C., Pandya C., Kharrazi H. Assessing patient and community-level social factors; the synergistic effect of social needs and social determinants of health on healthcare utilization at a multilevel academic healthcare system. J Med Syst. 2023;47:95. doi: 10.1007/s10916-023-01990-9. [DOI] [PubMed] [Google Scholar]
  • 38.Sotudian S., Afran A., LeBedis C.A., et al. Social determinants of health and the prediction of missed breast imaging appointments. BMC Health Serv Res. 2022;22:1454. doi: 10.1186/s12913-022-08784-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Lee T.C., Saseendrakumar B.R., Nayak M., et al. Social determinants of health data availability for patients with eye conditions. Ophthalmol Sci. 2022;2 doi: 10.1016/j.xops.2022.100151. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Hatef E., Rouhizadeh M., Tia I., et al. Assessing the availability of data on social and behavioral determinants in structured and unstructured electronic health records: a retrospective analysis of a multilevel health care system. JMIR Med Inform. 2019;7 doi: 10.2196/13802. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Shanklin R., Samorani M., Harris S., Santoro M.A. Ethical redress of racial inequities in AI: lessons from decoupling machine learning from optimization in medical appointment scheduling. Philos Technol. 2022;35:96. doi: 10.1007/s13347-022-00590-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Gianfrancesco M.A., Tamang S., Yazdany J., Schmajuk G. Potential biases in machine learning algorithms using electronic health record data. JAMA Intern Med. 2018;178:1544–1547. doi: 10.1001/jamainternmed.2018.3763. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Wang H.E., Weiner J.P., Saria S., et al. Assessing racial bias in healthcare predictive models: practical lessons from an empirical evaluation of 30-day hospital readmission models. J Biomed Inform. 2024;156 doi: 10.1016/j.jbi.2024.104683. [DOI] [PubMed] [Google Scholar]
  • 44.Blumenthal D.M., Singal G., Mangla S.S., et al. Predicting non-adherence with outpatient colonoscopy using a novel electronic tool that measures prior non-adherence. J Gen Intern Med. 2015;30:724–731. doi: 10.1007/s11606-014-3165-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Mason J.A., Friedman E.E., Devlin S.A., et al. Predictive modeling of lapses in care for people living with HIV in Chicago: algorithm development and interpretation. JMIR Public Health Surveill. 2023;9 doi: 10.2196/43017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Okada A., Hashimoto Y., Goto T., et al. A machine learning-based predictive model to identify patients who failed to attend a Follow-up visit for diabetes care after recommendations from a national screening program. Diabetes Care. 2022;45:1346–1354. doi: 10.2337/dc21-1841. [DOI] [PubMed] [Google Scholar]
  • 47.Ridgway J.P., Ajith A., Friedman E.E., et al. Multicenter development and validation of a model for predicting retention in care among people with HIV. AIDS Behav. 2022;26:3279–3288. doi: 10.1007/s10461-022-03672-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48.Ding X., Gellad Z.F., Mather C., 3rd, et al. Designing risk prediction models for ambulatory no-shows across different specialties and clinics. J Am Med Inform Assoc. 2018;25:924–930. doi: 10.1093/jamia/ocy002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Neighborhood Atlas - Home. https://www.neighborhoodatlas.medicine.wisc.edu
  • 50.Cai C.X., Michalak S.M., Stinnett S.S., et al. Effect of ICD-9 to ICD-10 transition on accuracy of codes for stage of diabetic retinopathy and related complications: results from the CODER study. Ophthalmol Retina. 2021;5:374–380. doi: 10.1016/j.oret.2020.08.004. [DOI] [PubMed] [Google Scholar]
  • 51.Harrigian K., Tran D., Tang T., et al. Improving the identification of diabetic retinopathy and related conditions in the electronic health record using natural language processing methods. Ophthalmol Sci. 2024;4 doi: 10.1016/j.xops.2024.100578. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary Material
mmc1.pdf (1.1MB, pdf)

Articles from Ophthalmology Science are provided here courtesy of Elsevier

RESOURCES