Deep learning algorithms to detect diabetic kidney disease from retinal photographs in multiethnic populations with diabetes

Abstract

Objective

To develop a deep learning algorithm (DLA) to detect diabetic kideny disease (DKD) from retinal photographs of patients with diabetes, and evaluate performance in multiethnic populations.

Materials and methods

We trained 3 models: (1) image-only; (2) risk factor (RF)-only multivariable logistic regression (LR) model adjusted for age, sex, ethnicity, diabetes duration, HbA1c, systolic blood pressure; (3) hybrid multivariable LR model combining RF data and standardized z-scores from image-only model. Data from Singapore Integrated Diabetic Retinopathy Program (SiDRP) were used to develop (6066 participants with diabetes, primary-care-based) and internally validate (5-fold cross-validation) the models. External testing on 2 independent datasets: (1) Singapore Epidemiology of Eye Diseases (SEED) study (1885 participants with diabetes, population-based); (2) Singapore Macroangiopathy and Microvascular Reactivity in Type 2 Diabetes (SMART2D) (439 participants with diabetes, cross-sectional) in Singapore. Supplementary external testing on 2 Caucasian cohorts: (3) Australian Eye and Heart Study (AHES) (460 participants with diabetes, cross-sectional) and (4) Northern Ireland Cohort for the Longitudinal Study of Ageing (NICOLA) (265 participants with diabetes, cross-sectional).

Results

In SiDRP validation, area under the curve (AUC) was 0.826(95% CI 0.818-0.833) for image-only, 0.847(0.840-0.854) for RF-only, and 0.866(0.859-0.872) for hybrid. Estimates with SEED were 0.764(0.743-0.785) for image-only, 0.802(0.783-0.822) for RF-only, and 0.828(0.810-0.846) for hybrid. In SMART2D, AUC was 0.726(0.686-0.765) for image-only, 0.701(0.660-0.741) in RF-only, 0.761(0.724-0.797) for hybrid.

Discussion and conclusion

There is potential for DLA using retinal images as a screening adjunct for DKD among individuals with diabetes. This can value-add to existing DLA systems which diagnose diabetic retinopathy from retinal images, facilitating primary screening for DKD.

Introduction

Diabetic kidney disease (DKD) is the leading cause of end-stage renal disease worldwide.¹ Early detection of DKD would allow prompt preventive actions to limit excess morbidity and mortality in patients with diabetes mellitus (DM).² Current guidelines recommend yearly blood tests to calculate estimated glomerular filtration rate (eGFR) from serum creatinine, or spot urine sampling for urine albumin/creatinine ratio (UACR).^3,4 In individuals with DM, more frequent monitoring may be appropriate because of increased risk of progressive kidney disease relative to individuals with no diabetes.^3,4 Furthermore, population screening for chronic kidney disease (CKD) among the DM subgroup was found to have acceptable cost effectiveness.⁵ However, studies in the United States,⁶ Australia,⁷ and Asia⁸ have reported suboptimal utilization or poor adherence to screening, and underdiagnosis of DKD. This remains a crucial barrier to early intervention.

Retinal photography is noninvasive and convenient, commonly used in primary care settings for screening of eye pathologies, particularly diabetic retinopathy. Because the retina and other end organs, including the kidneys, share similar structural, physiological (renin-angiotensin-aldosterone system), and pathogenic (inflammation, oxidative stress, endothelial dysfunction, microangiopathy) properties, the retinal vessels are an indirect representation of renal microvasculature.⁹ Clinically appreciable retinal microvascular changes have been associated with DKD,^10,11 suggesting that retinal images contain substantial representative information of the kidney’s function. As the prevalence of diabetes grows worldwide,¹² a noninvasive tool to screen for DKD would complement existing deep learning algorithm (DLA) systems to diagnose diabetic retinopathy from retinal images,¹³ facilitating primary care screening for complications of DM. Our group has previously developed and validated a DLA to detect CKD from retinal photographs in the general population in Singapore (RetiKid), which showed good performance in external datasets from Singapore and China.¹⁴ In this study, we developed and validated a DLA for detecting DKD (RetiKid-Diab) using retinal images from a clinic-based diabetic population. This model was compared with 2 other models—one using clinical risk factor (RF) data and another hybrid model combining a retinal imaging score and RF data—to assess if this could lead to improved DKD predictions compared to an image-only model.

Methods

This study was approved by the Centralized Institutional Review Board (CIRB) of SingHealth, Singapore and conducted in accordance with the Declaration of Helsinki. Patients’ informed consent was exempted by the CIRB for the use of deidentified health information and retinal images. We performed a conventional development, validation, and external testing study on 3 models (retinal images only; RF only; hybrid) using retinal images and clinical data collected from 3 population-based studies. We developed and internally validated the models using data from the Singapore Integrated Diabetic Retinopathy Program (SiDRP).¹⁵ External testing was performed on 2 independent datasets: (1) Singapore Epidemiology of Eye Diseases (SEED) study¹⁶ and (2) Singapore Macroangiopathy and Microvascular Reactivity in Type 2 Diabetes (SMART2D) in Singapore.

Definition of DKD

eGFR was calculated from serum creatinine using the CKD Epidemiology Collaboration (CKD-EPI) creatinine equation.¹⁷ Since SiDRP provided annual screening for retinopathy among individuals with diabetes since 2010, presence and absence of DKD were assessed in all visits where serum creatinine data were available. We included individuals with 4 or more screening visits. We defined DKD (cases) as those with eGFR < 60 mL/min/1.73 m² on ≥2 consecutive visits between 3 months to 2 years apart (in the SiDRP development cohort; in external test cohorts, DKD was defined by a single visit). We defined no DKD (controls) as eGFR of ≥60 mL/min/1.73 m² in all visits. Definitions of DKD in the external validation datasets (SEED and SMART2D) were the same as SiDRP.

Training dataset for the DLA

For development, data and retinal images were obtained from patients with DM who participated in SiDRP (2010-2019), a national-level, telemedicine-based program established in 2010 to optimize eye screening for a general urban diabetes population.¹⁵ For each patient, 2 retinal photographs (optic disc-centered and macula-centered) were taken from each eye after pupil dilation according to the Early Treatment for Diabetic Retinopathy Study (ETDRS) protocol using a digital retinal camera (TRC-NW200, Topcon, Japan). Figure 1 is a flowchart detailing the breakdown of participants and images included from SiDRP. 187 563 visits from 79 511 unique individuals were recruited and assessed for eligibility. After excluding samples missing creatinine or age data for eGFR calculation, having unstable CKD status, or poor image quality, 5356 cases (DKD positive visits) and 7928 controls (DKD negative visits) from SiDRP (total 6066 unique participants) were used for training and validation of the algorithm.

Figure 1.

Flowchart showing detailing inclusion and exclusion of participants and images from SiDRP.

External testing datasets for the DLA

Two datasets were used for external testing: SEED and SMART2D. SEED is an ongoing population-based study of Chinese, Malay, and Indian participants aged ≥40 years at baseline. Detailed methods of SEED have been published.^18,19 After excluding those missing eGFR data or with poor quality images, data, and images from 1885 participants in SEED (798 cases, 1171 controls) were used for external testing, providing a total of 3938 fundus photographs (Table 1). SMART2D was a cross-sectional study conducted between 2011 and 2014 including 2057 adults aged 21-90 years with Type 2 DM. Detailed methods of SMART2D have been published.²⁰ 1163 participants of SMART2D with eye screening visits were recruited for this study, of which 439 participants (227 cases, 485 controls) were eventually included, totaling 1424 fundus photographs.

Table 1.

Baseline characteristics of participants.

	SiDRP (2010-2019)	SEED Diabetes cohort		SMART2D (2011-2014)
	N = 13 284	N = 1969	P-value	N = 712	P-value
Number of unique participants	6066	1885		439
Number of images	26 568	3938		1424
Case: control	5356: 7928	798: 1171	.879	227: 485	<.001
Chronic kidney disease			<.001		<.001
Stage 3	4741 (88.5)	694 (87.0)		124 (54.6)
Stage 4	558 (10.4)	79 (9.9)		44 (19.4)
Stage 5	57 (1.1)	25 (3.1)		59 (26.0)
Age (years)	64.1 (10.8)	64.0 (9.2)	.914	57.4 (10.7)	<.001
Sex			.060		.208
Female	6526 (49.1)	922 (46.8)		332 (46.6)
Male	6758 (50.9)	1047 (53.2)		380 (53.4)
Ethnicity			<.001		<.001
Malay	1833 (14.3)	705 (35.8)		159 (22.8)
Indian	1145 (8.9)	868 (44.1)		165 (23.6)
Chinese	9821 (76.7)	396 (20.1)		374 (53.6)
Duration of Diabetes (years)	7.0 [3.0, 12.0]	7.0 [1.2, 14.7]	.306	13.0 [7.0, 20.0]	<.001
Hemoglobin A1C (%)	7.2 (1.2)	7.6 (1.6)	<.001	8.3 (1.8)	<.001
eGFR (mL/min/1.73 m2)	74.1 (27.1)	71.7 (25.8)	<.001	74.8 (33.6)	.504
Systolic blood pressure (mmHg)	130.2 (15.9)	143.8 (21.3)	<.001	143.5 (19.8)	<.001

Abbreviations: eGFR, estimated glomerular filtration rate; SiDRP, Singapore Integrated Diabetic Retinopathy Program; SEED, Singapore Epidemiology of Eye Diseases; SMART2D, Singapore Macroangiopathy and Microvascular Reactivity in Type 2 Diabetes data.

Data are n (%), mean (standard deviation, SD), or median [interquartile range, IQR]. Some numbers (eg, ethnicity) may not add up due to the presence of missing data. P-values were calculated between SiDRP and each external test set using Pearson’s chi-squared test, Student t-test, or Mann-Whitney U-test as appropriate for the variable.

Risk factors

We used 6 classic RF (age, sex, ethnicity, duration of diabetes, HbA1c, and systolic blood pressure [SBP]) as predictors for the RF model. Age, sex, ethnicity, and duration of diabetes were self-reported in all cohorts. HbA1c and SBP were obtained from physical examination or laboratory tests in all datasets.

Algorithm architecture and development

The image-only DLA was trained on 26 568 retinal images from 6066 SiDRP participants. The DLA models were based off the ResNet18²¹ neural network architecture (Figure 2). The input layer takes 2 standardized macula-centered images (1 image per eye per participant) with resolution of 512 × 512. The output layer was a binary classifier with one node predicting the presence of DKD. During the training process, network parameters were initialized with weights pretrained using a large-scale diabetic retinopathy dataset (https://www.kaggle.com/competitions/diabetic-retinopathy-detection/data) to improve generalizability. For each image, the prediction given by the neural network is compared with its ground truth label, and parameters updated via backpropagation to reduce prediction error. We used 5-fold cross-validation to evaluate model performance, preserving the ratio of DKD cases and controls from the original dataset. The validation set had no overlap with the training set. The performance of the trained DLA was evaluated on the validation set by calculating the AUC combining the five sets of scores. Heatmaps were generated to identify the most important regions in a retinal image contributing to the DLA classification decision.

Figure 2.

Convolutional neural network architecture for predicting diabetic kidney disease from retinal images.

Statistical analysis

We presented the characteristics of participants using number (%), mean (standard deviation, SD), or median [interquartile range, IQR] as appropriate for the variable. Model data from SiDRP were compared with each external validation set using Pearson’s chi-squared test, Fisher’s exact test, Student t-test, or Mann-Whitney U-test as appropriate (P-value <.05). We developed 3 models: (1) image-only model using retinal images; (2) RF-only model using multivariable logistic regression (LR) adjusted for age, sex, ethnicity, duration of diabetes, HbA1c, and SBP; (3) hybrid model using multivariable LR adjusted for age, sex, ethnicity, duration of diabetes, HbA1c, SBP, and standardized z-scores from the image-only model. Primary analysis was to evaluate the performance of the 3 models by calculating the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) at the optimal threshold defined by Youden’s J Index, on internal validation and external validation.

Supplementary analysis

We performed several supplementary analyses including: (1) because misclassification of DKD would be common among those with eGFR near the normal range (55-60 mL/min/1.73 m²), we tested all 3 models using an alternate definition of DKD (eGFR <45 mL/min/1.73 m²; G3b and above). (2) We recalculated model performance when sensitivity or specificity were fixed at 80%. (3) We performed subgroup analysis calculating model accuracy for individuals in different CKD severity stages. (4) We externally validated the DLA in 2 predominantly Caucasian cohorts—a high-risk cohort for coronary artery disease (CAD) from the Australian Eye and Heart Study (AHES),²² and an older cohort from the Northern Ireland Cohort for the Longitudinal Study of Ageing (NICOLA).²³ We performed these as supplementary analyses because the sample size and number of events were low. AHES is a cross-sectional study of 1680 participants (460 included in this study) presenting to the Westmead Hospital in Sydney for assessment of suspected CAD between 2009 and 2012.²² NICOLA is a cross-sectional study which collected health and lifestyle data from 8452 participants (265 included in this study) in Northern Ireland, aged ≥50 years. Baseline data were collected between 2013 and 2016. For AHES and NICOLA RF-only and Hybrid models, we did not adjust for ethnicity because a majority of AHES and NICOLA participants were Caucasian; in AHES, we also did not adjust for duration of diabetes as this information was unavailable. For missing values of duration of diabetes (in NICOLA), SBP and HbA1c, we used mean/mode imputation as per the SMART2D cohort. (5) Finally, we performed error analysis of false positive (FP) and false negative (FN) samples by key characteristics including albuminuria and presence of ocular diseases to gain some insights into the misclassification by the DLA.

Results

Participant characteristics of each dataset are shown in Table 1. Characteristics of participants in SiDRP and SEED were similar, except for higher mean SBP in SEED (SiDRP: 130.2 [SD 15.9] mmHg vs SEED: 143.8 [SD 21.3] mmHg). There were several differences between the SMART2D dataset and the SiDRP or SEED datasets: (1) Participants from SMART2D were younger (57.4 years [SD 10.7]) than SiDRP (64.1 years [SD 10.8]) and SEED (64.0 years [SD 9.2]). (2) In terms of eGFR, CKD cases were generally more severe in SMART2D (tertiary care patients) than SiDRP (primary care patients) and SEED (population-based). There was a higher proportion of stage G4 and G5 CKD patients in SMART2D (G4: 19.4%; G5: 26.0%) than SiDRP (G4: 10.4%; G5: 1.1%) and SEED (G4: 9.9%; G5: 3.1%). (3) Ratio of cases (DKD positive visits) to controls (DKD negative visits) were lower in SMART2D (1 case: 2.1 controls) than SiDRP (1 case: 1.5 controls) and SEED (1 case: 1.5 controls). (4) Participants in SMART2D had a longer duration of diabetes (Median 13.0 years [IQR 7.0, 20.0]) than SiDRP (7.0 years [IQR 3.0, 12.0]) and SEED (7.0 years [IQR 1.2, 14.7]). (5) HbA1c was higher in participants from SMART2D (8.3% [SD 1.8]) than SiDRP (7.2% [SD 1.2]) and SEED (7.6% [SD 1.6]).

Figure 3 shows the receiver operating characteristic (ROC) curve plots for the image-only DLA, RF-only multivariable LR model, and hybrid model. In SiDRP validation, AUC was 0.826 (95% CI 0.818-0.833) for image-only, 0.847 (0.840-0.854) for RF-only, and 0.866 (0.859-0.872) for hybrid. In external validation with SEED, AUC was 0.764 (0.743-0.785) for image-only, improving to 0.802 (0.783-0.822) for RF-only and 0.828 (0.810-0.846) for hybrid. In SMART2D external validation, AUC was 0.726 (0.686-0.765) for image-only, decreasing to 0.701 (0.660-0.741) in RF-only and improving to 0.761 (0.724-0.797) for hybrid. Table 2 provides additional performance metrics to compare both models in internal validation and external test sets, at the optimal cut-off point defined by Youden’s J Index. In SiDRP, the image-only model had a sensitivity of 76% and specificity of 75%. The RF-only and hybrid models had sensitivity ranging from 78% to 79% and specificity ranging from 76% to 77%. In SEED, the image-only model had a sensitivity of 70% and specificity of 71%. The RF-only and hybrid models had higher sensitivities at 79% and 76%, respectively, while specificity was 67% and 74%, respectively. In SMART2D, image-only model had a sensitivity of 64% and specificity of 71%. RF-only and hybrid models had higher sensitivities at 75% and 71%, respectively, while specificity was 61% and 72%, respectively. NPV for the image-only model was 82% in SiDRP, 78% in SEED, and 81% in SMART2D. In RF-only and hybrid models, NPV levels were generally higher, ranging from 82% to 86% in all datasets. PPV values were consistently lower than NPV in all models and all datasets.

Figure 3.

ROC curves for prediction of diabetic kidney disease in image-only, RF-only, and hybrid models.

Table 2.

Performance of the 3 models in internal and external test sets at the optimal thresholds by Youden’s J Index.

	AUC (95% CI)	Sensitivity	Specificity	PPV	NPV
SiDRP, n = 13 284
Image-only	0.826 (0.818-0.833)	0.76	0.75	0.67	0.82
RF-only	0.847 (0.840-0.854)	0.78	0.76	0.68	0.85
Hybrid	0.866 (0.859-0.872)	0.79	0.77	0.69	0.86
SEED, n = 1969
Image-only	0.764 (0.743-0.785)	0.70	0.71	0.62	0.78
RF-only	0.802 (0.783-0.822)	0.79	0.67	0.61	0.82
Hybrid	0.828 (0.810-0.846)	0.76	0.74	0.66	0.82
SMART2D, n = 712
Image-only	0.726 (0.686-0.765)	0.64	0.71	0.51	0.81
RF-only	0.701 (0.660-0.741)	0.75	0.61	0.47	0.84
Hybrid	0.761 (0.724-0.797)	0.71	0.72	0.55	0.84

Abbreviations: AUC, area under the receiver operating characteristics curve; SiDRP, Singapore Integrated Diabetic Retinopathy Program; SEED, Singapore Epidemiology of Eye Diseases; SMART2D, Singapore Macroangiopathy and Microvascular Reactivity in Type 2 Diabetes data.

RF-only: Logistic regression adjusted for age, sex, ethnicity, diabetes duration, HbA1c, and systolic blood pressure. Hybrid: Logistic regression adjusted for age, sex, ethnicity, diabetes duration, HbA1c, systolic blood pressure, and the Image-only predicted z scores.

Table 3 shows the results of sensitivity analysis using the alternate definition of DKD (eGFR <45 mL/min/1.73 m²). Model performance improved for all models in SiDRP and SEED, and the image-only model in SMART2D. In the image-only model, AUC improved from 0.826 (0.818-0.833) to 0.851 (0.843, 0.860) in SiDRP; from 0.764 (0.743-0.785) to 0.785 (0.757-0.813) in SEED, and from 0.726 (0.686-0.765) to 0.759 (0.718-0.800) in SMART2D. In hybrid models, AUCs improved from 0.866 (0.859-0.872) to 0.887 (0.880-0.895) in SiDRP; from 0.828 (0.810-0.846) to 0.851 (0.829-0.874) in SEED; and from 0.761 (0.724-0.797) to 0.765 (0.723-0.806) in SMART2D. The highest AUC in external validation was achieved in SEED hybrid model, with an AUC of 0.851 (0.829-0.874).

Table 3.

AUC (95% CI) of the 3 models excluding DKD Stage G3a (eGFR 45-60 mL/min/1.73 m²).

	Case: control	Image-only (95% CI)	RF-only (95% CI)	Hybrid (95% CI)
SiDRP, n = 10 311	2383: 7928	0.851 (0.843-0.860)	0.868 (0.860-0.877)	0.887 (0.880-0.895)
SEED, n = 1508	337: 1171	0.785 (0.757-0.813)	0.822 (0.797-0.848)	0.851 (0.829-0.874)
SMART2D, n = 645	160: 485	0.759 (0.718-0.800)	0.678 (0.631-0.725)	0.765 (0.623-0.806)

Abbreviations: AUC, area under the receiver operating characteristics curve; SiDRP, Singapore Integrated Diabetic Retinopathy Program; SEED, Singapore Epidemiology of Eye Diseases; SMART2D, Singapore Macroangiopathy and Microvascular Reactivity in Type 2 Diabetes data.

Optimal probability threshold chosen by Youden’s J Index.

We also calculated model performances when sensitivity or specificity were fixed at 80% (Table S1). When sensitivity was fixed at 80%; the image-only model had a specificity of 70% in SiDRP, 57% in SEED, and 51% in SMART2D. Specificities of the RF-only and hybrid models ranged from 51% to 77%. When specificity was fixed at 80%; the image-only model had a sensitivity of 70% in SiDRP, 58% in SEED, and 53% in SMART2D. Sensitivities of the RF-only and hybrid models ranged from 40% to 77%. Next, we performed subgroup analysis for individuals in different eGFR categories. When fixed at 80% sensitivity, the image-only model was able to detect 81%-82% of the cases in Stage G3B CKD, 87%-93% of the cases in Stage G4 CKD, and 82%-96% of the Stage G5 DKD.

In our error analysis of the image-only model, images from controls (non-DKD) are more likely to be labeled positive (FP) if they were taken from patients that were older (66.2 [7.3] vs 56.2 [8.1] years), of Chinese ethnicity, had longer duration of diabetes (6.0 [3.0, 9.0] vs 4.0 [2.0, 8.0] years), higher SBP (129.9 [15.8] vs 126.0 [14.4] mmHg), lower HbA1c (7.0 [1.0] vs 7.2 [1.2] %), or had cataracts (n = 28/1131, 2.5% vs n = 9/4007, 0.2%). Images from DKD cases were more likely to be labeled negative (FN) if they were taken from patients that were younger (65.1 [7.7] vs 75.1 [7.5] years), of Malay ethnicity, had lower SBP (128.0 [16.9] vs 131.7 [17.1] mmHg), or did not have cataracts (n = 1/535, 0.2% vs n = 68/985, 6.9%).

Finally, in our supplementary analysis on the AHES and NICOLA cohort, characteristics of AHES and NICOLA participants are provided in Table S2. Regarding model performance in AHES, the image-only model achieved an AUC of 0.670 (0.612-0.729), which was slightly lower than its performance in the 3 main datasets. Otherwise, the RF-only and hybrid models performed similarly, achieving AUCs of 0.685 (0.626-0.745) and 0.695 (0.640-0.751) respectively (Figure S1). Regarding model performance in NICOLA, the image-only model achieved an AUC of 0.638 (0.562-0.714), which was lower than its performance in the 3 main datasets. However, the RF-only and hybrid models performed well, reaching AUCs of 0.721 (0.652-0.790) and 0.710 (0.640-0.779) respectively (Figure S1).

Discussion

We developed and validated a DLA for detecting DKD from retinal images (RetiKid-Diab), aiming to determine if the models are sufficiently robust to screen individuals with diabetes in the primary care setting. To our knowledge, this is the first study that attempts to predict DKD from retinal images in a population with diabetes, augmenting existing reports of CKD diagnosis using fundus images in the general population.^14,24 Our models showed reasonable performance, faring well in internal validation (AUC image-only = 0.826 [0.818-0.833]), with moderate performance in external validation (AUC image-only = 0.764 [0.743-0.785] in SEED; 0.726 [0.686-0.765] in SMART2D). In particular, the image-only model performed comparably well in all datasets compared to the RF-only model. That being said, the hybrid models, comprising both retinal image and RF, performed better than the image-only or RF-only versions across all datasets. Taken together, these results suggest that there is potential for DLA using retinal images as a screening adjunct for DKD among individuals with diabetes, alongside standard screening methods. Inclusion of common, readily acquirable RFs will add some value to the performance of the algorithm, if it were translated into the primary care setting.

An important consideration for any artificial intelligence (AI)-based screening system is its clinical relevance. DKD has a long asymptomatic phase, and early detection is critical for optimal management. It is widely reflected in medical guidelines that people with diabetes should be regularly evaluated for DKD, typically with an annual urine test for albuminuria and a blood test for serum creatinine to estimate GFR.²⁵ However, current screening rates are suboptimal²⁶; in a systematic review exploring screening rates among individuals with diabetes for diabetes-related complications, de Jong et al²⁷ reported that two-thirds of studies described nephropathy screening rates of less than 70%. In the Korean National Health and Nutrition Examination Survey, only 40.5% of patients with diabetes received screening for diabetic nephropathy during the previous year, even though they knew that they had diabetes.²⁸ While an AI-based retinal image screening system for DKD may not replace current screening methods in the near future, it has potential to serve as a screening adjunct, to improve worldwide screening rates. Firstly, telemedicine for diabetic retinopathy screening among those with diabetes has remained strongly cost-effective compared with in-person office screening.²⁹ Secondly, AI-based diabetic retinopathy screening programs have begun real-world implementation³⁰; a noninvasive, low cost, point-of-care DKD screening tool that uses the same input (retinal fundus photographs) provides the opportunity for simultaneous screening of 2 major microvascular complications of diabetes (DR+DKD) at the population level. For example, in Singapore, patients on follow-up for diabetes are routinely screened for referable DR using a nationwide implemented deep learning software—Singapore Eye Lesions Analyzer (SELENA).³¹ Using the same retinal images, patients could be screened for DKD as well.

Our results had several notable trends. There was a reduction in performance of the image-only model on SMART2D external validation, with particularly poorer specificity and PPV. However, this was also noticed with the RF-only and Hybrid models. The proportion of DKD cases in SMART2D (31.9%) was lower than SiDRP (40.3%) or SEED (40.5%), which could explain the lower PPV. Comparing this to real-world rates, reported prevalence of DKD among patients with diabetes vary greatly from approximately 20%-40%,^32,33 depending on the study population and presence of cardiovascular comorbidities. To increase PPV in populations with low prevalence of DKD, our DLA could be applied to higher risk individuals, including individuals with diabetes that are older, have poorer glycemic control, or multiple cardiovascular comorbidities. We also suggest applying this tool as part of a 2-stage screening, where a noninvasive DLA pegged at higher sensitivity can be applied first; individuals who screen positive are recalled for further testing with serum creatinine, to exclude false positives. There were 2 other notable trends regarding the image-only model in our results. First, performance of the image-only model improved in all datasets when the stricter definition of DKD (eGFR <45 mL/min/1.73 m²) was used. Second, the accuracy of the image-only model when sensitivity was fixed at 80% increased with the severity of DKD. This would be beneficial for a community screening adjunct, to safeguard that moderate/severe cases are less likely to be missed. Next, in our error analysis of the image-only model, FP labels occurred in patients that were older, had a longer duration of diabetes, or higher SBP. Conversely, FN labels occurred in patients that were younger and had lower SBP. This trend suggests that a positive label is associated with patients with a higher cardiovascular risk profile (and vice versa), which is not unexpected. FP labels were more likely in patients with lower HbA1c, while FN labels more likely in patients with higher HbA1c. It seems contradictory that better glycemic control is associated with a positive label. FP labels were more likely to occur in patients with cataracts (and vice versa)—this could be related to the known association between CKD and cataracts.³⁴ The presence of cataracts can also affect the quality of fundus images, and in turn the performance of the DLA. Patients with albuminuria were more likely to have FP labels, suggesting that the DLA is identifying signals from those with early renal impairment (for example, stage 1 and 2 DKD which we classified as controls). Finally, on external validation, we observed slight reductions in AUCs for the image-only model. The small sample size of these external validation datasets (n = 460 in AHES, n = 265 in NICOLA) could explain the reduced, albeit modest, image-only model performance. With further training on more ethnically diverse datasets, our model has the potential to generalize well in populations of varying ethnic predominance, which is advantageous for any AI-based community screening device.

Several image-based DLAs have been created to screen for CKD,^14,24,35,36 but not for DKD (Table 4). The RetiKid-Diab algorithm we describe in this article is a “sister” algorithm to RetiKid,¹⁴ developed by our group in 2020. In our previous report,¹⁴ RetiKid was also tested on a subgroup of individuals with diabetes (ie, predicting DKD), achieving better AUCs than RetiKid-Diab on internal validation (RetiKid hybrid AUC: 0.925; RetiKid-Diab hybrid AUC: 0.876). The algorithms used were inherently different in architecture—RetiKid is a neural network, able to utilize nonlinear associations and interaction terms. RetiKid-Diab image-only was also a neural network. However, RetiKid-Diab RF-only and hybrid models are multivariable LR models, which cannot utilize interaction or high-order terms for prediction. We note that a direct comparison of AUCs does not provide a full picture of model performance, and we suggest further evaluation of existing retina-based CKD screening tools in diabetic cohorts.

Table 4.

Performance of deep-learning algorithms related to chronic kidney disease in current literature.

Study	Purpose	Limitations	Input	Model	Dataset	Validation	AUC	Sensitivity	Specificity
Betzler 2023a Current Study	To detect DKD in a diabetic population. DKD defined as eGFR < 60 mL/min/1.73 m2 (category G3 and above) on ≥2 consecutive visits between 3 months to 2 years apart.	Reduced performance when validated on Caucasian datasets.	Retinal fundus images	CNN	31 930 images SiDRP SEED SMART2D	Internal	0.854 (Image) 0.847 (RF) 0.876 (Hybrid)	0.77 (Image) 0.78 (RF) 0.81 (Hybrid)	0.78 (Image) 0.76 (RF) 0.79 (Hybrid)
						External 1	0.744 (Image) 0.802 (RF) 0.821 (Hybrid)	0.80 (Image) 0.79 (RF) 0.74 (Hybrid)	0.59 (Image) 0.67 (RF) 0.76 (Hybrid)
						External 2	0.724 (Image) 0.685 (RF) 0.745 (Hybrid)	0.77 (Image) 0.78 (RF) 0.74 (Hybrid)	0.63 (Image) 0.56 (RF) 0.68 (Hybrid)
Sabanayagam 202014	To detect CKD in a general population CKD defined as eGFR < 60 mL/min/1.73 m2 (category G3 and above) in either of 2 consecutive visits	No external validation on non-Asian datasets	Retinal fundus images	CNN	23 516 images SEED SP2 BES	Internal	0.911 (Image) 0.916 (RF) 0.938 (Hybrid)	0.83 (Image) 0.82 (RF) 0.84 (Hybrid)	0.83 (Image) 0.84 (RF) 0.85 (Hybrid)
						External 1	0.733 (Image) 0.829 (RF) 0.810 (Hybrid)	0.70 (Image) 0.73 (RF) 0.74 (Hybrid)	0.70 (Image) 0.80 (RF) 0.75 (Hybrid)
						External 2	0.835 (Image) 0.887 (RF) 0.858 (Hybrid)	0.75 (Image) 0.79 (RF) 0.79 (Hybrid)	0.75 (Image) 0.82 (RF) 0.79 (Hybrid)
						Internal, Diabetic Subgroup	0.889 (Image) 0.899 (RF) 0.925 (Hybrid)	—	—
Zhang 202124	To detect CKD in a general population CKD defined as eGFR <60 mL/min/1.73 m2 or eGFR >60 mL/min/1.73 m2 with albuminuria, confirmed in ≥2 visits separated by 3 months	Training and external validation sets were of similar ethnicity (Chinese)	Retinal fundus images	CNN	115 344 images CC-FII Guangdong COACS	External 1	0.918 (Image) 0.861 (RF) 0.930 (Hybrid)	—	—
						External 2	0.885 (Image) 0.842 (RF) 0.898 (Hybrid)	—	—
Kang 202035	To detect early renal function impairment in a general population Early renal function impairment defined as eGFR <90 mL/min/1.73 m2	No external validation	Retinal fundus images	CNN	25 706 images CGMH Taiwan	Internal	0.810 (Image)	0.830 (Image)	0.620 (Image)
Kuo 201936	To predict kidney function from kidney ultrasound images CKD defined as eGFR < 60 mL/min/1.73 m2 (category G3 and above)	No external validation	Kidney ultrasound images	CNN	4505 images CMUH Taiwan	Internal	0.904 (Image)	0.921 (Image)	0.606 (Image)

Abbreviations: AUC, area under the receiver operating curve; BES, Beijing Eye Study; CC-FII, China Consortium of Fundus Image Investigation; CGMH, Chang Gung Memorial Hospital; CKD, chronic kidney disease; CMUH, China Medical University Hospital in Taiwan; CNN, convolutional neural network; COACS, China Suboptimal Health Cohort Study; DKD, diabetic kidney disease; eGFR, estimated glomerular filtration rate; RF, Risk Factors; SiDRP, Singapore Integrated Diabetic Retinopathy Program; SEED, Singapore Epidemiology of Eye Disease; SMART2D, Singapore Macroangiopathy and Microvascular Reactivity in Type 2 Diabetes; SP2, Singapore Prospective Study Program.

^aThe current study is the only one conducted for DKD in a diabetic population. All other published studies listed here were conducted in a general population.

The strengths of our study include the validation of our DLA on independent cohorts with similar imaging and DKD diagnostic protocols. Next, we had a robust ground truth, where DKD was defined based on at least 2 measurements of eGFR on consecutive visits, potentially reducing misclassification of DKD cases. In addition, we explored the utility of hybrid models with additional clinical RF data, which showed improvement relative to image-only models. Nevertheless, our study has several limitations. First, because our study was based in an Asian population (Chinese, Indian, and Malay), our DLA would be more relevant in Asian countries with a high burden of diabetes and DKD. Further evaluation of its generalizability in non-Asian populations, and in more diverse demographic cohorts, may improve clinical utility and diagnostic accuracy. We explored this with our supplementary analyses on the predominantly Caucasian AHES (Australia) and NICOLA (Northern Ireland) cohorts, which demonstrated modest performance. Second, we did not have data on albuminuria for all participants, thus we could not incorporate albuminuria levels into the prediction. Third, we had a low representation of Stage G5 cases in the SiDRP training set, while Stage G5 cases were over-represented in SMART2D. Fourth, while heatmaps (Figure 4) indicated microvascular changes characteristic of retinopathy, it is unclear what specific features were used by the DLA to identify DKD. This is a problem faced by most existing image-based DLAs. A multistep algorithm that detects characteristic microvascular changes and uses these features to predict DKD is possible, although this might overcomplicate the prediction process without a substantial increase in performance. Fifth, our current model underperforms relative to our prior work in a general population.¹⁴ However, this could be because the training dataset of our prior work was under a SEED research dataset which has less noisy labels. Our current model was trained on a real-world dataset which could account for the difference in performance. We attempted other machine learning classifiers (such as random forest and support vector machine), but overall the results were suboptimal to LR. We also attempted vision transformers (ViT) as the deep learning image model, performance was comparable, but it was computationally intensive and the training took longer.

Figure 4.

Heatmaps of diabetic kidney disease controls and cases.

In conclusion, our study shows the potential of a retina image DLA to screen for DKD among individuals with diabetes. Since access to digital retinal photography is increasing at the primary care level, a retinal image-based DLA, if adopted, has the potential to improve DKD screening rates worldwide, as an adjunct to existing laboratory methods. Next steps might include validating the algorithm in non-Asian populations, among young patients with diabetes (ie, T1DM patients) and performing implementation and cost-effectiveness studies.

Author contributions

All authors contributed to the intellectual development of this article. C.S. conceived and planned the study. B.K.B. and C.S. wrote the manuscript. B.K.B. performed the literature review. E.Y.L.C. and F.H. performed the data analysis. B.K.B., E.Y.L.C., F.H., C.C.L., J.H., H.H., N.C.T., G.L., G.J.M.K., C.-Y.C., S.C.L., A.Y.L., T.Y.W., M.L.L., W.H., G.S.W.T., and C.S. contributed to the interpretation of the results and provided critical feedback to the manuscript. The final version of the article has been seen and approved by all authors.

Supplementary material

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

This study was supported by the National Medical Research Council (NMRC/STaR/003/2008, NMRC/OFLCG/001/2017, and NMRC/MOH-HCSAINV21jun-0001) and the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG-GC-2019-001-2A). We are grateful to all the participants of the NICOLA Study, and the whole NICOLA team, which includes nursing staff, research scientists, clerical staff, computer and laboratory technicians, managers, and receptionists. The Atlantic Philanthropies, the Economic and Social Research Council, the UKCRC Centre of Excellence for Public Health Northern Ireland, the Centre for Ageing Research and Development in Ireland, the Office of the First Minister and Deputy First Minister, the Health and Social Care Research and Development Division of the Public Health Agency, the Wellcome Trust/Wolfson Foundation, and Queen’s University Belfast provide core financial support for NICOLA. We also acknowledge staff and participants of SiDRP, SEED, SMART2D, and AHES. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Conflicts of interest

H.H. is a co-founder of start-up company EyRIS. G.L. has consulted for Bayer and Novartis and has received travel grants and research funding from Bayer. A.Y.L. reports grants from Santen, personal fees from Genentech, personal fees from US FDA, personal fees from Johnson and Johnson, grants from Carl Zeiss Meditec, personal fees from Topcon, personal fees from Gyroscope, non‐financial support from Microsoft, grants from Regeneron, outside the submitted work. T.Y.W. is a consultant for Aldropika Therapeutics, Bayer, Boehringer-Ingelheim, Genentech, Iveric Bio, Novartis, Oxurion, Plano, Roche, Sanofi, Shanghai Henlius. He is an inventor, holds patents, and is a co-founder of start-up companies EyRiS and Visre, which have interests in, and develop digital solutions for eye diseases. G.S.W.T. reported receiving travel support from Carl Zeiss Pte Ltd; receiving personal fees and consultancy fees from Novartis; receiving grants from Bayer AG and Santen Pharmaceutical Company, Ltd; and receiving travel support from Alcon outside the submitted work. The rest of the authors declare no conflicts of interest.

Data availability

As the study involves human participants, the data cannot be made freely available in the manuscript, the supplemental files, or a public repository due to ethical restrictions. Nevertheless, the data are available from the Singapore Eye Research Institutional Ethics Committee for researchers who meet the criteria for access to confidential data. Interested researchers can send data access requests to the Singapore Eye Research Institute using the following email address: seri@seri.com.sg.