OptiDON-Guard: An Ensemble Model for Identification of Dysthyroid Optic Neuropathy Using Optical Coherence Tomography Angiography

Abstract

Purpose

To explore the value of machine learning models based on optical coherence tomography angiography (OCTA) in identifying dysthyroid optic neuropathy (DON).

Methods

A retrospective, dual-center, cross-sectional study was conducted involving 346 thyroid eye disease patients (with/without DON) from two institutions. Participants underwent ophthalmic examinations, including OCTA scans centered on the macula and optic nerve head. The OCTA data from Center 1 were split into a training set and an internal validation set, while the data from Center 2 served as an external test set. Features were extracted from OCTA images, and a wrapper-based feature selection strategy identified the optimal subset. A two-tiered stacking modeling framework was used, incorporating baseline model construction, data augmentation using pseudo-labeling, and ensemble integration. Model performance (area under the curve [AUC], accuracy, sensitivity, specificity, precision) was evaluated alongside receiver operating characteristic, calibration, and decision curve analyses. Additionally, the SHapley Additive exPlanation method was used to interpret feature contributions.

Results

The ensemble model, OptiDON-Guard, demonstrated superior performance to individual models on both internal validation and external test sets. In the internal validation set, it achieved an AUC of 0.872, accuracy of 0.943, sensitivity of 0.667, specificity of 0.979, and precision of 0.943. In the external test set, it maintained strong performance with an AUC of 0.854, accuracy of 0.816, sensitivity of 0.767, specificity of 0.842, and precision of 0.854.

Conclusions

The OptiDON-Guard is a reliable, interpretable, and clinically applicable tool for identifying DON.

Translational Relevance

The OptiDON-Guard offers a valuable clinical tool to enhance the diagnostic precision of DON.

Introduction

Dysthyroid optic neuropathy (DON) is a vision-threatening complication of thyroid eye disease (TED), occurring in approximately 4%–8% of the cases.¹ This critical condition arises when increased orbital content compresses or stretches the optic nerve, leading to disruption of axonal flow or ischemia of the nerve.² DON typically presents with insidious and progressive vision loss that is often bilateral and asymmetric, along with dyschromatopsia and visual field deficits.³ Without timely diagnosis and intervention, DON-related vision loss can be permanent. However, current diagnostic paradigms for DON lack standardized criteria, primarily dependent on clinical evaluation of non-specific manifestations, including visual acuity decline, dyschromatopsia, relative afferent pupillary defect, visual field defects, and optic disc edema.² Crucially, these findings demonstrate poor specificity because they overlap significantly with other optic neuropathies, particularly glaucomatous and/or other types of optic neuropathy. This phenotypic ambiguity directly contributes to suboptimal diagnostic accuracy and failure to meet clinical requirements. To overcome this clinical void, there is an urgent requirement to develop an objective and reliable diagnostic system for DON, which would enable timely intervention to preserve vision.

Optical coherence tomography angiography (OCTA) is a noninvasive interferometric imaging technique that enables in vivo cross-sectional imaging of the retina and is capable of producing high-resolution three-dimensional images of retinal vascular and their surrounding tissues.^4,5 It has been widely used in ophthalmic diagnosis and monitoring, particularly for glaucoma and retinal diseases.^6,7 Studies have shown significant changes in retinal vascular density and retinal structure in patients with DON, including peripapillary vascular perfusion, the peripapillary retinal nerve fiber layer (pRNFL), and the ganglion cell complex (GCC).^8–10 This suggests the potential of OCTA for assessing optic nerve status in DON disease. Despite the valuable insights provided by OCTA-derived images and data, their translation into clinical decisions remains challenging because of the absence of automated analysis tools, which rely on subjective interpretation. Advanced decision support systems are urgently required to assist clinicians in interpreting OCTA data effectively and objectively, thereby improving diagnostic accuracy.

Machine learning (ML), a subset of artificial intelligence, empowers decision support systems to autonomously extract patterns from complex datasets and generate generalizable insights for data-driven decision-making. By leveraging ML's ability to model nonlinear relationships and adapt to evolving data, these systems significantly enhance diagnostic accuracy, reduce human bias, and optimize personalized therapeutic strategies.¹¹ Ensemble machine learning has emerged as a critical research domain in ML, primarily because of its demonstrated capacity to improve predictive performance through the strategic combination of multiple base models.¹² Among various ensemble methodologies, stacking represents a sophisticated approach that integrates diverse base models via a meta-model architecture. This hierarchical framework has shown particular promise in achieving performance optimization, as evidenced by its ability to leverage complementary strengths of constituent models while mitigating individual weaknesses.¹³

In this study, we introduced a stacking ensemble model named OptiDON-Guard, designed to identify DON by leveraging OCTA data. OptiDON-Guard contains a two-tiered stacking ensemble framework, which boosts model complementarity through stacked predictions while maintaining interpretability. To ensure robustness and generalizability, we externally test our model using data from an independent center. Furthermore, we used the SHapley Additive exPlanations (SHAP)¹⁴ method to interpret the model and provide insights into the relative contributions of features within the models. Our work demonstrates that OptiDON-Guard is an accurate, reliable, interpretable, and clinically applicable tool for identifying DON.

Methods

Study Design

The retrospective study enrolled consecutive patients with TED from the Eye Hospital of Wenzhou Medical University (Center 1) and Zhejiang Eye Hospital (Center 2) between October 2018 and February 2024. The study was approved by the hospital's ethics committee (Approval No. 202-067-K-58) and was conducted adhering to the tenets of the Declaration of Helsinki. Informed consent was waived by our institutional review board because of the retrospective nature of our study. TED was diagnosed based on the European Group on Graves’ Orbitopathy criteria¹⁵ and can be further divided into DON and non-DON groups. Three ophthalmologists (>10 years of experience) clinically assessed each patient. Varied diagnoses were discussed until a consensus diagnosis was reached. DON diagnosis was based on at least two of the following signs: decreased best-corrected visual acuity (BCVA; ≥0.1 logMAR), visual field loss with a mean deviation (<−5 dB), loss of color vision, optic disc swelling, and relative afferent pupillary defect.^2,16,17 In addition, all DON diagnoses were confirmed by orbital magnetic resonance imaging, presenting signs of orbital apical crowding.¹⁸ Patients were excluded if they met any of the following criteria: age < 18 years, severe refractive media opacity, uveitis, glaucoma, and retinal diseases. The flowchart of this study is shown in Figure 1.

Figure 1.

The flowchart of the study.

Ocular Assessment and OCTA Examination

Multiple ophthalmological examinations were collected on admission, including spherical equivalents with BCVA, slit-lamp, intraocular pressure (IOP), proptosis, perimetry, and OCTA. OCTA images were obtained with Optovue AngioVueTM (RTVue XR Avanti; Optovue Inc. Fremont, CA. USA) with a scanning speed of 70,000 A-scans per second, a central wavelength of 840 nm and a resolution of 304 × 304 pixels. These images were centered on two areas: the optic nerve head (ONH) with a region of 4.5 × 4.5 mm² and the macula with a region of 3.0 × 3.0 mm². ONH had a diameter of 4.0 mm and was divided into an inner disc with a diameter of 2.0 mm, and a peripapillary area, which was further subdivided into eight sectors according to the Garway-Heath map¹⁹ (Fig. 2a). The macula has a diameter of 3 mm divided into the fovea, with a diameter of 1.0 mm, and parafoveal area, which is further divided into four parafoveal quadrants according to the Quadrant Maps²⁰: superior (S), temporal (T), inferior (I), and nasal (N) (Fig. 2b). A masked reader reviewed all these images and excluded those with a signal strength index below 4/10, poor scan centering, or motion artifacts.

Figure 2.

Legend of Garway-Heath map of the optic nerve head and Quadrant Map of the macula. (a) High-definition disc Angio scan. (b) High-definition macular Angio scan. I, inferior; IN, inferior nasal; IT, inferior temporal; N, nasal; NI, nasal inferior; NS, nasal superior; S, superior; SN, superior nasal; ST, superior temporal; TI, temporal inferior; TS, temporal superior; T, temporal.

Extraction of Image Features

OCTA images provide detailed images of the retinal layers and their blood flow, enabling the assessment of structural thickness and vascular status. The changes in the thickness of the macular ganglion cell complex (mGCC) can reflect the health of retinal ganglion cells and their axons, which are associated with a variety of neurodegenerative diseases.²¹ The change of vascular density of the retinal capillary plexus (RCP) reflects the state of micro-circulation of the retina, and the disturbance of micro-circulation may lead to optic nerve disease.²² The pRNFL thickness and RPC vascular densities within ONH have been reported to be highly related to DON.²³ Based on the above research, from macular images, we extracted the thickness of the retina and mGCC, along with the vascular density of the RCP, including both the superficial retinal capillary plexus and deep retinal capillary plexus (DRCP). From ONH images, we extracted the thickness of pRNFL and RPC vascular densities. These parameters were automatically detected and measured using the built-in software of the OCTA device. Finally, 56 different thickness and vascular density features from the macula and ONH regions were extracted. These features are as follows: 14 macular thickness features (seven mGCC thickness features, seven macular retina thickness features), 10 ONH pRNFL thickness features, 14 macular vascular density parameters (seven superficial retinal capillary plexus features, seven DRCP features), and 18 ONH vascular density parameters.

Machine Learning Framework

Dataset Preparation

The study cohort comprised patients from two independent medical centers. Patients from Center 1 were randomly partitioned into a training set and an internal validation set at an 8:2 ratio for model development and hyperparameter tuning, respectively. Patients from Center 2 were reserved as an external test set to independently evaluate model generalizability. All datasets underwent Z-score normalization using the mean and variance derived exclusively from the Center 1 training set before partitioning.

Feature Selection

A wrapper-based feature selection strategy²⁴ was used to identify the optimal feature subset. Using a support vector machine (SVM) as the baseline classifier, a Monte Carlo randomization approach²⁵ was iterated 10,000 times to evaluate the feature subset. This process yielded a stable combination of 17 OCTA-derived features, which were retained for subsequent modeling.

Model Development

The modeling framework incorporated three stages: baseline model construction, data augmentation with pseudo-labeling, and ensemble integration. The Bayesian optimization²⁶ was used to select optimal hyperparameters for all machine learning models developed on the training set.

Stage 1: Baseline SVM Model

A primary SVM model (termed the “Teacher Model”) was initially trained on the 17 features selected before using the training set.

Stage 2: Pseudo-Label Augmentation for Student Model Training

To enhance model robustness and mitigate overfitting, a synthetic data generation strategy was implemented. Gaussian random sampling²⁷ was applied to the feature space to create virtual samples, which were then annotated using predictions from the “Teacher Model.” These synthetic samples, combined with the original training data, were utilized to train five distinct machine learning architectures as “Student Models”: Random Forest,²⁸ Support Vector Machine (SVM),²⁹ eXtreme Gradient Boosting (XGBoost),³⁰ Light Gradient Boosting Machine (LightGBM),³¹ and Categorical Boosting (CatBoost).³² This hybrid approach leveraged both real and augmented data to improve generalization capacity.

Stage 3: Stacking-Based Ensemble Model

A two-tiered stacking framework was designed to integrate predictions from the Teacher and Student Models. In the first Layer, all six models (Teacher + 5 Students) served as base models, generating probabilistic predictions on the training set. The base models’ outputs were concatenated as meta-features to form a meta-dataset. In the second layer, a linear-kernel SVM was trained as a meta-model to optimally combine the base-models’ predictions and produce the final ensemble output. When the ensemble model inference is performed, data is processed sequentially through the base models, with the meta-model synthesizing their probabilistic outputs into a consolidated prediction.

Model Evaluation and Interpretation

Model performance was assessed across training, internal validation, and external test sets using metrics including AUC, accuracy, sensitivity, specificity, and precision. Receiver operating characteristic (ROC) curves, calibration curves, and decision curve analysis were used to evaluate the clinical utility of models. Model interpretability was investigated via the SHAP method,³³ quantifying feature importance and decision logic across the ensemble framework.

Statistical Analysis

Continuous variables are presented as mean ± standard deviation. The Kolmogorov-Smirnov test was used to assess the normality of the data. For data that are normally distributed, the Student's t-test for continuous variables and the χ² test for categorical variables were used to evaluate differences between the two groups. In the case of non-normally distributed data, the Kruskal-Wallis test was used. Categorical variables were analyzed by the chi-square test and are shown as numbers (percentages). Statistical significance was defined as P < 0.05. All data analysis and visualization were carried out using SPSS (version 26.0) and Python (version 3.10.13).

Results

Demographic Characteristics and OCTA Features

The OCTA data of 259 eyes (231 non-DON and 28 DON) from Center 1 were used to form the training and internal validation sets, whereas the OCTA data of 87 eyes (57 non-DON and 30 DON) from Center 2 were used to form the external test set. The demographic and clinical characteristics are shown in Table 1. The heatmap plot shows the distribution of clinical characteristics and OCTA features (Fig. 3). The BCVA was significantly better in non-DON eyes compared to DON eyes in both centers (P < 0.001). Although the mean IOP was higher in DON eyes, this difference was not statistically significant in either center. A statistically significant difference in visual field mean deviation was observed between non-DON and DON eyes in both centers (P < 0.001). Additionally, a statistically significant difference was found in the proptosis between the two groups in Center 2 but not in Center 1.

Table 1.

Demographic Characteristics and Clinical Characteristic

	Center 1			Center 2
Characteristic	Non-DON (n = 231)	DON (n = 28)	P Value	Non-DON (n = 57)	DON (n = 30)	P Value
Age (years)	45.1 ± 12.6	60.0 ± 7.0	<0.001a	47.5 ± 13.7	57.8 ± 9.2	<0.001a
Sex			0.139b			0.894b
Male	113	9		27	13
Female	118	19		30	17
BCVA (LogMAR)	0.1 ± 0.2	0.4 ± 0.4	<0.001b	0 ± 0.1	0.4 ± 0.1	<0.001b
IOP (mm Hg)	17.6 ± 5.0	17.9 ± 8.0	0.077c	17.2 ± 2.9	20.5 ± 6.8	0.013c
PROP (mm)	18.9 ± 3.0	19.0 ± 2.9	0.840a	17.7 ± 2.7	18.9 ± 2.4	0.025a
VF-MD (dB)	−3.0 ± 4.4	−8.8 ± 8.8	<0.001c	−1.9 ± 1.6	−10.8 ± 9.9	<0.001c

PROP, proptosis; VF-MD, visual field-mean deviation.

^aStudent's t-test.

^bPearson's χ² test.

^cKruskal-Wallis test.

Figure 3.

The heatmap plot shows the distribution of clinical characteristics and OCTA features in the non-DON group and the DON group.

The Performance of Machine Learning Models

Table 2 summarizes the performance of the individual base machine learning models and the ensemble model, OptiDON-Guard, in identifying eyes with or without DON. All individual base models achieved AUC values exceeding 0.8 in both the internal validation set and the external test set. OptiDON-Guard outperformed these individual models with robustness and consistency across different datasets. Specifically, OptiDON-Guard demonstrated higher AUC values than any of the individual base machine learning models in both the internal validation set and the external test set. In the internal validation set, the OptiDON-Guard achieved an AUC of 0.872, accuracy of 0.943, sensitivity of 0.667, specificity of 0.979, and precision of 0.943. In the external test set, it maintained strong performance with an AUC of 0.854, accuracy of 0.816, sensitivity of 0.767, specificity of 0.842, and precision of 0.854. The ROC curves demonstrate the OptiDON-Guard's excellent classification performance, whereas calibration curves confirm that its predictive accuracy aligns well with actual outcomes. Decision curve analysis further shows that, compared to “all intervention” or “no intervention” strategies, the OptiDON-Guard achieves higher net benefits, indicating strong clinical utility across both the internal validation set and the external test set (Fig. 4).

Table 2.

The Performance of Different Models in Dysthyroid Optic Neuropathy Identification

Models	Accuracy	AUC	Sensitivity	Specificity	Precision
Internal validation set
Random Forest	0.717	0.805	0.833	0.702	0.263
CatBoost	0.792	0.840	0.833	0.787	0.333
Teacher SVM	0.792	0.851	0.500	0.830	0.273
SVM	0.792	0.798	0.833	0.787	0.333
LightGBM	0.717	0.869	1.000	0.681	0.286
XGBoost	0.868	0.809	0.667	0.894	0.444
Ensemble (OptiDON-Guard)	0.943	0.872	0.667	0.979	0.800
External test set
Random Forest	0.747	0.805	0.867	0.684	0.747
CatBoost	0.770	0.820	0.833	0.737	0.625
Teacher SVM	0.770	0.806	0.500	0.912	0.750
SVM	0.724	0.804	0.833	0.667	0.804
LightGBM	0.805	0.823	0.667	0.877	0.741
XGBoost	0.759	0.823	0.767	0.755	0.621
Ensemble (OptiDON-Guard)	0.816	0.854	0.767	0.842	0.719

CatBoost, categorical boosting; LightGBM, light gradient boosting machine; XGBoost, eXtreme gradient boosting.

Figure 4.

Performance of the individual models and the ensemble model for identifying DON in the internal validation set and the external test set. The ROC curves of models in the internal validation set (a) and the external test set (d). The calibration plots of models in the internal validation set (b) and the external test set (e). The decision curves of models in the internal validation set (c) and the external test set (f). LightGBM, Light Gradient Boosting Machine; XGBoost, eXtreme Gradient Boosting; CatBoost, Categorical Boosting.

Feature Importance and Model Interpretation

To interpret the OptiDON-Guard model, Figure 5 and Supplementary Figure S1 present the feature importance using SHAP values derived from the internal validation set and the external test set, respectively. The relative contributions of the 17 features to the model demonstrate parallel patterns, with the same six features emerging as the most significant across the datasets. This consistency highlights the reliability and robustness of these key features in driving the performance of the model.

Figure 5.

The significance of different features of OptiDON-Guard in the external test set. (a) The ranking of the importance of the 17 features. (b) The SHAP summary graph.

We further explored the influence of the six features on the DON prediction through the SHAP dependency plot. The results indicate that lower values of retinal thickness in the parafoveal nasal quadrant, average retinal thickness in the parafoveal region, and average mGCC thickness in the parafoveal region are associated with an increased prediction of DON (represented by negative SHAP values). Conversely, the higher values of retinal thickness in the parafoveal temporal quadrant and average mGCC thickness in the parafoveal temporal quadrant are associated with an increased prediction of DON (represented by positive SHAP values). Interestingly, we found that there was a U-shaped curve association between the average thickness of pRNFL and DON prediction (Fig. 6 and Supplementary Fig. S2).

Figure 6.

SHAP dependency plot for the features contributing to the OptiDON-Guard in the external test set. Each panel (a–f) shows a SHAP dependence plot for one feature. The x-axis represents the feature value input to the model. The y-axis represents the corresponding SHAP value. Each point corresponds to an individual observation. The gray histogram indicates the corresponding distribution of the shown feature values.

Discussion

In the early stages, DON typically manifests with subtle and nonspecific symptoms. Current diagnostic methods largely depend on the subjective interpretation of various test results by ophthalmologists, lacking both objectivity and standardization. This subjectivity frequently leads to the missed diagnosis of DON, resulting in delayed treatment for many patients and subsequently poorer prognoses. This study is the first in-depth exploration of using machine learning on OCTA features for DON identification, leading to the development of the OptiDON-Guard model. The model's innovation lies in its specially designed two-tiered stacking ensemble framework, which incorporates pseudo-labeling synthetic data augmentation and stacking-based hierarchical ensemble learning, leading to satisfactory and reliable model performance.

The identification of DON in TED eyes remains a significant clinical concern, and several studies have already explored the use of radiomics analysis to improve the diagnostic accuracy of DON. Ma et al.³⁴ found that the combination of muscle cone inner volume/bony orbit volume (%) and extraocular muscle volume/bony orbit volume (%) is a good indicator for the diagnosis of DON, with AUC of 0.862, with a sensitivity of 85.7% and a specificity of 76.1%. Zhang et al.³⁵ found that extraocular muscle volume measured according to the 3D reconstruction of magnetic resonance imaging scans had the highest AUC for the diagnosis of DON, at 0.850. A classification model utilizing muscle and fat volumes from orbit computed tomography images had an accuracy of 0.838.³⁶ However, the clinical adoption of these models remains limited due to challenges including insufficient validation of generalizability across diverse populations, restricted accessibility of advanced imaging modalities (computed tomography/ magnetic resonance imaging) in ophthalmology clinics, and suboptimal algorithmic stability in real-world clinical settings.

OCTA, as a noninvasive imaging modality, can be easily integrated into clinical practice, requiring only a few minutes per examination. It is already widely used in ophthalmology clinics to monitor subtle changes in retinal structure and microvasculature, offering valuable insights into the status of the retina and optic nerve. Zhang et al.³⁷ found a significant reduction in vascular density in the peripapillary area of DON eye. Wu et al. found the DRCP vascular densities of the DON eyes were decreased compared to the non-DON eyes.⁸ The value of GCC thickness combined with RCP vascular density in the diagnosis of DON is greater than that of a single index.⁸ Consistent with previous research, OptiDON-Guard, which uses multiple OCTA features, was extensively validated across independent internal and external datasets, showcasing its accuracy, reliability, and generalizability in identifying DON. The performance of OptiDON-Guard surpassed that of various classical machine learning models, including SVM, XGBoost, LightGBM, Random Forest, and CatBoost, highlighting its robustness. Additionally, we analyzed how features influence the model, facilitating the translation of OptiDON-Guard into clinical practice.

Machine learning models play a pivotal role in uncovering the determinants driving reliable predictions while simultaneously illuminating latent disease mechanisms or clinically relevant biomarkers. Interpretive analysis of OptiDON-Guard revealed that a low retinal thickness in the parafoveal nasal quadrant, reduced average retinal thickness in the parafoveal region, and decreased average mGCC thickness in the parafoveal region, along with a high retinal thickness in the parafoveal temporal quadrant and elevated average mGCC thickness in the parafoveal temporal quadrant, were significant predictors of DON. The GCC includes the inner plexiform layer, ganglion cell layer, and nerve fiber layer. Variations in GCC thickness can reflect the state of retinal ganglion cells and their axons, and previous studies have confirmed the significance.^10,38 It has been reported that DON patients presented lower overall GCC thickness compared with the non-DON patients through meta-anlysis.^9,37 Additionally, it has been found that cell atrophy and edema coexist in DON patients.⁸ The unique trajectory of nerve fibers in the macular region underlies their varying responses to mechanical compression. Nasal fibers, with a shorter and more direct course toward the optic nerve head, are more vulnerable to compression-induced damage, resulting in atrophy and thinning. Conversely, temporal fibers take a longer, curved path around the macula before entering the optic nerve, making them less sensitive to compressive forces. Under mechanical compression, nasal fibers sustain direct damage near the compression site, leading to thinning, whereas temporal fibers may exhibit transient thickening because of disruptions in axoplasmic transport. Similarly, our results show that in DON, the retinal and mGCC thickness in the parafoveal nasal quadrant increased, whereas it decreased in the parafoveal temporal quadrant. Overall, the retinal and mGCC thickness in the parafoveal region was reduced. Moreover, we observed a U-shaped relationship between the average thickness of pRNFL and DON prediction, which indicates that when the pRNFL thickness is beyond the normal range, the diagnostic probability of DON will increase. Our findings support the notion that both ganglion cell atrophy and edema coexist in patients with DON. In these patients, edema secondary to inflammation or vascular compression results in abnormal thickening, whereas ganglion cell atrophy leads to abnormal thinning, indicating different stages of DON progression. ML models not only achieved satisfactory performance in the identification of DON but also enabled the interpretation of critical features, offering clinical insights into the pathogenesis of DON.

This proof-of-concept study still has some limitations. First, a relatively small sample size for the training and validation set, and data from two medical centers located in one country. This sample size and geographic limitation may restrict the generalizability of the model to broader populations. To improve robustness and generalizability, future studies should include larger, more diverse datasets spanning multiple geographic regions, demographics, and clinical settings, thereby enhancing the model's reliability and clinical practice. Second, because the OCTA data used in this study were sourced from a single device, the model's ability to generalize across different devices is unknown. Third, although OptiDON-Guard effectively distinguishes between DON and non-DON cases in patients with TED, it does not incorporate a severity staging system, thereby limiting its ability to guide stage-specific therapeutic interventions. OptiDON-Guard warrants further validation through prospective, multicenter studies involving diverse populations with a full spectrum of DON severity. Additionally, incorporating OCTA data with other established clinical indicators to develop a multidimensional severity staging system will enhance clinical utility by providing a practical framework for guiding stage-specific therapy.

In conclusion, OptiDON-Guard represents a novel and validated model for the identification of DON, offering critical support for clinical diagnosis. By harnessing a stacking ensemble learning approach, the model achieves significant enhancements in both reliability and robustness. Furthermore, its interpretable design offers clinicians insights into prediction-driving factors, supporting evidence-based decisions.