Modular multi-task deep learning framework for prediction of treatment initiation in neovascular age-related macular degeneration modular AI for NVAMD treatment initiation study-MANTIS

Abstract

Purpose

The purpose of this study is to develop a fully automated system for determining the optimal time to begin anti-VEGF treatment, based on retinal changes observed in optical coherence tomography (OCT) images and visual acuity (VA) scores.

Methods

This retrospective study included two cohorts: patients with intermediate dry AMD who did not progress to NVAMD and those who progressed and received anti-VEGF treatment. Subjects had ≥3 consecutive visits. Two tasks were defined: (I) develop a deep learning (DL) model to identify the need for anti-VEGF treatment, and (II) estimate outer retina thickness (ORT) from Bruch’s membrane (BM) to the outer plexiform layer (OPL) as an imaging biomarker. A convolutional neural network (CNN) extracted OCT features related to ORT, and two long short-term memory (LSTM) models were trained on separate datasets to address Tasks I and II. treatment.

Results

A total of 122 patients (207 eyes) were included: 49 in dataset 1 (69 eyes; mean [SD] age 84 [8]; 25 [51%] female) and 100 in dataset 2 (138 eyes; 82 [7]; 54 [54%] female). Task I performance was evaluated by the model’s alignment with clinical decisions on anti-VEGF initiation, and demonstrated AUROC of 0.73 (95% CI, 0.59–0.87), AUPRC (precision-recall curve) of 0.51 (95% CI, 0.35–0.66), an accuracy of 0.87 (95% CI, 0.76–0.97), a precision of 0.73 (95% CI, 0.59–0.87), a sensitivity of 0.69 (95% CI, 0.54–0.83), and a specificity of 0.92 (95% CI, 0.84–1.00). Task II achieved NMAE 0.12 (95% CI, 0.04–0.20).

Conclusions

This DL framework provides a fully automated approach for determining optimal treatment timing, supporting earlier intervention to preserve visual acuity in NVAMD patients.

Introduction

Patients with early stages of non-exudative age-related macular degeneration (AMD) have more than a 25% risk of progression to neovascular AMD (NVAMD) within 24 months^1,2, frequently without noticeable visual changes. Thus, irreversible vision loss may already have occurred by the time of presentation.^3–8 On the other hand, anti-vascular endothelial growth factor (anti-VEGF) treatment has been proven effective in treating NVAMD, and several high-impact studies emphasize the importance of early intervention to preserve baseline vision.^2,9

Prior work has introduced image biomarkers that can be used to assess AMD classification and progression, including identification of the risk factors.^6,10,11 However, to the best of our knowledge, no algorithms are currently available to successfully automate the decision-making process with high precision regarding the optimal timing to initiate early anti-VEGF intervention.

In this work, we introduce a modular, multi-task, and multi-modality deep learning (DL) framework that can automatically determine when to initiate anti-VEGF interventions for patients with early conversion to NVAMD. This framework automatically extracts the OCT image and VA features needed for the decision-making process and extrapolates predictive image biomarkers for future visits. This framework aimed to eliminate the need for human-in-the-loop efforts during decision-making, including feature selection, image annotation, and label annotation. This framework was validated against existing “future” data obtained from routine patient follow-up visits. Consequently, this framework can be executed in a fully automated manner, utilizing an end-to-end framework that is generalized for processing patient data collected from healthcare facilities on a larger scale and making decisions in real-time.

Methods

Retinal Imaging and Dataset Formulation

This retrospective study analyzed 2,047 OCT retinal images from 122 patients (207 eyes; mean [SD] age, 82 [8] years; 65 [53%] female) diagnosed with intermediate AMD. Intermediate AMD was defined clinically by the presence of medium-to-large drusen and/or retinal pigmentary changes in the macula, without any signs of neovascularization or advanced atrophy. Patients were identified through an electronic health record query for the specified time frame, and all patients meeting the criteria (diagnosis of intermediate AMD and at least 3 consecutive visits with OCT imaging) were included in the cohort. Collected data included visits to the Duke Eye Center, a tertiary academic ophthalmology center, between January 2006 and November 2020, and were analyzed retrospectively from January 2020 to January 2024. Images were captured using a Heidelberg Spectralis OCT device (Heidelberg Engineering, Inc.; A-scan rate 40/85 kHz, axial digital size 496 pixels, axial resolution 3.9 μm/pixel, lateral resolution 5.7 μm/pixel, scan angle up to 55°, with a 4× “Moving Average” oversampling to produce the averaged final image) by trained Duke Eye Center ophthalmic photographers. The study received Institutional Review Board (IRB) approval from Duke University Health System and complied with the principles outlined in the Declaration of Helsinki. The requirement for written informed consent was waived by the IRB due to the retrospective use of de-identified patient data.

Each patient was followed across multiple visits (mean [SD] number of visits, 11 [8]). Visual acuity was recorded at each visit when available, with missing entries imputed by carry-forward of the last recorded value. In total, 137 out of 2,047 visit records (6.7%) had missing VA measurements that were imputed using this last-observation-carried-forward method. Following Chen et al.¹², for eyes that progressed to NVAMD (60 out of 207 eyes, 29%), the date of the initial anti-VEGF injection as recorded in the medical records was considered the clinical action label (proxy for treatment start) for the conversion to NVAMD .^8,12,13 All diagnoses of intermediate AMD, NVAMD, and decisions to initiate treatment were made by board-certified retina specialists as part of standard clinical care; no additional grading of images was performed for this study. For each OCT volume scan, only the central scan (i.e., the 31st B-scan out of 60 in a macular volume) was extracted (in JPEG format, 496×768 pixels, then resized to 299×299 to match the CNN input).

Using only the central B-scan for analysis has the advantage of focusing on the foveal region, which is typically most relevant to disease progression^14,15, but it also imposes a limitation. This approach assumes the scan is properly centered on the fovea and that the central slice is representative of the pathology. If an OCT volume was not perfectly centered or if significant pathology lay outside the central slice, our model would not leverage that information. We chose to use the central B-scan to maintain consistent input and reduce data complexity (as is common practice in some prior imaging analyses), assuming that standard macular OCT scans are centered on the fovea, so the central slice captures critical changes. We acknowledge that this may not always hold; our future work will improve upon this by incorporating multiple B-scans or the full OCT volume to increase robustness to possible misalignment and to capture peripheral (off-center) lesions. The overall dataset was randomly and uniformly divided at the single-eye level into two portions (dataset 1 and dataset 2) to train the different modules of the framework described below.

Model Design and Outcomes Measures

We propose a modular, multi-task, and multi-modality deep learning system that can automatically extract features (biomarkers) from the OCT images needed to facilitate the two decision-making modules that predict when to start anti-VEGF injection for the patients that have acutely converted to NVAMD (Modules I and II) and extrapolate the biomarkers needed for evaluation of the latest visit (t=T) given data collected from prior visits (t=0 to T−1), such as prognosis (Module III). As illustrated in Figure 1, a CNN model first extracted global average pooling (GAP) features and ORT from OCT images, which were used by the two downstream long short-term memory (LSTM) models, along with the VA, to predict at which step the anti-VEGF needs to be initiated (Task I) and the ORT for the latest visit (Task II). The features include (1) VA score; (2) outer retina thickness (ORT – area between Bruch’s membrane (BM) and outer plexiform layer (OPL), inclusive of both) predicted by a CNN model, and numerically quantified in terms of nanometers (converted from the number of pixels between the two layers averaged over the horizontal axis, given the OCT imaging device’s specification). We focused on the ORT as it represents the primary site of pathological changes in AMD progression. (3) GAP features were extracted from the CNN model. Note that though the system was designed to extrapolate the need of anti-VEGF and ORT for a future visit given data collected from existing visits, the system was evaluated retrospectively against the latest visit (t=T) with which the outcomes were already observed and documented - to compute the evaluation metrics introduced in the Validation of the Models section below, as opposed to conducting randomized controlled trials. This work serves as a proof of concept for the model and system design, with retrospective analyses aimed at verifying its effectiveness and assessing the feasibility of potentially extending and deploying it in future randomized controlled trials.

Figure 1. Overview of the Proposed Modular Multi-Task and Multi-Modality Design Methodology.

Our modular multi-task model consisted of three modules, aimed at achieving two tasks: (Task I) determining the time to start early anti-VEGF intervention and (Task II) predicting outer retina thicknesses (ORTs) for upcoming visits. Module I extracted image features that are used as predictors for Modules II and III, which were reused across tasks. Then, the downstream Modules II and III were responsible for tackling the two tasks, respectively.

Module I: The CNN Model for GAP and ORT Extraction

Typical transfer learning^16,17 pipelines were leveraged to train an inception-v3 model¹⁸ toward mapping OCT images to ORT for all visits, using squared losses applied against ground-truth ORT annotated by an adjudicated consensus system where at least 2 medical trainees (among J.A., A.D.R., J.R., T.L., J.M, J.M.C., P.P, A.K.) first reached agreement, followed by a confirmation from an attending physician (M.H.). The global average pooling layer outputs were referred to as the GAP features. More details for this module can be found in eMethods.

Module II: The LSTM Model for the Prediction of Anti-VEGF Initiation

Each visit was assigned a positive or negative label indicating whether anti-VEGF administration was administered, and 41 of 138 eyes (30%) in dataset 2 were treated at some point during the follow-up period. This has introduced an imbalanced label challenge to the model.¹⁹ At each visit, t, the recurrent unit in the LSTM took as input the memory unit from the previous visit, t−1 along with the three features obtained from the current visit. Then, it predicted if anti-VEGF needs to be initiated at the current visit, as well as updated the internal memory unit for use in the next visit, t+1. Weighted binary cross-entropy loss was used due to label imbalance, which up-weighed the positive class by 10×. Note that the current recurrent implementation encodes longitudinal information using visit order only (i.e., discrete sequence steps) and does not explicitly encode elapsed time between visits. Because follow-up intervals in routine care are irregular, this design implicitly learns trajectories from observed sequences but does not normalize or parameterize time gaps. Figure 1 summarizes the model and data pipeline. eMethods provide specific modeling information, such as layer configurations and hyper-parameters.

Module III: Prediction of the Latest ORT

An additional LSTM model was trained, using squared losses, to predict the ORT of the latest visit (t=T), given the three features identified pertaining to all previous visits (t=0 to T−1). Note that it reused Module I without fine-tuning to obtain the GAP features and historical ORT. Moreover, this setup resembled the scenario as it would take place in clinics, i.e., using all historical data up to the last visit (t=T−1) to predict the ORT for the upcoming visit (t=T).

Modularity and module interfaces

Each module’s backbone and architecture can be updated as appropriate based on the size and quality of the available data. Module I may be swapped for alternative OCT encoders (e.g., different CNN backbones) if they yield the same per-visit feature dimensionality (or can be projected to it). For example, vision transformers (ViTs)²⁰ could be leveraged by fine-tuning or post-training an open-source model originally developed for general-purpose tasks, provided that sufficient training data are available. Modules II/III may be swapped with alternative sequence processing models, such as self-attention²¹, that accept the same sequential feature stream. Moreover, Modules II and III are functionally independent and may be removed depending on the application. For example, a deployment focused solely on predicting treatment initiation would not require the ORT forecasting component. Because both modules rely on Module I only for feature extraction, removing either Module II or Module III does not affect the performance of the remaining module.

Validation of the Models

The performance of Modules I and II in accomplishing Task I was evaluated through two types of assessments: sequential and single-visit assessments. The former evaluated how well the model’s suggested anti-VEGF initiation step aligns with the clinician’s decision for a single-visit setting. The latter evaluated how the model predictions align with the clinician’s decisions overall, across all visits in the validation set, treating each visit as an individual sample. It was designed to capture the model’s performance holistically, regardless of the temporal relations among visits. Multiple metrics were used to quantify the discrepancy between when the LSTM model suggested initiating the anti-VEGF treatment against the clinicians’ choices, i.e., accuracy, sensitivity, specificity, precision, area under the receiver operating curve (AUROC), area under the precision-recall curve (AUPRC), from the precision-recall curve evaluated at the visit from which the LSTM model predicted a positive outcome the first time in a sequence. Figure 1 (lower right of the region pertaining to Task I) illustrates the temporal step at which such metrics were evaluated within a sequence. Specifically, true positives were achieved when the model accurately determined the time step at which to initiate anti-VEGF, while true negatives were identified when there was no need to initiate anti-VEGF and the model correctly predicted this. False positives and false negatives were obtained when the model suggested initiating anti-VEGF earlier or later than the clinicians’ discretion, respectively. The performance of using Modules I and III to accomplish Task II was validated using the normalized mean absolute error (NMAE) between the predicted and ground-truth ORT at t=T.

To better validate the effectiveness of our model, we compared its performance against two simplified baseline models. These baseline models were chosen to represent clinically relevant alternative approaches. Model Baseline A uses only a rudimentary imaging feature and VA, whereas Baseline B relies solely on VA without incorporating OCT features.

Specifically, Baseline A excluded the CNN-derived GAP features from the LSTM input (using only the ORT and VA inputs), and Baseline B used only VA as input to the LSTM (essentially ablating Module I entirely). By comparing our framework to these baselines, we can evaluate the added predictive value of the OCT-based biomarkers and our multi-module approach in a context analogous to typical clinical decision strategies.

Results

This study included a total of 122 patients (207 eyes): 49 in dataset 1 (69 eyes; mean [SD] age, 84 [8]; 18 [51%] female), 100 in dataset 2 (138 eyes; mean [SD] age, 82 [7]; 54 [59%] female). Dataset 1 was used to train Module I for feature extrapolation, while Dataset 2 was used to train and validate the two LSTMs (i.e., Modules II and III), with a training/validation ratio of 1:1. The specific characteristics of both datasets are documented in supplemental Table 1.

To evaluate the performance pertaining to Task I, under the sequential assessment, our model achieved AUROC, 0.73 (95% CI²², 0.59-0.87); AUPRC, 0.51 (95% CI, 0.35-0.66); accuracy, 0.87 (95%CI²³, 0.76-0.97); precision, 0.73 (95%CI, 0.59-0.87); sensitivity, 0.69 (95%CI, 0.54-0.83); specificity, 0.92 (95%CI, 0.84-1.00). Under the single-visit assessment, our model achieved AUROC, 0.88 (95% CI, 0.78-0.98); AUPRC, 0.77 (95% CI, 0.64-0.90); accuracy, 0.89 (95%CI, 0.83-0.96); precision, 0.89 (95%CI, 0.82-0.95); sensitivity, 0.62 (95%CI, 0.52-0.72); specificity, 0.98 (95%CI, 0.94-1.00). Confusion matrices, ROC curves, and precision-recall curves are presented in Figure 2, with detailed results documented in Table 1.

Figure 2. Methodological Illustrations.

(a) Top and bottom figures assessed the model performance under the sequential and single-visit assessment, respectively. Rows of the confusion matrices represent ground truth labels and columns represent the model.

(b) The blue bars in each sub-plot represent the ground-truth ORTs overall visits pertaining to a single eye. Green bars represent the ORTs as extracted by the CNN model (from Module I) for historical visits Orange bars represent the future ORTs predicted by Module III.

(c) Module I’s predictions of ORT achieved Pearson’s =0.73, with Module II’s predictions of ORT achieving Pearson’s =0.73.

Table 1.

Performance comparison between our model and two baselines achieving Task I.

	AUROC	AUPRC	Accuracy	Precision	Sensitivity	Specificity
Sequential Assessment
Our Model (p<0.001)	0.73 (95% CI22, 0.59-0.87)	0.51 (95% CI, 0.35-0.66)	0.87 (95%CI23, 0.76-0.97)	0.73 (95%CI, 0.59-0.87)	0.69 (95%CI, 0.54-0.83)	0.92 (95%CI, 0.84-1.00)
Baseline A (p<0.001)	0.57 (95% CI, 0.47-0.68)	0.44 (95% CI, 0.34-0.55)	0.66 (95% CI, 0.56-0.76)	0.31 (95%CI, 0.21-0.41)	0.75 (95%CI, 0.66-0.84)	0.64 (95%CI, 0.54-0.74)
Baseline B (p<0.001)	0.49 (95% CI, 0.39-0.60)	0.27 (95% CI, 0.18-0.37)	0.31 (95% CI, 0.21-0.40)	0.21 (95% CI, 0.13-0.30)	0.80 (95% CI, 0.71-0.89)	0.17 (95% CI, 0.09-0.25)
Single-Visit Assessment
Our Model (p<0.001)	0.88 (95% CI, 0.78-0.98)	0.77 (95% CI, 0.64-0.90)	0.89 (95%CI, 0.83-0.96)	0.89 (95%CI, 0.82-0.95)	0.62 (95%CI, 0.52-0.72)	0.98 (95%CI, 0.94-1.00)
Baseline A (p<0.001)	0.59 (95% CI, 0.49-0.70)	0.27 (95% CI, 0.18-0.37)	0.69 (95% CI, 0.59-0.79)	0.34 (95% CI, 0.25-0.45)	0.75 (95% CI, 0.66-0.84)	0.92 (95% CI, 0.87-0.98)
Baseline B (p<0.001)	0.55 (95% CI, 0.45-0.66)	0.27 (95% CI, 0.17-0.36)	0.49 (95% CI, 0.38-0.59)	0.26 (95% CI, 0.17-0.35)	0.66 (95% CI, 0.56-0.76)	0.43 (95% CI, 0.33-0.54)

Table 1. Performance Comparison between Our Approach, Baseline A (VA was excluded from the inputs to LSTM), and Baseline B (both GAP features and thickness were excluded from the inputs to LSTM) on dataset 1. ^*

Compared with the baseline models, our method significantly outperformed other models across most metrics (Table 1). Notably, our model’s precision was substantially higher, indicating far fewer false-positive treatment recommendations. The baseline models achieved marginally higher sensitivity than our model, but this was at the cost of drastically reduced precision and specificity. Specifically, Baseline A (which utilized ORT and VA but no CNN features) and Baseline B (VA only) both tended to recommend treatment more liberally, resulting in a higher number of false positives. Baseline A generally performed better than Baseline B on all metrics, highlighting the importance of incorporating even a single imaging biomarker into the computations. However, our model outperformed Baseline A, underscoring the added value of the CNN-extracted image features (the GAP features from Module I) in making accurate predictions. The improvements from Baseline B to Baseline A and from Baseline A to our model suggest that both the imaging biomarkers and the multi-task modular design contribute meaningfully to predictive performance. As evidence, the gap in performance between our model and Baseline A was larger than that between Baseline A and Baseline B, indicating that the rich CNN-derived features (beyond just ORT) were the most significant contributors to precise decision-making on treatment timing. Moreover, the method attained phenomenal specificity in both sequential and single-visit assessments. The importance of specificity in NVAMD conversion prediction was emphasized by Yim et al.⁸, which underscored the model’s ability to avoid unnecessary interventions for patients.

Under Task II, the LSTM model achieved an NMAE of 11.99% (95% CI, 4.2%-19.71%) toward predicting the latest ORT thicknesses (t=T), over the validation dataset (the clinical ground truth). Moreover, in assessing the performance of Module I over the validation dataset, the CNN model achieved an NMAE of 9.34% (95% CI, 7.2%-11.47%) toward extracting ORT thicknesses from historical OCT images. The ground-truth, CNN-extracted, and LSTM-predicted thicknesses over the validation dataset are visualized in Figure 2(b). Such results also demonstrated that the knowledge captured by Module I can be well shared across the two tasks we considered. Variations in ground-truth outer retinal thickness (ORT) were observed across longitudinal visits. This aligns with previous findings showing that AMD progression is often associated with thinning of the outer retina, measured from the outer boundary of the OPL to the external limiting membrane, and thickening of the retinal pigment epithelium–drusen complex (RPEDC), defined as the region between the inner border of the RPE (including drusen) and the outer border of Bruch’s membrane.⁸ Moreover, the Pearson’s correlation coefficient (r) between the ground-truth and predicted ORTs made by Modules I and III were both 0.73, indicating strong correlations. The correlation plot can be found in Figure 2(c).

These results demonstrate that the knowledge captured by Module I can be successfully shared across the two different tasks. The CNN-extracted features (including ORT) are helpful not only for scoring the need for treatment but also for forecasting a quantitative biomarker of disease progression. The sequential LSTM model (Module II) further improved robustness in Task I by integrating chronological information; our model’s performance advantage over the baselines was even more pronounced when considering the sequential (longitudinal) assessment compared to a single-visit assessment. This suggests that Module II effectively utilizes longitudinal trends to make more reliable treatment decisions over time.

Discussion

Age-related Macular Degeneration (AMD), particularly advanced neovascular form (NVAMD), is the leading cause of irreversible central vision loss in individuals 55 and older in developed countries^6–8. Early treatment upon conversion to neovascular AMD is imperative for preventing significant and permanent vision loss^2,9. Given frequently, subtle subjective visual changes at the time of conversion^2,6,8, patients might remain undiagnosed for longer than the recommended two weeks, which is necessary to ensure more favorable visual outcomes.

In this work, we present a fully automated, modular, and multi-task DL system that can automatically extract and transfer the knowledge needed to make precise decisions on when to initiate anti-VEGF treatment in cases of acute conversion to NAMD. We also introduce an automated DL system that can be used to predict changes in outer retinal thickness (ORT) in patients affected by intermediate AMD and NVAMD.

Various prior works have emphasized the importance of early intervention with anti-VEGF to maximize the preservation of central vision when progressing to NVAMD.^{2,4,5,7–10,24,25} As a result, substantial existing research efforts have been devoted to identifying image biomarkers indicative of early conversion^6,10,11, or evaluating risk factors for dry-to-wet AMD progression pertaining to a single time step^5,25. However, to the best of our knowledge, there lacks a DL system capable of sequentially determining the necessity for early anti-VEGF interventions across the full horizon (from t=0to T) and that can fully automate the decision process by significantly reducing the dependency on manual featuring engineering and label annotations. Existing works that leveraged sequential decision-making models and that are most relevant to our work included Yim et al.⁸ and Banerjee et al.⁴, which only targeted predicting the conversion to NVAMD over patients who have already been diagnosed with NVAMD in one eye or quantifying the risk scores for conversion within 3 and 21 months, respectively. On the other hand, Rudas et al.⁷ used logistic regression to predict conversion within a 2-year window.

Our framework is advantageous over prior efforts by introducing the reusable modular components in the system, following the recent advancement in modular DL^26,27, to achieve multiple tasks in parallel. This has significantly reduced the manual efforts needed for feature extraction or label annotations with the new data. Specifically, Module I was shared across two different tasks (Tasks I and II) without task-specific fine-tuning. Moreover, as Module I leveraged transfer learning techniques^16,17 during training (see eMethods), it did not require massive training data to extract the features needed by the downstream models. As a result, this framework can be directly applied to larger-scale datasets and has the potential to process data from diverse healthcare facilities without requiring retraining of the feature extractor from scratch. We further inspected Module I’s outcome using Grad-CAM²⁸. Specifically, Figure 3. presents the corresponding Grad-CAM heatmaps for the CNN in Module I. The visualizations indicate that the network generally attends to multiple regions across retinal layers when estimating ORT thickness, while focusing on affected regions when pathological features are present.

Figure 3. Grad-CAM visualization for interpreting Module I predictions.

Grad-CAM heatmaps are shown for three representative categories of input OCT images

(a) Images with significant AMD pathology. In general, the CNN in Module I attends pathological regions when predicting ORT thicknesses.

(b) Images without significant AMD pathology. It can be observed that CNN in Module I, in general, attends to all layers to extrapolate the ORT thicknesses.

(c) Blurry or low-quality images. It can be observed that the CNN in Module I generally attends to multiple regions across retinal layers, suggesting that the model leverages distributed features to maintain prediction accuracy and robustness under challenging imaging conditions.

Once trained, the system operates in a fully automated fashion. Following fine-tuning, it can be integrated into the clinical workflow to assist clinicians in real time, determining if and when to initiate early anti-VEGF interventions as soon as new OCT images and VA data become available for a patient. Because our framework does not require manual feature input or labeling at the point of care, it can run seamlessly in the background and provide decision support. In a real-world clinic scenario, for example, when an intermediate AMD patient comes in for a follow-up visit, the latest OCT scan and VA measurement could be fed into the model. The model would then automatically analyze these inputs (in the context of the patient’s historical data) and output a recommendation on whether an anti-VEGF injection should be initiated at that visit. Such a tool could alert the treating ophthalmologist to signs of conversion that might be subtle or overlooked, thereby facilitating timely intervention. In a busy retina clinic, this decision-support system might help prioritize patients for treatment by flagging those eyes that the model predicts are at high risk of needing injections, ensuring that these patients receive prompt attention. Over time, deploying this model could standardize the decision-making process for early NVAMD treatment initiation, reducing variability between clinicians and potentially improving patient outcomes by avoiding delays in therapy. Importantly, the model’s recommendations would augment, not replace, clinician judgment, providing an objective second opinion that the physician can consider alongside clinical exam and imaging review. By integrating our framework into electronic health records or imaging systems, it could perform continuous surveillance on follow-up patients and new patients alike, making the transition to real-world use feasible. Much of the additional work is needed to sophisticate the system and allow for its real-world applications.

Previous research has studied the change in thickness of various retinal layers as affected by AMD and demonstrated the use of retinal layer thickness as a biomarker for analyzing the progression of AMD.^3,29,30 The most relevant work done by Thiele et al.³⁰ analyzed various prognostic features for linear models and found that thickening RPEDC posed a significant risk of progressing to non-exudative advanced AMD; however, no quantitative layer thickness features were identified as significant predictors for progression to NVAMD. In our work, we demonstrated that both the ORT values and the high-level CNN-derived features (GAP features) were important for making precision decisions on initiating anti-VEGF treatment, with the CNN features proving to be an even more significant contributor than ORT alone. This indicates that the deep learning–extracted features likely capture composite imaging patterns (beyond a single layer thickness) that are predictive of conversion to NVAMD, as opposed to the linear models considered by Thiele et al.³⁰, which assumed that linear relationships existed between the outcome and predictors.

Our ablation experiments and baseline comparisons clearly illustrate the contribution of each component. Removing the CNN image features (Baseline A) caused a notable drop in performance, and removing both image features and ORT (Baseline B) caused an even larger drop, confirming that the imaging data carries critical predictive signal that VA alone does not provide. Additionally, the fact that our model retained high precision means it minimized false positives (unnecessary treatment suggestions), a key consideration in a clinical tool to avoid overtreatment. The baselines, especially Baseline B, would have led to many unwarranted treatments or missed treatments, highlighting that our full model strikes a better balance in managing the risk of missing an intervention versus the risk of an unnecessary intervention. This balance is crucial in practice, since excessively frequent injections, besides possessing the risk of macular atrophy, also carry patient burden and cost, whereas missing an injection can lead to irreversible vision loss, secondary to bleeding and fibrosis.

Clinical implications of sensitivity-specificity trade-off

Clinically, false negatives correspond to potential delays in treatment initiation relative to the reference standard, which may carry a greater risk than false positives, depending on the magnitude of the delay.² However, prior work⁸ has also emphasized the importance of maintaining high specificity, as it helps ensure fair access to healthcare resources by reducing unnecessary treatments that may impose additional burdens on patients and clinics, including financial barriers such as insurance authorization requirements. As a result, the proposed framework could be useful in clinical practice to achieve an appropriate balance between sensitivity and specificity by adjusting the decision threshold of the model’s output.

Anticipated challenges in deployment

For clinical deployment, key challenges include the reliable ingestion of OCT and structured clinical data, managing missing or delayed documentation, and integrating model outputs into clinical workflows without contributing to alert fatigue. Because alert override rates can be high in clinical decision support, outputs should be delivered at the right time and in a usable format (e.g., within imaging review workflows rather than interruptive pop-ups) and should support clinical oversight rather than automate orders. Regulatory classification may depend on the intended use and the transparency of the rationale; we therefore frame the system as decision support and anticipate the need for prospective evaluation and governance consistent with relevant FDA clinical decision support guidance.

Future work

Emerging multimodal foundation models and agentic workflows may complement specialized OCT encoders and temporal models. For example, LLM-based agents could assist with retrieving longitudinal context and generating clinician-facing summaries, but they require careful validation of reliability, safety, and governance in clinical workflows. We therefore view LLM/agentic integration as a future systems-engineering layer around validated predictive models, not as a substitute for task-specific modeling in this early-stage study. Moreover, techniques such as ViT could be used to enhance Module I’s outcome once a sufficient amount of data is acquired.

Limitations

Firstly, our approach used only the central B-scan from each OCT volume, which may not capture all pathological features if they reside outside the central slice. This strategy relies on proper foveal centering of OCT scans; in cases where the fovea was not perfectly centered or pathology was eccentric, our model’s inputs might omit relevant information. While central scans are generally informative for macular disease and are standard in clinical evaluation^14,15, future extensions of our work should consider utilizing multiple B-scans or the entire volume to improve robustness and account for variability in scan acquisition. Secondly, the data used for training and analysis were collected during routine care visits, where visit frequency and intervals were inherently irregular. We did not attempt to normalize the time intervals between visits, allowing the LSTM to handle variable timesteps, as this approach mirrors real-world scenarios. However, this irregularity can introduce noise in temporal modeling. Thirdly, the amount of data available for this study was limited by the single-center, retrospective design and the number of patients available. Larger patient cohorts with extended follow-up periods would further improve the model’s performance and generalizability, enabling more robust statistical validation of the findings. Lastly, the outcome labels were intrinsically imbalanced: only about 30% of intermediate AMD eyes progressed to NVAMD and received treatment. We addressed this by focusing on AUPRC and precision, and our model’s high values for these metrics indicate it handled the imbalance well. Nonetheless, class imbalance remains a challenge, and additional techniques (e.g,. tailored loss functions or data augmentation) could be explored to further mitigate its effects further. Finally, the ground truth labels for conversion and non-conversion were based on retrospective clinical decisions and documentation, not on a protocolized grading of images at set intervals. While this real-world labeling improves the applicability of our model to actual practice, it also means that some labels might be noisy (e.g. if a conversion was detected slightly later than it began). We assume the clinical record is an accurate reflection of when treatment was truly needed; however, future studies may incorporate expert image re-grading to confirm the exact visit of conversion. Lastly, this was a single-center, retrospective study based on routine-care data collected at a tertiary academic retina service; accordingly, performance estimates primarily reflect this local operational environment and may not fully generalize to other centers, devices, or care patterns. Nevertheless, transfer learning and few-shot learning approaches could facilitate adaptation of the framework to new datasets, enabling efficient model generalization under different clinical configurations.

Conclusion

In this study, we introduced and evaluated a multi-modular, multi-model DL framework that can automatically determine the need for early anti-VEGF intervention in acute NVAMD conversion with high precision, while simultaneously accomplishing multiple related tasks by reutilizing components of the model. Our framework achieved high accuracy, particularly in terms of AUPRC and precision, demonstrating its capability to manage imbalanced outcomes and minimize false positives in positive predictions. Moreover, the model significantly outperformed baseline approaches in terms of sensitivity, effectively reducing the risk of missing necessary anti-VEGF treatments for patients who require them. The modular design showed great potential for reducing manual effort in both training and decision-making, as shared components can be applied across different tasks without additional annotation. Our ablation studies justified the necessity of the modular components and corroborated the suitability of our model for adaptation to larger-scale healthcare data. Furthermore, our analysis identified the CNN-derived GAP features as an important factor for predicting the need for early intervention.

We believe our framework can facilitate automated decision-making regarding if and when to employ anti-VEGF therapy within a short decision window (early conversion to NVAMD), thus allowing patients to receive timely treatment and potentially preventing further irreversible vision loss. The model could also serve as a valuable educational tool for training medical students and residents in making treatment-initiation decisions, for example by integrating it into simulation environments to provide hands-on decision-making experience before managing real patients.

Importantly, this work represents a proof-of-concept demonstration of an AI-driven decision support system for NVAMD. The clinical utility and impact on patient outcomes have yet to be established. Further prospective studies and real-world trials will be necessary to validate the model’s performance in practice and to determine how best to integrate such a tool into ophthalmology clinics. As with any assistive technology, our DL framework is intended to augment, not replace, clinician judgment. With additional refinement and validation, this approach could become a component of the clinical workflow that helps ensure early detection of conversion to NVAMD and prompt treatment, ultimately aiming to improve patient visual outcomes.

Synopsis.

A modular deep learning framework utilizing longitudinal OCT and visual acuity data predicts anti-VEGF treatment initiation in neovascular AMD, integrating imaging biomarkers to enhance decision-making and support earlier intervention for improved visual outcomes.

Transitional Relevance.

An accurate DL framework that can help with identifying the best time to initiate therapy for NVAMD.

Key Messages.

What is already known on this topic

Timing of anti-VEGF initiation in neovascular AMD is critical yet remains variable and largely dependent on clinician judgment.

What this study adds

This study presents a fully automated deep learning framework that uniquely integrates longitudinal OCT and visual acuity data to predict treatment initiation and extract clinically relevant imaging biomarkers.

How this study might affect research, practice or policy

These findings support the integration of AI-driven decision tools to standardize treatment timing, enable earlier intervention, and advance personalized management strategies for neovascular AMD.