AI-based prediction of best-corrected visual acuity in patients with multiple retinal diseases using multimodal medical imaging

Abstract

Background/aims

This study evaluated the performance of artificial intelligence (AI) algorithms in predicting best-corrected visual acuity (BCVA) for patients with multiple retinal diseases, using multimodal medical imaging including macular optical coherence tomography (OCT), optic disc OCT and fundus images. The goal was to enhance clinical BCVA evaluation efficiency and precision.

Methods

A retrospective study used data from 2545 patients (4028 eyes) for training, 896 (1006 eyes) for testing and 196 (200 eyes) for internal validation, with an external prospective dataset of 741 patients (1381 eyes). Single-modality analyses employed different backbone networks and feature fusion methods, while multimodal fusion combined modalities using average aggregation, concatenation/reduction and maximum feature selection. Predictive accuracy was measured by mean absolute error (MAE), root mean squared error (RMSE) and R² score.

Results

Macular OCT achieved better single-modality prediction than optic disc OCT, with MAE of 3.851 vs 4.977 and RMSE of 7.844 vs 10.026. Fundus images showed an MAE of 3.795 and RMSE of 7.954. Multimodal fusion significantly improved accuracy, with the best results using average aggregation, achieving an MAE of 2.865, RMSE of 6.229 and R² of 0.935. External validation yielded an MAE of 8.38 and RMSE of 10.62.

Conclusion

Multimodal fusion provided the most accurate BCVA predictions, demonstrating AI’s potential to improve clinical evaluation. However, challenges remain regarding disease diversity and applicability in resource-limited settings.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Artificial intelligence (AI)-based best-corrected visual acuity (BCVA) prediction has been explored using single imaging modalities (eg, macular optical coherence tomography (OCT) or fundus images) in homogeneous cohorts with isolated retinal diseases (eg, diabetic macular oedema); however, these approaches lack generalisability to diverse retinal pathologies and fail to integrate complementary information from multimodal imaging.

WHAT THIS STUDY ADDS

• This study demonstrates that multimodal fusion of macular OCT, optic disc OCT and fundus images achieves superior BCVA prediction accuracy in a heterogeneous cohort spanning six retinal diseases.

• It validates average aggregation as the optimal fusion strategy and highlights the feasibility of AI-driven BCVA assessment in multidisease clinical settings.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• The results support integrating multimodal AI systems into clinical workflows to enhance BCVA evaluation efficiency, particularly for complex cases.

• For resource-limited settings, a simplified two-modality approach offers practical utility.

• Future efforts should prioritise device-agnostic models and improved low-BCVA predictions through expanded datasets and tailored algorithms.

Introduction

Retinal diseases refer to a diverse group of conditions such as age-related macular degeneration (AMD), diabetic retinopathy (DR) and retinal detachment.¹,³ These diseases can lead to severe vision impairment and even blindness if left untreated, which poses a significant burden on individuals, healthcare systems and society as a whole.^{4 5} For example, AMD is the main cause of low vision and even blindness in the elderly population.⁶ The global number of patients with AMD is expected to reach 288 million by 2040.^{7 8} The prevalence of AMD in the population aged 70 years and above in China is about 20.2%.⁷

With the intensification of the ageing population, the number of patients with retinal diseases is continuously increasing.⁹ Early detection and effective management of these conditions can help prevent further vision loss and slow disease progression.¹⁰ This underscores the importance of regular eye examinations, awareness campaigns and access to appropriate healthcare services, including ophthalmologists and retinal specialists.¹¹

Best-corrected visual acuity (BCVA) is a measurement of the highest level of achievable vision after receiving optimal correction.¹² BCVA represents the severity and prognosis of retinal disease damage and is also the most concerning issue for patients and doctors.¹³,¹⁵ It is determined through subjective self-report results during eye tests using charts like Snellen or Early Treatment Diabetic Retinopathy Study (ETDRS), which display letters or symbols of varying sizes.¹⁶ However, measurement of BCVA has certain limitations and drawbacks. First, subjectivity plays a role as individuals must communicate their ability to see the smallest letters or symbols clearly, human errors and inconsistencies can also affect results. Additionally, BCVA testing requires a specific environment, such as a doctor’s office or clinic, necessitating extra time and effort on the part of the patient. Moreover, this test requires experienced doctors and instruments such as optometers, which makes BCVA measurement more difficult.

With the help of artificial intelligence (AI), especially deep learning technology, comprehensive progress has been made in medical tasks.¹⁷,²⁰ There are several studies attempting to predict BCVA through medical imaging,²¹,²⁴ including forecasting the BCVA of patients with diabetic macular oedema (DME) via fundus or optical coherence tomography (OCT) images.^{25 26} Additionally, the BCVA prediction of highly myopic eyes after cataract surgery using OCT images is plausible.²⁷ Previous studies have demonstrated that medical macular OCT or fundus images can forecast BCVA. However, these studies have primarily focused on patients with single disease, highlighting potential labels in the training process of medical imaging, such as DME or high myopia. Furthermore, all previous studies primarily focus on predicting BCVA using a single medical imaging modality. It is important to note that different imaging modalities can capture distinct patient imaging features. For example, macular oedema can be effectively detected by OCT, while it can be challenging to be identified through fundus images. On the other hand, fundus images can easily detect retinal bleeding and other lesions that may be difficult to observe through OCT imaging.

In order to achieve more universally applicable prediction for BCVA, we first established a dataset including patients with multiple retinal diseases to better align with ophthalmic clinical practice. Then, we tried to train models to predict BCVA using multimodal medical imaging.

Methods

Data collection

The data for model development were collected at Beijing Tongren Hospital from September 2021 to August 2023. Each patient underwent ophthalmic examinations including OCT scans (both macular-centred and optic disc-centred) and macular-centred fundus images. The fundus images were acquired using the 45-degree non-mydriasis fundus camera (Topcon, Japan). The OCT scans were performed using the Heidelberg Spectralis SD-OCT system (Heidelberg Engineering, Germany) with a scanning wavelength of 870 nm. For macular scans, a star-shaped radial scanning protocol with high-resolution line scans centred on the fovea was applied to comprehensively capture structural details across the macular region. For optic disc scans, a circular scanning pattern (peripapillary circle scan with 3.4 mm diameter) combined with radial line scans (12 lines centred on the optic disc) was used to ensure full coverage of the peripapillary area. The enhanced depth imaging mode was activated to optimise choroidal visualisation. All scans were reviewed for quality, and only images with a signal strength ≥25 dB, minimal motion artefacts and clear delineation of retinal/choroidal layers were included. All examinations were performed by trained operators. The BCVA was recorded using the decimal visual acuity system. To facilitate deep learning, we converted decimal visual acuity to ETDRS results using the following formula:

$$\mathrm{E}\mathrm{T}\mathrm{D}\mathrm{R}\mathrm{S}\mathrm{}\mathrm{l}\mathrm{e}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{s}=⌊85+50\times {\log }_{10}(DecimalVisualAcuity⌋$$ (1)

Please refer to the online supplemental table 1 for common changes in decimal vision chart and ETDRS chart.

Data preparation and preprocessing

To protect patient privacy, sensitive information such as patient names, birthdays and contact information were removed from the medical records. We only selected the patient’s BCVA indicators and diagnostic results. The original medical images were in various formats, including DICOM and TIFF. We transcoded all medical images to JPG format for modelling.

For OCT, multiple frames of medical images from a single scan were saved during the scanning process.²⁸ Factors that affect the patient’s BCVA, such as macular oedema or drusen, might exist on particular frames of OCT. In other words, randomly selecting a single image from OCT scans for BCVA prediction was inaccurate. In order to obtain more comprehensive data information from OCT scans, we designed a random sampling strategy. We uniformly sampled 12 frames of OCT raw data for the macula or optic disc. If the original scan frame count was <12, we randomly filled it to 12 images. If the original scanning frame count exceeded 12, we randomly covered the entire scanning data to the maximum extent from beginning to end and selected 12 images. We had set up seeds for random sampling to ensure that the same random results were obtained every time for the same OCT examination queue. Please refer to online supplemental methods 1 for specific code.

Deep learning architecture for predicting BCVA

Due to the presence of macular OCT, optic disc OCT and fundus image, we attempted to predict the patient’s BCVA using different or a combination of multiple medical imaging modalities. We chose ResNet50 as the backbone network, a convolutional neural network (CNN)-based visual feature extraction model that uses the design of residual networks and has shown excellent performance in previous work.²⁹ We performed transfer learning based on ImageNet pretrained ResNet50 and then performed BCVA prediction.

Since OCT-based BCVA prediction was based on 12 OCT images, each image was individually extracted through the visual extractor. Therefore, we must perform intramodality feature fusion for macular OCT or optic disc OCT. We designed three different feature fusion methods, namely average aggregation, concatenation and reduction and maximum feature selection. The primary methods of these three methods are shown in figure 1. We also endeavoured to fuse features from different modalities to predict BCVA in a multimodal fashion. Using ResNet50 as the backbone visual extractor, we established distinct visual extractors for various medical image modalities and conducted multimodal feature fusion. We experimented with two different schemes for feature fusion of multiple modalities: average aggregation, concatenation and reduction in figure 1b–d. Figure 1a shows the deep learning framework simultaneously using macular OCT, optic disc OCT and fundus image to predict BCVA, a three-modalities method in BCVA prediction. Please refer to figure 1 to locate the positions of intramodal feature fusion and multimodal feature fusion in the model framework. For more information on feature fusion schemes, please refer to online supplemental methods 2.

Figure 1. Deep learning framework simultaneously using macular OCT, optic disc OCT and fundus image to predict BCVA and three different feature fusion methods. (a) Multimodal prediction of BCVA, (b) average aggregation prediction of BCVA, (c) maximum feature selection prediction of BCVA, (d) concatenation and reduction prediction of BCVA. BCVA, best-corrected visual acuity; OCT, optical coherence tomography.

In the single-mode BCVA prediction experiments, we also tried to simulate OCT images as a three-dimensional (3D) image and use 3D ResNet50 to predict BCVA. 3D ResNet50 extends the ResNet50 network structure to three dimensions by incorporating a space dimension into the convolutional layers.³⁰ Compared with traditional two-dimensional convolutional neural networks, 3D ResNet50 excels at capturing spatiotemporal information in 3D objects and performs superior tasks such as 3D structure classification.

We have adopted mean squared error loss as the loss function for all models. We employed the stochastic gradient descent optimiser to train our neural network. The optimiser was configured with a learning rate of 0.0001 and a momentum factor of 0.9. All of these experiments were developed using PyTorch.³¹ The code will be made publicly available at: https://github.com/AI-thpremed/bcva/

External prospective validation

We have deployed the three-modalities model as a Web API for reasoning based on PyTorch and deployed it locally within the Liangxiang Hospital of Beijing Fangshan District. We used this AI-based BCVA inference model for patients who required BCVA examination. Between November 2023 and October 2024, we examined a total of 1381 eyes of 741 patients using OCT (Heidelberg, Germany) and fundus photography (Topcon, Japan). Among them, there were 202 cases of cataracts, 72 cases of pathological myopia (PM), 211 cases of AMD, 162 cases of retinal vein occlusion (RVO), 358 cases of DR and 450 cases of refractive error (RE). According to WHO standards (visual impairment: BCVA from 20/63 to 20/400), 0.36% (5/1381) eyes were classified as visual impairment in the dataset. The detailed information is shown in table 1.

Table 1. Basic characteristics of the datasets.

Characteristics	Development dataset			External prospective dataset
	Training	Test	Internal validation
No. of patients	2545	896	196	741
Female (%)	1634 (64.20%)	580 (64.73%)	134 (68.36%)	386 (52.09%)
No. of eyes	4028	1006	200	1381
No. of images
Optic disc OCT	45 876	11 626	2330	16 848
Macular OCT	61 845	15 161	2941	15 881
Fundus photographs	4028	1006	200	1381
Best-corrected visual acuity letter score
Score≤30	114	20	4	6
30<Score≤60	395	86	18	496
Score>60	3519	900	178	879
Ophthalmic diseases
Cataract	208	57	7	202
Pathological myopia	98	26	7	72
Age-related macular degeneration	307	57	11	162
Refractive error	2817	731	139	450
Retinal vein occlusion	181	45	6	72
Diabetes retinopathy	279	86	16	358

OCT, optical coherence tomography.

Evaluation measures

In order to assess the overall performance of the proposed methods to predict BCVA, we employed the following comprehensive set of metrics:

Mean absolute error (MAE): this metric quantifies the average magnitude of errors between predicted and actual values, providing a straightforward assessment of the model’s accuracy, which is defined as:

$$\mathrm{M}\mathrm{A}\mathrm{E}=\frac{1}{\mathrm{N}}{\sum }_{\mathrm{i}=1}^{\mathrm{N}}\left|{\tilde{\mathrm{y}}}_{\mathrm{i}}-{\mathrm{y}}_{\mathrm{i}}\right|$$ (2)

Mean squared error (MSE): this metric measures the average of the squares of the errors between predicted and actual values, emphasising larger errors due to the squared term, which is defined as:

$$\mathrm{M}\mathrm{S}\mathrm{E}=\frac{1}{\mathrm{N}}{\sum }_{\mathrm{i}=1}^{\mathrm{N}}{\left({\tilde{\mathrm{y}}}_{\mathrm{i}}-{\mathrm{y}}_{\mathrm{i}}\right)}^2$$ (3)

Root mean squared error (RMSE): RMSE represents the square root of the MSE, offering an easily interpretable metric that reflects the typical magnitude of the errors, which is defined as:

$$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=\sqrt{\frac{1}{\mathrm{N}}{\sum }_{\mathrm{i}=1}^{\mathrm{N}}{\left({\tilde{\mathrm{y}}}_{\mathrm{i}}-{\mathrm{y}}_{\mathrm{i}}\right)}^2}$$ (4)

R-squared (R²) score: this is an evaluation metric for regression models, used to measure the accuracy of model predictions. The R² value ranges from 0 to 1, with 1 indicating complete prediction accuracy and 0 indicating no predictive value. This is defined as:

$${\mathrm{R}}^2\mathrm{}\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=1-\frac{{\mathrm{S}\mathrm{S}}_{\mathrm{r}\mathrm{e}\mathrm{s}}}{{\mathrm{S}\mathrm{S}}_{\mathrm{t}\mathrm{o}\mathrm{t}}}$$ (5)

The sum of squares of residuals, also called the residual sum of squares:

$${\mathrm{S}\mathrm{S}}_{\mathrm{r}\mathrm{e}\mathrm{s}}={\sum }_{\mathrm{i}=1}^{\mathrm{N}}{\left({\tilde{\mathrm{y}}}_{\mathrm{i}}-{\mathrm{y}}_{\mathrm{i}}\right)}^2$$ (6)

The total sum of squares (proportional to the variance of the data):

$${\mathrm{S}\mathrm{S}}_{\mathrm{t}\mathrm{o}\mathrm{t}}={\sum }_{\mathrm{i}=1}^{\mathrm{N}}{\left(\stackrel{-}{\mathrm{y}}-{\mathrm{y}}_{\mathrm{i}}\right)}^2$$ (7)

In the above formula, ${\tilde{\mathrm{y}}}_{\mathrm{i}}$ represents the predicted BCVA, ${\mathrm{y}}_{\mathrm{i}}$ represents the ground true BCVA and $\stackrel{-}{\mathrm{y}}$ represents the average of the ground true BCVA for all test datasets. By using this diverse set of evaluation measures, we aimed to comprehensively capture and compare the predictive performance of our method against state-of-the-art approaches, ensuring a robust assessment of its efficacy in addressing the targeted prediction task.

Results

Characteristics of dataset

A total of 5234 eyes from 2761 patients were included in this study, and patients did not have duplicate medical records. Among the patients, 1777 were female (64.36%). Table 1 provides more detailed information for the dataset. The total number of macular OCT scans was 79 947, with an average of 15 scans per examination. The total number of optic disc OCT scans was 59 832, with an average of 11 scans per examination. The diagnosis for the patients included RE (3687 (70.44%)), AMD (375 (7.16%)), DR (381 (7.28%)), PM (131 (2.5%)), cataract (272 (5.2%)) and RVO (232 (4.43%)). We first randomly selected data from 200 eyes as the internal validation dataset, and then divided the remaining data into training dataset and test dataset in an 8:2 ratio.

Performance of models with different modality

As shown in table 2, in single modality BCVA prediction, the predictive ability of macular OCT and fundus image was relatively close. The best results for using macular OCT were MAE of 3.851 and RMSE of 7.844, while the predicted results for using fundus images were MAE of 3.795 and RMSE of 7.954. The results of optic disc OCT were poor among various intramodality feature fusion methods. The best results of optic disc OCT were obtained with an MAE of 4.977 and an RMSE of 10.026 based on average aggregation. The experiment results showed that the performance of average aggregation was significantly better than other methods in the intramodality fusion of multiframe OCT images. The average aggregation method might be able to stably extract the overall features of the macula or optic disc. Additionally, experiments demonstrated that simulating multiple OCT scans as a 3D structure and using 3D ResNet to predict BCVA was generally ineffective.

Table 2. Performance of the models in test dataset.

Modal	Intramodality feature fusion	Multimodal feature fusion	MSE	MAE	RMSE	R2 score
Macular OCT	AA	/	61.521	3.851	7.844	0.650
	CR	/	87.778	5.343	9.369	0.567
	MFS	/	72.154	4.413	8.494	0.589
Macular OCT with 3D ResNet50	/	/	242.012	5.715	15.557	0.082
Optic disc OCT	AA	/	100.522	4.977	10.026	0.520
	CR	/	123.921	6.212	11.132	0.530
	MFS	/	154.358	6.060	12.424	0.414
Optic disc OCT with 3D ResNet50	/	/	702.312	8.196	26.501	−1.665
Fundus	/	/	63.269	3.795	7.954	0.640
Macular OCT+optic disc OCT	AA	AA	63.697	4.355	7.981	0.696
	AA	CR	98.937	4.914	9.946	0.528
Macular OCT+fundus	AA	AA	57.287	3.860	7.569	0.727
	AA	CR	68.337	4.114	8.267	0.674
Optic disc OCT+fundus	AA	AA	84.472	4.614	8.862	0.553
	AA	CR	88.945	4.777	9.191	0.597
Macular OCT+optic disc	AA	AA	38.806	2.865	6.229	0.815
OCT+fundus	AA	CR	55.171	3.663	7.428	0.686

AA, average aggregation; CR, concatenation and reduction; 3D, three-dimensional; MAE, mean absolute error; MFS, maximum feature selection; MSE, mean squared error; OCT, optical coherence tomography; RMSE, root mean squared error.

Table 2 presented the performance of multimodal BCVA prediction using real clinical data. We used average aggregation for intramodality feature fusion of OCT in all multimodal experiments. The experimental results demonstrated that multimodal learning using macular OCT, optic disc OCT and fundus images led to improved prediction of BCVA. The best results were obtained using the average aggregation strategy for single-modal and multimodal feature fusion, achieving an average MAE of 2.865, RMSE of 6.229 and R² score of 0.935. Despite using different visual extractors for handling various medical image modalities, the experiments showed that employing features extracted from different modalities through average aggregation still yielded better results.

In summary, the joint learning of the three modalities resulted in significant performance improvements. Fusing macular OCT and fundus images as double inputs achieved an MAE of 3.86 and RMSE of 7.569. Although the MAE index did not show a significant improvement compared with using macular OCT or fundus image alone, there were noticeable improvements in RMSE, indicating an enhanced correlation between the predicted BCVA and the ground truth values.

Online supplemental figure 1 presents the scatter plots of the predicted and ground truth values of BCVA using different medical image modalities and combinations in the test dataset. All models adopted the optimal solution under this mode, which involved performing intramodality and multimodal feature fusion through average aggregation. When using a single medical image modality for prediction, although the MAE indicators obtained by different methods were all within five ETDRS letters, there was a significant difference in correlation. Using optic disc OCT for prediction resulted in the weakest correlation.

In online supplemental figure 1d–g, we presented the results of BCVA prediction using multimodal methods. Overall, the combination of two modal models did not significantly improve predictive performance compared with a single medical image modality(online supplemental figure 1a–c). In the comparison between the two modalities, results of using both macular OCT and fundus images and one modality method, the BCVA prediction results in terms of MAE showed no significant difference. However, the two-modalities approach exhibited superior RMSE and R² scores, resulting in a closer correlation between predicted and ground truth values in the scatter plot (as shown in online supplemental figure 1e). Online supplemental figure 1g presents the scatter plot of the three-modalities model in the test dataset, which is currently the best-performing approach. Out of 1006 samples in the test dataset, there were 11 cases where the difference between predicted values and actual values exceeded 30 letters, while there were 63 cases with differences ranging from 10 to 30 letters. Encouragingly, out of the 932 eye examinations, the variance between the predicted and actual values was within 10 letters. This implied that in actual clinical data, when multiple diseases coexist, using three modalities for prediction resulted in a 92.64% probability of the difference between the ETDRS index and actual values within 10 letters. Online supplemental figure 1h displays the prediction results of an internal additional validation dataset of 200 samples using three modalities.

Figure 2 shows the scatter plots of the predicted results and the actual values of different disease labels using three-modalities method in the test dataset. Please see the attachment for the indicators in the three-modalities prediction with different disease labels. Among them, RE performed the best in MAE and RMSE indicators, with values of 1.063 and 1.776, respectively. Although some RE patients might also have other diseases, such as AMD or RVO, the majority of RE patients have relatively good BCVA, indicating the three-modalities model had a strong ability to diagnose BCVA in medical imaging without apparent lesions. Among other disease label groups, PM patients had the best prediction results, with MAE and RMSE of 5.265 and 7.177, respectively.

Figure 2. Scatter plot results of BCVA prediction for patients of different disease tags in the test dataset using three modalities model. (a) Prediction results of AMD test dataset, (b) prediction results of DR test dataset, (c) prediction results of RVO test dataset, (d) prediction results of cataract test dataset, (e) prediction results of pathological myopia test dataset, (f) prediction results of RE performed test dataset. AMD, age-related macular degeneration; BCVA, best-corrected visual acuity; DR, diabetes retinopathy; PM, pathological myopia; RE, refractive error; RVO, retinal vein occlusion.

Visualisation analysis

In online supplemental figure 2, we present four sample cases along with their Gradient-weighted Class Activation Mapping (Grad-CAM) heatmaps for macular OCT, optic disc OCT and fundus images, which were involved in the multimodal decision-making process. These samples were labelled with different diseases: RVO for sample a, RE for sample b and AMD for samples c and d. The ground-truth BCVA values and the predicted BCVA values by our multimodal model for these samples are shown in online supplemental table 2.

For sample b, which had no retinal lesions, the fundus image showed minimal heatmap activation, while the heatmaps for macular OCT and optic disc OCT were concentrated in the key areas of the foveal centre and the entire optic disc region. In contrast, for the other disease-affected samples (a, c and d), the heatmaps were more focused on the pathological regions.

Performance analysis in internal validation

Table 3 presented the performance metrics obtained by the optimal fusion scheme for different modalities using 200 reserved internal validation data. The strategy of average aggregation was employed for the simultaneous input of the three modalities, resulting in the best performance. The achieved metrics included an MAE of 2.829 and an RMSE of 6.127.

Table 3. Performance of the models in internal validation dataset.

Modal	MSE	MAE	RMSE	R2 score
Macular OCT	58.600	3.791	7.655	0.720
Optic disc OCT	79.657	4.551	8.925	0.535
Fundus	76.899	4.098	8.769	0.633
Macular OCT+optic disc OCT	63.697	4.355	7.981	0.696
Macular OCT+fundus	57.288	3.850	7.568	0.726
Optic disc OCT+fundus	85.014	4.487	9.220	0.504
Macular OCT+optic disc OCT+fundus	37.536	2.829	6.127	0.835

MAE, mean absolute error; MSE, mean squared error; OCT, optical coherence tomography; RMSE, root mean squared error.

In online supplemental table 3, the analysis demonstrates consistent predictive accuracy across all disease subtypes, including AMD (MAE=9.493, RMSE=13.543), DR (MAE=10.111, RMSE=14.175), RVO (MAE=8.262, RMSE=11.599), cataracts (MAE=8.385, RMSE=11.908), PM (MAE=5.265, RMSE=7.177) and RE (MAE=1.063, RMSE=1.776). These metrics confirm the model’s robust generalisation capabilities across diverse retinal diseases.

Performance analysis of external prospective experiment

In online supplemental table 4, the analysis illustrates the distribution of BCVA across retinal disease subtypes in the external validation dataset. The results show consistent variability in BCVA outcomes, with cataracts (median=65 ETDRS letters, range=50–89), PM (median=58, range=30–89), AMD (median=65, range=19–89), RVO (median=65, range=20–89) and DR (median=65, range=20–94) demonstrating comparable median values. Notably, RE exhibited a higher median BCVA (85 ETDRS letters, range=58–89) compared with other ocular diseases.

Online supplemental figure 3 shows the scatter plot of predicted values and actual values in the external prospective experiment. Overall, the MAE of prospective data was 8.38, and RMSE was 10.62. There was a significant difference in this indicator compared with the test dataset, mainly due to two reasons. First, prospective data were mainly validated for various types of eye diseases, with 502 examinations having ETDRS <60, accounting for 36.35% of the total. Compared with the test dataset, there were a total of 106 cases in the test dataset with ETDRS <60, accounting for 11.83%. This significantly increased the difficulty of prospective reasoning. Second, in the prospective experiment, we also included some situations that were not presented in the training and testing datasets, including fundus tumours and postoperative examination cases.

Discussion

This study shows that AI can evaluate BCVA from different medical images in patients with various retinal diseases. We choose ResNet50 as the backbone network, a CNN-based visual feature extraction model that uses the design of residual networks. We use transfer learning based on ImageNet pretrained ResNet50 and then perform BCVA prediction. When using a single modality medical image for prediction, the prediction performance of macular OCT alone is close to that of fundus image, and it is better than using optic disc OCT alone. This suggests that predicting BCVA solely through optic disc OCT is challenging. One possible reason for this difficulty is that the optic disc OCT can only provide partial information and may not offer sufficient imaging features to assess BCVA accurately.

Although the MAE and RMSE indicators show little difference in the effectiveness of BCVA prediction using macular OCT or fundus images in a single medical image modality prediction, the scatter plot reveals significant differences in the two mechanisms for predicting BCVA. From a statistical perspective, out of the 1006 samples in the test dataset, 615 predicted BCVA values obtained through fundus prediction were higher than the ground truth, while only 298 predicted by macular OCT were higher than the ground truth. This suggests that the lesion features obtained by macular OCT through a visual extractor tend to underestimate BCVA. In contrast, the features obtained from fundus images tend to overestimate BCVA. Alternatively, based on the current training dataset, more features obtained from visual extractors in fundus images may be needed to evaluate BCVA accurately in patients with low BCVA. From this standpoint, fusing different medical image modalities for rapid AI automatic prediction of BCVA is a relatively reasonable solution, as it can acquire more comprehensive feature information.

If the multimodal prediction is used, using both macular OCT and fundus images can achieve more stable results while using macular OCT, optic disc OCT and fundus images simultaneously yields the best results. The fusion of features from multiple medical imaging modalities for predicting relevant medical information has greater potential for performance improvement, which is in line with physiological logic and has been widely confirmed in previous research.³¹,³³ The experiment of this study demonstrates that more patient medical imaging data will lead to more accurate and stable results, which give a useful suggestion to the importance of data collection in daily life.

Due to the involvement of multiple images in a single OCT examination, we devised a method to extract 12 frames from a single OCT examination and attained optimal results by integrating multiple OCT images through average aggregation. Regarding multimodal feature fusion, experiments have demonstrated that average aggregation is also a more robust and effective approach for integrating different medical image modalities, which may be an illustrated idea for further study.

Similar studies have focused on single modality predictions, with limited results.²²,²⁶³¹ While some studies have shown comparable performance between different imaging modalities, others have found differences in predictive accuracy. However, the method of multimodal prediction using macular OCT, optic disc OCT and fundus images has not been extensively explored in previous research, which has contributed to the limited diagnostic ability of multiple diseases in daily clinical practice.

The significance of this study lies in its demonstration of the potential of AI to evaluate BCVA using different medical images in patients with various retinal diseases. By comparing the predictive performance of single and multimodal medical image usage, we have identified the most effective combination of imaging modalities for accurate and stable results. This study will greatly reduce the time and human resource costs involved in the examination of BCVA and improve the accuracy and precision of the outcomes. With more accurate and easily obtainable results, the prediction and treatment for retinal diseases will become more efficient. In addition, dynamic monitoring of changes in the patients’ condition can be improved. This is of great significance for disease prognosis, especially for conditions such as DR, where BCVA is an important factor in guiding further treatment.³¹

However, the multimodal approach provides the highest accuracy, the requirement for multiple imaging modalities can be a barrier in clinics with limited resources. To address this, we suggest exploring the performance of the model using subsets of these imaging modalities. For example, preliminary results show that using macular OCT and fundus images together already provides a significant improvement over single-modality approaches, with an MAE of 3.86 and RMSE of 7.569. This suggests that a two-modal approach could be a viable alternative in settings where acquiring all three types of images is not feasible. Additionally, optimising the workflow and integrating these imaging modalities into existing clinical practices can help reduce the overall cost and time required for BCVA prediction.

Over 92.64% of the difference between the predicted BCVA and ground truth based on three-modalities prediction is within 10 letters in the ETDRS chart. Although the effect of the three-modalities method is better for all patient groups, including DR, AMD and RVO, the prediction ability of patients with low BCVA is still poor, especially for patients with BCVA <30 letters in the ETDRS chart. Two main reasons may cause it. First, there is a relatively small proportion (2.63%) in the overall dataset where the ground truth is <30. This results in a scarcity of training samples, and the model needs more samples for feature learning. Second, our patient dataset comprises individuals with multiple retinal diseases. Hence, the specific factors contributing to poor BCVA in patients may be diverse, presenting challenges for visual extractors to learn imaging lesion features.

This study demonstrates the capability of different modalities to predict BCVA in the presence of multiple disease-labelled patients simultaneously. Extensive experiments confirm that fusing multiple medical image modalities enhances BCVA prediction performance. In the external prospective experiment, the model performance was challenged due to the inclusion of more samples with ETDRS <60. Based on the above results, we believe that this suggests more room for improvement in the future. First, we will collect more data samples, especially increasing the data collection for patients in the low BCVA group. Second, we can consider incorporating more information into the model based on specific clinical conditions, such as patient disease labels or age. This method can potentially improve model performance and obtain more accurate BCVA estimates. Third, we can improve algorithms or clinical decision-making strategies, including establishing branching inference models based on patient-specific disease information, thereby enhancing the ability to evaluate and analyse specific lesions using different medical imaging modes in the low BCVA group.

While acknowledging the increased resource demands of multimodal imaging, our approach addresses clinical translatability through technical necessity and functional synergy. Multimodal fusion (OCT and fundus photography) synergistically captures complementary structural perspectives. OCT provides high-resolution cross-sectional details (eg, retinal thickness, choroidal morphology), while fundus photography offers overall vascular patterns and macular integrity. By integrating 12 OCT frames per scan (spanning macular-centred and optic disc-centred regions), our model holistically evaluates key determinants of visual function. This dual-modality integration surpasses single-modality performance (MAE reduced from 3.851 to 2.865 in internal validation) by encoding both global retinal architecture and localised pathological features. To mitigate resource burdens, we propose centralised imaging workflows (shared OCT/retinal cameras) and lightweight AI models (ResNet50-based architectures), with tiered implementation offering a cost-effective dual-modality alternative (MAE: 3.86). This framework reduces subjective BCVA testing variability and aligns with precision medicine goals through remote monitoring capabilities.

Certainly, this study has some limitations. First, there may be significant differences in the medical images generated by different medical devices, which is a challenge for the deep learning model. More importantly, in order to provide a more accurate assessment of BCVA, it is necessary to simultaneously use fundus photography equipment and OCT equipment, which undoubtedly increases the difficulty of AI technology in practical applications. Although the development of AI technology has provided tools to alleviate the differences in medical images from different sources, such as transfer learning or federated learning, the application of these methods requires further exploration in the future.

Conclusions

We conducted experiments to predict ETDRS-based BCVA using real clinical patient cohorts. We employed a single medical imaging modality or a combination of different medical imaging modalities in conjunction with AI-based information fusion. The results of this study indicate that the most accurate estimation of BCVA can be achieved through the multimodal fusion analysis of macular OCT, optic disc OCT and fundus images. These findings emphasise the potential of AI in facilitating efficient and cost-effective evaluation of BCVA in clinical settings.

Data availability statement

Data are available on reasonable request.