An ultrasound foundation model for the stratification of vision impairment and eye cancer risk

Abstract

The prevalence of vision disorders and eye cancer risk, driven by global population aging, necessitates precise ophthalmological disease screening, yet standardized and robust approaches for risk stratification remain absent. Here, we present SonoEye, a vision-language ultrasound foundation model for eye disease risk stratification, pre-trained on 215,356 image-text pairs from 70,452 patients using contrastive learning and fine-tuned with consensus-confirmed patient-level labels. The model achieves 98.3% screening sensitivity and a mean accuracy of 96.3% in differentiating 18 diseases. We established the Eye Reporting and Data System (Eye-RADS), a 4-tier risk stratification framework (normal, low-vision risk, high-vision risk, and tumor risk), demonstrating excellent Cohen’s kappa agreement across internal (0.808) and external (0.677-0.685) test cohorts. Notably, incorporating age significantly improved Eye-RADS performance in elderly but not younger populations, indicating age-related features in eye care. Through image-text alignment, SonoEye generates structured clinical reports with interpretable attention-based visualizations, advancing automated vision risk stratification, particularly for aging populations in resource-limited settings.

Introduction

The eyes serve as vital sensory organs and uniquely accessible windows into systemic health¹. Owing to the non-invasive, inexpensive and scalable nature of clinical imaging techniques, eye-as-window applications such as detecting diabetes², cardiovascular diseases³, brain diseases⁴ and chronic kidney diseases⁵ have garnered increasing attention in recent years. However, dedicated screening and diagnosis of ocular diseases themselves has not kept pace. The convergence of aging populations and an expanding spectrum of vision disorders creates an urgent need for precise, fine-grained risk stratification tools. Delivering such capabilities is crucial for prevention of sight- and life-threatening complications, particularly in settings with limited access to specialized ophthalmic expertise.

Optical imaging techniques, such as fundus photography and optical coherence tomography (OCT), are standard clinical tools for ocular health assessment. However, optical approaches are inherently constrained by their requirement for media transparency, which is often compromised in aging populations due to conditions such as cataracts or vitreous opacities^6,7. By contrast, ocular ultrasound offers superior tissue penetration and broad accessibility, enabling holistic visualization of ocular structures from the vitreous to the retina-choroid complex and intraorbital anatomy⁸. Most importantly, ultrasound remains indispensable for the evaluation of posterior segment vision disorders⁹ and life-threatening malignant tumors^10–12.

Ultrasound imaging are typically associated with low contrast, operator-dependent scanning, and subjective interpretation, all of which contribute to substantial intra- and inter-observer variability¹³. With the continuous development of artificial intelligence (AI), especially deep learning-based models^14–20, AI-based approaches may be uniquely poised to help with this challenge. Despite early efforts exploring AI’s potential in ophthalmic ultrasound^21–27, its real-world clinical applicability remains unsolved by three key challenges. First, current AI studies have been limited by small-scale labeled training data, resulting in poor generalizability across different regions, manufacturers, and populations^21–23. Second, previous AI studies^24,25 have relied exclusively on image-level interpretation, which contradicts the dynamic nature of ultrasound interpretation, resulting in unfair and biased evaluation of heterogeneous characteristics. Third, existing AI models predominantly focused on simple binary, disease-specific tasks rather than comprehensive, multi-class differential diagnosis^26,27. This limitation is particularly critical in ophthalmic imaging, where many conditions present with overlapping imaging features and require explicit differentiation among multiple competing diagnoses (intraocular tumors need to differentiate neoplastic lesions from a range of non-neoplastic conditions²⁸).

In this study, we introduce SonoEye which mirrors the clinical workflow: (1) screening for eye abnormalities from asymptomatic population, followed by (2) multi-class differential diagnosis across 18 common ophthalmic disease entities. The most important contribution of SonoEye is to naturally aggregate multiple views/images and contextual information for patient-level reasoning rather than biased image-level interpretation. Inspired by well-adopted standardized Reporting and Data Systems (RADS) of the American College of Radiology, we propose Eye-RADS, a novel 4-tier (normal, low-vision risk, high-vision risk, and tumor risk) risk stratification taxonomy of ocular conditions for the first time. In multicenter test cohorts, SonoEye demonstrated robust performance in both screening and diagnosis scenarios, with excellent inter-rater agreements in eye care. To further enhance clinical utility, reader studies, gradient-based visualization, and template-based reporting were performed, making it a decision support tool for real-world practice.

Results

Figure 1 shows the overall study design of SonoEye for identifying a broad spectrum of ocular diseases and supporting automated eye risk stratification. Specifically, SonoEye leveraged contrastive language-image pre-training (CLIP) paradigm²⁹, which has recently demonstrated strong performance in medical imaging interpretation³⁰. It consists of a Swin Transformer v2-based image encoder³¹ and a ChineseCLIP-based text encoder³² (Supplementary Fig. 1a). Before training the vision-language foundation model, we designed a domain-specific ophthalmic tokenizer (EyeReportTokenizer) to extract and structure key findings and anatomical references from free-text reports, thereby maximizing clinical coverage while reducing redundancy. The pre-training of SonoEye was developed on 215,356 unlabeled image-text pairs from 70,452 patients, resulting in robust in-context representations between multi-perspective disease images and text descriptions.

Fig. 1. The study design of the SonoEye.

a A customized text tokenizer is first applied to raw ultrasound reports for generating image-text pairs, followed by pre-training of the foundation model through contrastive learning. b Each test image computes similarity scores against both image prototypes and knowledge base texts, which is then fused to generate the confidence score. For patient-level prediction, an attention-based multiple instance learning is employed to aggregate the features from multiple images. Overall, SonoEye is performed on (1) screening and diagnosis, (2) Eye-RADS risk stratification, (3) clinical reports and recommendations. c SonoEye is evaluated at image-level, patient-level, and reader studies in multicenter cohorts.

To enable differential diagnosis, we performed a clinician-in-the-loop few-shot learning adaptation in which experienced ophthalmologists first curated prototype candidates (that is labeled ultrasound image) for each consensus-confirmed disease category. These prototype candidates were then clustered to three representative prototypes for each category via K-means clustering, followed by the combination of optimized text prompts in a hybrid inference scheme for category-aware predictions. In other words, similarity is computed between each input ultrasound image and both image prototypes (19 categories from 18 diseases and 1 normal condition, 3 representative prototypes per category) and 19 text feature embeddings, as shown in Fig. 1b.

To evaluate the performance of SonoEye at image-level, we tested on an internal test cohort of Site A (n = 2,342 images) as shown in Fig. 2. Across 19 categories, SonoEye demonstrated robust feature extraction capabilities, achieving a mean Top-3 accuracy of 94.7% (95% confidence interval (CI): 93.6–95.5%). Performance was particularly high for three ultrasound-important conditions: retinal detachment, retinal tear, and intraocular tumors^9,33, with a Top-3 accuracy greater than 97.8% (95% CI: 96.5–99.9%), as shown in Supplementary Table 1. Crucially, given that characterizing malignancy is a primary indication for ocular ultrasound, we specifically investigated the model’s fine-grained capability in identifying tumor subtypes. As shown in Fig. 2d, SonoEye achieved excellent results for retinoblastoma, exhibiting 100% (95% CI: 10–100%) sensitivity and 94.7% (95% CI: 90.9–98.0%) specificity. Although rare benign tumors showed lower sensitivity due to data scarcity, the model maintained clinically relevant performance for the critical distinction between benign and malignant lesions, suggesting that the image encoder captures subtle, discriminative features essential for cancer-related decision support.

Fig. 2. SonoEye evaluation at image-level.

a, b, accuracy among 19 categories in the internal test cohort of Site A when considering (a) top-1 and (b) top-3. c, d The performance of SonoEye in tumor subtypes (c) benign and (d) malignant, the metric PPV refers to Positive Predictive Value and NPV refers to Negative Predictive Value.

The dynamic scanning characteristic of ultrasound naturally requires aggregating information from multiple views for comprehensive disease evaluation, which is different from bias-involved image-level interpretation. To bridge this gap and accelerate its clinical potential, we additionally designed and implemented an attention-based multiple instance learning (AttnMIL) fusion module³⁴ (Supplementary Fig. 1b). This module mimics the physician’s reasoning process by aggregating potential findings from variable numbers of ultrasound images to generate a unified, patient-level prediction. It should be noted that the patient-level labels were derived from the actual multi-disease combinations confirmed in the electronic medical records, rather than single image-level labels.

Screening and diagnosis of ocular diseases

We first evaluated the screening performance of SonoEye in a temporally independent screening cohort with consecutive patient enrollment from Site A. In this cohort, a participant was classified as abnormal if any ultrasound image from the examination was predicted as abnormal (Supplementary Fig. 2). SonoEye achieved strong discrimination between normal and abnormal cases, with an area under the receiver operating characteristic curve (AUC) of 98.9% (95% CI: 98.4%-99.3%), a sensitivity of 98.3% (95% CI: 97.4%-99.1%), and a specificity of 91.9% (95% CI: 89.8%-93.8%). The corresponding confusion matrix for the screening cohort is shown in Supplementary Fig. 3. These results indicate that the screening module can effectively identify participants requiring further review while minimizing missed abnormal cases in a real-world screening setting.

We next evaluated the full patient-level pipeline in an independent test cohort comprising 941 participants from Site A. In this setting, SonoEye first performed image-level screening and then aggregated information across multiple ultrasound views using the AttnMIL fusion module to generate final patient-level predictions, enabling differential diagnosis among 19 categories. SonoEye exhibited strong diagnostic accuracy across multiple clinically important ophthalmic diseases at the single-disease level (Fig. 3b). Specifically, an accuracy of at least 93.8% (95% CI: 92.2–95.2%) was achieved for three clinically critical conditions - retinal detachment, retinal tear, and intraocular tumors - while the remaining diseases also showed robust performance, all exceeding 81.8% (95% CI: 79.4%-84.4%), as shown in Supplementary Table 2.

Fig. 3. SonoEye evaluation at patient-level.

a The ROC curve of screening outcomes at Site A. b Confusion matrix of differential diagnosis on 19 categories at Site A. c The performance of our proposed Eye-RADS risk stratification.

Overall, SonoEye obtained the mean accuracy of 96.3% (95% CI: 95.3–97.2%), sensitivity of 63.8% (95% CI: 46.3–80.0%), and specificity of 98.5% (95% CI: 97.9–99.0%) for all 19 eye conditions at Site A. The generalizability of the model was further evaluated on two external test cohorts, with the mean accuracy of 94.1% (95% CI: 92.7–95.3%) at Site B and 96.2% (95% CI: 95.6%-96.8%) at Site C, indicating preserved diagnostic capability across diverse institutional settings. We also examined representative failure cases to characterize the main sources of diagnostic error (Supplementary Fig. 4). To further characterize the distributional differences across sites, we visualized feature distributions from the three sites using t-distributed Stochastic Neighbor Embedding (t-SNE) and Uniform Manifold Approximation and Projection (UMAP), which showed partially distinct clustering patterns across centers (Supplementary Fig. 5).

To assess whether the two-stage design introduces clinically relevant diagnostic gaps, we analyzed the agreement between the screening module (pretrained CLIP) and the downstream diagnostic module (CLIP+AttnMIL) in the screening cohort. The overall inconsistency rate was 4.58% (95% CI: 3.59–5.64%). The clinically most relevant inconsistency (cases classified as normal by the screening module across all images but assigned at least one disease label by CLIP+AttnMIL) occurred in 3.68% (95% CI: 2.35–5.15%) of participants. These results support the rationale for the two-stage design: the screening module operates as a high-sensitivity priority while the diagnostic stage leverages aggregated multi-image evidence for fine-grained differentiation.

Eye-RADS based risk stratification

To better align with clinical decision-making and management of risk stratification, we aggregated all 19 eye conditions into a communicable taxonomy and proposed a 4-tier risk stratification system, including normal (1), low-vision risk (2), high-vision risk (3), and tumor risk (4), termed as Eye-RADS.

Figure 3c shows the predicted Eye-RADS of all 941 participants whose reference standards are confirmed by ophthalmologists’ consensus. Based on confusion matrix analysis (Fig. 4a), Eye-RADS obtained a linear weighted Cohen’s kappa of 0.808 (95% CI: 0.778-0.837) for Site A, reflecting almost perfect agreement. The generalizability of Eye-RADS is further validated on two external cohorts, with Cohen’s kappa of 0.685 (95% CI: 0.616–0.747) at Site B and 0.677 (95% CI: 0.646–0.707) at Site C, showing substantial inter-rater agreement. These results demonstrate that SonoEye does not merely classify images but effectively stratifies patients into actionable risk groups, supporting its practical utility in complex diagnostic scenarios where multiple diseases often co-exist. Because Eye-RADS is derived from a predefined rule-based mapping rather than directly predicted as a probabilistic output, we evaluated model calibration at the disease prediction level. Calibration curves for all 18 disease categories demonstrate that the predicted probabilities are reasonably aligned with observed outcomes across most conditions (Supplementary Fig. 6).

Fig. 4. Confusion matrix analysis of Eye-RADS at patient-level.

a With the consideration of age. b Without considering age.

Age-adaptive Eye-RADS performance

Ocular anatomy profiles are intrinsically dynamic across the lifespan, yet previous AI models often treated patients as a uniform demographic. To account for this, we incorporated patient age into Eye-RADS risk stratification through an age-adaptive thresholding mechanism (see Methods), in which age was used to calibrate patient-level decision boundaries rather than to alter image feature extraction. Such a stratification is crucial because the same ultrasound image can imply vastly different risks depending on age. For instance, a vitreous echo, while potentially indicating disease in a child, is often clinically insignificant in elderly patients due to normal age-related physiological changes³⁵.

We divided patients into four distinct groups: adolescents (< 18 years), young adults (18–39 years), middle-aged (40–59 years), and elderly (≥ 60 years). As displayed in Fig. 4, Cohen’s kappa improved from 0.791 to 0.808 (Site A), 0.647 to 0.685 (Site B), and 0.609 to 0.677 (Site C), all of which demonstrate that age is an important factor for improving Eye-RADS risk stratification.

Beyond considering all age groups together, we also performed the analysis on four sub-age groups. We observed that the accuracy improvements were statistically significant (P < 0.05) for all cohorts in the elderly population (Supplementary Table 3). It indicates that the elderly population benefits most from age-specific adjustments, as their physiological aging often leads to diagnostic ambiguity. These findings suggest that incorporating age-specific priors improves risk stratification in age groups where physiological changes increase diagnostic ambiguity.

Retrieval-augmented report and interpretability

Owing to the alignment of image and text representations, SonoEye is naturally able to generate retrieval-augmented structured reports that integrate multi-label predictions with evidence-based clinical knowledge. The generated report follows a standardized format comprising several key sections: (1) observation section: presenting conditions ranked by clinical severity using predefined priority rules, (2) classification section: providing risk stratification and urgency recommendations, and (3) recommendation section: offering condition-specific clinical insights and management guidance derived from our curated knowledge base (Supplementary Fig. 7).

Rather than relying on generative large language models (LLMs) that may produce inconsistent or inaccurate clinical content³⁶, our approach utilizes a template-filling methodology where each diagnosed condition is mapped to pre-validated clinical descriptions and evidence-based management recommendations. This design ensures that all clinical content in the reports is derived from established medical knowledge and expert consensus, significantly enhancing reliability for clinical decision-making.

In terms of interpretability, the incorporation of CLIP’s multimodal capabilities allows the model to leverage both visual features and textual descriptions, creating a more comprehensive understanding of ophthalmic conditions. For cases that are difficult to diagnose, such evaluations often require extensive clinical experience and can be highly variable across different practitioners. By providing quantitative attention-based assessments, our model can mark the areas it focuses on through heatmaps (Fig. 5), alerting doctors to pay special attention to those regions.

Fig. 5. Examples of SonoEye outcomes.

For each test case associated with single image or multiple images, we provided image-level heatmaps, and patient-level Eye-RADS. These cases were selected from the reader study which were missed or partially missed by readers.

Clinical outcomes of solo readers versus AI-assisted readers

We conducted a comprehensive reader study to evaluate AI performance and its impact on clinical decision-making. Six readers participated, stratified into three experience groups: two non-specialists, who were generalist ophthalmologists without dedicated subspecialty training in ocular ultrasound, but with general clinical familiarity with ocular imaging; two junior specialists (2-3 years of experience); and two senior specialists (10-15 years of experience).

At the image-level, SonoEye’s performance was generally comparable to that of experienced human readers across the disease spectrum, with average accuracies ranging from 61.7% to 80.3% for individual readers (Fig. 6a). When AI assistance was provided, diagnostic accuracy improved across most readers and disease classes (Fig. 6b). To formally quantify this effect, we performed mixed-effects logistic regression with AI assistance as a fixed effect and random intercepts for both readers and cases. Across all six readers, AI assistance was associated with a statistically significant improvement in diagnostic accuracy (odds ratio (OR) = 1.76, 95% CI: 1.51–2.05, P < 0.001). Subgroup analysis revealed that the benefit was most pronounced for non-specialist readers (OR = 3.42, 95% CI: 2.63–4.45, P < 0.001) and junior readers (OR = 1.98, 95% CI: 1.38-2.86, P < 0.001), while the improvement for senior readers did not reach statistical significance (OR = 1.12, 95% CI: 0.83–1.52, P = 0.45).

Fig. 6. Reader study evaluation.

a Image-level differential diagnosis performance between readers and SonoEye. b Image-level performance changes of readers with SonoEye assistance, each point represents the mean accuracy of readers within each experience group (non-specialist, junior, and senior). c The Eye-RADS risk stratification comparison between readers and SonoEye at patient-level. d Patient-level performance changes of readers with SonoEye assistance, each point represents the mean accuracy of readers within each experience group.

At the patient-level, SonoEye performed comparably or better than individual readers across disease classifications, with its predicted risk distributions highly consistent with reference labels (Fig. 6c). AI assistance again yielded significant overall improvement (OR = 1.82, 95% CI: 1.49–2.22, P < 0.001). The effect was largest for non-specialist readers (OR = 5.86, 95% CI: 3.85–8.92, P < 0.001), followed by junior readers (OR = 1.55, 95% CI: 1.05–2.28, P = 0.026), while the effect for senior readers was not statistically significant (OR = 1.40, 95% CI: 0.90–2.16, P = 0.13), as shown in Fig. 6d. Detailed individual performance changes with SonoEye assistance for each of the six readers are shown in Supplementary Fig. 8. These findings indicate that AI assistance most benefits readers with less specialized experience in ocular ultrasound, consistent with the intended deployment scenario of SonoEye in settings with limited access to subspecialty expertise.

Discussion

Despite ophthalmic ultrasound is often regarded as complementary to frontline optical imaging, it is becoming increasingly important in the setting of aging population globally. Its low cost, wide accessibility, and unique ability in detecting sight- and life-threatening ocular tumors, make it particularly valuable when optical imaging is limited by media opacity. However, the clinical potential of ophthalmic ultrasound has long been constrained by challenges of low image contrast and operator dependency. Here, we present SonoEye, a vision-language foundation model to advance ultrasound-based screening and differential diagnosis.

A key bottleneck in developing medical AI models is the scarcity of annotated data. Previous ophthalmic ultrasound AI models were typically trained on only hundreds to a few thousands manually labeled images via supervised learning²⁵, resulting in the well-known distribution shift issue when tested on a different setting. To overcome this limitation, we combined contrastive learning on large-scale unlabeled data (215,356 image-text pairs) with weakly supervised adaptation using a small labeled data, thereby enabling training at much greater scale with better generalizability. Owing to the pre-trained backbone, SonoEye could support multiple clinical tasks simultaneously, without the need to build and maintain separate specialist models for each task (that is 18 binary classifiers for 18 individual diseases).

A major advance of SonoEye is its ability to make predictions at the patient-level, rather than from a single image as in conventional image-level analysis³⁷. This design more closely reflects real-world clinical practice, in which specialists integrate information across multiple images and scanning planes before reaching a diagnosis. In other words, SonoEye provides a more clinically meaningful assessment of disease status, especially for the common heterogeneity.

To facilitate the standardized clinical deployment of SonoEye, we developed Eye-RADS, a 4-tier risk stratification system comprising normal, low-vision risk, high-vision risk, and tumor risk, derived from the 19 ocular conditions evaluated in this study. By linking ultrasound findings to management priorities, from routine follow-up for low-risk abnormalities to urgent oncological referral, Eye-RADS translates model predictions into a clinically actionable framework.

We further incorporated age as a physiological factor within this framework because risk interpretation in ophthalmology is inherently context-dependent (similar ultrasound findings may have different diagnostic implications across age groups). This addition provides clinically relevant background information and shifts the SonoEye from image pattern recognition alone towards a more biologically grounded mode of inference that better reflects expert reasoning. Importantly, age was used only to calibrate decision thresholds at the final stratification stage, not to modify image feature extraction.

Although generative LLMs are increasingly used in clinical AI, their susceptibility to hallucination³⁶ poses an unacceptable risk in medical settings. We therefore adopted a retrieval-based, template-driven reporting system, in which the model uses validated terminology and management recommendations drawn from a curated medical knowledge base. This design enables the generation of consistent reports with standardized clinical guidance, thereby improving reliability and reducing the risk of unsafe outputs.

A missed diagnosis at the screening stage may result in irreversible vision loss or delayed oncologic management, whereas a false-positive case would potentially increase specialist workload and patient anxiety. Therefore, a key consideration for large-scale deployment is the balance between screening sensitivity and the downstream burden imposed by false-positive referrals. In our screening cohort, SonoEye achieved a sensitivity of 98.3% and a specificity of 91.9% at a sensitivity-prioritized operating point. This trade-off reflects the clinical rationale that false-negative findings carry greater risk, whereas false-positive cases would be resolved upon subsequent specialist review without lasting harm to the patient.

SonoEye was developed using Chinese-language reports and ChineseCLIP³² pretrained weights, reflecting the language of the source data rather than an inherent limitation of the architecture. Cross-lingual adaptation should be feasible with language-specific tokenizer customization and modest paired image-report data for fine-tuning, but this requires future validation.

Despite the strengths outlined in our study, we recognize several limitations. First, SonoEye exhibited relatively lower sensitivity for several conditions. These conditions are challenging because they often present with subtle or heterogeneous ultrasound features that are less consistently captured across frames. Second, our current study is focused only on B-mode ultrasound. Future work could extend SonoEye to incorporate complementary modalities into a unified multimodal diagnostic framework, potentially improving sensitivity for conditions that are ambiguous on ultrasound alone. Third, although we validated the model on multicenter retrospective cohorts, prospective validation in real-world clinical workflows remains essential before clinical deployment.

In summary, our study establishes an ultrasound-based vision-language foundation model as a scalable paradigm for ophthalmic AI. It offers a transferable framework for clinically aligned deployment. As aging populations and unequal access to specialists continue to strain ophthalmic care, such an approach may help transform ocular ultrasound into a broadly deployable tool for vision preservation.

Methods

Ethics approval

Our retrospective study in this investigation was approved by the institutional review board (IRB) of The Eye and ENT Hospital of Fudan University (No. 2024147-1) with a waiver granted for the requirement of informed consent. All images and clinical metadata were de-identified before being used in this investigation.

The ophthalmic ultrasound dataset

The primary dataset was retrospectively collected from The Eye and ENT Hospital of Fudan University (Site A). Following standard clinical protocols, each patient examination included one diagnostic report and 1 to 10 ultrasound images. Diagnostic reports were processed to create unlabeled image-text pairs. The resulting training dataset comprises 215,356 image-text pairs from 70,452 patients, collected between January 2, 2017 and October 29, 2024. This dataset was used both for CLIP pretraining and for training downstream patient-level components, including the AttnMIL aggregation network and the age-adaptive thresholding module. No data from the internal or external test cohorts were involved in any stage of model development. Separately, a manually annotated dataset was curated by domain experts, consisting of 2550 labeled images. Within this set, 208 images were designated as candidate prototypes across all categories, while the remaining 2342 images were reserved as the image-level test cohort. The distribution of images across disease categories for both the prototype pool and the image-level test cohort is summarized in Supplementary Table 4.

For the patient-level evaluation cohorts, images from the same patient were grouped using patient identifiers. Dataset partitioning between the pretraining and testing datasets was performed at the patient level to prevent potential data leakage. To support different evaluation objectives, we constructed separate cohorts for screening-oriented and diagnostic-oriented assessments.

For screening evaluation, we constructed a temporally independent screening cohort from Site A to better approximate real-world deployment. This cohort consists of 1614 consecutive patients enrolled between May 29, 2025 and August 12, 2025, thereby reflecting the natural case distribution in routine clinical practice.

For diagnostic evaluation, we used both internal and external test cohorts. The internal patient-level test cohort comprises 3,835 images from 941 patients, selected to include all disease categories and risks to the greatest extent possible. To further assess generalizability, two external test cohorts are collected from the Department of Ophthalmology, Zunyi First People’s Hospital (Site B), and The Eye and ENT Hospital of Fudan University (Pujiang branch) (Site C). The cohorts comprise 511 patients (1028 images) and 1666 patients (7747 images), respectively.

Regarding exclusion criteria, cases were excluded if (1) the ultrasound images were of insufficient quality (e.g., severe noise, low resolution, or incomplete visualization of ocular structures), or (2) the associated clinical reports lacked a clear or interpretable diagnostic description. No exclusions were made based on patient demographics, disease type, or clinical severity.

The development of SonoEye model

The backbone architecture

The backbone CLIP model adopts Swin Transformer v2³¹ as the image encoder, taking advantage of its refined windowing mechanism and strengthened scale-aware feature extraction, which collectively yield superior robustness to low contrast, speckle noise, and operator-dependent variability which were commonly encountered in clinical ultrasound workflows. For text encoding, we employed ChineseCLIP³², which is tailored to learn aligned representations between ophthalmic ultrasound images and corresponding clinical Chinese reports (Supplementary Fig. 1a). To project image and text features into a shared semantic space, the model applies modality-specific linear layers followed by L2 normalization. For each image-text pair (x, r), the model computes normalized image features ${\hat{\mathrm{\Phi }}}_v\left(x\right)\in {\mathrm{ℝ}}^d$ and text features ${\hat{\mathrm{\Phi }}}_t\left(r\right)\in {\mathrm{ℝ}}^d$, and evaluates their similarity. A bidirectional contrastive loss (CLIP-style) is used to align matched image-report pairs while separating mismatched ones.

Pretraining CLIP model

The model is fine-tuned using NVIDIA A100 80GB GPU. The learning objective is to minimize the multimodal contrastive loss to align visual and textual representations. The optimization process is performed by an AdamW optimizer with a weight decay of 0.01, utilizing an effective batch size of 32 (per-device batch size of 8 with 4 gradient accumulation steps) and an initial learning rate of 5e-5. A linear warm-up of 500 steps is performed, followed by a linear decay learning rate schedule. Mixed precision (FP16) training is employed to enhance computational efficiency, and the best model is automatically saved based on the lowest validation loss recorded during periodic evaluations. Notably, this stage focuses on learning joint image-text representations rather than performing direct disease classification. The contrastive learning objective operates on image-report pairs without explicit disease labels, and therefore class imbalance among the downstream disease categories does not directly affect the optimization process at this stage.

Text tokenization

We developed EyeReportTokenizer to effectively process ophthalmic ultrasound reports’ structured and specialized text. The tokenizer first utilizes a mainstream Chinese medical word model (ChineseCLIP³²) in the initialization stage, and combines expert knowledge to construct an entity word list and a prioritized word map covering common ophthalmic ultrasound lesions. The preprocessing stage consists of three components. First, quantitative and qualitative descriptors frequently used in ultrasound reports are standardized using regular-expression-based rules. For example, non-standard expressions such as “low-moderate” and “low + ” are uniformly converted to the standardized format “low to moderate” to standardize descriptions of echogenicity and extent. Second, anatomical references (e.g., anterior, middle, or posterior segments) are detected and mapped to standardized location tags, ensuring consistent representation of spatial information across reports. Third, ophthalmic ultrasound disease entities are used to identify diagnostic findings mentioned in the report text. In addition, priority rules were manually defined according to clinical diagnostic hierarchies established by ophthalmology experts. These predefined priority rules were introduced to reflect hierarchical clinical relationships between certain findings (e.g., a retinal detachment may co-occur with vitreous opacity as a secondary sign). These rules do not remove subordinate findings but preserve them to provide the downstream model with explicit co-occurrence context.

Through this preprocessing pipeline, EyeReportTokenizer standardizes heterogeneous report language while preserving clinically relevant information, providing consistent textual representations for downstream tasks such as disease classification, report retrieval, and multimodal alignment.

Hybrid reasoning for image-level interpretation

To address the inherent heterogeneity and intra-class variability in real-world clinical ultrasound data, we developed a hybrid mechanism that captures the diverse morphological characteristics within each disease category. Importantly, inference uses only ultrasound images as patient input and does not require diagnostic reports.

For the image branch, we constructed a set of visual prototypes to enable few-shot, prototype-based disease classification without conventional supervised training. Rather than relying on a single class centroid, each disease category was represented by multiple visual prototypes to capture intra-class morphological variation. Specifically, ophthalmology experts first annotated and verified 208 ultrasound images spanning all disease categories, forming a candidate prototype pool. Given that many ophthalmic diseases exhibit heterogeneous imaging appearances, we further adopt a k-means clustering approach to characterize the visual feature distribution comprehensively in the embedding space, resulting in the representative prototypes (each representing a distinct morphological pattern within the disease class). In this study, we selected k = 3 based on a trade-off between available candidate images and observed intra-class diversity. During inference, test image features are compared against all prototypes of each category (that is 19*3), with the maximum cosine similarity serving as the class representative score, subsequently normalized to yield prototype-based class probabilities. The maximum operation is applied within each disease category, rather than across all categories, so that each class independently yields its own representative score prior to normalization.

For the text branch, we further developed a knowledge-driven prompt learning framework to complement visual pattern recognition with clinical expertise. We construct a comprehensive disease knowledge base containing key ultrasound findings, discriminative diagnostic features, and clinical terminology for each disease class. All textual descriptions are predefined and independent of the test images. From this knowledge base, we automatically generate diverse candidate prompts that capture different aspects of each disease’s semantic information. For each disease, the prompt achieving the highest average similarity with prototype images is selected as the optimal textual representation. This selected prompt is then encoded by the text encoder of SonoEye to produce the final text embedding for each class; no text is generated from or conditioned on the test image itself. At test time, similarities between test image features and each class’s optimal prompt embedding are computed and normalized as text-based class probabilities. Finally, working in conjunction with the visual prototype-based inference module, these semantic predictions are fused to yield final image-level diagnostic probabilities.

AttnMIL module for patient-level interpretation

To address the challenge of integrating heterogeneous information from multiple ultrasound images, an AttnMIL module³⁴ (Supplementary Fig. 1b) that comprises three key architectural components is proposed. The feature transformation module projects image-level probability vectors (extracted from the same model architecture as the screen module) through a fully connected layer to generate higher-dimensional feature representations, enabling richer semantic encoding of diagnostic information. Subsequently, the attention mechanism employs a two-layer fully connected network to compute attention scores for each image, followed by softmax normalization, which enables adaptive weighting of individual images based on their diagnostic relevance. The patient-level representation is then obtained through attention-weighted feature aggregation and subsequently processed by a multilayer perceptron for final classification.

Given the long-tailed distribution of several ocular conditions in the dataset, class imbalance was further addressed during the patient-level training stage. Specifically, the classification objective was implemented using a binary cross-entropy loss with class-specific positive weights. This strategy increases the penalty for false negatives in rare conditions and helps improve sensitivity for clinically important but infrequent diseases.

The evaluation of SonoEye model

We evaluated the full patient-level pipeline for screening, fine-grained diagnosis, and Eye-RADS risk stratification (Supplementary Fig. 2). Patients were first screened as normal or abnormal at the image level, and abnormal cases were then processed by the AttnMIL module to integrate multi-view information and produce final diagnostic predictions, with the normal category retained from the screening stage. These predictions were mapped to a 4-tier risk stratification, that is normal, low-vision risk, high-vision risk, and tumor risk according to a tree-structured taxonomy. The full mapping rules are provided in Supplementary Table 5. When multiple abnormal conditions co-occurred, category assignment followed a priority hierarchy reflecting clinical urgency.

To further enhance model adaptability and clinical interpretability, we introduce an age-adaptive thresholding module into the patient-level classification framework. Specifically, we partition patients into discrete age groups (e.g., < 18, 18–39, 40–59, ≥60), and for each group and each disease category, the model learns a distinct decision threshold in the logit space during training. The model first produces disease logits based solely on image-derived features. Age was then incorporated through an adaptive threshold module that adjusts the decision boundary for each disease class after the model generates image-based logits. These thresholds are optimized jointly with the model during training. Therefore, age does not directly influence feature learning but instead serves as a calibration factor for risk stratification. During inference, the predicted logits for each patient are adjusted according to the learned threshold corresponding to their age group and disease class, thereby enabling the model to account for potential differences in disease prevalence, diagnostic ambiguity, and risk profiles across age strata.

Reader study

To evaluate the clinical value of SonoEye and its potential as a risk stratification tool, we conducted a comprehensive reader study at both image-level and patient-level. Six readers participated in this study, stratified into three groups based on their experience in ocular ultrasound interpretation: (1) two non-specialists, who were generalist ophthalmologists without dedicated subspecialty training in ocular ultrasound but with general clinical familiarity with ocular imaging; (2) two junior specialists, who were certified sonographers with 2 to 3 years of experience in ocular ultrasound; and (3) two senior specialists, who were certified sonographers with 10 to 15 years of experience in ocular ultrasound imaging. The reader study was designed in two phases to assess: (1) the standalone performance of AI compared to human readers, and (2) the impact of AI assistance on reader diagnostic accuracy.

In the first phase, readers independently reviewed all cases without AI assistance, providing their diagnostic assessments based on their clinical expertise. The second phase was conducted where readers reviewed the same cases with AI predictions provided as a reference. Readers were instructed that they could agree with, modify, or override the AI suggestions based on their clinical judgment. To prevent order effects, the sequence of case presentation was randomized between phases.

At the image-level, readers evaluated individual images for the presence of abnormalities across different disease categories. At the patient-level, readers synthesized findings from multiple images to provide an overall diagnostic assessment and risk stratification for each patient. All reader interpretations were recorded using a standardized reporting template, and inter-reader agreement was calculated using Cohen’s kappa. The reference standard for all cases was established by consensus of expert ophthalmologists, which served as the ground truth for performance evaluation. To formally assess the effect of AI assistance, we performed mixed-effects logistic regression analyses for image-level and patient-level correctness across all readers and within each experience subgroup.

Report generation

Aiming to support intelligent clinical decision-making aligned with real-world practice, we develop a comprehensive structured report generation system that transforms the two-stage diagnostic outputs into clinically meaningful documentation, inspired by prior work on ophthalmic report generation³⁸. For cases flagged as abnormal during the initial screening stage, the system systematically integrates the screening result with subsequent multi-label disease predictions, ranking all diagnostic labels according to predefined priority rules. Subsequently, patients will be stratified by Eye-RADS (4 risk categories), which include normal, low-vision risk, high-vision risk, and tumor risk. This approach ensures that both primary diagnoses and secondary findings are appropriately identified and presented in order of clinical significance, while maintaining clear documentation of the initial screening determination. Furthermore, an integrated knowledge base-driven recommendation engine provides personalized clinical guidance tailored to individual patient profiles. This intelligent system delivers specific management suggestions based on the screening outcome, individual disease labels, combined abnormal conditions, and corresponding risk stratifications (Supplementary Fig. 7). The recommendation engine encompasses stratified follow-up protocols, treatment considerations, and monitoring guidelines, thereby creating a comprehensive decision-support framework that aids both patients and clinicians in understanding the clinical implications of ocular ultrasound findings and facilitating evidence-based clinical management.

Statistical analysis

We used AUC and accuracy as the primary indicators for model performance evaluation. The Cohen’s linear weighted kappa agreement levels were measured as evaluation indicators of the confusion matrix. As suggested by Cohen, kappa results should be interpreted as follows: 0.61–0.80 as substantial and 0.81–1.00 as almost perfect agreement. The 95% CI for all metrics was calculated using 1000 non-parametric bootstrap methods. For the reader study, we performed mixed-effects logistic regression to evaluate the effect of AI assistance on diagnostic accuracy. Correctness of each diagnosis was modeled as the binary outcome, AI assistance as a fixed effect, and random intercepts were included for both readers and cases to account for inter-reader variability and case-level difficulty. All statistical tests were two-sided. All statistical analyses were performed in the Python environment using scikit-learn, NumPy, and Matplotlib libraries.

Author contributions

X.Q. conceived, designed, and supervised the project. T.Z. and J.G. provided clinical expertise and jointly supervised the study. Z.Z, D.Y., and Q.S. pre-processed the raw image data, developed the deep learning framework and software tools necessary for the experiments. X.C., Q.C., Y.Z. and J.Q. collected the raw ultrasound image, clinical diagnostic report, and defined the clinical labels. Z.Z and X.Q. executed the research, performed statistical analysis, and wrote the manuscript. All authors contributed to the review and editing of the manuscript.

Data availability

The authors declare that the main data supporting the results in this study are available within the paper and its supplementary information. All ultrasound datasets from The Eye and ENT Hospital of Fudan University and Zunyi First People’s Hospital are protected because of patient privacy yet the test data can be made available from the corresponding author upon reasonable request and with permission of hospitals, for academic purposes.

Code availability

The codes used in this study are available at https://github.com/Qian-IMMULab/SonoEye. Custom codes and the annotation tool for the deployment of the AI system are available for research purposes from the lead corresponding author upon reasonable request (qianxj@shanghaitech.edu.cn).

Competing interests

X.Q., T.Z., J.G., Z.Z., and D.Y. are co-inventors on a provisional patent application (No. 202511043125.7) encompassing the work described. The other authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41746-026-02870-5.