A multitask framework for automated multi-frame right upper quadrant ultrasound interpretation and clinical decision support

Abstract

Ultrasound is a cornerstone of emergency and hepatobiliary imaging, yet its interpretation remains highly operator-dependent and time-sensitive. Here, we present a multitask vision-language agent (VLM) developed to assist with right upper quadrant (RUQ) ultrasound interpretation across the diagnostic workflow. The system was trained on data from a single tertiary academic medical center (9189 cases, 594,099 images) and evaluated on datasets from two independent institutions (1704 cases, 90,616 images; and 108 cases, 3240 images). The agent integrates frame-level visual understanding with report-grounded language reasoning to perform three tasks: (i) classification of 16 clinical findings, (ii) generation of diagnostic reports, and (iii) surgical decision support based on ultrasound findings and clinical data. For classification, the model achieved macro areas under the receiver operating characteristic curve (AUROCs) of 0.820, 0.794, and 0.775 on the internal, University of Colorado, and Stanford cohorts, respectively, and an internal micro AUROC of 0.865. It also generated post-edited reports that were difficult for radiologists to distinguish from expert-written versions in blinded evaluations. The agent further predicted cholecystectomy using generated reports and clinical data. These results support further evaluation of vision-language models for improving diagnostic consistency, reporting efficiency, and surgical triage in ultrasound practice.

Authors developed an AI system for interpreting multi-frame right upper quadrant ultrasound examinations. Across three institutions, it classified findings, generated reports, and predicted the need for cholecystectomy.

Introduction

Ultrasound of the right upper quadrant (RUQ) is the preferred imaging modality for evaluating hepatobiliary disease, serving as the key tool for diagnosing and characterizing gallbladder and biliary pathology¹. Patients with suspected findings on CT often undergo confirmatory RUQ ultrasound to assess features, such as gallstones, wall thickening, and bile duct dilation, which directly guide surgical decision-making². However, RUQ ultrasound interpretation remains highly operator-dependent and limited by image noise, artifacts, and variability in acquisition and expertise. These factors can reduce diagnostic reliability and consistency across readers^3,4. A vision-language model (VLM) designed for RUQ ultrasound could help overcome these limitations by integrating visual and clinical context to standardize interpretation, highlight key findings, and generate structured, decision-ready reports that support accurate and timely surgical management.

Early advances in AI have shown substantial promise in ultrasound interpretation across diverse applications, including the detection and classification of thyroid nodules^5–7, breast masses^8–10, cervical lymph nodes¹¹, and liver lesions^9,12,13. However, conventional convolutional neural network (CNN)-based systems are inherently task-specific, typically generating narrow outputs, such as binary classifications or segmentation maps. This design limits their ability to integrate contextual information, perform multiple tasks, or emulate the broader clinical reasoning required for comprehensive diagnostic decision-making¹⁴.

Recent advances in transformer-based VLMs have enabled joint learning from visual and textual modalities, offering flexible, generalizable performance across diverse medical imaging tasks^15–18. In radiology, VLMs have shown strong capabilities in disease classification from chest X-rays¹⁹, automated report generation for CT^20,21, and visual question answering across multiple imaging domains^22–24—often performing well even in zero-shot settings where no task-specific training data are available²⁵. However, existing research has not yet extended these capabilities to ultrasound cine loops—dynamic, variable-length sequences that capture real-time anatomy and physiology, forming the foundation of RUQ ultrasound interpretation²⁶. Nor have current models demonstrated true multitask generalization across the full scope of ultrasound use, from image-based diagnosis to structured reporting and decision support. Addressing these gaps is crucial for developing VLMs that can operate as robust, clinically integrated assistants—capable of supporting interpretation and guiding surgical decision-making in real-world RUQ imaging workflows.

In this study, we introduce a VLM agent for multi-frame RUQ ultrasound interpretation, designed to support realistic clinical workflows in emergency care. Our work makes three key contributions. First, we curated a large-scale, multi-center dataset (n = 11,001), including a primary cohort from Johns Hopkins University (JHU) and two external validation cohorts from Stanford University and the University of Colorado (CU). Second, we developed a unified three-stage framework that reflects the typical RUQ ultrasound workflow: (1) abnormality classification to automatically detect and localize pathologic findings; (2) diagnostic report generation to produce coherent, clinically grounded narratives consistent with expert radiology standards; and (3) treatment decision guidance to predict downstream management steps. Third, we performed blinded expert evaluations to assess the realism, diagnostic accuracy, and clinical acceptability of the model’s outputs, demonstrating its potential for integration into real-world diagnostic pipelines. Overall, this work establishes an integrated framework that connects clinical motivation, data scale, model design, and rigorous evaluation, paving the way for practical deployment of vision-language agents in ultrasound interpretation (Fig. 1).

Fig. 1. Overview of the right upper-quadrant (RUQ) ultrasound workflow and the proposed US-Agent model.

a Clinical workflow of RUQ ultrasound examination, illustrating the sequential steps from clinical evaluation and ultrasound acquisition to radiologist interpretation, diagnostic reporting, and surgical decision-making. b Workflow of the US-Agent model. The internal Johns Hopkins University (JHU) dataset is used for both training and inference, while external datasets from Stanford University and the University of Colorado (CU) are used for independent testing. The trained agent supports three downstream tasks: abnormality classification, automated report generation, and treatment decision guidance. A radiologist Turing test was performed to assess realism and clinical accuracy. c Detailed structure of each downstream task. The abnormality classification module (USFM, ultrasound foundation model) predicts 16 diagnostic labels; the report generation module (Qwen VLM) produces structured radiology reports; the treatment decision module integrates image, report, disease, and clinical features to identify patients who require cholecystectomy; and the Turing test compares model-generated versus radiologist-written reports to assess realism and diagnostic fidelity. US ultrasound, VLM vision-language model.

Results

Study patients and label extraction

To simulate a realistic ED workflow, we curated a multi-center dataset comprising 11,001 RUQ ultrasound text-image series. This included a primary cohort from JHU (9189 cases, 594,099 images), along with two external validation cohorts from Stanford University (108 cases, 3240 images) and CU (1704 cases, 90,616 images). The original ultrasound reports followed templated narrative structures with predefined sections and standardized terminology, but lacked explicit structured labels. To enable supervised learning, we employed a locally deployed Llama 3 language model²⁷ to extract binary diagnostic labels across predefined hepatic, biliary, renal, pancreatic, vascular, and other abdominal categories. Biliary (45%) and hepatic (30%) findings were the most common, followed by renal (15%) and miscellaneous categories, such as ascites and pleural effusion (10%). In total, 37,849 positive findings were identified, corresponding to an average of 3.4 findings per patient.

To ensure the reliability of our automated pipeline, we rigorously evaluated the Llama 3 extraction model against a manually reviewed subset of 100 ultrasound reports. The automated extraction demonstrated high performance, achieving a label-level micro-accuracy of 99.60% (4183/4200) and a report-level strict accuracy of 83.00%. For positive findings, the model maintained high micro-precision (96.50%), micro-recall (98.90%), and micro-F1 (97.68%), while the macro-average F1-score was 97.87%. Across the 4200 label evaluations, there were only 17 total errors (13 false positives and 4 false negatives), indicating a highly reliable extraction process.

Abnormality classification

The classifier based on the ultrasound foundation model (USFM) achieved a macro AUROC of 0.820 and a micro AUROC of 0.865. At the per-disease level (Table 1), conditions such as right pleural effusion (AUROC = 0.926), ascites (AUROC = 0.904), and hepatic steatosis/fatty liver (AUROC = 0.904) achieved high performance (AUROC > 0.90). Twelve additional conditions demonstrated good discriminative ability with AUROCs between 0.70 and 0.90. Only pancreas poorly visualized (0.670) showed more limited performance. Pearson correlation analysis revealed no significant association between disease prevalence and model performance (r = − 0.040, p = 0.874).

Table 1.

Internal Johns Hopkins University (JHU) – full clinical metrics (n = 917)

Label	Positives %	AUROC	AP	Sens@90%Spec	Sens@95%Spec
Ascites	11.3%	0.904	0.677	0.740	0.606
Biliary sludge	30.5%	0.818	0.733	0.579	0.450
Cholecystitis	8.7%	0.826	0.377	0.525	0.350
Cholelithiasis	55.5%	0.895	0.917	0.752	0.585
Coarse hepatic echotexture	8.6%	0.745	0.248	0.367	0.177
Common bile duct dilation	11.0%	0.775	0.327	0.396	0.248
Gallbladder distention	32.1%	0.738	0.558	0.316	0.184
Gallbladder wall thickening	19.1%	0.823	0.583	0.509	0.400
Hepatic steatosis/fatty liver	21.5%	0.904	0.765	0.772	0.695
Hepatomegaly	24.1%	0.772	0.520	0.389	0.235
Increased hepatic echogenicity	21.8%	0.871	0.696	0.720	0.590
Increased renal echogenicity	7.6%	0.845	0.503	0.629	0.543
Pancreas poorly visualized	32.3%	0.670	0.454	0.186	0.098
Pericholecystic fluid	10.1%	0.782	0.313	0.387	0.280
Renal cyst	11.8%	0.826	0.574	0.639	0.556
Right pleural effusion	11.1%	0.926	0.712	0.814	0.618

AUROC area under the receiver operating characteristic curve, AP average precision, Sens sensitivity, Spec specificity.

To evaluate the model’s clinical utility under high-specificity conditions, we selected fixed per-label classification thresholds on the full internal JHU test set at target specificities of 90 and 95%. These JHU-derived thresholds were then applied unchanged to the CU and Stanford external validation cohorts, without external threshold retuning. On the internal JHU set, sensitivity at the 90% specificity operating point was high for several key findings, including right pleural effusion (0.814, 95% confidence interval (CI): 0.727–0.877), hepatic steatosis/fatty liver (0.772, 95% CI: 0.708–0.825), cholelithiasis (0.752, 95% CI: 0.713–0.788), and ascites (0.740, 95% CI: 0.649–0.815). These results support the model’s utility as a triage tool that preserves clinically useful sensitivity while maintaining low false-positive rates.

External validation on two independent datasets further assessed model generalizability. The USFM-based classifier achieved a macro AUROC of 0.775 on the Stanford dataset (n = 108) and 0.794 on the CU external dataset (n = 1704). Figure 2 summarizes classifier performance and cross-dataset generalization; the corresponding ablation comparing the USFM and ResNet-50 backbones is provided in Supplementary Table S1. Per-label metrics for the internal JHU cohort and external CU cohort are summarized in Tables 1 and 2. Full per-label results for JHU, CU, and Stanford are provided in Supplementary Tables S2–S4, and the per-label precision-recall (PR)/ROC curves across all three cohorts are shown in Supplementary Fig. S1.

Fig. 2. Performance of the automated abnormality classifier and cross-dataset generalization.

a Per-label comparison of classification metrics between the ultrasound foundation model (USFM) and the ResNet-50 baseline across representative organ systems. Blue and orange denote USFM and ResNet-50, respectively. USFM improved overall performance relative to the ImageNet-pretrained ResNet, with label-specific variation. b Comparison of disease prevalence (bars) and corresponding AUROC (lines) across the cohorts. Bars and lines are color-coded by dataset: internal Johns Hopkins University (JHU; blue), Stanford University (orange), and University of Colorado (CU; green). Despite substantial variations in disease prevalence, model performance remained stable across datasets. Source data are provided as a Source Data file. c Correlation between disease prevalence and AUROC for the internal and external validation cohorts. No statistically significant association was observed (p > 0.05), confirming the model’s robustness to label imbalance and institutional variability. AUROC area under the receiver operating characteristic curve.

Table 2.

External University of Colorado (CU) – full clinical metrics (n = 1704)

Label	Positives %	AUROC	AP	Sens@90%Spec	Sens@95%Spec
Ascites	3.2%	0.937	0.544	0.673	0.618
Biliary sludge	10.1%	0.606	0.291	0.314	0.221
Cholecystitis	4.2%	0.846	0.287	0.254	0.211
Cholelithiasis	25.2%	0.873	0.779	0.684	0.447
Coarse hepatic echotexture	6.9%	0.768	0.236	0.407	0.212
Common bile duct dilation	8.1%	0.715	0.280	0.304	0.167
Gallbladder distention	6.6%	0.739	0.274	0.292	0.239
Gallbladder wall thickening	11.0%	0.827	0.463	0.351	0.255
Hepatic steatosis/fatty liver	31.3%	0.875	0.801	0.725	0.620
Hepatomegaly	21.0%	0.744	0.427	0.319	0.162
Increased hepatic echogenicity	31.8%	0.858	0.760	0.703	0.561
Increased renal echogenicity	0.4%	0.823	0.155	0.429	0.286
Pancreas poorly visualized	36.8%	0.575	0.422	0.212	0.108
Pericholecystic fluid	3.2%	0.786	0.183	0.278	0.204
Renal cyst	5.9%	0.785	0.419	0.624	0.554
Right pleural effusion	1.3%	0.940	0.278	0.773	0.682

AUROC area under the receiver operating characteristic curve, AP average precision, Sens sensitivity, Spec specificity.

Automated radiology report generation

Radiology report generation lies at the center of the diagnostic workflow, serving as the key mechanism through which imaging findings are communicated as clinically actionable information. In the emergency department (ED), where timely and standardized interpretation is essential, report generation provides a structured synthesis of image-derived evidence that guides downstream decision-making. To assess the ability of VLMs to produce clinically meaningful ultrasound reports directly from multi-frame inputs, we trained three instruction-tuned architectures, including the M3D visual encoder paired with either Llama-3.1 (8B) or Phi-3-mini-4k (3.8B), and the Qwen2.5-VL (7B) model, resulting in three variants: UltrasoundLlama, UltrasoundPhi, and UltrasoundQwen.

Traditional natural language generation (NLG) metrics are widely used to capture fluency and lexical overlap between the generated reports and the ground-truth radiology reports. As demonstrated in Fig. 3, all models achieved strong semantic fidelity on the internal JHU test set. Among them, UltrasoundQwen achieved superior overall performance across all five lexical metrics relative to both UltrasoundLlama and UltrasoundPhi (Holm-adjusted P < 0.001 for all ten pairwise comparisons; exact values in Supplementary Table S7), with(p < 0.001; BLEU-1 = 0.5902, BLEU-4 = 0.3940, ROUGE-1 =  0.6572, ROUGE-L = 0.5527, and METEOR = 0.5506). Between the M3D-based variants, UltrasoundPhi achieved higher BLEU-1, whereas UltrasoundLlama showed stronger ROUGE-L and METEOR on the internal cohort.

Fig. 3. Cross-institutional comparison of automated report generation performance.

Bar plots summarizing the performance of UltrasoundLlama, UltrasoundPhi, and UltrasoundQwen across the internal Johns Hopkins University (JHU) dataset and two external validation cohorts from Stanford University and the University of Colorado (CU). Bars are color-coded by model: UltrasoundLlama (teal), UltrasoundPhi (orange), and UltrasoundQwen (blue). Each metric represents the mean of three independent runs, with standard deviations shown as error bars. Text similarity metrics (BLEU, ROUGE, METEOR, BERT-F1) capture linguistic and semantic fidelity, whereas clinical realism metrics (DocLens²⁸ and the FORTE²⁹ categories) assess factual accuracy and structural correspondence to expert-written reports. UltrasoundQwen consistently achieved the highest DocLens scores and strong overall clinically focused performance across datasets. Statistical comparisons between models were performed using a two-sided Welch’s t-test (unequal variances assumed) for each pairwise comparison (UltrasoundLlama vs. UltrasoundPhi, UltrasoundPhi vs. UltrasoundQwen, and UltrasoundLlama vs. UltrasoundQwen) within each institution and metric; P values were adjusted for multiple comparisons within each institution-metric family of three pairwise comparisons using the Holm–Bonferroni method. Statistical significance is denoted by asterisks (ns, not significant; *P < 0.05, **P < 0.01, and ***P < 0.001). Exact P values, t statistics, degrees of freedom, 95% confidence intervals of the mean difference, and Cohen’s d effect sizes for all comparisons are provided in Supplementary Tables S7, S8, and S9.

To assess generalizability, we further benchmarked the models on two external datasets from Stanford University and CU. Consistent with internal validation findings, UltrasoundQwen achieved the strongest performance on most lexical metrics. On the CU cohort, UltrasoundQwen achieved BLEU-1 = 0.4082, BLEU-4 = 0.0945, ROUGE-1 = 0.4564, ROUGE-L = 0.3317, METEOR = 0.3270, and BERT-F1 = 0.8798. Full quantitative comparisons are presented in Supplementary Table S5.

Since NLG metrics often fail to reflect the clinical accuracy and informational density required for diagnostic use, we further evaluated all models using two clinically oriented frameworks: DocLens²⁸, an LLM-as-a-judge claim-verification score, and the Feature-Oriented Radiology Task Evaluation (FORTE)²⁹, which quantifies the inclusion of key clinical information across four dimensions (degree, landmark, feature, and impression). On the internal JHU dataset, UltrasoundQwen achieved a DocLens score of 0.6094 and a FORTE average F1-score of 0.6539, outperforming UltrasoundLlama and UltrasoundPhi. On the external Stanford dataset, UltrasoundQwen achieved the highest DocLens score (0.5470), while UltrasoundPhi showed the highest FORTE average score (0.5292). On the external CU cohort, UltrasoundQwen again led with a DocLens score of 0.5904 and a FORTE average of 0.5337. Full clinically focused comparisons are provided in Supplementary Table S6.

Cholecystectomy treatment recommendation

To assess the agent’s ability to support downstream clinical decision-making, we evaluated whether a large language model (LLM) could predict the need for cholecystectomy using multimodal clinical inputs. We implemented a Llama 3 model and focused the analysis on a cohort of 2170 patients with confirmed cholecystitis, where the surgical decision is clinically critical.

We first established a benchmark using ground-truth data that included patient demographics, International Classification of Diseases (ICD) diagnostic codes before the cholecystitis diagnosis (e.g., comorbidities), and the original radiologist reports. As shown in Table 3, the benchmark model achieved its highest performance when all input features were combined, yielding an accuracy of 0.843 and a recall of 0.665. Ablation analysis revealed that each modality contributed a meaningful signal to the model’s performance, with the combination of ICD codes and narrative reports providing the greatest improvement in recall compared with patient variables alone.

Table 3.

Performance of large language model (LLM)-based cholecystectomy treatment recommendation model

	Original report		Generated report
Input features	Accuracy	Recall	Accuracy	Recall
Variables	0.869	0.126	0.868	0.128
Variables + ICD	0.890	0.515	0.814	0.500
Variables + Report	0.787	0.431	0.794	0.438
Variables + ICD + Report	0.843	0.665	0.718	0.602

Comparison of model accuracy and recall using original human-written radiology reports versus vision-language model (VLM)-generated reports as input. Each configuration combines different data modalities, including patient demographics, International Classification of Diseases (ICD) diagnostic codes, and either human-written or generated reports.

We then tested whether our AI agent could replicate this predictive capability using only machine-generated inputs. To do so, we replaced the human-written radiology reports with those generated by our VLM and repeated the analysis. The results demonstrated that the VLM-generated reports served as an effective proxy for ground-truth narratives. The model incorporating these generated reports, along with demographic and ICD features, achieved an accuracy of 0.718 and a recall of 0.602. Although a performance gap remains compared with the benchmark, the model retained strong recall, indicating that the VLM captured and conveyed the essential clinical information required for this complex downstream prediction task.

Expert evaluation of the VLM agent

Blinded Turing tests were conducted to assess the ability of radiologists to distinguish between AI-generated and human-written ultrasound reports. Radiologists identified unedited AI-generated reports with an average accuracy of 0.85 (95% CI: 0.76–0.93), while accuracy decreased to 0.46 (95% CI: 0.30–0.62) when evaluating post-edited AI-generated reports (p < 0.05). The average evaluation time was 1.18 (95% CI: 0.76–1.32) minutes per case for unedited AI reports and 1.82 (95% CI: 1.42–2.01) minutes per case for post-edited AI reports (p < 0.05). For the combined set of unedited and edited AI reports, the average accuracy of identifying the human-written report as ground truth was 0.61 (95% CI: 0.34–0.98) when only text was provided and 0.69 (95% CI: 0.47–0.88) when both text and corresponding ultrasound images were available (p > 0.05). When stratified by experience level, expert radiologists achieved accuracies of 0.56 (95% CI: 0.40–0.70) (text-only) and 0.69 (95% CI: 0.604–0.850) (text plus image), whereas trainees achieved 0.68 (95% CI: 0.44–0.72) (text-only) and 0.64 (95% CI: 0.34–0.70) (text plus image) (p > 0.05). These results are summarized in Fig. 4 and Table 4; additional analyses of evaluation time and inter-rater agreement are provided in Supplementary Fig. S2. Representative unedited, post-edited, and ground-truth reports are provided in Supplementary Tables S10–S12.

Fig. 4. Turing test evaluation results.

Boxplots show accuracy (a), recall (b), and F1 score (c) for radiologist judgments in the AI-clinician preference (blue) and AI-clinician collaboration (pink) settings, with and without ultrasound images. Each data point represents one independent radiologist (n = 12), with the individual radiologist serving as the unit of analysis; no technical replicates were included. Box plots show the median (center line), interquartile range (25th–75th percentiles; box), and minimum and maximum values (whiskers). Asterisks denote the statistical comparisons displayed in the figure (***p < 0.001). AI artificial intelligence.

Table 4.

Turing test performance in distinguishing AI from human-generated reports

Comparison	Condition	Accuracy
Report type
	Initial AI report	0.85
	Post-edited AI report	0.46
Imaging context
	Without images	0.61
	With images	0.69
Expertise level (with vs. without image)
Expert	Without image	0.56
Expert	With image	0.69
Trainee	Without image	0.68
Trainee	With image	0.64

Accuracy represents the proportion of cases in which radiologists correctly identified the human-written report as the ground truth. Lower accuracy indicates greater indistinguishability between AI-generated and expert-written reports.

AI artificial intelligence.

Discussion

In this study, we developed and validated a vision-language agent framework for automated interpretation of multi-frame RUQ ultrasound, designed to emulate the complete clinical workflow from image analysis to decision support. Our results show that our multitask system can accurately perform three core tasks traditionally requiring physician expertise—abnormality classification, diagnostic report generation, and treatment recommendation—thereby establishing a foundation for practical physician-AI integration in emergency care.

Our agent trained on the USFM demonstrated higher performance than a generic ResNet-50 backbone. This highlights the value of large-scale, domain-specific pre-training for medical imaging. Similar to recent ultrasound foundation and vision-language systems, such as USFM³⁰, URFM³¹, EchoVLM³², and UltraSAM³³, our results confirm that representation learning on ultrasound data yields gains that cannot be matched by generic visual encoders. These studies collectively show that domain-specialized visual encoders not only enhance classification but also support report generation and downstream reasoning tasks. Nevertheless, the observed performance gap across institutions, particularly on the CU external cohort, underscores the persistent challenge of cross-site generalization, consistent with prior reports on ultrasound domain shift and cross-site transferability^34,35. Differences in patient populations, scanning conventions, and device vendors remain key sources of variation. Future work should incorporate domain-adaptation and adversarial learning strategies³⁶ as well as federated or continual learning pipelines³⁴ to promote robustness across global clinical settings.

The diagnostic reports generated by UltrasoundLlama, UltrasoundPhi, and UltrasoundQwen represent a step beyond conventional text-similarity metrics commonly used in medical natural language generation. Earlier report-generation systems—such as LLAUS³⁷, EchoVLM³², and USFM³⁰—have advanced the integration of visual reasoning with narrative synthesis, yet their evaluation has largely depended on surface-level fluency measures (e.g., BLEU, ROUGE). In contrast, by applying clinically weighted evaluation frameworks, such as FORTE²⁹ and DocLens²⁸, we assessed factual precision and informational completeness rather than linguistic overlap alone. The strong performance of the UltrasoundQwen model on the internal cohort (FORTE F1 = 0.65, DocLens = 0.61) demonstrates that its generated narratives accurately capture key diagnostic content, characterize pathologic findings, identify anatomic landmarks, and maintain impression consistency—approaching the standard of expert radiology reporting.

Importantly, the output from Ultrasound Qwen encodes clinically decisive information to be actionable in our large-language-model classifier to predict the need for cholecystectomy with high recall (0.602). This extends the utility of vision-language systems from description to inference, paralleling recent work that couples report generation with clinical decision support and outcome prediction^21,38. The ablation results further reveal that report generation is not merely cosmetic but transforms raw visual data into an interpretable and clinically relevant representation suitable for downstream reasoning, an approach aligned with emerging paradigms of reasoning-centric VLMs in medicine³⁵.

The blinded expert evaluation illustrates how human oversight enhances model credibility. Although unedited AI reports were generally identifiable, minimal post-editing (68.4 s ± 21.0 s on average per report in this paper) rendered them statistically indistinguishable from human-written reports, suggesting a realistic AI-copilot paradigm in which radiologists verify and refine AI-drafted outputs rather than authoring from scratch. There are several possible reasons why clinician-AI collaboration (AI-copilot paradigm) can be effective. First, even with a baseline level of inter-rater variability in preference and error correction, AI assistance provides a consistent reference that helps reduce subjective differences among clinicians. Second, collaboration between clinicians and AI can enhance both efficiency and diagnostic precision when the AI system provides complementary insights rather than competing judgments. Developing robust strategies for when and how to present AI-generated drafts is, therefore, crucial to maximizing the advantages of assistance. Third, it is plausible that revising an AI-generated report can substantially reduce reporting time compared to writing from scratch. Although this was not explicitly quantified in our study, time savings represent an important direction for future work, especially given that reporting speed depends on clinical context, expertise, and case complexity. Such collaboration could reduce documentation burden while preserving professional accountability. The differential performance between expert and trainee readers—experts benefiting more from image context, suggesting that clinical experience remains essential for adjudicating subtle findings, whereas AI contributes scale, speed, and consistency.

Several limitations warrant acknowledgment. The retrospective design and label extraction using a large language model introduce weak supervision that may omit fine-grained diagnostic nuance. We also acknowledge that our current model prioritized acute and emergent biliary and hepatic conditions (e.g., cholecystitis, biliary obstruction) and excluded certain critical oncologic findings, such as hepatic masses and gallbladder polyps. While the system serves as a triage and decision-support tool rather than a comprehensive cancer screening system, the inclusion of oncological findings remains a primary objective for future iterations of the model. The cholecystectomy prediction model was trained on a limited subset of patients, and broader, prospective validation is needed to confirm generalizability across diverse institutions and clinical scenarios. In addition, although the revised fixed-threshold analysis applies JHU-derived decision thresholds unchanged to the CU and Stanford cohorts, these thresholds should be interpreted as retrospectively evaluated operating points rather than deployment-ready clinical cutoffs; prospective evaluation is still required to determine threshold stability across scanners, acquisition protocols, patient populations, and clinical workflows before real-world implementation.

Overall, this work establishes an integrated framework linking image perception, linguistic reasoning, and therapeutic guidance within a single multimodal agent. By coupling scalable foundation models with expert oversight, such systems could accelerate radiologic reporting, standardize interpretation quality, and extend diagnostic capacity to resource-limited settings. The next critical step is prospective clinical evaluation to determine how these physician-AI partnerships influence diagnostic accuracy, workflow efficiency, and ultimately patient outcomes in real-world emergency care.

Methods

Ethics statement

All study protocols were reviewed and approved by the Institutional Review Boards of Johns Hopkins University (No. IRB00424745), Stanford University (No. 77734), and the University of Colorado (No. 24-1825). Given the retrospective design and the use of fully de-identified data, the requirement for informed consent was waived by all institutional ethics committees.

Study design and data acquisition

Multi-center cohort description

This retrospective study utilized a multi-center dataset comprising 11,001 RUQ ultrasound text-image series. The primary cohort was obtained from JHU and included 9189 ultrasound examinations after quality filtering. Data collection for this cohort spanned from July 31, 2016, to December 31, 2023. Patients were included if they underwent a clinically indicated RUQ ultrasound examination. Exclusion criteria included incomplete data or technically limited images that precluded diagnostic interpretation. For external validation, two additional cohorts were incorporated. The first external cohort consisted of 108 cases (3240 images) from Stanford University, and the second external cohort comprised 1704 cases (90,616 images) from CU.

Participant demographic characteristics were retrieved from the electronic health record (EHR) for the patient populations underlying each cohort; quality filtering was applied at the examination level. For the primary JHU cohort, the patient population comprised 8864 unique patients (median age 55 years, interquartile range [IQR] 38–67); among the 8858 patients with a recorded binary sex/gender designation, 4977 (56.2%) were female and 3881 (43.8%) were male. For the external CU cohort, the patient population comprised 2317 unique patients (median age 48 years, IQR 32–65); among patients with a recorded binary sex/gender designation, 1313 (56.7%) were female and 1001 (43.3%) were male. Sex/gender and age were determined as documented in the EHR at the time of patient registration (recorded from institutional registration data rather than ascertained by self-report or assigned specifically for this study). Sex/gender and age were not used as inclusion/exclusion criteria, design variables, or stratification factors in any of the study tasks, and no sex- or gender-specific subgroup analyses were performed, as sex/gender was not a study endpoint; all analyses were conducted on the full cohort. Consistent with the SAGER guidelines, overall demographic characteristics are reported here and in the Nature Portfolio Reporting Summary, and individual-level demographic data are provided in the Source Data file accompanying this manuscript.

Automated report processing and label extraction with a local LLM

To generate structured labels for model training and evaluation, we developed an automated natural language processing pipeline to analyze all 11,001 free-text RUQ ultrasound reports corresponding to the image datasets. The pipeline employed a locally deployed Llama 3 70B Instruct model³⁹ to parse the narrative Findings and Impression sections of each report. Using optimized prompt engineering, the system extracted and standardized diagnostic information into 40 predefined medical categories spanning the hepatic, biliary, renal, pancreatic, and vascular systems. This automated extraction process identified a total of 37,849 positive findings, corresponding to an average of 3.4 findings per patient across the entire cohort. While this process identified 40 distinct categories (the full prevalence distribution is provided in Supplementary Table S13), a preliminary analysis revealed that many had a very low prevalence. To create a robust dataset for the subsequent classification task and mitigate severe class imbalance, we selected only the 16 labels with a prevalence greater than 5% for model training.

VLM agent framework

Image preprocessing and standardization

Our dataset is composed of clinical ultrasound cine loops (video sequences), which are inherently variable in length, ranging from under 20 to over 200 frames. To create a uniform input for our deep learning model, we developed a preprocessing pipeline to standardize every cine loop to a fixed length of 32 frames.

The core of our method is uniform temporal sampling. For longer videos, we deterministically select 32 frames evenly spaced across the entire duration of the scan. For the few videos shorter than 32 frames, we perform cyclical temporal padding by repeating the sequence to meet the required length. This two-pronged approach ensures that the standardized input is a faithful temporal representation of the original scan, capturing information from its beginning, middle, and end phases. This process preserves the temporal dynamics critical for our analysis while creating a fixed-size input required by the model. A detailed, step-by-step description of this standardization algorithm is provided in the Supplementary Material (Section S1).

Following frame selection, an automated workflow was used to remove artifacts and standardize the image inputs. This included isolating the diagnostically relevant fan-shaped scanning sector and eliminating non-imaging elements, such as embedded text overlays and display markers. The selected frames were then resized to a uniform resolution of 224 × 224 pixels using bicubic interpolation, and pixel intensities were normalized to a standardized range of [0.0, 1.0] to ensure consistent input for downstream model training.

Multi-task agent architecture

To emulate the clinical workflow of ultrasound interpretation, we developed a multitask agent framework comprising three sequential stages: (1) abnormality classification, (2) automated report generation, and (3) treatment recommendation. The architecture was designed so that the output from each stage serves as a conditional input for the subsequent one, enabling progressive information transfer across the diagnostic and decision-making pipeline. In the abnormality classification stage, a multi-label classification model processes the 32 standardized ultrasound frames to generate patient-level predictions for 16 clinically relevant pathologies. The model produces both categorical disease labels and 768-dimensional image embeddings, which are passed to the next stage. In the automated report generation stage, a VLM generates a structured diagnostic report conditioned on the visual features from the ultrasound frames and the structured predictions from the classification model. This stage links quantitative image interpretation with natural-language narrative synthesis. In the final treatment recommendation stage, a predictive model for cholecystectomy integrates multimodal information—combining the image embeddings from the classifier with text embeddings derived from either human-written or VLM-generated reports—to estimate the likelihood of surgical intervention.

Abnormality classification module

Model architecture and training

To enable automated abnormality classification, we developed a multi-component disease classification module consisting of three core elements. The first component was a visual feature extractor, implemented using the USFM, a Vision Transformer (ViT-B/16) backbone pretrained on a large-scale, domain-specific ultrasound dataset. This backbone processed standardized input images of 224 × 224 pixels (patch size 16 × 16) and generated 768-dimensional feature embeddings for each of the 32 frames per patient. The second component was an attention-based feature aggregator designed to handle the variable number of frames across patient studies. This module learned adaptive weighting of frame-level embeddings to produce a single consolidated 768-dimensional patient-level representation. The final component was a multi-label classification head, consisting of a dropout layer (rate = 0.1) followed by a linear layer that mapped the 768-dimensional feature vector to 16 disease probabilities. For the high-specificity analysis, per-label operating thresholds were selected once on the full internal JHU test set at target specificities of 90 and 95% and then applied unchanged to both external cohorts.

To address severe class imbalance in the training dataset, we employed asymmetric loss (ASL) as the primary objective function. Optimization was performed using AdamW (initial learning rate = 1 × 10⁻⁴) with a ReduceLROnPlateau scheduler that reduced the learning rate by a factor of 0.5 after 10 epochs without improvement in validation loss. Additional stabilization measures included gradient clipping (maximum norm = 5.0) and gradient surgery, which projected conflicting task gradients orthogonally to prevent inter-task interference. Data augmentation was implemented using Multi-Label Mixup with a Beta distribution parameter (α = 0.4) applied at a 50% probability. To further mitigate class imbalance, we applied adaptive sample reweighting, assigning double weight to hard samples (top 30% by loss) based on an exponential moving average (β = 0.9) of per-sample losses. Class weights were also incorporated, scaled inversely to class frequency, and clamped between 1.0 and 10.0. A 5-epoch warm-up phase preceded activation of the full training strategy.

Model evaluation

The performance of the final classifier was evaluated on 917 patient studies from the internal JHU test cohort. Evaluation was conducted across 16 disease categories with a prevalence greater than 5% in the training data. To assess generalizability, the model was further evaluated on two external validation cohorts. The first included 108 cases from Stanford University, and the second comprised 1704 cases from CU. For both external datasets, performance was reported for the same 16 disease categories used in the primary cohort.

Ablation study design

To validate the contribution of the domain-specific USFM backbone, a comparative ablation study was conducted. A baseline model was constructed using a general-purpose ResNet-50 architecture, pre-trained on the ImageNet dataset, as the visual feature extractor. All other components of the framework, including the attention-based aggregator and the multi-label classification head, remained identical to the primary model. This baseline model was trained and evaluated under the exact same conditions, utilizing the same 8269-sample training set, loss function, optimizer, learning rate schedule, and all other specified hyperparameters to ensure a direct and fair comparison of the backbones’ performance.

Automated report generation module

Vision-language model selection and fine-tuning

To enable multimodal report generation, we fine-tuned a suite of VLMs for the automated ultrasound reporting task. The candidate architectures included Qwen2.5-VL-7B, a 7-billion-parameter model with a Qwen2-VL⁴⁰ Vision Transformer backbone, and M3D⁴¹, a 3D medical VLM evaluated with the Phi-3-mini-4k (3.8B)⁴² and Llama-3.1-8B³⁹ language backbones. All models were fine-tuned on the internal training dataset (n = 8269) using a mixed-supervision, instruction-tuning strategy.

The training data were organized in a conversational format with three components: (1) the full findings section from the original human-written report, serving as the ground-truth response to a general instruction prompt to generate a diagnostic report; (2) a series of binary (Yes/No) visual question-answer pairs derived from the 13 most common Llama 3-extracted disease labels, designed to reinforce diagnostic accuracy; and (3) the complete set of 32 standardized ultrasound frames for each patient. For Qwen2.5-VL-7B, fine-tuning was performed using Low-Rank Adaptation (LoRA) with a rank of 16 and an alpha value of 32. The model was trained for three epochs with a learning rate of 1 × 10⁻⁴ and an effective batch size of eight (one sample per device, eight gradient accumulation steps), using bfloat16 precision. The M3D models underwent a two-phase training pipeline consisting of a vision-language alignment pretraining phase followed by a visual instruction-tuning phase, also using LoRA (rank = 16, alpha = 32) with a learning rate of 5 × 10⁻⁵ for five epochs. All training was conducted on a NVIDIA DGX workstation equipped with four A100 GPUs.

Report quality assessment with traditional, clinical, and AI-driven metrics

The quality of the generated reports was assessed across the internal JHU test set (n = 917) and the two external validation cohorts from Stanford University (n = 108) and CU (n = 1704). To quantify report quality, we employed a multi-faceted evaluation strategy combining traditional natural language generation (NLG) metrics with two advanced, clinically oriented frameworks designed to overcome the limitations of surface-level text similarity.

Traditional NLG metrics

To assess linguistic fluency and content overlap with ground-truth reports, we used a standard panel of metrics:

• ROUGE-1 (Recall-Oriented Understudy for Gisting Evaluation) ⁴³: measured the recall of unigrams.

• METEOR (Metric for Evaluation of Translation with Explicit ORdering) ⁴⁴: evaluated alignment based on exact unigrams, stemmed tokens, and synonyms.

• BLEU-1 and BLEU-4 (Bilingual Evaluation Understudy) ⁴⁵: measured the precision of 1-g and 4-g matches to assess grammatical structure.

• BERT-F1⁴⁶: computed F1 scores using contextualized embeddings to evaluate semantic similarity.

Clinically-oriented evaluation (FORTE)

To directly measure the clinical value of generated reports, we implemented the Feature-Oriented Radiology Task Evaluation (FORTE) framework²⁹. FORTE deconstructs a report’s clinical essence into four key components, each governed by a dedicated keyword lexicon. Following Li et al.²⁹, we curated the RUQ ultrasound keyword bank, where each category-specific lexicon encompasses both primary terms and clinically equivalent synonyms to capture lexical variability and differences in reporting style.

• Degree: describes the size, scope, and intensity of findings (e.g., dilation/distended/distension/enlarged/prominent, extensive, absent/negative/no/not, limited/poorly/suboptimally).

• Landmark: pinpoints the anatomical location of findings (e.g., pancreas/pancreatic, kidney/renal/urinary, interpolar/mid/midportion).

• Feature: details the primary imaging characteristics of the findings (e.g., lesion/mass/masses, size/volume, coarse/coarsened/heterogeneous).

• Impression: captures the final diagnostic conclusions (e.g., cholelithiasis, hepatic steatosis, and cirrhosis).

For evaluation, keywords from each category were extracted from both the AI-generated and ground-truth reports. By comparing the keyword matches, an F1-score was calculated for each dimension, and a final average F1-score quantified the report’s overall clinical information density and accuracy. (For the keyword details, please see Supplementary Table S14)

AI-driven factual accuracy evaluation (LLM-as-a-judge)

To simulate an expert review and validate the factual accuracy of our VLM’s outputs, we utilized an LLM-as-a-judge approach with the DocLens²⁸ tool on a local Llama 3 LLM environment. This automated process unfolds in three steps:

• Extract claims: DocLens parses the AI-generated report and extracts each distinct medical assertion.

• Verify presence: it then cross-references each claim against the ground-truth human report to determine if it is factually supported.

• Compute score: a final claim-verification score is calculated from the proportion of verified claims among all extracted claims. This score measures the factual reliability of the generated report by penalizing unsupported or hallucinatory statements.

Treatment recommendation for cholecystectomy

Patient cohort definition

To develop a clinically relevant prediction model for cholecystectomy, we extracted a subset of patients with a confirmed diagnosis of cholecystitis from the primary dataset using corresponding ICD diagnosis codes. The final cohort (n = 2170) was randomly partitioned into a training set (n = 1736; 80%) and a test set (n = 434; 20%). This filtering process improved class balance from less than 1% in the overall cohort to approximately 15% in the cholecystitis-specific subset. The ground-truth outcome—receipt of cholecystectomy—was determined by identifying the corresponding procedural codes in the patients’ electronic health records.

Predictive model development and optimization

To predict the need for cholecystectomy, we implemented a Llama 3 large language model in a few-shot classification setting. The task was formulated as a binary classification problem, prompting the model to output Yes or No regarding the necessity of surgical intervention. Each input prompt included combinations of engineered features serialized into a structured textual format. A full prompt consisted of patient demographics, relevant ICD codes, and the full text of either the human-written or VLM-generated radiology report. Relevant ICD codes were defined as diagnosis codes assigned prior to the index ultrasound examination and before any formal diagnosis of acute cholecystitis or related biliary inflammatory conditions. These codes primarily captured baseline comorbidities (e.g., metabolic, cardiovascular, hepatic, and gastrointestinal conditions) and prior medical history, while explicitly excluding ICD codes directly indicating cholecystitis or other acute gallbladder inflammatory diagnoses to prevent label leakage. Model performance was evaluated on the test set using accuracy and recall as primary metrics. The evaluation emphasized the ability of VLM-generated reports to serve as a viable substitute for human-written reports in downstream decision-making tasks.

Ablation study

To assess the relative contribution of each data modality to the predictive performance, a systematic ablation study was conducted. The model was evaluated using four configurations of input features: (1) patient variables alone, (2) variables plus ICD codes, (3) variables plus the radiology report text, and (4) all three feature types combined. Each configuration was tested using both original human-written and VLM-generated reports, enabling direct comparison of model performance across feature subsets and report sources. This analysis quantified the incremental predictive value contributed by each data modality and the degree to which AI-generated reports captured the clinical information necessary for surgical decision support

Expert evaluation of the VLM agent and physician-AI collaboration

Blinded reader study design

To evaluate the clinical acceptability and quality of AI-generated reports—and to explore potential collaboration between physicians and AI systems—we conducted a blinded reader study modeled after a Turing test framework⁴⁷. The study included two independent reader groups representing different levels of clinical experience: one group evaluated the initial AI-generated reports, and a separate group evaluated the post-edited versions. For the initial evaluation, a set of 60 non-overlapping patient cases was randomly selected from the test set. Each case included two reports presented side by side in randomized order: (1) the unedited AI-generated report produced by the Qwen2.5-VL-7B model and (2) the ground-truth human-written report authored by a radiologist. The evaluation followed a two-stage protocol. In the first stage, readers reviewed the text-only reports and were asked to identify which of the two had been generated by AI. In the second stage, readers were provided with the full ultrasound image series for each case and asked to reassess their decision. This two-step process enabled evaluation of both textual plausibility and image-grounded clinical reasoning. The study design is illustrated in Fig. 5.

Fig. 5. Turing test evaluation of AI-generated radiology reports.

a AI versus clinician preference. Radiologists were presented with ultrasound images alongside two reports—the original human-written version and the AI-generated version—and asked to identify which they believed to be the authentic human report. b AI + Clinician Collaboration. Radiologists proofread and post-edited AI-generated reports, correcting factual, locational, and severity errors and adding missing or clinically significant findings. The edited reports were then compared with the originals by an independent expert to evaluate indistinguishability and clinical accuracy. AI artificial intelligence.

Evaluation of post-edited reports

To assess whether human refinement could improve the perceived quality and realism of AI-generated reports, a second blinded reader study was conducted. For this evaluation, the AI-generated reports from a different subset of cases were manually post-edited by a board-certified radiologist with 7 years of post-fellowship experience (HXB) to correct factual inaccuracies, refine phrasing, and add clinically relevant details. A new group of six radiologists then compared these post-edited AI reports with the original human-written reports for the same cases. The evaluation protocol mirrored that of the first study, consisting of two stages: initial review of text-only reports, followed by re-evaluation after viewing the corresponding ultrasound image series. This design enabled a direct comparison of indistinguishability between the unedited and post-edited AI reports, quantifying the impact of human-AI collaboration on report realism and clinical acceptability

Statistical analysis

Normally distributed variables (e.g., report performance or accuracy) were compared using two-tailed paired t-tests, while non-normally distributed data were analyzed with the Wilcoxon signed-rank test. Between-group comparisons (e.g., expert vs. trainee, text-only vs. text + image) used independent t-tests as appropriate. Ninety-five percent confidence intervals (CIs) were estimated via bootstrap resampling. All tests were two-sided, with p < 0.05 considered statistically significant.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Author contributions

Conceptualization: H.G., C.Y.L., Y.W., R.W. and H.B. Methodology: H.G., C.Y.L., Y.W. and R.W. Investigation: C.Y.L., Y.W., Q.P., D.C. and C.T.L. Visualization: H.G., C.Y.L., Y.W. and R.W. Supervision: Y.D., J.W. and H.B. Writing-original draft: H.G., C.Y.L., Y.W. and Y.D. Writing-review & editing: H.G., C.Y.L., Y.W., R.W., Y.D., Q.P., D.C., Z.Z., T.V., L.Z., C.Z., C.T., J.S., S.K., F.S., J.A., J.Fa., J.Fe., D.O., U.H., A.Z., T.M., M.L., C.T.L., Z.J., I.K., J.W., L.Y. and H.B.

Peer review

Peer review information

Nature Communications thanks Lie Cai, Atefeh Zeinoddini and the other anonymous reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Data availability

The clinical imaging datasets generated and/or analysed during this study are not publicly available because they contain potentially identifiable patient information and are subject to Institutional Review Board (IRB) requirements, patient privacy regulations, and institutional data-sharing policies. Access to de-identified data may be considered for qualified researchers conducting non-commercial scientific research. Requests for data access should be directed to the corresponding author, Dr. Harrison Bai. Following receipt of a request, an application for data sharing will be submitted to the appropriate IRB for review. If the request is approved by the IRB and the applicable institutional data-use agreement has been executed, the requested de-identified data will be made available through a secure institutional mechanism in accordance with institutional policies. Requests will be reviewed as promptly as possible on a case-by-case basis. Approved investigators will be granted access to the shared data for the duration specified in the applicable data-use agreement. Source data are provided with this paper.

Code availability

The code developed for this study, including the vision-language model framework and analysis scripts, is publicly available at https://github.com/Mikeghm/ruq-ultrasound-vlm. The version of the code used in this study (v1.0) has been permanently archived in Zenodo at https://doi.org/10.5281/zenodo.21841033⁴⁸.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41467-026-77498-w.