Large language models for structuring knee ultrasound reports-A comparative evaluation of DeepSeek R1, gemini 2.5 flash, and GPT-4o

Abstract

Objectives

To evaluate three LLMs (DeepSeek R1, Gemini 2.5 Flash, and GPT-4o) for generating structured, clinically interpretable outputs from free-text knee ultrasound reports.

Materials and Methods

We retrospectively analyzed 359 de-identified knee ultrasound reports using four prompt strategies targeting (Prompt 1) information extraction, (Prompt 2) structured report generation, (Prompt 3) impression generation, and (Prompt 4) clinical recommendation generation. Two experienced physicians rated outputs using prespecified 5-point Likert rubrics for extraction quality, structuring quality, diagnostic usefulness, recommendation quality, readability, and subjective repeatability. Objective evaluations included Prompt 2 structural completeness and stability under a fixed slot schema (schema compliance, missing-item rate, and slot-level exact match), Prompt 3 objective repeatability via mapped binary findings (percent agreement and Cohen’s κ across repeated runs), and Prompt 3 diagnostic performance against an expert binary reference standard (sensitivity/specificity, F1 score, balanced accuracy, and MCC; 95% confidence intervals via bootstrapping). For Chinese recommendation metrics, BLEU/ROUGE-L were computed using character-level tokenization.

Results

Across 359 knee ultrasound reports, GPT-4o demonstrated the strongest overall performance across extraction, diagnostic impression generation, and clinical recommendation tasks. It achieved the highest physician-rated scores for NER (4.0 ± 0.6) and diagnostic summarization (3.8 ± 0.7), along with the best diagnostic performance (F1-score = 0.79 vs 0.54 for DeepSeek R1 and 0.45 for Gemini 2.5 Flash, with the highest balanced accuracy and MCC). DeepSeek R1 showed competitive performance in structured reporting (4.0 ± 0.5) and repeatability (3.41 ± 0.06), with stable schema compliance and run-to-run consistency. Gemini 2.5 Flash consistently underperformed, particularly in NER accuracy (3.2 ± 0.7) and diagnostic recall (0.37). Objective analyses further confirmed superior structural completeness, repeatability, and semantic quality of recommendations for GPT-4o compared with the other models.

Conclusion

GPT-4o consistently outperformed DeepSeek R1 and Gemini 2.5 Flash, supporting its potential to improve knee ultrasound report consistency and clinical utility.

Graphical abstract

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-026-03415-2.

Background

Ultrasound has become a widely used imaging modality in clinical practice owing to its noninvasiveness, real-time capability, cost-effectiveness, and accessibility [1–3]. In musculoskeletal applications, knee ultrasonography enables effective assessment of periarticular structures (e.g., soft tissues and osseous abnormalities) and provides limited yet clinically valuable visualization of key intra-articular structures, such as the menisci. It plays an important role in the early detection, dynamic monitoring, and postoperative follow-up of knee injuries.

Structured reporting has been increasingly advocated in radiology, including musculoskeletal ultrasound, to improve report completeness, consistency, and downstream clinical communication. A recent systematic review highlighted that structured reporting templates are associated with improved reporting quality and standardization across imaging subspecialties [4]. In parallel, controlled vocabularies and medical ontologies such as RadLex and SNOMED CT have been developed to standardize imaging and clinical terminology [5], providing an important semantic foundation for automated information extraction and report structuring in radiology.

Despite its clinical utility, knee ultrasound reports are predominantly documented in unstructured free-text format, with substantial variability in descriptive style and terminology among physicians [6, 7]. The lack of standardized formats and consistent terminology for key findings not only hampers cross-sectional comparison and longitudinal tracking but also poses a significant obstacle to large-scale data integration, structured analysis, and the development of intelligent ultrasound-based decision support systems.

In recent years, large language models (LLMs), such as DeepSeek R1, GPT-4o, and Gemini 2.5 Flash, have demonstrated strong capabilities in natural language processing (NLP) and medical information extraction [8–10]. Given the highly variable and unstructured nature of free-text knee ultrasound reports, leveraging LLMs’ strengths in language comprehension and clinical concept recognition—combined with iterative prompt engineering—offers a promising solution to enhance information extraction accuracy and the consistency of structured outputs [11–13]. This approach holds substantial potential for promoting standardization and intelligent transformation of ultrasound reporting and for supporting large-scale medical data platforms.

The primary objective of this study is to systematically evaluate and compare the performance of three state-of-the-art LLMs—DeepSeek R1, Gemini 2.5 Flash, and GPT-4o—in processing free-text knee ultrasound reports. The evaluation focuses on six key dimensions: named entity recognition (NER), structured report generation, ultrasound impression generation, clinical recommendation quality, repeatability, and readability. The selected models represent distinct design paradigms in contemporary LLMs, including reasoning-oriented, efficiency-optimized, and general-purpose architectures. This study focuses on task-level performance under realistic clinical usage scenarios, rather than maximal reasoning capability. The findings aim to inform intelligent ultrasound report analysis and AI-assisted clinical workflows.

Materials and methods

This retrospective study was approved by the ethics committee of The First Affiliated Hospital of USTC (NO.2025-RE-162). Informed consent from study participants was not required since we used anonymized retrospective data.

Study design and data collection

This study evaluated the performance of three large language models (LLMs)—DeepSeek R1 (DeepSeek), Gemini 2.5 Flash (Google), and GPT-4o (OpenAI)—in structuring and interpreting free-text knee ultrasound reports. De-identified knee ultrasound reports were retrospectively collected from the First Affiliated Hospital of USTC between January 2024 and January 2025. Exclusion criteria include: free text contains obvious spelling errors, textual/logical inconsistencies and incomplete descriptions.

The dataset included normal findings and abnormalities such as meniscal tear, effusion, ligament injury, and synovitis. Clinical indications comprised suspected meniscal or ligament injury, joint effusion, and inflammatory conditions. All reports described standard anatomical regions, including the suprapatellar bursa, patellar tendon, quadriceps tendon, medial/lateral meniscus, and collateral ligaments.

The study followed the Developmental Evaluation Guidelines for Applied LLM Benchmarking (DEAL-B), with the completed checklist provided in the Supplementary E1 [14].

Prompt design and experimental setting

All included reports were processed by three large language models (LLMs)—DeepSeek R1 (DeepSeek), Gemini 2.5 Flash (Google), and GPT-4o (OpenAI)—using four prompts designed for clinically relevant tasks: (1) named entity recognition (NER), (2) structured report generation, (3) ultrasound impression generation, and (4) clinical recommendation formulation.

For each report, the free-text section was extracted and entered into each LLM with the four predefined prompts. Each prompt for each report was submitted in a separate, independent chat session to eliminate cross-case context contamination and sequential prompting effects. All inputs (prompts) and outputs (responses) were in Chinese. Detailed wording of the four prompts is provided in Supplementary E2. Model outputs were independently scored by two board-certified ultrasound physicians, blinded to the model identity, using a 5-point Likert scale (1 = very poor, 5 = excellent) [15, 16]. For descriptive and comparative analyses, Likert ratings were aggregated by averaging the two physicians’ scores. Evaluation dimensions included NER accuracy, report structuring accuracy, diagnostic concordance with expert judgment, and appropriateness of clinical recommendations, with detailed scoring criteria presented in Supplementary Tables S1–S4. For statistical analysis, Likert scores were dichotomized to facilitate group comparisons, with scores of 4–5 classified as satisfactory and scores of 1–3 as unsatisfactory, consistent with commonly adopted practices in prior studies [17]. All models were accessed via their publicly available web interfaces. To improve transparency under web-UI constraints, we added a model access log documenting the evaluation window, access method, and session handling, and explicitly noting the unavailability of decoding controls and model identifiers in the UI (Supplementary Table S5).

Prompt 2: readability, and structural completeness

Readability (Prompt 2, subjective). Readability was assessed on structured reports generated under Prompt 2. Two experienced physicians independently rated report clarity, fluency, and coherence using a 5-point Likert scale (Supplementary Table S6).

Structural completeness and stability (Prompt 2, objective). Prompt 2 outputs were evaluated against a predefined structured-report schema with K = 14 required slots (Supplementary Table S7). The Findings section comprised six mandatory sub-sections (soft tissue, joint cavity, tendon, ligament, meniscus, and other). A slot was coded as present (1) if it contained clinically meaningful content or an explicit placeholder (e.g., normal/none/NA/not assessed), and coded as missing (0) only when the slot was entirely absent. Schema compliance was defined as all K slots being present, and missing-item rate was defined as the proportion of missing slots across all cases (missing slots/[N×K]). We quantified Prompt 2 stability using schema compliance, missing-item rate, slot-level exact match, and all-slots identical. With three runs per case, each metric was computed for run pairs (1–2, 1–3, 2–3) and summarized as the pairwise mean across the three pairs (detailed metric definitions and computation procedures are provided in the Supplementary E3). Ninety-five percent confidence intervals were estimated using 1000 case-level bootstraps (Supplementary Table S8).

Although repeatability was originally emphasized for diagnostic impressions (Prompt 3), multiple independent generations had already been obtained for Prompt 2 within the same predefined evaluation window. Leveraging these existing repeated outputs, we further examined the stability of structured report generation under Prompt 2, including schema compliance, missing-item rate, and slot-level exact-match metrics. This analysis does not involve any additional model executions, but instead provides a complementary characterization of structural robustness under real-world web-based usage conditions.

Prompt 3: repeatability and diagnostic performance

Repeatability (subjective and objective). Repeatability was evaluated based on diagnostic conclusions generated under Prompt 3. Each model produced three independent outputs per input. Two physicians rated across-run consistency using a 5-point Likert scale (Supplementary Table S9). In parallel, we quantified objective repeatability by mapping free-text impressions to a predefined multi-label set of binary findings (0/1) using a rule-based lexicon with negation and uncertainty handling (Supplementary Table S10). For each finding, we computed percent agreement and Cohen’s κ for each run pair (1–2, 1–3, 2–3) and summarized three-run repeatability as the pairwise mean across the three run pairs (Supplementary E4.5, E4.6). Ninety-five percent confidence intervals were estimated using 1000 case-level bootstraps.

Diagnostic performance (Prompt 3). Diagnostic performance under Prompt 3 was evaluated against a report-level binary expert reference standard (abnormal vs normal, Supplementary E4.1). Two board-certified musculoskeletal ultrasound physicians independently reviewed the original de-identified free-text reports and assigned reference labels (normal vs abnormal); disagreements were resolved by consensus. For model-generated impressions, two board-certified ultrasound physicians independently assigned predicted binary labels (abnormal vs normal) based solely on the generated impression text using prespecified criteria; disagreements were resolved by consensus (Supplementary E4.2). Using the reference and predicted labels (positive class = abnormal), we categorized each output as true positive (TP), false positive (FP), false negative (FN), and true negative (TN) and computed accuracy, sensitivity (recall), specificity, precision, F1 score, Matthews correlation coefficient (MCC), and balanced accuracy, defined as (sensitivity + specificity)/2, to account for outcome imbalance (Supplementary E4.3, 4.4). To avoid dependence across repeated generations, diagnostic performance was computed using the first run per case, while additional runs were reserved for repeatability analyses. Ninety-five percent confidence intervals (CIs) were estimated using non-parametric report-level bootstrapping (1000 resamples) as detailed in Supplementary E4.7. Inter-rater agreement for reference labeling and output labeling was assessed using Cohen’s κ.

Semantic evaluation of clinical recommendations

Semantic quality of clinical recommendations (Prompt 4) was assessed using five reference-based natural language generation (NLG) metrics: Jaccard index, Bilingual Evaluation Understudy (BLEU), Recall-Oriented Understudy for Gisting Evaluation – Longest common subsequence (ROUGE-L), Character F-score (ChrF), and MoverScore [18–20]. Each model generated three independent outputs per case. For each output, NLG metrics were computed between the generated recommendation and the expert reference. For Chinese NLG metrics, BLEU and ROUGE-L were computed using character-level tokenization (each Chinese character treated as one token), applied consistently to both references and outputs; ChrF was computed at the character/subword level, and MoverScore is an embedding-based semantic similarity metric [21]. Metric estimates were summarized at the case level, and uncertainty was quantified using 95% confidence intervals from 1000 case-level bootstrap resamples (Supplementary E5).

Evaluation metrics and overall performance profiling

To visualize overall model performance, radar plots were constructed based on six core evaluation dimensions: (1) Extraction (NER accuracy, Prompt 1); (2) Structuring (Prompt 2); (3) Diagnostic generation (Prompt 3); (4) Recommendation quality (Prompt 4); (5) Readability (Prompt 2); and (6) Repeatability (Prompt 3). Detailed definitions, scoring criteria, and calculation methods for each dimension are provided in the Supplementary E6.

Statistical analysis

Statistical analyses were performed using R (version 4.3.2). Analyses were conducted by outcome family, with statistical tests selected based on data type and distributional characteristics. Likert-scale outcomes were assessed for normality and homogeneity of variance as appropriate. Normally distributed data were compared across models using one-way analysis of variance (ANOVA) with Tukey’s post-hoc tests, whereas non-normally distributed data were analyzed using the Kruskal–Wallis test followed by pairwise Wilcoxon rank-sum tests (Mann–Whitney U) with Bonferroni adjustment. Binary satisfaction rates derived from dichotomized Likert scores were compared using chi-square tests. Diagnostic classification metrics and reference-based natural language generation (NLG) metrics were treated as continuous variables and compared across models using one-way ANOVA, followed by Tukey’s post-hoc test for pairwise comparisons (two-sided; post-hoc adjusted p values reported where applicable). For NLG metrics, comparisons were performed on per-case metric values, and 95% confidence intervals were estimated using 1000 case-level bootstrap resamples. For Prompt 3 diagnostic metrics, 95% confidence intervals were estimated using non-parametric bootstrapping at the report level (1000 resamples). Multiple comparisons were controlled within each outcome family, and results across fundamentally different outcome families were interpreted descriptively. To assess the consistency of the scoring system, Weighted Cohen’s kappa coefficients were calculated for the Likert-scale ratings provided by the two evaluators across the six evaluation dimensions, with 95% confidence intervals reported. A post hoc power analysis was conducted as a descriptive assessment of sample size adequacy (Supplementary E7). Statistical significance was set at two-sided p < 0.05. A concise mapping of outcomes to statistical tests is provided in Supplementary Table S11.

Results

Patient characteristics

The initial dataset consisted of 417 de-identified free-text knee ultrasound reports. Following the application of exclusion criteria, 359 reports were included in the final analysis (Fig. 1). The cohort of 359 patients comprised 179 males and 180 females, with a median age of 37 years. Among the reports, normal findings were present in 69 reports (0.19), while fluid accumulation (0.50) and multiple concurrent abnormalities (0.28) were the most common abnormalities. Baseline characteristics are presented in Table 1. The cohort was stratified into mutually exclusive categories, including reports with no abnormal findings (normal), reports with a single abnormal finding (e.g., fluid accumulation), and reports with multiple concurrent abnormalities (multimorbidity).

Fig. 1.

Flow diagram of free-text knee ultrasound reports recruitment and inclusion

Table 1.

Demographic and Clinical Characteristics of Patients

Characteristic	Value, n
Sex
Male/Female	179 (0.50)/180 (0.50)
Age (y)	37 (19,56)
Pathology/Normal
Normal	69 (0.19)
Fluid accumulation	179 (0.50)
Ligament injury	3 (0.01)
Fluid accumulation with synovial hyperplasia	3 (0.01)
Meniscus injury	2 (0.01)
Multiple concurrent abnormalities	103 (0.29)

Note. For the “Age” row, values are expressed as median = 37 (Q1 = 19, Q3 = 56). Multiple concurrent abnormalities refers to samples who have two or more diseases

Performance evaluation of LLMs under four Prompt strategies

Across the four prompt strategies (Please refer to the Supplementary E8 for detailed specific examples), GPT-4o achieved the highest mean scores in Prompt 1 (4.0 ± 0.6) and Prompt 3 (3.8 ± 0.7). One-way ANOVA revealed significant group effects for both prompts (F = 8.76 and 10.13; both p < 0.01). Post-hoc Tukey tests showed that GPT-4o performed significantly better than DeepSeek R1 (3.5 ± 0.8) and Gemini 2.5 Flash (3.2 ± 0.7) in Prompt 1 (both Tukey-adjusted p < 0.01), and similarly outperformed DeepSeek R1 (3.2 ± 0.6) and Gemini 2.5 Flash (3.3 ± 0.8) in Prompt 3 (both Tukey-adjusted p < 0.01). In Prompt 2, DeepSeek R1 achieved the highest mean score (4.0 ± 0.5), significantly exceeding GPT-4o (3.6 ± 0.6) and Gemini 2.5 Flash (3.5 ± 0.7) (ANOVA F = 6.95, p < 0.01; Tukey-adjusted p < 0.01 for both comparisons). For Prompt 4, Gemini 2.5 Flash showed the highest mean score (3.9 ± 0.6), but differences among models were not statistically significant (ANOVA F = 1.87, p = 0.27; all Tukey-adjusted p > 0.05). Mean scores (± SD) are summarized in Fig. 2A, while score distributions are illustrated in Fig. 2B(median and interquartile range). Regarding satisfactory response rates (Fig. 2C; scores 4–5 vs 1–3), GPT-4o achieved the highest proportions across prompts, particularly in Prompt 1 (92%) and Prompt 3 (90%). Overall chi-square tests indicated significant between-model differences for Prompts 1 and 3 (both p < 0.01). Pairwise comparisons showed that GPT-4o outperformed DeepSeek R1 (78 and 75%; χ² = 9.47 and 11.68; both p < 0.01) and Gemini 2.5 Flash (65 and 62%; χ² > 15; p < 0.01). In Prompt 2, DeepSeek R1 showed a numerically higher satisfactory rate than GPT-4o (89% vs 87%), but the difference was not significant (χ² = 0.38, p = 0.54). Gemini 2.5 Flash consistently yielded lower satisfactory rates in Prompts 1–3 (χ² > 10; all p < 0.01).

Fig. 2.

Performance comparison of three large language models (LLMs) across four prompt strategies. (A) Heatmap of mean physician Likert scores (1–5) for DeepSeek R1, Gemini 2.5 Flash, and GPT-4o under each prompt. (B) Violin plots showing the distribution of Likert scores across models and prompts (n = 359 reports per model). The inner box denotes the interquartile range (IQR; P25–P75), and the center line indicates the median. Mean ± SD are reported in the text to support ANOVA-based between-model comparisons, whereas medians and IQRs are shown here to visualize score distributions and reduce sensitivity to outliers. (C) Stacked bar plots showing the proportion of satisfactory (scores 4–5) versus unsatisfactory (scores 1–3) responses for each model. Prompts correspond to clinically relevant tasks: prompt 1 = named entity recognition (NER), Prompt 2 = structured report generation, Prompt 3 = ultrasound impression generation, Prompt 4 = clinical recommendation formulation

Evaluation of readability and repeatability in LLM-Based ultrasound reporting

For readability (Fig. 3A), GPT-4o obtained the highest scores (median = 4.0, IQR = 3.6–4.5), followed by DeepSeek R1 (median = 3.5, IQR = 3.0–4.0), whereas Gemini 2.5 Flash showed the lowest and most variable scores (median = 3.0, IQR = 2.4–3.8). Kruskal–Wallis testing indicated significant group differences (H = 14.83, p < 0.01). Post-hoc pairwise Mann–Whitney U tests confirmed that GPT-4o significantly outperformed Gemini 2.5 Flash (Bonferroni-adjusted p < 0.001) and DeepSeek R1 (Bonferroni-adjusted p = 0.012). Objective structuring stability under Prompt 2 is summarized in Table 2. Across three independent runs per case, GPT-4o demonstrated the highest schema compliance (96%, 95% CI: 94–98) and the lowest missing-item rate (7%), followed closely by DeepSeek R1. Gemini 2.5 Flash showed lower compliance and higher missing rates across mandatory Findings sub-sections. Slot-level exact-match analysis further indicated greater run-to-run stability for GPT-4o and DeepSeek R1 compared with Gemini 2.5 Flash.

Fig. 3.

Readability and repeatability of outputs from DeepSeek R1, Gemini 2.5 Flash, and GPT‑4o. (A) Violin plots of readability scores for structured report generation (Prompt 2; Likert 1–5). (B) Violin plots of repeatability scores for diagnostic conclusions (Prompt 3; Likert 1–5), reflecting physician-rated across-run consistency of repeated generations for the same input. Each violin represents n = 359 reports per model. The embedded box indicates the interquartile range (IQR; P25–P75) and the center line denotes the median; violin width represents the kernel-density distribution

Table 2.

Prompt 2 schema compliance and stability

Model	Schema compliance % (95% CI)	Missing-item rate % (95% CI)	Slot-level exact match % (95% CI)	All-slots identical % (95% CI)
GPT-4o	96 (94–98)	7 (5–10)	93 (90–95)	81 (77–85)
DeepSeek R1	93 (90–95)	11 (8–15)	90 (87–93)	76 (71–81)
Gemini 2.5 Flash	91 (88–94)	14 (10–18)	88 (85–91)	72 (67–77)

Notes. Prompt 2 outputs were parsed into a fixed set of required slots (K = 14) defined in Table S7. A slot was coded as present (1) if the model provided either an abnormal description or an explicit normal/NA statement; absence of a required slot was coded as Missing (0) and counted as non-compliance. Metrics are the pairwise mean across run pairs (1–2, 1–3, 2–3). 95% CIs were estimated using 1000 case-level bootstraps

For repeatability (Fig. 3B), GPT-4o (median = 3.4, IQR = 3.3–3.5) and DeepSeek R1 (median = 3.4, IQR = 3.3–3.5) both yielded higher consistency compared with Gemini 2.5 Flash (median = 3.2, IQR = 3.1–3.4). Group differences were again significant (H = 9.62, p = 0.008, Kruskal–Wallis test). Post-hoc Mann–Whitney U analysis showed that both GPT-4o and DeepSeek R1 performed significantly better than Gemini 2.5 Flash (all p < 0.05), while no significant difference was observed between GPT-4o and DeepSeek R1 (p > 0.05). Objective repeatability based on mapped binary diagnostic findings is shown in Table 3. Across individual findings (effusion, meniscal tear, ligament injury, and synovitis), GPT-4o achieved the highest macro-averaged Cohen’s κ (0.73), followed by DeepSeek R1 (0.70), whereas Gemini 2.5 Flash showed lower agreement and κ values. Agreement rates exceeded 88% for GPT-4o and DeepSeek R1 across most findings, indicating stable diagnostic conclusions across repeated generations.

Table 3.

Prompt 3 repeatability using mapped binary findings

Model	Metric	Effusion	Meniscal tear	Ligament injury	Synovitis	Macro-average
GPT-4o	Cohen’s κ (95% CI)	0.78 (0.70–0.85)	0.74 (0.66–0.81)	0.69 (0.58–0.78)	0.72 (0.63–0.80)	0.73 (0.66–0.79)
GPT-4o	Agreement % (95% CI)	92 (89–95)	90 (87–93)	88 (84–91)	89 (86–92)	90 (88–92)
DeepSeek R1	Cohen’s κ (95% CI)	0.75 (0.66–0.82)	0.70 (0.61–0.78)	0.64 (0.52–0.74)	0.69 (0.60–0.77)	0.70 (0.63–0.76)
DeepSeek R1	Agreement % (95% CI)	91 (88–94)	88 (85–92)	86 (82–90)	88 (85–91)	88 (86–90)
Gemini 2.5Flash	Cohen’s κ (95% CI)	0.72 (0.63–0.80)	0.68 (0.59–0.76)	0.60 (0.48–0.71)	0.66 (0.57–0.74)	0.67 (0.60–0.73)
Gemini 2.5Flash	Agreement % (95% CI)	90 (87–93)	87 (84–91)	84 (80–88)	87 (84–90)	87 (85–89)

Notes. Prompt 3 free-text impressions were mapped to binary finding labels (0/1) for each category. Three runs were performed per case. For each model and each finding, repeatability was summarized as the pairwise mean across run pairs (1–2, 1–3, 2–3) for (i) Cohen’s κ and (ii) percent agreement. Macro-averages are unweighted means across findings. 95% confidence intervals were estimated via 1000 case-level bootstraps

US diagnostic performance evaluation

As shown in Table 4, GPT-4o achieved the highest overall performance in ultrasound impression generation. It attained the highest accuracy (0.84), precision (0.95), recall (0.70), specificity (0.96), and F1-score (0.81), and also demonstrated the highest balanced accuracy (0.83) and MCC, indicating robust overall diagnostic performance, substantially outperforming both DeepSeek R1 and Gemini 2.5 Flash. DeepSeek R1 showed moderate performance across metrics (accuracy: 0.66; F1-score: 0.57), with relatively high specificity (0.83) but lower recall (0.48), resulting in a lower MCC and reduced overall discriminative ability. In contrast, Gemini 2.5 Flash demonstrated the lowest scores across all evaluated metrics (accuracy: 0.57; F1-score: 0.45), indicating poor sensitivity (recall: 0.37) and reduced overall diagnostic robustness, and the lowest MCC among the three models.

Table 4.

Ultrasound Diagnostic Performance of LLMs (Prompt 3; abnormal vs normal)

Model	TP	TN	FP	FN	Accuracy (95% CI)	Precision (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)	F1-score (95% CI)	Balanced accuracy (95% CI)	MCC (95% CI)
DeepSeek R1	82	155	32	90	0.66 (0.61–0.71)	0.72 (0.64–0.80)	0.48 (0.40–0.55)	0.83 (0.77–0.88)	0.57 (0.50–0.65)	0.65 (0.59–0.72)	0.33 (0.23–0.42)
GPT-4o	121	180	7	51	0.84 (0.80–0.88)	0.95 (0.91–0.98)	0.70 (0.64–0.77)	0.96 (0.94–0.99)	0.81 (0.75–0.87)	0.83 (0.79–0.88)	0.69 (0.63–0.76)
Gemini 2.5 Flash	63	142	45	109	0.57 (0.52–0.62)	0.58 (0.48–0.67)	0.37 (0.29–0.44)	0.76 (0.70–0.82)	0.45 (0.37–0.53)	0.56 (0.50–0.63)	0.14 (0.04–0.24)

Note. Positive class was defined as abnormal. TP = true positive, TN = true negative, FP = false positive, FN = false negative. Balanced accuracy was defined as (sensitivity + specificity)/2. MCC = Matthews correlation coefficient. Diagnostic metrics were computed using the first run per case (N = 359); additional runs were reserved for repeatability analyses. 95% confidence intervals (CIs) were estimated using non-parametric report-level bootstrapping (1000 resamples)

Clinical recommendation generation evaluation

As shown in Fig. 4A–E, GPT-4o consistently achieved the highest performance across all five evaluation metrics, with mean values of Jaccard 0.55, BLEU 0.61, ROUGE-L 0.69, ChrF 0.66, and MoverScore 0.87 (95% confidence intervals shown in Fig. 4). One-way ANOVA revealed significant differences among the models for each metric (Jaccard: F = 9.82; BLEU: F = 7.64; ROUGE-L: F = 5.47; ChrF: F = 4.93; MoverScore: F = 6.85; all p < 0.01). Tukey post-hoc tests showed that GPT-4o significantly outperformed Gemini 2.5 Flash on all metrics (Jaccard, BLEU, ROUGE-L, and MoverScore: all p < 0.01; ChrF: p = 0.014). DeepSeek R1 showed comparable performance to GPT-4o for BLEU, ROUGE-L, and ChrF (all p > 0.12), while significantly outperforming Gemini 2.5 Flash in Jaccard and MoverScore (both p < 0.01), indicating stronger lexical overlap and semantic similarity in recommendation generation.

Fig. 4.

Performance in clinical recommendation generation (Prompt 4) across five natural language generation (NLG) metrics. (A) Jaccard index. (B) BLEU. (C) ROUGE-L. (D) ChrF. (E) MoverScore. For Chinese NLG metrics, BLEU and ROUGE-L were computed using character-level tokenization (each Chinese character treated as one token), applied consistently to both reference texts and model outputs. ChrF was computed at the character/subword level, and MoverScore is an embedding-based semantic similarity metric. Error bars indicate 95% confidence intervals estimated via 1000 bootstrap resamples

Overall performance comparison across six evaluation dimensions

As shown in Fig. 5A–C, GPT-4o demonstrated the most balanced and highest overall performance across six dimensions: Extraction, Structuring, Diagnosis, Recommendation, Readability, and Repeatability. Across all six evaluation dimensions, inter-rater agreement for the original 5-point Likert ratings reached a consistently substantial level (Weighted Cohen’s κ = 0.68–0.77), supporting the reliability of the expert-based scoring framework used in this study (Supplementary Table S12). In Extraction, GPT-4o achieved the highest score (4.6 ± 0.3), significantly outperforming Gemini 2.5 Flash (4.0 ± 0.4; mean diff = 0.60, p < 0.01, Tukey HSD), and marginally surpassing DeepSeek R1 (4.4 ± 0.3; mean diff = 0.20, p = 0.041). In Diagnosis, GPT-4o (4.7 ± 0.2) outperformed Gemini 2.5 Flash (4.0 ± 0.5; mean diff = 0.70, p < 0.01) and showed no significant difference compared to DeepSeek R1 (4.6 ± 0.2; mean diff = 0.10, p = 0.20). In Recommendation, GPT-4o (4.6 ± 0.2) significantly exceeded Gemini 2.5 Flash (4.1 ± 0.4; mean diff = 0.52, p < 0.05), and was comparable to DeepSeek R1 (4.5 ± 0.3; mean diff = 0.08, p = 0.26). In Readability, GPT-4o (4.5 ± 0.4) and DeepSeek R1 (4.4 ± 0.5) both outperformed Gemini 2.5 Flash (4.0 ± 0.6). The mean difference between GPT-4o and Gemini was 0.49 (p = 0.014), while the difference between GPT-4o and DeepSeek R1 was not significant (mean diff = 0.12, p = 0.48). In Repeatability, both GPT-4o and DeepSeek R1 scored 3.4 ± 0.1, significantly higher than Gemini 2.5 Flash (3.2 ± 0.1; F = 4.65, p = 0.011, ANOVA). Post-hoc comparisons showed a mean difference of 0.18 (GPT-4o vs. Gemini, p = 0.015) and 0.19 (DeepSeek vs. Gemini, p = 0.011), with no difference between GPT-4o and DeepSeek R1 (mean diff = 0.02, p = 0.89). In Structuring, DeepSeek R1 (4.5 ± 0.3) slightly outperformed GPT-4o (4.3 ± 0.4; mean diff = 0.18, p = 0.12), and significantly exceeded Gemini 2.5 Flash (4.0 ± 0.5; mean diff = 0.52, p < 0.01). The difference between GPT-4o and Gemini was also significant (mean diff = 0.34, p < 0.05).

Fig. 5.

Radar plots of model performance across six evaluation dimensions. Radar plots summarize the overall performance profiles of three large language models: (A) DeepSeek R1, (B) Gemini 2.5 Flash, and (C) GPT-4o. The six axes represent extraction, structuring, Diagnosis, recommendation, readability, and repeatability. Values indicate mean physician ratings on a 5-point Likert scale (higher scores denote better performance), based on independent assessments by two experienced physicians using prespecified rubrics. All dimensions were min–max normalized across models for visualization purposes, with equal weighting applied. The radar plots are intended for descriptive comparison rather than statistical inference

Qualitative error analysis

To gain deeper insight into model failure modes, we systematically reviewed all outputs rated as “unsatisfactory” (Likert score ≤ 3). Two researchers independently categorized each case into one or more of the following error types according to predefined criteria (Supplementary Table S13): (1) hallucination, (2) omission of key information, (3) logical or terminological errors, and (4) formatting inconsistencies. Disagreements were resolved through discussion to reach consensus.

Table 5 summarizes model-specific error patterns. DeepSeek R1 errors were dominated by key information omission (32.4%), aligning with lower recall. Gemini 2.5 Flash showed mixed failure modes, with frequent logical or terminological errors (30.2%) and formatting inconsistencies (29.0%). GPT-4o exhibited fewer information omissions (13.4%) and logical errors (16.0%), but more hallucinations (35.4%) and formatting inconsistencies (35.2%), mainly in higher-level generative tasks. These patterns provide a qualitative explanation for the quantitative differences observed across models.

Table 5.

Distribution of qualitative error types across models

Error type	DeepSeek R1, n (%)	Gemini 2.5 Flash, n (%)	GPT-4o, n (%)
Hallucination	210（27.7%）	92（11.7%）	230（35.4%）
Omission of key information	246（32.4%）	229（29.1%）	87（13.4%）
Logical or terminological errors	215（28.3%）	238（30.2%）	104（16.0%）
Formatting inconsistencies	88（11.6%）	228（29.0%）	229（35.2%）
Total unsatisfactory outputs	759（100.0%）	787（100.0%）	650（100.0%）

Note. Qualitative error analysis was performed on all outputs rated as unsatisfactory (Likert score ≤ 3). Errors were categorized according to predefined criteria (Supplementary Table S13). Since a single case may be classified into multiple error types, the sum of cases for each type exceeds the total number of “unsatisfactory” cases

Discussion

This study provides a unified, multi-dimensional comparison of three LLMs (DeepSeek R1, Gemini 2.5 Flash, and GPT-4o) for interpreting Chinese free-text knee ultrasound reports using four task-specific prompts spanning extraction (Prompt 1), structuring (Prompt 2), impression generation (Prompt 3), and recommendation drafting (Prompt 4). Across these tasks, performance was consistently task- and model-dependent, highlighting that model choice and prompt design should be guided by the intended clinical use case rather than a single “best model” assumption.

For extraction (Prompt 1) and impression generation (Prompt 3), GPT-4o showed the strongest alignment with expert references, achieving the highest mean Likert scores (Prompt 1: 4.0 ± 0.6; Prompt 3: 3.8 ± 0.7) and the highest satisfactory rates (Prompt 1: 92%; Prompt 3: 90%). These findings suggest that GPT-4o is relatively better at semantic abstraction and synthesizing report-level impressions from narrative content [22, 23]. In contrast, structuring (Prompt 2) favored DeepSeek R1 (mean 4.0 ± 0.5), indicating that its strengths may lie in information organization and template-constrained generation [24]. For recommendation drafting (Prompt 4), the mean Likert differences were not statistically significant, but objective NLG metrics favored GPT-4o and DeepSeek R1 over Gemini 2.5 Flash, with GPT-4o achieving the highest similarity to expert references (e.g., MoverScore 0.87, ROUGE-L 0.69, BLEU 0.61), supporting better lexical/semantic alignment when generating recommendation text. We note that overlap-/similarity-based NLG metrics quantify agreement with reference text rather than clinical faithfulness, and should therefore be interpreted as auxiliary measures [25].

Subjective readability further differentiated models: GPT-4o achieved the highest readability distribution (median 4.0, IQR 3.6–4.5), exceeding DeepSeek R1 (median 3.5) and Gemini 2.5 Flash (median 3.0). Importantly, we complemented subjective readability with objective Prompt 2 structural completeness and stability across three runs. GPT-4o demonstrated high schema compliance (96%, 95% CI 94–98) and low missing-item rates, followed closely by DeepSeek R1, whereas Gemini 2.5 Flash more frequently omitted mandatory Findings sub-sections. Together, these results indicate that “readable” outputs are not necessarily “complete” outputs, and that structural constraints require dedicated compliance metrics beyond subjective scoring [26].

Because models were accessed via web interfaces without version pinning or visible decoding controls, we evaluated repeatability as stability under realistic interactive use rather than deterministic replication. Subjectively, GPT-4o and DeepSeek R1 showed higher repeatability (both median 3.4) than Gemini 2.5 Flash (median 3.2). Objectively, Prompt 3 impressions were mapped to per-finding binary labels (effusion, meniscal tear, ligament injury, synovitis), and GPT-4o achieved the highest macro-averaged κ (0.73), followed by DeepSeek R1 (0.70). This finding is practically relevant because run-to-run variability can undermine trust even when average performance is acceptable.

For Prompt 3, diagnostic performance was evaluated against an expert-derived report-level reference standard (normal vs abnormal) based on the original ultrasound reports; therefore, results should be interpreted as agreement with expert report references, not outcome-verified diagnostic accuracy [27]. Under this framing, GPT-4o showed the strongest overall classification performance (F1 0.81, balanced accuracy 0.83, high specificity 0.96 with recall 0.70), while DeepSeek R1 exhibited moderate performance with lower recall (0.48), and Gemini 2.5 Flash showed the weakest sensitivity (0.37) and lowest overall robustness.

The radar plot provides an accessible overview of multi-dimensional performance but can obscure trade-offs; therefore, we provide raw metric values and uncertainty estimates (95% CIs where applicable) alongside the aggregate visualization. Inter-rater agreement for 5-point Likert ratings was substantial (weighted κ 0.68–0.77), supporting reliability of the expert scoring framework. Qualitative error analysis further contextualized quantitative differences and highlights that high average scores do not eliminate clinically relevant failure modes, reinforcing the need for clinician review before deployment.

In summary, this work establishes a structured benchmark for evaluating LLMs on free-text knee ultrasound reports, clarifies task- and prompt-dependent model strengths across clinically relevant subtasks, and proposes a reusable, multi-dimensional evaluation framework aligned with recent recommendations for testing and evaluation of clinical LLM applications [28]. At their current stage, LLMs should be positioned as clinician-supervised drafting or decision-support tools, and their outputs should not be used for independent diagnosis. Consistent with prior work on structuring/extracting information from free-text radiology or clinical narratives and real-world clinical triage support evaluations [27, 29, 30], our findings support the feasibility of LLM-assisted reporting while underscoring the need for external validation in Chinese knee ultrasound corpora.

This study has limitations. First, it used Chinese-language knee ultrasound reports from a single center, which may limit generalizability to other institutions, reporting styles, and languages, and performance may degrade under domain shift. Second, models were evaluated via public web interfaces without version pinning or explicit decoding controls; silent updates and stochastic generation may affect reproducibility over time, so our repeatability analyses reflect robustness during the documented evaluation window rather than version-locked replication. Third, we did not perform a formal fairness/bias audit because de-identified reports lacked reliable demographic attributes, and residual annotation variability may still influence subjective ratings despite inter-rater agreement assessment. Finally, overlap-based NLG metrics capture similarity rather than clinical faithfulness or safety, and operational considerations (latency/cost) were not systematically evaluated.

Despite these limitations, this work provides a unified, multi-dimensional framework to compare state-of-the-art LLMs on a real-world ultrasound reporting task, offering a methodological reference for future validation across institutions and languages. Future studies will (i) validate the framework on multi-center, noise-preserved datasets to test robustness under domain shift; (ii) incorporate safety- and factual-consistency–oriented evaluations tailored to clinical text generation; and (iii) prospectively assess LLM-generated drafts in clinical workflows with attention to safety, editability, and accountability. Integration of vision–language models may further enable end-to-end ultrasound interpretation by jointly modeling images and reports.

Conclusion

GPT-4o demonstrated the most robust and expert-aligned performance in interpreting knee ultrasound reports across multiple tasks and prompt strategies, with DeepSeek R1 showing competitive results and Gemini 2.5 Flash performing less favorably. Importantly, diagnostic performance in this study reflects agreement with expert-derived report references rather than validation against downstream clinical outcomes. These findings, derived from Chinese-language knee ultrasound reports, highlight the task-dependent nature of model performance, the importance of prompt design, and the role of ontology-aligned structuring in improving the clinical usability of LLM-generated reports. In practice, such models are best positioned to support report drafting and clinical workflows under appropriate professional oversight.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Abbreviations

LLMs

large language models

NER

named entity recognition

NLG

natural language generation

DEAL-B

Developmental Evaluation Guidelines for Applied LLM Benchmarking

BLEU

Bilingual evaluation understudy

ROUGE-L

Recall-Oriented Understudy for Gisting Evaluation-Longest common subsequence

ChrF

Character F-score

ANOVA

Analysis of variance

MCC

Matthews correlation coefficient

Author contributions

M T and L X: manuscript drafting or manuscript revision for important content, data curation, experimental design, experimental supervision, and thesis modification; LB Z and A M: data collection and formal analysis; XZ Z: experimental design, experimental supervision, data analysis, and thesis modification; C P: critical revision of the manuscript for important intellectual content, methodological and statistical guidance, experimental supervision, and co-corresponding author oversight during the revision process; L H: experimental design, experimental supervision, methodology, manuscript drafting, and manuscript modification. All authors read and approved the final manuscript.

Funding

The authors declare that this study received no external funding.

Data availability

The de-identified knee ultrasound reports used in this study are not publicly available due to institutional and ethical restrictions related to patient privacy. Access to the data may be granted upon reasonable request to the corresponding author, subject to institutional approval and compliance with applicable ethical and regulatory requirements. In addition, representative synthetic or fully de-identified examples of model inputs and outputs are provided in the Supplementary Materials to illustrate the evaluation workflow without exposing protected information.

Declarations

Ethics approval and consent to participate

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. The study was approved by the Bioethics Committee of the First Affiliated Hospital of USTC(NO.2025-RE-162).

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.