ABSTRACT
Septic arthritis is an important cause of morbidity in calves, and radiographic interpretation may vary between observers, especially when structured scoring systems are used by less experienced readers. This single‐center retrospective reader study evaluated whether general‐purpose multimodal large language models (LLMs) could support rubric‐based radiographic scoring of presumptive septic carpal arthritis in calves. Fifty calves aged 0–3 months with clinical findings consistent with presumptive septic carpal arthritis were included, and one carpal radiograph per case was assessed. Two novice veterinary surgeons and three multimodal LLMs, ChatGPT‐5, Gemini‐2.5 Pro, and Claude Sonnet‐4, independently scored each case using a Constant‐based ordinal radiographic scoring framework. Expert consensus served as the operational reference standard. Agreement with the reference was assessed using exact agreement, agreement within one score category, and quadratic weighted kappa. Novice 1 showed the highest concordance, with a mean exact agreement of 55.6%, mean ±1 agreement of 89.8%, and substantial agreement (mean κ w = 0.68; 95% CI, 0.56–0.80). ChatGPT‐5 was the best‐performing model, achieving moderate agreement (mean exact agreement, 53.0%; mean ±1 agreement, 82.6%; mean κ w = 0.58; 95% CI, 0.41–0.72). Claude Sonnet‐4 and Gemini‐2.5 Pro showed slight agreement overall. The best‐performing novice reader remained more reliable than the evaluated models, although ChatGPT‐5 performed comparably to, or better than, one novice evaluator in selected parameters. Selected LLMs may have limited adjunctive value for structured review or training, but not as replacements for human interpretation.
Keywords: artificial intelligence, bovine imaging, musculoskeletal radiology, observer agreement, ordinal scoring
1. Introduction
Artificial intelligence (AI) is increasingly being integrated into human and veterinary medicine to support workflow efficiency, standardization, and diagnostic confidence. In human radiology, AI has been shown to improve the detection of anatomical and pathological abnormalities, in some settings approaching or exceeding novice‐level performance [1]. In veterinary medicine, its use is also expanding, particularly in digital pathology and imaging interpretation [2, 3] Within this broader context, large language models (LLMs) have attracted growing interest because of their ability to combine textual and visual information and generate structured outputs [1].
Although the LLMs evaluated in the present study were not developed specifically for veterinary radiographic interpretation, they are increasingly used in clinical and educational settings as general‐purpose support tools. This raises important questions about their reliability when applied outside their intended domain, particularly in unsupervised real‐world use [1, 3, 4]. In veterinary practice, such systems are not intended to replace expert interpretation, but they may still be used by trainees or early‐career clinicians for preliminary assessment, structured learning, or review. Their performance therefore needs to be evaluated within clearly defined use cases [1, 4].
Septic arthritis is an important cause of morbidity in calves and commonly presents with joint swelling, lameness, fever, and reduced mobility. Although reported prevalence varies according to herd type, case definition, and study population, joint ill/septic arthritis has been reported in approximately 2.47% of crossbred dairy calves in a farm‐based study, and a recent clinical case series described 248 calves treated for septic arthritis, highlighting its continued clinical relevance in calf practice [5, 6]. Infection may occur via hematogenous spread or secondary to umbilical and respiratory disease [7].
The radiographic features of septic arthritis in calves have been described in detail. Constant et al. [8] reported joint‐space widening, articular surface irregularity, and subchondral lysis as common findings, whereas Yurdakul [9] showed that acute cases are more often associated with swelling and joint‐space enlargement, whereas chronic cases may progress to narrowing, periosteal proliferation, and ankylosis. Radiography is therefore important for suspicion and severity assessment, although its sensitivity may be limited in very early disease [8, 9]. Because interpretation may vary between observers, structured scoring systems, such as the Constant et al. [8] scale, may help improve reproducibility, especially outside expert settings.
In the present study, we evaluated three LLMs for rubric‐based radiographic scoring of presumptive septic carpal arthritis in calves and compared their performance with that of novice veterinary surgeons in a standardized multimodal framework using single‐view JPEG radiographs, a Constant‐based scoring rubric, and a brief clinical prompt. To our knowledge, no previous study has compared LLMs with novice readers for single‐view mediolateral (ML) or dorsopalmar (DP) calf carpal radiographs within a defined ordinal scoring system. Our aim was therefore to assess agreement, reproducibility, and concordance with expert‐derived reference scores in order to define the potential role and limitations of LLMs as adjunct readers in bovine musculoskeletal imaging.
2. Materials and Methods
2.1. Study Design
This single‐center, retrospective reader study was conducted at the Atatürk University Faculty of Veterinary Medicine Teaching Hospital, Erzurum. The case accrual spanned January 2022–April 2025. The protocol was approved by the Atatürk University Local Ethics Committee for Animal Experiments (Decision No.: 213; Date: 2025/08), in accordance with institutional and national guidelines for the use of animals in research. Given the retrospective analysis of de‐identified clinical data and images, owner consent was waived by the committee.
Calves aged 0–3 months, regardless of breed or sex, with presumptive septic carpal arthritis were included if ≥2 of the following were documented: joint swelling, lameness, fever, anorexia, leukocytosis, or elevated acute‐phase reactants. Exclusion criteria were non‐carpal joint disease, traumatic carpal fracture/dislocation, motion or processing artifact precluding scoring, or incomplete clinical data/radiographs. The final dataset comprised 50 unique cases (one radiograph per case), selected from the eligible clinical archive on the basis of complete clinical records and diagnostically usable radiographs. No formal a priori sample size calculation was performed because this was an exploratory retrospective reader study. The sample size was determined by the number of eligible cases available during the predefined study period. When both ML and DP projections were available in the clinical archive, one representative projection was selected for standardized scoring on the basis of image availability and diagnostic usability. Paired ML and DP projections from the same calf were not analyzed; therefore, projection‐stratified analyses were performed as exploratory between‐case comparisons rather than direct within‐subject comparisons.
2.2. Image Acquisition and File Handling
Radiographs were acquired using a POSKOM beam‐limiting device (Model PCMAX‐100, maximum tube voltage 100 kV, input 240 V/AC, 1500 W). Exposure parameters followed routine calf protocols: 60–70 kV, 5–8 mAs, and source‐to‐image distance (SID) 80–100 cm. Positioning targeted standard ML or DP views with tight collimation centered on the carpus. Animals were physically restrained without image‐guided adjustments; no post‐acquisition enhancement was performed. Images were exported directly as JPEG files via USB from the radiography system without post‐processing, resizing, or contrast manipulation. Original DICOM metadata were not available after export; all files were de‐identified and renamed using Okur et al. [10] study IDs. Projection labels (ML/DP) were withheld from the raters. However, because radiographic anatomy could have allowed implicit recognition of the imaging view, the procedure was regarded as label‐blinding rather than complete projection blinding. Accordingly, blinding was described as label‐blinding rather than true projection blinding.
2.3. Case Presentation
Each case packet contained three standardized components: (i) the original‐resolution JPEG radiograph, (ii) a Word document embedding the Constant et al. [8] scoring framework for septic arthritis in calves (10 predefined soft‐tissue and osseous parameters on an ordinal scale), and (iii) a standardized clinical prompt summarizing signalment and key presenting signs (swollen/painful joint, lameness, fever, anorexia) consistent with presumptive septic arthritis. Thus, scoring was performed in a standardized multimodal assessment setting that combined radiographic appearance with structured written criteria and limited clinical context, rather than as unrestricted image‐only interpretation.
2.4. Evaluators and AI Models
Scoring was performed by two novice veterinary surgeons, each with 2 years of postgraduate surgical training, and three LLMs: ChatGPT‐5 (OpenAI, August 2025), Gemini‐2.5 Pro (Google, June 2025), and Claude Sonnet‐4 (Anthropic, April 2025). The number of novice readers was determined by the availability of early‐career veterinary surgeons with comparable postgraduate surgical training who could complete the full standardized scoring workflow. All models were evaluated zero‐shot, without additional training or veterinary radiograph exposure, reflecting general reasoning and vision capabilities. The reference scores were established by a two‐member expert panel consisting of a veterinary surgeon and a radiologist with 4 and 10 years of experience, respectively, to combine clinical surgical experience with specialist imaging expertise. Both experts first scored all radiographs independently using the same Constant‐based rubric. Discrepant items were then reviewed jointly, and final reference scores were assigned by consensus. This consensus‐based score set was used as an operational reference standard for agreement analysis.
Each evaluator was allowed up to 2 min per case to standardize case review time across raters and to simulate a constrained first‐pass assessment environment. The purpose of this limit was to ensure comparability of exposure across human and LLM evaluations, rather than to reproduce the workflow of a full specialist radiographic interpretation. The 50 radiographs, comprising 25 ML and 25 DP views, were equally distributed and randomized before presentation, and projection labels were withheld from evaluators.
For each LLM, all cases were evaluated through the commercially available user‐facing chat interface rather than through API or sandbox access. This approach was selected to reflect a structured real‐world use scenario in which veterinary trainees or early‐career clinicians may interact with general‐purpose multimodal LLMs. Each case was opened in a separate cold‐start conversation window, and no feedback was provided during scoring. Where applicable, system memory was disabled, and no prior case information was available to the model. For each model, all 50 case packets were uploaded and scored during a single evaluation session on the same day. The same standardized prompt, radiographic image, clinical summary, and Constant‐based scoring rubric were provided for each case. Model outputs were recorded immediately and transcribed verbatim into the study database. The following standardized prompt was used for every case:
You are a veterinary radiologist. Provided are a carpal radiograph of a calf and the Constant et al. scale designed to systematically score articular radiographs in calves. The calf presents with swollen and painful joints, lameness, fever, and anorexia, findings consistent with a presumptive diagnosis of septic arthritis. Using only the image and these clinical details, assign the radiographic scores according to the scale.
2.5. Scoring and Radiographic Evaluation
The Constant et al. [8] framework comprises 10 parameters assessing soft‐tissue and osseous changes, including swelling, joint‐space height, articular surface irregularity, bone proliferation, perichondral erosion, subchondral lysis/sclerosis, and an “others” category (joint‐space reduction, osteophytes, ankylosis, fracture, luxation, and intra‐articular gas). Two composite indices were calculated: Osseous (0–24) and Overall (0–27).
2.6. Blinding and Response Collection
Raters were blinded to the reference and to each other's assessments. Projection labels (ML/DP) were withheld, although the projection itself may have been inferable from the radiographic appearance. Case packets were pre‐assembled and randomized (fixed seed), and data were double‐checked for completeness before analysis. No feedback was given during scoring. Human raters entered scores in Excel, and LLM outputs were transcribed verbatim by an independent researcher. The final dataset comprised 50 cases scored by five evaluators (2 novices, 3 LLMs) plus the expert reference. Because microbiological confirmation was not available for all cases, the consensus scores were treated as an expert‐derived operational reference for rubric‐based radiographic evaluation rather than as a definitive ground truth.
2.7. Statistical Analysis
All analyses were performed in IBM SPSS Statistics 26.0 (IBM Corp., Armonk, NY, USA) and R 4.3.2 (R Foundation for Statistical Computing, Vienna, Austria). The primary endpoint was exact agreement between each rater and the expert‐consensus reference for each parameter; secondary endpoints were ±1 agreement and quadratic weighted Cohen's kappa (κ_w), interpreted according to Landis and Koch [11]. Agreement patterns across the 10 scoring parameters were summarized using violin plots (Figure 1). Pairwise inter‐rater reliability was assessed using κ_w, exact agreement, and ±1 agreement, whereas overall multi‐rater agreement was described with Fleiss’ κ. Misclassification patterns were examined using confusion matrices (Figure 2; detailed parameter‐ and composite‐score confusion matrices are presented in Figures S1–S10), and case‐level dispersion and scoring deviations across raters were assessed using the range and variance of scores (Figure 3) and visualized with a case‐level deviation heatmap (Figure 4).
FIGURE 1.
Distribution of quadratic weighted κ (κ w) across 10 Constant‐based radiographic scoring parameters for each novice reader and large language model relative to the two‐expert consensus reference standard. Violin plots display the distribution of κ w values for each rater/model. The white dot indicates the median, the thick bar represents the interquartile range, the thin line represents the full range, and the horizontal line denotes the mean κ w. The reference standard was established by consensus between a veterinary surgeon and a radiologist.
FIGURE 2.
Confusion matrices for the overall score comparing each rater/model with the two‐expert consensus reference standard. Each panel shows the distribution of assigned composite scores (x‐axis) versus expert‐consensus reference scores (y‐axis) for all 50 cases. Darker squares indicate a higher number of cases in each cell. Exact agreement is reported for each rater/model. Novice 1 demonstrated the highest exact agreement (22%), followed by Claude Sonnet‐4 (10%) and ChatGPT‐5 (8%), whereas Novice 2 (2%) and Gemini‐2.5 Pro (0%) showed very low exact agreement. The dispersion of points away from the diagonal illustrates systematic misclassification patterns and highlights the difficulty of achieving exact agreement on the composite scoring system.
FIGURE 3.
Case‐level dispersion analysis for the overall score across all novice readers and large language models. Blue bars represent the observed range of assigned scores, and orange bars represent the sample variance across raters/models for each case. Cases are ordered by descending range to highlight cases with greater inter‐rater/model disagreement. Higher range and variance values indicate greater scoring dispersion relative to the structured overall scoring task.
FIGURE 4.
Case‐level deviation heatmap for overall score across all raters. Each cell represents the deviation of a rater's score from the gold standard (rater—gold standard). The diverging color scale is centered at zero, with negative values (underestimation) shown in cooler colors and positive values (overestimation) in warmer colors. This visualization highlights systematic tendencies and misclassification patterns across cases.
Projection‐specific performance was evaluated separately for ML (n = 25) and DP (n = 25) views using the overall and osseous scores. For each rater and projection, exact agreement, ±1 agreement, and κ_w were calculated against the reference. Differences between ML and DP were tested with two‐sided Fisher's exact test, and absolute differences were reported with Newcombe–Wilson 95% confidence intervals. Confidence intervals for κ_w were estimated by nonparametric bootstrap (5000 replicates, fixed seed). All tests were two‐sided, p values were considered descriptive, and no adjustment was made for multiple comparisons. Randomization seeds, analysis scripts, and de‐identified score tables are provided in Supporting Information 2.
3. Results
Novice 1 showed consistently strong performance across parameters. Exact agreement values were 42% (swelling), 40% (joint space), 74% (surface irregularity), 76% (bone proliferation), 58% (perichondral erosion), 62% (subchondral lysis), 78% (sclerosis), 62% (others), 42% (osseous), and 22% (overall). Agreement was predominantly substantial to almost perfect, with moderate agreement observed for swelling and joint space. Overall, Novice 1 showed the highest concordance with the reference. Novice 2 showed more variable performance, with exact agreement values of 50% (swelling), 42% (joint space), 26% (surface irregularity), 10% (bone proliferation), 18% (perichondral erosion), 22% (subchondral lysis), 4% (sclerosis), 18% (others), 2% (osseous), and 2% (overall), demonstrating moderate agreement for selected parameters but fair agreement overall. Direct comparison between the two novice evaluators also showed notable inter‐reader variability. Direct novice‐to‐novice comparison showed a mean exact agreement of 16.4%, mean ±1 agreement of 56.8%, and mean quadratic weighted κ of 0.33 across the 10 scoring parameters, confirming notable inter‐reader variability despite comparable postgraduate experience. Although Novice 1 demonstrated the highest overall concordance with the expert‐consensus reference, Novice 2 showed lower and more variable agreement across several parameters.
ChatGPT‐5 was the strongest performing LLM and, in selected parameters, approached novice‐level performance. Exact agreement values were 46% (swelling), 30% (joint space), 52% (surface irregularity), 80% (bone proliferation), 68% (perichondral erosion), 70% (subchondral lysis), 74% (sclerosis), 64% (others), 38% (osseous), and 8% (overall). Agreement was generally moderate to substantial, with near‐perfect agreement for bone proliferation and comparatively strong performance for osseous.
Claude Sonnet‐4 showed inconsistent performance. Exact agreement values were 44% (swelling), 46% (joint space), 20% (surface irregularity), 52% (bone proliferation), 18% (subchondral lysis), 54% (sclerosis), with comparatively lower exact agreement for the “others,” osseous, and overall scores. Agreement was slight in most parameters, reflecting low overall reliability. Gemini‐2.5 Pro showed the weakest results overall. Exact agreement values were 28% (swelling), 20% (joint space), 16% (surface irregularity), 18% (bone proliferation), and low across the remaining parameters, with mostly slight agreement, including osseous and overall (all detailed analysis tables are available in Supporting Information 2).
Across all parameters, Novice 1 showed the highest overall concordance with the expert‐consensus reference and remained the best‐performing evaluator overall, whereas ChatGPT‐5 followed with moderate agreement. Novice 2 showed fair agreement, whereas Claude Sonnet‐4 and Gemini‐2.5 Pro showed the weakest overall agreement (Table 1). In projection‐stratified analysis, Novice 1 achieved higher exact and ±1 agreement on DP than ML overall, with κ_w increasing from the substantial to the almost perfect range. ChatGPT‐5 also showed higher performance on DP for osseous, with ±1 agreement increasing from 56% to 84% and stronger agreement metrics overall. Other raters showed minimal, nonsignificant differences. Projection‐specific agreement metrics for the overall and osseous scores are presented in Tables 2 and 3, respectively. Selected projection‐specific differences were observed, particularly for Novice 1 and ChatGPT‐5, with DP views showing numerically higher agreement metrics in some analyses. Case‐level deviation heatmaps illustrated the magnitude and direction of scoring deviations from the expert‐consensus reference across individual cases for each rater (Figure 4).
TABLE 1.
Mean agreement metrics across 10 Constant‐based radiographic scoring parameters for each novice reader and large language model relative to the two‐expert consensus reference standard.
| Rater/Model | Mean exact (%) | Mean ±1 (%) | Mean κ_w (95% CI) a | Mean n (cases) | Agreement level b |
|---|---|---|---|---|---|
| Novice 1 | 55.6 | 89.8 | 0.68 (0.56–0.80) | 50.0 | Substantial |
| Novice 2 | 19.4 | 55.8 | 0.37 (0.28–0.46) | 50.0 | Fair |
| Claude Sonnet‐4 | 34.6 | 76.4 | 0.09 (0.05–0.14) | 50.0 | Slight |
| ChatGPT‐5 | 53.0 | 82.6 | 0.58 (0.41–0.72) | 50.0 | Moderate |
| Gemini‐2.5 Pro | 21.0 | 50.2 | 0.11 (0.05–0.17) | 50.0 | Slight |
Note: Values represent mean exact agreement, ±1 agreement, and quadratic weighted Cohen's κ (κ_w) across all 10 parameters for each rater/model.
a κ_w confidence intervals were obtained via 5000 nonparametric bootstrap replicates with a fixed random seed to ensure reproducibility.
bAgreement levels are classified according to Landis and Koch (slight <0.20, fair 0.21–0.40, moderate 0.41–0.60, substantial 0.61–0.80, almost perfect >0.80). The reference standard was established by consensus between a veterinary surgeon and a radiologist after independent review.
TABLE 2.
Projection‐specific exact agreement, ±1 agreement, and quadratic weighted κ (κ w) for the overall score relative to the two‐expert consensus reference standard, stratified by mediolateral (ML) and dorsopalmar (DP) projections.
| Rater/Model | ML exact (%) | DP exact (%) | Δ a Exact (95% CI) | Fisher p b (exact) | ML ±1 (%) | DP ±1 (%) | Δ ±1 (95% CI) | Fisher p b (±1) | ML κ_w c (95% CI) | DP κ_w c (95% CI) |
|---|---|---|---|---|---|---|---|---|---|---|
| Novice 1 | 12.0 | 32.0 | 20.0↓ (0.0–47.4) | 0.171 | 56.0 | 72.0 | 16.0↓ (0.0–48.6) | 0.377 | 0.90 (0.76–0.95) | 0.94 (0.85–0.97) |
| Novice 2 | 4.0 | 0.0 | 4.0↑ (0.0–19.5) | 1.000 | 8.0 | 0.0 | 8.0↑ (0.0–25.0) | 0.490 | 0.39 (0.1–0.58) | 0.46 (0.25–0.60) |
| Claude Sonnet‐4 | 12.0 | 8.0 | 4.0↑ (0.0–27.7) | 1.000 | 20.0 | 20.0 | 0.0↑ (0.0–30.3) | 1.000 | 0.01 (−0.21 to 0.25) | 0.29 (0.06–0.51) |
| ChatGPT‐5 | 4.0 | 12.0 | 8.0↓ (0.0–29.2) | 0.609 | 32.0 | 40.0 | 8.0↓ (0.0–42.1) | 0.769 | 0.67 (0.40–0.81) | 0.85 (0.55–0.92) |
| Gemini‐2.5 Pro | 0.0 | 0.0 | 0.0↑ (0.0–13.3) | 1.000 | 0.0 | 4.0 | 4.0↓ (0.0–19.5) | 1.000 | 0.14 (−0.06 to 0.35) | 0.09 (−0.02 to 0.22) |
Δ = Absolute difference between ML and DP; ↑ = better ML, ↓ = better DP. Exact and ±1 differences are accompanied by Newcombe–Wilson 95% CIs.
Fisher p is two‐sided.
κ_w CIs were obtained via nonparametric bootstrap with 5000 replicates (fixed seed).
TABLE 3.
Projection‐specific exact agreement, ±1 agreement, and quadratic weighted κ (κ w) for the osseous score relative to the two‐expert consensus reference standard, stratified by mediolateral (ML) and dorsopalmar (DP) projections.
| Rater/Model | ML exact (%) | DP exact (%) | Δ a Exact (95% CI) | Fisher p b (Exact) | ML ±1 (%) | DP ±1 (%) | Δ ±1 (95% CI) | Fisher p b (± 1) | ML κ_w c (95% CI) | DP κ_w c (95% CI) |
|---|---|---|---|---|---|---|---|---|---|---|
| Novice 1 | 48.0 | 36.0 | 12.0↑ (0.0–46.3) | 0.567 | 80.0 | 84.0 | 4.0↓ (0.0–32.7) | 1.000 | 0.94 (0.80–0.97) | 0.91 (0.71–0.96) |
| Novice 2 | 0.0 | 4.0 | 4.0↓ (0.0–19.5) | 1.000 | 4.0 | 8.0 | 4.0↓ (0.0–24.3) | 1.000 | 0.39 (0.20–0.56) | 0.40 (0.21–0.59) |
| Claude Sonnet‐4 | 16.0 | 12.0 | 4.0↑ (0.0–30.5) | 1.000 | 36.0 | 40.0 | 4.0↓ (0.0–39.0) | 1.000 | 0.01 (−0.26 to 0.37) | 0.13 (−0.14 to 0.47) |
| ChatGPT‐5 | 40.0 | 36.0 | 4.0↑ (0.0–39.0) | 1.000 | 56.0 | 84.0 | 28.0↓ (0.0–56.5) | 0.062 | 0.75 (0.42–0.88) | 0.87 (0.64–0.95) |
| Gemini‐2.5 Pro | 12.0 | 4.0 | 8.0↑ (0.0–29.2) | 0.609 | 24.0 | 12.0 | 12.0↑ (0.0–39.3) | 0.463 | 0.16 (−0.05 to 0.39) | 0.12 (−0.04 to 0.30) |
Δ = Absolute difference between ML and DP; ↑ = better ML, ↓ = better DP. Exact and ±1 differences are accompanied by Newcombe–Wilson 95% CIs.
Fisher p is two‐sided.
κ_w CIs were obtained via nonparametric bootstrap with 5000 replicates (fixed seed).
4. Discussion
In the present study, the best‐performing novice reader showed closer agreement with the expert‐consensus reference than the evaluated general‐purpose LLMs. However, novice‐reader performance was not uniform. Novice 2 showed lower and more variable agreement, and ChatGPT‐5 performed comparably to, or better than, this novice evaluator in selected parameters. Overall performance was strongest for Novice 1, whereas ChatGPT‐5 was the best‐performing model among the LLMs; however, no model matched the consistency of the best‐performing novice reader across all parameters. These findings indicate that, within this structured radiographic scoring task, general‐purpose LLMs may provide partial support but do not yet perform at a level that would justify their use as stand‐alone readers.
Among the evaluated models, ChatGPT‐5 showed the most favorable agreement profile and, for some conspicuous osseous features, approached novice‐level performance. This pattern is important because it suggests that LLM performance was not uniformly poor, but rather depended on the type of lesion being scored. In our dataset, agreement was higher for more overt structural abnormalities such as bone proliferation, subchondral lysis, sclerosis, and the osseous composite score, whereas softer or potentially earlier radiographic changes, including swelling and joint‐space abnormalities, were assessed less consistently. Importantly, the comparison with novice readers should be interpreted at the individual‐rater level. Although Novice 1 remained the strongest overall evaluator, ChatGPT‐5 performed comparably to, or better than, Novice 2 in selected parameters. This finding is consistent with the broader radiographic literature showing that observer agreement may vary across readers, lesion types, and scoring tasks, particularly when ordinal or structured radiographic assessments are used [12, 13]. Under structured rubric‐based conditions, selected LLMs may, therefore, provide performance similar to some novice interpreters for conspicuous osseous abnormalities, while still remaining unreliable as stand‐alone readers. This interpretation is in line with the radiographic literature indicating that septic arthritis in calves is associated with changes such as joint‐space alteration, articular irregularity, and subchondral lysis, but that observer reproducibility is not uniform across lesion types and disease stages [8, 9]. It is also consistent with broader imaging literature showing that reader and model agreement tends to be higher for visually conspicuous abnormalities than for subtle or early‐stage findings [14, 15].
Exact agreement should also be interpreted with caution in this setting. To our knowledge, there is no universally accepted clinically meaningful threshold for adequate exact agreement between novice and expert readers for Constant‐based radiographic scoring of septic carpal arthritis in calves. Exact agreement is a stringent metric, because even a one‐category difference on an ordinal scale is counted as disagreement. Therefore, in the present study, exact agreement was interpreted together with ±1 agreement and quadratic weighted κ, which provide additional information on near‐category agreement and the magnitude of ordinal disagreement. This combined interpretation is particularly important for structured radiographic scoring tasks, in which adjacent‐category discrepancies may not carry the same practical meaning as larger scoring errors.
The difference between ChatGPT‐5 and the other evaluated LLMs should, therefore, be interpreted cautiously. The better performance of ChatGPT‐5 does not imply diagnostic equivalence to trained readers; rather, it likely reflects better consistency in mapping conspicuous radiographic features onto a predefined ordinal rubric. This distinction is important because the present study assessed agreement with an expert‐derived scoring framework, not unrestricted diagnostic interpretation. In that sense, the observed performance may relate as much to rubric adherence and structured multimodal reasoning as to raw image interpretation itself. This is compatible with recent reviews emphasizing that multimodal LLM outputs in medical imaging are highly task‐dependent and influenced by prompt structure, image quality, and the specificity of the requested output format [1, 4].
Our findings are also broadly consistent with previous veterinary AI literature suggesting that AI performance in radiographic interpretation remains variable, particularly in heterogeneous clinical material and outside task‐specific training conditions [2, 13, 15]. At the same time, our results appear more conservative than reports describing stronger AI or multimodal model performance in other medical imaging contexts [16, 17]. This difference is defensible and likely reflects major differences in study design rather than a contradiction. In the present study, the evaluated systems were general‐purpose, zero‐shot LLMs applied to single‐view JPEG calf carpal radiographs and required to assign ordinal scores within a veterinary‐specific framework. This represents a narrow and demanding task, and performance in such a setting should not be expected to mirror results obtained in other species, imaging modalities, anatomical regions, or task‐specific AI pipelines.
Projection‐stratified analyses showed numerically higher agreement for DP views in selected comparisons, particularly for Novice 1 in overall scoring and for ChatGPT‐5 in osseous scoring. However, these differences were inconsistent across raters and endpoints, and the corresponding confidence intervals were wide. These findings should, therefore, be considered exploratory and interpreted in light of the projection‐selection limitations described below.
Several limitations should be considered when interpreting these findings. First, the study included only 50 cases from a single center. Second, the reference standard was based on expert consensus rather than an independent biological gold standard; therefore, the results reflect agreement with expert‐defined rubric scores rather than absolute diagnostic truth. Because the reference standard was generated by two experts, this may limit the stability and generalizability of the reference classification, although discrepant items were resolved through consensus review. Accordingly, the reference standard represents the outcome of the consensus process rather than independent expert ratings. Third, not all cases had microbiological confirmation, and the analysis therefore concerns presumptive septic arthritis rather than uniformly confirmed infection. Fourth, the human comparator group consisted of only two novice surgeons, which limits generalization both across different experience levels and within the same novice experience level, given the observed variability between the two novice evaluators. Fifth, only one radiographic projection was analyzed per case, and paired ML and DP radiographs from the same calf were not evaluated. Projection selection was based on clinical availability and diagnostic usability rather than random or paired sampling. Consequently, the projection‐stratified analyses represent between‐case comparisons, may have been affected by selection bias, and cannot establish that either projection provides superior diagnostic or model performance. Sixth, the evaluated LLMs were not trained specifically on veterinary radiographs, and the analysis was performed using single‐view JPEG images within a structured prompt‐and‐rubric environment. In addition, LLM evaluation was performed through user‐facing chat interfaces rather than through API‐ or sandbox‐based access. Therefore, model parameters could not be fully controlled, and exact reproducibility of model behavior cannot be guaranteed. Repeated evaluations for intraobserver variability were not performed, and control assessments using text‐only or image‐only inputs were not included. Consequently, the present findings should be interpreted as exploratory results from a structured real‐world chat‐interface workflow rather than as validation of a fully controlled diagnostic AI system.
In conclusion, the present results show that the best‐performing novice reader showed the highest overall concordance with the expert‐consensus reference for rubric‐based radiographic scoring of presumptive septic carpal arthritis in calves. ChatGPT‐5 showed the most promising performance among the tested models, particularly for conspicuous osseous changes, and performed comparably to, or better than, one novice evaluator in selected parameters. However, ChatGPT‐5 remained less consistent than the best‐performing novice reader and less dependable for subtle findings and composite scores. These findings support a cautious view in which selected LLMs may have value as adjunctive tools for training or structured review, but not as replacements for human interpretation in this setting.
Author Contributions
Tuğçe Kartal: software, investigation, writing – review and editing. Ayşe Gölgeli Bedir: methodology, writing – original draft, writing – review and editing. Ferda Turgut: data curation, resources. Sıtkıcan Okur: formal analysis, supervision, writing – review and editing. Esra Modoğlu: conceptualization, writing – review and editing, data curation. Ömer Tarık Orhun: conceptualization, visualization, writing – review and editing. Uğur Ersöz: resources, investigation. Büşra Kibar: visualization, validation. Latif Emrah Yanmaz: formal analysis, supervision, writing – review and editing. Yasemin Akçora: writing – review and editing, methodology, investigation. Yakup Kocaman: conceptualization, validation. Mümin Gökhan Şenocak: formal analysis, resources, writing – review and editing.
Funding
The authors have nothing to report.
Disclosure
This retrospective observational veterinary study was prepared in accordance with the STROBE‐VET guidelines, where applicable.
Conflicts of Interest
The authors declare no conflicts of interest.
Supporting information
Supporting File 1: vru70252‐sup‐0001‐figureS1‐S10.docx
Supporting File 2: vru70252‐sup‐0002‐tables.docx
Acknowledgments
ChatGPT‐5 (OpenAI), Gemini‐2.5 Pro (Google), and Claude Sonnet‐4 (Anthropic) were used to evaluate the radiographic scoring of 50 calf cases with presumptive septic arthritis according to the Constant et al. [2] system. All outputs generated by these models were critically reviewed, verified, and approved by the authors.
Data Availability Statement
The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy and ethical restrictions.
References
- 1. Lundervold A. S. and Lundervold A., “An Overview of Deep Learning in Medical Imaging Focusing on MRI,” Zeitschrift Fur Medizinische Physik 29, no. 2 (2019): 102–127, 10.1016/j.zemedi.2018.11.002. [DOI] [PubMed] [Google Scholar]
- 2. Pacholec C., Flatland B., Xie H., and Zimmerman K., “Harnessing Artificial Intelligence for Enhanced Veterinary Diagnostics: A Look to Quality Assurance, Part I Model Development,” Veterinary Clinical Pathology 54 (2025): S30–S42, 10.1111/vcp.13401. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Pereira A. I., Franco‐Gonçalo P., Leite P., et al., “Artificial Intelligence in Veterinary Imaging: An Overview,” Veterinary Science 10, no. 5 (2023): 320, 10.3390/vetsci10050320. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Bradshaw T. J., Tie X., Warner J., Hu J., Li Q., and Li X., “Large Language Models and Large Multimodal Models in Medical Imaging: A Primer for Physicians,” Journal of Nuclear Medicine 66, no. 2 (2025): 173–182, 10.2967/jnumed.124.268072. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Orhun Ö. T., Kocaman Y., Okur S., et al., “Comparison of the Effect of Two Different Joint Lavage Techniques on Survival Rate in Calves With Septic Arthritis: 248 Cases,” Large Animal Review 29, no. 4 (2023): 181–185. [Google Scholar]
- 6. Jalal M. S., Dutta A., Islam K. M. F., Sultana J., Sohel M. S., and Ahad A., “A Study on the Prevalence and Etiology of Joint Ill in Calves of Cross‐Breed Dairy Cattle in Six Dairy Farms of Bangladesh,” Research Journal for Veterinary Practitioners 4, no. 4 (2017): 66–70, 10.14737/journal.rjvp/2016/4.4.66.70. [DOI] [Google Scholar]
- 7. Weaver A. D., Atkinson O., Jean G. St., and Steiner A., Bovine Surgery and Lameness, 3rd ed. (Wiley, 2018). [Google Scholar]
- 8. Constant C., Masseau I., Babkine M., et al., “Radiographic Study of Haematogenous Septic Arthritis in Dairy Calves,” Veterinary and Comparative Orthopaedics and Traumatology 31, no. 4 (2018): 252–260, 10.1055/s-0038-1641732. [DOI] [PubMed] [Google Scholar]
- 9. Yurdakul İ., “Evaluation of Clinical, Radiological, Ultrasonographic and Microbiological Findings of Septic Arthritis in 50 Calves,” Revista Mexicana de Ciencias Pecuarias 10, no. 1 (2019): 254–266, 10.22319/rmcp.v10i1.4727. [DOI] [Google Scholar]
- 10. Okur S., Modoğlu E., Baykal B., et al., “Comparison of Diagnostic Performance Between Large Language Models and Veterinary Evaluators in Feline Ocular Diseases Based on Clinical Summaries and Anterior Segment Photographs,” Veterinary Ophthalmology 29, no. 1 (2026): e70052, 10.1111/vop.70052. [DOI] [PubMed] [Google Scholar]
- 11. Landis J. R. and Koch G. G., “The Measurement of Observer Agreement for Categorical Data,” Biometrics 33 (1977): 159–174, 10.2307/2529310. [DOI] [PubMed] [Google Scholar]
- 12. Argue B. J. and Ahern B. J., “Evaluation of Agreement for Radiographic Lesions and Risk for Racing in Thoroughbred Yearling Sale Repository Radiographs,” Frontiers in Veterinary Science 11 (2024): 1430993, 10.3389/fvets.2024.1430993. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Ball E., Uhlhorn M., Eksell P., Olsson U., Ohlsson Å., and Low M., “Repeatability of Radiographic Assessments for Feline Hip Dysplasia Suggest Consensus Scores in Radiology Are More Uncertain Than Commonly Assumed,” Scientific Reports 12 (2022): 13916, 10.1038/s41598-022-18364-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Ndiaye Y. S., Cramton P., Chernev C., Ockenfels A., and Schwarz T., “Comparison of Radiological Interpretation Made by Veterinary Radiologists and State‐of‐the‐Art Commercial AI Software for Canine and Feline Radiographic Studies,” Frontiers in Veterinary Science 12 (2025): 1502790, 10.3389/fvets.2025.1502790. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Norman B., Pedoia V., Noworolski A., Link T. M., and Majumdar S., “Applying Densely Connected Convolutional Neural Networks for Staging Osteoarthritis Severity From Plain Radiographs,” Journal of Digital Imaging 32, no. 3 (2019): 471–477, 10.1007/s10278-018-0098-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Agbareia R., Omar M., Soffer S., Glicksberg B. S., Nadkarni G. N., and Klang E., “Visual‐Textual Integration in LLMs for Medical Diagnosis: A Preliminary Quantitative Analysis,” Computational and Structural Biotechnology Journal 27 (2025): 184–189, 10.1016/j.csbj.2024.12.019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Kim S. H., Wihl J., Schramm S., et al., “Human‐AI Collaboration in Large Language Model‐Assisted Brain MRI Differential Diagnosis: A Usability Study,” European Radiology 35, no. 9 (2025): 5252–5263, 10.1007/s00330-025-11484-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supporting File 1: vru70252‐sup‐0001‐figureS1‐S10.docx
Supporting File 2: vru70252‐sup‐0002‐tables.docx
Data Availability Statement
The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy and ethical restrictions.