A multi-agent large language model framework to automatically assess performance of a clinical AI Triage tool

Abstract

Radiology reports can be used as a surrogate for performance of clinical AI tools. Radiology reports were analyzed by an ensemble of eight open-source LLM models and a internal version of GPT-4o using a single multi-shot prompt that assessed for presence of ICH. Performance of the open-source models, consensus of models and GPT-4o were compared to human report review. Three ideal consensus LLM ensembles were tested for rating the performance of the triage tool. The capability of each LLM varied. The highest AUC performance was achieved with llama3.3:70b and GPT-4o. Using MCC the ideal combination of LLMs were: Full-9 Ensemble, Top-3 Ensemble and consensus. No statistically significant differences were observed between Top-3, Full-9, and consensus. An ensemble of open-source LLMs provides a more consistent and reliable method to derive a ground truth retrospective evaluation of a clinical AI triage tool over a single LLM alone.

Introduction

Many users of clinical artificial intelligence (AI) tools have little objective evidence to determine if the tools are regularly utilized or whether the tools effectively augment clinical care. Not all vendors monitor local performance of their models, and even when they do, it is invariably incomplete. Ongoing monitoring of commercial AI tools is crucial to detect potential drift, as studies indicate that over 90% of AI models can experience temporal performance degradation^1,2. The etiology of AI performance drift is multi-factorial, but it is most often attributed to subtle changes in the input data (e.g., new protocols, image processing, device addition or replacement). Alterations in the local patient population and disease prevalence can also contribute to the performance drift of an AI tool^3,4.

It is widely held that there are disparities in the level of performance of clinical AI tools compared to the reported performance at the time of filing to the FDA. Moreover, pre-validation of a model’s performance on local data during a pilot phase does not guarantee future performance. It has been estimated that 90% of the tools that have been cleared by the FDA demonstrate negative performance drift¹. The monitoring methods that some vendors provide offer incomplete assessment of true performance and focus more on positive results rather than offering a comprehensive assessment that provides transparency to all types of failure modes. Even if a vendor provided a comprehensive analytics platform to each customer, there is a need to have a fully independent and unbiased means to measure the statistical performance against a reference standard and evaluate for internal biases. Recent additions to Health and Human Services (HHS) Section 1557 of the Affordable Care Act which prohibits bias on the basis of race, gender, ethnicity and disability now applies to all clinical decision support tools including those that use artificial intelligence. Providers and health systems that use these tools are responsible for evaluating whether there is implicit bias on their patient populations and implementation of a mitigation plan when bias is detected. All of these factors underscore that health systems need to have full control of their data and an independent method to probe the data for performance drift and bias.

The resources required to assess real-time performance of an AI tool are not trivial, especially when establishing a reference standard requires reviewing electronic medical record (EMR) data, radiology reports, or the original imaging^5,6. The time and expertise required to manually review data retrospectively to obtain ground truth verification of an AI prediction are prohibitive. Requesting radiologists to verify AI results in real-time is often met with resistance, and compliance can be poor. As an alternative the final radiology report has served as a surrogate for ground truth verification. Natural language processing (NLP) techniques have been employed as a means to avoid human expert review of reports; however, they heavily rely on encoding all possible scenarios, making them less effective to variations in reporting style⁷. Large language models (LLMs) have been shown to successfully extract and normalize concepts from heterogeneous free text in radiology reports although performance varies with the model used and the task presented^8–11. A number of LLMs are freely available in the open-source community and can be locally deployed to mitigate patient privacy concerns related to the exposure of protected health information (PHI) to outside commercial entities. An ensemble of LLMs with a majority consensus has been shown to exhibit superior performance over a single LLM for certain tasks¹².

Because of the variability in observed individual LLM performance, the purpose of this study was to determine if an ensemble of open source LLMs instead of a single LLM provides a more reliable and consistent method to obtain ground truth. This methodology is evaluated as a practical application to automate measuring the performance of a commercial intracranial hemorrhage (ICH) AI triage detection tool.

Results

Dataset

The data cohort is summarized in Table 1 consisting of 29,766 non-contrast CT head examinations performed between July 2024 and April 2025, derived from 14 hospitals in two-state health systems. DICOM routing gave preference to emergency studies (26,165), however, 1986 inpatient and 615 outpatient head CTs were also included in the cohort. The cohort consisted of 14,564 males, 15,189 females and 12 of unknown gender. Age ranged from 15 to 100 with a mean age of 62.76 ± 20.09 (95% CI: 62.53, 62.99).

Table 1.

Demographics of the final head CT dataset that underwent inference using a commercial grade intracranial hemorrhage computer vision tool.

Exam type:	Non-contrast head CT
AI Feature being assessed:	Presence/absence of intracranial hemorrhage
Total:	29,766
Gender
Male:	14,564
Female:	15,189
Unknown:	12
Age
Mean ± Std Dev	62.76 ± 20.09
Range (years)	14.9–102 yrs
Setting
Emergency:	26,165
Inpatient:	2986
Outpatient:	615

The Viz.ai ICH AI model detected ICH in 1469 of 28,297 (4.93%) of exams. The number of exams that failed inference was not captured, as this would have required direct access to the Viz.AI vendor database.

Assessing concordance between LLMs

Cohen’s kappa [Fig. 1a] and Jaccard similarity [Fig. 1b] were calculated for each pair of LLMs and separately calculated between each LLM and the consensus vote (four or more in agreement). Overall, we found that Llama3.2:1b had the lowest agreement with the other models. The overall agreement exceeded 0.7 for many of the medium and large-sized models with between eight and seventy billion parameters. GPT-4o alone showed excellent agreement with the ensemble consensus and DeepSeek-r1 alone.

Fig. 1. Overview of the technology stack for the Real-time AI Data Assessment and Reporting (RADAR) LLM automated performance engine.

The vendor ICH detector receives the DICOM images for non-contrast head CT examinations and transmits a simple HL7 message with the boolean results for the presence or absence of hemorrhage. The HL7 result message is processed by a generic MIRTH receiver, which parses the message to add a row in a local MySQL database. Once a day, a script extracts reports from the reporting system using the Powerscribe API, it parses the impression from the report, and passes the impression, combined with a specific prompt, to each of eight LLMs. A consensus is generated when four or more models are in agreement.

Comparing performance of LLM to human reviewed reports

A total of 1726 report impressions were manually reviewed for the presence/absence of acute ICH. Of these reports, 236 (14%) were judged to be either incomplete or ambiguous to make a valid determination. The remaining 1490 that were reviewed were definitive as either positive or negative for ICH and served as a conservative reference standard for assessing LLM performance. Performance characteristics and composite score of the nine large language models and consensus (four or more of eight in agreement) to identify the concept of hemorrhage in the impression of the report was compared to human review (n = 1490) [Table 2]. Llama3.2:1b was the lowest performing model, whereas llama3.3:70b had the highest F1 score (0.81) and recall (0.85). Llama3.3:70b also had greater precision (0.78), specificity (0.72), and MCC (0.57) compared to others. Deepseek-r1 was not a top-performer relative to other models.

Table 2.

Performance characteristics with confidence intervals (95% lower-upper) of the nine large language models and consensus (four or more of eight in agreement) to identify the concept of hemorrhage in the impression of the report compared to human review (n = 1490) including a calculated composite score.

LLM	Accuracy	Precision (PPV)	Recall (Sensitivity)	Specificity	NPV	F1-score	Cohen’s Kappa	MCC	Composite Score
Llama3.2:1b	0.51 (0.48–0.53)	0.55 (0.51–0.58)	0.55 (0.51–0.58)	0.45 (0.42–0.49)	0.46 (0.42–0.49)	0.55 (0.52–0.57)	0.00 (-0.05–0.05)	0.00 (-0.05–0.05)	0.44 (0.41–0.47)
Llama3.2:3b	0.69 (0.67–0.72)	0.70 (0.67–0.73)	0.75 (0.72–0.78)	0.62 (0.58–0.66)	0.67 (0.64–0.71)	0.72 (0.70–0.75)	0.37 (0.32–0.42)	0.37 (0.32–0.42)	0.65 (0.62–0.68)
CodeLlama:7b	0.66 (0.63–0.68)	0.74 (0.71–0.78)	0.57 (0.53–0.60)	0.77 (0.74–0.80)	0.60 (0.56–0.63)	0.64 (0.61–0.67)	0.32 (0.28–0.37)	0.34 (0.29–0.39)	0.61 (0.59–0.64)
Llama3.1:8b	0.69 (0.67–0.72)	0.65 (0.63–0.68)	0.95 (0.94–0.97)	0.38 (0.34–0.42)	0.87 (0.83–0.90)	0.77 (0.75–0.79)	0.35 (0.31–0.40)	0.42 (0.38–0.46)	0.67 (0.65–0.69)
Granite3-dense:2b	0.67 (0.64–0.69)	0.65 (0.62–0.68)	0.86 (0.83–0.88)	0.44 (0.41–0.48)	0.72 (0.68–0.77)	0.74 (0.71–0.76)	0.31 (0.27–0.36)	0.33 (0.29–0.39)	0.63 (0.60–0.65)
Llama3.3:70b	0.79 (0.77–0.81)	0.78 (0.75–0.81)	0.85 (0.82–0.87)	0.72 (0.69–0.75)	0.80 (0.76–0.83)	0.81 (0.79−0.83)	0.57 (0.53−0.61)	0.57 (0.53−0.61)	0.76 (0.74−0.78)
Granite3-dense:8b	0.72 (0.69–0.74)	0.75 (0.72–0.78)	0.72 (0.69–0.75)	0.71 (0.67–0.74)	0.68 (0.64–0.71)	0.73 (0.71−0.76)	0.43 (0.38−0.47)	0.43 (0.38−0.47)	0.68 (0.65−0.70)
DeepSeek-r1	0.67 (0.64–0.69)	0.65 (0.62–0.68)	0.83 (0.81–0.86)	0.47 (0.43–0.50)	0.70 (0.66–0.74)	0.73 (0.71−0.75)	0.31 (0.26−0.35)	0.32 (0.28−0.37)	0.62 (0.60−0.65)
GPT-4o	0.78 (0.75–0.80)	0.83 (0.80–0.85)	0.75 (0.72–0.78)	0.81 (0.78–0.84)	0.73 (0.70–0.76)	0.79 (0.76−0.81)	0.56 (0.51−0.60)	0.56 (0.51−0.60)	0.75 (0.72−0.77)
Consensus	0.65 (0.62–0.67)	0.62 (0.59–0.65)	0.88 (0.86–0.90)	0.37 (0.33–0.40)	0.72 (0.67–0.77)	0.73 (0.71−0.75)	0.26 (0.21−0.30)	0.29 (0.24−0.34)	0.60 (0.58−0.63)

The highest performing model in each category is in bold. Llama3.2:1b was the lowest performing model, whereas Llama3.3:70b had the highest F1 score (0.81) and accuracy (0.79). An intermediate-sized model (Llama3.1:8b) demonstrated the highest recall (0.95) and NPV (0.87). GPT-4o had the highest precision (0.83) and specificity (0.81) compared to others. DeepSeek-r1 was not a top-performer relative to other models. Because precision, recall, and F1-score are interdependent, the composite score places modest additional emphasis on positive class performance.

Receiver-operator characteristics for each large language model and consensus in predicting ground-truth using a human observer review of 1490 report impressions as reference [Table 3]. The lowest discriminating power was observed for Llama3.2:1b (AUC = 0.50). The highest performance was achieved with Llama3.3:70b and GPT-4o (AUC = 0.78). Precision-Recall characteristics for the LLMs and consensus are also shown in Table 3. The average precision was lowest for Llama3.2:1b (AP = 0.55) highest for Llama3.3:70b and GPT-4o (AP = 0.75 & 0.76).

Table 3.

Area under the curve for receiver-operator (ROC AUC) and precision-recall (PR AUC) for each large language model and consensus in predicting ground-truth using a human observer review of 1490 report impressions as reference.

LLM	ROC AUC (95% CI)	PR AUC (95% CI)
Llama3.2:1b	0.501 (0.476–0.526)	0.546 (0.518–0.574)
Llama3.2:3b	0.684 (0.661–0.707)	0.663 (0.634–0.691)
CodeLlama:7b	0.666 (0.641–0.689)	0.657 (0.627–0.685)
Llama3.1:8b	0.668 (0.648–0.689)	0.645 (0.619–0.670)
Granite3-dense:2b	0.651 (0.628–0.673)	0.634 (0.605–0.663)
Llama3.3:70b	0.782 (0.761–0.803)	0.746 (0.719–0.773)
Granite3-dense:8b	0.715 (0.692–0.739)	0.691 (0.663–0.718)
DeepSeek-r1	0.649 (0.625–0.671)	0.633 (0.604–0.662)
GPT4o	0.780 (0.759–0.802)	0.756 (0.728–0.785)
Consensus	0.623 (0.602–0.645)	0.615 (0.586–0.643)

The lowest discriminating power was realized for llama3.2:1b (AUC = 0.50). The highest performance was achieved with llama3.3:70b and GPT-4o (AUC = 0.78). The average precision was lowest for llama3.2:1b (AP = 0.55) highest for llama3.3:70b and GPT-4o (AP = 0.75 & 0.76).

Gauging change in apparent performance of the AI ICH tool by varying configuration of ensembles

To simulate how ensemble configuration might introduce variability into the assessment of ICH AI tools’ performance, we created four different configurations of LLM consensus as a reference standard and used each to evaluate the performance of the Viz.ai ICH model [Table 4]. The configurations tested included: (1) an ensemble of the top three performing LLM (Llama3.3:70b, GPT-4o and Granite3-dense:8b), (2) a consensus of all nine models (including GPT-4o, using five or greater agreement), (3) consensus agreement of the eight LLM agents (four of more) and (4) GPT-4o alone using the entire dataset (n = 29,766). Using all 29,766 cases, we compared AI result performance against four alternate ground truth definitions (Top-3 Ensemble, Full-9 Ensemble, GPT4o, and Consensus) using MCC. Bootstrap resampling (1000 replicates) was used to generate 95% confidence intervals and perform paired comparisons. MCC (95% CI) values were: Full-9 Ensemble 0.571 (0.552–0.591), Top-3 Ensemble 0.558 (0.537–0.579), Consensus 0.556 (0.539–0.574), and GPT4o 0.522 (0.500–0.543). Paired bootstrap testing indicated significantly lower MCC for GPT4o compared with Top-3 (p = 0.026), Full-9 (p < 0.001), and Consensus (p = 0.020), with no significant differences between Top-3, Full-9, and Consensus (p > 0.05).

Table 4.

Variation in calculated performance of the ICH detection model depending on ensemble configuration of LLMs used.

Metric	Top-3 LLM Ensemble	Full-9 LLM Ensemble	GPT-4o	Consensus
Accuracy	0.95	0.95	0.95	0.94
Precision (PPV)	0.62	0.71	0.59	0.74
Recall (Sensitivity)	0.53	0.50	0.51	0.46
Specificity	0.98	0.98	0.98	0.99
F1 Score	0.57	0.59	0.55	0.57
Cohen’s Kappa	0.55	0.56	0.52	0.54
MCC	0.55	0.57	0.52	0.56

An ensemble of the top three performing LLM (Llama3.3:70b, GPT-4o and granite3-dense:8b), a consensus of all nine models (using five or greater agreement), consensus agreement of the eight Ollama LLM (four of more) versus GPT-4o alone using the entire dataset (n = 29,766). Paired bootstrap testing revealed significant differences in MCC for GPT-4o compared with Top-3 Ensemble (p = 0.026), Full-9 Ensemble (p < 0.001), and Consensus (p = 0.020). No statistically significant differences were observed between Top-3, Full-9, and Consensus (p > 0.05).

Discussion

The objective of this study was to determine whether an ensemble of LLM agents could be more effective than a single LLM to verify a finding in the clinical report. The rationale was to use this ensemble to automatically monitor the performance of a commercial ICH detection AI model analogous to a multi-observer consensus process. Retrospective review of the original images is probably the best source of ground truth when verifying the performance of an image-based AI system, however the cost in time and resources makes this impractical for continuous monitoring. Under ideal circumstances, access to multiple visual models could serve to adjudicate a single commercial visual model, but the cost and access for this type of solution is prohibitive.

The radiology report is produced during clinical care, and it is perhaps the highest quality reference that reflects the actual visual findings in the imaging study. Manual retrospective human review of clinical reports is also time-consuming and impractical for continuous monitoring. An automated solution for reviewing reports has been proposed as a reasonable surrogate to generate ground truth data as a means to calculate real-time performance of a clinical AI tool. An LLM can be substituted for human review of the report especially when the identification task is simple and unambiguous. Some commercial AI vendors have incorporated LLM review of reports to provide feedback to customers on the local performance of their AI solution. However, the performance characteristics of the LLM are usually not exposed to the customers, and different LLMs exhibit variable levels of performance depending upon the task¹³. It has been shown that an ensemble of LLM agents serves as a surrogate for multiple observers, providing a means for self-correction^12,14. In this study, an ensemble of LLMs evaluated only the impression section of the report, with the expectation that a group of LLMs would make up for the deficiencies of any single LLM. The task was simplified by focusing on the impression section content alone.

Rohren et al. used a HIPAA-compliant Microsoft Azure hosted version of GPT-4o to review radiologist reports from 332,809 head CT exams from 37 radiology practices and compared the LLM derived results as a means to automate post-deployment performance of a commercial ICH detector¹⁵. A total of 13,569 (4%) exams were labeled positive for ICH by the ICH detection model. Two hundred of the positive exams were randomly selected for manual review of both images and reports to derive performance metrics for the ICH detector, LLM, and radiologists. They found a high diagnostic accuracy in identifying the correct concept of ICH in the reports with only a single false negative and no ambiguous report content. Ambiguity in our cohort was far more prevalent, suggesting that variation in report quality may significantly impact a retrospective review of reports by either human or LLM.

The decision to use open-source models was intentionally aimed at determining if the solution could be operationalized internally without the need to employ a HIPAA-compliant commercial version of an LLM. We compared the performance of our open-source ensemble solution to the HIPAA-compliant version of GPT-4o and found that their performance was comparable or superior.

Nearly fourteen-percent of the exams with discordant results included reports with ambiguous findings on secondary review. Ambiguous language has not been consistently addressed in the literature. Lack of clarity can exist both with the images themselves and the reports depending upon image quality, artifacts, surgical changes, non-acute fluid collections and unrelated findings (e.g., mineralization). Despite the relative simplicity of the task (i.e., hemorrhage present or absent in the impression) there was substantial variability in performance across the collection of LLMs. As expected, the larger models with the most parameters Llama3.1:8b (8 billion parameters), Llama3.3:70b (70 billion parameters), and GPT-4o (1.76 trillion parameters) provided the most consistent results, whereas some of the smaller models Llama3.2:1b and CodeLlama:7b created more disparate results from consensus or the larger models. Moderate performance was realized with smaller models Llama3.2:3b and Granite3-dense:2b suggesting that model design was a performance factor.

Prompt construction is a major contributor to LLM performance^8,10,16–18. For example, our study employed a few-shot prompt, providing examples for the five subtypes of hemorrhage under investigation. The prompt only specified “intracranial” and the five intracranial subtypes of hemorrhage to define the anatomic region specifically and as a result, swelling or hematomas on the scalp were often misclassified as intracranial. This underscores the requirement to perform testing and continuous monitoring of the LLM to identify patterns where the majority of the LLMs fail. Rohren et al. designed their LLM ICH detection prompt using a corpus of reports prior to deployment¹⁵. The use of the ensemble method does provide some capacity to correct for certain types of errors that were more prevalent with a single model and can correct for random misclassifications and biases, yielding a more stable result by consensus similar to any multi-observer study. Interestingly, the larger models were not consistently more reliable than some of the medium sized models.

This prototype for assessment of a single clinical AI solution served as a successful proof-of-concept for real-time monitoring of any clinical AI tool that has dichotomous outputs. The majority of deployed clinical AI tools are triage-based and fit this category (e.g., brain hemorrhage, pulmonary embolism, spine fracture, long bone fracture, cerebral aneurysm detection, stroke, etc). Our goal is to extend this same capability across to clinical AI tools and provide real-time and historical performance characteristics to assess for performance drift. Because performance can vary widely by patient population, acquisition device, and acquisition protocol, the reporting platform for assessment is filterable by location, vendor, and even software version. This provides a mechanism for troubleshooting specific system issues that may lead to degraded performance. In large health systems with disparate and non-uniform acquisition protocols and acquisition devices, having access to this type of data analysis is imperative if a high level of quality control is maintained.

Focusing on discordant exam/report pairs was done deliberately as this seemed to be the optimal approach for assessing true performance of the LLM particularly when the prevalence of a positive exam is relatively low. Random sampling of concordant examples in the dataset (n = 58) was therefore not as productive and seemed to over-estimate true LLM performance in which even the underpowered LLMs performed reasonably well. Focusing on the 1668 discordant exam/report pairs offered a better distribution of textual complexity including ambiguous reports.

Ambiguity in report text is an expected consequence of radiology reporting. There are conditions where expressing uncertainty is appropriate; e.g., limited or degraded imaging study and this type of report was removed because neither the images or the report were interpretable. The vagaries of speech, variations in terminology, incomplete information and expressions of uncertainty cloud the assessment of a report by both human observers and LLM. The 14% ambiguous reports in our discordant group was surprisingly low as the expectation was that report ambiguity was going to account for substantially more disparate results. The 14% also represents a “real-world” representation of ambiguity for this particular clinical use case as it comprises a combined result of three community based practices plus an academic core where the initial draft of a report may be created by a trainee. The rationale for excluding these reports from the analyzed cohort was to optimize the dataset used for LLM verification as including them would potentially introduce negative bias in the performance characteristics of the LLM and the hemorrhage detector. Further enhancements to the prompt engineering could return an uncertainty/ambiguity score that might obviate manual review in the future.

Limitations of this study include the nature of the dataset. ICH has a low prevalence in the general population and this is reflected by the large class imbalance in our data (4.7%) which is somewhat lower than the general estimates in the emergency department population (6.7%)¹⁹. The lower prevalence of hemorrhage inflates specificity and accuracy values. There is an element of incorporation bias as the interpreting radiologist had access to the AI outputs. An evaluation-set bias is also present because we primarily adjudicated exams in which the AI result was discordant with the report. This makes the assumption that concordant exams were inherently correct - both the initial interpretation (radiology report by the radiologist) and the LLM interpretation of the radiologists report. The benefit to this approach is that it provided a conservative estimate of the LLM performance rather than artificially enhancing performance. There is also an element of automation bias as the AI result was presented to the radiologist at the time of interpretation and there was no process in place to evaluate whether the AI result influenced the interpreting radiologists report.

Retrospective review of clinical radiology reports through consensus of an ensemble of open-source (HIPAA-compliant) LLMs can provide a more consistent means for assessing performance of some types of computer vision clinical AI tools compared to a single LLM alone in certain clinical circumstances. This system offers a relatively inexpensive, vendor neutral method for continuous monitoring of AI performance over time and to help identify sources of drift.

Methods

Dataset

This study was exempted by the Jefferson University institutional review board for this HIPAA-compliant retrospective review of data (iRISID-2024-1664). The Real-time AI Data Assessment and Reporting or RADAR was constructed using off-the-shelf components initiated by the receipt of an inference message (HL7 ORU message) from the commercial AI vendor which triggered collection of the final radiology report for each exam in a database for subsequent LLM analysis [Fig. 2]. The data collected focused on prospective collection of results from an ICH detector developed by Viz.AI (San Francisco, CA). Inference was performed on consecutive emergency room and outpatient non-contrast head CTs from fourteen hospitals which included two community practices supported by teleradiology and an academic core. DICOM routing to the AI inference engine was predominately focused on emergency room exams, however some inpatient and outpatient exams were included in the cohort [Table 1 and Fig. 3]. All AI results were made available to the radiologist at the time of dictation and could be used or dismissed at the discretion of the radiologist. The radiologists did not provide any direct feedback on the AI results.

Fig. 2. Data selection inclusion and exclusion criteria for this analysis.

29,825 non-contrast head CT exams run through Viz.AI ICH inference engine over a 14 month period. 45 exams had missing reports or impressions and 14 failed GPT-4o analysis and were removed from the cohort. 29,766 report impressions were rated by all nine LLM. 1726 report impressions (6%) were manually graded and 236 were redacted because of ambiguous or incomplete information. 1490 reviewed reports were used to assess performance of the nine LLMs.

Fig. 3. Agreements between the large language models (LLM) also compared to the consensus score.

Cohen’s kappa (a) and Jaccards similarity index (b) comparing the outputs of the nine large language models, a consensus score (4 or more LLM in agreement) for identifying presence or absence hemorrhage in the impression section of the reports shows agreement ranging from poor to excellent.

Report parsing and LLM ensemble evaluation

The impression text was parsed from the report and combined with a set of multi-shot prompt instructions (Supplement A) requesting a specific response if the concept of acute intracranial hemorrhage was present or absent in the impression text. The prompt specifies that a JSON object be returned with the presence of hemorrhage (Boolean) and if positive, the type of hemorrhage. If positive, the key-value pair is provided as [hemorrhage: < true/false > , subtype: <subarachnoid, intraparenchymal, subdural, epidural, intraventricular hemorrhage > ]. Otherwise, the key-value pair is provided as [hemorrhage: false]. The combined prompt and impression text was processed by local versions of eight LLMs of various sizes and types (Llama3.2:1b, Llama3.2:3b, CodeLlama:7b, Llama3.1:8b, Granite3-dense:2b, Llama3.3:70b, Granite3-dense:8b, and DeepSeek-r1) managed through the Ollama engine API (Meta, San Francisco, CA). Each LLM processed the same report in succession with the temperature hyperparameter set to zero. An additional score, a consensus was derived based upon agreement of four or more models.

An internal, HIPAA-compliant Microsoft Azure (Redmond, WA) hosted version of GPT-4o (OpenAI, San Francisco, CA) was the ninth model evaluated on the same corpus of report impressions and multi-shot prompt as a separate exercise. The results of GPT-4o were matched and compared to each of the original eight LLMs and the consensus of eight.

Assessing agreement across LLM

In order to assess the level of agreement among the LLMs in the identification of the hemorrhage concept in the impression text, Cohen’s kappa and Jaccard similarity²⁰ were employed for each LLM paired against each other, and the LLM ensemble consensus.

Establishing ground truth and LLM performance

A sample of the radiology report impressions was reviewed by two of the authors: a neuroradiologist with over 30 years of experience and a cardiothoracic radiologist with over 15 years of experience. A random sample (n = 58) of result-report pairs were assessed for concordance. All of the discordant result-report pairs (n = 1668), defined as disagreements between the LLM consensus and result from the ICH AI tool were scrutinized primarily to provide a conservative estimate of the LLM performance overall in addition to a small sample of the concordant exam-report pairs. Reports were categorized by the reviewers as absolute positive, absolute negative, incomplete information, or indeterminate (where the concept of hemorrhage was not discernible because of ambiguous descriptions or incomplete information (e.g., no change). Exams with ambiguous or incomplete report impressions for the concept of hemorrhage were excluded from the performance analysis.

Receiver-operator and precision-recall analysis were generated for all nine LLM agents to determine how consistently each LLM predicted the presence of hemorrhage compared to the human-reviewed reports.

Using the human-reviewed reports as ground truth, performance characteristics were calculated for each LLM, yielding recall, specificity, accuracy, precision, Cohen’s kappa, MCC and F1 score. A composite performance score was derived as the arithmetic mean of the eight standard metrics (accuracy, precision, recall, specificity, negative predictive value, F1-score, Cohen’s κ, and MCC). Because precision, recall, and F1-score are interdependent, the composite score places only modest additional emphasis on positive class performance and therefore was not provided as the sole basis of comparison.

Various LLM configurations were evaluated to identify an ideal combination to use as ground truth assessment tool in evaluating the performance of the ICH detection algorithm. This included the consensus of all eight LLM models where four or more were in agreement, a smaller ensemble using the top three performing LLM models, GPT-4o alone, and a consensus of the entire ensemble of nine models where five or more models were in agreement. Paired bootstrap comparison of the Matthews correlation coefficient (MMC) was performed for each scenario to determine if there was a significant difference in performance characteristics between each ensemble.

All of the manuscript text was generated by the authors. GPT-4o and GPT-5 (OpenAI, San Francisco, CA) was used to construct and verify some of the statistical Python code.

Author contributions

A.F. concept, performed data collection and wrote initial draft and prepared the figures. P.L. contributed to data curation and design. R.B., L.P., and P.M. developed the initial study design and data analysis methodology. Y.P., G.S., H.L., and E.C. provided additional methodologic analysis and supplemented references and discussion. All authors reviewed the final submission.

Data Availability

Outputs from the LLMs and the clinical AI tool that formed the basis of the analysis are available for download from the following URL: https://github.com/budflanksteak/RADAR

Code Availability

Python statistical code that formed the basis of the analysis are available for download from the following URL: https://github.com/budflanksteak/RADAR

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s44401-026-00100-4.