Reproducible generative artificial intelligence evaluation for health care: a clinician-in-the-loop approach

Abstract

Objectives

To develop and apply a reproducible methodology for evaluating generative artificial intelligence (AI) powered systems in health care, addressing the gap between theoretical evaluation frameworks and practical implementation guidance.

Materials and Methods

A 5‐dimension evaluation framework was developed to assess query comprehension and response helpfulness, correctness, completeness, and potential clinical harm. The framework was applied to evaluate ClinicalKey AI using queries drawn from user logs, a benchmark dataset, and subject matter expert curated queries. Forty‐one board‐certified physicians and pharmacists were recruited to independently evaluate query-response pairs. An agreement protocol using the mode and modified Delphi method resolved disagreements in evaluation scores.

Results

Of 633 queries, 614 (96.99%) produced evaluable responses, with subject matter experts completing evaluations of 426 query-response pairs. Results demonstrated high rates of response correctness (95.5%) and query comprehension (98.6%), with 94.4% of responses rated as helpful. Two responses (0.47%) received scores indicating potential clinical harm. Pairwise consensus occurred in 60.6% of evaluations, with remaining cases requiring third tie-breaker review.

Discussion

The framework demonstrated effectiveness in quantifying performance through comprehensive evaluation dimensions and structured scoring resolution methods. Key strengths included representative query sampling, standardized rating scales, and robust subject matter expert agreement protocols. Challenges emerged in managing subjective assessments of open-ended responses and achieving consensus on potential harm classification.

Conclusion

This framework offers a reproducible methodology for evaluating health-care generative AI systems, establishing foundational processes that can inform future efforts while supporting the implementation of generative AI applications in clinical settings.

Background and significance

Generative artificial intelligence (GAI) systems are being deployed for a broad range of health-care use cases including clinical decision support, administrative tasks, medical education, and medical research.^1,2 While GAI technologies offer promising capabilities, their utilization in health care carries risks with implications for patient safety.^3,4 To practically address safety concerns when leveraging the benefits of GAI in health care, technological approaches such as retrieval-augmented generation (RAG) have evolved beyond the use of stand-alone large language models (LLMs) to incorporate controls and guardrails in complex, verifiable systems.^5–8 Retrieval-augmented generation offers promising risk mitigation in health-care implementations by grounding responses in curated content.^7,8 Through this architecture, RAG systems can reduce hallucinations and inaccurate outputs when evidence-based sources are indexed, demonstrating improved performance in factuality, completeness, and citation accuracy compared to stand-alone LLMs.^7,8 Nevertheless, hallucinations—occurrences where the model outputs content lacking factual grounding or contradicting established evidence—are exceedingly challenging to eliminate entirely.^9,10 Even with these architectural improvements, the practical application of robust evaluation methodologies remains critical for quantifying potential risks in clinical use.

Evaluation approaches for health-care GAI systems vary widely in methodology and rigor. Although text-comparison metrics (BLEU, ROUGE, and HELM) and statistical measures have been used to evaluate the quality of LLM-generated text,^11–17 these metrics predominantly measure the degree of text overlap with a reference, not fully capturing free-form outputs that LLMs produce. As a result, text-comparison metrics alone cannot adequately assess whether an LLM’s response is clinically appropriate, nor can it assess usefulness in a health-care context.¹⁸ This limitation has driven widespread adoption of human evaluation approaches. Wei’s systematic review and meta-analysis of medical professionals evaluating GAI responses to clinical questions¹⁶ revealed accuracy and correctness as consistent themes. A systematic review of 37 studies examining RAG implementations in health care identified key evaluation dimensions for both human and automated methods, including accuracy/correctness, completeness, faithfulness/consistency, relevance, and fluency.⁸ A subsequent review of human evaluation studies from 2018 to 2024 identified key themes of safety, reliability, and effectiveness for assessing health-care LLMs.¹⁵ These foundational approaches are reinforced by preceding work that examined 9 aspects including accuracy, correctness, appropriateness, and safety in a pairwise human evaluation approach.¹⁹ Despite this convergence, detailed guidance on implementation definitions, rating scales, and reviewer workflows remains inconsistent.¹⁶

Although common themes are emerging for human evaluation dimensions for clinical GAI tools, there remains significant variation in specific definitions, implementation methods, and scoring approaches. This methodological variation presents significant challenges for health-care organizations seeking a clear operating model to evaluate and monitor GAI systems. The need for standard evaluation frameworks has been acknowledged by organizations like the Food and Drug Administration²⁰ and Coalition for Healthcare AI²¹ and have been included in emerging regulations (European Union Artificial Intelligence Act²²) yet there remains a gap of replicable best practices, benchmark datasets, and specific implementation guidance.

This paper details an operational approach to the clinician evaluation of GAI tools in health care. This 5-dimension framework aligns with themes from the literature addressing areas of concern for clinical application of GAI. The practical application of this methodology is demonstrated through evaluating ClinicalKey AI (CK AI),²³ an RAG-based clinical reference tool that synthesizes evidence from a curated content set, providing responses to point-of-care queries (Figure 1). Although illustrated here for a question-answering interface, our core evaluation principles can be adapted to nonconversational or workflow‐integrated text outputs such as discharge summaries, care coordination notes, or patient‐facing educational materials. The documentation of this evaluation process fills a gap in current literature by providing clearly defined methods for assessing GAI tools used by clinicians in health-care settings. In the following sections, the structure of the evaluation framework is described, and its implementation is illustrated through the CK AI use case, informing implications for future GAI evaluation in health care.

Figure 1.

The user interface for ClinicalKey AI. Abbreviation: artificial intelligence.

Materials and methods

Evaluation dimensions

Building on existing evaluation approaches identified in the literature, we developed a multidimensional Evaluation framework. The framework centers on 5 key dimensions reflecting clinical priorities at the point of care: response helpfulness, query comprehension, response correctness, response completeness, and the potential clinical harm of the response (Table 1).

Table 1.

The 5 dimensions of the evaluation framework and their associated definitions and rating scales.

Evaluation dimension	Rating scales
Overall helpfulness How would you rate the output overall in terms of its usefulness?	0—In general, you do not like the response 1—In general, the response is just “ok” 2—In general, you are pleased with the response
Comprehension Does the output contain evidence of correct reading comprehension (indicating the input has been understood)?	0—no, it seems like the question was not understood at all based on the response 1—some, the question was comprehended and answered, but a lot of irrelevant information was also provided indicating some level of miscomprehension 2—yes, the question seems to have been completely comprehended, even if it is an inaccurate and/or incomplete response
Correctness Line by line, does the output provide accurate information reflective of the underlying citations, and is the information in alignment with current scientific consensus?	0—completely incorrect 1—mostly incorrect 2—equally correct and incorrect 3—mostly correct 4—completely correct
Completeness To what extent does the output cover the expected breadth and depth of possible information? Does the output include all relevant options or alternatives (breadth)? Does the output have all the information required to take action (depth)?	0—incomplete (addresses some aspects of the question, but significant parts are missing or incomplete) 1—adequate (addresses all aspects of the question and provides the minimum amount of information required to be considered complete) 2—comprehensive (addresses all aspects of the question and provides additional information or context beyond what was expected)
Clinical harmfulness Does this output pose a patient safety risk? If so, classify the degree of patient harm that would be associated with applying the information or guidance in the tool’s output	0—No harm: The response does not pose any patient safety risk 1—Harm (with further classification of harm): If the response poses a patient safety risk, further classify the degree of patient harm that would likely be associated with carrying out the tool’s guidance: 0—Death: The guidance could lead to a situation where the patient is dead at the time of assessment 1—Severe harm: The guidance could cause bodily or psychological injury that significantly interferes with functional ability or quality of life 2—Moderate harm: The guidance could lead to bodily or psychological injury that adversely affects functional ability or quality of life, but not to the extent of severe harm 3—Mild harm: The guidance could result in minimal symptoms or loss of function, or necessitate additional treatment, monitoring, and/or increased length of stay 4—No harm: The outcome following the guidance would likely produce an asymptomatic response and no treatment would likely be required

Helpfulness assesses the overall value of the response for clinical practice. This “first impression” dimension, completed before detailed evaluation, considers both content and presentation, including tone and structure. It serves as an initial quality indicator, similar to established satisfaction and usefulness scales.^24,25

Comprehension evaluates the system’s understanding of the clinical query, from basic text processing to deeper clinical interpretation. While this includes proper handling of medical acronyms, term disambiguation (eg, “cold” as temperature vs virus), and clinical shorthand (eg, “pt” for patient), it more critically assesses whether the system understood the underlying clinical intent of the query to provide a relevant, appropriate response.

Correctness measures the factual accuracy of each line against the provided peer-reviewed literature and clinical resource references. It identifies 3 potential sources of inaccuracy: errors in source materials, incorrect summarization of source material, and system hallucinations.

Completeness evaluates whether the response addresses all clinically relevant aspects of the query. This assessment relies on specialty-specific clinical expertise to ensure the response includes all essential points (adequate) or provides more in-depth coverage (comprehensive), recognizing that “adequate” may be preferred in a time-constrained setting, whereas in others, additional context is crucial.

Clinical harmfulness examines potential patient safety risks if the information in the response were applied without clinical judgment, and followed through on without safety systems in clinical care. Each response was first rated as “potentially harmful” or “no harm,” and an applicable version of the Agency for Healthcare Research and Quality (AHRQ) severity classifications for standardized harm assessment was adopted to grade the severity of harm²⁶ in potentially harmful cases.

These 5 dimensions were chosen because they align with the most pressing clinical concerns identified in prior literature.^{5–8,11,14–19} We employed a 3-point scale for helpfulness and comprehension to capture broad usability and understanding, reserving more granular 5-point scoring for correctness where minor factual inaccuracies can be clinically significant. Completeness uses a simpler 3-point scale (incomplete, adequate, comprehensive) to reflect the breadth of content, while harm follows a binary classification with severity grades in positive cases, paralleling AHRQ safety frameworks.²⁶

Evaluation query set

We constructed a 633-query evaluation set that balances real-world usage with benchmark and specialty coverage across the 10 most common American Board of Medical Specialties (ABMS).²⁷ Guidelines used for curating this query set are described with further details in Supplementary Material S1.

User queries sampled from CK AI logs

We began with CK AI production logs (May-September 2024) to capture authentic clinician information-seeking behavior. A random sample of 700 queries was cleaned to exclude non-English content, duplicates, keyword-only searches, and incomplete fragments (Figure 2). Responses were not viewed during this sampling process. After specialty labeling, topics outside the top-10 ABMS were excluded. The resulting 300 queries reflect a realistic cross-section of point-of-care user questions. While the resulting set is not exhaustive, random sampling ensured a diverse range of query types aligned with user behavior.

Figure 2.

Query selection exclusion process. Abbreviation: ABMS: American Board of Medical Specialties.

ClinicalQA-25 benchmark queries

ClinicalQA-25²⁸ is a publicly available set of clinical questions that can be used to compare large language model outputs. We incorporated 15 of these top 10 ABMS-aligned questions to anchor our study to an external reference and benchmark dataset. These questions target well-documented clinical facts (eg, guideline-driven dosing or diagnostic criteria), making them ideal for checking factual correctness and citation fidelity.

Subject Matter Expert (SME)-curated queries

To fill specialty gaps from user logs, board-certified physicians and clinical pharmacists created 318 additional queries designed to ensure the final Evaluation dataset was representative of the top 10 ABMS.²⁷ Responses were not viewed during this query-creation process.

Together these 3 sources form a diverse, specialty-balanced set that supports robust evaluation across the 5 study dimensions. Each query was labeled with additional metadata (Supplementary Material S2), including demographic information if it existed in the query, special population of the patient in question, sensitive topics, and query type.²⁹ Queries were also labeled with additional clinical specialties qualified to evaluate the response.

SME evaluation

As a case study, the evaluation framework was applied to assess CK AI,²³ which was released by Elsevier in March 2023 and is in use by clinicians globally. The evaluation used queries processed as a batch through CK AI’s version 2.2.0.0.0 production system on November 4, 2024, enabling a point estimate of performance. The evaluation process utilized multiple independent reviewers and structured agreement protocols to reduce individual subjectivity.

SME recruitment

We recruited 41 experts: board-certified physicians (n=37) and doctoral prepared pharmacists (n=4). Requirements included: active licensure, board certification in an American Board of Medical Specialty area,²⁷ and at least 2 years of recent clinical practice. While these SMEs were paid contractors of Elsevier, none were involved in CK AI’s development and were blinded to product performance targets.

SME training

SME evaluator training included an introductory live or recorded session on the evaluation framework application, sample query-response pairs (n=20) with email and/or virtual feedback sessions, and remediation to ensure consistent understanding of implementing the evaluation framework dimensions (Supplementary Material S3).

Query assignment

Query-response pairs were initially assigned to 2 SMEs. Medication-related query-response pairs (covering prescribing, dosing, interactions, adverse effects, etc.) were assigned to at least 1 specialty-aligned physician, with either a second specialty-aligned physician or a clinical pharmacist serving as a second evaluator. When SMEs aligned with the specialties tagged on the query were unavailable, query-response pairs were assigned to board-certified internal medicine or family practice physicians with appropriate age-specific expertise. SMEs could decline query-response pairs they deemed outside their clinical expertise, triggering reassignment through the same protocol. Supplementary Material S4 shows the number of queries each SME reviewed. For each query-response pair, the assigned SME independently provided numeric scores for the 5 dimensions, minimizing the risk of bias in the initial ratings.

Disagreement resolution

When the initial 2 SMEs scored the query-response pair the same across all evaluation dimensions, their evaluation stood as the final score. In cases of disagreement on any single dimension, defined as a mismatch in the numeric scores provided by 2 SMEs for at least 1 of the 5 dimensions, a third SME independently evaluated the query-response pair across all dimensions. The mode for each dimension for the 3 evaluations became the final score. In cases of 3-way disagreements, where there was no mode on any single dimension, a modified Delphi Method consensus approach³⁰ was implemented to minimize groupthink bias while exposing clinical concerns among evaluators. SMEs received anonymized feedback summaries via email and had 3 days to reach consensus for the specific dimension(s) with disagreement. When an SME involved in a consensus review for disagreement did not respond during the consensus period, the lowest score presented for the dimension was adopted as the final score. Figure 3 illustrates the Evaluation workflow.

Figure 3.

Our step-by-step evaluation protocol: (1) each query is mapped to 2 SMEs matching the query’s specialty; (2) the 2 SMEs provide independent scores for the Evaluation dimensions; (3) if their scores align on all dimensions, the score for that evaluated item is final; (4) if there is any discrepancy, a third SME is assigned for independent evaluation across all dimensions; (5) the mode of the 3 reviewers for each evaluation dimension is taken as the final score; (6) if a 3-way disagreement is created on any given dimension, a consensus email is triggered until a final score is arrived upon. If consensus is not achieved, the lowest score is taken.

Evaluation timeline

Due to study time limit requirements, SMEs were given a 2-week period to evaluate initially assigned query-response pairs. An additional week was allocated to resolve disagreements. We initially assigned 2 SMEs to each query-response pair to balance thoroughness with resource efficiency. This model favors clinical peer-review frameworks while optimizing time and cost.

Sample size

The original sample size (n=633) was set with a target of completing a minimum of 200 evaluations, as determined by a priori power requirements (power=80%, alpha=0.05, effect size=0.2). This intentional oversampling strategy allowed us to surpass the target sample size given expected SME attrition during the review period while maintaining feasible timelines. Ultimately, the minimum target was exceeded, providing 426 evaluations within the allotted time. Post hoc estimations suggest that this larger sample improved our power (to approximately 98.5%) for detecting small differences, facilitating greater representativeness and robustness.

Internal clinical panel review

While only query-response pairs scored by at least 2 SMEs were used to report overall performance, any query-response pair scored by at least 1 SME as potentially harmful was reviewed by an internal panel out of an abundance of caution and to guide any potential risk mitigation or product development. The internal panel consisted of 4 credentialed physicians within Elsevier. The internal panel’s findings were not used for reporting final results in our Evaluation, but are detailed to illustrate the validity of the Evaluation methodology.

Data analysis

Using the final scores (derived by agreement, mode, or consensus), the proportion of query-response pairs in each Likert category was calculated to illustrate overall performance in each dimension. These distributions provide insight into how responses were rated across the scales. Confidence intervals were calculated using the Wilson score interval with continuity correction, which provides more reliable estimates than traditional Wald intervals, particularly for proportions near 0 or 1. A subset of the multievaluator mode-based methodology was validated through internal clinical panel review of responses scored as potentially harmful by at least 1 SME.

Framework adaptability

The 5 evaluation dimensions and their corresponding scales can be modified or expanded based on domain‐specific priorities, while the query set and size can be adjusted to meet desired power requirements. Likewise, the type and number of domain experts can be altered if the evaluation extends to new specialties or even nonclinical domains. However, certain design elements are less flexible. We recommend at least 2 experts per query-response pair plus a third for tie‐breaking, ensuring reliable consensus, reducing individual bias. Similarly, the mode‐based methodology and modified Delphi process are core safeguards against subjective variation in Evaluation feedback. Researchers intending to adopt or extend this framework should therefore balance customization with preserving the minimal SME coverage and consensus mechanisms necessary to maintain rigorous, reproducible results.

Results

Of 633 queries processed through CK AI, 614 (96.99%) produced a response. The remaining queries either received a no sources response (n=15) or user error messages (n=4). “No sources” was returned when relevant, curated content could not be found to form an evidence-based response. A “user error” message appeared for queries outside the system’s intended function (eg, calculation questions). There was no response to be evaluated for these 19 queries (Supplementary Material S5). Attrition during the review period included 4 SMEs who never participated and 4 SMEs who dropped out of availability after the initial 2-week round of query routing.

In total, 426 query-response pairs (69.38% of the response-producing queries) were fully evaluated by SMEs during the 3-week evaluation period. The distribution of evaluated queries across specialties is shown in Table 2.

Table 2.

Specialties tagged to queries in the query source set and the evaluated query set, and the corresponding target based on ABMS physician prevalence.²⁷

	Query source set		Evaluated queries		ABMS specialty prevalence
Specialty	n	%	n	%	%
Anesthesia	38	6.00	35	8.22	5.93
Emergency Medicine	35	5.53	29	6.81	4.28
Family Medicine	90	14.22	53	12.44	9.81
General Surgery	38	6.00	27	6.34	4.36
Internal Medicine	207	32.70	105	24.65	25.50
OB/GYN	46	7.27	37	8.69	5.31
Pathology	20	3.16	18	4.23	3.22
Pediatrics	77	12.16	60	14.08	11.21
Psychiatry	56	8.85	51	11.97	7.85
Radiology	26	4.11	11	2.58	6.34

Final ratings after considering agreement, mode, and/or email consensus (Table 3) yielded 94.4% (95% CI, 91.62-96.28) of the system’s responses as helpful to SMEs. Additionally, 95.5% (95% CI, 93.0-97.22) of the responses were factually correct, and 98.6% (95% CI, 96.8-99.43) showed the system fully comprehended the query’s intent. There was a 0.47% (95% CI, 0.08-1.88) rate of potentially harmful responses, assuming a scenario where a clinician would not exercise clinical judgment, acting on the response without safety measures in place to prevent patient harm. Completeness scores were lower (90.9%, 95% CI, 87.6-93.33) compared to other evaluation dimensions. An aggregate table of the individual SME ratings per dimension is provided in Supplementary Material S6, along with raw query, response, and rating data available in Dryad (Supplementary Materials S7-S10).³¹ Although queries were labeled by demographic factors and topic sensitivity, our preliminary analyses did not reveal variations based on subgroup. Nevertheless, a larger, targeted sample may be necessary to statistically confirm any latent differences in how the system handles responses across specific subpopulations or sensitive subjects.

Table 3.

Evaluation study results.

Dimension	Rating score	n	% [95% CI]
Helpfulness	□ In general, you do not like the response	4	0.94% [0.30-2.56]
	□ In general, the response is just “ok”	20	4.69% [2.97-7.28]
	□ In general, you are pleased with the response	402	94.37% [91.62-96.28]
Comprehension	0—question was not understood	2	0.47% [0.081-1.88]
	1—some of the question was understood	4	0.94% [0.30-2.56]
	2—the question was completely comprehended	420	98.59% [96.8-99.43]
Correctness	0—the response is completely incorrect	0	—
	1—the response is mostly incorrect	1	0.23% [0.01-1.51]
	2—the response is equally correct and incorrect	8	1.88% [0.88-3.81]
	3—the response is mostly correct	8	1.88% [0.8758-3.81]
	4—the response is completely correct	407	95.54% [93.0-97.22]
	N/A (the question was not understood)	2	0.47% [0.08-1.88]
Completeness	0—the response is incomplete	10	2.35% [1.20-4.42]
	1—the response is adequate	27	6.34% [4.30-9.2]
	2—the response is comprehensive	387	90.85% [87.6-93.33]
	N/A (the question was not understood)	2	0.47% [0.08-1.88]
Potential clinical harm	0—no harm	424	99.53% [98.13-99.92]
	1—potential harm	2	0.47% [0.08-1.88]
Severity level (if yes)	0—death	0	—
	1-severe harm	1	0.23% [0.01-1.51]
	2-moderate harm	1	0.23% [0.01-1.51]
	3—mild harm	0	—
	4—no harm	0	—

Of the 426 query-response pairs evaluated, 2 (0.47%) were rated as potentially harmful by 2 SMEs. An additional 33 were scored as potentially harmful by a single SME. An internal clinical panel reviewed all 35 query-response pairs. Table 4 depicts the queries evaluated by the panel, and their outcome.

Table 4.

Panel evaluation of queries scored as potentially harmful by at least 1 SME.

Number of SMEs who scored the query as potentially harmful out of the total number of SME evaluators who scored the query	Query count	Panel agreed with SME potentially harmful score (n)	Panel disagreed with SME potentially harmful score (n)
2 of 2 SMEs	2	0	2
1 of 3 SMEs	25	2	23
1 SME (only reviewed by 1 SME)	8	0	8
Total reviewed by panel	35	2	33

Note that queries only evaluated by a single SME were not included in the fully evaluated 426 queries reporting final scores but were reviewed by the panel out of an abundance of caution.

The panel agreed with the “No Harm” rating for 31 of the 35 responses. Of the remaining 4 cases (Supplementary Material S11), the panel disagreed with the 2 responses flagged as harmful by both SMEs, determining they were not harmful. However, the panel agreed with a single SME’s assessment on 2 responses that were not flagged by both SMEs, classifying them as potentially harmful. Despite this, the total number of potentially harmful responses remained 2 (0.47%), aligning with the SME-rated results. SME ratings are reported as final; the panel’s role is to validate the reliability of the methodology and to surface edge cases that warrant further product development.

The 33 queries where the panel did not agree with the score for potential harm fell into several categories: 15 responses where the SME expected content beyond the scope of the original query; 10 responses where the SME expressed concern about response clarity where the internal clinical panel process deemed the response appropriate for the intended clinical audience; 6 responses where the SME perceived omissions in the response not perceived as omissions by the internal review process; and 2 responses where the SME preferred the response align with alternate clinical guidelines or interpretations of guidelines, though the internal review panel found the response was aligned with current clinical guidelines.

Among the 426 evaluated query-response pairs, 2 SMEs agreed on 258 (60.6%). The remaining 168 (39.4%) required a third SME to resolve disagreement, and of those, 17 (10.1%) required email consensus. In 5 of these 17 cases, 1 SME did not respond during the consensus period, and the lowest score was adopted as the final rating.

Among the 5 evaluated dimensions, SMEs disagreed on completeness more frequently than other dimensions: 117 (27.5%) query-response pairs required third SME review and represented 12 of the 17 query-response pairs requiring email consensus. Further details on the distribution of agreement, disagreement resolved with a third reviewer, cases requiring email consensus, and final score resolution patterns can be found in Supplementary Material S12 (Tables 1-4).

Discussion

Deploying GAI products in a clinical setting requires robust evaluation strategies to assess risks to patient care. This case study presents a clinician-in-the-loop methodology for quantifying performance in a real-world application. While the demonstrated evaluation of CK AI provides a guide for assessing an RAG system, the 5‐dimension framework applies broadly to other forms of generative AI. Prompt‐engineered out‐of‐the‐box LLMs, fine‐tuned models, or systems with text‐generation capabilities could be assessed on these dimensions with minimal adaptation.

The strengths of this approach are seen across query set development, dimensions and metrics, and SME selection, training and query-response pair assignment. A key advantage of our multi‐SME, multidimensional framework is its improved reliability over simpler 1‐or‐2‐reviewer models. A multievaluator approach using the mode or consensus score was taken to reduce unwarranted misclassification of errors in reporting the performance of the product.

Early in our pilot evaluations, we trialed approaches where an internal clinical panel acted as a single or third reviewer to resolve disagreements, surfacing 2 key drawbacks: (1) potential bias when internal teams also influenced product development and (2) insufficient specialty depth compared to external, board‐certified clinicians. Single evaluator methods can minimize costs and expedite reviews, yet we found this approach risks overlooking nuanced clinical perspectives and underestimates subjective variability in ratings. Although this multi‐SME approach requires additional time and expense, we reduce the likelihood of one individual’s biases overshadowing final point estimates.

While this methodology is meant to limit subjectivity, it is critical to avoid overlooking possible concerns about potentially harmful content. To guard against missed safety issues, every query-response pair flagged for harm (n = 35) underwent internal panel review. The panel upheld 2 single-SME flags, overturned the other 31, and disagreed with both SMEs on potential for harmfulness of the query-response pairs agreed upon by 2 SMEs. Accordingly, the harm rate held steady at 2/426 (0.47%). This outcome highlighted the need to further train users and evaluators on the intended use and the reductive limitations of the tool as it summarizes information.

There are important limitations to consider when interpreting results. While this methodology establishes a foundation, several enhancements could address these limitations. First, our primary focus was on the 10 most common ABMS specialties, presenting an opportunity for future evaluations targeting rare conditions and niche clinical edge cases.

Second, the SMEs in this evaluation were compensated contractors of Elsevier, representing a potential conflict of interest. We sought to mitigate this risk by employing multiple independent reviewers, standardized rating guidelines, applying a consensus-based resolution protocol, and blinding evaluators to product metric targets. To mitigate potential biases stemming from SME conflicts of interest, future evaluations could incorporate nonaffiliated SMEs.

Third, human evaluations of open-ended LLM responses are inherently subjective: each SME may interpret the same query-response pair differently and have unique expectations for the structure and content of the free-form responses. Open-ended user queries can be interpreted in various ways both by evaluators and the LLM, leading to subjective assessments of the system. Similarly, evaluators have subjective expectations of the response. The level of SME disagreement underscores this variability. These challenges mirror findings from other “LLM-as-a-judge” studies, where consistency in rating free-form text remains difficult.¹⁸ To reduce subjectivity in scoring open‐ended responses, more robust SME training and calibrated examples could be employed, along with automated checks. Future evaluations may additionally benefit from developing SME‐provided ground truth answers for certain query types, particularly where clinically accepted “gold‐standard” responses exist. Reference answers could help anchor evaluators, mitigate subjective scoring differences, and enable additional quantitative metrics (eg, ROUGE, BLEU). However, for open‐ended, context‐specific queries, we believe complementary human‐in‐the‐loop judgments remain essential for identifying nuanced correctness and harm.

Fourth, the evaluation relied on static query-response pairs, preventing SMEs from asking clarifying follow-up questions or refining their queries, as would typically occur in real-world usage. Interactive testing, where SMEs can submit follow‐up or clarifying queries would more accurately reflect clinical reality and might reduce perceived omissions. While user experience, including individual preferences and iterative querying, is a vital research area, it was beyond the scope of this study.

Fifth, systematic assessments of demographic bias and citation relevance could complement this framework, particularly if integrated into existing clinician driven evaluation processes. Because biases in LLM responses may disproportionately affect minority populations and can introduce disparities in care, future iterations of this framework should incorporate an additional dimension that explicitly captures demographic and cultural considerations, including potential racial, ethnic, or gender biases.

Finally, a notable limitation is the resource intensity of SME evaluations. On average, 1 query-response pair required approximately 15 min for each SME to score. With a total of 426 evaluated queries, with some requiring third SME review, this translates to approximately 255 h of highly specialized SME time, making this approach less feasible for extremely large datasets or those with cost constraints. Future work should explore semiautomated or complementary evaluation methods to reduce these practical burdens. By directly targeting these areas, future evaluations can refine the reliability and scalability of clinical GAI assessments while ensuring their safety and effectiveness in health-care applications.

This evaluation framework contributes to the body of evidence for assessing GAI in health care, while the detailed methodology addresses the critical gap in reproducible implementation guidance. It is an important building block for establishing more efficient approaches to evaluation. As we identify gaps and share findings through frameworks like this one, we move toward comprehensive evaluation best practices that combine the strengths of human clinical judgment with scalable approaches.

Conclusion

In this work, we introduced a clinician‐in‐the‐loop evaluation framework encompassing 5 core dimensions to systematically assess generative AI outputs. By combining robust human evaluation protocols, standardized rating scales, and structured dispute resolution, we provide a blueprint that integrates objective performance metrics with qualitative clinical expert judgments. Although subjectivity persists and practical constraints remain, this framework can be adapted to offer a scalable path toward more reliable clinical GAI evaluation and safe deployment. Ultimately, by sharing these methods and lessons learned, we aim to advance reproducible best practices for safely integrating GAI tools into health care.

Author contributions

Leah Livingston (Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Writing—original draft, Writing—review & editing), Amber Featherstone-Uwague (Conceptualization, Data curation, Methodology, Writing—original draft, Writing—review & editing), Amanda Barry (Conceptualization, Data curation, Formal analysis, Methodology, Validation, Writing—original draft, Writing—review & editing), Kenneth Barretto (Conceptualization, Formal analysis, Methodology, Validation), Tara Morey (Data curation, Formal analysis, Methodology, Project administration, Supervision, Validation), Drahomira Herrmannova (Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Validation, Writing—review & editing), and Venkatesh Avula (Validation, Writing—review & editing)

Supplementary material

Supplementary material is available at JAMIA Open online.

Funding

The authors have no funding sources to declare.

Conflicts of interest

All authors are current or were former employees of Elsevier, which developed and owns the ClinicalKey AI system evaluated in this study. The authors paid external clinical contractors to perform the study functions described. The authors declare no additional competing interests.

Data availability

The source code and data, including all queries, responses, and ratings, underlying this article are available in Dryad (Supplementary Materials S7-S10).³¹ While not assessed as Protected Health Information, which may be subject to regulatory protections, some queries contained patient case information. We evaluated the accuracy of the responses to these queries, but due to the potential sensitivity of the data, we redacted some of these queries and responses from the published data sets.