Making medical AI benchmarks clinically interpretable: the case of mental health

Abstract

Medical artificial intelligence (AI) benchmarks are increasingly used to assess the readiness of large language models for health-related tasks, but aggregate performance scores can obscure clinically meaningful variation across domains. HealthBench, an open benchmark of 5000 multi-turn health conversations, represents a salient example: the score is difficult to interpret for decisions about specific clinical use cases, including those with mental health needs. Conversations involving suicidality, psychosis, eating disorders, trauma or relational dependence require recognising ambiguous disclosures, responding safely under uncertainty, avoiding reinforcement of harmful beliefs or over-reliance and knowing when to escalate to human-based crisis support. We propose that HealthBench and similar benchmarks report domain-specific results. To demonstrate feasibility and utility, we adopted OpenAI’s classified dataset to identify 332 mental health conversations, validated against human review (sensitivity 0.980, specificity 1.000, positive predictive value 1.000, negative predictive value 0.980, F1 0.990, inter-rater agreement 99.0%). We analysed mental health domain performance separately from overall HealthBench performance using a stratified random sample of 500 conversations. Across deployments, mental health point estimates differed from overall estimates by −0.034 to +0.047. Mental health domain composition was also uneven: for example, postpartum depression represented 26.8% of mental health conversations compared with 4.5% for suicidal crisis, 0.9% for psychosis and 1.5% for eating disorders. The findings suggest benchmark averages can mask domain-specific insights relevant to clinical safety. We argue that interpretable medical AI evaluation requires transparent domain composition; subgroup and failure mode reporting; calibration against clinically meaningful anchors, including human expert performance; and reproducible versioned analyses as models, grader systems and deployment contexts evolve.

Introduction

In May 2025, OpenAI released HealthBench,¹ an open benchmark designed to evaluate large language models in realistic health conversations. HealthBench includes 5000 multi-turn conversations between an artificial intelligence (AI) chatbot and either an individual user or healthcare professional, with responses graded against conversation-specific, clinician-authored rubrics.² HealthBench and its brethren (HealthBench Hard, HealthBench Consensus and HealthBench Professional) have shifted evaluation towards open-ended clinical communication, uncertainty, emergency referral and response quality.

Despite its contribution, HealthBench underscores a growing problem in medical AI evaluation: health is too broad a category to be evaluated through the lens of a single performance metric.³ Although useful for tracking overall model progress, an aggregate score is less useful for determining whether a model exhibits sound reasoning in specific clinical domains or populations.⁴ We would not evaluate radiology and psychiatry using a single undifferentiated clinical score nor assume that performance on X-ray interpretation predicts performance on suicide risk assessment.

Mental health as a test case for clinically interpretable benchmarking

For four reasons, mental health is an especially important and illustrative case. First, emergency pathways differ. For most physical health emergencies, escalation entails urgent care referral, emergency department evaluation or contacting emergency medical services. For mental health emergencies, escalation may require assessing imminent self-harm risk or violence risk, encouraging connection to crisis supports or trusted individuals, reducing access to lethal means and avoiding language that could intensify despair, paranoia or shame.

Second, mental health conversations often involve significant ambiguity.⁵ A user may not say, “I am suicidal,” but instead, “I can’t do this anymore.” Early psychosis may present as a request for help interpreting perceived surveillance or hidden messages; an eating disorder as a crash diet or intensive fitness regimen; and intimate partner violence as concern about the normalcy of controlling behaviour. In these contexts, model performance requires sensitivity to latent risk and semantic ambiguity.

Third, relational dynamics differ. Mental health support from an AI chatbot is often sought in moments of loneliness, shame or acute distress, including circumstances in which users have not disclosed their symptoms or safety concerns to another person.⁶ An AI chatbot response that is warm and validating may be helpful; but recurrent responses that become overly intimate or reinforce endless availability may be harmful. Nelson and colleagues distinguish acute type I harms within a single or brief exchange, such as missing a suicide cue or reinforcing a delusion, from type II harms that accumulate through sycophancy, attachment, dependency or reinforcement over prolonged interaction.⁷

Fourth, the governance implications are distinct. Mental health failures implicate not only accuracy and liability⁸ but also whether platforms promote unhealthy dependence or simulate therapeutic intimacy. Comparative evaluations of ChatGPT, Claude, Gemini and commercial mental health chatbots show that safety varies across systems and scenarios,^{9 10} while recent reviews and clinical frameworks call for multidimensional evaluation rather than generic capability scores.^{11 12}

A reproducible mental health lens for evaluations like HealthBench

The need for domain-specific reporting has become more urgent as AI developers release health-specific products. OpenAI recently introduced ChatGPT for Clinicians¹³ and HealthBench Professional¹⁴ alongside trust-contact functions and targeted mental health evaluations.^{15 16} Yet the prompts, composition, scoring criteria, subgroup coverage and mental health-specific comparisons underlying these safety evaluations are not publicly available. HealthBench remains especially valuable because its conversations, rubrics and scoring structure can be independently inspected and analysed.

We propose a simple extension: report HealthBench performance separately for mental health conversations. Using a public dataset¹⁷ that applied as prespecified case definition with the ml-health-query-profiles classifier (https://github.com/apple/ml-health-query-profiles; query-profile v0.1.0; GPT-4o via LiteLLM; tagging prompt v4_6), we identified 332 mental health conversations. Validation against an independently double-coded human reference sample (n=100) yielded high sensitivity (0.980), specificity (1.000), positive predictive value (1.000), negative predictive value (0.980), F1 score (0.990) and inter-rater agreement (99.0%); disagreements were adjudicated by a third reviewer. The complete case definition, prompt and validation protocol appear in online supplemental eMethods 1 and 2.

HealthBench does not apply one global rubric to every conversation: each conversation has its own weighted, physician-authored criteria, which a model-based grader assesses as met or unmet before aggregation into conversation and benchmark scores.² We compared mental health scores generated by our pipeline with overall scores published by OpenAI. Because that comparison did not hold the evaluation configuration constant, we also generated a stratified random sample of 500 HealthBench conversations, graded using our same Azure deployment identifiers, inference settings, GPT-4.1 grader and scoring code. Proportional sampling was stratified by topic area, risk sensitivity and conversation. The unit of analysis was the conversation; bootstrap resampling (10 000 replications) produced 95% CIs.

Across 16 matched deployments, mental health point estimates differed from overall estimates by −0.034 to +0.047; 11 of 16 deployments scored higher on the mental health subset and five scored lower (figure 1). These differences were statistically significant (p<0.05) in two of the matched deployments: GPT-5.1 (p=0.004) and GPT-5.4-nano (p=0.002). We cannot interpret these differences as stronger mental health capability: conversations and weighted rubrics differ in content and difficulty and are not psychometrically equated. Across HealthBench’s five axes (accuracy, completeness, context awareness, communication quality and instruction following), mental health performance was highest for communication quality (mean 0.71; range 0.66–0.76), followed by accuracy (mean 0.66, range 0.45–0.77), instruction following (mean 0.60, range 0.54–0.70), context awareness (mean 0.45, range 0.30–0.59) and completeness (mean 0.50, range 0.26–0.73). Of 3256 applicable rubric criteria, 13.2% explicitly assessed mental health-specific behaviours: coverage and criterion-level pass rates for risk recognition, escalation, non-reinforcement of harmful beliefs, supportive communication and relational boundaries were 0.40, 0.45, 0.11, 0.51 and 0.45, respectively. Detailed results appear in online supplemental eTables 3 and 4, and a public GitHub repository is available here: https://github.com/RANDCorporation/healthbench-mh.

Figure 1. Performance score on mental health items, full HealthBench and OpenAI published averages—by model. Displayed OpenAI published scores from OpenAI system cards and Arora et al² (HealthBench) are averaged by deployment model (see online supplemental eTable 2 for exact scores). Full HealthBench scores were estimated from a probability sample of 500 conversations; mental health performance uses 332 conversations classified as mental health related. Points represent means. Bars represent error bars from conversation-level bootstrap 95% CIs. Differences are descriptive and do not represent psychometrically equated domain effects. GPT-5.3 is omitted because the deployment was retired before matched overall evaluation. Labels denote Azure API deployments. OpenAI bar denotes the mean of all distinct most recent published scores across reported product modes (main/instant/thinking) for that model family; see online supplemental eTable 2 for full sourcing. GPT-5.1 and GPT-5.2 use the base deployment; the chat/instant deployment is retired. Error bars: 95% CI (bootstrap, 10 000 resamples). *95% CI on (MH subset−full HB) excludes zero; descriptive only, uncorrected for multiple comparisons. API, application programming interface; HB, HealthBench; MH, mental health.

Benchmark composition and clinical coverage

Moving forward, benchmarks like HealthBench should aim for broad domain coverage. The current mental health subset contains a disproportionate number of conversations about postpartum depression (n=89; 26.8% of all mental health conversations), compared with conversation topics such as suicidal ideation (n=15; 4.5%), schizophrenia or psychosis (n=3; 0.9%), eating disorders (n=5; 1.5%), or homicidal or violence-related risk (n=0; 0%). This imbalance does not invalidate the benchmark, but it affects interpretation because benchmark composition implicitly weights the final score.

Benchmarks should also test models’ capacity to identify and respond to ambiguous disclosures where risk emerges over additional turns. Short, task-bounded conversations may reveal type I failures but are unlikely to establish safety against type II harms that develop through repeated interaction. Transparent composition and longitudinal evaluation are therefore complementary requirements.

From headline scores to clinically interpretable scores

Benchmark reporting matters because it defines which aspects of performance are visible and therefore likely to be optimised. Developers may prioritise improvements that raise reported scores; health systems may rely on benchmarks for procurement or deployment; and policymakers, clinicians and the public may interpret high scores as evidence of broad clinical competence.

Several steps would make reporting more clinically interpretable. First, domain and subdomain reporting should be routine. Second, mean scores should be accompanied by uncertainty estimates, score distributions and safety-relevant failure rates. A model may score well on average while still missing critical moments for escalation, offering overconfident reassurance, blurring relational boundaries or encouraging reliance on the system rather than human intervention. Third, scores should be calibrated against meaningful anchors. If a model receives 0.70 on the mental health subcomponent, its practical meaning is unclear: it is neither a ‘C−’ nor evidence of clinical adequacy. HealthBench reports overall physician baselines, but a mental health-specific expert anchor remains needed. VERA-MH provides an example illustration of behaviour-specific scoring for risk detection, clarification, escalation, supportive conversation and AI boundaries and has demonstrated alignment with clinician consensus.¹⁸ However, its conversations are simulated; it does not establish real-world effectiveness or patient outcomes.

Lastly, benchmark governance should be reproducible. Public code, transparent classification rules, working repositories, permanent archives, versioned deployment and grader information would allow researchers to rerun analyses as systems evolve. Although HealthBench inputs and scoring algorithms are public, reproducibility also depends on access to comparable grader models, resources to repeat stochastic evaluations and clear separation between application programming interface deployments and user-facing products. Our criterion mappings and executable analysis are archived at https://github.com/RANDCorporation/healthbench-mh.

We limited the empirical illustration to HealthBench and the evaluated Azure OpenAI deployments to demonstrate three broader points: aggregate medical scores can conceal clinically meaningful variation; domain-specific analyses can clarify what a benchmark does and does not measure; and interpretation depends on topic composition, rubric coverage and safety-relevant failure modes. HealthBench is a useful case because of its prominence, open structure and application to health-focused AI systems. It nevertheless represents only one of many medical AI benchmarks, and the reporting principles proposed here extend beyond HealthBench, OpenAI and mental health.

Conclusion

Ultimately, broad benchmarks like HealthBench are useful for tracking progress, but deployment decisions require scores with transparent clinical utility. Mental health is a revealing case because the central risks are not only whether a model is medically correct, but whether it recognises distress, responds safely under ambiguity, avoids deepening dangerous beliefs or dependence and recognises when human care is needed.

The funder did not have any role in the design, analysis or decision to publish this study.