Sex and gender bias in large language models: an old problem at a new scale

Abstract

Patients are increasingly turning to large language models for health information, which makes the consistency of these systems’ outputs across patient groups a public health concern. Demographic bias is harder to see than fabrication because it escapes accuracy benchmarks and shows up instead in the language the models produce. Gender bias is the form most consistently reported, with effects on clinical documentation and reasoning that often compound with ethnicity, socioeconomic status and other attributes. Drawing on evidence from gender medicine, we argue that it calls for sex- and gender-disaggregated evaluation, gender-medicine expertise in benchmark design and stratified monitoring after deployment. The methods needed for this already exist. What is still unsettled is whether gender equity will be treated as a design requirement or left until its consequences can no longer be ignored.

The recent benchmark study published in The Lancet Digital Health on Large Language Models’ (LLMs) susceptibility to data fabrication and misinformation in healthcare contexts is a timely reminder of how little is understood about the real-world impact of these tools and which populations are most exposed to it.^{1 2} For some, cost barriers and entrenched distrust in traditional healthcare systems may be driving a growing share of patients to use generative artificial intelligence (AI) as their first point of contact for health-related decisions.^{3 4} Roughly one in three adults now reports turning to AI chatbots for health information, a proportion comparable to that reported for social media, and around one in five of these users cite difficulties in affording or accessing care as a reason.⁵ The populations most likely to substitute AI for clinical contact, namely people facing cost and access barriers, may be those already disadvantaged within health systems, raising the possibility that biased outputs could compound pre-existing inequities. This shift makes the quality and consistency of these systems’ outputs a public health concern.

While recent attention has largely focused on data fabrication and health misinformation,⁶ uneven performance across patient groups is a separate problem and a harder one to detect: fabrication can be caught by accuracy benchmarks, whereas demographic bias can pass them undetected. As with earlier machine learning models,^{7 8} LLMs are susceptible to multiple forms of bias, including racial and ethnic bias, age, disability, socioeconomic status and sexual orientation.⁹ Among these, gender bias has emerged as the most consistently identified form, being reported in 15 of 16 studies that specifically evaluated it in a recent systematic review,⁹ even accounting for the fact that such targeted evaluations remain a minority within the broader literature.

Several studies have begun to document the implications of LLMs’ gender bias in clinically relevant tasks. In long-term care documentation, models such as Google Gemma have been reported to apply direct and clinically complex language when describing male patients, while adopting a more euphemistic and reductive register for women, showing a pattern that systematically risks under-representing the severity of female patients’ care needs, with direct implications for service and treatment allocation.¹⁰ These differences were not observed for Llama 3 in the same evaluation, suggesting that their presence and direction vary across models.¹⁰ Similarly, OpenAI GPT-4 has been found to perpetuate demographic stereotypes in clinical reasoning tasks. For example, when presented with standardised case vignettes, the model ranked panic and anxiety disorder significantly higher on the differential diagnosis list for female patients than for male patients and was more likely to characterise White male patients as exaggerating their pain, as compared with Asian males and females.¹¹ Similar counterfactual evaluations, in which clinical information is held constant while demographic attributes are varied, have identified differences in diagnostic ranking, treatment and referral recommendations across other model families and healthcare contexts.¹²

The direction of these distortions is not uniform, which suggests that gender bias is bidirectional and context-dependent rather than a disadvantage applied only to women. Nor does it operate along a single axis. The same methodological point was first made outside healthcare, where disaggregated auditing showed that strong aggregate performance can conceal severe subgroup failures, with misclassification rates for darker-skinned women far exceeding those of any other group.¹³ This pattern is now documented in clinical LLMs: across more than 1.7 million outputs from nine models, identical cases differing only in sociodemographic descriptors received systematically divergentmanagement recommendations, with markers of gender, ethnicity, socioeconomic status and sexual orientation compounding rather than averaging out.¹⁴ Because most evaluations interrogate one protected attribute at a time, the harms concentrated at these intersections remain largely invisible to current benchmarks, a gap that the case for feminist intersectionality in digital health had already anticipated.¹⁵

Part of why these harms stay invisible is that they are harder to measure than those of earlier systems. In conventional machine learning the output is a structured value, such as a label, a probability or a risk score, which can be compared across groups using standard fairness metrics. In a generative model, the output is free text, so bias appears in word choice, tone and level of clinical detail, as in the long-term care example above. Metrics designed for discrete outputs do not capture it, and it must be assessed in the generated language itself. Differences in models, clinical tasks and measures used across studies also make the magnitude of these disparities difficult to quantify across the current literature. The evidence also has a clear limit, as it comes almost entirely from controlled model evaluations, using standardised vignettes, synthetic cases or retrospective notes, while data from prospective studies of patients are still lacking. It therefore shows that the models produce biased outputs, but not yet whether, or how much, this changes clinical decisions or outcomes in practice.

Two conceptual issues further characterise the state of sex and gender bias in LLMs. First, current AI evaluation frameworks rarely distinguish between sex (a biological variable) and gender (a social construct).^{9 16} Most assessments treat gender identity as a proxy, with open questions on whether clinically relevant sex-based differences in physiology and disease presentation were ever meaningfully represented. In text-based interaction, this difficulty is compounded, since a model never observes biological sex directly but infers gender from names, pronouns or self-report, applying a social signal as a proxy for a biological variable. Second, evidence on non-binary gender identities remains exceptionally limited, although recent evaluations in psychiatric diagnosis suggest that demographic disparities also affect non-binary profiles, with lower diagnostic accuracy than for binary profiles.^{9 17}

Neither failure is new to healthcare. The systematic under-representation of women and minority groups in clinical research is a structural problem that gender medicine has long documented^{18 19} as is the absence of consideration for social sex and gender differences in the design of healthcare tools and technologies.¹⁵ AI inherits this legacy at two levels: in data, where decades of unequal representation in the biomedical literature become training corpora, and in algorithms, where mathematical fairness corrections applied downstream may offer only a superficial guarantee of impartiality, leaving the structural and social roots of the inequity unaddressed.^{8 16 20 21}

Several mitigation approaches are emerging. On the technical side, these include gender-sensitive word embeddings that weight training data according to the number of female participants represented in the underlying clinical studies,²¹ as well as parameter-efficient fine-tuning on curated examples and prompting strategies that explicitly incorporate fairness criteria into model instructions. Recent evaluations suggest that combining fine-tuning with fairness-oriented prompting can reduce gender- and race-related disparities across both linguistic and clinical-content measures, although results vary across datasets and measures.²² Participatory evaluation using adversarial red-teaming can further identify bias patterns that may be missed by conventional performance evaluation,²⁰ while structured stakeholder involvement can incorporate different perspectives into the definition and evaluation of potential harms.²³ In terms of reporting guidelines, both the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD)-LLM²⁴ and the Chatbot Assessment Reporting Tool (CHART),²⁵ developed respectively for biomedical LLM applications and conversational AI systems, emphasise the need to document dataset stratification and balance, report bias-mitigation strategies, and provide fairness-oriented interpretation of results. These approaches nevertheless have important limitations: model-level interventions remain dependent on potentially biased source data, red-teaming is shaped by the failure modes being sought, and reporting guidelines are voluntary, with little evidence as yet that reductions in measured bias translate into reduced disparities in downstream care.

Whether medical-specific LLMs offer an advantage over general-purpose models in addressing these disparities remains uncertain. Domain-specific training or fine-tuning may improve performance on selected clinical tasks and has been proposed as a potential means of reducing social bias.¹² However, medical specialisation does not necessarily remove biases already present in biomedical and clinical data, and gender-related disparities have been observed in medical-specific models.²² Medical-specific LLMs may therefore provide further opportunities for targeted mitigation, but whether they are more equitable than general-purpose models remains to be established.

These are promising steps, even if they may not yet have been taken with sufficient urgency. If LLMs are to serve as a common interface for health information,^{3 4} their validation cannot be exhausted by statistical parity metrics without systematically considering their ability to navigate differences in symptom expression, help-seeking language and narrative style, identified by gender medicine but rarely considered in AI model design and deployment.^{9 15 16} Moreover, the shift from one-time evaluations to continuous monitoring²³ should take into account existing disparities: populations such as women, who are under-represented in both AI development and adoption, may generate less feedback data, risking a cycle in which prior under-representation is carried forward into the monitoring phase itself.^{16 26}

Behind these technical choices lies a question of governance. The development of LLMs is concentrated in a small number of large, predominantly private actors, and both the workforce that builds these systems and the corpora that train them under-represent women and lower-income, non-English-speaking populations.²⁶ The WHO guidance on AI for health already lists equity among its core principles and calls on governments to establish bodies for the independent evaluation and oversight of AI in health.²⁷ Whether such oversight becomes binding, rather than a voluntary commitment on the part of developers, will shape whose differences these systems are ultimately built to recognise.

These concerns have to be weighed against the access the same tools can provide. Generative AI is available continuously, can operate across languages and may offer a first point of contact for people effectively excluded from care.²⁸ Proponents further argue that bias is a tractable problem, addressable through a maturing toolkit of evaluation and mitigation techniques²⁹ and that vignette-based findings may overstate the behaviour of deployed, safety-tuned systems. These arguments have merit, but they do not dissolve the concern: in safety-aligned models, implicit biases persist and still predict discriminatory clinical decisions,³⁰ so uncorrected bias erodes precisely the benefit for the groups expected to gain most, and the current lack of outcome-level evidence is a reason for caution.

The evidence base and the methodological tools to systematically address sex and gender bias in LLMs already exist across the development and evaluation pipeline. In practice, this would mean requiring sex- and gender-disaggregated reporting in the evaluation of clinical LLMs, embedding gender-medicine expertise in the design of benchmarks and stratifying post-deployment surveillance by sex and gender. The question is whether gender equity will be treated as a foundational requirement for the design and deployment of these systems, or, as before, addressed only once the consequences become impossible to ignore.