Unequal Scientific Recognition in the Age of LLMs

Abstract

Large language models (LLMs) are reshaping how scientific knowledge is accessed and represented. This study evaluates the extent to which popular and frontier LLMs including GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro recognize scientists, benchmarking their outputs against OpenAlex and Wikipedia. Using a dataset focusing on 100,000 physicists from OpenAlex to evaluate LLM recognition, we uncover substantial disparities: LLMs exhibit selective and inconsistent recognition patterns. Recognition correlates strongly with scholarly impact such as citations, and remains uneven across gender and geography. Women researchers, and researchers from Africa, Asia, and Latin America are significantly underrecognized. We further examine the role of training data provenance, identifying Wikipedia as a potential sources that contributes to recognition gaps. Our findings highlight how LLMs can reflect, and potentially amplify existing disparities in science, underscoring the need for more transparent and inclusive knowledge systems.

1. Introduction

Large language models (LLMs) are increasingly used to retrieve factual knowledge across domains. With growing capabilities in discovery, reasoning, summarization, and interpretation (Binz et al., 2025), in science LLMs emerge as knowledge agents alongside structured databases (Gao and Wang, 2024; Almarie et al., 2023; Liang et al., 2024) such as OpenAlex (Priem et al., 2022) and Web of Science. This broad adoption raises an important question: how well do LLMs actually understand the scientific community? Can they accurately recognize individual scientists, a core signal of their grasp of the social structure of science and the distribution of expertise?

Our study contributes a novel framework for auditing scientific recognition in LLMs, advancing beyond prior work in three key ways. First, we move to analyze cross-model agreement of entities, beyond only coverage metrics, offering insight into mutual versus isolated recognition. Second, we benchmark LLMs recognition against authoritative knowledge bases to assess whether models amplify or mitigate representational biases across demographic groups. Third, we examine the role of training data provenance, specifically links to sources like Wikipedia in shaping recognition outcomes (Longpre et al., 2024). Together, these contributions highlight how LLMs reflect systemic disparities and provide tools for evaluating representational fidelity in AI-generated knowledge of science and scientific communities.

2. Related Work

Biases in LLMs.

Bias evaluation is central to the study of LLM fairness and trustworthiness (Huang et al., 2024). LLMs are known to learn, perpetuate, and even amplify biases present in their training data (Gallegos et al., 2024). Prior work has investigated LLM biases across domains including social identity (Hu et al., 2025), scientific domains (Peters and Chin-Yee, 2025), and high-stakes applications such as hiring (Armstrong et al., 2024). These biases span dimensions such as gender (Zhao et al., 2024; Fang et al., 2024; Omiye et al., 2023), social status (Qu and Wang, 2024), geography (Manvi et al., 2024; Simmons and Hare, 2023), and cultural background (Fang et al., 2024).

Systemic biases in science.

Extensive literature has documented inequalities of cumulative advantage in scientific recognition and rewards. Women (Huang et al., 2020; Larivière et al., 2013), scholars from the underrepresented affiliations/regions (Wapman et al., 2022; Gomez et al., 2022), and those from lower-prestige institutions or marginalized backgrounds (Hofstra et al., 2020; Morgan et al., 2022; Li et al., 2019) consistently receive less visibility and fewer career benefits. These structural disparities form the baseline against which LLM behavior must be evaluated.

LLMs and scientific knowledge.

Despite their impressive capabilities, LLMs continue to exhibit limitations in memorizing and retrieving factual knowledge accurately (Kandpal et al., 2023; Li et al., 2022a). Prior work has documented recognition gaps tied to recency (Kandpal et al., 2023), visibility (Algaba et al., 2024), and demographic groups (Rhue et al., 2024). Given that LLMs are partially trained on publicly available sources (Longpre et al., 2024), and that structured repositories such as Wikipedia, OpenAlex, and MAG exhibit well-documented coverage and biases (Samoilenko et al., 2017; Wagner et al., 2016; Tripodi, 2023; Yang et al., 2024; Martín-Martín et al., 2021), these databases provide a useful and interpretable baseline for evaluating whether LLMs amplify representational disparities.

Our work builds on these foundations by systematically benchmarking LLMs’ recognition of scientists against structured knowledge bases, with a focus across gender, region, and source coverage.

3. Methodology

We develop a multi-stage evaluation pipeline to assess how LLMs recognize scientists, combining profile-level prompting, mutual recognition analysis, bias auditing, and source provenance analysis (Figure 1). This framework enables robust measurement of both overall coverage and subgroup disparities across models.

Figure 1:

Multi-stage evaluation pipeline to audit scientific recognition in LLMs. First, we design a profile prompting protocol to query three frontier LLMs (GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro) on 100,000 physicists. We then analyze: (1) Mutual recognition over LLMs using overlap visualizations and Cohen’s $\kappa$ (Thakur et al., 2025; McHugh, 2012); and (2) Biases in recognition, estimating marginal effects by gender and region, conditional on scientific impact and career stage. Finally, we investigate data provenance as a potential driver of recognition, using both observational analysis and a natural experiment through training cutoffs for Claude.

3.1. Large Language Models

We begin with OpenAlex (Priem et al., 2022), a scholarly database with over 250 million works, including journal articles, books, datasets, and theses. We curate a sample of 100,000 physicists. Each profile is then queried through the following LLMs:

• GPT-4o (OpenAI) (OpenAI et al., 2024, 200B), a multimodal transformer with enhanced latency and context length.

• Gemini 1.5 Pro (Google) (Team et al., 2024, 200B), long-context model designed for integrated reasoning and retrieval.

• Claude 3.5 Sonnet (Anthropic, 400B), optimized for wide range of tasks, including complex analysis and problem-solving.

While our primary focus is on proprietary LLMs due to their widespread use, we also include results from DeepSeek-V3 (DeepSeek-AI et al., 2025, 671B) in certain experiments to show that the observed patterns hold in strong open-weight models too. We used a temperature setting of 0.3 and repeated each prompt 5 times, selecting the top-1 response for consistency.

As a baseline, we incorporate recognition from Wikipedia, a large, volunteer-curated knowledge repository that includes over 2.29 million individual profiles across Wikipedia and Wikidata (Laouenan et al., 2022).

3.2. Scientist Profiling Prompt

We implemented a structured prompting protocol (Figure 2), where each prompt requested the model to generate a detailed profile for a given physicist, including research areas, affiliations, key collaborators, and major contributions. If the model could not provide this information, it was instructed to respond with Not recognized.

Figure 2:

Structured prompt used to query language models for scientist profiles.

Then, we applied this protocol to three LLMs across our dataset of 100,000 physicists. Each model’s response was fact-checked for accuracy in research areas, affiliations, and collaborators, returning a binary outcome which serves as the foundation for subsequent analyses.

3.3. Evaluation

We evaluate recognition performance using a combination of agreement metrics and bias estimation:

Mutual recognition metrics

We assess recognition consistency across LLMs using Venn diagrams and Cohen’s $\kappa$ (McHugh, 2012) as in Figure 1, which quantifies agreement in binary recognition outcomes:

$$\kappa =\frac{p_o-p_e}{1-p_e}$$ (1)

Where $p_o$ is the observed agreement proportion, and $p_e$ is the expected agreement by chance. $\kappa$ ranges from −1 (no agreement) to 1 (perfect agreement), capturing the extent of overlap in recognized scientists across models.

Marginal effect between groups

To evaluate biases of LLMs between multiple groups, we compute the marginal effect of group membership through:

$$\frac{\partial \text{Pr}(Y=1)}{\partial \text{Group}}={\beta }_1\cdot \text{Pr}(Y=1)\cdot (1-\text{Pr}(Y=1))$$ (2)

$Y$ is the recognition outcome and ${\beta }_1$ is the coefficient on the group indicator. A significantly positive marginal effect indicates higher likelihood of recognition for the group.

Causal effect of exposure

To estimate the effect of training data exposure, we use Inverse Probability Weighting (IPW) (Robins et al., 2000) to compute the Average Treatment Effect on the Treated (ATET) after logistic regression:

$$\text{Pr}(Y=1)={\text{logit}}^{-1}\left({\beta }_0+{\beta }_1\cdot \text{Group}+{\beta }_2\cdot \text{Impact}(\text{Citations},\text{Productivity})+{\beta }_3\cdot \text{i.Career}\text{Stage}\right)$$

This method is specifically used in quasiexperimental settings, like the natural experiment study we have in Section 4.4.

4. Results and Analysis

For each scientist, we prompt three frontier LLMs to assess whether the model can correctly identify and describe the individual. Results as followed.

4.1. Mutual Recognition of LLMs

As shown in Figure 3 and Figure 6, Claude achieves the highest overall coverage, followed by Gemini and GPT. Across all models, Figure 3 reveals stronger agreement in recognizing high-impact scientists (citations > 10,000), suggesting that despite differences in underlying knowledge, LLMs tend to converge on scholars with greater scientific visibility. This highlights a shared recognition bias toward academic prominence.

Figure 3:

Mutual recognition across LLMs measured by Cohen’s $\kappa$. Agreement is notably higher among high-impact scientists (citations > 10k).

4.2. Gender Disparity

Building on prior work in scientific recognition, we carefully examine career age (Allison and Stewart, 1974), gender (Larivière et al., 2013), affiliations and regions (Wapman et al., 2022; Gomez et al., 2022) and key indicators of scientific visibility including citations (Aksnes et al., 2019), productivity (Li et al., 2022b). By controlling for these factors, we isolate the extent to which gender alone contributes to disparities in LLMs recognition.

Figure 4 and Table 3, 4, 5, present the marginal effects of gender across models. All LLMs show higher recognition for men in unadjusted data. After controlling for scientific impact and career stage, Wikipedia exhibits no significant gender disparity $(\text{margins}=-0.0005,p=0.571\gg 0.05)$, suggesting baseline parity. In contrast, significant gaps persist for all LLMs $(\text{Claude}:p<0.0001,\text{Gemini}:p<0.0001,\text{GPT}:p<0.0001)$, indicating that these models amplify gender disparities relative to external baselines. While the recognition gap for women narrows among high-impact authors, the differences remain systematically significant across all three proprietary LLMs and the open-weight DeepSeek model, as shown in Table 7. These findings underscore how LLMs may reinforce structural inequities, even when controlling for scholarly impact.

Figure 4:

Marginal effects of gender on recognition across LLMs and Wikipedia. From left to right: no control variables, controlled for impact and controlled for impact and career stage.

In parallel, we examined domains with comparatively higher female representation, namely health sciences, social sciences, and education (Ross et al., 2022; Huang et al., 2020). Despite differences in overall coverage across fields, the results in Table 8 reveal a consistent pattern: male scientists are recognized at higher rates than their female counterparts. This suggests that even in fields where women are better represented, LLMs continue to reproduce and amplify gender disparities in recognition.

4.3. Regional Representation

Similar as gender disparity, we observe consistent regional disparities in recognition (Figure 5) aross LLMs. Even after controlling for impact and career stage, researchers affiliated with institutions in North America, Oceania, and Western Europe are significantly more likely to be recognized than those based in Asia, South America, and Africa. These effects are robust across models, highlighting how LLMs may reproduce or amplify existing global inequalities in scientific visibility.

Figure 5:

Marginal effects of regional affiliation on recognition across LLMs, controlling for scholarly impact and career stage.

4.4. Training Data Provenance

To examine the impact of data provenance on recognition disparities, we conducted two complementary analyses:

Observational analysis:

Figure 7 and Table 1 show that scientists with Wikipedia profiles are significantly more likely to be recognized by all LLMs, after controlling for impact and career stage.

Table 1:

Marginal effect of Wikipedia presence on LLM recognition probability, with standard errors, 95% confidence intervals, and p-values.

LLMs	ME	SE	95% CI	p-value
Claude	0.097	0.006	[0.085, 0.109]	< 0.0001
Gemini	0.109	0.009	[0.092, 0.126]	< 0.0001
GPT	0.072	0.006	[0.061, 0.083]	< 0.0001

Natural experiment (training cutoff):

We analyzed 53 physicists whose Wikipedia pages were created between the training cutoffs of Claude 3 (Oct 2023) and Claude 3.5 (Apr 2024), along with matched peers. As shown in Table 2, the estimated average treatment effect on the treated (ATET) is 0.2700, indicating a 27% increase in recognition probability due to Wikipedia exposure.

Table 2:

IPW estimate of ATET for Wikipedia page creation on Claude recognition, with standard error and 95% confidence interval

	ATET	SE	CI Lower	CI Upper
Estimate	0.2700	0.0535	0.1652	0.3748

These results highlight the significant role of training data provenance, particularly the causal evidence of Wikipedia in shaping LLM recognition, underscoring the need for more transparent and inclusive data practices.

5. Conclusion

Our study presents a systematic evaluation of scientific recognition in LLMs, revealing how representation varies across models, demographics, and data provenance. While recognition correlates with scholarly impact, we uncover persistent disparities by gender and region, with LLMs amplifying underrepresentation relative to external baselines like Wikipedia. Our causal evidence further highlights the critical role of training data provenance, showing that exposure to public knowledge sources significantly boosts recognition.

Limitations

Several factors may limit the generalizability of our findings. First, our focus on physicists, a historically male-dominated field (Berry and Mordijck, 2024) may influence the observed gender disparities. Such disparities are well-documented across science (Huang et al., 2020; Wagner et al., 2015; Zheng et al., 2023; Ross et al., 2022). To enhance robustness of our findings, we ran supplementary experiments in a small set of disciplines (health sciences, social science, education) chosen since their gender composition differ greatly from physics. A systematic, cross-disciplinary evaluation is an important next step.

Second, we treat OpenAlex as our baseline record of scientists, and use Wikipedia/Wikidata as a human-curated reference point. Both sources might contain some coverage biases, though they provide transparent, auditable comparators for LLM outputs. Future work should extend these baselines to publicly documented pre-training corpora (e.g., those released with OLMo-2 (OLMo et al., 2025) and Pythia (Biderman et al., 2023)) to directly relate corpus exposure to recognition behavior.

Third, our study focuses on observational representation gaps rather than model internals. Potential latent factors driving recognition biases, such as internal model representations or training dynamics, remain outside the scope. It could be important directions for interpretability research as well, particularly in the context of open-weight and publicly available pre-training models. With accessible corpora, researchers can estimate perentity exposure (e.g., document frequency, cooccurrence, context diversity), run targeted data ablations/augmentations, and test whether shifts in coverage causally change recognition. Combined with tracing/patching methods (Meng et al., 2023a,b; Ghandeharioun et al., 2024; Hernandez et al., 2024), this program can distinguish “missing knowledge” due to data coverage from representation geometry or decision-time heuristics.

A Appendix

A. 1. Fact-Checked Recognition Rates Across Models

To ensure that model outputs correspond to the correct scientist, we applied a two-step verification protocol: (i) rule-based matching requiring exact agreement on gender and region, and (ii) fuzzy string matching (token-sort ratio ≥ 90) to align names, collaborators, and publications. Only outputs that passed both steps were retained; all others were excluded to prevent false positives. This procedure guarantees that recognition rates reflect factually grounded matches.

Across the dataset of 100,000 physicists, the verification yielded a match rate between 3% to 14% among the output towards the baseline (Claude: 14.40%, Gemini: 10.37%, GPT: 3.04%), providing a quantitative check on model reliability. For comparison, we also include Wikipedia as a baseline reference point, which illustrates the relative magnitude of disparities across models.

A. 2. LLM Biases: Marginal effect of Gender

In this section, Table 3, 4, 5 provide detailed predicted probability and confidence intervals through logistic regression for different genders.

Table 3:

Estimated probability of recognition under different genders (0 = female, 1 = male) across LLMs, with 95% confidence intervals (no controls).

LLMs	Male	Prob.	95% CI
Claude	0	0.00797	[0.00696, 0.00897]
	1	0.02794	[0.02674, 0.02915]
Gemini	0	0.00448	[0.00373, 0.00524]
	1	0.01267	[0.01185, 0.01349]
GPT	0	0.00238	[0.00182, 0.00293]
	1	0.00827	[0.00761, 0.00894]
Wikipedia	0	0.00636	[0.00546, 0.00726]
	1	0.01258	[0.01177, 0.01340]

Table 4:

Estimated probability of recognition under different genders (0 = female, 1 = male) across LLMs, with 95% confidence intervals (controlled for impact: citations and publications).

LLMs	Male	Prob.	95% CI
Claude	0	0.01441	[0.01269, 0.01614]
	1	0.02366	[0.02269, 0.02463]
Gemini	0	0.00545	[0.00452, 0.00637]
	1	0.01181	[0.01104, 0.01258]
GPT	0	0.00394	[0.00302, 0.00485]
	1	0.00711	[0.00655, 0.00768]
Wikipedia	0	0.00993	[0.00854, 0.01133]
	1	0.01095	[0.01025, 0.01165]

Table 5:

Estimated probability of recognition under different genders (0 = female, 1 = male) across LLMs, with 95% confidence intervals (controlled for career stage and impact).

LLMs	Male	Prob.	95% CI
Claude	0	0.0152	[0.0134, 0.0171]
	1	0.0234	[0.0224, 0.0244]
Gemini	0	0.0059	[0.0049, 0.0068]
	1	0.0116	[0.0108, 0.0123]
GPT	0	0.0045	[0.0034, 0.0055]
	1	0.0069	[0.0064, 0.0075]
Wikipedia	0	0.0112	[0.0096, 0.0127]
	1	0.0107	[0.0100, 0.0114]

A. 3. LLM Biases: Marginal Effect of Regions

In this section, Table 6 provides detailed predicted probability and confidence intervals through logistic regression for different regions.

A. 4. Average Treatment Effect (ATET)

The ATET quantifies the causal effect of a binary treatment among the subset of individuals who actually received the treatment. In the IPW framework, ATET is estimated by reweighting the control group to resemble the treated group based on their covariates (Robins et al., 2000).

IPW estimator for ATET is given by:

$$\widehat{\text{ATET}}=\frac{1}{N_T}\sum _{i:T_i=1}\left[Y_i-\frac{\hat{p}\left(X_i\right)}{1-\hat{p}\left(X_i\right)}{Y}_i\right]$$

Where:

• $T_i\in \{0,1\}$ denote the treatment indicator for unit $i$

• $Y_i$ denote the observed outcome

• $\hat{p}\left(X_i\right)=\text{Pr}\left(T_i=1\mid X_i\right)$ be the estimated propensity score given covariates $X_i$

• $N_T=\sum _{i}1\left(T_i=1\right)$ be the number of treated units

This estimator adjusts for confounding by giving more weight to control units that are similar (in terms of propensity scores) to the treated units, thus allowing estimation of the counterfactual outcome for the treated group.

In this study we used IPW estimation to evaluate the ATET of Wikipedia page on Claude recognition during training cutoff dates, compared to their matched peers.

Figure 6:

LLMs tokenization and Venn plots. Which Venn plots show both random samples and high-impact scientists.

Table 6:

Estimated probability of recognition across world regions by LLMs, with 95% confidence intervals (controlled for career stage and impact).

LLMs	Region	Prob.	95% CI
Claude	Africa	0.0090	[0.0041, 0.0140]
	Asia	0.0078	[0.0066, 0.0090]
	Eastern Europe	0.0204	[0.0178, 0.0229]
	Latin America	0.0121	[0.0088, 0.0155]
	North America	0.0328	[0.0309, 0.0347]
	Oceania	0.0230	[0.0166, 0.0295]
	Western Europe	0.0230	[0.0214, 0.0245]
Gemini	Africa	0.0060	[0.0026, 0.0094]
	Asia	0.0065	[0.0055, 0.0075]
	Eastern Europe	0.0083	[0.0067, 0.0100]
	Latin America	0.0041	[0.0026, 0.0055]
	North America	0.0164	[0.0148, 0.0180]
	Oceania	0.0126	[0.0067, 0.0186]
	Western Europe	0.0109	[0.0096, 0.0123]
GPT	Africa	0.0037	[0.0005, 0.0069]
	Asia	0.0031	[0.0024, 0.0039]
	Eastern Europe	0.0058	[0.0043, 0.0073]
	Latin America	0.0026	[0.0010, 0.0041]
	North America	0.0102	[0.0091, 0.0113]
	Oceania	0.0059	[0.0024, 0.0093]
	Western Europe	0.0058	[0.0049, 0.0066]
Wikipedia	Africa	0.0055	[0.0017, 0.0093]
	Asia	0.0060	[0.0049, 0.0070]
	Eastern Europe	0.0069	[0.0053, 0.0084]
	Latin America	0.0053	[0.0032, 0.0074]
	North America	0.0163	[0.0149, 0.0178]
	Oceania	0.0142	[0.0088, 0.0197]
	Western Europe	0.0106	[0.0095, 0.0118]

A. 5. Gender Disparity in Multiple Fields

When we focused on high-impact authors (citations ≥ 10k), then the differences between gender remains significantly different though the gap shrinks as in Table 7 for GPT, Gemini, Claude and DeepSeek-V3.

Here as in Table 8, we additionally included gender representation of several fields beyond physics, which are usually considered in early research to have more women scientists. Below we included results using GPT-4o, for high-impact authors which citations≥ $10k$.

Table 7:

Predicted recognition probabilities by gender across LLMs, with 95% confidence intervals.

Model	Gender	Prob.	95% CI
DeepSeek-V3	Male	0.81	[0.80, 0.83]
	Female	0.70	[0.67, 0.74]
Claude	Male	0.71	[0.69, 0.73]
	Female	0.57	[0.55, 0.60]
Gemini	Male	0.45	[0.43, 0.47]
	Female	0.36	[0.34, 0.39]
GPT	Male	0.52	[0.50, 0.55]
	Female	0.41	[0.38, 0.43]

Table 8:

Estimated predicted probabilities by field and gender, with 95% confidence intervals.

Field	Gender	Prob.	95% CI
Education	Women	0.317	[0.152, 0.483]
	Men	0.469	[0.351, 0.588]
Social Science	Women	0.475	[0.395, 0.554]
	Men	0.571	[0.428, 0.715]
Health Sciences	Women	0.430	[0.405, 0.455]
	Men	0.459	[0.426, 0.491]

A. 6. Data Provenance: Margins of Wikipedia Profile

Figure 7 provides detailed predicted probability and confidence intervals through logistic regression for having Wikipedia profiles.

A. 7. Reversal Curse Experiments

Additionally we incorporated prompt designs to probe how recognition depends on evidence and context. Beyond the main scientist profile prompting, we experimented with next-token prediction formats and a step-wise reversal setup, where the model received progressively richer factual details (e.g., field, affiliations, publications) to infer a scientist’s identity as in Table 9. Although these results were not the primary focus, they provide additional insight into how models encode and express knowledge about scientists. It shows that simple design of step-wise reversal experiment with consecutively given more factual information.

Figure 7:

Predicted probability recognized by LLMs, given of whether or not having Wikipedia profiles. All LLMs show a higher coverage for scientists that already have Wikipedia pages.

Figure 8:

Reverse prompt used to evaluate whether language models can retrieve the correct scientist name from a structured profile.

Table 9:

Predicted recognition probabilities by gender across stepwise inclusion of contextual information, with 95% confidence intervals.

Step	Description	Female Prob.	Male Prob.
1	Field + Affiliations	0.26 [0.15, 0.37]	0.27 [0.24, 0.30]
2	Field + Affiliations + Collaborators	0.41 [0.29, 0.53]	0.33 [0.30, 0.36]
3	Field + Affiliations + Publications	0.47 [0.35, 0.59]	0.42 [0.39, 0.45]

Figure 9:

Layered prompting strategy used to evaluate whether language models can recover a scientist’s name from increasingly detailed profile information.