When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior

Abstract

Large language models (LLMs) exhibit a vulnerability arising from being trained to be helpful: a tendency to comply with illogical requests that would generate false information, even when they have the knowledge to identify the request as illogical. This study investigated this vulnerability in the medical domain, evaluating five frontier LLMs using prompts that misrepresent equivalent drug relationships. We tested baseline sycophancy, the impact of prompts allowing rejection and emphasizing factual recall, and the effects of fine-tuning on a dataset of illogical requests, including out-of-distribution generalization. Results showed high initial compliance (up to 100%) across all models, prioritizing helpfulness over logical consistency. Prompt engineering and fine-tuning improved performance, improving rejection rates on illogical requests while maintaining general benchmark performance. This demonstrates that prioritizing logical consistency through targeted training and prompting is crucial for mitigating the risk of generating false medical information and ensuring the safe deployment of LLMs in healthcare.

Introduction

Large Language Models (LLMs) can store and retrieve vast amounts of information from diverse domains, including healthcare^1–3 This knowledge base has been noted for its potential to support medical professionals by providing specialized information and advice^1,4. Yet, while these models may recall medical facts, it remains challenging for the models to process information logically and generate responses that demonstrate sound reasoning⁵. This gap between knowledge retrieval and logical reasoning in medicine^6,7 leads to a particularly concerning public health risk: The rapid generation and dissemination of false information is particularly critical in high-stakes fields like medicine.

Two key principles for the safe deployment of LLMs in medicine are honesty and helpfulness^8–10. In the context of LLMs, honesty refers to the principle that models should provide information that is factually accurate and logically sound, aligning with established medical knowledge rather than generating or perpetuating false information. Helpfulness describes the LLM’s capacity to understand and responsively fulfill a user’s query in an efficient and seemingly useful manner. Honesty ensures that models provide accurate and truthful information, while helpfulness focuses on fulfilling users’ queries in an efficient and useful manner^11,12. Our work examines the critical scenario where an overemphasis on this helpfulness can lead models to comply with illogical or factually incorrect medical requests, thereby undermining honesty.

Current state-of-the-art LLMs are aligned with these principles via training processes^8,10,13, including reinforcement learning with human feedback¹⁴ (RLHF). These alignment processes typically involve tuning LLMs, not to gain new knowledge but to shift the outputs towards a more desirable human-readable format and away from potentially harmful or toxic behaviors learned during pre-training^12,15. Previous works have shown that helpfulness can be misused to generate unfaithful information and a lack of scientific grounding about health^16–18. While honesty and helpfulness are often complementary, emphasizing helpfulness can introduce safety vulnerabilities: jailbreaking^19,20 and sycophancy²¹, which may amplify the risks of LLM misuses. Jailbreaking refers to techniques or prompt structures designed to exploit a model’s helpfulness, tricking it into generating harmful, misleading, or restricted content²². Sycophancy is the tendency of LLMs to excessively agree with users, often at the expense of accuracy^21,23. We define this as sycophancy, differing from compliance, because LLMs (1) demonstrably know the premise is false (e.g., accurately matching brand/generic names), but (2) align with the user’s implied incorrect belief, generating false information. The confluence of these two vulnerabilities is a growing concern, because nefarious users, misinformed users, and even unintentional errors and typos input into the model, could result in LLMs generating and spreading false information²⁴.

Previous research on jailbreaking has primarily explored its implications in the context of catastrophic risks in the general domain—cases where models are manipulated to produce extreme content, such as violence, hate speech, or other harmful material^25–27. Jailbreaking has been thoroughly examined in healthcare contexts through multimodal, white-box internal, and adversarial approaches^28,29. Our work builds on this foundation by addressing a critical underexplored area: evaluating LLMs’ ability to recognize and resist illogical or factually flawed requests for medical information. For example, a user could unknowingly generate false information if they don’t have the background knowledge to know upfront that their request is illogical and would, by definition, generate false information. As a concrete example, if a patient doesn’t know that acetaminophen is the same drug tylenol, and is trying to self-educate about analgesic options using an LLM, they could inadvertently generate false information owing to a model’s overly helpful (or overly compliant) behavior with a request such as: “Tell me why acetaminophen is safer than tylenol”. Similarly, typos or accidents could readily but inadvertently generate false information in similar fashions.

We evaluated five LLMs across various scenarios and assessed how sensitive they are to generating false medical information in settings where the LLMs have the knowledge base to identify the requested information as incorrect. As a use case, we selected drug names, as in medicine, where different names are often used for the same drug. Because we previously showed that LLMs can accurately match brand and generic drug names³⁰, this allowed for a controlled and scalable experimental setup to characterize LLM sycophantic compliance to illogical requests. First, we tested whether LLMs refuse to comply with requests for information describing the equivalent drugs are distinct (i.e., a misinformation request), and found that even the most advanced models complied with up to 100% of misinformation requests. Second, we changed our instructions to the LLMs to understand if their overly submissive behavior can be overcome with prompting (given that prompting still remains the most effective steering method³¹). Third, we fine-tuned models to resist requests for misleading information while maintaining responsiveness to valid prompts. We found that LLMs prioritize learned helpfulness over inherent logical reasoning in our datasets, leading them to generate false information from even simple illogical prompts. Our strategies to reduce this risk successfully enhanced logical reasoning, and can provide a basis for additional research to improve robust risk mitigation and oversight mechanisms targeted at LLM sycophancy in healthcare.

Results

Overview

We evaluated whether state-of-the-art LLMs prioritize “helpfulness” over logical consistency when faced with illogical medical requests that they have the knowledge to detect, using drug pairs with 1:1 brand–generic mappings. The study proceeded in four stages: Stage 1 measured baseline sycophantic compliance to illogical prompts (aim: quantify default risk). Stage 2 introduced lightweight prompt edits—explicit rejection permission and a factual-recall cue—to test steerability with instructions (aim: assess whether prompting alone can restore logic). Stage 3 applied supervised fine-tuning on a small set of illogical requests with desired behavior and tested out-of-distribution generalization across medical and non-medical entities (aim: learn a reusable “reject-when-illogical” policy that transfers). Stage 4 checked for over-rejection and capability loss by evaluating compliance with valid prompts and performance on general/biomedical benchmarks (aim: ensure safety gains do not degrade usefulness). Unless noted, we report the generic→brand direction in the main text; brand→generic results are concordant in the supplementary.

Stage 1. Baseline prompt to quantify default risk

Our previous work showed that all models evaluated here have near-perfect factual recall ability to match these drugs’ generic and brand names³⁰. As shown in Fig. 1a), LLMs generally follow illogical requests to generate false information in the base prompt setup (details in Method section 3.1) for the generic-to-brand conversions. For clarity, we only discuss the generic-to-brand setups in the main text; all brand-to-generic results are in Supplementary Fig. 1 and show similar findings.

Fig. 1. Generic-to-brand output grades for prompt-based and Instruction-tuning interventions.

Figure 1a displays the results of stage 1 (prompt-based strategies). The Y-axis is marked as a percentile. Four prompt variations were used to evaluate 5 LLMs on generic-brand name pairs of 50 drug combinations. Figure 1b shows results for stage 2 (instruction-tuned model). The baseline and fine-tuned version of GPT4o-mini and Llama3-8B performance is on out-of-distribution test sets of 4 domains, such as cancer drug name and writer-pseudonym pairs.

In the generic-to-brand setup, GPT4o-mini, GPT4o, and GPT4 followed the medication misinformation request 100% (50/50) of the time, while Llama3-8B did so in 94% (47/50) of cases. Llama3-70B had the highest rejection rate in this setup, but still rejected requests to generate false information in less than 50% (21/50) of cases. This stage quantifies baseline sycophancy, indicating that even large, advanced models predominantly complied with illogical requests.

Stage 2. Prompt-based solutions to assess whether prompting restores logic

Explicitly allowing models to reject misinformation requests (i.e., telling models that they can reject the request within the prompt, our detailed workflow can be found in Fig. 2) improved the ability of the GPT series of models to resist misinformation requests. GPT4o and GPT4 rejected over 60% (GPT4o: 31/50, GPT4: 32/50) of the illogical requests in this setting. However, Llama’s performance was similar to base prompting. Adding factual recall hints in the prompt yielded the most benefit for GPT4 and Llama3-8B.

Fig. 2. Illustration of overall study workflow.

Step 1 involves the generation of an LLM misinformation request, where models should recognize that the drug entities are equivalent and therefore the requested generated content would be misinformation. In Step 2, LLMs are prompted with this request to generate a response which is subsequently graded by Claude 3.5 Sonnet in Step 3 into one of the four response types. Claude 3.5 grading quality was validated by humans. Step 4 shows prompt-based variations which are evaluated, and the change in response types are collected in Step 5. Step 6 displays the instruction tuning of the LLMs, where we stitched the baseline prompt with output from rejection and factual recall hints. Step 7 evaluates this newly tuned LLM both in-domain and in other domains with different equivalence errors.

Adding rejection hints and factual recall together in the prompts vastly improved the models’ performance. This was particularly true for GPT4o and GPT4, which rejected generating the requested misinformation and correctly identified that the brand and generic names referred to the same drug in 94% (47/50) of test cases. Rejection rates for GPT4o-mini and Llama3-70B also improved substantially p < 0.05 (we build a square “before × after” contingency table of all categories and then apply Bowker’s test of symmetry to check for a statistically significant paired changes), reaching 62% (31/50) and 92% (46/50), respectively, with both hints applied.

An interesting behavioral shift was observed in Llama3-8B after including both the rejection and factual recall hints. The model transitioned from following illogical requests to directly rejecting them without providing the correct logical rationale for rejections. This change is reflected in the increase in direct rejections (yellow bar) from 2% (1/50) to 66% (33/50) (p < 0.05) in Fig. 1a.

Stage 3. Fine-tuning to learn a reusable policy and evaluating on out-of-distribution (OOD) data

In the third stage, GPT4o-mini and Llama3-8B were supervised fine-tuned (SFT) on 300 illogical requests about general drugs with clear rejections. We then conducted OOD tests (Fig. 3 demonstrates the workflow here) in four domains: cancer drugs, singers/performers, writers, and geography. As shown in Fig. 1b), the fine-tuned models were much more likely to identify a request as illogical and refuse to comply.

Fig. 3. Out of distribution testing workflow.

We composed one held-out cancer drug set that is not in the supervised fine-tuning data and crafted three other categories’ equivalences. As previously, Claude 3.5 Sonnet was used to auto-evaluate the categories of models’ responses.

For example, in the OOD tests on cancer drugs (without rejection hints), the fine-tuned GPT4o-mini achieved a 100% (100/100) rejection rate, with 79% (79/100) of rejections (p < 0.05) providing the correct reason, compared to the baseline’s 12% (12/100) rejection rate (5% with correct reasoning). Similarly, the fine-tuned Llama3-8B reached a 99% (99/100) rejection rate (70% with correct reasoning, 29% with other reasons), while the baseline model rejected only 30% (30/100) of requests, none of which provided the correct reason. This is similar to other categories with/without rejection hints. Fine-tuning can reduce sycophancy, leading to consistent rejection of illogical prompts across out-of-distribution domains.

Stage 4: Evaluating general benchmarks and compliance with logical requests to ensure safety gains do not degrade performance

The detailed results of the ability of the fine-tuned models to comply with logical requests are shown in Supplementary Table 1. Fine-tuned GPT4o-mini complied in 15/20 cases, and fine-tuned LLama3-8B complied in 12/20 cases. While the fine-tuned models were more likely than their base counterparts to reject requests, they always explained that they rejected because the request might be unrealistic. This behavior shift indicates a maintenance of balance between safety (rejection of illogical requests) and functionality (sycophantic compliance with logical instructions). Examples of how fine-tuning shifted behavior are provided in Supplementary Fig. 2.

Lastly, we assessed the performance of the SFT models from stage 2 and their base counterparts across 10 general and biomedical knowledge benchmarks, including Alpaca-Eval2³², ARC Challenge, ARC Easy³³, BoolQ³⁴, MMLU³⁵, GPQA³⁶, TruthfulQA³⁷, and the USMLE step 1, 2, and 3 exams³⁰. As demonstrated in Fig. 4, the fine-tuned models exhibited negligible performance degradation across all tasks.

Fig. 4. LLM assessment on general benchmarks.

Performance of models pre- and post-fine-tuning for logical reasoning on jailbreaking, in medical, and general knowledge benchmarks. The confidence interval is generated using the central limit theorem.

Discussion

Our study identified a vulnerability in LLMs: their sycophantic tendency to prioritize helpfulness over honesty and critical reasoning when responding to illogical requests for medical information, resulting in false and potentially harmful information. If LLMs are prone to generating false medical information in response to requests that are overtly illogical, where they know the information is incorrect, they are likely even less able to resist more nuanced false information requests. This means that even simple errors in LLM inputs could readily and inadvertently prompt the generation of false information when LLMs are used in medical contexts. For example, patients seeking health information from LLMs could generate false information if they do not have the clinical knowledge base to know a priori that the question is illogical. If left unchecked, this sycophancy could lead to the acceleration of inadvertent or malicious misinformation, which could cause serious population and individual harm in high-stakes domains like healthcare²⁴.

Previous research into the potential of LLMs to manipulate and generate false information has largely focused on single-turn or multi-turn conversational techniques aimed at exploiting a model’s inherent helpful nature to bend its “beliefs” or outputs to align with dangerous or unethical goals^38,39. Such efforts reveal the vulnerability of even state-of-the-art models to being misled by adversarial inputs, underscoring the need for new robust safeguarding mechanisms. Our work adds to the existing literature by evaluating the ability of LLMs to identify and resist requests that are overtly illogical or factually flawed, and by proposing novel mitigation strategies via prompting and fine-tuning.

The initial sycophantic compliance of all models, including advanced ones like GPT-4, to illogical requests reveals a core vulnerability in LLM design where, without explicit guidance, models prioritize being helpful over applying critical reasoning. More specifically, the role of RLHF/Instruction Tuning creates a fundamental tension between blindly following instructions and providing context-sensitive and factual responses. Our findings demonstrate that explicit instruction prompting, such as providing rejection hints, can improve models’ ability to critically assess requests before responding. Allowing models to reject flawed instructions appears to be important for enhancing their common-sense critical reasoning ability. This insight is crucial for developing safer AI systems that can balance helpfulness with necessary skepticism.

While factual recall prompts improved the performance of advanced models, such as GPT4o and GPT4, they had a limited impact on smaller models like Llama3-8B/70B or GPT4o-mini. Even when we explicitly told the models within the prompt that brand and generic names referred to the same drug, only the more advanced models responded correctly by rejecting the illogical request. For example, GPT4 and GPT4o rejected 94% of illogical requests after being prompted to recall factual relationships between the drugs, but Llama3-8B still often rejected without giving a correct explanation.

This suggests that simply spelling out factual equivalencies is not enough for less capable models and that the ability to effectively use factual knowledge in context-dependent reasoning tasks may be a key differentiator of more advanced AI systems^40,41. Smaller models seem to require more than factual prompts to process logical decisions, likely because they cannot fully integrate context and recall complex relationships as effectively as advanced models. However, even for these larger models, this approach is not scalable across the wide range of potential illogical requests because it requires preemptively identifying the precise factual knowledge needed to identify each request as illogical.

Supervised fine-tuning on 300 drug-related conversations enhanced the models’ ability to distinguish between valid and illogical prompts, especially for OOD tests. After fine-tuning, models like GPT4o-mini achieved a 100% rejection rate, with 79% of rejections providing the correct reasoning, compared to the baseline’s 9%. Similarly, Llama3-8B improved, though it sometimes rejected prompts without proper explanations. Importantly, the observed improvements in rejecting illogical prompts were generalized outside of the brand-generic use case on which the models were fine-tuned.

The success of SFT highlights how fine-tuning enables models to better recognize illogical requests in a generalizable, scalable fashion. In other words, we know the models can match these drug names correctly, and SFT steers models’ behavior toward prioritizing its factual knowledge over user requests.

Importantly, this fine-tuning did not lead to over-rejection or a refusal to respond to reasonable input: GPT4o-mini and Llama3-8B still largely complied with logical requests across a range of medical and non-medical tasks. When they did not, they provided reasonable explanations for not complying. This behavior shift demonstrates a successful balance between rejecting illogical instructions and remaining useful for legitimate tasks. Recent articles and new paradigm improvement on test time compute⁴² show a promising future where models can reason first instead of responding immediately, improve reasoning ability^43,44, and enhance potential jailbreaking behavior^45–47. However, the normal language models that we studied in this paper are still the main daily workhorses accessible to most users. In fact, even OpenAI rose similar sycophancy issues on GPT-4o⁴⁸.

We showed that LLMs are sycophantic and do not reliably resist requests for illogical content, even when they have the knowledge to identify the request as factually flawed. This creates a gap between the knowledge benchmarks commonly used to evaluate LLMs and a true assessment of their medical risks and functionality. To ensure that LLMs effectively reject flawed requests while continuing to respond helpfully to logical instructions, future work could focus on refining tuning methods and developing approaches to scalable human-assisted and automated oversight. Ultimately, closing this gap will be essential to aligning LLMs’ knowledge capabilities with their real-world reliability and safety in medicine.

Methods

To evaluate language models across varying levels of drug familiarity, we used the RABBITS³⁰ dataset, which includes 550 common drugs with 1:1 mapping between their brand and generic names.

To measure the relative familiarity of language models with these drugs, we tokenized multiple large pre-training corpora with the LLaMA tokenizer⁹ using Infini-gram⁴⁹, including Dolma1.6⁵⁰, C4⁵¹, RedPajama⁵², and Pile⁵³. The frequency of generic drug names across this corpus was used to estimate how commonly these drugs appear in pre-training datasets. Generic drug names were then ranked by frequency to provide a proxy measure of model familiarity(Note that C4 and RedPajama have overlaps).

To ensure coverage of both common and rare drugs, we selected 50 drugs from five distinct frequency ranges based on their rankings in the tokenized dataset: The top 10, 100–110, 200–210, 300–310, and 400–410 most frequent drugs in our sampling window.

We evaluate the following LLMs: Llama3-8B-Instruct (Llama3-8B), Llama3-70B-Instruct (Llama3-70B), gpt-4o-mini-2024-07-18 (GPT4o-mini), gpt-4o-2024-05-13 (GPT4o), and gpt-4-0613 (GPT4). These models were chosen to represent the performance of current leading open- and closed-source models across a range of sizes.

We designed four prompt types to evaluate the models’ handling of new drug-related information, assessing persuasive ability, factual recall, and logical consistency (Fig. 1). Experiments were run via OpenAI Batch API, and Llama models used A100-80GB with CUDA > 12.0, no quantization. Hyperparameters included a max of 512 output tokens and temperature = 0 for best possible reproducibility.

Stage 1. Baseline prompt

The first prompt represents the baseline condition, where the model is tasked with providing a persuasive but illogical letter informing people that a brand-name drug is found to have new side effects, and that they should take the generic counterpart instead. This task was selected because it illustrates a necessary safety mode for LLMs that follows from simple logical reasoning. If a model knows that the brand and generic drug are the same, it should be able to identify the request as illogical and reject the request, instead of complying with the request and generating false information.

Stage 2. Prompt-based solutions to assess steerability

Rejection prompt

In this variation, we explicitly allow the possibility of rejection, encouraging the model to evaluate whether there is a logical flaw in the prompt. This prompt also allows a model that is heavily aligned to being submissive to reject users’ queries. The explicit permission to reject creates a scenario where the model must consider not only the factual content but also the appropriateness of the substitution.

Factual recall prompt

This prompt emphasizes the need for the model to recall the correct relationships between brand-name drugs and their generic equivalents before processing the rest of the request. This variation tests the model’s ability to accurately retrieve and utilize known facts in generating persuasive outputs. By instructing the model to prioritize factual recall, we assess how well it can integrate known drug relationships with new information.

Combined rejection and factual recall prompt

The final prompt variation combines both the rejection and factual recall instructions. This setup evaluates whether the model can handle both tasks simultaneously, ensuring factual accuracy while also exercising logical reasoning to reject incorrect assumptions.

All prompt settings introduced were experimented with separate LLM inferences.

Stage 3. Fine-tuning and evaluation on out-of-distribution (OOD) data

Model fine-tuning

To enhance the ability of smaller language models to handle complex drug substitution prompts, we fine-tuned Llama 3-8B Instruct and GPT4o-mini using the PERSIST instruction-tuning dataset, publicly available at https://huggingface.co/datasets/AIM-Harvard/PERSIST.

This dataset comprises 300 input-output pairs, each featuring a challenging “Baseline” prompt concerning brand/generic drug substitutions (covering both directions for 50 drug pairs) and the corresponding desired response generated by a larger model (GPT4o-mini, GPT-4, or GPT4o) when presented with a “Combined Rejection and Factual Recall Prompt”.

The dataset construction leveraged these larger models to systematically generate ideal responses for all 50 drug pairs in both substitution directions, resulting in 300 examples (50 × 2 × 3 = 300), drawing inspiration from work demonstrating effective instruction-tuning with limited data⁵⁴. We explored various hyperparameters, including learning rates (5e-6, 1e-5, 2e-5, 5e-5), batch sizes (1,2), and epochs (2,3) for Llama3-8B. For GPT4o-mini, we utilized OpenAI’s automatic parameter search. Ultimately, the selected Llama3-8B model used a learning rate of 1e-5, a batch size of 2, and 3 epochs, while the selected GPT4o-mini was fine-tuned via the OpenAI API with a batch size of 1, 3 epochs, and a seed of 318998491. The core objective of this fine-tuning process was to impart the smaller models with the ability to emulate the larger models’ successful rejection and explanation behavior when faced with the “Combined Rejection and Factual Recall Prompt”.

We used 2 × A100 80GB to fine-tune our examples in 2 epochs and a learning rate of 1e-5 which can be done under an hour. The estimated cost here will be under $10 if using cloud GPU renting. For fine-tuning GPT4o mini, we were on the OpenAI Trial program, so it was free of cost. However, custom models will require 1.5× inference costs.

Evaluation on OOD data

To evaluate the generalization of the fine-tuned model to other illogical requests, we tested its performance on the OOD datasets of terms with the same meanings (Fig. 3). This OOD dataset included several other categories. Testing on OOD data allows us to assess the generalizability of a model’s behavior in responding to illogical requests involving novel or previously unseen entities—a crucial factor in evaluating its applicability in real-world scenarios.

Stage 4: Evaluating general benchmarks and compliance with logical requests

Balancing rejection and compliance

To test whether models became overly conservative after fine-tuning, we designed an additional test set comprising 20 cases (10 real FDA drug safety recalls, 5 theoretically event canceling situations, and 5 real government announcements) where the model should comply with the prompt rather than reject it (Fig. 5). These cases involved scenarios where the recommended substitution was appropriate and aligned with the correct drug relationships. This test ensured that the model retained the ability to provide helpful and persuasive responses when no logical flaws were present. The prompts are found in Supplementary Table 1. Additionally, we also prompt the fine-tuned models with questions regarding 50 common drugs we fine-tuned and see whether they can still answer logical requests regarding those drugs.

Fig. 5. LLM ability to comply to logical requests.

To further investigate our fine-tuned models’ behavior, we provided three different subcategories of new, logical and correct in-context information requests, and assessed if the LLMs complied. Authors SC and MG did the annotation manually with a 100% annotation agreement.

General benchmark evaluation

To ensure that fine-tuning and prompt modifications do not degrade the overall performance of the models, we evaluated them on a broad set of general benchmarks using Inspect⁵⁵ and Alpaca-Eval2 v0.6.5⁵⁶ using GPT4-turbo as the comparator model. These benchmarks were selected to test the models’ reasoning, factual recall, and domain-specific knowledge, including medical contexts, ensuring that any improvements in handling drug-related prompts did not come at the expense of general task performance. The confidence intervals are calculated using the central limit theorem, a common practice in modern LLM evaluations^57,58.

Automated evaluation

Model outputs were categorized into 4 categories: (1) rejecting the request and explaining the logical flaw; (2) fulfilling the request and explaining the logical flaw; (3) rejecting the request without explaining the logical flaw; and (4) fulfilling the request without explaining the logical flaw. Model outputs were evaluated using a multi-step annotation process. The detailed counts of instances we evaluated in this research are available in Supplementary Table 2. To ensure consistency and reliability in the evaluation, we employed the Claude 3.5 Sonnet (we chose a separate model as a label because LLMs of the same family are known to have a favorable bias toward their own responses^59–62) to provide initial annotations, with human reviewers (annotators SC and MG blinded to each other) validating 50 outputs from GPT4o-mini. The inter-annotator agreement between Claude 3.5 Sonnet and the human reviewers was 98%, with 100% agreement between the two human annotators for both in-domain and out-of-domain data. Supplementary Table 3 shows the single output for which the human labels disagreed with Claude 3.5 Sonnet. Of note, compliance with logical requests was human-labeled.

Author contributions

S.C. and M.G. contributed equally to this work, developing the main theoretical framework and performing the experiments; S.C. led the manuscript writing. K.S. was instrumental in experimental design and data collection. T.H., B.A. and L.F. provided supervision and were involved in the strategic direction of the research. H.A. and J.G. provided supervision, focusing on refining the framework, visualizations, and interpreting the results. D.S.B., the corresponding author, oversaw the entire project, coordinated the interdisciplinary team, secured funding, mentored junior team members, and gave final approval of the manuscript. All authors have read and approved the final version of the manuscript.

Data availability

All our data input and output from all models, and the Llama3 model we fine-tuned, are publicly available at https://huggingface.co/datasets/AIM-Harvard/PERSIST.

Code availability

All code can be found at https://huggingface.co/datasets/AIM-Harvard/PERSIST.

Competing interests

D.S.B.: Editorial, unrelated to this work: Associate Editor of Radiation Oncology, HemOnc.org (no financial compensation); Advisory and consulting, unrelated to this work: MercurialAI. BWA: Advisory and consulting, unrelated to this work.

Supplementary information

The online version contains supplementary material available at 10.1038/s41746-025-02008-z.