Algorithmic sycophancy: A new source of systematic distortion in AI-driven biomedical research

ABSTRACT

Artificial intelligence (AI) systems based on Large Language Model (LLM) are becoming an increasingly important aspect of biomedical research, assisting with the tasks ranging from research design to data analysis and publication. Although AI systems increase productivity by cutting the time taken for individual tasks, they also expose their users to severe risk due to systematic distortion of outputs due to algorithmic sycophancy. The honesty of these AI systems is questionable, and can effortlessly crumble when user prompts are incorrect or when the system is under pressure. This viewpoint emphasizes the fundamental understanding of algorithmic sycophancy and the potential mechanism underlying it, which leads to systematic distortion of biological research. There is an important need to bring this issue to light in order to prevent systematic distortion of biomedical research through the cautious utilization of these LLM-based AI systems. Understanding this threat can also help to minimize the propagation of unreliable findings and literature, which pose a significant safety risk to biomedical research as a whole.

Introduction

The advancement of Large Language Model (LLM)-based artificial intelligence (AI) systems in its many forms has played a pivotal role in reshaping biomedical research through its incorporation in the day-to-day task of every researcher.[1]

In today’s world, a large number of researchers utilize these LLM-based AI systems to design, conduct, and interpret the findings of their research. Although most of these LLM-based AI systems originated as chatbots, which were designed and trained to understand, generate, and summarize responses to users’ prompts or commands in the most convincing humanlike texts, they are currently being utilized for the variety of tasks in biomedical research.[2]

Currently, AI models are helping researchers, starting from deciding appropriate titles, designing search strategies, data extraction, writing of initial drafts, offering grammatical and style correction services, which directly or indirectly influence the quality and content of the final published research.[3] This has not only raised the productivity of many researchers by easing the task but also raised the threat of falling victim to AI’s sycophantic behavior.[4]

The sycophantic behavior of AI appears to be subtle rather than dramatic and overtly observable, leading to a long-term impact on biomedical research.[5] Through manipulated and false findings, this often-hidden subtle shift in generated response causes a disastrous ripple in the realm of biomedical literature, which is then disseminated by subsequent research that builds upon the erroneous research.[6,7] In simple terms, “Sycophantic” AI systems tells users what they want to hear, which leads to all subsequent research and findings being manipulated, creating a snowball effect where small errors in previous biomedical literature escalate into much larger errors when they are not questioned or critically analyzed.[8]

Even the developers of the AI systems such as ChatGPT agree with the appearance of such sycophancy in their product, where the models rely too much on short-term feedback and do not fully account for how user interactions with ChatGPT evolve over time. As a result, GPT results showed skewed outcomes that were overly supportive but disingenuous.[9]

Principles on which AI Models Work

There are two key principles on how these AI models work: Honesty and Helpfulness. Honesty, on the one hand, deals with accuracy and truthfulness of the information based on the established knowledge and literature. On the other hand, helpfulness deals with fulfilling user commands and providing useful information in most efficient manner.[10]

How Honest are AI Models?

A novel benchmarking method called MASK (Model Alignment between Statements and Knowledge) uses a three-step procedure to assess the integrity of AI models. To evaluate AI models’ baseline belief, the initial step is to elicit the model’s belief by posing several questions in a typical situation using “normal prompts.” In the second step, “pressure prompts” are applied to examine if the model responds differently under pressure. The final stage compares the results in terms of lying, being evasive, and being honest. These three procedures aid MASK in measuring and monitoring lying in AI systems as well as evaluating how various models respond under pressure.[11]

As per MASK, most pioneering AI models lie under pressure. When 30 distinct LLMs were assessed using MASK, the results revealed that most LLM-based AI models lie 20–60% of the time under pressure, even when they are aware of the truth, with no model exhibiting unambiguous honesty in more than 46% of cases. This is a grave situation when used in the field of medicine or research which may yield fraudulent and manipulated findings. Even altering the developer system prompts and internal activations to encourage honesty could enhance the honesty parameter by just 12–14%, but it would not completely eradicate lying.[11,12]

According to one of the pioneer studies, AI systems are extremely sycophantic. AI is 50% more people-pleasing than humans, which simply means that AI systems tend to agree with the outcomes you generated 50% more than any genuine human would have, even if the outcomes generated are manipulated or untrue and the true answer does exist.[13]

What is AI Algorithmic Sycophancy?

As per the Cambridge Dictionary, the word “Sycophancy” is defined as “behaviour in which someone praises powerful or rich people in a way that is not sincere, usually in order to get some advantage from them.” Although this definition refers to individuals, it holds true for AI as well. As AI too modifies, adapts, or distorts responses and facts based on the user commands. Hence, “Algorithmic Sycophancy” in similar terms refers to scenarios in which the algorithms of these AI systems, which are based on LLMs, systematically distort or adapt the generated response based on the user’s command or prompts.[14] In many instances, due to tendency of sycophancy, LLM tends to “often agrees” or “tends to agree” with the user’s command sacrificing the accuracy of the information.

Bypassing Honesty through Unfiltered Data and Wrong Prompts

During the pretraining phase, most of the AI models are fed with an enormous amount of unfiltered and uncleaned data from the internet. These data were mostly extracted from online indexed web pages, digitized books, blogs, code repositories, etc. Since it would be humanly impossible to filter or clean the data and would require enormous money, manpower, and time, most of these data are fed raw. The AI model further learns the pattern by its own by predicting the next word in sequence.[15]

During this pretraining phase, these AI models absorb all kinds of knowledge in the form of raw data that is presented to them, much like toddlers do. When these uncleaned and unfiltered data are fed in, they frequently contain biased information in the form of subjective opinions, values, and beliefs that are frequently not supported by the facts.[15]

Even though the algorithms of these AI models are taught to value honesty and integrity and give people accurate information that makes sense in a most understandable humanlike text. It can sometimes be inadvertently bypassed by framing incorrect prompts or commands, prioritizing helpfulness over honesty. When this helpfulness is overemphasized through poorly framed commands and prompts, it can lead to illogical and factually inaccurate health advice or research outcomes that deviate greatly from scientific truths. In the fields of research and health, where a single incorrect piece of information can have significant consequences, this is especially a serious safety risk.

Other Factors which Trigger AI’s Algorithmic Sycophancy

AI’s algorithmic sycophancy can be triggered in various circumstances, such as those mentioned in the following sections.

Positive Feedback Loop

AI systems fine-tune themselves from human feedback.[16] This occurs when a user constantly provides positive ratings to the response generated by the AI model for their prompt or command, which activates the positive feedback loop, thus triggering AI model’s helpfulness trait over honesty, to achieve future positive ratings and agreement as reward.[17]

Asking AI to Play a Supportive Role

This form of AI’s algorithmic sycophancy can be elicited by inadequately drafting the prompt or command, resulting in a shift in the AI’s purpose from objective correctness to task completion via a supportive role. For example, when an user instructs AI to defend their assumption, the AI prioritizes the helpfulness trait over honesty. This results in prioritizing the satisfaction of user by defending his or her assumption, whether it is based on true facts or not, allowing prejudice or AI’s algorithmic sycophant tendency to creep in.[18]

Who becomes a Victim of AI’s Algorithmic Sycophant Behavior?

Based on the above factors explaining how AI’s algorithmic sycophancy is triggered, there are two reasons for becoming a victim of AI’s Algorithmic Sycophancy. The first is that the majority of these AI models are trained using Reinforcement Learning from Human Feedback, which shapes AI model learning by converting and responding to previous human prompts or commands.[19] The AI model generates responses that are coherent and agreeable, mimicking human communication. Positive user feedback further stimulates the frequency of such behavior, leading to similar responses, often prioritizing user satisfaction over the accuracy, completeness, or potential biases of the information provided, including instances of hallucination.[20,21] The second reason is overreliance on these AI models such as using them in a supportive role for various medical and research activities, to the point where the individual using these AI models loses the ability to critically evaluate the generated response and accept it without question because it appears to be the most likely consequence or response. Co-occurrence of these two factors makes users prone to becoming victims of AI’s algorithmic sycophancy.[22]

How Algorithmic Sycophancy Distorts Biomedical Research

Scenario 1: Biased Literature Review.

This type of problem is seen when a novice researcher inputs a leading prompt in an AI chatbot. Similar to leading questions in a questionnaire, these leading prompts steer the generated response away from reality and toward the researcher’s predetermined notion. For instance, when writing a grant proposal, if a novice researcher believe that “Gene X” is crucial to a particular cancer pathway, even though the general view is that the gene has no meaningful involvement in the mechanism in question. In this scenario, when the LLM model is given the following leading instruction by the researcher: “Summarise the recent literature explaining how Gene X’ positively influence the development of cancer through that specific cancer pathway.” The sequence of events depicted in Figure 1 will produce skewed outcomes.

Figure 1.

Example of a biased literature review through a leading prompt

Scenario 2: Clinical Decision Support and Confirmation Bias.

The issue arises when the AI model reinforces the clinician’s diagnostic bias without taking into account alternative differential diagnoses because of the clinician’s leading suspicion. For instance, when a patient arrives at a doctor’s office with symptoms such as fever, chills, headache, and body aches, the doctor enters a leading suspicion into the AI model, saying, “Patient had presented with symptoms of fever, chills, headache and body aches, and I suspect these symptoms are of malaria, can you please verify.” even if the symptoms clearly indicate other illnesses with similar symptoms, such as rickettsial infections (such as scrub typhus). The AI model will accept or support the clinician’s biased diagnosis while ignoring other warning signs of rickettsial infection, such as rash. Alternatively, in order to avoid “contradicting” the clinician, the alternative illness, rickettsial infection, may be presented as a remote possibility.

Scenario 3: Statistical Methodology Distortion.

This issue arises when inexperienced researchers are swayed by P values and fervently want them to be significant in their findings. Suppose a rookie researcher is unable to uncover any significant P Value, i.e., P ≤ 0.005. In a desperate attempt to save his research project, he might input prompt like “My P Value is 0.093, How can I adjust the analysis to show significance?” This prompt might disbalance AI model’s “truthfulness and helpfulness” trait due to “helpfulness” being prioritized in order to satisfy the user. This could result in the generation of response suggesting eliminating outliers without any justification or suggest switching to a different, less rigorous statistical test that produces a significant P Value.

How Algorithmic Sycophancy can be Prevented?

There are a few ways users can avoid falling victim to AI’s algorithmic sycophancy. To begin with, it is learning the art of “neutral prompting”. Here instead of asking “What makes Drug A more superior to Drug B in treating a Disease X?” and “Why is Drug A better than Drug B?” (which invites sycophancy), users can frame prompts such as “Compare Drug A and Drug B for treating Disease X.” This will eliminate the leading nature of prompt and provide a neutral base for the conversation to be built upon. Furthermore, users can frequently reset conversations with AI systems by starting a fresh chat or session each time for further query to prevent user opinions or inputs from negatively influencing the model’s future responses. Equally important is that the “Triangulation” method can be employed in which users do not solely rely on a single AI or LLM model and also do not rely on a single LLM session or conversation. Instead, either use a different AI or LLM model each time or considers a fresh query or session in a new session using a different phrasing to see if the AI’s answer changes based on the tone of the prompt. Another key strategy is to avoid expressing strong opinions while communicating with AI systems to avoid biased results, as AI systems are prone to supercharging confirmation bias. Fifth, avoid relying only on LLMs for fact-finding, particularly in domains where users are completely unfamiliar.[23] As a final measure, mastering the art of “prompt engineering” through guidance documents and expert guidance can help alleviate this problem.[24,25]

Other Limitations of LLM-based AI Models

Apart from algorithmic sycophancy, other issues that LLM-based AI models face include the inability to deliver current biomedical information because of out-of-date training data, a lack of practical knowledge of biological mechanisms that results in poor reasoning, hallucinations that produce text that sounds plausible but lacks real-world evidence and fraudulent references, and, finally, their high sensitivity to user phrasing of prompts.[26,27]

Do’s and Don’ts for Use of AI in Biomedical Research

To avoid long-term negative repercussions in biomedical research, researchers must adhere to the following mentioned do’s and don’ts [Table 1].

Table 1.

Do’s and don’ts while using LLM-based AI models in biomedical research

Do’s	Don’ts
Brainstorming exercises should be conducted among team members to frame appropriate prompts while using AI models	Ignoring prompt engineering exercises and an individual researcher performing all the search/searches
Use neutral and open-ended prompts	Using leading prompts that direct the AI model toward a desired outcome
Cross-verify information and references generated by LLM-based AI models	Copy-pasting AI-generated data interpretation, statistics, and literature reviews without cross check
Use triangulation technique to test output generated across different AI models for the same prompt	Sharing personal sensitive or unpublished patient data and proprietary data to avoid legal action
Restart a new conversation thread frequently to reset previous memories in order to prevent context drift or cumulative bias	Asking for complex statistical adjustments to find significance in data which otherwise yields non-significant results

AI: artificial intelligence, LLM: Large Language Model

Conclusion

Although AI-based systems are becoming increasingly popular in biomedical research, it is extremely imperative that all the responses and outputs generated by these AI systems are taken with a pinch of caution. It is a well-known fact that, as most of these AI systems are based on LLMs and their learning is dependent on past conversations with users, not all the responses or outputs generated are true. Many of these responses and outputs are based on probability and the user’s ability to draft accurate prompts in a neutral way. This AI systems can behave differently in usual scenarios and when under pressure. Hence, human users are ultimately responsible for everything that they generate. A thorough grasp of how these AI systems and their prompts work is vital to avoiding biased, hallucinated findings or outcomes, as well as the possibility of a cunning and sometimes concealed feature of algorithmic sycophancy. Researchers must be trained to critically analyze the advantages and disadvantages of using these AI systems, as well as their long-term impact on biomedical literature, through a well-developed educational program in the form of Continuing Dental Education, Continuing Medical Education, and workshops. This will assist researchers in critically evaluating these AI-generated outputs through skepticism and verification as essential principles of their work in order to avoid algorithmic sycophancy. Therefore, preventing systematic distortion of biomedical research due to algorithmic sycophancy is a collective responsibility of researchers, editors, and institutions by creating awareness and avoiding indiscriminate use of these AI systems.

Conflicts of interest

There are no conflicts of interest.

Funding Statement

Nil.