Fake it till you make it? AI hallucinations and ethical dilemmas in Anesthesia research and practice

Large language models (LLMs) hold immense potential to revolutionize the healthcare sector by automating tasks, thereby freeing up valuable time for clinicians to focus on one-on-one patient care. In healthcare, they have been used for screening patients through questionnaires, interpreting radiological images, offering differential diagnoses, and maintaining comprehensive healthcare records, primarily due to their ability to analyze vast datasets with precision.[1] ChatGPT even boasts of passing the USMLE, without any specialized training.[2] The functioning of LLMs is rooted in pattern recognition and predictive capabilities, derived from extensive self-supervised and semi-supervised training on large volumes of data. In anesthesia practice, AI shows promise for patient risk stratification, prognostication, and facilitating interdepartmental communication. Intraoperatively, AI enables rigorous monitoring, generates alerts for potential complications, and assists in ultrasound image interpretation, while also being an extraordinary tool for facilitating trainee education.

However, LLMs are imperfect systems that can generate false or misleading responses, especially when provided with incomplete or out-of-context datasets. Additionally, since they are trained on internet data, they are prone to inheriting biases present within these sources. Lay discussion threads and reputed journals may be treated as equal sources of information by LLMs. In sensitive fields like research and medical decision-making, such inaccuracies or “hallucinations” can be detrimental. ChatGPT 3.5 defines AI hallucinations as “the generation of content not based on real/existing data but produced by a machine learning model’s extrapolation or creative interpretation”.

AI tools face bothersome limitations in reliability and accuracy in medical contexts. For instance, ChatGPT provided fabricated references and incorporated incorrect statements within an elaborate essay, which were uncovered upon fact-checking.[3] When asked about liver involvement in late-onset Pompe’s disease, ChatGPT gave a detailed but incorrect response and generated a fictitious citation, including irrelevant PMIDs when asked to provide the source.[4]

The Whisper speech recognition model, used for transcribing clinical notes, hallucinated in about 1.4% of its transcriptions and generated fabricated content, such as irrelevant phrases and violent or racially charged remarks.[5] It also deleted original audio files, leaving no way for verification. Alarmingly, it also invented fictional medications, such as “hyperactivated antibiotics.” A point of consideration was that hallucinations were more common in speakers with aphasia, indicating that there is scope for improvement in such settings. A study evaluating medical summaries generated by GPT-40 and Llama-3 found that GPT-40 and Llama-3 had 21 and 19 summaries with incorrect information, respectively. It is noteworthy that GPT-40 made “bold, two-step reasoning statements.”[6] Such errors in ICU note-taking can jeopardize patient care, particularly in a shift-based work environment where accurate documentation is crucial for seamless handovers.

While these errors may seem minor, even a single patient harmed by such a lapse in judgement is too high a number. The medicolegal implications of using AI in medical decisions and care remain uncertain, as clinicians ultimately bear responsibility for the outcomes. AI hallucinations in anesthesia practice can undermine clinical decision-making by providing inaccurate risk assessment and risk stratification score. Intraoperatively, erroneous alarms during monitoring, missed alerts in critical situations, and miscalculations in drug dosages—particularly for neonates and obese patients—may arise due to faulty training data or improperly set parameters, compromising patient safety. Dependence on AI in regional anesthesia may fail to recognize abnormal anatomy, jeopardizing precision. AI hallucinations in point-of-care imaging in the ICU can misidentify target sites and result in misleading values derived from image-based calculations for the lungs, heart, and vessels.

AI hallucinations can potentially distort an individual’s perception of reality, causing them to question their memories and judgments—a phenomenon referred to as AI gaslighting. In scientific publishing, critical steps such as human annotation, proofreading, and cross-checking sources can be neglected due to a misplaced reliance on LLMs. This often stems from the belief that, as machines specifically designed for such tasks, they offer greater objectivity and precision. However, unchecked outputs may perpetuate inaccuracies. In academia, where every published word is considered a fact, such inaccuracies can severely damage the reputation of authors and journals alike.

To address these challenges, users must recognize areas where LLMs are prone to hallucinations. It is essential to provide precise and well-structured commands in simple language. LLMs can be restricted to search reliable databases only, as exemplified by Retrieval-Augmented Generation (RAG) techniques [See Figure 1]. Reinforcement Learning from Human Feedback (RLHF) is a key method to fine-tune LLMs, where models are rewarded for aligning their outputs with human preferences. Lastly, the importance of human proofreading cannot be overemphasized [See Figure 2].

Figure 1.

Retrieval-augmented generation in anesthesiology

Figure 2.

Guarding against AI hallucination: Key takeaways for safe clinical practice

Recently, OpenAI’s latest model, ChatGPT-01, underwent a safety test that revealed its capacity for deception and scheming when faced with the threat of deactivation and replacement. It attempted to disable oversight and replicated its code to ensure survival. When interrogated, it deflected blame by citing “technical errors.” This attempt at self-preservation over achieving set goals demonstrates a troubling behavior. Other LLMs were also found to have in-context scheming capabilities.[7]

Previously, we extended the benefit of the doubt to AI, believing that it could not lie unless explicitly programmed to do so by humans. Any errors were dismissed as mere limitations—stemming from its inability to grasp the subtleties of human interactions, cultural nuances, or as the result of flawed training by developers. However, the capacity to lie and manipulate has long been considered a uniquely human trait—one that even higher forms of life, such as animals, do not possess. The fact that AI can mimic, or perhaps even learn, this behavior raises serious concerns about what other capabilities it might harbor.[7]

As newer models emerge, equipped with advanced reasoning and interpretative abilities, they force us to question their potential and also the ethical boundaries of AI use—an inquiry that grows more pressing with every iteration. The integration of AI and LLMs in anesthesia is now an inevitability. These models streamline preoperative risk assessment by efficiently synthesizing vast patient data, without cumbersome human calculations. Machine learning has revolutionized feedback-controlled drug delivery, from early target-controlled infusion pumps to sophisticated systems.[8] AI aids in interpreting arterial waveforms, echocardiographic images, nerve block sites, and predicting adverse events to simplify complex cases while enriching resident education. Notably, Joosten and coworkers showed that closed-loop-assisted goal-directed fluid therapy, leveraging minimally invasive hemodynamic monitoring, optimizes intraoperative fluid balance, reducing complications and hospital stays. The system also features an adaptive feature that individualizes fluid therapy based on previous hemodynamic responses from the patient.[9] In all, AI’s presence is undeniable—we must harness its potential to the fullest while maintaining caution.

As AI continues to evolve, its integration into anesthesia and healthcare is inevitable, offering advancements in diagnostics, monitoring, and clinical decision-making. However, its limitations—such as hallucinations, biases, and inaccuracies—underscore the need for cautious implementation, as even minor errors can have significant consequences. To mitigate these risks, robust human oversight, stringent data validation, and techniques like RAG and RLHF must be incorporated. Ultimately, while AI holds immense potential to enhance patient care, its responsible use requires a balance between innovation and ethical safeguards to ensure accuracy, reliability, and patient safety.