Artificial intelligence for clinical reasoning: the reliability challenge and path to evidence-based practice

Abstract

The integration of generative artificial intelligence (AI), particularly large language models (LLMs), into clinical reasoning heralds transformative potential for medical practice. However, their capacity to authentically replicate the complexity of human clinical decision-making remains uncertain—a challenge defined here as the reliability challenge. While studies demonstrate LLMs’ ability to pass medical licensing exams and achieve diagnostic accuracy comparable to physicians, critical limitations persist. Crucially, LLMs mimic reasoning patterns rather than executing genuine logical reasoning, and their reliance on outdated or non-regional data undermines clinical relevance. To bridge this gap, we advocate for a synergistic paradigm where physicians leverage advanced clinical expertise while AI evolves toward transparency and interpretability. This requires AI systems to integrate real-time, context-specific evidence, align with local healthcare constraints, and adopt explainable architectures (e.g. multi-step reasoning frameworks or clinical knowledge graphs) to demystify decision pathways. Ultimately, reliable AI for clinical reasoning hinges on harmonizing technological innovation with human oversight, ensuring ethical adherence to beneficence and non-maleficence while advancing evidence-based, patient-centered care.

Introduction

The rise of artificial intelligence (AI) marks a transformative era that looks set to reshape medical practice. With the advent of generative large language models (LLMs), especially those capable of producing chain-of-thought (CoT) narrative,¹ AI’s role seems to be shifting from merely supporting to potentially executing clinical reasoning. However, this raises a critical question: can CoT AI authentically replicate the intricate clinical reasoning required of human clinical decision-making?

Certainly, the literature continues to expand with studies applying generative AI to different problems in healthcare. However, followers of the topic will also be aware that many AI studies exaggerate the benefits and potential and downplay the drawbacks of AI. Overall, the specific value and reliability of generative AI in clinical reasoning remain underexplored and poorly understood.² Clinical settings are highly diverse, varying according to geography, resourcing, and the unique interpersonal dynamics of physician-patient relationships. Reliable clinical reasoning is fundamental for safe, high-quality care delivery in heterogeneous clinical settings, while reliable clinical AI for physician lies in the authentic clinical reasoning capability.

Here, we explore the current landscape of the reliability of generative AI for clinical reasoning, especially with respect to its potential to replicate and enhance human clinical decision-making. Our hope is to call for a reliable AI system towards future evidence-based practice.

What is the ‘reliability challenge’ of AI for clinical reasoning?

Clinical reasoning is a sophisticated process that integrates knowledge and experience to formulate accurate diagnoses and management plans. This dynamic and iterative process requires the synthesis of multifaceted and multimodal ‘data’ regarding individual symptoms, physical signs, imaging, laboratory test results, pathological examinations, risk factors, and medical histories. It takes years of rigorous training, consolidated with hands-on experience, for medical professionals to master this intricate skill. With the advent of LLMs, which appear capable of ‘reasoning’ based on giving answers to clinical questions entered as prompts into chatbots, some have speculated that AI looks poised to replace human clinicians by delivering faster and more accurate diagnoses. However, diagnostic inaccuracies from AI may constitute a clinical risk that is too high to be acceptable.

So what is the evidence base for AI being able to perform clinical reasoning? The existing literature highlights some impressive capabilities for LLMs in clinical reasoning tasks. For example, LLMs have been shown to pass the United States Medical Licensing Exam (USMLE).³ In further clinical case tests, LLMs were comparable to attending physicians and residents in terms of diagnostic accuracy and cannot-miss diagnoses. Indeed, LLMs achieved a median inclusion rate of 66.7% for cannot-miss diagnoses, similar to attending physicians (50.0%) and matching residents (66.7%).⁴

Despite this promise shown in clinical reasoning tests, LLMs also have notable limitations.⁵ LLMs commit more diagnostic errors than medical residents⁴ and misunderstand clinical terminology, such as illness scripts and problem lists, necessitating additional explanations.⁶ Furthermore, physician-AI chatbot (GPT-4) collaboration does not outperform physicians using conventional resources (e.g. UpToDate, Google).⁷ While AI may be more accurate in certain scenarios, it also generates a higher volume of errors, highlighting its practical shortcomings in clinical settings.

Critically, generating reasoned text or passing a medical exam does not equate to genuine logical reasoning. Researchers from Apple remind us that LLMs do not truly reason but instead mimic reasoning steps based on training data. A single additional clause in a question (prompt) can degrade performance by up to 65%,⁸ resulting in inconsistent and unreliable outputs. This sensitivity to prompts has also been observed in clinical reasoning examinations.⁶ Moreover, while LLMs can produce coherent reasoning texts, they lack access to high-quality regional medical data and up-to-date medical knowledge to ensure implementation of an optimal treatment plan. Retraining these models with the latest medical information is not only costly but also risks catastrophic loss or inadvertent assimilation of unreliable or misleading contents from the open web.

Call for AI toward evidence-based practice

Diagnostic inaccuracies from generative AI not only compromise patient safety in clinical settings but may also lead to inefficiency and misguided clinical decision-making.⁷ These errors violate the core ethnical principles of beneficence and non-maleficence, threatening patient safety and trust. Two critical factors have historically overcome these challenges: an improvement in digital tools and their effective implementation by humans.² Building on this foundation, we, therefore, propose a transformative paradigm where evidence-based practice hinges on a synergistic collaboration between physicians and AI. In this paradigm, physicians must remain equipped with the latest medical knowledge and sophisticated clinical reasoning skills, while AI must evolve to be more interpretable and transparent. To achieve this, AI systems must not only develop authentic clinical reasoning capabilities that mirror human cognitive processes but also access and present the most recent evidence aligned with local clinical contexts, ensuring transparency through traceable sourcing and relevance to region-specific healthcare challenges and resource constraints.

It is worth remembering that interpretable AI models predate the emergence of LLM. For example, Liang et al.⁹ developed a hierarchical logistic regression classifier that systematically classified the electronic health record, starting with broad organ systems, followed by anatomic categories, and finally specific diagnosis groups. Similarly, reasoning-based AI, supported by expert clinical knowledge, has been designed to emulate the deliberate and methodical nature of human clinical reasoning. Causal graph knowledge, for example, explicitly delineates causal relationships with uncertainties and subsequently performs probabilistic reasoning.¹⁰

A promising trend, rapidly gaining momentum, is the integration of interpretable components into LLMs. Emerging initiatives aim to combine LLMs with multi-step reasoning¹¹ or clinical reasoning graph knowledge to address the ‘black-box’ nature of purely diagnostic models.¹² These advances are critical because clinicians need to first comprehend the rationale behind AI-generated recommendations to trust and integrate these tools in their decision-making processes.

Conclusions

The reliability of AI for clinical reasoning is a critical challenge that can be effectively resolved through a collaborative approach between technology and human clinicians. The rapid advancement of AI in clinical reasoning, when combined with critical clinical thinking, could unlock the transformative potential of AI for evidence-based practice.

Author contributions

H.X. was responsible for the writing of the first draft of the thesis; H.X., Y.W., and Y.X. participated in the revision and text finishing of the first draft; Y.J. and R.S. were responsible for the selection of the thesis, structural design, data and information review, revision, quality control and review of the first draft of the article.

Conflict of interest: None declared.

Funding

National High Level Hospital Clinical Research Funding (2022-PUMCH-A-017); the non-profit Central Research Institute Fund of Chinese Academy of Medical Sciences (2022-ZHCH330-01); the non-profit Central Research Institute Fund of Chinese Academy of Medical Sciences (2021-RC330-004).

Patient consent for publication

Not applicable.

Data availability

Not applicable