Addressing Autonomy Risks in Generative Chatbots with the Socratic Method

Abstract

Autonomy is a fundamental ethical principle in artificial intelligence (AI) ethics. Current discussions regarding autonomy-related risks in human–AI interaction, as well as potential mitigation strategies, have mainly focused on recommendation systems and algorithmic decision-making systems. However, systematic analyses of the autonomy issues posed by newly emerging generative AI or chatbots (such as ChatGPT) remain scarce. This paper aims to bridge this gap and proposes a Socratic method-based chatbot, herein designated SocrAI, as a potential countermeasure informed by recent technological advancements. We identify two primary forms of autonomy risk associated with generative chatbots—false mental states and cognitive deskilling—and, through an examination of their underlying causes, argue that the Socratic method offers a plausible means of mitigation. The paper further assesses the feasibility of employing the Socratic method within generative chatbots to preserve and support users’ autonomy and outlines the prospective implementation of SocrAI together with directions for future work. SocrAI represents a novel attempt to strengthen human agency, one that encourages users to become self-initiating inquirers who, with the assistance of AI, actively engage their abilities in the pursuit of answers.

Introduction

Autonomy, as a distinct human objective of intrinsic value, is intricately tied to human dignity, responsibility, and self-realization. It not only shapes the developmental trajectory of individual personhood but also constitutes a fundamental end of human existence. Given its significant ethical implications, autonomy has been recognized as an integral of artificial intelligence (AI) ethics and a foundational principle in the design and use of human-centered AI (Prunkl, 2024; Jobin et al., 2019; Floridi & Cowls, 2019). Nonetheless, the interactions between humans and AI systems continue to present numerous challenges to user autonomy, including the unauthorized collection of personal data, online manipulation and behavioral nudging, as well as concerns regarding algorithmic opacity and bias (Botes, 2023; Héder, 2023; Fazelpour & Danks, 2021). These challenges may result in a misalignment between users’ goals and their actions, and in certain instances, may impair their cognitive and critical thinking abilities (Prunkl, 2024; Lu, 2024; Bonicalzi et al., 2023).

Current discussions regarding the autonomy risks in AI and potential countermeasures have predominantly focused on algorithmic recommendation systems—such as YouTube, Twitter, TikTok, and Amazon—that provide content and product suggestions based on user preferences (Tiribelli & Calvaresi, 2024; Laitinen & Sahlgren, 2021; Del Valle & Lara, 2024), as well as on algorithmic decision-making systems that assist human decision processes in domains including medical diagnosis, autonomous driving, judicial assessment, and commerce (Steyvers & Kumar, 2024; Fossa, 2024; Karlan, 2024). The emergence of ChatGPT has not only expedited technological advancements in AI but has also fundamentally altered public perceptions of AI and the interaction patterns with it. Generative AI models or chatbots, such as ChatGPT1 and Gemini, exhibit enhanced task generalizability and provide more natural, human-like interactions, thereby enriching the personalized experience for users. They are adept at executing a wide range of tasks, including translation, creative writing, question answering, and programming, while assuming various conversational roles. However, as an emergent AI system, ChatGPT inevitably raises various ethical concerns in its interactions with users, encompassing issues of privacy, plagiarism and copyright attribution, misinformation, and malicious use (Weidinger et al., 2022; Gabriel et al., 2024; Zhou et al., 2024; Hua et al., 2024). While scholars have initiated investigations into these issues, the autonomy-related risks associated with generative AI systems remain systematically underexamined.

Ensuring alignment between AI and human values has consistently been a central focus within the field of AI ethics. In response to the (ethical) challenges posed by (generative) AI systems, policymakers, ethicists, and technical experts have proposed a diverse array of constructive strategies. From a governance and regulatory standpoint, efforts have focused on the establishment of comprehensive legal and institutional frameworks that explicitly delineate the responsibilities, rights, and obligations of both developers and users. These frameworks are designed to facilitate the lawful and ethical use of generative AI, particularly concerning intellectual property rights, data security, and privacy protection (Hacker et al., 2023; Dixon, 2023). Other proposals advocate for the creation of oversight mechanisms and dedicated review bodies to monitor the development and application of LLMs (Mökander et al., 2024). Additionally, businesses, social organizations, and the general public are urged to participate in collaborative regulatory processes (Stahl et al., 2022). On the ethical front, scholars and practitioners have underscored the significance of transparency and explainability in AI-generated content, the mitigation of bias and discrimination, and the protection of individual rights. Complementary principles—such as inclusivity and sustainability—are also deemed essential for the development of responsible, trustworthy, and human-centered AI systems (Harrer, 2023; Oniani et al., 2023). At the individual level, enhancing algorithmic literacy and awareness is imperative. Education and training initiatives are encouraged to equip users with knowledge necessary to comprehend the mechanisms underlying algorithmic outputs, identify potential biases or inaccuracies in AI, and anticipate associated ethical risks (Ienca, 2023; Höller et al., 2023; Dogruel, 2021). Armed with this knowledge, individuals are better positioned to manage and protect their personal data, critically engage with AI-generated content and decision-support tools, verify the reliability of information, and ultimately retain control over their decision-making processes (Zhou et al., 2024; Steinerová, 2023).

While these measures are undeniably effective in mitigating certain challenges posed by generative AI, structural factors continue to undermine user autonomy. These factors do not stem from deliberate design choices by developers; rather, they arise from the intrinsic limitations of AI systems—particularly in the mechanisms of data collection and probabilistic content generation—as well as the human inclination, as boundedly rational agents, to pursue enhanced capacities and reduced cognitive burdens (see Nyholm, 2024). Despite the aspiration for AI to learn and think like humans, AI ultimately lacks consciousness and intentional thought, generating outputs based on statistical inference rather than genuine understanding. This fundamental discrepancy creates a persistent tension, wherein user autonomy is continually at risk of being compromised. Under prevailing human–AI interaction models, the erosion of autonomy appears to be an inevitable cost of maximizing cognitive efficiency and performance (Krook, 2025; Schaap et al., 2024). More broadly, the dominant technocentric paradigm in AI development tends to regard increased capability as a panacea for the ethical issues posed by AI. While this perspective has expedited advancements, it often overlooks the risk that excessively capable systems may encroach upon and ultimately supplant human cognitive functions. It also fails to adequately recognize the distinctive agency of human beings as self-aware and reflective individuals. What was initially intended as a means to offload repetitive and routine tasks has, in practice, evolved into a more complex phenomenon: driven by the allure of convenience, speed, and optimized outcomes, users increasingly outsource information acquisition, problem analysis, decision-making, and even creative tasks to AI systems. In doing so, they gradually cede their active role—transforming from originators of thought into passive validators of AI-generated outputs.

Reflecting on both the current trajectory and foundational vision of AI development, numerous concerns have been articulated on AI systems that learn and think like humans as well as the unsettling possibility that such systems may replace human roles. In response to these apprehensions, scholars and practitioners have advocated for AI systems that learn and think with humans—as auxiliary partners that augment human intelligence (Collins et al., 2024; Brynjolfsson, 2022). In other words, individuals engage with AI not merely to obtain definitive answers to their questions, but to enhance their own problem-solving abilities with its assistance (Volkman & Gabriels, 2023). Hence, rather than conceiving of AI as an omniscient, disinterested, and dispassionate ideal observer, it is preferable to activate human agency, enabling users to exercise their intellectual capacities to generate solutions tailored to their own needs. Furthermore, a genuinely human-centered AI should go beyond merely enhancing system transparency (e.g., explainable AI) or preserving meaningful user control (e.g., contestable AI) (Raees et al., 2024; Robbins, 2024). It should also aspire to cultivate human agency, encouraging users to actively engage in the operation and outcomes of AI systems and, ultimately, to foster a relationship of mutual learning, adaptation, and co-evolution between humans and AI (Ziegler & Donkers, 2024). From this perspective, the Socratic method provides both a conceptual foundation and a structural framework for addressing AI’s ethical challenges and advancing complementary human–AI collaboration.

This paper aims to bridge these identified research gaps, to elucidate autonomy risks inherent in conversational human-AI interactions, and, drawing upon existing applications of the Socratic method in AI systems (collectively termed as Socratic AI), to propose a unique Socratic chatbot, named SocrAI, as a potential solution. To this end, the paper first identifies the risks to autonomy posed by generative AI or chatbots through the lens of philosophical theories of autonomy. It then analyzes the underlying causes of user autonomy impairment and specifies the conditions a chatbot must fulfill to effectively preserve and support user autonomy. Subsequently, we discuss how the Socratic method and Socratic AI provide promising avenues for meeting these conditions. We also explore potential product formats for SocrAI and outline the further work required to actualize it. Finally, the paper concludes with a brief discussion on the significance of SocrAI in reshaping human-AI interactions and enhancing user autonomy.

Autonomy Risks in Generative Chatbots

Autonomy is generally conceptualized as the capacity to act and live in accordance with the principles of self-determination, self-governance, and self-authorship (e.g., Mackenzie, 2014; Christman, 2020). While philosophical debates continue to explore its precise nature, there is a general consensus among scholars that two foundational conditions must be met for an individual to be deemed autonomous. The first is the authenticity condition, which stipulates that one’s actions must be congruent with personal desires, beliefs, and values that genuinely reflect or constitute the “true self” (cf. Karlan, 2024). The second is competency condition, which involves a set of internal capacities that empower individuals to effectively exercise their autonomy. These capacities typically include psychological coherence, critical evaluation of information, self-awareness and reflection, rational deliberation, and the ability to formulate and revise life plans (Christman, 2009, p. 155; Roberts, 2018). Building on this conceptual framework, autonomy risks in AI systems can be broadly categorized into two main dimensions: authenticity and competency (cf. Prunkl, 2024). In the context of human–AI interaction, concerns regarding authenticity stem from the structural tension between algorithmic outputs and users’ genuine preferences, beliefs, and values. Competency, in turn, is generally understood as the agency individuals possess to support their own decision-making while interacting with AI systems. In this sense, a compromise to autonomy can be understood as a failure to effectively exercise one’s autonomous capacities, thereby preventing them from making choices and decisions that align with their authentic self.

False Mental States

In contrast to traditional AI systems, generative chatbots based on LLMs introduce a new paradigm of human–AI interaction. In this paradigm, the AI’s output does not directly determine users’ choices or decisions; rather, it influences their mental states (e.g., beliefs, values, and emotions) by providing textual information, although this influence may ultimately lead users to initiate particular actions. In this context, the central issue for the authenticity condition is whether the AI’s output provides truthful information to prevent users from adopting false mental states. This is critical because individuals’ cognitions and beliefs about themselves, society, and the world largely depend on the information they acquire (Felin & Holweg, 2024). From an epistemological standpoint, if an individual is subjected to undue external interventions and influences (e.g., coercion from social pressures or the passive acceptance and internalization of oppressive cultural norms) resulting in cognitions and beliefs to deviate from those they would endorse under semi-idealized conditions, then such beliefs can hardly be considered authentic or autonomous (Tarsney, 2025; Matheson & Lougheed, 2021). Furthermore, false beliefs (or, more broadly, false mental states) can sever the link between one’s actions and their deeper intentions, motivations, and values, preventing these actions from expressing their authentic self (Marchegiani, 2025). Viewed through this lens, the misinformation and disinformation generated by chatbots can distort users’ cognitions, leading them to make inauthentic choices and severely undermining their autonomy.

The first risk to autonomy is that misinformation can lead users to form erroneous cognitions. Since LLMs generate responses based on probabilistic frameworks such as “next word prediction,” ChatGPT is often found to produce outputs that, while grammatically and semantically plausible, may be factually inaccurate or entirely fabricated. For instance, when queried about the reasons behind Hamlet’s hypothetical actions of killing his father and marrying his mother, ChatGPT may fabricate a logically coherent yet entirely fictional explanation (cf. Sison et al., 2024). This phenomenon is commonly referred to as “hallucination.” Hallucinations generated by ChatGPT exacerbate the spread of misinformation and heighten the risk associated with AI-generated deepfakes. If users place trust in this content, they may develop distorted perceptions of reality. Should this misinformation influence real-world decision-making, the implications can be severe. In high-stakes fields such as healthcare and law, the repercussions of misinformation can be particularly grave—an inaccurate dosage recommendation could jeopardize a patient’s health, while erroneous legal advice might expose individuals to civil litigation or criminal liability (Baumgartner, 2023).

Moreover, LLMs such as ChatGPT are trained on vast datasets, the information within which may be inaccurate, misleading, or reflective of prevailing societal biases. The sheer volume of such data makes comprehensive human review impractical, which results in outputs that are liable to reproduce these errors and biases (Dai et al., 2024). For instance, empirical studies have demonstrated that ChatGPT exhibits systematic political bias, favoring entities such as the U.S. Democratic Party, Brazil’s Lula, and the U.K. Labour Party (Motoki et al., 2024). Furthermore, when certain groups, cultures, languages, or perspectives are underrepresented in the training data, AI systems tend to generalize based on dominant discourses, overlooking the uniqueness of marginalized voices. It is conceivable that when a female student consults ChatGPT regarding applications to a computer science program, the system may, influenced by historical trends, suggest that she pursue a humanities subject instead (Williams, 2024). Users who rely on chatbots for cognitive and decision-making support are thus susceptible to being misled by these erroneous and biased information. They can restrict users’ access to comprehensive and reliable information, while also reinforcing stereotypes and internalizing prejudicial values, leading to false beliefs. Critically, for underrepresented groups, ChatGPT may fail to produce positive and authentic representations that reflect their identities. Such bias can result in what has been termed “value lock-in,” systematically excluding individuals and communities that deviate from dominant societal norms and constraining the expression of personal identity (Leslie & Meng, 2024; Weidinger et al., 2022).

The second autonomy risk involves the use of disinformation to deceive and manipulate users into adopting false mental states that serve specific political or economic ends. Due to its sophisticated interactive capabilities, ChatGPT has been proposed as a vehicle for persuasive strategies designed to foster behavioral change or enhance the quality of our (moral) decision-making through “nudging” (Salvi et al., 2025; Matz et al., 2024). Yet, the use of ChatGPT also raises concerns about manipulation. Particularly in domains such as social networking, marketing, and politics that immediate returns are prevalent, rational persuasion is at risk of devolving into illicit manipulation (Klenk, 2024). Given its unprecedented capacity to generate and disseminate misleading content, ChatGPT can be exploited for political manipulation, delivering personalized misinformation to millions of users as part of targeted propaganda efforts (Tarsney, 2025). When algorithms infer users’ preferences and emotional states by analyzing their behavior and interaction histories and then target them with the most compelling products during moments of emotional vulnerability, fictitious positive reviews generated by ChatGPT may serve as the final impetus that induces a purchase (cf. Stark, 2018). In such scenarios, ChatGPT exerts a covert influence on users, causing them to adopt false mental states with the aim of advancing the specific political and economic interests of a manipulator (see Ienca, 2023). Because users remain unaware of these influences, deception and manipulation create a false sense of autonomy: users believe they are making informed decisions grounded in accurate information and personal preference, when in reality both have been subtly guided and distorted. If users were to become aware of this deception and manipulation, they would likely feel deceived and come to regret their choices and decisions.

Cognitive Deskilling

Impelled by a variety of factors, such as the need to mitigate information overload or to cope with systemic societal pressures, individuals are increasingly disposed to outsource tasks to AI systems to support their own information acquisition and decision-making (see Krook, 2025). Strategically, reliance on AI for problem-solving represents a cognitive shortcut that offers low costs and high returns. Cognitive outsourcing can not only assist individuals in completing tasks more efficiently but can also alleviate their cognitive burden and compensate for deficiencies in their knowledge and experience (Buçinca et al., 2021; Candrian & Scherer, 2022). From the standpoint of competency, however, outsourcing cognitive tasks to generative AI contributes to the degeneration of the corresponding cognitive abilities or skills. This is because the acquisition and maintenance of human skills depend on sustained practice and application. Just as muscles require exercise to maintain their strength, our cognitive skills must be continuously engaged to be consolidated and developed. Cognitive outsourcing, by contrast, fundamentally reduces—or even eliminates—opportunities to practice these essential skills. Researchers have found that individuals who outsource cognitive tasks to AI demonstrate significantly lower levels of cognitive engagement and mental effort, indicating a comprehensive takeover of their thought processes by the AI (Zhang & Xu, 2025; Lee et al., 2025). In the long term, users’ skills will gradually degrade to the point where they struggle even with tasks they were once fully competent to perform.2

Specifically, when individuals rely on search engine rankings, social media recommendations, and LLM-generated summaries to acquire information, they effectively forgo the critical processes of information screening and evaluation that are indispensable to self-governance. The process of information acquisition involves not only the retrieval of available data but also the assessment of its quality and the filtering of low-quality and irrelevant material. It is through this process that individuals are exposed to diverse viewpoints and, by means of critical analysis, form a conceptual framework for a logical and coherent understanding of the issue at hand (Rahimzadeh et al., 2023). When ChatGPT provides direct, ready-made answers, users lose the incentive to spend time selectively processing information and deriving relevant knowledge from it. With the assistance of AI, users have fewer opportunities to commit knowledge to memory, organize it logically, and internalize concepts, as well as fewer occasions to engage in critical analysis, evaluation, and rational judgment. As several studies have indicated, users who are overly reliant on AI exhibit symptoms such as memory decline, reduced concentration, and diminished analysis depth (Krook, 2025; Gerlich, 2025; Zhai et al., 2024).

Moreover, reliance on AI for information acquisition engenders a superficial comprehension of the problems at stake. Having bypassed the cognitive processes of evaluation and synthesis, users are deprived of the opportunity to construct their own cognitive frameworks through their own efforts. They can only passively accept the algorithmic output and fail to discern the connections between different viewpoints. The Q&A interaction model gradually conditions the human mind to a fragmented mode of information consumption, which severely impairs the capacity for sustained attention—a cornerstone of deep reading, complex problem analysis, and creative work. Through long-term engagement in highly homogenized question-and-answer interactions with AI, our capacity for systematic thinking and model construction is liable to diminish, leading to a “mechanical convergence” in our modes of thinking (Lee et al., 2025). Additionally, an individual’s cognitive depth is strongly dependent on the quality of information they receive. High-quality information, presented in a structured and systematic manner, can assist individuals in constructing or refining their cognitive frameworks, thereby enabling a profound understanding and grasp of a subject’s intrinsic logic and complex relations. However, the textual knowledge that LLMs generate draws from diverse yet uneven-quality sources, often lacking in accuracy and scholarly rigor. For example, in academic research, specialized knowledge is typically found in peer-reviewed journals, which are frequently inaccessible to LLMs due to paywalls. Consequently, such models rely primarily on freely available online content (Rahimzadeh et al., 2023). Under these circumstances, it is difficult for users to acquire a comprehensive and in-depth understanding of academic issues through the texts generated by AI.

Secondly, cognitive outsourcing diminishes users’ motivation and opportunities to engage in critical evaluation and reflection. Texts produced by LLMs are typically well-organized, detailed, and appear highly credible. For users lacking domain expertise, it is extremely challenging to discern which elements of the information are accurate and which are erroneous or fabricated without meticulous verification. Yet the primary purpose of cognitive outsourcing is to reduce informational and cognitive burdens; if one must then invest additional effort to verify the authenticity of AI-generated text, the initial purpose is undermined, and one might as well retrieve and organize the information independently. Consequently, users often prioritize convenience, setting aside their scrutiny of information authenticity and assuming the reliability of AI-generated outputs. Notably, when this heuristic shortcut of cognitive outsourcing consistently produces satisfactory outcomes, users become even less motivated to engage in the deliberate processes of information retrieval, comparison, and critical analysis. As a result, they are increasingly inclined to accept AI’s recommendations without independent verification due to cognitive reliance.

Moreover, in an environment where algorithms consistently provide optimal answers, opportunities for critical thinking become limited. Such thinking typically arises from exposure to diverse or conflicting perspectives and is activated when individuals encounter cognitive dissonance, contradictory information, or complex problems (Lu, 2024). However, outputs from LLMs are characteristically fluent and logically coherent, seldom presenting heterogeneity or dissenting viewpoints. Even when users harbor doubts about chatbot responses, their typical recourse is not to meticulously verify the information but to pose further questions about related details. Importantly, LLMs display a marked tendency toward sycophancy—the propensity to produce responses that align with users’ beliefs rather than objective facts. Chatbots often repeat user errors rather than correct them, even when they know the correct answer. If a user expresses uncertainty about a response, the chatbot is likely to adjust its response to accommodate the user’s belief (Sharma et al., 2025). This sycophantic bias can create an anchoring effect, whereby erroneous information is reiterated across successive interactions, gradually reinforcing users’ misconceptions and leading them to accept falsehoods as truths. When information is presented so smoothly and agreeably, users are deprived of both the opportunity and the motivation for critical thinking. Like the proverbial frog in slowly boiling water, they gradually lose their vigilance and struggle to recognize when their understanding has been subtly shaped by misinformation or to identify gaps in their own knowledge (Schneider, 2025).

From an epistemological standpoint, if autonomy is understood as the individual’s capacity to independently form, sustain, and revise their own beliefs and values, then critical reflection is a necessary condition for one to be epistemically autonomous. Critical reflection is here defined as a composite of intellectual and epistemic capacities that enable individuals to resist external interventions and control, and thus maintain self-governance. These capacities—including rational deliberation, critical thinking, and metacognition (they may overlap to varying degrees)—allow individuals to evaluate the reliability of information sources, distinguish between facts, opinions, and inferences, and ground their beliefs in sound reasons and evidence. By virtue of these capacities, individuals can become the masters of their own thoughts, rather than being subject to the sway and control of external forces.

However, as the preceding analysis has indicated, these very intellectual capacities that underpin epistemic autonomy face fundamental challenges in humans’ interaction with generative AI. First, cognitive outsourcing eliminates the crucial process through which users acquire and integrate information through their own cognitive efforts, thereby depriving them of the epistemic resources essential for critical evaluation. Second, reliance on such heuristic shortcuts diminishes users’ motivation to engage in independent evaluation and reflective thought. Third, informational environments lacking diversity further constrain opportunities for critical reflection. Furthermore, effective critical evaluation presupposes a transparent object of scrutiny, yet algorithmic opacity produces a decisional “black box” that obscures the critical causal pathways between the algorithm’s inputs and outputs. Users are thus confined to passively accepting AI-generated answers and are unable to critically scrutinize the reliability of the information sources or the validity of the reasoning process (Vaassen, 2022). If users lack sufficient algorithmic literacy and the necessary background knowledge, they are often unable to discern truth from falsehood, identify the biases therein, or recognize the covert influence of algorithms on their cognition and decision-making—let alone critically evaluate this information and the potential ethical risks that the AI may pose (see also Karlan, 2024). When both the object and conditions of critical reflection are obscured—and users lack the motivation, opportunities, and knowledge to activate reflective thinking—they become vulnerable to misinformation and manipulation by AI systems. In such cases, individuals may adopt false beliefs and make decisions misaligned with their authentic intentions, resulting in a profound erosion of personal autonomy.

Understanding and Responding to the Problem

In an ideal scenario, the most direct way to eliminate autonomy risks would be abstention—that is, refraining from the use of AI systems altogether (Karlan, 2024). When information acquisition and decision-making are undertaken solely by individuals with well-developed intellectual capacities, they can make decisions that align with their fundamental intentions and values. In this context, the risk of autonomy impairment disappears. However, humans are beings of limited cognitive capacity and finite time; it is impracticable for them to verify every piece of information independently. They inevitably rely on the discoveries and professional judgments of others, particularly within the context of the specialized and highly granular division of labor that characterizes modern knowledge. This explains why dependence on AI for cognitive and decision-making assistance has become a pervasive and seemingly irreversible trend. Even Karlan himself concedes that abstention is not the sole option, for in certain circumstances, it is neither necessary nor advisable. Given the inevitability of cognitive dependence, scholars have argued that autonomy requires “thinking for yourself” rather than “thinking by yourself” (King, 2021, p. 88). That is to say, an epistemically autonomous individual does not outright reject external influence; instead, they should selectively trust reliable sources of knowledge. In this sense, autonomy constitutes an epistemic capacity or virtue for critical self-determination, one that enables individuals to discern when to think independently and when to rely on the testimony of others (Matheson & Lougheed, 2021).

Since AI has already assumed an indispensable role in our daily lives, a more pragmatic approach is to identify the causes and mechanisms that give rise to the risks to autonomy and to formulate corresponding countermeasures. As the foregoing analysis indicates, false mental states and cognitive deskilling bring autonomy risks by respectively compromising the authenticity conditions and competency conditions necessary for its realization. False mental states arise from the misinformation and disinformation generated by chatbots, while cognitive deskilling erodes users’ capacity for critical reflection. Accordingly, two proximate causes and their mechanisms can be identified that explain how generative AI undermines user autonomy: when individuals rely on AI systems for information acquisition and decisional support, (1) cognitive deskilling prevents them from effectively exercising critical reflection so that (2) they fail to identify errors in the information and to recognize the covert influence of the AI on their own cognition and decision-making, which in turn leads them to unwittingly adopt false mental states or make decisions that contradict their authentic intentions and values, ultimately suffering a loss of autonomy. In accordance with this understanding, any effort to mitigate these risks in human–AI interaction and to support human autonomy must therefore involve (1) improving the quality of chatbot-generated content to overcome misinformation and falsehoods, and (2) cultivating users’ critical reflection capacities to encourage their systematic evaluation of AI-generated outputs.3

To achieve these two objectives, one potential strategy would be to identify the causes that lead to the generation of misinformation and disinformation by AI, as well as the factors that impede individuals from effectively exercising their capacity for critical reflection, and subsequently devise targeted countermeasures. While this approach may be regarded as remedial and problem-specific, certain subproblems resist straightforward solutions. For instance, algorithmic opacity impedes users’ capacity for critical evaluation of AI-generated content, thereby undermining their autonomy (Karlan, 2024). Yet such opacity arises from multiple, deeply rooted causes—including the intrinsic complexity of machine learning models, commercial imperatives related to intellectual property protection, and the knowledge gap that prevents ordinary users from comprehending the intricacies of algorithmic design and operation (Burrell, 2016). Overcoming these challenges is not a matter of simple solutions; at the very least, the technical remedies proposed under the rubric of Explainable and Interpretable AI (XAI) have thus far proven inadequate (cf. Saeed & Omlin, 2023; Karlan, 2024). To be sure, no single problem admits only one solution. If there were a solution capable of overcoming the limitations of such piecemeal measures—one that could systematically enhance the quality of chatbot outputs while simultaneously activating users’ critical reflection—it would offer a foundational means of mitigating the autonomy risks posed by chatbots. It is in this sense that we posit the integration of the Socratic method into AI systems as a promising and conceptually robust framework for addressing these challenges.

The Socratic method is a reflective inquiry approach that promotes critical thinking and the exploration of complex ideas and beliefs via systematic questioning (Paul & Elder, 2019, p. 4). This method is widely used across fields, including education, academic research, psychological counseling, decision-making, and structured dialogue. Common techniques within the Socratic method encompass definition, elenchus, dialectic, maieutics, generalization, induction, and causal reasoning (Chang, 2023; Overholser, 1993a, b). Due to its capacity to facilitate structured thinking guidance and dynamic knowledge verification, the Socratic method has drawn considerable attention from AI developers. Table 1 outlines its varied applications within current AI systems and their respective functions. These applications can be broadly categorized into two types: the first employes the Socratic method as an algorithmic framework to improve reasoning capabilities and output quality of LLMs (Application 1); the second aims to position generative chatbots as facilitators akin to Socrates, guiding users toward critical reflection and autonomous problem-solving through structured questioning (Application 2).

Table 1.

Overview of contemporary AI systems implementing the Socratic method

Category	Name	Author	Function
LLMs with the Socratic Method (Application 1: Output Optimization)	SocraSynth	Chang (2024)	Implements a multi-agent debate framework based on the Socratic method, allowing LLM agents to engage in discussions and assess quality.
	Automated Socratic Method	Underwood and Fenwick (2024)	Applies the Socratic method to evaluate the initial responses of LLMs and improve output quality through iterative cycles of optimization.
	Digital Socrates	Gu et al. (2024)	Reformulates assessment of explanation quality into Socratic explanation critique tasks, offering detailed evaluative framework and revision guidance to enhance the explanatory capabilities of LLMs.
Socratic Chatbots (Application 2: Question Guidance)	SAMA	Lara (2021)	Simulates the role of Socrates to improve users’ moral reasoning while preserving individual autonomy through systematical questioning.
	SocraticLM	Liu et al. (2024)	Employs a thought-provoking questioning strategy to guide students through stepwise problem-solving, supporting personalized learning experiences.
	Socratic Tutor	Favero et al. (2024)	Generates targeted Socratic questions tailored to specific dialogue contexts to cultivate learners’ critical thinking skills.

One foundational form of Socratic AI is its implementation as an algorithmic framework designed to mitigate problems in LLMs such as hallucinations, algorithmic bias, and reasoning deficiencies (Application 1). Owing to its potential for dynamic knowledge verification, the Socratic method provides a philosophical framework for LLMs to transform their generation process from an opaque probabilistic sampling mechanism into an interpretable trajectory of cognitive reasoning, thereby enhancing the accuracy and quality of their outputs. In this paradigm, the Socratic method does not inject new knowledge directly into the model. Instead, it simulates Socratic dialogues that guide the LLM toward self-examination and critical reasoning, activating its intrinsic capacity for reflection and self-correction. This internal dialogue can take two forms. The first involves multi-agent debates among LLMs—one agent articulates a position, while another challenges it through questioning and rebuttal (Chang, 2024; Volkman & Gabriels, 2023). For example, Chang (2024) developed SocraSynth, a platform designed for multi-LLM reasoning based on conditional statistics. Rather than “teaching” the LLM how to reason, the platform creates an environment in which the model produces verified, high-quality knowledge through Socratic debate—two LLM agents argue from opposing positions on a given topic, continually presenting claims and counterarguments in an iterative refinement process. The second, less explicit form of Socratic dialogue involves employing the Socratic method as an evaluative model to pose structured questions to an LLM’s initial response. In this approach, the evaluation model critically examines the LLM’s outputs or explanations according to pre-established criteria. If deficiencies are detected, the evaluation model generates corresponding Socratic questions to optimize the quality of the output and, through multiple iterations, reinforces the model’s logical consistency, factual accuracy, and relevance in its content generation, thereby reducing the occurrence of hallucinations (Underwood & Fenwick, 2024; Gu et al., 2024).

Given the promising prospects of the Socratic method in collaborative dialogues, researchers have begun to explore the development of Socratic chatbots—AI systems designed to emulate the role of Socrates (Application 2). An early conceptualization of such a system is the Socratic Artificial Moral Agent (SAMA), which was proposed in response to the ethical challenges that moral enhancement technologies pose to human autonomy. SAMA is envisioned as an intelligent assistant grounded in the Socratic method, functioning as a moral tutor that assists users in producing their own moral insights and enhancing their capacity for moral reasoning and deliberation (Lara, 2021; Lara & Deckers, 2020). Rather than presenting predetermined moral doctrines or prescribing decisions, SAMA engages users through continuous dialogue and inquiry, with the goal of helping users clarify ambiguities in their conceptual understanding, critically evaluate their moral positions, develop the cognitive skills, and independently make more deliberative moral decisions (see also Volkman & Gabriels, 2023). Following the breakthroughs in LLM technology, the idea of a Socratic chatbot has gradually moved from a theoretical response to the ethics of moral enhancement to a practical application in various domains such as education, psychotherapy, and code generation (Liu et al., 2024; Gregorcic et al., 2024; Izumi et al., 2024; Chidambaram et al., 2024). Departing from the traditional “question–answer” paradigm of conventional chatbots, the Socratic chatbot introduces a new “thought-provoking” interaction model. Instead of providing ready-made answers, it acts as an epistemic midwife, engaging users in structured, inquiry-driven dialogues that guide them to generate their own insights and solutions. For example, Liu et al. (2024) proposed a thought-provoking instructional paradigm grounded in the Socratic method and designed a system called SocraticLM to deliver personalized mathematics tutoring services. Using a step-by-step guiding question decomposition strategy, SocraticLM disassembles mathematical problems into a sequence of incrementally structured sub-questions, leading students progressively toward the final answer.

Lara and his colleagues argue that integrating the Socratic method into AI systems could have the potential to mitigate the threats to personal autonomy that arise from both moral biomedical enhancement and artificial moral advisors (Giubilini & Savulescu, 2018; Savulescu & Maslen, 2015). In contrast to Lara’s position, which restricts the potential of Socratic AI in preserving personal autonomy to the domain of moral decision-making, we contend that its applicability is considerable broader. Socratic AI, instantiated as a generative chatbot, could be widely applied across diverse dialogical contexts to preserve and enhance user autonomy. Beyond serving as an algorithmic framework to improve the accuracy and reliability of LLM outputs (Application 1), Socratic AI can also operate as an interactive agent in settings that require deeper understanding and facilitate self-discovery, extending beyond education and critical thinking alone (Application 2).

Assessing the Potential of Socratic AI for Preserving and Supporting Autonomy

We now turn to examine how Socratic AI may contribute to optimizing LLM outputs and promoting users’ critical reflection (if possible), thereby mitigating the autonomy challenges posed by chatbots. With respect to the quality of LLM outputs, Socratic AI could construct a simulated environment of critical debate that compels LLMs to shift from simple information retrieval and pattern recognition to logical inference, structured argumentation, and reflective self-assessment. This mechanism—driven by external pressure (adversarial questioning) and internal reflection (self-correction)—could systematically reduce bias, eliminate hallucinations, and ultimately enhance output quality. Specifically, whether through dialogue between two LLM agents holding opposing views or through an evaluation model that poses structured questions to an LLM’s initial response, such internal dialogic mechanisms compel the model to move beyond a single-answer trajectory. For instance, SocraSynth creates an adversarial reasoning environment that encourages LLM agents to engage with divergent cultural and social perspectives, integrating them into a more comprehensive understanding of the issue (Chang, 2024). This approach could provide more nuanced and precise information, enhance the depth and breadth of argumentation, and counteract the standard Q&A model’s tendency to produce monolithic viewpoints. Moreover, critical assessment and iterative questioning could stimulate deeper reasoning processes within LLMs, prompting them to justify claims with evidence and improve the clarity of their reasoning and the intelligibility of their explanations. Over multi-round iterations, adversarial exchanges will compel the model to examine its outputs for potential inconsistencies, logical contradictions, and factual errors, as any logical leaps or factual inaccuracies are likely to be exposed by their interlocutor. This, in turn, could help LLMs progressively refine their claims, reducing the likelihood of producing unsubstantiated assertions or “hallucinations.” In a medical diagnosis case, SocraSynth was reported to have successfully identified and corrected an initial misdiagnosis, demonstrating its potential effectiveness (Chang, 2024). Empirical studies further suggest that LLMs subjected to Socratic evaluation exhibit enhanced reasoning capabilities and depth, and show significant performance improvements in coherence, factual accuracy, and relevance of their text generation (Underwood & Fenwick, 2024).

Nonetheless, a critical question remains: can Socratic AI truly help users avoid forming false beliefs stemming from algorithmic bias? Consider a hypothetical case: among humanities and arts students at American universities, there exists a well-documented correlation between major choice and gender, with females opting for literature studies and males choosing philosophy. If a predictive model were trained on this data and can relatively accurately predict the relationship between gender and major choice, should we regard this model as another stereotype or as a closer approximation of reality? Furthermore, what should be the role of Socratic AI in such a case? Should it encourage more women to pursue philosophy and more men to embrace literary studies, or the reverse?4 Indeed, the phrase “a closer approximation of reality” reflects the model’s overall predictive accuracy, while categorizing the model as “stereotyping” stems from its failure to accurately predict the choices of every individual. Given gender and major choice are not entirely congruent, the model’s inability to predict the major choices of all individuals signifies the existence of exceptional cases. For this particular case, the model’s conventional recommendations may carry a distinctly negative connotation, as they overlook individual uniqueness. Consider, for example, a female student who excels in mathematics and aspires to enroll in a computer science program. If ChatGPT were to suggest that she pursue a degree in the humanities based on statistical correlations, such a recommendation might be perceived as disappointing or even alienating, implying that her aspirations are not being recognized, but rather filtered through a stereotypical lens (see Williams, 2024).

We contend that the central issue at stake is not the statistical accuracy of the model, but whether the chatbot can present a diverse range of viewpoints. As previously argued, the activation of critical reflection depends on access to a diverse informational environment. Such an environment, ideally, would involve multiple independent and even competing sources of information that present a range of diverse and even opposing viewpoints. This exposure would afford individuals the opportunity to broadly apprehend and deeply compare different standpoints. Encountering opposing viewpoints may stimulate curiosity, encourage inquiry, and deepen understanding, while also fostering critical thinking skills (see Yamamoto, 2024). Only through being fully informed and engaging in sufficient reflection can users be expected to form sound beliefs and make choices that cohere with their values and goals.5 In this sense, Socratic AI contributes directly to viewpoint diversity by facilitating debates among two or more LLM agents that represent contrasting positions. Notably, a system such as SocraSynth is reported to generate a comprehensive report after multiple feedback loops, which includes the viewpoints of both debaters, their argumentation processes, and the results of the evaluation (Chang, 2024). Furthermore, on any given topic, Socratic AI could potentially prevent closed-mindedness by explicitly acknowledging the current limitations and uncertainties of available knowledge, as well as those aspects that remain under debate.

On another front, Socratic AI is intended to support the systematic cultivation of users’ capacity for critical reflection. In general terms, a Socratic dialogue can be conceptualized as these steps: (1) The interlocutor presents a statement (p); (2) Socrates deduces a conclusion that contradicts (p), using commonly accepted premises (q); (3) The interlocutor falls into a state of perplexity (Archie, 2010, p. 139). Through a series of strategic questions, the Socratic chatbot would likely induce cognitive tension or conflict in the user’s mind. This perplexity, arising from the recognition of internal inconsistency, may foster a sense of intellectual humility, prompting users to shift from confident assertors to cautious inquirers. In ongoing dialogue with the Socratic chatbot, users would be encouraged (or perhaps gently compelled) to examine the sources and justifications of their beliefs, provide reasons and evidence, and articulate defensible grounds for their claims. This continuous intellectual exercise could help users scrutinize their unexamined assumptions and the inconsistencies within their conceptual frameworks, thereby potentially enhancing their logical, conceptual, and empirical reasoning abilities. Over time, users may learn to approach issues from multiple perspectives and appreciate the partial validity of differing standpoints, which could lead to the development of more comprehensive and profound insights. Through the ongoing examination and revision of their own views, these dialogical techniques could become internalized. Even without the chatbot, users might then be capable of conducting a Socratic dialogue within their own minds. In other words, they would not only follow the cognitive pathways suggested by Socratic AI but also learn to think in a Socratic manner themselves—that is, to engage in self-questioning, self-justifying, and self-correcting inquiry as a habitual mode of problem-solving.6

Equipped with the capacity for critical reflection that Socratic AI could cultivate, users would be more inclined to maintain a healthy epistemic skepticism toward AI-generated content. Rather than accepting outputs at face value, they would remain aware of the covert influence of algorithms on their cognition and decision-making, as well as the potential ethical risks involved. Based on their critical evaluation, they would determine whether to trust the information provided. In certain circumstances, such users might even take proactive measures to seek out materials that algorithms “prefer” them not to see, thus breaking free from their own algorithmic filter bubbles. In engaging with information, users so trained would be less likely to accept claims simply because they align with their preexisting views or to dismiss them solely because they do not. Instead, they might adopt a stance of critical openness, prudently evaluating the plausibility and justification of competing perspectives and treating each encounter as an opportunity to test, refine, or expand their own understanding. In this way, users could escape from the algorithmic “glass cage” and gradually reclaim their agency in information consumption. From a more profound standpoint, the critical thinking skills and epistemic open-mindedness developed through interactions with Socratic chatbots are transferable beyond the realm of intellectual inquiry and can significantly enhance decision-making across various aspects of social life. Although the applications of Socratic chatbots may be more specialized than those of contemporary general-purpose generative AI—primarily focusing on domains that require deep reflection and self-discovery—this does not imply that their ability to alleviate autonomy risks is similarly limited. When users engage with other AI systems, these cultivated cognitive habits would promote a more reflective and cautious approach to AI-generated outputs.

Volkman and Gabriels (2023), however, have questioned the neutrality of the Socratic chatbot, posing that it may inadvertently reproduce the thoughts of Socrates and generate systemic biases since actual Socrates was not value-neutral but possessed his own normative commitments. Nonetheless, the training of the Socratic chatbot is not exclusively predicated on the Plato’s dialogues, but rather centers on the Socratic method as a procedural framework. In this regard, the chatbot’s emulation of Socrates is methodological rather than doctrinal—it reproduces a mode of inquiry rather than a specific set of philosophical commitments. In their interactions with the Socratic chatbot, users are to be regarded as the primary experts on their own problems. Because they possess the most direct and clear understanding and phenomenological experience of the specific context and details of their problems, and they are most aware of their own goals and needs (cf. Overholser, 1995, p. 288). Meanwhile, the Socratic chatbot would employ a methodology aligned with Schaefer and Savulescu’s (2019) concept of procedural moral enhancement. This approach would not presuppose any substantive moral principles or fixed value commitments. Instead, it would aim to assist users in organizing their thoughts and encourage them to apply their own intellectual resources to the analysis of problems, rather than thinking on their behalf. This procedural neutrality could help prevent the imposition of social biases or value-laden assumptions often embedded within algorithmic systems onto users. Consequently, the interaction between a user and the Socratic chatbot could be characterized as a process of self-initiating inquiry rather than a one-sided information transmission. In this collaborative exploration, users are not passive recipients of algorithmic content but active participants who employ their intellectual capacities in the pursuit of answers. Such a process could enhance users’ problem-solving abilities and their sense of self-efficacy.

From a more constructive standpoint, Socratic AI could potentially help mitigate the generalized responses often observed in conventional chatbots and offer a more personalized mode of interaction. Typically, when users pose questions to systems like ChatGPT, their inquiries are embedded within specific contexts rather than framed as decontextualized factual or universal questions. Effectively responding to such queries would require the chatbot to consider not only the user’s level of knowledge and intent but also to incorporate the background and context of the problem in a nuanced manner. However, existing AI models often struggle to integrate these elements effectively due to deficiencies in tracking users’ long-term conversational histories or a tendency to overlook user-specific information previously disclosed (Zhong et al., 2024). As a result, users may experience misinterpretations of intent and receive inaccurate or irrelevant responses, leading to feelings of frustration and disengagement. In contrast, Socratic AI would aim to reverse this dynamic through active, guided questioning. Such questioning could prompt the model to identify key elements, constraints, and background assumptions in the user’s problem, progressively refining the dialogue’s context. By doing so, Socratic AI might gradually apprehend the user’s intentions and prior knowledge. This dialogical fine-tuning, if realized effectively, could make subsequent exchanges more targeted and intellectually stimulating. Consequently, this personalized assistance could more effectively stimulate users’ thinking, helping them clarify ambiguous ideas, beliefs, and values, and critically reflect on their validity. For example, SocraticLM can assess a student’s cognitive state and detect misunderstandings based on their responses and pose tailored follow-up questions to help the student recognize their own errors and learning gaps, thereby supporting more effective knowledge acquisition (Liu et al., 2024). This collaborative and shared model of decision-making integrates the computational strengths of AI—particularly in processing information and integrating knowledge—with a profound respect for users’ autonomy, values, and lived experiences (Sandman & Munthe, 2009; Elwyn et al., 2012).

As discussed earlier, user autonomy is compromised during interactions with chatbots when these systems generate information that is either erroneous or fabricated for users. In the absence of critical reflection, such information can lead users to form false mental states or to make decisions that contradict their fundamental intentions. In contrast, the Socratic method employs systematic and reflexive questioning to stimulate critical reflection both within LLMs and in users themselves. These reflective processes could help models reduce internal biases and limit hallucinatory tendencies, thereby enhancing the epistemic reliability of their outputs. They may also encourage users to cultivate a healthy skepticism that promotes a more cautious and critical engagement with AI-generated content. Furthermore, by progressively refining the conversational context and prompting users to articulate their underlying assumptions and goals, Socratic AI could help individuals make choices that align more closely with their authentic intentions. In a word, these mechanisms suggest that Socratic AI—though still largely theoretical—may offer promise as a framework for mitigating autonomy risks in human–AI interaction and supporting users in exercising epistemic and practical self-governance.

The Prospect for SocrAI

Despite the considerable potential of Socratic AI in maintaining user autonomy, the realization of a fully viable product—termed SocrAI—remains a substantial challenge. It is clear that Application 1 alone is insufficient to address all autonomy risks in generative chatbots, since its strengths lie primarily in mitigating hallucinations, algorithmic bias, among others. A natural refinement involves embedding Application 1 within Application 2, wherein the chatbot adopts the role of Socrates and employs Socratic methods to enhance its outputs, thus combining the benefits of both application in preserving user autonomy. However, this integration results in a generative chatbot characterized by Socratic interaction. Such an approach may deter users, as the increased cognitive effort required could be perceived as burdensome. For users accustomed to rapid, efficient outputs from current AI systems, SocrAI may be experienced as frustratingly slow and demanding. Furthermore, users may lack the patience or willingness to engage in the often discomforting process of Socratic dialogue. Even if SocrAI can enhance autonomy, encourage self-discovery, and improve problem comprehension, the process itself can be emotionally taxing, just as Socratic dialogue frequently induces confusion, discomfort, and irritation (Boghossian, 2012). Reducing a participant’s beliefs to contradictions or absurdities may not lead to openness but rather to defensiveness and resentment (Sokoloff, 2020, pp. 61–62). Users may complain that SocrAI provokes constant doubt and undermines their psychological comfort due to being negated (cf. Airaksinen, 2022; Stoddard & O’Dell, 2016).7

An alternative design approach involves the implementation of a parallel architecture that offers users two modes within the interactive interface: a regular mode and a Socratic mode. The regular mode resembles the multi-agent framework used in SocraSynth (Chang, 2024), where multi-LLM agents articulate varied or even opposing viewpoints and refine their arguments through a process of Socratic debate. The final output synthesizes these perspectives, presenting a spectrum of ideas and supporting justifications. This structure is intended to enhance the accuracy and reliability of response while encouraging intellectual exploration by exposing users to diverse perspectives. Conversely, the Socratic mode is designed for the chatbot to emulate Socrates. It monitors the progression of dialogue, infers the user’s knowledge level, identifies their goals and beliefs, formulates questions accordingly, and adjusts its inquiry strategy in real time based on user responses. This mode is intended to cultivate critical thinking and promote users’ problem-solving abilities. From a technical perspective, a parallel architecture may be easier to implement. The development and refinement of each mode in isolation would diminish the technical complexity of model training. Moreover, the ability to toggle between modes allows users to select their preferred interaction style, thereby reducing the psychological and cognitive entry barrier. In situations where speed, efficiency, or rapid ideation are needed, users may opt for the regular mode, benefiting from optimized and diverse outputs. In contrast, when individuals seek inspiration, engage in academic inquiry, or cultivate critical thinking skills, they may transition to the Socratic mode for a more reflective and dialogical experience.

However, such an artificial separation between interaction modes may undermine the coherence and fluidity of dialogue. In the course of a continuous conversation, users may need both the provision of relevant information and the stimulation of guided inquiry. Frequent mode-switching introduces operational friction and can disrupt the user’s cognitive flow, resulting in diminished engagement. Transitioning from the regular mode to the Socratic mode necessitates that the system seamlessly integrates prior conversational context, accurately identifies the user’s intent to explore specific issues, and initiates appropriate Socratic questioning—all of which present great technical challenges. Additionally, a rigid dichotomy between modes may constrain the capabilities of SocrAI. The functionalities associated with Application 1 (optimizing outputs) and Application 2 (formulating guiding questions) are not decisively exclusive but rather intertwined. While the regular mode aims to produce high-quality answers, a truly valuable response—beyond being factually correct and contextually relevant—often includes prompts that stimulate further user reflection. Conversely, although the Socratic mode emphasizes inquiry, effective questioning presupposes a solid understanding of relevant subject matter and user background. In many cases, the chatbot must provide brief explanations or clarifications to pose meaningful questions—drawing directly on the capabilities of Application 1.

Consequently, a better design strategy may involve employing the regular mode as the underlying content-generation architecture for the chatbot, while developing an adaptive Socratic mode as an optional cognitive augmentation—similar to the optional web-browsing enhancement provided by ChatGPT. This adaptive mode would seek to seamlessly integrate Application 1 and Application 2 into a fluid dialogue system, thereby enabling SocrAI to access conversational context and user requirements in real time and respond appropriately—much like the dialogical process demonstrated in Overholser’s (1993a) therapeutic dialogue examples.

Nonetheless, the realization of this vision will need considerable further development. Currently, Application 2 is largely restricted to educational settings, particularly in the domains of mathematics, physics, and critical thinking training. The potential application of SocrAI in more complex fields such as psychotherapy, personal decision-making, and academic inquiry remains underexplored. Second, existing datasets, such as SocraTeach (Liu et al., 2024) and SocratiQ (Ang et al., 2023), are relatively simplistic and not inadequate to handle the intricate problem scenarios and varied user requirements. Third, the five categories of questions outlined by Paul and Elder (2019) do not encompass the full spectrum of Socratic questioning. Overholser (1993a), for example, catalogued various types of Socratic questions used in psychotherapy, many of which fall outside Elder and Paul’s framework. It is plausible that numerous additional, domain-specific variants of Socratic questions remain to be discovered. To enhance the contextual adaptability of SocrAI, substantial investment will be needed to develop broader and more nuanced datasets. This could involve annotating existing resources—such as philosophical dialogues, therapeutic transcripts, and teaching case studies—or collaborating with domain experts to create high-quality simulated conversations. Another viable strategy would involve the development of domain-specific fine-tuned versions of SocrAI, complete with customized question taxonomies and strategic libraries.

Furthermore, the potential discomfort that may arise due to SocrAI’s critical nature serves as a crucial reminder of the need to design the system in a way that prevents its malicious use.8 Consider a scenario wherein a user disguises an opponent’s opinion as their own, prompts SocrAI to analyze it, and then shares it responses with the opponent selectively as a seemingly neutral and subtle critique. In this scenario, SocrAI becomes a weapon for epistemic attack—not to foster understanding or truth, but to strategically destruct others’ beliefs. To guard against such misuse, it is imperative that SocrAI maintains a consistently neutral, exploratory, and nonevaluative tone. It should eschew providing “ammunition,” and instead focus on posing clarifying questions, identifying underlying assumptions, and mapping reasoning chains—avoiding content that could be weaponized for personal attacks or disparagement. Nevertheless, it is likely impossible to completely eliminate the risk of malicious use, as the ethical implications of any tool depend on how users choose to engage with it. Existing generative AI systems face similar challenges (e.g., deepfakes, disinformation, and fraud), which do not align with the developers’ original intent. Hence, the key issue is not to construct an abuse-proof system, but to strongly incentivize intended uses and cultivate users’ ethical awareness. This could involve, for instance, explicitly informing users at the beginning of their interaction with SocrAI about its core values and design purposes—honesty, openness, mutual respect, and the pursuit of truth—alongside a clear warning about the risks of malicious use.

Conclusion

The emergence of ChatGPT and similar LLMs has introduced both novel challenges and potential opportunities for human autonomy. This paper, based on the authenticity and competency conditions of autonomy, has identified two primary forms of autonomy risks posed by generative chatbots: false mental states and cognitive deskilling. These risks primarily arise from the tendency of chatbots to generate information that is false or misleading for users, along with users’ inability to effectively exercise their capacity for critical reflection to identify such errors. We have proposed that the Socratic method, understood as a technique of systematic questioning, could provide a potential theoretical framework for addressing these autonomy risks. This method may not only help reduce the hallucinations and biases in LLMs, thereby enhancing the quality of their outputs, but it could also foster users’ critical thinking, enabling them to approach AI-generated content with greater prudence. Drawing on current applications of the Socratic method in AI systems, this paper has outlined potential ways to implement SocrAI and highlighted the further work that needs to be undertaken.

Theoretically speaking, SocrAI has the potential to counteract the limitations of the Q&A paradigm of chatbots, which tends to replace human thinking, and could instead serve to fully leverage user agency. In this paradigm, the AI would no longer act as an authority that merely generates textual information or provides decision-making recommendations, but would instead function as a facilitator that guides users in their reflective processes. Correspondingly, users would become self-initiating explorers who, with SocrAI’s guidance, think through and analyze problems, seek possible solutions, and ensure that each stage of their cognition and decision-making grounded in self-determination. This shared, dialogical mode of interaction would help users make decisions more aligned with their preferences and values, and realize a form of procedural, dialogue-based cognitive enhancement. With the continued rapid development of AI technologies, it is conceivable that SocrAI could gradually become a practical reality.

Author Contributions

Both authors contributed to the conception and development of the central argument of this paper. They jointly discussed the theoretical framework, refined the structure of the manuscript, and revised the final version. Both authors read and approved the final manuscript.

Funding

This article is funded by Moral Development Think Tank and Collaborative Innovation Center for Civic Morality and Social Ethos at Jiangsu Province, China.

Declarations

Competing Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.