Engagement patterns of middle school students with AI teachable agents in mathematics learning

Abstract

 This study investigates how secondary students engage with an AI teachable agent (TA) during mathematics learning, with particular focus on learners whose performance declined after interacting with the TA system. Using a mixed-methods design, we analyzed dialogue logs from two subgroups: a Declined Group (DG; n = 206), whose post-test scores decreased, and an Improved Group (IG; n = 327), whose scores increased. Analyses examined interaction modes and behavioral, emotional, and cognitive engagement. Passive interaction was most prevalent in DG (36 %), whereas IG more frequently demonstrated constructive interaction (62.78 %). DG exhibited high variability in behavioral engagement: although they completed more sessions and generated more utterances per session, their completion rate (0.35) was lower than that of IG (0.58). Regarding emotional engagement, boredom was the most frequent non-neutral emotion in DG (50.4 %) and tended to rise as sessions progressed, whereas IG expressed more positive (30 %) than negative emotions (17.94 %). For cognitive engagement, most students displayed surface-level knowledge acquisition with limited application to novel or complex tasks. Notably, within DG, greater behavioral activity and positive emotions were sometimes associated with lower learning gains, often when such activity reflected off-task dialogue or superficial goal completion. These findings highlight classroom challenges in AI-supported learning and suggest design implications for TAs that scaffold proactive interaction, detect emerging boredom, and redirect high-volume yet low-yield behaviors toward meaningful engagement.

Introduction

The integration of pedagogical agents in educational settings has earned increasing attention as a way to enhance student engagement and learning outcomes^1,2. Teachable agents (TAs), as pedagogical agents grounded in the learning-by-teaching paradigm, typically function as virtual peers designed to simulate less knowledgeable learners, thereby prompting students to engage in the cognitive processes involved in both learning and teaching³. TAs such as Betty’s Brain engage students in constructing and refining a concept map, which helps them understand subject matter more deeply and develop metacognitive skills⁴. The system was found beneficial when applied in a middle school biology setting⁵. Researchers also found that TAs can effectively teach younger children scientific reasoning, improving their understanding of organisms, taxonomies, and ecosystems⁶.

Recently, with the advancement of Generative AI (GenAI) technology, AI-supported TAs have introduced new opportunities in education. GenAI offers personalized learning experiences by providing tailored feedback, scaffolding, and interaction based on individual student needs^1,7,8. With recent advancements in natural language processing (NLP), many empirical studies have found that AI teachable agents play a supportive role for language learners, enhancing their engagement, scaffolding their learning, and helping them become self-regulated learners^8–10. Some studies also demonstrate the use of AI teachable agents for programming education were beneficial^11–13. Prior research has demonstrated that intelligent tutoring systems including AI teachable agent can be nearly as effective as human tutors in improving learning outcomes¹⁴. Studies have also shown that emotionally adaptive AI systems can reduce frustration and increase student motivation, leading to enhanced emotional engagement¹⁵, while interactions with AI tutors have been found to foster cognitive engagement and improve learning compared to traditional instruction¹⁶. Despite the increasing interest in AI teachable agents, there remains a gap in understanding how these systems impact the different dimensions of student engagement (i.e., behavioral, emotional, and cognitive) and how these engagement patterns relate to learning outcomes.

According to Fredricks and colleague’s work¹⁷, student engagement is a critical component of the learning process. Behavioral engagement involves active participation in learning tasks, such as attending sessions and interacting with instructional content^18,19. Emotional engagement encompasses students’ affective responses to their learning experience, including emotions like boredom, frustration, and curiosity²⁰. Cognitive engagement refers to the depth of students’ thinking and the strategies they employ to understand and apply knowledge²¹. Understanding the relationship between these dimensions of engagement and learning outcomes is essential for designing AI teachable agents that effectively support student needs. Besides, a recent review highlights the lack of clarity regarding how students engage with pedagogical agents in classroom settings and emphasizes the need for more qualitative or mixed methods research, which has been underrepresented in recent studies on pedagogical agents^1,7.

However, much of the existing research has focused on the overall functionality of AI systems (e.g., ), with limited attention in learning analytics on students’ interactions with these systems. Thus, the present study seeks to address the above gaps by exploring the interaction and engagement patterns of students using an AI teachable agent for math learning, specifically focusing on students who exhibited declines in learning performance after using the agent. Focusing on this subgroup is crucial because they are often the most vulnerable learners and require targeted scaffolding; if overlooked, educational technologies may fail to narrow gaps and can even widen inequalities across groups^22,23. At the same time, average gains attributed to AI tools can mask heterogeneous effects across student subgroups, so we must examine who actually benefits from a fairness perspective^24,25. Prior work also shows that when their needs are not addressed explicitly, certain groups (e.g., low- or high-achieving students) may see limited benefits or even negative effects²⁶. This study aims to uncover the engagement patterns that may have contributed to their lack of learning improvement and provide insights into the design of more effective AI teachable agents. We proposed three research questions:

1. RQ1: What interaction modes do students exhibit when engaging with an AI teachable agent during math learning?

2. RQ2: What are the patterns of student engagement (i.e., behavioral, emotional, and cognitive) in math learning with an AI teachable agent?

3. RQ3: How is student engagement related to their learning performance when using an AI teachable agent?

To address RQ1, we coded student utterances using the ICAP framework²⁷ and reported the frequency and proportion of each interaction mode. For RQ2, we computed behavioral engagement indicators (e.g., session count, mean session length, completion rate, and proactive-interaction count from RQ1). We then conducted an emotion analysis using seven categories commonly observed in mathematics learning (anxiety, frustration/confusion, boredom, neutral, curiosity, sense of accomplishment/enjoyment, and excitement). Cognitive engagement was examined at three levels built on previous studies⁷. This study contributes to the learning-analytics literature by providing a fine-grained account of student engagement with AI teachable agents. By identifying interaction and engagement patterns, it clarifies how AI agents can differentially affect learning, particularly for students who struggle in mathematics, and highlights the need for improved educational AI design.

Related work

AI teachable agents and its benefits and challenges

TA systems represent a sophisticated pedagogical innovation rooted in the learning-by-teaching model³. AI-supported TA systems, however, adapt this model by having learners teach a virtual peer powered by artificial intelligence⁷.

TAs offer a range of benefits and challenges that make them both a promising and complex tool for enhancing education^3,7,28. Research has shown that TAs can play various roles in supporting student learning. These agents are designed to provide personalized, flexible, adaptive assistance, facilitating students’ engagement with content in multiple ways^29–31. Whether acting as virtual peers, tutors, or learning companions, TAs can help reinforce subject knowledge, promote active learning through teaching, and encourage metacognitive processes. By offering valued feedback, prompting reflection, and enabling students to explore concepts interactively, TAs contribute to deeper understanding and improved academic outcomes across diverse educational settings. Research has demonstrated the effectiveness of TAs, such as Betty’s Brain, in fostering student engagement, metacognitive skills, and knowledge retention through the learning-by-teaching approach⁴. TAs facilitate deeper learning by encouraging students to reflect on and restructure their knowledge, while also promoting self-regulated learning and responsibility³². However, there are notable challenges in the design and implementation of AI-powered TAs. One key issue relates to the technical complexities of creating GenAI-powered TAs^7,33 can adapt to students’ needs and respond to open-ended inputs, as many systems struggle to fully comprehend students’ responses, limiting meaningful interaction^33,34. Additionally, the development of pedagogical models to effectively integrate TAs into learning environments is critical, as poorly designed AI cannot lead to significant educational gains³⁵. More studies are needed to identify limitations in AI teachable agents.

Students’ engagement in AI supported learning environment

With the integration of AI into education, the context of student engagement has expanded to include how students interact with AI tools, intelligent tutoring systems, and adaptive learning platforms. AI-supported environments have the potential to significantly influence all three dimensions of engagement by providing personalized, adaptive learning experiences that respond to individual student needs in real time³⁶. AI systems are increasingly being employed to facilitate personalized learning experiences, adaptive feedback, and real-time assessment of student progress. Several studies have explored the role of AI in promoting engagement. For instance, Holstein et al.¹⁶ found that students interacting with AI tutors showed increased cognitive engagement and better learning outcomes compared to traditional instruction. Similarly, research demonstrated that emotionally adaptive AI systems can reduce frustration and increase student motivation, improving emotional engagement¹⁵.

Recently, the rapid development of Generative AI provides empower in education fields, and many research have tried to integrate generative AI into a learning system to provide students different types of support and feedback. For instance, a study explores the integration of the ChatGPT API with the GPT-4 model and Microsoft Copilot Studio on the Microsoft Teams platform to promote students’ motivation and engagement³⁷. Another study investigates the use of AI-powered multi-role chatbots as a means to enhance learning experiences and foster engagement in computer science education, providing insight into the potential of AI-powered multi-role chatbots in K-12 education³⁸. These studies represent preliminary explorations of AI chatbots and tutoring systems; however, limited studies employ comprehensive analyses of student learning processes within AI tutor systems to elucidate how students engage with these platforms.

Methods

AI teachable agent

The AI teachable agent used in this study was developed by our team, incorporates five main features, as illustrated in Fig. 1. Grounded in the learning-by-teaching theory³, the AI teachable agent provides students with an instructor-like experience by functioning as a virtual peer, prompting students to explain specific mathematical concepts or solve problems. The AI teachable agent initiate the conversation and ask questions to the students, the students students interact by typing their responses in the input box (Fig. 1 (3)) using natural language. The AI teachable agent may provide feedback or ask follow-up questions based on the student’s input. The agent is integrated into a math learning platform (named Math Nation) to assist students in their learning.

Fig. 1.

The interface of the AI teachable agent.

Students have the option to personalize their teaching experience by selecting math problems from three difficulty levels (i.e., simple, medium, and hard) and choosing from nine themes (e.g., environmental issues, as shown in Fig. 1 (1)) based on their interests. The virtual peer’s profile is displayed in Fig. 1 (2). When the virtual peer presents a math problem, students interact by typing their responses in the input box (Fig. 1 (3)) using natural language. The AI teachable agent may provide feedback or ask follow-up questions based on the student’s input. The ranking board (Fig. 1 (4)) provides real-time updates on students’ ranks and points earned for teaching the virtual peer. Lastly, Fig. 1 (5) presents an introduction to the virtual peer, allowing students to choose peers with varying personalities to enhance their teaching experience. More information can be found in the supplementary document Appendix A.

Study context and participants

We focused on students whose performance declined after learning with the AI teachable agent; however, we also analyzed the improved group to provide comparison and context. To determine whether students’ performance declined or improved, participants completed the same 15-item multiple-choice mathematics test on the Math Nation platform (https://www.mathnation.com/) before and after using the AI teachable agent. The test aligned with the Florida Benchmarks for Excellent Student Thinking (B.E.S.T.). Scores were calculated as the proportion correct (0–1). We defined a change score as and classified students as improved () or declined (). Appendix B of the supplementary document reports group sizes, proportions, and mean changes. This study was approved by the Institutional Review Board at University of Florida (IRB No. IRB202301838). All research procedures followed relevant ethical guidelines, and informed consent was obtained from the legal guardians of all participants.

Data description

We analyzed 1397 dialogue sessions from 206 students whose performance declined after using the AI teachable agent, and an additional 1332 sessions from students whose performance improved. The logs contain time-stamped textual interactions with the agent, including problem-solving attempts and the agent’s responses. Each record includes a unique student ID and a session ID, allowing multiple sessions per student. For analysis, we organized the conversation data by (student ID, session ID) and then ordered messages chronologically using timestamps to form coherent dialogue sequences. These organized sequences constitute the corpus for all subsequent analyses and coding. Figure 2 shows an example snippet and the resulting data structure.

Fig. 2.

The data collected from the AI teachable agent.

Interactive mode analysis

To understand how students interacted with the AI teachable agent, this study employed a deductive coding approach to analyze students’ interaction modes. The Interactive Constructive Active Passive (ICAP) framework²⁷ aligns well with the context of learning with an AI teachable agent in our study. ICAP classifies learning activities on a continuum, proposing four dimensions (i.e., interactive, constructive, active, and passive), each represented by a specific code. The framework suggests that learning outcomes improve as students progress from passive to interactive engagement. Previous studies have applied the ICAP framework to analyze students’ interactions and engagement in online discussions³⁹. In this study, we utilized the ICAP framework to identify interaction patterns in students’ engagement with the AI teachable agent in math learning. Table 1 outlines the ICAP framework as applied in earlier studies, alongside the coding scheme we adopted for this study. The last column provides examples of students’ responses corresponding to the different interaction modes. Building upon previous frameworks, our coding scheme is specifically tailored to the context of students’ interactions with the AI teachable agent in mathematics learning.

Table 1.

Interaction coding schema.

Literature	ICAP category
	Passive	Active	Constructive	Interactive
27	Listen to explanations without any doing anything else	Taking notes; highlight content	Self-explaining; comparing and contrasting	Discuss with a peer; drawing a diagram
39	Receives information without overtly doing anything	Engages in any overt actions	Produce additional outputs; shows evidence of knowledge construction	Engage in substantive dialoguing; shows evidence of knowledge construction
Our study	The student listens to the AI teachable agent without providing any responses or only gives non-meaningful responses	The student responds to the AI teachable agent with math-related content, such as calculating, copying solution steps, summarizing, or giving simple feedback	The student asks questions, reflects on the problem, explains mathematical concepts or processes, or compares and contrasts different approaches	The student defends their solutions, engages in arguments, asks and answers comprehensive questions, or participates in debates and reasoning
Example	Student: Okay	Student: that’s correct! you were right!	Student: OK so x=101 right and when you get the inequality that means greater than or equal to so that mean 101 is greater than 100!	Student: use bar graphs in every situation because it is good overall but for specific information that you might want to have more in depth then use the line graph

Engagement analysis

Behavioral engagement

Behavioral engagement encompasses the observable actions and participation of students during learning tasks. Key indicators of behavioral engagement in this context include the total number of sessions a student attends, the duration of each session, the completion status of sessions, and the frequency of proactive interactions by the student. Higher values for these variables are typically associated with greater levels of behavioral engagement. We calculated session counts based on unique student IDs and unique session IDs. Session length was determined by counting the number of students’ responses in each session . The completion status was analyzed by checking whether students helped the AI teachable agent solve the problem presented at the beginning of the session. If a student participated in sessions and completed sessions, the completion status is defined as . Proactive interaction was calculated based on the coding results in Table 1. We counted all dialogue modes except for passive modes as proactive interactions, and the total number of proactive interactions was obtained by summing these (see Fig. 3).

Fig. 3.

The analyzed features for the three engagement aspects. To gain a comprehensive understanding of students’ behavioral, emotional, and cognitive engagement when interacting with the AI teachable agent, we employed a mixed-methods approach to analyze these three dimensions.

Emotional engagement

Emotional engagement refers to the emotional responses students exhibit during learning activities. We focus on the frequency of non-neutral emotional types, the types of emotions, and changes in emotions throughout the dialogue (see Fig. 3). This part of the analysis employs a mixed-method approach. First, researchers manually labeled the emotional states of all dialogues using a 7-point scale, where each point represents a distinct emotional state: anxiety, frustration/confusion, boredom, neutral, curiosity, sense of accomplishment/enjoyment, and excitement. These emotions were selected as they are commonly observed during math learning^40,41. The unit of analysis is a student utterance. Discrepancies were resolved through discussion between the two coders to reach consensus; the consensus label was used for all subsequent analyses. We then calculated the frequency and change of emotional states based on these emotion labels. A table defining each emotion category with examples is provided in Appendix C of the supplementary material.

Cognitive engagement

Cognitive engagement refers to the mental effort students invest in learning, reflected in efforts to understand content, reason through ideas, solve problems, and apply knowledge. This conception aligns with the mathematical literacy literature and prior frameworks. For example, the Department of Basic Education frames mathematical literacy around content, competencies/skills, and real-world context⁴²; Niemi et al. emphasize conceptual understanding, communication about mathematics, and usefulness in life and application⁴³; and PISA describes the constructs as formulate, employ, and interpret/evaluate⁴⁴. Building on a recent analysis of students’ online discussions that adopts a similar three-part structure (knowledge and content; competencies and skills; real-world application)⁷, we use this tripartite lens to assess cognitive engagement in our setting. Specifically, we coded students’ interactions with the AI teachable agent into three levels (see Fig. 3): (i) knowledge acquisition and content understanding (CK), (ii) competency to apply concepts and solve problems (CC), and (iii) application of knowledge in new or complex situations (CA). Appendix D in the supplementary document provides definitions and examples for each category.

Manual coding process

Two researchers conducted a deductive content analysis using the coding schema in Table 1 and Section Engagement analysis (see the supplementary document for more details). The corpus consisted of students’ time-stamped responses to the AI teachable agent (Fig. 2b). Each response was coded for three dimensions: interaction mode, emotion type, and cognitive engagement level. For the declined-performance group (1397 sessions), the two coders first reviewed the scheme and independently coded an initial set of 120 sessions. Initial inter-rater reliability (IRR; Cohen’s kappa) was 0.65 for interaction mode, 0.70 for emotion type, and 0.58 for cognitive engagement. After reconciling disagreements, they independently coded another 120 sessions; IRR improved to 0.84, 0.90, and 0.81, respectively. Having met the target reliability, the remaining sessions were divided between coders: one coded 820 sessions and the other coded 577 sessions. The average coding rate was approximately 50 sessions per hour per coder. The same procedure was applied to the improved-performance group (1332 sessions); each coder coded 666 sessions.

Results

Descriptive results

Table 2 reports descriptive statistics for all variables examined in the study (more descriptions can be found at Appendix E in the supplementary document ).

Table 2.

Comparison of two groups on key variables (mean and standard deviation per student).

Variables		Performance-declined group Mean (SD)	Performance-improved group Mean (SD)
Interaction mode	Passive	14.82 (26.57)	3.10 (5.16)
	Active	9.88 (14.20)	5.63 (6.28)
	Constructive	9.77 (14.12)	16.17 (14.10)
	Interactive	4.46 (6.51)	0.85 (1.57)
Behavioral engagement	Session count	6.66 (7.41)	3.79 (3.00)
	Session length	52.84 (67.31)	24.62 (21.19)
	Completion rate	0.35 (0.31)	0.58 (0.34)
	Proactive interactions	30.85 (29.91)	22.66 (19.74)
Emotional engagement	Negative	069 (2.21)	1.31 (1.51)
	Neutral	4.76 (5.56)	3.41 (2.40)
	Positive	0.31 (0.76)	2.18 (1.79)
Cognitive engagement	CK	15.68 (22.36)	24.18 (21.35)
	CC	6.50 (8.58)	1.47 (2.36)
	CA	1.90 (3.66)	0.11 (0.40)
Learning performance	Pre score	0.48 (0.18)	0.34 (0.19)
	Post score	0.29 (0.16)	0.58 (0.23)
	Learning gain	− 0.19 (0.13)	0.24 (0.19)

Interaction mode

We aggregated interaction modes from the coded data; Table 3 reports the distributions. In the declined-performance group, the passive mode is most prevalent (36.0%), exceeding the other three interaction modes; chat (non-mathematical) interactions account for 5.5%. At the student level, 70 students (33.98%) exhibited more passive interactions than any other mode. A chat-dominant subgroup (n = 26) engaged primarily in off-topic exchanges (e.g., inquiries about the AI’s hobbies, place of residence, and similar personal topics); illustrative examples are summarized in Appendix F of the supplementary document. In the improved group, the constructive mode is most prevalent, accounting for 62.78%.

Table 3.

Interaction modes by group: proportions.

Group	Passive + chat	Active	Constructive	Interactive
Performance-declined group	36% + 5.5%	24%	23.7%	10.8%
Performance-improved group	12.04%	21.87%	62.78%	3.31%

Engagement analysis

Behavioral engagement

Table 2 summarizes per-student session count, mean session duration, completion rate, and number of proactive interactions for the two groups. In the performance-declined group, students completed on average 6.66 sessions, with a mean session duration of 52.84 min, a mean completion rate of 0.34, and 30.85 proactive interactions per student. By contrast, students in the performance-improved group completed fewer and shorter sessions but achieved a higher completion rate (0.58).

Emotional engagement

Table 4 reports emotional engagement by group. In the Performance-Declined Group, non-neutral emotions comprised 19.32% of all responses. Within the non-neutral subset, boredom was most prevalent (9.73%), followed by curiosity (5.32%) and frustration/confusion (1.57%); anxiety (0.27%), sense of accomplishment/enjoyment (1.42%), and excitement (1.01%) were infrequent. By contrast, the Performance-Improved Group showed a more activated profile: positive emotions (i.e., curiosity, sense of accomplishment/enjoyment, and excitement) occurred in 30.09% of responses, exceeding negative emotions.

Table 4.

Emotional engagement categories by group: proportions.

Group	Anxiety	Frustration/confusion	Boredom	Neutral	Curiosity	SA/enjoyment	Excitement
Performance-declined group	0.27%	1.57%	9.73%	80.68%	5.32%	1.42%	1.01%
Performance-improved group	1.16%	14.81%	1.97%	51.97%	16.03%	13.41%	0.65%

Given our focus on the Performance-Declined Group, we further analyzed the emotional changes declined-performance group students experienced throughout each session. Four examples were sampled from sessions of different lengths (i.e., less than 10, 10–20, 20–50, and greater than 50), as illustrated in Fig. 4. While neutral emotions were the most frequently observed and persisted throughout the sessions, boredom notably increased as the response index grew. This trend was consistently observed across all four subplots, indicating that as the session progressed, students were more likely to experience boredom.

Fig. 4.

Student Emotional engagement pattern. The x-axis represents the index of students’ responses in sequential order within a session, while the y-axis denotes the corresponding emotional states.

Cognition engagement

Table 2 shows that students in both groups display low engagement at the CC and CA levels. In the Performance-Declined Group, the per-student means are CK = 15.68, CC = 6.50, and CA = 1.90. Across groups, CK exceeds CC and CA and also exhibits greater variability (variance> 20). Compared with the Declined Group, the Performance-Improved Group shows higher CK and lower CC.

Engagement and the learning performance

We further group the Performance-Declined Group students based on their behavior, emotional and cognition engagement patterns and compared their learning performance to see if there is any correlation between different engagement patterns and the learning performance. The results are shown in Fig. 5. For the behavioral pattern, we clustered students based on their four behavioral engagement indicators (i.e., session counts, average session length, completion rate, and proactive interaction counts) using K-means. The clustering results revealed two groups with high and low behavioral engagement, as shown in Fig. 5 (a). A subsequent t-test analysis indicated a significant difference in learning performance between the two groups (t =6.23, p = 0.01***). The high-group exhibited much higher behavioral engagement, such as an average session count of 17.31 compared to 4.16 in the low behavioral engagement group. However, their performance was significantly lower. For the emotional pattern, we categorized students into three groups: positive emotion dominant, balanced, and negative emotion dominant. ANOVA revealed a significant difference in learning performance across the three groups (F = 3.76, p = 0.03**). Further post-hoc analysis using Tukey’s HSD test indicated a significant difference between the positive dominant and balanced groups (p = 0.0486**< 0.05). In terms of cognitive engagement, students were classified as CK, CC, or CA dominant, with a tied group representing students who were balanced across all three aspects. ANOVA showed no significant difference in learning performance among these groups.

Fig. 5.

Student engagement pattern and learning performance. Performance was measured as the difference between students’ post-test and pre-test scores.

The same cluster analysis and comparative tests were also conducted for the Performance-Improved Group. Figure A2 in supplementary document presents the results for behavioral and emotional dimensions. For behavioral engagement, no significant difference in performance gains was observed between the high- and low-engagement clusters (t = -1.53, p = 0.13). Similarly, for the emotional groups, neither the positive-dominant nor the negative-dominant students differed significantly from the balanced group in terms of learning gains (F = 0.22, p = 0.80). Regarding cognitive engagement, the majority of students were classified as CK-dominant, with only two students identified as CC-dominant or tied cognition, resulting in limited variability and no significant differences in performance outcomes.

Discussion

RQ1: What interaction modes do students exhibit when engaging with an AI teachable agent during math learning?

The findings revealed clear differences in the interaction modes between performance-declined and performance-improved students. Students in the performance-declined group were more likely to engage in passive or off-task chat interactions, whereas constructive interactions dominated in the performance-improved group. These results align with the ICAP framework²⁷, which emphasizes that constructive engagement is more conducive to deep learning. In contrast, passive engagement or off-task behaviors may reduce opportunities for meaningful knowledge construction. The chat-dominant subgroup further illustrates how learners may redirect the system toward social or irrelevant topics, potentially undermining intended learning outcomes.

Several factors may explain why passive interactions were so prevalent in the performance-declined group. First, acting as the “teacher” of an AI peer without sufficient guidance may have increased students’ cognitive load. Research shows that excessive cognitive load can discourage active learning strategies, as learners may struggle to analyze and internalize information effectively^45,46. Second, prior studies suggest that students often find it challenging to use chatbots for complex tasks such as mathematics⁴⁷, and the reliance on typing and text-based exchanges in this study may have further reduced comfort and engagement. Third, the AI teachable agent frequently offered detailed explanations before asking students to confirm correctness, a pattern that may have unintentionally positioned students as passive recipients rather than active contributors. As⁴⁸ argued, simpler and more concise responses may be more effective in sustaining human-agent dialogue.

These findings echo broader concerns that students often act as passive recipients when interacting with AI tutors or pedagogical agents, especially when the system dictates a pre-defined learning path⁴⁹. Without intentional scaffolding or pedagogical interventions, the mere presence of an AI agent does not necessarily promote more active learning⁵⁰. Consequently, the results highlight the importance of designing AI teachable agents that not only provide explanations but also deliberately prompt and support students to take proactive roles in constructing knowledge. Future designs should carefully consider the learning environment, learner characteristics, and the pedagogical strategies embedded within the agent to ensure that interactions foster agency and meaningful engagement^2,51.

RQ2: What are the patterns of student engagement (i.e., behavioral, emotional, and cognitive) in math learning with an AI teachable agent?

Behaviorally, performance-declined students invested more time and interactions, yet their completion rates were substantially lower than performance-improved students. This pattern suggests inefficient engagement strategies, where more input does not necessarily translate into higher learning gains. Such behaviors resemble prior observations of students “gaming the system,” where frequent but superficial actions fail to contribute to genuine learning progress⁵². In this study, students in the performance-declined group completed on average 6.66 sessions, however, their completion rate remained low (0.34). One possible explanation is that students had just been introduced to the mathematical content, and their limited prior knowledge made it difficult to sustain meaningful progress. Prior knowledge has been shown to influence how students interact with intelligent tutoring systems, with learners who start from a weaker foundation often achieving smaller learning gains in technology-supported environments⁵³.

Emotionally, the performance-declined group was characterized by high levels of boredom and neutrality, while the performance-improved group expressed substantially more curiosity and enjoyment. These findings support prior work showing that positive emotions can sustain attention and motivation, whereas boredom tends to accumulate over time and hinder persistence. In this study, boredom frequently appeared in the latter half of sessions, consistent with research suggesting that unresolved frustration can escalate into boredom¹⁵. The prevalence of boredom likely reflects the absence of adequate scaffolding when some students faced complex mathematical problems. By contrast, the performance-improved group’s greater expression of curiosity and accomplishment highlights the importance of positive affective states in sustaining engagement and promoting achievement.

Cognitively, both groups demonstrated limited engagement beyond the knowledge level (CK), with minimal evidence of higher-order reasoning (CC, CA). Although the performance-improved group exhibited higher CK and lower CC, the overall low representation of higher-level cognition underscores a broader challenge in promoting complex thinking in AI-supported environments. A likely explanation is that the predominantly text-based interactions encouraged students to demonstrate surface-level understanding of concepts posed by the AI, while applying knowledge in novel or complex situations proved far more challenging. While emotionally adaptive AI systems have been shown to reduce frustration and increase motivation¹⁵, and intelligent tutoring systems have the potential to enhance cognitive engagement compared to traditional instruction, our findings reveal that students working with an AI teachable agent did not consistently reach higher-order cognitive engagement. This highlights the need for adaptive scaffolding strategies that deliberately encourage deeper reasoning processes.

RQ3: How is student engagement related to their learning performance when using an AI teachable agent?

The relationship between engagement and performance proved to be more complex than a simple positive correlation. For the performance-declined group, students who exhibited higher behavioral engagement actually demonstrated lower performance outcomes. This paradox suggests that raw behavioral intensity, such as longer sessions and more interactions, does not necessarily equate to productive engagement. Instead, the quality of engagement appears to be more important. Emotional engagement was significantly associated with performance in the performance-declined group. Yet students who expressed more positive emotions (e.g., curiosity) did not achieve better learning outcomes than those who maintained neutral emotions. A likely explanation is that much of this emotional activation arose in the context of off-task conversations with the AI (e.g., about its personal attributes), which stimulated curiosity but did not contribute to mathematics learning.

Prior work in educational data mining underscores this finding. Observable increases in activity can reflect maladaptive behaviors such as gaming the system or inefficient help-seeking, which inflate interaction counts but undermine learning by reducing opportunities for practice^52,54,55. Similarly, off-task behavior and mind-wandering generate visible engagement without commensurate gains, ultimately lowering performance in computer-based environments^55,56. From an affective perspective, control-value theory posits that emotions shape learning through appraisals of control and value rather than valence alone, so positive emotions are not uniformly beneficial⁵⁷. The broaden-and-build theory further suggests that enjoyment and curiosity can foster exploration, but in open-ended tutoring dialogues such exploration may drift toward seductive details, thereby diverting attention from core content and diluting immediate learning^58–60.

Moreover, the limited cognitive engagement observed in dimensions such as Conceptual Clarification (CC) and Application of Knowledge (CA) helps explain why students with high behavioral engagement and positive emotions did not achieve improved performance. Behavioral engagement alone, particularly when characterized by off-task interactions, does not guarantee effective knowledge construction^61,62. These findings resonate with research on intelligent tutoring systems, which has shown that the effectiveness of such systems depends not on the quantity of interactions but on the depth and task relevance of those interactions¹⁴.

In contrast, for the performance-improved group, neither behavioral nor emotional engagement showed significant associations with learning gains. One possible explanation is that these students were already employing relatively effective learning strategies, resulting in less variability in how engagement dimensions related to outcomes. The relatively balanced distribution of engagement types in this group suggests that once students possess adequate strategies for interacting with the AI teachable agent, their performance may no longer depend heavily on differences in behavioral or emotional engagement.

Taken together, these results highlight that the engagement-performance relationship is contingent on both the quality and the context of engagement. For struggling students, scaffolding that redirects off-task behavior, sustains positive emotions, and promotes higher-order cognitive engagement is critical. For students already performing well, the priority should shift toward maintaining effective strategies and deepening reasoning opportunities. Future AI teachable agents should therefore integrate adaptive mechanisms that not only foster constructive engagement but also mitigate unproductive or curiosity-driven diversions, ensuring that student effort is aligned with meaningful learning outcomes^35,63.

Design implications for AI teachable agents

Here we present three recommendations aim to discourage off-task conversations and foster deeper engagement when students interact with AI teachable agents.

Our results showed that some students, particularly those in the performance-declined group, became bored after several rounds of interaction and engaged in off-task behaviors. One strategy to prevent such disengagement is to co-set a time box for problem solving before the interaction begins. This can be implemented in two complementary ways: (1) teachers announce a fixed duration for working with the AI teachable agent; and (2) the system interface displays a session goal, a countdown timer, and a visible progress indicator. Such features provide a clear, clickable task structure to keep students focused on solving the math problems. Prior research indicates that goal setting and time boxing can strengthen persistence⁶⁴, while visible progress indicators can enhance task completion and the user experience⁶⁵.

Another strategy is to incorporate real-time mechanisms for detecting off-task behavior. This can be achieved using discourse features or multimodal signals. For example, an utterance may be flagged as potentially off-task if its semantic similarity to the current goal or the most recent turns falls below a threshold (e.g., 0.45). Students could be informed that their off-topic ideas have been added to a “curiosity list,” with the promise of a short free-chat window after completing the math task. This design acknowledges student curiosity while preserving task relevance. Evidence from affect-sensitive systems such as AutoTutor demonstrates that adaptive detection of learner states, combined with multimodal scaffolding and redirection, leads to deeper learning⁶⁶.

When off-task behavior occurs, effective redirection is critical. The system should first acknowledge the “interesting but unrelated” idea, then provide a bridging prompt that connects it back to the task (e.g., “Rewrite your example as an inequality that includes ‘at least’.”). A staged escalation policy may be applied: on the first occurrence, issue a gentle prompt with several on-task options; on the second, restate the goal, add the curiosity list, and propose a micro-task; on the third, apply a lightweight lock for 2 min where only task-relevant options are available, coupled with a brief rationale. If these measures prove ineffective, requesting teacher support is also necessary. This design echoes prior work on real-time classroom orchestration tools that empower teachers to intervene effectively⁶⁷.

Conclusion, limitations and future work

This study conducted an exploratory analysis of students’ interaction and engagement patterns with an AI teachable agent, comparing students whose learning performance declined with those whose performance improved. The findings revealed clear contrasts between the two groups. For the performance-declined group, passive interaction mode was predominant, and high levels of behavioral engagement did not translate into improved learning outcomes. Instead, much of this engagement reflected off-task conversations that stimulated curiosity but did not contribute to mathematics learning. Boredom also emerged as the most persistent non-neutral emotion, becoming more frequent as sessions progressed. In terms of cognitive engagement, students in both groups primarily focused on CK, with minimal involvement in CC and CA, limiting opportunities for deeper reasoning and transfer. By contrast, students in the performance-improved group engaged more constructively with the AI and expressed more positive emotions, but in this group, neither behavioral nor emotional engagement was significantly associated with learning gains, likely because these students already employed relatively effective strategies.

Several limitations must be considered when interpreting the findings. First, the study focused on short-term interactions, which may not capture the full impact of AI-supported learning on long-term outcomes or skill development. Second, although human annotation of students’ emotions and interaction logs provided valuable insights, it may not fully capture students’ internal affective or cognitive states without complementary self-reports. Finally, the study examined only interaction and engagement patterns, without accounting for external factors such as classroom context, peer dynamics, or teacher involvement that could have influenced students’ experiences.

Future research should explore how to design AI teachable agents that not only increase interaction frequency but also enhance task-relevant engagement and foster higher-order cognitive processing. This could involve integrating adaptive learning pathways that adjust to students’ prior knowledge and learning trajectories, thereby preventing off-task behavior and strengthening conceptual understanding. Longitudinal studies with larger and more diverse samples are needed to examine how AI teachable agents affect student engagement and learning performance over time. In addition, future work should investigate emotionally adaptive features that can mitigate negative states such as boredom and promote sustained curiosity and enjoyment. Ultimately, advancing the design of AI teachable agents requires a holistic approach that considers behavioral, emotional, and cognitive dimensions simultaneously to ensure that engagement leads to meaningful learning gains.

Author contributions

Conceptualization: Z.L., W.X. Data collection: Z.L., B.N., X.J. Methodology: Z.L., W.X. Visualization: Z.L., S.J., C.L. Writing—original draft: Z.L., B.N. Writing—review & editing: Z.L., W.X., B.N., X.J., S.J., C.L.

Funding

This work is supported by the National Science Foundation (NSF) of the United States under grant numbers 1503196 and 2105695. Any opinions, findings, and conclusions or recommendations expressed in this paper, however, are those of the authors and do not necessarily reflect the views of the NSF.

Data availability

The data that support the findings of this study are not publicly available due to privacy or institutional restrictions but are available from the corresponding author upon reasonable request.

Declarations

Competing interests

The authors declare no competing interests.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-025-24841-8.