Skip to main content
PMC home page
Frontiers in Artificial Intelligence logoLink to Frontiers in Artificial Intelligence
. 2022 Nov 3;5:1015660. doi: 10.3389/frai.2022.1015660

Automatic modeling of student characteristics with interaction and physiological data using machine learning: A review

Fidelia A Orji 1,*, Julita Vassileva 1
PMCID: PMC9670110  PMID: 36406472

Abstract

Student characteristics affect their willingness and ability to acquire new knowledge. Assessing and identifying the effects of student characteristics is important for online educational systems. Machine learning (ML) is becoming significant in utilizing learning data for student modeling, decision support systems, adaptive systems, and evaluation systems. The growing need for dynamic assessment of student characteristics in online educational systems has led to application of machine learning methods in modeling the characteristics. Being able to automatically model student characteristics during learning processes is essential for dynamic and continuous adaptation of teaching and learning to each student's needs. This paper provides a review of 8 years (from 2015 to 2022) of literature on the application of machine learning methods for automatic modeling of various student characteristics. The review found six student characteristics that can be modeled automatically and highlighted the data types, collection methods, and machine learning techniques used to model them. Researchers, educators, and online educational systems designers will benefit from this study as it could be used as a guide for decision-making when creating student models for adaptive educational systems. Such systems can detect students' needs during the learning process and adapt the learning interventions based on the detected needs. Moreover, the study revealed the progress made in the application of machine learning for automatic modeling of student characteristics and suggested new future research directions for the field. Therefore, machine learning researchers could benefit from this study as they can further advance this area by investigating new, unexplored techniques and find new ways to improve the accuracy of the created student models.

Keywords: machine learning, student characteristics, learning interaction data, student physiological data, student modeling, learner characteristics, adaptation

Introduction

Artificial intelligence (AI) is a field of computer science concerned with developing smart software systems that mimic human capabilities in understanding and responding to tasks. AI is referred to as “computing systems that are able to engage in human-like processes such as learning, adapting, synthesizing, self-correction and use of data for complex processing tasks” (Popenici and Kerr, 2017). The implementation of the capabilities is made possible using AI techniques. The techniques allow inferencing, learning, natural language processing, and perception abilities to be implemented in software systems. Artificial intelligence comprises various strategies and methods, including machine learning in generating, as well as applications of these techniques in solving specific problems. Machine learning is a field of AI which refers to computational methods that improve the performance of a task or make predictions based on available data about the task. The machine learning approach allows software systems to learn from data and to make informed decisions about a specific task based on information learned (Popenici and Kerr, 2017). The approach uncovers valuable insights from data through a combination of concepts from computer science, probability, statistic, and optimization theory. The insights assist in supporting the execution and automation of processes in online educational systems. Typically, data obtained via interaction of students with their online educational systems are crucial in predicting their needs, learning patterns, and progress.

Several factors that affect students' academic performance, for example, include emotion, motivation, engagement, and cognitive ability (Yukselturk and Bulut, 2007; Castillo-Merino and Serradell-López, 2014; Cho and Heron, 2015; Lei et al., 2018). Studies reported that emotion, motivation, and engagement have more effect on students' learning with technological systems compared to face-to-face educational settings (Tobarra et al., 2014; Stark, 2019). In the studies, teachers' presence was identified as a mediating factor. This created a need for different methods for automatic assessments of student characteristics as students engage with online educational systems using objective measures (such as students' learning behavior) to be investigated. Being able to predict student characteristics at run-time is crucial for an educational system to automatically and continuously change to adapt to each student's current state. As a result, computer and education researchers are harnessing artificial intelligence (AI) techniques in online educational systems design to make them more engaging, influential, and beneficial.

The increasing learning activities performed on online educational systems results in the generation of a huge volume of learning data. The application of machine learning techniques aimed at harnessing learning data to understand the real learning behavior of students and determine factors that improve learning success is a growing research field. Useful information about learning processes and students' behaviors extracted from learning data is used as input to machine learning techniques for modeling various student characteristics. The role of the characteristics such as student motivation, affective states, and engagement in learning has been recognized by various studies seeking to understand their effect on student learning and performance in different educational contexts. Assessing and identifying the effects of the characteristics on students is important for online educational systems because they have been shown to affect the willingness and ability of students to acquire new knowledge. Prior research has shown that motivated students actively use their learning materials and carry out their assignments (Chan and Ahern, 1999; Schunk et al., 2008). Such students are passionate about their learning activities.

The process of modeling with machine learning techniques produces models which can automatically predict student characteristics based on their learning behavior. Over the past years, several machine learning models targeted at one or more learning characteristics have been developed. The models can broadly be classified into two: models for modeling student characteristics based on physiological data and those based on interaction data. The physiological approach is targeted at behaviors undertaken by students through facial expression, eye-gaze, and neural specificity (Taylor et al., 2004), posture and body language (Boulay, 2011), and wearable sensory devices such as electroencephalography (EEG) and smartwatches (Bauer et al., 2019) are usually employed to capture some of the behaviors. The approach determines behaviors based on complex interactions among physiological, biological, and cognitive systems (Beauchaine, 2001). Examples of student characteristics modeled through the approach include motivation (Taylor et al., 2004), boredom and curiosity (Jaques et al., 2014), and engagement (Monkaresi et al., 2017). Interaction-based approach uses learning interaction data of students to build models for predicting various student characteristics such as valence and arousal (Salmeron-Majadas et al., 2018); motivation (Aluja-Banet et al., 2019); confusion, frustration, and eureka/delight (Graesser et al., 2008); and engagement (Gledson et al., 2021). Each of the two approaches has attracted attention in research.

Interest in developing and using machine learning technology to promote education and learning is increasing. Therefore, it is essential to carry out a literature review on applications of machine learning techniques for modeling various student characteristics automatically. The review will help to identify significant trends, best practices, gaps, and potential opportunities for improving online educational systems using automatic modeling of student characteristics. Additionally, analysis of current literature can help to bring different machine learning techniques, concepts, and processes used for automatic modeling of student characteristics with interaction-based or physiological data together. The modeling process plays an important role in determining the need for learning intervention that can help students.

This review presents novel work by collating current works on automatic modeling of student characteristics through interaction-based and physiological data using machine learning techniques. It answers the following research questions.

  • RQ1: What are student characteristics targeted for automatic modeling with machine learning techniques?

  • RQ2: What are effective machine learning techniques for modeling student characteristics based on interaction-based or physiological data?

  • RQ3: What are the commonly used techniques for capturing students' physiological data?

Background

Student characteristics

In order to enhance teaching and learning, a variety of studies discovered and investigated different student characteristics to identify their influence on learning progress and academic performance (Arroyo et al., 2014; Halawa et al., 2016; Liu and Ardakani, 2022). Some of the characteristics include student learning motivation, engagement, affective states, etc. Detecting the characteristics can contribute to supporting and presenting personalized learning content and activities to students. Typically, three different methodological approaches are widely exploited in learning technology research to measure the characteristics: based on self-report, interaction-based, and based on physiological measures. The self-report approach has been widely applied due to its reliability, ease of administration, and validity. The self-report tools vary in length and design and range of construct covered. For example, a tool such as the Motivated Strategies for Learning Questionnaire (MSLQ; Pintrich et al., 1993) integrated expectancy belief, value judgement, affect, and intrinsic/extrinsic orientation so that multiple constructs on student characteristics could be measured. Also, the National Survey on Student Engagement (NSSE; Kuh, 2009) tool is employed in assessing student engagement. Results of self-report tools demonstrated strong links between student characteristics, learning behaviors and academic performance. However, using self-report in assessing student characteristics during learning could be distracting. As a result, current research efforts are geared toward additional methods such as interaction-based and physiological methods that could facilitate dynamic modeling of the characteristics in various learning contexts. Physiological measures assess student characteristics through physical responses such as eye gaze, facial expression, body movement, etc. that occur naturally during learning. Interaction-based measures examine learning logs of students' activities to extract information on learning behaviors patterns, for example, number of accesses to a learning system, time spent, views of contents and assessments, number of assessments completed, etc. are employed in understanding the influence of student characteristics during learning based on context.

Nowadays physiological-based devices capture cognitive and non-cognitive student data to support the building of student models. For example, eye-tracker, cameras for capturing facial expressions, and electroencephalography (EEG) headsets have been used to obtain data employed in building student models that determine their current state during learning processes. Eye-tracking was employed to estimate the concentration level or attention patterns of students during learning process (Conati and Merten, 2007; Joe Louis Paul et al., 2019). The nature of students' concentration was analyzed to provide the basis for supporting and adapting learning components to students. Also, students' facial expressions were captured and analyzed to determine changes in their emotions during learning process (Llanda, 2019; Tonguç and Ozaydin Ozkara, 2020). The studies acknowledge the role of emotion in making students gain more knowledge and in improving their engagement in learning. Furthermore, El Kerdawy et al. (2020) in a study for detecting a student's cognitive profiles used facial expressions and EEG (as physiological monitoring tools) data to build models that detected two cognitive states (instantaneous attention and engagement) and three cognitive skills (planning, shifting, and focused attention). The researchers indicated that EGG and facial expressions provided important features that could be applied for dynamic cognitive state modeling. Consequently, there is a need to review recent works that use the physiological-based approach in a systematic way to provide an overview of the achievements and gaps which can serve as a useful starting point for advancing research in this area.

Furthermore, using interaction-based data for student characteristics modeling is becoming increasingly popular. The main advantage of interaction logs is that sensors are not required in collecting data and data collection does not distract or obstruct students learning processes. The data are captured on online educational systems as students are learning without distraction. Several researchers harnessed learning logs data of students in building models that detected varied student characteristics (Botelho et al., 2017; Hmedna et al., 2020). Information on patterns of student learning behaviors and other learning descriptors were used by the researchers in predicting student states. Thus, the use of log data in generating features for detecting student characteristics has been recognized as a viable approach which can provide a more scalable solution that could be applied in designing and adapting instructional materials to support the needs of students. Therefore, there is a need for a systematic review of current works on the interaction-based approach to analyze and classify scattered research studies in this area under useful headings that could help to identify progress made and aid in advancing research in this area.

A variety of data about students' behavior are being tracked in technology-enhanced learning systems during student learning. Student modeling process employs the tracked data to infer valuable information about student learning patterns, progress as well as characteristics. Student modeling is essential as it identifies the individual needs of the students and their current working context so that learning content and teaching could be adapted based on their needs. Two broad categories of student modeling are static and dynamic modeling. Static student modeling describes a method in which variables used in building student models are initialized just once while the dynamic approach often updates variables used in building student models, enabling a system to react instantly to changes in the examined student characteristics. The dynamic approach enables systems to incrementally learn student characteristics, discover and consider exceptional behavior of students, and respond to changes in a characteristic by updating the student model. The ability of the dynamic modeling process to present a more precise approximation of students' needs is vital for the provision of relevant support to students. Thus, the use of machine learning with physiological and learning interaction data aid in building dynamic student models because student data can be captured at various intervals during learning process. Considering dynamic modeling of student characteristics in technology-enhanced learning can have many benefits for students such as delivering to them adapted learning materials and guidance based on their learning needs.

Machine learning

Recently, research on developing student models that detect various student characteristics automatically from their learning behavior uses machine learning techniques to facilitate monitoring student behavior during learning process and updating built student models to reflect current states of students. Machine learning is concerned with development of models that enable a system to learn from observational data and draw conclusions automatically. Data acquired by observing students' behaviors and actions during teaching and learning are applied to construct student models. Considerable works have been done in using machine learning techniques to automatically construct student models for various student characteristics. The range of student characteristics investigated in literature includes emotion, motivation, engagement, learning style, and affective states (Botelho et al., 2017; Aissaoui et al., 2019; Hung et al., 2019; Raj and Renumol, 2022; Wang et al., 2022). The most common approach adopted by the majority of these studies is the use of supervised machine learning techniques in creating a model of student characteristics under investigation using features collected from either physiological or interaction log data of students. The studies extracted various features and investigated different machine learning algorithms such as random forest, support vector machine, convolutional neural network, decision tree, etc. Useful results were obtained in the studies with various algorithms and features for several student characteristics. For instance, Hung et al. (2019) automatically discover student emotions using convolutional neural network and facial expressions features. The convolutional neural network model predicted student emotion with an accuracy of 92%.

This review focuses on exploring how various student characteristics are predicted in studies using interaction-based and/or physiological data, thus supervised machine learning approach is the most appropriate method. As a result, we considered only studies that applied supervised machine learning approach to model student characteristics based on interaction-based and/or physiological data. Student characteristics have been identified as an important factor that affects their academic performance—studies that applied supervised approach of machine learning to predict academic performance were excluded (as our focus is on student characteristics modeling and there are other systematic reviews on papers predicting academic performance).

Materials and methods

Systematic literature reviews relating to a specific research domain provide a means to identify, analyze and understand research gaps and progress in the domain. In this review, we followed the PRISMA guidelines for systematic literature review (Moher et al., 2016), which enabled us to compare, contrast and categorize papers according to different themes and concepts related to student characteristics modeled with machine learning methods. The procedures we followed for the review process were: (1) identified the need to summarize existing evidence on the application of machine learning techniques for automatic modeling of student characteristics, (2) formulated the research questions that the literature review will address, (3) developed a review protocol, (4) identified research databases for our search, (5) identified, selected and saved papers using Mendeley software, (6) screened the papers based on inclusion and exclusion criteria, (7) extracted data from selected papers. (8) performed data synthesis and analysis and (9) reported the review results.

Resources and search strategy

For our literature search in this review, we used the Elsevier Scopus database, IEEE Xplore, Google Scholar, and the ACM Digital Library for our search. Searching various databases ensures good coverage of studies in this area. We employed the following search items: machine learning and learner model OR machine learning and student model,” “machine learning and learner modeling OR machine learning and student modeling,” and “machine learning and student characteristics OR machine learning and learner characteristics.” Only papers that were published in English between 2015 and 2022 were included in the literature search as we wanted to identify current development and progress made concerning automatic modeling of student characteristics with interaction-based and physiological data using machine learning methods. Moreover, the referenced lists of the selected literature were scanned to find more potentially relevant studies.

Eligibility criteria for selected papers

The following inclusion and exclusion criteria were defined to ensure that selected papers are in line with the purpose of this review.

Inclusion criteria

  • Peer-reviewed and published research papers.

  • Any paper that models one or more student characteristics with machine learning techniques and learning interaction and/or physiological data.

  • The paper must report the data types and machine learning algorithms used.

  • The paper employed supervised machine learning techniques.

Exclusion criteria

  • Papers on the prediction of academic performance with machine learning techniques and learning interaction and/or physiological data.

  • Papers that modeled student characteristics with machine learning but did not use learning interaction or physiological data of students.

  • Papers that created student profiles using unsupervised machine learning techniques.

  • Duplicate papers with similar contributions by the same author.

Paper selection process

Figure 1 summarizes the paper selection process. The search on relevant databases (the Elsevier Scopus database, IEEE Xplore, Google Scholar, and the ACM Digital Library) resulted in 1,732 articles with unique titles. The titles of the articles were screened and those that we considered out of the scope of this study were not selected. Five hundred and twenty-one articles considered relevant were downloaded and their abstracts were read. A total of 150 articles were selected based on review of the abstracts and the exclusion criteria. Furthermore, the full texts of the 150 articles were read and based on our inclusion and exclusion criteria, 38 papers were selected for inclusion in this study. The articles were saved in Mendeley for easy management and referencing.

Figure 1.

Figure 1

Open in a new tab

Selection process for the reviewed papers.

Data extraction from the selected papers

For saving extracted data from the articles, we created an excel document with the following columns: Xteristic_modeled, author_name, publication_year, feature_type (interaction or physiological or both), learning_environments (type of systems), number_of_participants, collection_methods (eye-tracker, EEG headsets, cameras, etc.), signal_used (brainwaves, videos of facial expressions, etc.), ML_algorithm_used, and model_accuracy.

Data synthesis and analysis

Based on the extracted data from the selected articles in an excel document, we analyzed and classified the data into different themes that will help to reveal insights and trends relevant to our research questions.

Results

The analysis of the selected papers revealed some trends and interesting insights. Our findings are presented below under various categories.

Data collection and analysis of trend

The results suggest a growing research interest in interaction-based (also called sensor-free) and physiological-based (also referred to as sensor-based) methods for capturing data on students learning behavior over the past few years. Students using online educational systems generate a wealth of data that is stored in log files. The logs record students' behaviors, view counts of different topics and contents, time spent in the system, and other information. Interaction-based features are often derived from the log files containing students' interactions with online educational systems. The interaction-based features used by the studies in this review include view events, assessment attempts, click activities, number of questions answered correctly, number of help requests, navigation order, number of accesses, mouse movements, keystrokes, clickstreams, and time spent on learning. A summary of studies that applied interaction-based features for modeling student characteristics is shown in Table 1. Some of the studies evaluated one machine learning model while a few assessed more than one model. For studies that evaluated a couple of machine learning models, we selected the best-performing models and presented them in Tables 1–3.

Table 1.

Automatic modeling of student characteristics in learning studies using interaction-based data.

Studies Number of students Learning environments Features Student characteristic modeled Machine learning models Models accuracy
Botelho et al. (2017) 646 students ASSISTments learning platform Interaction features Affective states Recurrent neural network AUC is 78%
DeFalco et al. (2018) 119 university students TC3Sim game learning environment Interaction features Affective states Logistic regression A' is 85%
Ghaleb et al. (2019) 23 university students Technology-enhanced learning system Interaction features Affective states Support vector machine Precision is 67%
Hutt et al. (2019) 69,174 high school students Online math learning platform Interaction features Affective states Genetic algorithm None
Khan et al. (2019) 81 students Learning management system (Moodle) Interaction features Affective states Bayesian network Precision is 67.9%
Salmeron-Majadas et al. (2018) 41 students Essay writing tool named MOKEETO Mouse and keyboard interactions Affective states Random forest 70%
Wang et al. (2019) 269 undergraduates Cloud Classroom Interaction features Affective states K* F score is 70%
Edmond Meku Fotso et al. (2020) 3,617 university students MOOC Interaction features Engagement Recurrent neural network 89.2%
Erkan et al. (2020) 12,447 university students MOOC (Peer-review participation) Interaction features, Student performance Engagement Random forest 80%
Raj and Renumol (2022) 7,775 university students Virtual learning environment course Interaction features, student performance Engagement Random forest 95%
Aissaoui et al. (2019) 1,235 students MOOC Interaction features Learning style Naïve bayes 89%
Amir et al. (2017) 200 university students Learning management system Interaction features Learning style Support vector machine 83.6%
Hmedna et al. (2020) 52,735 university students Edx course Interaction features Learning style Decision Tree 98%
Lwande et al. (2019) 199 students eLearning course Interaction features Learning style K-nearest neighbors None
Al-Shabandar et al. (2018) 7,000 university students MOOC (Edx) Interaction features Motivation Decision tree 75.5%
Babić (2017) 129 university students Learning management system Interaction features Motivation Neural network 76.9%
Abyaa et al. (2018) 48 university students Online learning platform (piazza) Interaction features Personality Random forest 83.3%
Open in a new tab

Table 3.

Automatic modeling of student characteristics in learning studies using multimodal data.

Studies Number of students Learning environment Signal types Collection devices/methods Student characteristic modeled Machine learning models Models accuracy
Altuwairqi et al. (2021) 110 university students Students performing tasks in a university computer lab Video clips of facial expressions, mouse movement and keystrokes Webcam and mouse/keyboard behavior recorders Engagement Naïve Bayes 95.2%
Henderson et al. (2020) 119 undergraduates Game-based learning environment Interaction features, posture and upper-body movement data Microsoft Kinect motion tracking sensor Affective states Multi-layer perception 89.5%
Lallé et al. (2021) 79 university students MetaTutor Eye gaze data and interaction features Tobii 60 eye tracker Emotion Linear regression 48%
Sharma et al. (2020) 32 undergraduates Web-based self-assessment environment Interaction features, fixations and saccades, EEG data, video clips of facial expression, and heart rate data Tobii X3-120 eye tracker, head-mounted portable EEG cap, Logitech webcam, and Empatical E4 wristband Motivation Hidden Markov models, support vector machine 90 and 80%
Open in a new tab

The physiological method uses sensors to record signals of physiological or neurobiological responses of students during learning. The physiological responses involve visible behaviors such as facial expression, eye gaze, and posture based. Cameras were employed in recording videos focused on students' faces as they were learning. Facial action units (FAU), head position, and facial points features were computed from the videos using specialized software such as OpenFace library and Kinect SDK's face tracking engine. For eye gaze, eye trackers (e.g., Tobii eye tracker) were used in capturing fixations and saccades of students' real-time eye responses during learning. To capture students' posture data during learning, Microsoft Kinect motion-tracking sensors were utilized to capture skeletal vertex coordinates based on upper-body movement and posture. The neurobiological responses measure neurobiological changes such as brainwave patterns and heart rates. Wearable EEG headsets were employed in collecting brainwave data. Empatica E4 wristbands were used to measure heart rate, electrodermal activity (EDA), body temperature, and blood volume pulse. An overview of studies that applied physiological data for modeling various student characteristics is presented in Table 2.

Table 2.

Automatic modeling of student characteristics in learning studies using physiological data.

Studies Number of students Learning environments Collection devices/ methods Signal types Student characteristic modeled Machine learning models Models accuracy
Bixler and D'Mello (2016) 178 undergraduates Research texts on research methods presented on computer screens Tobii TX 300 and Tobii T60 eye tracker Fixations and saccades Affective states Bayesian network 72%
Shi et al. (2019) 82 students MOOC platform Logitech C920 webcam Video clips of facial expression Affective states Convolutional neural networks and support vector machine 93.8%
Ashwin and Guddeti (2020) 50 students Classroom environment Camera Video clips Affective states Convolutional neural networks 95.6%
Hung et al. (2019) 4 university students Students learning in a class Camera Video records Emotion Convolutional neural networks 84.6%
Li and Wang (2018) 10 students Intelligent education system Camera Video clips of facial expressions (blink frequency) Emotion Convolutional neural networks Not reported
Liu and Ardakani (2022) 15 students Affective learning system EMOTIV EPOC and EEG headset Brain waves pattern Emotion K-nearest neighbors 74.3%
Yang and Qi (2020) 70 students Not reported Camera Pictures of students' facial expressions Emotion Convolutional neural networks 97%
Booth et al. (2018) 10 students Interactive computer tasks EPOC and EEG headset Brainwave data Engagement Random forest 62.5%
Dubbaka and Gopalan (2020) 5 adults MOOC Logitech C920 webcam Video clips of facial expressions Engagement Convolutional neural networks 95%
El Kerdawy et al. (2020) 109 university students Psychology continuous performance tasks Camera and EEG headset Video clips of facial expressions and brainwaves data Engagement •Random forest (for EEG data) •Convolutional neural networks (for facial data) 86 and 82%
Liu et al. (2018) 8 students Intelligent class environment Overhead camera in a wide classroom Visual focus of attention (VFOA) and head pose estimation Engagement Hybrid multilayered random forest 70%
Mohamad Nezami et al. (2020) 20 high school students Virtual world learning environment named Omosa Camera Video clips of facial expressions Engagement Convolutional neural networks 72.4%
Monkaresi et al. (2017) 22 students Students writing essays using google document Microsoft Kinect Sensor Videos of their faces and upper bodies (used for estimation of facial expression and heart rate) Engagement Naïve Bayes AUC 0.76%
Aggarwal et al. (2021) 12 undergraduates MOOC EEG headset Brainwave data Motivation Support vector machine 94%
Chattopadhyay et al. (2021) 30 students Game environment EEG known as brainmarker Brainwave data Motivation Convolutional neural networks 89%
Santos et al. (2020) 45 high school students Students performing experiments in physics lab Camera Video clips of facial expressions Motivation Convolutional neural networks 85%
Wang et al. (2022) 25 university students OGAMA software Tobii X120 tracker and EEG headset Fixation, pupil size, saccades, and Brainwave data Motivation Logistic regression 92.8%
Open in a new tab

Modeling student characteristics using interaction-based or physiological data has shown promising results when used separately, however, studies are investigating whether a multimodal approach (combination of the two data sources) can produce models of student characteristics with improved accuracy. Table 3 lists studies that present a multimodal approach for modeling different student characteristics. Ninety percent (34) of the reviewed papers proposed either interaction-based (17 papers) or physiological methods (17 papers), each accounting for 45% of the total number of papers. A fully multimodal approach, combining interaction-based and physiological methods is the least explored method, employed by only four papers (10% of all the studies) and they are more recent (from 2020 to 2021). The studies that applied the multimodal approach combined eye-tracking and interaction logs; eye-tracking, EEG, wristband, facial expressions, and interaction logs; facial expression, mouse movement and keystrokes; posture-based and interaction logs to build multimodal models for predicting students' affective states, emotion, motivation, and engagement, respectively. Some of these studies revealed that the multimodal method performed better than the model based on a single approach.

Techniques used with physiological method

Figure 2 summarizes the major technological techniques employed by studies that used the physiological approach to model learner characteristics. Some techniques were more often applied than others. For example, among the studies investigated, tracking facial expressions emerged as the most frequently utilized technique (used in 41% of all studies). It is followed by eye tracking (24%) and brainwave measure with EEG (21%). Tracking body posture (7%) and heart rate (7%) are the least employed techniques. The studies that provided their physiological data through them were few and they combined the obtained data with other physiological data sources such as facial expression, eye tracking, and EEG. Generally, tracking facial expressions, EEG, and eye tracking emerged as top techniques that are commonly employed in the physiological method for student characteristics modeling.

Figure 2.

Figure 2

Open in a new tab

Physiological techniques used in studies.

Modeled student characteristics category

As can be seen from Figure 3, the reviewed papers in this study fundamentally focused on modeling automatically six major student characteristics. The characteristics include student affective state, emotion, engagement, learning style, motivation, and personality. Among the 38 studies that modeled different student characteristics using machine learning techniques and interaction-based and/or physiological data, affective states led the list with a total of 29% of all the studies followed by engagement with 26%. Motivation comes third with 18%, and emotion with 13% took fourth place.

Figure 3.

Figure 3

Open in a new tab

Trend of student characteristics modeled.

Effective machine learning techniques for modeling student characteristics

Tables 4, 5 summarize various student characteristics modeled using either interaction-based or physiological data and machine learning algorithms. Based on the results from the analyzed studies, effective models capable of predicting students' affective state, emotion, engagement, learning style, and motivation can be developed using interaction-based data. With respect to models' performance, measured as accuracy in predicting different student characteristics, most of the studies analyzed reported impressive model performance (70% and above). Neural networks and random forest models predicted more characteristics effectively with interaction data. Furthermore, the results from the analyzed studies revealed that useful models that can predict students' affective states, emotions, motivation, and engagement can be implemented using physiological data. Interestingly, overall models' accuracy for predicting various student characteristics with physiological data ranges from 62.5 to 97%. Convolutional neural network models effectively predicted more characteristics using physiological data than the other models.

Table 4.

Student characteristics modeled using interaction-based data.

Student characteristics modeled Studies that used interaction-based data Machine learning algorithms with effective performance Accuracy of the models
Affective states Recurrent neural network, logistic regression, random forest, and K* Ranges from 70 to 85%
Botelho et al., 2017; DeFalco et al., 2018; Salmeron-Majadas et al., 2018; Ghaleb et al., 2019; Hutt et al., 2019; Khan et al., 2019; Wang et al., 2019
Engagement Edmond Meku Fotso et al., 2020; Erkan et al., 2020; Raj and Renumol, 2022 Recurrent neural network and random forest Ranges from 80 to 95%
Learning style Amir et al., 2017; Aissaoui et al., 2019; Lwande et al., 2019; Hmedna et al., 2020 Naïve bayes, support vector machine, and decision tree Ranges from 83.6 to 98%
Motivation Babić, 2017; Al-Shabandar et al., 2018 Decision tree and neural network 75.5 and 76.9%
Personality Abyaa et al., 2018 Random forest 83.3%
Open in a new tab

Table 5.

Student characteristics modeled using physiological data.

Student characteristics modeled Studies that used physiological data Machine learning algorithms with effective performance Accuracy of the models
Affective states Bixler and D'Mello, 2016; Shi et al., 2019; Ashwin and Guddeti, 2020 Bayesian networks, convolutional neural networks and support vector machine Ranges from 72 to 95.6%
Emotion Li and Wang, 2018; Hung et al., 2019; Yang and Qi, 2020; Liu and Ardakani, 2022 Convolutional neural networks and k-nearest neighbor Ranges from 74.3 to 97%
Engagement Monkaresi et al., 2017; Booth et al., 2018; Liu et al., 2018; Dubbaka and Gopalan, 2020; El Kerdawy et al., 2020; Mohamad Nezami et al., 2020 Convolutional neural networks, random forest, and naïve bayes Ranges from 70 to 95%
Motivation Santos et al., 2020; Aggarwal et al., 2021; Chattopadhyay et al., 2021; Wang et al., 2022 Support vector machine, convolutional neural networks, and logistic regression Ranges from 85 to 94%
Open in a new tab

The variations in the accuracy of the models for each characteristic are probably due to many possible factors (ranging from dataset size, number of features, type of features, and their preparation methods) which could affect machine learning models.

Discussion

The results of this literature review provide valuable insight for designers of educational systems and researchers about dynamic modeling of various student characteristics. Several machine learning techniques have been employed in detecting student characteristics using learning interaction logs, facial expressions, eye-gaze, electroencephalogram (EEG), and other physiological features.

Effectiveness of machine learning techniques for modeling student characteristics automatically

Based on the reviewed literature, machine learning techniques are effective at modeling various student characteristics automatically based on interaction-based or physiological data. Among the reviewed papers, 82% reported impressive model accuracy (from 70% and above) in modeling different student characteristics. The results suggest that various student characteristics can be modeled dynamically from their interaction and/or physiological responses during learning.

Targeted student characteristics for automatic modeling with machine learning techniques

Research indicated that getting useful and reliable student information to support adaptive decision-making is a challenge (Shute and Zapata-Rivera, 2012). This literature review revealed how machine learning methods have been applied to provide useful information about student characteristics which can help in adaptation of teaching and learning. The results of current research trends in Figure 3 revealed six main categories of student characteristics targeted for automatic modeling with machine learning techniques and students' interaction and/or physiological data. The categories include affective states, emotion, engagement, learning style, motivation, and personality. Based on our results, the commonly modeled student characteristics are affective states, engagement, and motivation.

Different researchers define student affective states, engagement, and motivation differently. The term “affective states of students in learning” has been used to denote a range of variables related to their motivation, engagement, and attitude. For example, in Wang et al. (2019), the affective states comprise confusion, engagement, off-task, and frustration. According to Shute and Zapata-Rivera (2012), the affective states consist of motivated, attentive, engaged, and frustrated. Similarly, the term “engagement” has been used to include different affective states by different authors. For instance, Dewan et al. (2019) consider learning gain, delight, boredom, neutral, confusion, and frustration as a scale for engagement levels. Despite the difference in their definition, researchers have the same specific strategies for assessing them. However, motivation is distinct from engagement. Motivation has been characterized as the psychological processes that underpin the energy, purpose, and long-term sustainability of learning activities whereas engagement has been defined as the outward manifestation of motivation (Skinner et al., 2009). Engagement can manifest itself in the form of observable behaviors (for example, involvement in learning, on-task behavior, etc.). “Therefore, when motivation to pursue a goal or succeed at an academic task is put into action deliberately, the energized result is engagement” (Wang and Degol, 2014). For instance, Xiong et al. (2015) applied structural equation modeling to MOOC data and revealed that motivation is a strong predictor of student engagement in a course.

Below we discussed the three commonly modeled student characteristics: affective states, engagement, and motivation. The discussion covers studies with high model accuracy and some studies that demonstrated how the models can be used in online educational systems.

Affective states

Students' affective states during learning impact their interaction, progress, and performance. Detecting positive and negative affective states of students is important in improving and supporting their learning processes and progress. According to research, learning affective states include students' concentration, confusion, frustration and off-task behaviors (Wang et al., 2019). A number of studies conducted in the area of affective computing in education have recognized the role of affective states especially the learning affective states in facilitating learning activities (Wiggins et al., 2015; Jiménez et al., 2018; Standen et al., 2020). For example, Standen et al. (2020) used multimodal sensor data and machine learning to build models that recognized three learning affective states (engagement, frustration, and boredom) and based on the state detected, the presentation of learning contents was adapted to maximize the student's learning rate. The authors used intervention and controlled experiments in evaluating their adaptive system. The results of the evaluation indicated that more engagement and less boredom were experienced in the intervention than in the control session. The results suggest that tailoring learning contents based on students' affective states resulted to increased engagement in learning activities and promoted learning. Liu and Ardakani (2022) collected students' brainwave data using portable electroencephalogram and applied k-nearest neighbor (KNN) machine learning algorithm to recognize affective states in real-time. Based on the recognized states, learning contents were automatically recommended to students. The authors evaluated their system using experimental and control groups. The results of the t-test analysis demonstrated that their eLearning system model enhanced students' satisfaction but did not have a significant impact on student engagement and learning. Furthermore, Wiggins et al. (2015) identified students' affective states and learning styles automatically from their learning behavior and preferences within a course. Based on the results of the automatic detection, the researchers developed a tool which teachers can use to identify students' affective states and learning styles. These studies demonstrated that the incorporation of affective states models into online educational systems can be used to improve students learning progress and performance and to assist teachers in understanding their students' conditions in a learning context, thus effective affective states models are important in detecting student's various states on different learning contexts.

Students' engagement

Student engagement with online educational systems is important in understanding their learning experience with the systems. The ability to recognize engagement levels automatically plays a role in designing and responding to student engagement issues. Student engagement tracking based on interaction and/or physiological data is crucial for the automatic detection of engagement in online educational systems. Data collected via these approaches are employed in modeling and designing adaptive learning environments that will support student learning. A variety of studies (Monkaresi et al., 2017; Dubbaka and Gopalan, 2020; Erkan et al., 2020; Raj and Renumol, 2022) reported many students' online learning behaviors and physiological responses that can be extracted and applied in building models that will automatically predict students' engagement levels. The studies reported the accuracy of their models to indicate the extent to which engagement levels could be dynamically detected using their approach. For example, a study by Raj and Renumol (2022) predicted student engagement with an accuracy of 95% using learning interaction data and a random forest model. This study achieved the best performance in terms of accuracy among all the reviewed papers that predicted student engagement levels using learning interaction logs. Hussain et al. (2018) developed a dashboard which incorporated a model of student engagement built using their interaction data, grade scores, and machine learning into an online education system to help instructors in assessing student engagement levels in online courses in relation to various activities and resources, and to offer additional interventions for students prior to their final exam. Dubbaka and Gopalan (2020) reported an accuracy of 95% for the prediction of student engagement using facial expression data and convolutional neural network model. The research achieved the best performance in terms of accuracy among all the reviewed papers that predicted student engagement levels using physiological data. To provide robust and more effective student engagement measures, Altuwairqi et al. (2021) combined learning interaction and physiological data (facial expression data, mouse movement and keystrokes) of students to build a naïve bayes model that predicted student engagement with an accuracy of 95.2%. The accuracy level of the two approaches suggested that automatic student engagement detection can effectively track issues on engagement in learning environments.

Based on this literature review, the most predominant approach for detecting engagement using physiological data is facial expression tracking. A number of studies (Monkaresi et al., 2017; Dubbaka and Gopalan, 2020; Mohamad Nezami et al., 2020; Altuwairqi et al., 2021) analyzed student engagement during interaction with online educational systems using facial expressions. The studies focused on capturing and analyzing facial expressions to identify disengagement and engagement states of students. Facial expression recognition is less invasive and integrated cameras on computer systems are easily accessible in majority of learning platforms.

Student motivation for learning

Motivation is an important factor that influences students learning. Research has shown that student motivation is a fundamental factor that affects their success (Yukselturk and Bulut, 2007). Muilenburg and Berge (2005) in a survey-based study identified poor motivation as one of the main barriers to students' online learning. Several studies revealed that motivated students are more likely to be actively engaged, embark on challenging tasks, and demonstrate improved performance and outcomes (Chan and Ahern, 1999; Schunk et al., 2008). As a result, the need to detect and support students to overcome motivation issues based on learning context has led researchers to explore various ways student motivation can be predicted automatically during teaching and learning. In line with the need for dynamic modeling of student motivation, studies have demonstrated that student motivation can be predicted from their learning interaction and/or physiological data using machine learning (Al-Shabandar et al., 2018; Santos et al., 2020; Aggarwal et al., 2021). Among the papers on student motivation we reviewed, Babic obtained a better accuracy of 76.9% for predicting student motivation with learning interaction logs and neural network models (Babić, 2017). Wang et al. (2022) reported a better accuracy of 92.8% in predicting student motivation with physiological data (EEG and eye gaze) and logistic regression model. Also, an accuracy of 90% was obtained in predicting student effort using multimodal data (consisting of learning interaction logs, eye-tracking, EEG, wristband, and facial expressions) and Hidden Markov Models (Sharma et al., 2020). These studies have revealed valuable insights into the usefulness of learning interaction logs, physiological data, and machine learning methods for dynamic modeling of student motivation. More details on modeling student motivation and adaptation of teaching and learning based on it are provided in Orji and Vassileva (2021).

In general, considerable progress has been made in using student interaction logs or their physiological data for detecting student affective states, engagement, and motivation in learning through machine learning techniques. The techniques that have been applied in developing various models range from simple (classification) to sophisticated ones (convoluted neural network) that can diagnose and predict student needs during learning process. The transformational potential of modeling techniques facilitates a more accurate estimation of student needs so that appropriate learning intervention could be delivered to support the student. The models have been found useful in designing smarter educational systems that serve better the students, educators, and administrators. The information provided by the models is used by online educational systems to adapt their responses dynamically to each student through providing personalized learning content, help, instructions, and feedback. It is also employed in understanding the effect of different learning processes and behaviors.

The importance of online educational systems in supporting teaching and learning cannot be overemphasized, however, to make the systems more effective and reduce the issues of high attrition rates of students, the need for dynamic adaptation of teaching and learning based on each student's needs has been highlighted. Adaptation is accomplished based on student models and the models identify student needs through their characteristics. Thus, this study identified student characteristics that are being modeled automatically and how. Researchers, educators, and online educational systems designers will benefit from this study as it could be used as a guide for decision-making when creating student models for adaptive educational systems that could detect students' needs during learning processes and adapt appropriate learning interventions based on the detected needs. Moreover, the study revealed progress made in the application of machine learning for automatic modeling of student characteristics. Therefore, machine learning researchers could benefit from this study as they can further advance this area by investigating how to improve the accuracy of the models and other techniques that have not been explored.

Future work

With advances in computers and human-computer interaction, the use of interaction or physiological data has been shown to be a viable solution for creating adaptive educational systems with suitable algorithms. The current state-of-the-art data acquisition methods and machine learning techniques offer promising opportunities to show that effective student characteristics models can be developed and applied for various decision-making in learning contexts. Thus, there are many opportunities for future research in this area including:

  • Further examining various learning features, preparation methods, and machine learning techniques that may improve the accuracy of student characteristics models developed with interaction or physiological data.

  • The link between interaction and physiological data features with student characteristics has been established, it will be interesting to investigate if the features generalize across learning contexts.

  • Furthermore, few researchers build student models with a combination of different data modalities (interaction and physiological data) to detect a variety of student characteristics. The researchers revealed that multimodal models that combine different feature metrics such as facial expression and interaction data provided improved accuracy than unimodality models. Robust models of student characteristics could be achieved using multimodal data, thus building more effective multimodal models, and determining the extent to which adaptations based on these models enhance learning and performance is an important research direction.

Conclusion

This paper provides a review of the trends and effectiveness of machine learning for automatic modeling of student characteristics. Our review results demonstrated that interest in automatic modeling of a wide range of student characteristics (especially student affective states, engagement, motivation, learning style, and emotion) using machine learning continues to grow. A research trend on dynamic modeling of the characteristics become prominent in the past years and considerable progress has been made in this area, with the goal of developing more effective models that can identify student needs during teaching and learning to improve adaptivity and student learning. Detecting student characteristics during the learning process and adapting teaching tactics will help to support better student learning. Thus, researchers are investigating automatic modeling methods such as machine learning techniques for dynamically assessing in real-time student characteristics using their digital traces and physiological responses obtained during learning processes with online educational systems. Current advances in technology such as cheap miniature sensors, digital cameras, and machine learning algorithms, enable unobtrusive continuous measurement of physiological and interaction data of students during learning, offering the ability to predict student characteristics during various learning contexts. The ability of the models to detect the precise needs of the students will improve the success of adaptive learning interventions that depend on them. The paper presented an analysis and classification of student characteristics and the most prevailing machine learning algorithms for modeling them over the last 8 years, along with some future trends.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Author contributions

Both authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

Funding

This work was supported by NSERC through the Vanier Canada Graduate Scholarship CGV-175722 of FO and the Discovery Grant program RGPIN-2021-03521 of JV.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

  1. Abyaa A., Khalidi Idrissi M., Bennani S. (2018). “Predicting the learner's personality from educational data using supervised learning,” in In Proceedings of the 12th International Conference On Intelligent Systems: Theories and Applications, Rabat, Morocco, October 2018. 10.1145/3289402.3289519 [DOI] [Google Scholar]
  2. Aggarwal S., Lamba M., Verma K., Khuttan S., Gautam H. (2021). A preliminary investigation for assessing attention levels for Massive Online Open Courses learning environment using EEG signals: an experimental study. Hum. Behav. Emerg. Technol. 3, 933–941. 10.1002/hbe2.274 [DOI] [Google Scholar]
  3. Aissaoui O., El El Madani, Y. E. A., Oughdir L., Allioui Y. E. (2019). Combining supervised and unsupervised machine learning algorithms to predict the learners' learning styles. Procedia Comput. Sci. 148, 87–96. 10.1016/j.procs.2019.01.012 [DOI] [Google Scholar]
  4. Al-Shabandar R., Hussain A. J., Liatsis P., Keight R. (2018). Analyzing learners behavior in MOOCs: an examination of performance and motivation using a data-driven approach. IEEE Access 6, 73669–73685. 10.1109/ACCESS.2018.2876755 [DOI] [Google Scholar]
  5. Altuwairqi K., Jarraya S. K., Allinjawi A., Hammami M. (2021). Student behavior analysis to measure engagement levels in online learning environments. Signal Image Video Process 15, 1387–1395. 10.1007/s11760-021-01869-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Aluja-Banet T., Sancho M. R., Vukic I. (2019). Measuring motivation from the virtual learning environment in secondary education. J. Comput. Sci. 36, 100629. 10.1016/j.jocs.2017.03.007 [DOI] [Google Scholar]
  7. Amir E. S., Sumadyo M., Sensuse D. I., Sucahyo Y. G., Santoso H. B. (2017). “Automatic detection of learning styles in learning management system by using literature-based method and support vector machine,” in 2016 Int. Conf. Adv. Comput. Sci. Inf. Syst. ICACSIS 2016 (Malang: ), 141–144. 10.1109/ICACSIS.2016.7872770 [DOI] [Google Scholar]
  8. Arroyo I., Woolf B. P., Burelson W., Muldner K., Rai D., Tai M. (2014). A multimedia adaptive tutoring system for mathematics that addresses cognition, metacognition and affect. Int. J. Artif. Intell. Educ. 24, 387–426. 10.1007/s40593-014-0023-y [DOI] [Google Scholar]
  9. Ashwin T. S., Guddeti R. M. R. (2020). Automatic detection of students' affective states in classroom environment using hybrid convolutional neural networks. Educ. Inf. Technol. 25, 1387–1415. 10.1007/s10639-019-10004-6 [DOI] [Google Scholar]
  10. Babić I. D. (2017). Machine learning methods in predicting the student academic motivation. Croat. Oper. Res. Rev. 8, 443–461. 10.17535/crorr.2017.002828496266 [DOI] [Google Scholar]
  11. Bauer M., Bräuer C., Schuldt J., Niemann M., Krömker H. (2019). “Application of wearable technology for the acquisition of learning motivation in an adaptive E-learning platform,” in Advances in Intelligent Systems and Computing (Berlin: Springer Verlag; ), 29–40. 10.1007/978-3-319-94619-1_4 [DOI] [Google Scholar]
  12. Beauchaine T. P. (2001). Vagal tone, development, and gray's motivational theory: toward an integrated model of autonomie nervous system functioning in psychopathology. Dev. Psychopathol. 13, 183–214. 10.1017/S0954579401002012 [DOI] [PubMed] [Google Scholar]
  13. Bixler R., D'Mello S. (2016). Automatic gaze-based user-independent detection of mind wandering during computerized reading. User Model. User-adapt. Interact. 26, 33–68. 10.1007/s11257-015-9167-1 [DOI] [Google Scholar]
  14. Booth B. M., Seamans T. J., Narayanan S. S. (2018). “An evaluation of EEG-based metrics for engagement assessment of distance learners,” in Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. EMBS 2018-July (Honolulu: ), 307–310. 10.1109/EMBC.2018.8512302 [DOI] [PubMed] [Google Scholar]
  15. Botelho A. F., Baker R. S., Heffernan N. T. (2017). “Improving sensor-free affect detection using deep learning,” in International Conference on Artificial Intelligence in Education (Berlin: Springer Verlag; ), 40–51. 10.1007/978-3-319-61425-0_4 [DOI] [Google Scholar]
  16. Boulay B. d. (2011). “Towards a motivationally intelligent pedagogy: how should an intelligent tutor respond to the unmotivated or the demotivated?,” in New Perspectives on Affect and Learning Technologies (New York, NY: Springer; ), 41–52. 10.1007/978-1-4419-9625-1_4 [DOI] [Google Scholar]
  17. Castillo-Merino D., Serradell-López E. (2014). An analysis of the determinants of students' performance in E-learning. Comput. Hum. Behav. 30, 476–484. 10.1016/j.chb.2013.06.020 [DOI] [Google Scholar]
  18. Chan T. S., Ahern T. C. (1999). Targeting motivation - adapting flow theory to instructional design. J. Educ. Comput. Res. 21, 151–163. 10.2190/UJ04-T5YB-YFXE-0BG2 [DOI] [Google Scholar]
  19. Chattopadhyay S., Zary L., Quek C., Prasad D. K. (2021). Motivation detection using EEG signal analysis by residual-in-residual convolutional neural network. Expert Syst. Appl. 184, 115548. 10.1016/j.eswa.2021.115548 [DOI] [Google Scholar]
  20. Cho M. H., Heron M. L. (2015). Self-regulated learning: the role of motivation, emotion, and use of learning strategies in students' learning experiences in a self-paced online mathematics course. Distance Educ. 36, 80–99. 10.1080/01587919.2015.1019963 [DOI] [Google Scholar]
  21. Conati C., Merten C. (2007). Eye-tracking for user modeling in exploratory learning environments: an empirical evaluation. Knowl. Based Syst. 20, 557–574. 10.1016/j.knosys.2007.04.010 [DOI] [Google Scholar]
  22. DeFalco J. A., Rowe J. P., Paquette L., Georgoulas-Sherry V., Brawner K., Mott B. W., et al. (2018). Detecting and addressing frustration in a serious game for military training. Int. J. Artif. Intell. Educ. 28, 152–193. 10.1007/s40593-017-0152-1 [DOI] [Google Scholar]
  23. Dewan M. A. A., Murshed M., Lin F. (2019). Engagement detection in online learning: a review. Smart Learn. Environ. 6, 1–20. 10.1186/s40561-018-0080-z [DOI] [Google Scholar]
  24. Dubbaka A., Gopalan A. (2020). “Detecting learner engagement in MOOCs using automatic facial expression recognition,” in IEEE Glob. Eng. Educ. Conf. EDUCON 2020-April (Porto: ), 447–456. 10.1109/EDUCON45650.2020.9125149 [DOI] [Google Scholar]
  25. Edmond Meku Fotso J., Batchakui B., Nkambou R., Okereke G. (2020). “Algorithms for the development of deep learning models for classification and prediction of behaviour in MOOCS,” in Proc. 2020 IEEE Learn. With MOOCS, LWMOOCS 2020 (Porto: ), 180–184. 10.1109/LWMOOCS50143.2020.9234363 [DOI] [Google Scholar]
  26. El Kerdawy M., El Halaby M., Hassan A., Maher M., Fayed H., Shawky D., et al. (2020). The automatic detection of cognition using EEG and facial expressions. Sensors 20, 1–32. 10.3390/s20123516 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Erkan E., Eduardo G.-S., Miguel L. B.-L., Yannis D., Juan I. A.-P. (2020). Generating actionable predictions regarding MOOC learners' engagement in peer reviews. Behav. Inf. Technol. 39, 1356–1373. 10.1080/0144929X.2019.1669222 [DOI] [Google Scholar]
  28. Ghaleb E., Popa M., Hortal E., Asteriadis S., Weiss G. (2019). “Towards affect recognition through interactions with learning materials,” in Proc. - 17th IEEE Int. Conf. Mach. Learn. Appl. ICMLA 2018 (Orlando, FL: ), 372–379. 10.1109/ICMLA.2018.00062 [DOI] [Google Scholar]
  29. Gledson A., Apaolaza A., Barthold S., Günther F., Yu H., Vigo M. (2021). “Characterising student engagement modes through low-level activity patterns,” in Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization (New York, NY: Association for Computing Machinery; ), 88–97. 10.1145/3450613.3456818 [DOI] [Google Scholar]
  30. Graesser A. C., D'Mello S. K., Craig S. D., Witherspoon A., Sullins J., McDaniel B., et al. (2008). The relationship between affective states and dialog patterns during interactions with auto tutor. J. Interact. Learn. Res. 19, 293–312. [Google Scholar]
  31. Halawa M. S., Hamed E. M. R., Shehab M. E. (2016). “Personalized E-learning recommendation model based on psychological type and learning style models,” in 2015 IEEE 7th International Conference on Intelligent Computing and Information Systems, ICICIS 2015 (Piscataway, NJ: Institute of Electrical and Electronics Engineers Inc.), 578–584. 10.1109/IntelCIS.2015.7397281 [DOI] [Google Scholar]
  32. Henderson N., Rowe J., Paquette L., Baker R. S., Lester J. (2020). “Improving affect detection in game-based learning with multimodal data fusion,” in International Conference on Artificial Intelligence in Education (New York, NY: Springer; ), 228–239. 10.1007/978-3-030-52237-7_19 [DOI] [Google Scholar]
  33. Hmedna B., El Mezouary A., Baz O. (2020). A predictive model for the identification of learning styles in MOOC environments. Cluster Comput. 23, 1303–1328. 10.1007/s10586-019-02992-4 [DOI] [Google Scholar]
  34. Hung J. C., Lin K. C., Lai N. X. (2019). Recognizing learning emotion based on convolutional neural networks and transfer learning. Appl. Soft Comput. 84, 105724. 10.1016/j.asoc.2019.105724 [DOI] [Google Scholar]
  35. Hussain M., Zhu W., Zhang W., Abidi S. M. R. (2018). Student engagement predictions in an e-learning system and their impact on student course assessment scores. Comput. Intell. Neurosci. 2018, 6347186. 10.1155/2018/6347186 [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Hutt S., Grafsgaard J. F., D'Mello S. K. (2019). “Time to scale: generalizable affect detection for tens of thousands of students across an entire school year,” in Conference on Human Factors in Computing Systems - Proceedings (Glagsow: ACM Digital Library; ). 10.1145/3290605.3300726 [DOI] [Google Scholar]
  37. Jaques N., Conati C., Harley J. M., Azevedo R. (2014). “Predicting affect from gaze data during interaction with an intelligent tutoring system,” in International Conference on Intelligent Tutoring Systems (Berlin: Springer Verlag; ), 29–38. 10.1007/978-3-319-07221-0_4 [DOI] [Google Scholar]
  38. Jiménez S., Juárez-Ramírez R., Castillo V. H., Ramírez-Noriega A. (2018). Integrating affective learning into intelligent tutoring systems. Univers. Access Inf. Soc. 17, 679–692. 10.1007/s10209-017-0524-1 [DOI] [Google Scholar]
  39. Joe Louis Paul I., Sasirekha S., Uma Maheswari S., Ajith K. A. M., Arjun S. M., Athesh Kumar S. (2019). “Eye gaze tracking-based adaptive e-learning for enhancing teaching and learning in virtual classrooms,” in Information and Communication Technology for Competitive Strategies (Singapore: Springer; ), 165–176. 10.1007/978-981-13-0586-3_17 [DOI] [Google Scholar]
  40. Khan F. A., Akbar A., Altaf M., Tanoli S. A. K., Ahmad A. (2019). Automatic student modelling for detection of learning styles and affective states in web based learning management systems. IEEE Access 7, 128242–128262. 10.1109/ACCESS.2019.2937178 [DOI] [Google Scholar]
  41. Kuh G. D. (2009). The national survey of student engagement: conceptual and empirical foundations. New Dir. Institutional Res. 2009, 5–20. 10.1002/ir.283 [DOI] [Google Scholar]
  42. Lallé S., Murali R., Conati C., Azevedo R. (2021). “Predicting co-occurring emotions from eye-tracking and interaction data in metatutor,” in International Conference on Artificial Intelligence in Education (Berlin: Springer Science and Business Media Deutschland GmbH; ), 241–254. 10.1007/978-3-030-78292-4_20 [DOI] [Google Scholar]
  43. Lei H., Cui Y., Zhou W. (2018). Relationships between Student Engagement and Academic Achievement: a meta-analysis. Soc. Behav. Pers. 46, 517–528. 10.2224/sbp.705431099030 [DOI] [Google Scholar]
  44. Li G., Wang Y. (2018). “Research on leamer's emotion recognition for intelligent education system,” in Proc. 2018 IEEE 3rd Adv. Inf. Technol. Electron. Autom. Control Conf. IAEAC 2018 (Chongqing: ), 754–758. 10.1109/IAEAC.2018.8577590 [DOI] [Google Scholar]
  45. Liu X., Ardakani S. P. (2022). A machine learning enabled affective E-learning system model. Educ. Inf. Technol., 1–22. 10.1007/s10639-022-11010-x [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Liu Y., Chen J., Zhang M., Rao C. (2018). Student engagement study based on multi-cue detection and recognition in an intelligent learning environment. Multimed. Tools Appl. 77, 28749–28775. 10.1007/s11042-018-6017-2 [DOI] [Google Scholar]
  47. Llanda C. J. R. (2019). “Video tutoring system with automatic facial expression recognition: an enhancing approach to e-learning environment,” in ACM International Conference Proceeding Series (New York, NY: Association for Computing Machinery; ), 5–9. 10.1145/3321454.3321462 [DOI] [Google Scholar]
  48. Lwande C., Muchemi L., Oboko R. (2019). “Behaviour prediction in a learning management system,” in 2019 IST-Africa Week Conf. IST-Africa 2019. (Nairobi). 10.23919/ISTAFRICA.2019.8764877 [DOI] [Google Scholar]
  49. Mohamad Nezami O., Dras M., Hamey L., Richards D., Wan S., Paris C. (2020). “Automatic recognition of student engagement using deep learning and facial expression,” in Joint European Conference on Machine Learning and Knowledge Discovery in Databases (Berlin: Springer; ), 273–289. 10.1007/978-3-030-46133-1_17 [DOI] [Google Scholar]
  50. Moher D., Shamseer L., Clarke M., Ghersi D., Liberati A., Petticrew M., et al. (2016). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Rev. Esp. Nutr. Humana y Diet. 20, 148–160. 10.1186/2046-4053-4-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Monkaresi H., Bosch N., Calvo R. A., D'Mello S. K. (2017). Automated detection of engagement using video-based estimation of facial expressions and heart rate. IEEE Trans. Affect. Comput. 8, 15–28. 10.1109/TAFFC.2016.2515084 [DOI] [Google Scholar]
  52. Muilenburg L. Y., Berge Z. L. (2005). Students barriers to online learning: a factor analytic study. Distance Educ. 26, 29–48. 10.1080/01587910500081269 [DOI] [Google Scholar]
  53. Orji F. A., Vassileva J. (2021). “Modelling and quantifying learner motivation for adaptive systems: current insight and future perspectives,” in International Conference on Human-Computer Interaction (Berlin: Springer Science and Business Media Deutschland GmbH; ), 79–92. 10.1007/978-3-030-77873-6_6 [DOI] [Google Scholar]
  54. Pintrich P. R., Smith D. A. F., Garcia T., Mckeachie W. J. (1993). Reliability and predictive validity of the motivated strategies for learning questionnaire (Mslq). Educ. Psychol. Meas. 53, 801–813. 10.1177/0013164493053003024 [DOI] [Google Scholar]
  55. Popenici S. A. D., Kerr S. (2017). Exploring the impact of artificial intelligence on teaching and learning in higher education. Res. Pract. Technol. Enhanc. Learn. 12, 1–13. 10.1186/s41039-017-0062-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Raj N. S., Renumol G. V. (2022). Early prediction of student engagement in virtual learning environments using machine learning techniques. Sage J. 2022, 204275302211080. 10.1177/20427530221108027 [DOI] [Google Scholar]
  57. Salmeron-Majadas S., Baker R. S., Santos O. C., Boticario J. G. (2018). A machine learning approach to leverage individual keyboard and mouse interaction behavior from multiple users in real-world learning scenarios. IEEE Access 6, 39154–39179. 10.1109/ACCESS.2018.2854966 [DOI] [Google Scholar]
  58. Santos P. B., Bhowmik C. V., Gurevych I. (2020). “Avoiding bias in students' intrinsic motivation detection,” in International Conference on Intelligent Tutoring Systems (Berlin: Springer; ), 89–94. 10.1007/978-3-030-49663-0_12 [DOI] [Google Scholar]
  59. Schunk D. H., Pintrich P. R., Meece J. L. (2008). “Motivation in education: theory, research, and applications, 3rd Edn,” in Upper Saddle River NJ Pearson/Merrill Prentice Hall. Available online at: http://www.sciepub.com/reference/202995 (accessed May 6, 2021).
  60. Sharma K., Papamitsiou Z., Olsen J. K., Giannakos M. (2020). “Predicting learners' effortful behaviour in adaptive assessment using multimodal data,” in ACM Int. Conf. Proceeding Ser. (Frankfurt: ), 480–489. 10.1145/3375462.3375498 [DOI] [Google Scholar]
  61. Shi Z., Zhang Y., Bian C., Lu W. (2019). “Automatic academic confusion recognition in online learning based on facial expressions,” in 14th Int. Conf. Comput. Sci. Educ. ICCSE 2019 (Toronto: ), 528–532. 10.1109/ICCSE.2019.8845348 [DOI] [Google Scholar]
  62. Shute V. J., Zapata-Rivera D. (2012). “Adaptive educational systems,” in Adaptive Technologies for Training and Education (Cambridge: Cambridge University Press; ), 7–27. 10.1017/CBO9781139049580.004 [DOI] [Google Scholar]
  63. Skinner E. a., Kindermann T. A., Connell J. P., Wellborn J. G. (2009). Engagement and disaffection as organizational constructs in the dynamics of motivational development. Handb. Motiv. Sch. 223–246. [Google Scholar]
  64. Standen P. J., Brown D. J., Taheri M., Trigo M. J. G., Boulton H., Burton A., et al. (2020). An evaluation of an adaptive learning system based on multimodal affect recognition for learners with intellectual disabilities. Br. J. Educ. Technol. 51, 1748–1765. 10.1111/bjet.13010 [DOI] [Google Scholar]
  65. Stark E. (2019). Examining the role of motivation and learning strategies in student success in online versus face-to-face courses. Online Learn. J. 23, 234–251. 10.24059/olj.v23i3.1556 [DOI] [Google Scholar]
  66. Taylor S. F., Welsh R. C., Wager T. D., Phan K. L., Fitzgerald K. D., Gehring W. J. (2004). A functional neuroimaging study of motivation and executive function. NeuroImage 32, 1045–1054. 10.1016/j.neuroimage.2003.10.032 [DOI] [PubMed] [Google Scholar]
  67. Tobarra L., Robles-Gómez A., Ros S., Hernández R., Caminero A. C. (2014). Analyzing the students' behaviour and relevant topics in virtual learning communities. Comput. Human Behav. 31, 659–669. 10.1016/j.chb.2013.10.001 [DOI] [Google Scholar]
  68. Tonguç G., Ozaydin Ozkara B. (2020). Automatic recognition of student emotions from facial expressions during a lecture. Comput. Educ. 148, 103797. 10.1016/j.compedu.2019.103797 [DOI] [Google Scholar]
  69. Wang M.-T., Degol J. (2014). Staying engaged: knowledge and research needs in student engagement. Child Dev. Perspect. 8, 137. 10.1111/cdep.12073 [DOI] [PMC free article] [PubMed] [Google Scholar]
  70. Wang R., Chen L., Ayesh A. (2022). Multimodal motivation modelling and computing towards motivationally intelligent E-learning systems. CCF Trans. Pervasive Comput. Interact. 2022, 1–18. 10.1007/s42486-022-00107-4 [DOI] [Google Scholar]
  71. Wang Y., Liu Q., Stein D., Xia Q. (2019). Measuring students affective states through online learning logs — an application of learning analytics. Int. J. Inf. Educ. Technol. 9, 356–361. 10.18178/ijiet.2019.9.5.1226 [DOI] [Google Scholar]
  72. Wiggins J. B., Boyer K. E., Baikadi A., Ezen-Can A., Grafsgaard J. F., Ha E. Y., et al. (2015). “JavaTutor: an intelligent tutoring system that adapts to cognitive and affective states during computer programming,” in Proceedings of the 46th ACM Technical Symposium on Computer Science Education (New York, NY: Association for Computing Machinery; ), 599–599. 10.1145/2676723.2691877 [DOI] [Google Scholar]
  73. Xiong Y., Li H., Kornhaber M. L., Suen H. K., Pursel B., Goins D. D. (2015). Examining the relations among student motivation, engagement, and retention in a MOOC: a structural equation modeling approach. Glob. Educ. Rev. 2, 23–33. [Google Scholar]
  74. Yang L., Qi C. (2020). “Study on learner's facial expression recognition,” in Proc. - 2020 Int. Conf. Netw. Netw. Appl. NaNA 2020 (Haikou City: ), 174–179. 10.1109/NaNA51271.2020.00038 [DOI] [Google Scholar]
  75. Yukselturk E., Bulut S. (2007). Predictors for student success in an online course. Educ. Technol. Soc. 10, 71–83. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Articles from Frontiers in Artificial Intelligence are provided here courtesy of Frontiers Media SA

RESOURCES