A systematic literature review on adaptive content recommenders in personalized learning environments from 2015 to 2020

Abstract

In personalized learning, each student gets a customized learning plan according to their pace of learning, instructional preferences, learning objects, etc. Hence the content recommender system in Personalized Learning Environment (PLE) should adapt to learner attributes and suggest appropriate learning resources to aid the learning process and improve the learning outcomes. This systematic literature review aims to analyze and summarize the studies on learning content recommenders in adaptive and personalized learning environments from 2015 to 2020. The publications were searched using proper keywords and filtered using the inclusion and exclusion criteria, which resulted in 52 publications. This paper summarizes the recent trends in research on different aspects of the recommender systems, such as learner attributes, recommendation methods, evaluation metrics, and the usability tests used by the researchers. It is observed that cognitive aspects of learners like learning style, preferences, knowledge level, etc., are used by most studies than non-cognitive aspects as social tags or trust. In most cases, recommendation engines are a hybrid of collaborative filtering, content-based filtering, ontological approaches, etc. All models were evaluated for the correctness of the prediction done, and a few studies have also done evaluations based on learner satisfaction or usability.

Introduction

Technology's rapid progression has affected many facets of our lives, especially those related to education. The impact of the COVID-19 pandemic resulted in the drastic shift of learning from traditional classrooms to e-learning environments (Basilaia & Kvavadze, 2020; Sun et al., 2020). Electronic gadgets, especially mobile devices, applications, learning environments, and internet usage, have made learning simpler and quicker and improve learning. The virtual learning environments find more popularity in higher education and distance learning, providing free access and immersive participation to learners worldwide through the internet and other technologies (Bylieva, 2020).

As  Chrysafiadi and Virvou (2013) have shown, the target learners are no longer just undergraduate students; these courses are open to anyone who desires to learn. So, the learner community is a large number of heterogenous enrolees. But when learning resources are delivered to learners systematically, the "one-size-fits-all" approach may not be beneficial. In addition, the vast number of learning resources available on the internet could contribute to knowledge overload (Raj & Renumol, 2018). The learning environment should be tailored to improve learners' learning performance and satisfaction (Hwang et al., 2020b). Personalized learning shows how each learner's pace of learning, instructional preferences, and learning objects are tailored to their specific needs. Adaptivity is the ability of the system to adapt to the changing needs of the learners as they progress in their studies (Dorca  et al., 2017). Adaptive/Personalized learning maximizes learning and assists learners in effectively completing course objectives in less time and at a lower cost. The use of adaptivity in assessment, focusing on remediation, is linked to a significant increase in learning gains while having no significant impact on drop-out rates (Rosen et al., 2018). Xie et al. (2019) reviewed the literature on PL from 2007 to 2017, where they have analyzed the major research issues such as the parameters of adaptive/personalized learning, learning supports, learning outcomes, subjects, participants, hardware, etc. From the study, the authors concluded that the spectrum of personalized learning is getting more comprehensive with the advancements in Artificial Intelligence, Cloud Computing, Wearable Computing, and Virtual Reality. Thus, for developing personalized learning environments, various aspects of individual learners need to be analyzed. The data acts as a guide for the instructors to design pedagogy and learning materials (Moreno-León et al., 2017; Essalmi et al., 2010). Machine learning and data mining techniques need to be applied to generate helpful information (Piety et al., 2014).

The learning environments logs data regarding the various learning aspects of the students, which can be used for gauging the student's performance, grouping them based on their similarities, detecting undesirable behaviors, and also recommending courses and course materials to them (Romero & Ventura, 2010; Baker et al., 2014). Content authoring and content recommendation are two essential features of a learning environment. In e-learning, a recommender system's job is to suggest suitable learning materials to learners and assist them in making decisions. Recommender systems are a type of information retrieval in which learning resources are filtered and presented to students (Aguliar et al., 2017). The available data are mined to provide suitable recommendations to the learners. Several indications demonstrate the efficacy of adaptation and personalization in e-learning environments. The authors recommend indicators such as making course content available at the student's choice and allowing the learner to study at their own space and pace (Lerís et al., 2017; Raj & Renumol, 2019).

This paper provides a critical review of research works that have applied data analytics in e-learning, especially in adaptive and personalized content recommendation. The article also tries to identify the popular techniques associated with these recommendation models. The primary objectives of this review are:

1. To analyze and summarize the research happenings in personalized learning environments from 2015 to 2020.

2. Identify the different recommendation techniques, personalization parameters, models, algorithms, evaluation metrics for e-learning content recommendation systems.

The organization of this paper is as follows. The “Related works” secion deals with the similar literature reviews conducted in e-learning content recommender systems by other researchers. The “Adaptive content recommender system—a general framework” section presents the system framework of e-learning content recommender systems. The “Methodology” section discusses the methodology used in this paper. The “Analysis of literature on content recommender systems” section compares and summarizes different content recommendation models reviewed as part of this study, and the “Analysis of results” section projects the inferences from the review. The “Discussion” section discusses the results concerning the study’s objective, section and the limitations of the study, and the “Conclusion” section concludes the entire process.

Related works

In the last few years, e-learning researchers have written several review articles on e-learning recommender systems. This segment describes some of the current reviews. The appropriate suggestions made by the recommender systems help learners in decision-making toward self-regulated learning (Fatahi et al., 2016; Aguilar & Riofrio, 2017). Klašnja-Mili'cevi'c et al. (2015) conducted an inclusive review of recommender systems in e-learning environments. The study focused on essential requirements and challenges in designing recommender systems in e-learning environments. The paper summarized tag-based recommender systems as a future scope and expected more research in possible extensions with prototypes for tagging activities. But from later studies, it is evident that tag-based systems showed declination in popularity among researchers. Tarus et al., (2018a, 2018b) conducted a literature review on ontology-based recommender systems for e-learning from 2005 to 2014. The paper discusses different hybridization of techniques. The authors detail the various aspects of a knowledge-based recommender system. They also identify that the hybrid scenarios will help to overcome issues like data sparsity and cold start. The authors found that ontology working with other knowledge structures like knowledge vectors, case-based reasoning, social knowledge, and constraint-based reasoning will be future research trends.

George and Lal (2019) conducted a systematic literature study based on the papers from 2010 to 2018, which compared the non-ontology and ontology-based techniques for providing the content recommendation. The non-ontology methods discussed in the article are matrix factorization-based, machine learning-based, tag-based, group-based, and user-based recommendation systems. The paper concludes that among the other techniques used for recommendations in the learning domain, the research works that have used ontology showed better results. But the review is not discussing the reason behind the better acceptance of ontology-based systems. Both learner and learning object characteristics are factored in using ontologies on the papers under their discussion. By hybridizing the techniques, researchers are trying to overcome the margins of one process. Traditional systems try to combine different styles either to feed data in the system or to develop algorithms. The authors also observed that using ontology with other techniques produces better results.

The review also discusses several techniques to calculate learner similarity based on their interests. Zhong and Xie (2019) summarized five assessment aspects of e-learning recommender systems. They are the metrics for the e-learning system, the evaluation metrics for the recommendation algorithms, the recommendation filtering technology, the phases of the recommendation process, and the system's learning outcomes. They concluded that most e-learning systems would adopt the adaptive mechanism as the central aspect and accuracy as a vital performance index for the algorithms. Mangaroska and Giannakos (2019) analyzed 43 articles from the literature from 2010 to 2017. This article highlights the knowledge of relating learning analytics with learning design to develop tools, methods, and models to benefit instructors and learners. The authors insist on using LA to design personalized learning and feedback and lessen the conventional mode of lecturing, reading, or watching videos.

Drachsler et al. (2015) reviewed 82 recommender systems from 35 countries and classified them using a specific classification process. According to their characteristics, the examined systems were divided into seven clusters and analyzed. The clusters are (a) Cluster 1: Recommending resources for learning based on CF (b) Cluster 2: Improving CF algorithms with TEL domain particularities (c) Cluster 3: Educational constraints as a source of information (d) Cluster 4: Exploring non-CF techniques to find successful educational recommendations (e) Cluster 5: Considering contextual information (f) Cluster 6: Assessing the educational impact of recommendations (g) Cluster 7: Recommending courses. The paper summarized the frameworks used in recommender systems and the possible challenges in designing and developing the frameworks.

The literature reviews on e-learning recommender systems show that the studies are done and published on recommendation techniques like collaborative filtering, content-based and hybrid recommendation techniques. These reviews include investigating different recommendation models that use conventional recommendation techniques, ontology-based strategies, machine learning algorithms, and comparisons between different recommendation strategies. But these reviews included recommender systems where data from systems other than PLE is also analyzed. The authors were unable to locate any study that focuses only on e-learning content recommendation. This study is intended to bridge this gap by consolidating the recent research works, focusing on adaptive and personalized content recommender studies conducted from 2015 to 2020.

Adaptive content recommender system—a general framework

The primary goal of a content recommender system is to provide suggestions to both the learners and the instructors who interact with the system. The learners can get adaptive content recommendations, and instructors can use the inferences for course design and content authoring. The basic framework of a content recommendation system is given in Fig. 1 (Raj & Renumol, 2018).

Fig. 1.

A general framework of an adaptive personalized content recommender

When the learner is active for the first time, the basic preferences are collected using a questionnaire, and Learner Modeling Unit develops an initial Learner Model (LM). The input attributes are specific learner parameters such as their learning style, media preference, and level of knowledge. Later, as the learner becomes more active, the Learner Modeling Unit should alter the learner model accordingly by adapting to the changing preferences of the learner. The Learner Monitoring Unit assesses the learner's performance, logs the interactions, and analyzes the changes in the preferences. The instructor provides the learning objects. The Content Managing Unit contains the learning object model and the recommender engine. The recommender engine is supported with recommendation algorithms. Various measures are used to find the similarity between the learner attributes, to cluster them, or to cluster the learning patterns to recommend appropriate learning objects (Joy & Renumol, 2020). Based on this similarity measure, learner and learning object mapping is done in the content managing unit, and the recommender engine will recommend relevant learning objects to the learners. The monitoring unit gets feedback from the learners so that the system can adapt to changing needs of the learner.

Thus, the steps involved in developing an adaptive, personalized content recommender are:

1. Collect data and develop the learner and learning object (LO) model.

2. Group the learners in terms of their similarities. Identify the rules by which the learners and LOs can be mapped.

3. Generate top 'N' learning object recommendations.

4. Get feedback from the learners. Identify the learning paths from the learner activity log.

5. Revisit the mapping between learner and LO based on feedback.

Methodology

This survey paper aims to analyze and summarize the research happenings in personalized learning environments from 2015 to 2020. The systematic review (Fig. 1) is done in three phases (Xiao & Watson, 2019).

1. Search the repositories with selected keywords.

2. Apply inclusion and exclusion criteria.

3. Detailed analysis and summarization of the content.

The authors analyzed the literature in the databases Scopus (https://www.scopus.com) and the Web of Science Core Collection (www.webofknowledge.com), libraries of the Institute of Electrical and Electronics Engineers (IEEE, https://ieeexplore.ieee.org), and the Association for Computing Machinery (ACM, https://dl.acm.org/). Google Scholar is used as the seed search engine with the keywords, and later the literature is imported from the above-said repositories. The filter search is done using the keywords like "recommender system", "e-learning", "hybrid recommender system", "ontology", "ontology-based recommender system", "intelligent tutoring system", "knowledge-based systems", "machine learning for recommender". Always the filter search is done with the keywords mentioned above combined with strings as "personalized" and "adaptive learning." The papers were selected based on their relevance to the domain under review. Few records are removed from this set of records as the full-text version of those articles was inaccessible.

After retrieving the publications, the authors analyzed the abstract, conclusion, methodology, and keywords to select the most relevant papers.

The articles were chosen based on the following inclusion criteria:

• Proposing recommendation techniques for personalized content recommendation.

• Research is an empirical study.

• Offering detailed discussion on methodology, design, development, and evaluation of the recommendation models.

• Converging on adaptive/personalized e-learning environments

• Published in the journals from 2015 to 2020, considering the indexing and impact factors of the journal. Articles from leading conference proceedings are also included as they show high relevance in the domain under study.

Again, a filtering step is carried out to exclude few articles based on the exclusion criteria:

Authors excluded those articles with all or any one of the following criteria:

• The recommended items are neither learning resources nor learning activities.

• The methodology, design, and evaluation of recommendation models are vague.

• The article is not written in English.

• The article is from the proceedings of Seminars/Workshops.

Further, the content of the selected papers was thoroughly read, analyzed, and summarized. The following section discusses the publications based on the models, recommendation techniques, evaluation criteria, and usability studies. Figure 2 represents the stages in the literature review methodology adopted.

Fig. 2.

Stages of literature review

Analysis of literature on content recommender systems

An adaptive or personalized e-learning system tries to understand learners' individual needs and preferences for supporting teaching–learning activities. Literature shows that there are many techniques by which researchers attempted to achieve this goal. A content recommender system should fundamentally decide two strategies, (i) Learner/Learning Object Model and (ii) the recommendation technique, including algorithm and evaluation methods used. This section exposes these aspects of the recommender systems concerning the recent publications. Table 1 discusses the recommendation methods, input attributes, and summary of the publications selected for the current study.

Table 1.

Comparison and summary of publications in e-learning content recommendation during 2015–2020

Citation	Method	Attributes	Summary
Aeiad and Meziane (2019)	Hybrid (Ontology Model, Rule-Based)	Learning style, Knowledge level	The architecture of a customized e-learning framework to collect open-source LO is developed. To retrieve learning tools from the internet, Google API is used. The learning concepts and the semantic relationships between them are stored hierarchically using an ontology
Al Abri et al. (2020)	The skill-based system with collaborative filtering on an ontology base	FSLSM learning style Knowledge level	The LMS's interaction layer uses web crawling strategies to gather learner interaction data from social media networks. The collected data are subjected to various data mining and text mining techniques to build a domain model that incorporates the learners' expertise
Albatayneh et al. (2018)	Content-Based	Learner's negative ratings	Analyzed the discussions to know the learner needs and recommended learning objects with an improved learning performance of 9.8%. Latent Semantic Analysis is done with cosine similarity for recommendation
Anuradha et al. (2020)	Agent-based hybrid system	Learner requirements/needs	A multi-agent-based clustering technique is proposed to increase personalization and facilitate individual learning performance. The Firefly optimization algorithm is combined with a differential evolution algorithm in their model. The hybridization technique is used to solve clustering issues that arise during the distribution of materials
Araújo et al. (2017)	Rule-Based	FSLSM Learning style, IEEE LOM	The proposed architecture allows students' learning styles to be assessed using the FSLSM's probabilistic proposal
Benhamdi et al. (2017)	Hybrid of Collaborative and Content-Based Filtering	Learner preferences, Knowledge level	Introduced NPReL, a new recommendation system based on collaborative and content-based filtering (New multi-Personalized Recommender for e-Learning)
Bhaskaran and Santhi (2019)	Trust-based system with rule mining	Learning style, Learner behavior	In their model, learners are clustered based on the learning style attribute using k-means and firefly algorithms. The learner preferences and behavior are analyzed using the AprioriAll algorithm
Bouihi and Bahaj (2019)	Ontology Model/Semantic Web-based model	Learning context	Also, the architecture has four sets of Semantic Web Rule Language (SWRL) rules that aids in filtering learning objects based on their relevance and weightage. The set of rules are Learning History Rules (LHR), Learning Performance Rules (LPR), Learning Social Network Rules (LSNR), and Learning Pathway Rules (PR)
Bourkoukou and El Bachari (2016b)	Collaborative Filtering	FSLSM Learning style	Some modules for personality detection and choosing an appropriate learning scenario for the learner's personality are described in this model. The dataset from PSLC DataShop is used for experimentation. K-Means and KNN algorithms are used along with Euclidean distance and Pearson Similarity measures. The performance of the model is checked against MAE and execution time
Bourkoukou et al. (2017)	Hybrid of Collaborative Filtering and Pattern Mining	Access patterns	KNN and GSP Algorithms are used for finding the similarity in access patterns. This is a hybrid system combining collaborative filtering and Sequential Pattern Mining
Bourkoukou et al.(2016a)	Collaborative Filtering	FSLSM Learning style	The learning model is designed to generate personalized learning experiences by selecting and sequencing the most appropriate learning materials
Christudas et al. (2018)	Genetic Algorithm with Ontological Model	FSLSM Learning style, Knowledge level, Interactivity level	Input parameters are tuned using a Compatible Genetic Algorithm (CGA), triggered when a new learning instance happens. The standard genetic algorithms are taken as the baseline methods, and in comparison, with the baseline, the proposal showed promising results. The model usability study is done based on the scores of the students
Deng et al. (2018)	Collaborative filtering and Trust Analysis	Content features, Learner ratings, Social-trust data	The proposed model combined Deep Neural Network (DNN) and Collaborative Topic Regression (CTR) model to enhance the Collaborative Filtering technique's Performance by integrating learner ratings with content features. Later, by incorporating the social trust information into the rating prediction, the model could balance the learners' personal preferences and trusted friends' interests
Dorca et al. (2016)	Rule-Based	FSLSM Learning style, IEEE LOM	The study's main contribution is the set of rules by which the learner characteristics and LO attributes can be mapped. The authors themselves identify that the work lacks detailed experimentation with any metrics
Dorca et al. (2017)	Rule-Based	FSLSM Learning style, IEEE LOM	They grouped the learners based on their learning style into 16 different clusters. The experiments were done over 1600 LOs where they had 100 LOs corresponding to each learning style cluster
Dwivedi and Bharadwaj (2015)	Collaborative Filtering	Knowledge levels, FSLSM learning style, Learner ratings	A Genetic Algorithm (GA) is adopted to group the learners based on their attributes, and knowledge levels are integrated using simple mean calculation. Here recommendation is made to a group of learners who shows similarity in individual preferences. Evaluation is done by finding Mean Absolute Error (MAE), precision, and recall
Dwivedi et al. (2018)	Content-Based	FSLSM Learning style, Knowledge level	Recommended a learning route, a sequence of learning materials in proper order with a starting and finishing point. To improve learners' learning capacities, the proposed sequence also reflected the learner's preferences
Fraihat and Shambour (2015)	Ontology Model	Learner requirements, LO attributes	Make use of intra and extra semantic links between LOs and the learner's needs. Used shortest path algorithm
Hwang et al.(2020a)	Hybrid (Fuzzy Logic Based, Rule-Based)	Learners' affective and cognitive factors	Through Ontology Model serving the learner's learning behaviors, the expert system determines the learner's affective states, and the learner's cognitive status is registered. To suggest pre-classified LOs based on the above criteria, a set of fuzzy inference rules is used
Ibrahim et al. (2020)	Hybrid (Ontology Model, Fuzzy Logic Based)	Learning goals, Learner Interests	The contents' relevant class is chosen based on the frequency concept and weight concept for a requested topic in the class identification module. Later, a subclass for the requested matter is identified with the association rule mining technique's help. Finally, the similarity between the user request query and course ontologies is calculated using fuzzy logic to recommend personalized and learning goal-based content to a target learner
Imran et al. (2016)	Attribute-based	learning styles, expertise level, prior knowledge, Performance	Presented PLORS, which can be plugged in with LMSs to recommend LOs. The recommendation techniques used are neighborhood algorithm and associate rule mining. The usability of learning Objects is computed based on the search history of learners
Jagadeesan and Subbiah (2020)	Skill-based	Learner skills	Emphasis is given to the knowledge level of the learners. Learners with advanced skills are provided with cutting-edge content, intermediate learners with moderate content, and beginners or slow learners with the primary content
Joy et al. (2019)	Ontology-based model	FSLSM, Prior Knowledge, Leaner Preference	The research aims at solving the cold-start issue in recommending LOs. SWRL queries are used to make predictions with a Learner and Learning Object Ontology framework. User ratings are used for evaluating the performance
Klašnja-Milićević et al. (2018a)	Collaborative Filtering	Learning style, Preferences, Learning patterns	Compared the suitability of various strategies for implementing tag-based suggestions in e-learning. To obtain the most efficient recommendation results, the most appropriate model ranking, based on tensor factorization methodology, has been updated
Klašnja-Milićević et al. (2018b)	Hybrid of tag-based and association rule mining models	Learners personal, Performance, and history data, Learning patterns, Social tags	The AprioriAll algorithm is used to identify learning patterns. Following the learners' grouping, an initial tensor is generated by specifying a collection of tags that they use. After that, by dividing and reducing the initial tensor, the factorized tensor is computed. Finally, the device generates a list of the top N tags related to the target learner's unique needs
Kolekar et al. (2019)	Rule-Based	FSLSM learning style	The proposed model creates design maps for each learner to indicate a learning direction. In different learner classes, the system's output is assessed using statistical methods
Kouis et al. (2020)	Correlation-based	FSLSM learning style	Created an e-learning system focused on the FSLSM learning style for accessing learning material in an LMS. A correlation matrix has been added to their model to measure the compatibility of e-learning content and volume with learning styles
Labib et al. (2017)	Ontology Model	Learning style	The aim was to semantically link different learning style dimensions with learner characteristics to find appropriate learning material
Murad et al. (2020)	Hybrid (Collaborative Filtering, Rule-Based)	Learning outcome scores, Contextual information	The qualitative knowledge is derived from the learners' results on the entrance exam. A collection of decision rules is combined with the predicted learning outcome in the recommendation engine to suggest appropriate learning materials for the target learner. The model's benefit is that it can be applied directly in academic departments
Nabizadeh et al. (2020)	Content-Based	Estimated learner score and time	The LOs have sequenced with Depth First Search (DFS) algorithm. The model calculates the scores and time requirements before sequencing the LOs. The advantage of the model is that whenever the learner fails to complete a lesson with the estimated score, auxiliary LOs are provided. Usability of the system is checked using the learner scores and evaluating learner satisfaction
Nafea et al. (2018)	Hybrid (Collaborative and Item Content-based Filtering)	FSLSM Learning style	They have applied the k-means clustering algorithm to group the learners and improve the recommendation system's Performance. The study consists of a small LO set of 20 learning a student dataset of size 80. Also, the details of the recommendation strategy are vague in this publication
Nafea et al. (2019)	Hybrid (Collaborative, Content-Based Filtering)	FSLSM Learning style	Improved their earlier architecture of hybrid recommendation strategy consisting of Collaborative Filtering and Content-Based techniques. They have used IEEE LOM to model learning objects and FSLSM to model learners. Addressed data sparsity and cold-start issues. The adaptivity and dynamicity of the system are not detailed in both papers
Nihad et al. (2020)	Hybrid (Content-based Reasoning, Fuzzy-based Logic)	FSLSM learning style	They have created a MAALS (multiagent adaptive learning system) to make real-time learning decisions based on the current learning situation. The model has the advantage of having adequate decision-making capabilities at all times. Fuzzy logic is used for recommendation
Ouf et al. (2017)	Ontology Model	Learner characteristics	Suggested Learner model ontology, LO ontology, learning tasks ontology, and teaching methods ontology as the four components of their domain ontology
Perumal et al. (2019)	Fuzzy Logic-based pattern mining model	Domain knowledge, Portal hit similarity,	The study looks at how user interests shift over time to come up with repeated learning trends. The learner cluster is built using their portal hit similarity and domain awareness
Rahman and Abdullah (2018)	Group-based	Academic, behavioral, contextual information, learning goal	The recommender system act as an interface between the Google Search Engine and the institutions' portal. Adaptive learner grouping is done based on preferences and similarities. Thus, a group-based algorithm serves to filter out personalized learning materials from the search engine results. Based on the learner group, the algorithm to re-ranks and prioritizes the search
Riyahi and Sohrabi (2020)	Hybrid (Collaborative Filtering, Content Based)	Learner ratings, Content features, Tags	The most similar users to the active user are identified with the users' implicit ratings in the Collaborative Filtering phase. The sparsity problem is eliminated in their model using the content features in the recommendation engine
Saleena and Srivatsa (2015)	Ontology Model	Learning concepts	The learning document is retrieved after cross-ontology similarities between learning concepts are measured using the feature set and neighborhood set and the similarity ranking
Sarwar et al. (2019)	Hybrid of case-based reasoning and neural networks	Learning style, Knowledge level	Machine learning techniques have been used to develop a mathematical model for learner categorization. CourseOntology was used to annotate the learning material, and three academic courses (each for the languages of C +  + , C#, and JAVA) were modeled for the learners. A dynamic rule-based recommender has been provided
Segal et al. (2019)	Collaborative Filtering	Learner preferences	Presented EduRank, an algorithm that combines a collaborative filtering algorithm with voting mechanisms to personalize educational content for students
Senthilnayaki et al. (2015)	Fuzzy Logic applied in the ontology-based system	Learning style	The proposed ontology discovery method was practically tested in an e-Learning environment against student scores to teach the subject Database Management Systems. KNN with Jaccard Similarity is used
Shi et al. (2020)	Semantic-based	Learner needs	To store and organize LOs in different groups, the model includes a multidimensional information path structure. Information graphs are designed with semantic relationships to meet pre-defined learning needs and propose personalized learning paths
Tarus et al. (2017)	Collaborative Filtering in an Ontology Model	FSLSM learning style, Access patterns, Learner ratings	The authors have combined a sequential pattern mining (SPM) technique with the ontology model to identify the learners' historical sequential patterns. IEEE LOM is used to model LO
Tarus et al., (2018a, 2018b)	Collaborative Filtering with Pattern Mining	Context awareness, Sequential access patterns	Compared to the author's earlier work, the SPM is replaced with a Generalized Sequence Pattern (GSP) algorithm to understand the sequential access patterns of the learners. Their approach performs better in sparse conditions by using contextual information and sequential patterns in the absence of learner ratings
Vanitha and Krishnan (2019)	Population-based	Learning Objectives, Cognitive capability, Emotional state, Performance	LOs are represented as nodes in the artificial ant model, and ants are modeled as learners. Learning goals, emotional state, cognitive capacity, and learners' success are part of their learner model
Venkatesh and Sathyalakshmi (2020)	Hybrid (Collaborative Filtering, Content-Based)	Learner behavior, Content features	Suggested personalized bee recommender for e-learning (PBReL) based on artificial bee colony (ABC) optimization and uses K-means clustering to generate a recommendation structure. Experiments are carried out using the Moodle-based learning management system's web linkages and materials (LMS)
Wan and Niu (2016)	Rule-Based	Learning goals, Competency, Attitude, Content, media preference	The LOs are recommended by combining a self-organization-based (SOB) recommendation approach with sequential pattern mining
Wan and Niu (2018)	Content-Based recommendation in Learner and LO Ontology framework	Learning goals, Learning style, Learner behavior	The proposed recommendation approach's Performance is compared against the genetic algorithm, Markov Chain approach, and traditional teaching method. The usability is checked using different factors as Learner scores, learning time, LO utilization, Entropy, Fitness, Satisfaction level
Wan and Niu (2019)	Hybrid of collaborative filtering and pattern mining	Learner influence, Access patterns	They have proposed a hybrid recommendation approach based on learners' influence propagation. Their Recommendation model uses a learner influence model (LIM) to compute the influence of a learner on others
Xiao et al. (2018)	Hybrid of content-based and association rules with Collaborative Filtering	Learner behavior, Learner ratings	The suggestion method considers the learners' learning preferences and browsing history and the amount of time spent on each learning material when using the tutoring framework
Zhang et al. (2019)	Deep Learning-based	Learner and content attributes	Presented an LO recommendation model (MOOCRC). The method employs a Deep Belief Networks (DBN) classification model to detect and extract the learner features in the MOOC environment. The model's performance and accuracy are not evaluated using actual learner data instead of using the MovieLens dataset
Zhu et al. (2018)	Semantic-based Knowledge Graph	Learner needs	They suggested a method for generating learning paths that involve the specification of starting and ending nodes. Multi Constraints are used to define learners goal/needs and to recommend a suitable learning path

Analysis of results

This study examined research works on the adaptive content recommendation in e-learning environments, published from 2015 to 2020. The literature repository contains 52 papers which are analyzed and summarized in the previous section. This review process focuses on finding the different attributes/techniques that the research works adopted in learner/learning object modeling, recommendation process, and evaluation. On summarizing the results, the researchers are curious to know the systems' adaptivity and dynamicity. Like any other system, the recommender systems also need to be checked based on the system's usability. Here, the authors try to mine the works to know if they do a usability check or not?

1. Personalization Parameters and Models

2. Recommendation Techniques

3. Usability of the system

Personalization parameters and models

This subsection discusses the primary models used in the e-learning content recommendations. The learner modeling attributes are learning style, learner preferences, knowledge level, learning paths and patterns, learner skills, pre-defined tags, context, etc.

Table 2 represents the set of prevalent parameters used for learner modeling in the content recommender systems under review.

Table 2.

Set of prevalent parameters used for learner modeling in content recommender systems

Parameters	Citation
Learning style/Learner Preferences	Dwivedi and Bharadwaj (2015), Senthilnayaki et al. (2015), Imran et al. (2016), Dorca et al. (2016), Bourkoukou and El Bachari (2016), Bourkoukou et al.(2016), Tarus et al.(2017), Labib et al. (2017), Ouf et al. (2017), Araújo et al., (2017), Nafea et al. (2018), Christudas et al.(2018), Wan and Niu (2018), Dwivedi et al. (2018), Klašnja-Milićević et al. (2018b), Aeiad and Meziane (2019), Kolekar et al. (2019), Joy et al. (2019), Segal et al. (2019), Sarwar et al. (2019), Bhaskaran and Santhi (2019), Perumal et al. (2019), Nafea et al. (2019), Kouis et al. (2020), Al Abri et al. (2020), Nihad et al. (2020)
Knowledge level	Dwivedi and Bharadwaj (2015), Imran et al. (2016), Benhamdi et al. (2017), Nafea et al. (2018), Christudas et al. (2018), Dwivedi et al. (2018), and Joy et al. (2019)
Learning path/patterns	Bourkoukou et al. (2017), Tarus et al., (2018a, 2018b), Klašnja-Milićević et al. (2018a), Klašnja-Milićević et al. (2018b), Wan and Niu (2019), and Zhu et al. (2018)
Performance/ Score	Imran et al. (2016), Nabizadeh et al. (2020), Rahman and Abdullah (2018), and Jagadeesan and Subbiah (2020)
Learner Ratings	Dwivedi and Bharadwaj (2015), Tarus et al. (2017), and Deng et al. (2018)
Portal Hit Similarity	Perumal et al. (2019),
Social Tags	Klašnja-Milićević et al. (2018a)
Social Trust	Deng et al. (2018)
Learning need/goal	Anuradha et al. (2020), Ibrahim et al. (2020), Wan and Niu (2016), Shi et al. (2020), and Zhu et al. (2018)
Cognitive/Emotional States	Vanitha and Krishnan (2019) and Hwang et al.(2020a),

Felder-Silverman Learning Style Model (FSLSM) is the most popular modeling technique used among the works analyzed. Felder and Silverman (Felder, 1988) presented a theoretical model in which each student can be classified according to four dimensions: perception, input, processing, and organization. (1) Perception classifies learners based on how they perceive the contents, and the dimension has two classes Sensitive (Sen) and Intuitive (Int). (2) Input, where the classification is done based on the format of the content presented to study, and the classes are Visual (Vis) and Verbal (Ver). (3) Processing, which indicates the measure of the active involvement of the student toward the content presented, and the classes are Active (Act) and Reflective (Ref). (4) Organization: Classifies students as sequential, if they prefer content exhibited progressively and more restricted view; are Sequential (Seq) and Global (Glo). To find each learner's dominant learning style, a survey with 44 questions, the Felder -Soloman Index of Learning Style questionnaire, is used (Graf et al., 2007, Joy et al., 2019). The questions are analyzed, and a probabilistic learner model is constructed and fed as an input to the system. Even though the FSLSM is not an adaptive model, it can alleviate the model's cold-start issue. Many recommender systems use the FSLSM as an initial fuel and later switch to learning path or learning pattern analysis for being adaptive.

Aeiad and Meziane (2019) focused on another learning style, VARK (Visual, Auditory, Reading/Writing, Kinaesthetic). VARK Learning Styles Theory is introduced by Fleming, N.D. and Mills, C. (Othman & Amiruddin, 2010). The VARK proposes four learning attributes. (1) Visual (V): learning by viewing a picture, diagram, and graphs (2) Auditory (A): learning by listening to explanations or group discussion. (3) Read/Write (R): learning by reading or writing. (4) Kinaesthetic (K): learning by experience or simulation. VARK has lesser acceptance compared with FSLSM as the latter is a probabilistic model.

Few recommender systems took a survey on learner preferences about learning objects explicitly, on the system's initiation. For example, questions like "Do you prefer visual or verbal learning content?" are asked. The knowledge level is historical data that the learner gives at the system's entry point. The learners are classified as, Novice, Intermediate, Advanced based on their subject knowledge. This parameter is assessed by explicitly surveying the learner by conducting a pre-test. The learning path is identified after the initiation of the recommender system. This method cannot be used under cold-start conditions. A path is a sequence of learning objects visited by the learner in time (Zhu et al., 2018). The paths are recommended for similar users who specify the initial or/and final learning objects to be studied.

Few studies have evaluated the learners' performance at regular intervals, and recommendations are made based on the score (Table 2). Learner's ratings, both positive and negative, are taken as a learner modeling parameter in few studies. Similar learners behave similarly in rating the learning objects. Motivated by the business domain, few studies adopted social tags/trust for classifying learners and recommend learning materials to them. Recently, research shows interest in using cognitive states of learners by analyzing how they interact with the system, how engaged they are, and their emotional state and relationship between these states and learning patterns.

The content recommender always tries to find the relation between learners and learning objects. So, it is equally important to model the learning objects appropriately for the system. The IEEE LOM is observed to be of high popularity among the researchers who believe in modeling the LOs (Araújo et al., 2017, Dorca et al. 2016; Dorca et al., 2017; Tarus et al., 2017). Some studies used specific content features and media attributes (Wan and Niu 2016; Deng et al., 2018; Venkatesh and Satyalakshmi 2020). The Learning Object Metadata Schema, created by IEEE Working Group P1484.12, is one of the most promising metadata schemas (Shen et al., 2002). It was primarily inspired by IMS and ARIADNE's work (Alliance of Remote Instructional Authoring and Distribution Networks for Europe). IEEE LOM has a wide range of data field categories. The Educational Category was crucial for the recommendation systems because it would enable the system to focus on defining learning materials' pedagogical features. Learning management system data or the recommender system data are used by most researchers in their studies.

Recommendation techniques

The use of efficient and effective recommendation techniques is crucial to solving the problem of retrieving relevant learning objects for learners. The different aspects of recommendation techniques discussed here are

1. Recommendation Method

2. Algorithms Used

3. Similarity Measures Used

4. Evaluation techniques checking the correctness of the algorithm

The past years viewed many recommendation techniques and algorithms as collaborative filtering, content-based filtering, fuzzy-based systems, context-aware systems, tag-based systems, group-based systems, ontology-based systems, rule-based, trust-aware systems, and social networking-based systems.

The collaborative filtering strategy leverages the learner's feedback (rating history) to cluster comparable learners and provide relevant recommendations for the future, which is the most prevalent RS design technique (Bourkoukou et al., 2016). The basic idea is that if users' tastes were similar in the past, they would have similar tastes in the future. The rating history is the essential factor in determining how similar two users are. The principle of content-based recommendation is based on the similarity computation of the item features associated with the compared objects (Dwivedi et al., 2018). Both collaborative filtering and content-based models heavily depend on the similarity measures used in the implementation.

Ontology is used to represent knowledge in ontology-based systems (Tarus et al., 2018a, b). Semantic relations are established between learners and learning objects. They are the best solution for cold start and sparsity problems. But the disadvantage is it's challenging, expensive, and time-consuming to build the ontologies. Group-based models analyze the behavior of a group of learners and make recommendations. They use different learner characteristics for grouping the learners. In contrast, the skill-based models observe the similarity in skills among the learners and cluster them based on their abilities. Both group-based and skill-based systems use collaborative or content-based methods to recommend the learning objects after initial clustering (Jagadeesan and Subbiah, 2020; Rahman  and Abdullah, 2018).

In this period, 2015–2020, the models based on ontologies emphasize the inclusion of fuzzy techniques and hybrid methods that include algorithms genetics to enhance adaptive learning. Hybrid approaches are observed to have mostly collaborative or content-based models combined with an ontological framework. The following table, Table 3, shows that researchers show affinity toward hybrid models compared to others. The publications are grouped based on different recommendation methods.

Table 3.

Most used recommendation techniques in the e-learning domain

Recommendation method	Citation
Ontology-Based	Fraihat and Shambour (2015), Saleena and Srivatsa (2015), Labib et al. (2017), Ouf et al. (2017), Bouihi and Bahaj (2019), and Joy et al. (2019)
Collaborative Filtering	Dwivedi and Bharadwaj (2015), Bourkoukou and El Bachari (2016), Bourkoukou et al. (2016), Klašnja-Milićević et al. (2018a), and Segal et al. (2019)
Content-Based	Albatayneh et al. (2018), Dwivedi et al. (2018), and Nabizadeh et al. (2020)
Hybrid	Senthilnayaki et al. (2015), Tarus et al.(2017), Bourkoukou et al. (2017), Benhamdi et al. (2017), Tarus et al.,(2018a, 2018b), Nafea et al. (2018), Christudas et al.(2018), Wan and Niu (2018), Klašnja-Milićević et al. (2018a), Xiao et al. (2018), Deng et al. (2018),Aeiad and Meziane (2019), Sarwar et al. (2019), Bhaskaran and Santhi (2019), Wan and Niu (2019), Nafea et al. (2019),Riyahi and Sohrabi (2020), Ibrahim et al. (2020), Venkatesh and Sathyalakshmi (2020), Murad et al. (2020), Al Abri et al. (2020), Nihad et al. (2020), Anuradha et al. (2020), and Hwang et al.(2020a)
Group-based	Rahman and Abdullah (2018)
Rule-Based	Wan and Niu (2016), Dorca et al. (2016), and Kolekar et al. (2019)
Skill-based	Jagadeesan and Subbiah (2020)

Recommendation techniques give better results when it is combined with suitable machine learning algorithms (Table 4). The algorithms cluster the learners, recognize learning patterns, and map learners with learning objects.

Table 4.

Machine learning algorithms

Machine learning algorithm	Citation
K-Nearest Neighbor	Dwivedi and Bharadwaj (2015), Bourkoukou and El Bachari (2016), Bourkoukou et al. (2016), Bourkoukou et al. (2017), Dwivedi et al. (2018), Sarwar et al. (2019), Murad et al. (2020), and Nihad et al. (2020)
K-Means	Senthilnayaki et al. (2015), Bourkoukou and El Bachari (2016), Vanitha and Krishnan (2019), Bhaskaran and Santhi (2019), Nafea et al. (2019), and Venkatesh and Sathyalakshmi (2020)
AssociationRule Mining/Apriori	Imran et al. (2016), Klašnja-Milićević et al. (2018a), and Ibrahim et al. (2020),
Sequence Mining Algorithm	Tarus et al. (2017)
Pattern Mining	Tarus et al., (2018a, 2018b)
Genetic Algorithm	Christudas et al. (2018), Dwivedi et al. (2018), and Anuradha et al. (2020),
Shortest Path Algorithm	Fraihat and Shambour (2015)
LO based Self-Organizing Algorithms	Wan and Niu (2018)

It is observed that the majority of the research works use K-Nearest Neighbor (KNN) and K-Means algorithms to cluster the items. Both of them are simple and powerful grouping algorithms. KNN is used in supervised learning and K-Means in unsupervised learning. Tarus et al., in their publications in 2017 and 2018, tried sequence and pattern mining algorithms, respectively. They were working with non-real-time data and attempted to explore patterns of learning objects ranked by the learners. Studies also use genetic algorithms for grouping learners/learning objects. Learning path explorers also tried to find the shortest path between sequences of learning things to fulfill the learning needs/goals of the learners. Figure 3 shows the distribution of the algorithms used in the e-learning recommender system.

Fig. 3.

Distribution of machine learning algorithms

The similarity measures used in the works are Euclidean distance similarity, Jaccard Coefficient, Cosine Similarity, Ontological Similarity, Pearson Correlation Coefficient, learner parameters-based similarity. Algorithmic calculations are also used for assuming similarities between different entities in the system. However, few studies have used multiple similarities for clustering either the learners or the learning objects or both. When the data are represented using an ontological framework, an ontological similarity measure is used. The research works with input data as learning style/learner preferences used Pearson Coefficient, Cosine Similarity, Euclidean Distance, or Jaccard Coefficient as the similarity measure.

The similarity measures, along with the publications, are given in Table 5.

Table 5.

Similarity measures

Similarity measure	Citation
Euclidean Distance Similarity	Imran et al. (2016), Bourkoukou and El Bachari (2016), Venkatesh and Sathyalakshmi (2020), Murad et al. (2020), Vanitha and Krishnan (2019)
Jaccard Coefficient	Senthilnayaki et al. (2015), Wan and Niu (2018), and Bourkoukou et al. (2016)
Cosine Similarity	Albatayneh et al. (2018), Bourkoukou et al. (2016), Tarus et al. (2017), Wan and Niu (2018),Nafea et al. (2019), Bourkoukou et al. (2017), Joy et al. (2019), Venkatesh and Sathyalakshmi (2020), and Riyahi and Sohrabi (2020)
Ontological Similarity	Saleena and Srivatsa (2015)
Pearson Correlation Coefficient	Tarus et al., (2018a, 2018b), Nafea et al. (2019), Riyahi and Sohrabi (2020), and Bourkoukou et al. (2016)
Learner Parameters-based Similarity	Dwivedi et al. (2018),

The most popular similarity measure is cosine similarity. It is used mainly with both hybrid and non-hybrid models. The data used in these cases are from learning management systems which are recorded or real-time learner logs. The preprocessed data are removed of null values. For comparison, studies have used Jaccard Coefficient and Pearson Correlation coefficient with many hybrid models to measure learner similarity. Euclidean distance similarity measures serve to cluster learners based on their preferences in collaborative filtering models. The study takes content and learner features as input data uses learner parameter-based similarity (Dwivedi et al., 2018). The models analyzing learning path or patterns also uses the correlation coefficient.

The evaluation methods in content recommenders aim to check the recommendation technique's correctness (Fazeli et al., 2017). The following table, Table 3, categorizes the references based on their recommender evaluation method.

The analysis shows that MAE, MSE, RMSE, accuracy, recall, and f-measure are the most commonly used standardized measurement measures in e-learning content recommender systems. Aside from that, the run-time complexity of some of the established procedures is assessed. In addition, many researchers used pre-and post-tests to evaluate student academic progress. The popularity assessment of different evaluation methods is shown in Fig. 4.

Fig. 4.

Prevalent evaluation methods

Usability

A recommendation's usefulness to the system or the user is referred to as utility (Fazeli et al., 2017). The learner could decide the utility of a recommendation explicitly (e.g., in learner-defined ratings), or the system can compute it by observing learner attributes (e.g., click-stream data). The utility of a recommendation can be determined by examining the learners' subsequent behavior, such as interacting with the recommendation or using prescribed learning objects. However, the studies are not focusing on increasing the system's usability; few studies use learner rating (Albatayneh et al., 2018; Deng et al., 2018; Tarus et al., 2017; Xiao et al., 2018). The user rating is primarily used as a learner/learning object attribute to train the systems.

Some studies use the learner's score and satisfaction level (Table 6) as an evaluation criterion of the model. Learner rating and satisfaction level are direct feedback, and scores are indirect feedback given by the learner (Shi et al., 2020). So, the input from the users in these cases is reflecting on the usability of the systems.

Table 6.

Evaluation methods

Evaluation method	Citation
MAE, Precision, Recall, RMSE	Dwivedi and Bharadwaj (2015), Bourkoukou and El Bachari (2016), Bourkoukou et al.(2016), Tarus et al.(2017), Bourkoukou et al.(2017), Tarus et al.,(2018a, 2018b), Albatayneh et al. (2018), Klašnja-Milićević et al., (2018a, 2018b), Xiao et al. (2018), Deng et al. (2018), Zhang et al. (2019), Bhaskaran and Santhi (2019), Nafea et al. (2019), Riyahi and Sohrabi (2020), Ibrahim et al. (2020), Venkatesh and Sathyalakshmi (2020), Jagadeesan and Subbiah (2020, Murad et al. (2020), Anuradha et al. (2020), and Nabizadeh et al. (2020)
Learner Score	Senthilnayaki et al. (2015), Wan and Niu (2016), Benhamdi et al. (2017), Christudas et al. (2018), Wan and Niu (2018), Dwivedi et al., (2018, Aeiad and Meziane (2019), Segal et al. (2019), Vanitha and Krishnan (2019), Sarwar et al. (2019), Jagadeesan and Subbiah (2020), Hwang et al.(2020a), and Nabizadeh et al. (2020)
Usage of LO	Wan and Niu (2016) and Wan and Niu (2018)
Run Time	Bourkoukou and El Bachari (2016), Christudas et al. (2018), Klašnja-Milićević et al., (2018a, 2018b), and Bhaskaran and Santhi (2019)
Learner Satisfaction	Tarus et al. (2017), Zhu (2018) Christudas et al.(2018), Wan and Niu (2018), Rahman and Abdullah (2018), Klašnja-Milićević et al., (2018a, 2018b), Nafea et al. (2019), Kouis et al. (2020), Shi et al. (2020), and Nabizadeh et al. (2020)
Learning Time	Wan and Niu (2018)

Discussion

The journal papers reviewed, rated, and categorized in this survey were excellent, particularly considering that most of them were downloaded from the Science Citation Index (SCI) and Scopus indexed journals (Table 1). The selected papers have more comprehensive content and are subject to a more stringent peer-review process. From the analysis, it is evident that the research work in the field is active throughout the review period. The development of research in the broader area of recommender systems and the acceptance of e-learning as a teaching and learning method by more institutions of higher learning can be attributed to this substantial growth and increase in research interest.

From Table 2, it is observed that both cognitive and non-cognitive aspects of the learners contribute toward learner modeling in the content recommendation. Learning style, pattern, knowledge level are examples of cognitive factors, while ratings, social tags, the polarity of hit, social trust are non-cognitive aspects. Learning styles are fundamental in achieving adaptivity in the system (Shemshack et al., 2021). The probability-based learning style, FSLSM, has received the most popularity among others. Most of the works do the initial questionnaire-based learning style or knowledge level survey to handle the cold-start issue. At an advanced stage of recommendation, learners' paths or patterns or ratings adapt to the learner's changing needs. Ratings are a strong indicator of the usability of the system also. Fuzzy learners use cognitive attributes to train the model (Hwang et al., 2020a). The adaptivity and dynamicity of the designs are ensured by exploring learning the paths and patterns within the system.

Table 3 displays that the hybrid systems are the most popular among other recommendation techniques. That includes the hybrid of ontology and knowledge-based systems. Most of the studies which followed hybridization have used collaborative filtering as one knowledge-based method. This trend shows that researchers are trying to unveil the hidden patterns and the diversity in learner behavior. Collaborative filtering clusters the learners based on their similarity or dissimilarity, thereby exposes common likeliness. Collaborative filtering is combined with trust-based, content-based, rule-based, pattern-based, and item-based filtering methods for recommending LOs. KNN and K-means are the popular learner grouping algorithms used across the publications under review. KNN, as it is a non-parametric and lazy learner, goes well with learner data, where data points are separated into clusters to interpret new samples. KNN relies only on feature similarity; it will calculate the distance of the learner attribute of one learner with all others and returns top N similar learners. So, in a feature similarity study, KNN performs well. General appreciation of the K-Means algorithm as an unsupervised classifier makes it a good performer in the e-learning recommendation.

On analyzing the similarity measures used in the recommender engine, if there are no null references in the input dataset, the cosine similarity measure performs well. Similarly, when data is usually distributed, Pearson Correlation Coefficient produces the best output. The popularity of Jaccard Coefficient and Euclidean distance similarity stems from their ease of use. The most common associates are K nearest neighbors and cosine similarity "fit for the Collaborative filtering method." Content-based strategies are commonly associated with space vector models. Analyzing the hybrid methods is based on integrating at least two techniques, using weighted algorithms; from commutation, cascade, and magnification functions. Based on semantic recommendation techniques used, it is observed that the hybridization is inclined to respond to the main limitations of the methods previously associated with cold-start (George & Lal, 2019). The recommendations carried out in the adaptive environment do not depend on the students' evaluations but are based on knowledge of the domain; this integrates relevance and inference feedback methods. The recommendation correctness is mainly calculated in Mean Absolute Error, Precision, or Recall (Table 6).

Instead of recommending top-N or top-1 LO, there are studies which recommend a sequence or pattern of LOs for a lesson using a knowledge graph (Shi et al., 2020; Zhu et al., 2018). These sequences are called a learning path. Here, the learners should specify the target topic (knowledge unit, KU) or the lesson they need to study. Then the model recommends a learning path, which starts from the initial KU, through the intermediate KUs till the final KU. For each KU, an LO is recommended based on the learner's performance or learner goals. These models are highly interactive and experimented with live data, whereas the methods which study the frequent patterns or rule-based patterns work with recorded data (Tarus et al., 2018a, 2018b). Interactive environments analyze students' performance or satisfaction and non-interactive environments analyze the learner ratings to evaluate the system's usability.

Unlike the earlier reviews, this study is confined to the content recommender system from the e-learning domain alone. The observations made in the current study, the classification of recommender model support systems, are in line with earlier studies (Zhong & Xie, 2019; Mangaroska & Giannakos, 2019). This means there is a continuity and correlation in the trends that the researchers follow. Zhong and Xie (2019) concluded that recommender systems would adopt the adaptive mechanism. The current study observed that the recent recommender models show adaptivity with the changing user needs. Mangaroska and Giannakos (2019) observed the growing demand for learning analytics in designing personalized learning environments. The current study observed that the selected publications apply machine learning algorithms to analyze the learner logs/feedback and predict suitable learning objects. Tarus et al., (2018a, 2018b) conducted exclusive studies with ontology-based systems, but the current research includes different recommendation strategies. Both studies converge on the observation that hybrid models are most popular in recommender systems.

There are few limitations to this review study as listed below:

• Due to potential subjectivity and a lack of relevant knowledge, bias in collecting databases, papers, and publications.

• Since keywords are discipline and language-specific, there is a bias in the search string.

• The bias from rejecting non-English papers; the works in other languages are not considered for the study, even though they seemed appropriate from the abstracts.

• The emphasis was on observational analysis, so; it was difficult to draw broader conclusions.

• The bias and vagueness in data extraction because only the two authors conducted it

• The bias from the interpretation of specific results, processes, or approaches since some parts of the research from the selected studies was not adequately represented.

On the other hand, the authors attempted to ensure an impartial review procedure by planning a study protocol with a pre-determined research objective. The search keywords are created based on the research objectives and consider the lack of standardization in keywords, varying by discipline and language. The authors performed a detailed search in the journals, repositories, and previous related reviews. Applying the literature review method described in Section “Methodology” the authors have done an in-depth study in the narrower area of content recommendation in e-learning.

Conclusion

An adaptive and personalized e-learning system aims to support the learners' individual needs and interests to facilitate the learning process. The researchers in this area have tried a variety of techniques to achieve this goal. This paper presents a systematic analysis of 52 publications on e-learning recommender systems published between 2015 and 2020. This review investigated the methods used, attribute selection, and model evaluation metrics used across the publications.

A hybrid of two or more recommendation methods like ontology-based, collaborative filtering-based is observed as the most commonly used technique. In contrast, the usage of non-hybrid ontology systems has been decreased in these years. Also, the designs are relying on explicit learner attributes. Some recommender system studies analyze the cognitive features of the learners as an implicit attribute. In many studies, the input attribute selection is based on learning style, knowledge level, and learning preferences. The learning pattern and learning path are computed in these works for recommending the content. Apart from MAE, MSE, RMSE, accuracy, recall, and f-measure, the models also use run-time, learner satisfaction, and learner performance to evaluate the models.

According to Shi et al. (2020) there is an increase in the usage of mobile devices in learning and the need for generating context-aware recommendations is relatively higher. The new research studies in this domain would be ubiquitous and autonomous systems using the knowledge in recommender systems.

Biographies

Nisha S. Raj

is a Ph.D. Scholar in the division of Information Technology, School of Engineering, Cochin University of Science and Technology, India. Her research interests include applications of EDM/LA in MOOCs, designing Personalized Learning Environments, and Recommender Systems. She is an activist in the field of novel practices in e-learning. Towards the same, she has delivered workshops and talks in Innovative Tools and Practises in e-learning ecologies.

V. G. Renumol

is a Professor in Information Technology, School of Engineering, Cochin University of Science and Technology, India. She secured her doctoral degree from IIT, Madras, India, and Post-Doctoral degree from IIT, Bombay, India. Her research interests include Computing Education, Cognitive Psychology, Personalised Learning, Educational Technology, Special Education, etc. She possesses several national and international publications as conference proceedings and journals