Integrating ontology and knowledge graphs for intelligent assessment and feedback in E-learning systems

Abstract

In many online courses, assessment is still reduced to a numerical score accompanied by brief, generic feedback. Learners with different difficulties often receive similar responses, while instructors have limited support for identifying at-risk students early. In this study, we present an ontology-driven knowledge graph framework for adaptive assessment and feedback. At its core is a reusable ontology that models learning outcomes, assessment tasks, learner characteristics, and feedback strategies. This ontology is instantiated as a knowledge graph and continuously updated with traces of learner activity, representing interactions and performance as semantically structured data. Assessment-to-learning-outcome alignment is defined through predefined mappings, enabling explicit reasoning over learner performance. Using SWRL rules and SPARQL queries, the system identifies at-risk learners, generates tailored feedback, and links each recommendation to its underlying evidence and reasoning steps. We evaluate the framework on the OULAD dataset. Compared with a simple threshold baseline, the approach achieves higher precision for detecting at-risk learners (0.80 vs. 0.68) and an overall F1 score of 0.77. In an expert-based evaluation, the generated feedback is judged more relevant (average gain of 0.7 points) and more explainable (average gain of 1.3 points), with substantial inter-rater agreement (κ = 0.69). On a knowledge graph of approximately 4.2 million triples, query times remain under one second, indicating that the approach can support near real-time feedback generation at institutional scale. These results suggest that ontology-driven knowledge graphs can provide a transparent and practical foundation for adaptive, explainable feedback in e-learning systems.

Introduction

Teachers and researchers usually agree on one thing: assessment and feedback matter. They sit close to the center of learning, shaping how students build knowledge, notice weaknesses and move toward intended outcomes^1,2. In online and blended courses, assessment is also expected to do extra work. It should not only certify performance at the end, but also steer adaptive learning paths and support more personalized teaching³. In practice, we still see something much simpler. On many e-learning platforms, feedback appears as a single score and a short, generic remark. Large logs of clicks, submissions and interactions are collected, but are rarely integrated in ways that meaningfully support instructors.

Part of the problem lies in how learner progress is modelled. Many systems rely on fixed thresholds or black-box machine learning models to decide who is doing well and who might be at risk. These approaches can hit reasonable accuracy on a test set, but they seldom represent the links between learning objectives, assessment tasks and a learner’s evolving knowledge state⁴. When we tried to interpret such models on LMS data, we often ended up with predictions that were difficult to explain to a tutor in concrete, pedagogical terms. This lack of interpretability limits their usefulness in real educational settings, where instructors need to understand not only what the system predicts, but also why those predictions are made.

A second difficulty is the way data and models are fragmented. Information about learners is scattered across different tools and databases, sometimes even within the same institution. That makes it hard to reuse knowledge across courses, and also hinders the generation of feedback that is personalized, context-aware and clearly justified at scale. Semantic web technologies, especially ontologies and knowledge graphs, offer one route out of this fragmentation⁵. Ontologies introduce a shared vocabulary for a domain, spelling out competencies, assessments and pedagogical strategies in a machine-readable form. Knowledge graphs then tie together heterogeneous data (student interactions, performance records, course structure) into a linked network that can support automated reasoning. Used together, they can align observed performance with intended learning outcomes and highlight learning patterns that are hard to see in flat tables or dashboards.

Several strands of work have moved in this direction, but mostly in a piecemeal way. Some studies use ontologies to model curricula or competencies. Other work builds knowledge graphs mainly for tasks such as resource recommendation⁶. A further line integrates ontologies into knowledge tracing models to improve interpretability⁴. When we looked across this literature, we found useful pieces but no single, scalable framework that connects a formal ontology and a dynamically instantiated knowledge graph and then uses that combination to drive end-to-end assessment and the generation of personalized, explainable feedback. In particular, existing approaches either emphasize predictive performance without transparency, or semantic modeling without fully operationalizing feedback generation, leaving a gap at the intersection of reasoning, explainability, and practical deployment.

More broadly, prior work tends to treat learner modelling, data integration, reasoning, and feedback generation as loosely coupled components, often optimized independently. As a result, the connection between data-driven inference and pedagogically grounded feedback remains implicit and difficult to interpret in practice. This paper proposes such a framework. Our goal is to make feedback in e-learning systems adaptive and easy to justify to both learners and instructors. Rather than introducing new reasoning algorithms, we focus on designing an operational framework that connects semantic modeling, reasoning, and feedback generation (see Sect. 2.4 for a detailed formulation of the contribution). This design enables traceable and pedagogically aligned decisions, while avoiding reliance on isolated predictions or purely descriptive analyses. In addition, instead of handling modelling, data integration and feedback generation as separate problems, we treat them as parts of one pipeline and make four main contributions:

1. Unified ontological model. At the base, we define a shared ontology that describes learners, assessments, learning outcomes and feedback strategies, designed to support reuse across educational contexts. The schema is intended to remain consistent across courses and platforms while preserving semantic meaning. In this study, the evaluation is limited to a higher-education dataset, and transferability to other contexts remains to be investigated.

2. Semantically enriched knowledge graph. On top of this ontology, the framework builds a knowledge graph that links heterogeneous learner interaction data back to the formal concepts. In our implementation, this step transformed noisy LMS logs into a structured graph that can be queried and used for reasoning. The instantiation relies on explicit mappings between assessment items and learning outcomes. Unlike typical KG-based educational systems focused on recommendation or data integration, the graph here is explicitly designed to support diagnostic reasoning and feedback generation grounded in pedagogical concepts.

3. Context-aware, explainable feedback engine. Using SWRL rules and SPARQL queries over the graph, the system generates feedback that adapts to the learner’s situation and can be traced back to specific ontology elements and data points. Each recommendation is accompanied by a clear rationale, which we found important when discussing results with instructors. The design emphasizes transparency and traceability of the reasoning process rather than increasing algorithmic complexity. This establishes a direct reasoning–feedback loop, where inference outputs are operationalized into structured, explainable feedback.

4. Prototype and empirical study. We implement the framework in OWL and GraphDB and test it with anonymized LMS data. The prototype suggests that the approach is not only conceptually coherent but also practical to deploy with a realistic, institution-scale dataset. Along the way, we encountered practical but important obstacles, such as cleaning log data and tuning query performance, that we also report. The evaluation combines quantitative performance metrics with expert-based assessment of feedback quality, providing an initial but limited validation of the approach.

Beyond the technical design, the framework gives a concrete way to connect semantic representation and reasoning with principles of formative assessment. Because feedback is generated directly from ontological concepts and explicit rules, we can begin to articulate design guidelines for AI-supported assessment that are transparent and learner-centered rather than opaque. We treat scalability and pedagogical interpretability as linked concerns, not as separate add-ons, and in that sense, we try to narrow the gap between data-driven analytics and established educational theory. The study is organized around three research questions:

1. RQ1 (Detection accuracy). How well does the ontology-driven reasoning engine identify struggling learners when compared with simpler baseline methods?

2. RQ2 (Feedback usefulness and explainability). How do human experts judge the feedback produced by the system in terms of relevance, actionability and clarity of explanation?

3. RQ3 (Robustness and scalability). How does the framework behave under ablation, and can it handle institution-scale datasets while keeping computational costs acceptable?

While these questions focus on system performance and expert evaluation, broader validation in terms of long-term learning outcomes and cross-domain generalization remains an important direction for future work. Issues such as long-term effects on learner outcomes or deployment in live classrooms are recognized as important, but they fall outside the scope of this initial study and are left for future work. The remainder of the paper is structured as follows. Section 2 surveys related work on ontologies in education, knowledge graphs for learning analytics and automated feedback systems, and highlights the gaps that motivate our approach. Section 3 presents the proposed framework, including its architecture, ontology extensions, instantiation pipeline and reasoning mechanisms. Section 4 describes the experimental design and reports results on detection accuracy, expert judgments of feedback and system scalability. Section 5 discusses these findings in the context of the broader literature and system-level adoption studies. Section 6 reflects on implications, limitations and directions for future work, and Sect. 7 concludes with a brief summary and outlook.

Related work

We build on three main strands of prior work. First, we look at how ontologies have been used in educational settings. Then we turn to knowledge graphs in learning analytics. Finally, we review intelligent assessment and feedback systems. This review allows us to position our framework and identify remaining gaps.

Ontologies in educational contexts

Ontologies have been part of e-learning research for quite a while. They are often used to describe knowledge domains, learning outcomes and teaching strategies in a structured, machine-readable way, usually with interoperability in mind⁷. Early work by⁸ highlighted the role of semantic web technologies and linked data in enabling interoperability and personalized learning experiences in e-learning environments. In particular, ontology-based representations were identified as key enablers for semantic retrieval, reasoning, and adaptive content delivery, supporting the transition from keyword-based systems to concept-driven learning infrastructures. In subjects like computer science and mathematics, for example, researchers have built ontologies to structure course content and support adaptive teaching scenarios⁹. Once these models exist, learning resources can be tagged with ontological concepts, which helps with targeted content delivery and keeps terminology and structure reasonably consistent across learning management systems (LMSs).

More recent work has explored the use of ontology-based knowledge bases for intelligent querying and retrieval in educational settings. For instance, Nguyen et al.¹⁰ proposed an ontology-driven framework for semantic search and knowledge querying, where user queries are interpreted through concept–relation structures and matched against a structured knowledge base. Their approach highlights how ontologies can support deeper semantic interpretation of learner queries compared to keyword-based methods, although reasoning capabilities remain largely constrained to predefined schema and rule structures.

This work, however, tends to stop at the level of content and curriculum. Processes such as ongoing assessment, diagnosis of learner difficulties and the generation of tailored feedback are often mentioned but not developed in much detail. Another pattern we noticed when reading this literature is the strong link to specific subject areas. Many educational ontologies are deeply tied to a single discipline, which makes them difficult to reuse when the course topic changes. In our own experiments with adapting existing ontologies, we often ended up rewriting large parts of the schema. This limitation becomes particularly critical when moving toward scalable learning analytics systems, where reuse across courses and contexts is essential. The framework we propose takes these issues seriously and starts from a design oriented toward reuse across contexts, whose core concepts are intended to transfer across subjects and instructional settings, while still being rich enough to support assessment and feedback.

Knowledge graphs for learning analytics

Knowledge graphs (KGs) have recently become popular in learning analytics as a way to store educational data in a connected, semantically rich form¹¹. Instead of spreading student records, logs and resources across separate tables, KGs bring heterogeneous data (performance scores, clickstreams, forum posts, learning materials) into a single graph. With this unified structure, it becomes easier to observe how particular activities, resources and behaviors group together for different types of learners. For learner modelling, this is attractive: dynamic profiles built from ongoing activity can be used to predict performance and support adaptive learning pathways¹².

In practice, the power of these graphs is often limited by weak semantic underpinnings. A KG without a clear ontological schema can still show that two nodes are connected, but it rarely explains what the connection means in educational terms. We have seen this in our own work with institutional data: the graph looked rich, yet our attempts to express “why this learner might be struggling” stayed vague. Without explicit modelling of learning objectives, misconceptions or instructional strategies, patterns and recommendations remain at a shallow level. This lack of formal semantic grounding makes it hard to generate feedback that is genuinely contextual, useful for teachers, and clearly aligned with pedagogical reasoning¹³.

A representative ontology-driven reasoning approach is presented by Reda et al.¹⁴, which demonstrates how OWL ontologies combined with SWRL rules can enable knowledge-driven reasoning beyond traditional trigger–action paradigms. Their approach highlights how semantic inference over structured data can support complex, context-aware decision-making while maintaining interpretability. In contrast to purely rule-based automation, such systems operate on formalized knowledge representations, enabling higher-level abstractions and improved interoperability. However, despite these advances, the integration of such semantic reasoning mechanisms into end-to-end educational feedback pipelines remains limited. Existing work also tends to assume that semantic alignments are predefined, without detailing how such mappings are constructed in practice.

Intelligent assessment and feedback systems

Research on automated assessment and feedback has grown quickly with the spread of machine learning (ML) and natural language processing (NLP). Current systems can grade assignments, scan discussion forums and produce formative comments for large cohorts with limited human effort¹⁵. More recently, Large Language Models (LLMs) have been plugged into this workflow, and many prototypes now rely on them to draft feedback that appears tailored to each learner and is generated within seconds¹⁶.

The trade-off is familiar. Many of these tools operate as black boxes. They can produce fluent and detailed feedback, but the reasoning behind a particular judgement is hidden from both teachers and students¹⁷. When we asked instructors to trust such systems on real courses, the lack of a clear explanation often became a sticking point. A small number of studies respond to this by combining ontologies and knowledge graphs with automated assessment to bring more structure and meaning into the pipeline¹⁸. From what we could find, though, there are still very few systems that go from live assessment data all the way to feedback that is both automated and explicitly explainable. Moreover, engagement- or affect-related signals are often incorporated implicitly or through heuristic features, with limited transparency regarding how such constructs are operationalized.

Research gap and our contribution

The three strands above leave a noticeable gap. Table 1 summarizes the main types of approach and the trade-offs we see in each case.

Table 1.

Comparison of existing approaches in learning analytics.

Approach type	Focus area	Strengths	Limitations	Key references
Ontology-based	Curriculum design, resource organization	Provides semantic structure, improves interoperability	Often domain-specific, minimal focus on dynamic assessment and feedback	7,9
KG-based	Learner modeling, recommendation	Captures relationships in heterogeneous data, supports personalization	Lacks semantic rigor without an ontology schema, focused on linking, not reasoning	11,12
Ontology-KG Integration	Intelligent tutoring, semantic search	Combines semantic rigor with dynamic graph reasoning	Few works explicitly address explainable assessment and feedback generation	13,18
ML/LLM-based	Automated grading, formative feedback	Highly scalable, can process unstructured data	Opaque “black-box” models, low interpretability, feedback lacks transparent reasoning	15,16

• Ontology-based systems offer strong semantic structure and help with interoperability, but they rarely handle dynamic assessment and detailed feedback.

• KG-based systems are effective at connecting heterogeneous data and supporting personalization, yet often rely on weak or implicit pedagogical semantics and focus more on linking than on reasoning.

• Ontology–KG integrations show that it is possible to combine semantic rigor with graph reasoning, but only a few explicitly target explainable assessment and feedback.

• ML/LLM-based systems scale well and can process unstructured data, but their internal logic is hard to inspect, and the feedback they generate usually comes without a transparent chain of reasoning.

More importantly, these approaches tend to treat semantic modeling, data integration, reasoning, and feedback generation as separate or loosely coupled processes, rather than as a single, coherent and operational pipeline. As a result, the link between data-driven inference and pedagogically grounded feedback remains implicit and difficult to interpret in practice.

Our work targets this gap. Rather than proposing another isolated model, we build an ontology-driven knowledge graph framework that keeps the whole assessment–feedback process in view. The ontology provides a shared and precise language for learners, tasks and outcomes. The knowledge graph connects that language to live learner data as it accumulates during a course. On top of both, semantic reasoning turns this information into feedback that is personalized and comes with a clear explanation.

The contribution of this work can be summarized along four complementary aspects:

1. the operationalization of semantic representations into a feedback-generating system;

2. the explicit coupling between reasoning and feedback generation;

3. the use of multi-hop inference across learner performance, engagement, and learning outcomes;

4. the integration of provenance-aware mechanisms that allow each feedback instance to be traced back to its underlying data and rules.

These elements define an operational pipeline in which semantic modeling, data integration, reasoning, and feedback generation are treated as interdependent stages rather than separate components. This enables decisions that are interpretable and pedagogically grounded, rather than purely predictive or descriptive.

When we trialed the framework with LMS data, this design allowed us to show instructors not only which learners were flagged as struggling, but also why the system reached that conclusion. In doing so, we make key modeling assumptions (e.g., outcome alignment and engagement indicators) explicit rather than implicit. This makes the reasoning process inspectable and supports the translation of inference results into structured, outcome-aligned feedback. Our aim is to move data-driven analytics a little closer to the kinds of assessment practices that teachers can understand, question and adapt. At the same time, recent developments in AI suggest that this gap is increasingly being approached through hybrid architectures that combine structured reasoning with flexible language generation. The next section situates our work within this emerging direction.

Symbolic reasoning and LLM-based surface realization

In addition to the approaches discussed above, recent work has explored the use of LLMs in educational feedback systems. Their ability to generate fluent and context-aware text makes them particularly appealing for feedback generation, where clarity and tone matter as much as content^19–21. In several recent prototypes, they are already used to draft comments on assignments or summarize learner progress, often in real time.

At the same time, their limitations become visible as soon as we look more closely at how feedback is produced. LLMs do not reason over structured representations in a strict sense; instead, they generate responses by estimating likely continuations of text^22,23. In educational settings, this can lead to a familiar tension. A piece of feedback may read well and sound convincing, yet still fail to reflect the underlying learning objectives or misinterpret the learner’s actual difficulties. Similar issues have been reported in other structured domains, where outputs that appear plausible can still violate constraints or drift away from formal knowledge representations^24,25.

Symbolic approaches offer a different set of trade-offs. When learning outcomes, assessment criteria and instructional strategies are explicitly modeled in an ontology, and connected through a knowledge graph, the reasoning process becomes much easier to follow. Rules and queries operate on well-defined structures, and their outcomes can be traced back to specific pieces of evidence. In our framework, this traceability is not only a technical feature but also a practical requirement: it supports meaningful interaction with instructors, who need to understand and sometimes challenge the system’s decisions. More generally, such representations support consistency, constraint checking and logical closure in a way that purely generative systems struggle to guarantee^26–28.

Rather than treating these paradigms as competing alternatives, recent work has started to bring them together. A common pattern is to use structured representations (ontologies, knowledge graphs or databases) to ground the generation process^29,30. In some systems, relevant graph fragments are retrieved and passed to the model; in others, natural language requests are translated into formal queries, or constraints are applied during generation to ensure validity^31–33. What these approaches share is a division of labor: symbolic components handle consistency and inference, while LLMs focus on expressing results in a form that is accessible to users.

Seen in this light, LLMs can be understood less as reasoning engines and more as a surface realization layer. The underlying logic (identifying a struggling learner, linking performance to outcomes, selecting an appropriate intervention) remains anchored in the knowledge graph and its associated rules. The language model then turns this structured output into feedback that is readable and context-sensitive. This separation is particularly relevant in educational settings, where both interpretability and communication matter. By keeping reasoning explicit while allowing flexibility in expression, hybrid approaches offer a way to combine reliability with usability.

For our work, this perspective also helps clarify the design choices made in the framework. The emphasis is placed on symbolic reasoning for detection and decision-making, with the possibility of integrating LLMs at a later stage to enhance how feedback is communicated. This keeps the core of the system transparent and inspectable, while leaving room for more natural interaction styles as these technologies continue to evolve.

While our current framework focuses on symbolic reasoning to ensure transparency and pedagogical alignment, this line of work highlights a natural extension: incorporating LLMs as a surface realization layer for feedback delivery. We therefore position our approach as a structured foundation that could support such hybrid architectures, rather than as a fully integrated neuro-symbolic system. In this context, our framework can be seen as providing the structured, interpretable backbone required for such hybrid approaches, where symbolic reasoning ensures correctness and traceability, and generative models may support flexible feedback presentation.

Proposed framework

Architecture overview

Figure 1 shows the overall shape of the framework. We organized it into several layers that pass information upwards and back again, so raw educational data can gradually turn into personalized, explainable feedback. The layers are linked closely: modifications to one component typically required corresponding adjustments in others. At the bottom is the Educational Ontology Layer, written in OWL. It is meant as a general schema designed to support reuse across contexts, not a model for a single course. We designed it with reuse in mind, with the intention that its abstraction level (e.g., Learner, LearningOutcome, AssessmentItem) supports transfer across educational contexts, although this has not yet been empirically validated. This layer defines the main concepts: Learner, AssessmentItem, LearningOutcome, Feedback and the relations that connect them. On top of this, the Knowledge Graph Layer turns those classes into concrete nodes and edges. Student activity and performance traces are mapped into the graph so that the course environment appears as a single integrated structure rather than relying on many separate tables and CSV files.

Fig. 1.

Architectural overview of the proposed ontology-driven knowledge graph framework.

The next level is the Semantic Reasoning Layer. Here we use SWRL rules and SPARQL queries as a deterministic reasoning engine grounded in explicit semantic representations. It searches the graph for patterns, identifies potential areas of difficulty and assembles candidate feedback messages. Rather than introducing new reasoning formalisms, this layer illustrates how standard semantic web reasoning can be applied in an operational setting for feedback generation when supported by a structured ontology and knowledge graph. Ensuring that this layer ran efficiently on our data required several iterations, especially around query design.

At the top, the Application Layer is responsible for what users actually see. We distinguish two views: a learner dashboard that focuses on individual progress and feedback, and an instructor console that aggregates information at cohort level. These interfaces are designed to expose both feedback and its underlying rationale, supporting inspection and potential adjustment by instructors. Both are intended to be used in a human-in-the-loop setting, where teacher judgements can prompt changes in the rules or even adjustments to the ontology when something seems off.

Educational ontology design

For our purposes, the ontology forms the base layer of the framework. It fixes the meaning of core concepts so that all later components (ETL scripts, graph mappings, rules) talk about the same things. We needed this layer to do more than list courses and resources. It also had to support explicit reasoning about performance and feedback, including fairly fine-grained signals.

There is already a long list of educational ontologies in the literature. Examples include Learning Object Metadata (LOM), the Curriculum Knowledge Graph Ontology, EduKG and the Simple Event Model (SEM). These cover resource metadata, curriculum structure and event tracking reasonably well. Direct application to our problem revealed several gaps that prevented their use without substantial extension. In particular, none of them gave us a clean way to represent performance metrics, different kinds of feedback or engagement-driven reasoning (see Appendix 1, Table 9).

Table 9.

Comparative analysis of representative educational ontologies and knowledge-graph models.

Ontology	Primary focus	Strengths	Gaps/limitations	Relevance to framework
Learning Object Metadata (LOM)	Standardized metadata for learning resources	Widely adopted; strong support for resource description (difficulty, format, intended user)	Does not model learner performance, assessment metrics, or adaptive feedback	Useful for aligning resource metadata, but insufficient for reasoning or feedback
Curriculum KG Ontology	Curriculum structures and outcome alignment	Represents courses, prerequisites, and learning outcomes	Limited modeling of performance tracking or feedback mechanisms	Good for curriculum mapping; requires extension for analytics and reasoning
EduKG	Large-scale graph of subjects and prerequisites	Rich concept and curriculum dependencies	Mostly data-oriented; limited modelling of learners or assessments	Helpful for concept alignment and enrichment, but not as a base ontology
Simple Event Model (SEM)	domain-agnostic event modelling intended for reuse	Captures actors, events, time and context; reusable across domains	Generic; lacks education-specific constructs	Useful for LMS log/temporal data, but needs coupling with educational ontologies
OULO	Pedagogical structure (courses, modules, outcomes, assessments)	Strong educational grounding; domain-agnostic design intended for reuse; OWL-based	Lacks explicit modelling of performance metrics, engagement and feedback	Best candidate for extension: covers core concepts, extendable for reasoning and feedback

We therefore chose the Open University Learning Ontology (OULO) as a starting point. OULO is widely used and has solid pedagogical grounding; it already models courses, modules, assessments and learning outcomes in some detail. On top of this base we added a small number of extensions that turned out to be necessary in practice:

• PerformanceMetric: This class holds detailed evidence about how a learner performs on a given assessment item. We record attributes such as score, attemptCount, timeOnTask, errorRate and dateRecorded. Each metric instance is linked to both a Learner and an AssessmentItem.

• Feedback: Feedback is treated as a first-class entity. We created subclasses such as HintFeedback, ResourceRecommendation, RemediationPlan and EncouragementFeedback. These instances connect to PerformanceMetric and LearningOutcome via properties like triggers, appliesTo, recommendedResource and deliveredThrough, which lets us point to the exact trigger and target of each message.

• EngagementMetric: Performance alone did not give us a good picture of learners who were quietly drifting away. We therefore added a class for engagement signals, with fields like motivationLevel, engagementScore, confidenceRating and engagementTimeStamp, linked to the Learner through an engagesIn relation.

These additions hold the reasoning layer together. They give rules and queries enough structure to pick up recurring learning patterns and to locate where understanding appears to break down. Importantly, they also enable multi-hop reasoning across the graph (e.g., linking EngagementMetric and PerformanceMetric through LearningOutcome to infer compounded risk patterns), rather than relying on isolated conditions. When we removed the learning-outcome links or the engagement metrics in ablation tests, detection performance dropped in a measurable way (Sect. 4.4; Appendix 1, Table 10). That experience convinced us that the extensions are not decorative; they are central to robust reasoning. For readers who want to inspect the design in detail, Appendix 2 includes a taxonomy diagram of the extended ontology (Fig. 5), a table that lists the new classes and properties (Table 11) and a short OWL/Turtle snippet (Listing 1) that shows how the model looks in code.

Table 10.

Precision, recall and F1 under ablation conditions (averaged across outcomes).

Condition	Precision	Recall	F1
Full system (baseline)	0.80	0.74	0.77
Learning outcome alignment	0.72	0.70	0.71
Engagement metrics	0.77	0.72	0.73
SHACL validation	0.79	0.74	0.76

Fig. 5.

Hierarchical taxonomy of OULO core classes and framework extensions.

Table 11.

Summary of ontology extensions.

Construct	Type	Description	Role in framework
PerformanceMetric	Class	Represents fine-grained learner performance on assessment items. Includes properties: score, attemptCount, timeOnTask, errorRate, dateRecorded. Linked to Learner and AssessmentItem	Enables precise evaluation of learner progress and supports rule-based reasoning about deficiencies and mastery
Feedback	Class	Represents adaptive responses triggered by learner performance. Subclasses: HintFeedback, ResourceRecommendation, RemediationPlan, EncouragementFeedback	Provides differentiated, explainable feedback pathways tailored to learner needs
Feedback → LearningOutcome (appliesTo)	ObjectProperty	Links feedback to the targeted learning outcome	Ensures feedback is pedagogically aligned
Feedback → FeedbackChannel (deliveredThrough)	ObjectProperty	Specifies the channel of delivery (dashboard, chatbot, email)	Supports role-specific and multi-modal feedback delivery
Feedback → Resource (recommendedResource)	ObjectProperty	Associates feedback with specific learning resources	Enables adaptive content recommendations
EngagementMetric	Class	Represents learner engagement indicators. Includes motivationLevel, engagementScore, confidenceRating, engagementTimeStamp, interactionType. These engagement indicators are implemented as engineered proxy variables derived from interaction logs (e.g., activity frequency, time-on-task) and are used as indicative signals rather than direct measurements of affective states	Provides context-aware reasoning that integrates cognitive and affective factors
Learner → EngagementMetric (engagesIn)	ObjectProperty	Links learners to their engagement data	Connects behavioral/affective indicators with performance-based reasoning

Listing 1.

Excerpt of the ontology serialisation in Turtle syntax, showing the formal definitions of the new classes and properties.

Knowledge graph design

The knowledge graph is where the ontology meets actual data. We instantiate learners, assessments, performance records and engagement metrics as RDF triples and store them in a single graph. In the current prototype, we load information from several sources:

• student records (demographics, prior study, identifiers) → Learner.

• assessments (quizzes, exams, assignments) → AssessmentItem.

• performance logs (scores, attempts, submission dates) → PerformanceMetric.

• learning resources and activities (videos, forums, readings) → Resource.

• engagement traces (click logs, time-on-task measures) → EngagementMetric.

Some entities, such as LearningOutcome and Feedback, are defined in the ontology but do not appear directly in raw data. They are linked in through rules and queries at runtime. Assessment-to-learning-outcome mappings were defined using an external mapping table, manually constructed by the authors based on course structure and assessment descriptions. These mappings serve as a modeling layer and were iteratively reviewed for consistency, although not independently validated by domain experts.

Engagement-related constructs (e.g., engagementScore, motivationLevel, confidenceRating) are implemented as engineered proxy variables derived from interaction logs (e.g., activity frequency, time-on-task), and are used as indicative signals rather than direct measurements of affective states. Although the mapping rules were authored manually, this approach made it straightforward to trace and correct issues observed in later queries. In addition, the graph structure enables the exploration of structural patterns (e.g., clusters of learners sharing similar performance–engagement profiles), which complements rule-based reasoning with graph-level insights.

Core reasoning and feedback generation

The reasoning engine is the active part of the framework. It combines inference rules and SPARQL queries so that the output is not a static report but feedback that someone can act on. We think of its work in two strands. The first strand is evidence discovery. Here, queries scan the graph to find learners who fit patterns we care about: repeated low scores on assessments linked to a particular outcome, signs of disengagement over several weeks, or missing key activities. These patterns often emerge through multi-step traversals of the graph, where evidence from different nodes (e.g., engagement, performance, and activity completion) is combined to form a more complete picture of learner state. The queries were revised multiple times as our understanding of the data improved; early versions were overly strict and produced very few alerts.

The second strand is feedback materialization. Once a pattern is detected, SWRL rules infer feedback-related classifications and relations, which are then used to materialize Feedback instances and attach them to the relevant PerformanceMetric and LearningOutcome nodes. This linkage means that when a message appears, we can always see which scores and which outcomes it depends on. To illustrate this process concretely, consider a simplified multi-hop reasoning chain implemented through SPARQL queries and rules:

• a PerformanceMetric instance associated with a Learner indicates low performance, for example below a defined threshold and, in some cases, across multiple attempts;

• the corresponding AssessmentItem is linked via the evaluates relation to a specific LearningOutcome;

• an EngagementMetric associated with the same Learner, through the engagesIn relation, indicates low or declining activity levels;

• the combined pattern triggers a remediation-oriented response rather than a simple HintFeedback.

This reasoning involves traversing multiple nodes and relations (Learner → PerformanceMetric → AssessmentItem → LearningOutcome, combined with Learner → EngagementMetric), and results in feedback that reflects both performance and engagement evidence rather than a single condition.

A simplified SWRL rule capturing this logic can be expressed as follows:

Learner(?l) ^ PerformanceMetric(?p) ^ belongsTo(?p,?l) ^ hasScore(?p,?s) ^ swrlb: lessThan(?s, 50) ^ forAssessment(?p,?a) ^ AssessmentItem(?a) ^ evaluates(?a,?lo) ^ EngagementMetric(?e) ^ engagesIn(?l,?e) ^ engagementScore(?e,?g) ^ swrlb: lessThan(?g, 0.4) -> RemediationNeededFor(?l,?lo).

A simple example: imagine a learner who has scored below a threshold three times on assessments aligned with CriticalThinking. The rules mark this learner as a StrugglingLearner for that outcome. In a more complex multi-hop case, this condition can be combined with low engagement signals (e.g., declining activity over time) to trigger a different type of feedback, such as a structured remediation plan rather than a simple hint. Additional rules then generate one or more feedback items: short hints on the next attempt, links to remedial resources, or a more structured remediation plan. In the graph, each message is linked to the specific scores and activities that triggered it, rather than remaining a generic “try harder” comment.

Application layer

The application layer is the point where all this machinery becomes visible to users. In our design there are two main entry points. The learner dashboard follows each student’s progress, displays tailored feedback as they work and suggests possible next steps or alternative routes through the material. The instructor console looks outward at the cohort, showing clusters of struggling learners, common error patterns and locations in the course where many students appear to stall.

For this study we did not build polished interfaces. We used simple outputs (JSON objects, simple HTML pages and views inside the triple store) because we wanted to focus on the backbone: the ontology, the graph and the reasoning pipeline. During meetings with instructors, these simple views were sufficient to walk through individual cases and discuss whether the system’s explanations made sense.

Advantages of the framework

Compared with a more traditional e-learning stack, the architecture brought a few practical benefits in our tests:

• Shared meaning across systems: Because all components rely on the same ontology, different data sources and tools refer to learners, tasks and outcomes consistently. We spent less time clarifying what a “quiz attempt” or a “module completion” meant.

• More genuinely personal feedback: Reasoning takes place over a rich graph, not just over isolated scores. This allowed feedback to reflect a learner’s past activity and engagement history, instead of sending a single generic message to everyone who falls below a threshold.

• Transparency by construction: Every recommendation can be traced back through the graph: from a feedback instance to the rules that fired and the data they used. In conversations with teachers, this trace often became the starting point for refining both rules and pedagogy.

• Room to grow: The graph structure made it straightforward to plug in extra data sources, define new learning outcomes or roll out additional feedback types. We also found that the same design can handle millions of triples in our experiments without requiring a major redesign, which gives some confidence about applying it at institutional level.

Experiments and results

In this section we explain how we set up and ran the experiments. We start with the environment and data preparation, then describe how we built the knowledge graph and evaluated the reasoning engine. The evaluation follows the three research questions (RQ1-RQ3): detection accuracy, perceived quality of feedback, and robustness and scalability. All experiments use the OULAD dataset. Learners are split by presentation to avoid temporal leakage: detection thresholds are tuned once on a development split and then kept fixed for testing. OULAD is a widely used benchmark in learning analytics that provides rich, large-scale traces of learner activity, enabling a realistic evaluation of the framework under complex and heterogeneous conditions. Extending the evaluation to additional educational contexts (e.g., K-12, MOOCs, corporate training) represents a natural direction for future work.

Definition of evaluation metrics

To ensure methodological clarity and reproducibility, we explicitly define the evaluation dimensions used in this study and the corresponding measurement procedures. These dimensions are aligned with the research questions (RQ1–RQ3) and provide a structured basis for assessing system performance.

• Detection Performance (RQ1): Detection performance is evaluated using standard classification metrics: precision, recall, and F1-score, computed on the task of identifying struggling learners. In this context, a “struggling learner” is operationally defined based on ground-truth labels derived from assessment outcomes and threshold-based criteria applied consistently across all compared methods (ontology-based, threshold baseline, and ML baseline). These metrics are defined as:

  • Precision = TP/(TP + FP).
  • Recall = TP/(TP + FN).
  • F1-score = 2 × (Precision × Recall)/(Precision + Recall).

where TP, FP, and FN denote true positives, false positives, and false negatives, respectively. These metrics are computed at the learner level and aggregated across learning outcomes, as reported in Table 3.

Table 3.

Detection performance (Precision, Recall, F1) for struggling learners.

Approach	Overall P	Overall R	Overall F1	Critical thinking F1	Algebraic reasoning F1	Data literacy F1
Ontology+Rules	0.80	0.74	0.77	0.78	0.76	0.77
Threshold baseline	0.68	0.78	0.72	0.70	0.71	0.74
ML baseline (LogReg)	0.76	0.74	0.75	0.74	0.73	0.75

• Explainability (RQ2): Explainability is assessed through expert-based evaluation. Three domain experts rated generated feedback using a 5-point Likert scale based on:

  • clarity of reasoning,
  • transparency of supporting evidence,
  • traceability of the decision process.

Scores are aggregated (mean) across experts and evaluated feedback instances. Inter-rater agreement is assessed using Krippendorff’s α (primary measure) and Cohen’s κ (pairwise agreement). Higher scores indicate greater perceived explainability.

• Personalization: Personalization is defined as the degree to which feedback is context-specific and adapted to the individual learner’s performance and context. It is evaluated through the same expert-based protocol as explainability, using ratings of:

  • relevance (alignment with learner performance),
  • actionability (usefulness of suggested interventions).

While assessed using the same evaluation procedure, personalization captures a distinct dimension related to the appropriateness and usefulness of feedback, rather than its interpretability. Operationally, personalization reflects the system’s ability to generate differentiated feedback instances across learners with distinct performance–engagement profiles, as observed in the variation of feedback types (e.g., HintFeedback, RemediationPlan) produced by the reasoning engine. Although primarily evaluated through expert judgment, personalization is grounded in observable variation in feedback content across different learner states produced by the reasoning process. Higher scores indicate stronger personalization. These ratings correspond to the “Relevance” and “Actionability” dimensions reported in Table 4.

Table 4.

Expert evaluation of feedback usefulness and explainability.

Criterion	Ontology+Rules (mean ± σ)	%≥4	Threshold baseline (mean ± σ)	%≥4
Relevance	4.3 ± 0.6	82%	3.6 ± 0.8	58%
Actionability	4.1 ± 0.7	76%	3.4 ± 0.9	52%
Explainability	4.5 ± 0.5	88%	3.2 ± 0.9	44%
Inter-rater agreement	κ = 0.69; α = 0.71	-	κ = 0.69; α = 0.71	-

• Engagement: Engagement is modeled using engineered proxy variables derived from learner interaction data, including activity frequency, time-on-task, and interaction patterns. These variables are aggregated into an engagement score used within the reasoning process. Formally, engagement is treated as a latent construct approximated through observable behavioral indicators and incorporated as an input feature in the knowledge graph via the EngagementMetric class. Engagement is not evaluated as a standalone outcome metric in this study. Instead, it is treated as an input variable that contributes to learner modeling, detection performance (RQ1), and feedback generation (RQ2).

• Scalability: Scalability is evaluated through system-level performance metrics, including:

  • knowledge graph size (number of RDF triples),
  • ETL processing time,
  • SPARQL query latency,
  • feedback generation time per learner cohort.

These metrics are measured empirically during experimentation (Sect. 4.5) and assess the framework’s ability to operate efficiently under increasing data volume and complexity. All reported values (Table 6) correspond to averaged measurements over multiple runs, ensuring robustness and reproducibility of system-level performance.

Table 6.

Efficiency and scalability metrics of the ontology-driven framework on the OULAD dataset (averaged over 20 runs).

Metric	Value
Knowledge graph size	4.2 M triples
ETL pipeline runtime (CSV→RDF)	~ 3 min (for 30k learners)
Median SPARQL query latency (Q1: single learner profile)	180 ms
Median SPARQL query latency (Q2: struggling learners by LO)	240 ms
Median SPARQL query latency (Q3: cohort-level misconceptions)	280 ms
Feedback materialization time	2.8 s/1,000 learners

Finally, we distinguish between quantitative performance metrics (e.g., detection performance, scalability), and qualitative evaluation dimensions (e.g., explainability, personalization). This distinction reflects the hybrid nature of the proposed framework, which combines predictive modeling, semantic reasoning, and expert-based evaluation of feedback quality. This dual evaluation strategy ensures alignment between algorithmic performance (RQ1), human-centered assessment of feedback quality (RQ2), and system-level operational viability (RQ3).

Experimental setup

All runs were carried out on a single workstation: Intel Core i7-11700 K (3.6 GHz, 8 cores), 32 GB RAM and a 1 TB SSD, running Ubuntu 22.04 LTS. The ontology was authored in Protégé and stored in a GraphDB triple store. ETL scripts and the reasoning pipeline were written in Python, using rdflib and related semantic web tooling to create RDF triples and issue SPARQL queries (SPARQLWrapper, pySHACL). We focused on two of the larger OULAD modules (DDD and FFF), which together cover about 30,000 students, roughly 10 million VLE interaction records and detailed assessment outcomes. The main mappings were:

• studentInfo.csv → Learner.

• assessments.csv → AssessmentItem.

• studentAssessment.csv → PerformanceMetric.

• vle.csv → Resource.

• studentVle.csv → EngagementMetric.

OULAD does not define learning outcomes or feedback explicitly. These come from the ontology and are connected during reasoning. Aligning CSV fields with ontology properties required several iterations, and early runs failed due to issues such as swapped column names. Assessment-to-learning-outcome mappings were defined externally (see Sect. 3.3) and kept fixed during experimentation to ensure consistency. To reduce instability, we fixed the ontology mappings, decision thresholds and query templates before the final test runs, and then kept them unchanged across the reported experiments.

RQ1: Detection accuracy

RQ1 asks: How accurately does the ontology-driven reasoning engine identify struggling learners compared with simpler baselines? We compared three approaches:

• Ontology+Rules (ours): Semantic reasoning over the extended OULO ontology using SWRL rules and SPARQL queries.

• Threshold baseline: A single rule (score < 50%), no learning-outcome alignment, no engagement features.

• Simple ML baseline: Logistic regression on a small set of tabular features: prior average score, most recent score and 7-day engagement activity. This reflects the kind of model that is easy to plug into existing dashboards.

To illustrate how the ontology enters the picture, we use SPARQL query Q1 (Fig. 2). Q1 returns learners who scored below 50% on assessments aligned with the CriticalThinking outcome. Because the alignment is explicit in the ontology, the query does not just look at raw scores; it flags learners in relation to a specific intended outcome.

Fig. 2.

SPARQL query Q1 for identifying struggling learners in Critical Thinking.

A small excerpt is shown in Table 2. For example, learner kg: learner/12,345 scored 42% on kg: assessment/A1, and this assessment evaluates kg: lo/CriticalThinking. The reasoning engine follows a simple chain:

Table 2.

Excerpt of SPARQL Q1 results for learners under the mastery threshold in Critical Thinking.

Learner	Assessment	Score	Learning outcome
kg: learner/12,345	kg: assessment/A1	42.0	kg: lo/CriticalThinking
kg: learner/67,890	kg: assessment/A3	37.5	kg: lo/CriticalThinking

• PerformanceMetric: kg: learner/12,345 → hasScore 42%, linked to kg: assessment/A1 via the forAssessment relation.

• AssessmentItem: kg: assessment/A1 → evaluates → kg: lo/CriticalThinking.

• Rule condition: hasScore < 50% fires the StrugglingLearner rule.

• Inference: the learner is marked as struggling on Critical Thinking.

This traceable chain reflects the design goal of making each inference step explicit and inspectable, rather than relying on implicit model behavior. This simple case is useful as a starting point, but the framework also supports more complex multi-hop reasoning. For instance, the same learner state can be refined by combining low performance on outcome-linked assessments with low engagement traces and missing key activities, yielding a different feedback path from the one triggered by score alone. In those cases, the reasoning process performs multi-hop traversal across interconnected nodes, combining evidence from PerformanceMetric, AssessmentItem, LearningOutcome, EngagementMetric and Resource usage, to construct a more context-sensitive interpretation of learner state before generating feedback.

We evaluated each approach using precision, recall and F1-score across all assessments and three outcomes: Critical Thinking, Algebraic Reasoning and Data Literacy. Results are shown in Table 3. Our ontology-based framework reached an overall F1 of 0.77, showing higher observed performance than both the threshold rule (0.72) and logistic regression (0.75). The biggest gap is in precision; explicitly tying assessments to learning outcomes appears to help reduce false alarms.

The threshold rule behaved as expected: it caught more struggling learners but misclassified many who were not really at risk. Logistic regression gave a more balanced trade-off between recall and precision, but it had no semantic view of what each task was meant to assess. We repeated the experiments with different random splits; the rankings stayed the same, suggesting stable relative ordering across sampled partitions. We did not perform formal statistical significance testing and therefore do not claim statistical superiority.

RQ2: Feedback usefulness and explainability

RQ2 asks: How do human experts judge the feedback produced by the system? We were particularly interested in three aspects: relevance, actionability and explainability. We sampled 120 feedback instances (40 for each outcome) and asked three university lecturers with assessment experience to rate them on a 5-point Likert scale. The experts independently evaluated the feedback using a predefined Likert-scale rubric assessing relevance, actionability, and explainability. Prior to participation, the experts were informed about the purpose of the study, the evaluation procedure, and the use of their anonymized evaluations for research and publication purposes. Verbal informed consent was obtained from all participating experts. The identities of the experts were not recorded or disclosed, and no personal or sensitive data were collected.

Cohen’s κ and Krippendorff’s α indicated substantial agreement (κ ≈ 0.69, α ≈ 0.71). Getting three busy lecturers to sit down with 120 feedback messages was challenging; in the end we split the work into shorter sessions to avoid fatigue. The threshold baseline could only offer generic messages such as “Your score is below 50%.” By contrast, our ontology-driven framework generated feedback with explicit provenance: each recommendation pointed back to the triggering PerformanceMetric, the aligned LearningOutcome and, indirectly, to the SPARQL query that surfaced the relevant evidence. Table 4 summarizes the scores.

To make the notion of “explainable feedback” less abstract, Table 5 follows two messages from learner and outcome through to the evidence used by the rules.

Table 5.

Example explainability trace for feedback generation.

Learner	PerformanceMetric (Score)	LearningOutcome	Triggered feedback type	Provenance evidence
kg: learner/12,345	42.0 (Assessment A1)	Critical Thinking	RemediationPlan (“Step 1, review concept map on logical fallacies; Step 2, attempt guided exercises A2–A4”)	SWRL low-score rule + SPARQL Q1 result
kg: learner/67,890	37.5 (Assessment A3)	Critical Thinking	ResourceRecommendation (“Watch supplemental video: Logical Reasoning, Week 4”)	SWRL low-score rule + SPARQL Q1 result + low engagement metric

During the review sessions, experts reported that access to the underlying chain of evidence made it easier to judge when to trust the system and when to override it. This suggests that the provenance information was both visible and practically useful. It also indicates that the value of the framework lies not only in prediction quality, but in making the path from evidence to recommendation inspectable in ways that instructors can work with.

RQ3: Robustness and scalability

The third question turns to robustness and scale. We wanted to know: What happens when we remove specific ontology components that were added in Sect. 3.2? Can the framework cope with institution-level data without slowing to a crawl? To answer the first part, we carried out an ablation study. Figure 3 shows the F1 change relative to the full system. Removing learning-outcome alignment had the largest effect. Overall F1 dropped from 0.77 to about 0.71 (ΔF1 ≈ − 0.06), with precision falling from 0.80 to 0.72 and recall from 0.74 to 0.70. Once assessments are no longer tied to explicit outcomes, the reasoning is clearly less sharp. Dropping engagement metrics also reduced performance, though not as dramatically (ΔF1 ≈ − 0.04). Precision decreased by about 0.03 and recall by 0.02. In practice, these engagement signals appear to help catch learners who were drifting away before their scores collapsed. Disabling SHACL validation had only a small impact on F1 (around − 0.01) but produced more inconsistent triples in the store, which made later debugging less pleasant. Detailed precision and recall numbers are given in Appendix 1 (Table 10).

Fig. 3.

Ablation study showing F1-score variation relative to the full system under component removal (learning outcome alignment, engagement metrics, and SHACL validation).

These results suggest that the framework’s contribution lies not only in system integration, but also in how the pipeline is structured, from data representation to reasoning and feedback generation. This structure supports transparency, inspectability, and operation at scale in the present evaluation.

To complement the ablation results, we also inspected structural patterns in the graph at cohort level. Although this was not treated as a standalone graph-analytics experiment, the graph made it easier to observe recurring combinations of low outcome mastery, reduced engagement and missing resource interaction across groups of learners. These recurring structures informed several of the rule refinements used in the final system and illustrate how graph organization contributes to interpretation beyond what flat feature tables provide.

For scalability, we looked at the OULAD-based graph summarized in Table 6. The instantiated graph contains about 4.2 million RDF triples, which we consider realistic for a medium-sized institution. The ETL pipeline processed data for 30,000 learners in just under three minutes. SPARQL queries stayed under one second even for more complex cohort-level questions (Q1: 180 ms; Q2: 240 ms; Q3: 280 ms, each averaged over 20 runs). Feedback materialization took roughly 2.8 s per 1,000 learners, so generating recommendations for a full cohort is a matter of minutes rather than hours. We did not attempt to aggressively optimize the system for minimal latency. The aim was simply to see whether the design is usable without special hardware.

Our overall impression from RQ3 is that the framework behaves as intended. When we remove ontology elements that were added for performance and engagement, accuracy drops in a way that matches their role in the design. At the same time, the system handles millions of triples on mid-range hardware without excessive latency. These results support the practical viability of the proposed pipeline, while broader validation across additional educational contexts remains an important direction for future work.

Comparative context with literature

Our work sits between ontology-based representations, KG analytics and research on feedback and engagement. To keep things readable, we group earlier work into four areas: (1) technical foundations, (2) feedback systems, (3) adoption and digital readiness, and (4) system-level benchmarks. This structure also made it easier for us to see where our own design follows existing lines and where it departs from them. In doing so, we aim to position our contribution not only in terms of performance, but also in terms of how reasoning, data integration, and feedback generation are combined within a single, operational pipeline. In particular, our analysis emphasizes how existing systems often optimise individual components (e.g., prediction, data integration, or feedback generation) in isolation, whereas our framework explicitly connects these components into a unified, interpretable workflow.

Ontologies, KGs and learner modelling

One of the most comprehensive KG-building efforts is KnowEdu¹⁸. It mines curriculum standards, textbooks and assessment data from 4,488 middle-school learners. For concept extraction, CRF, LSTM and GRU models reach mid-0.70s F1 with 50% training data (0.75–0.77). Prerequisite relations are discovered with association-rule mining (AUC ≈ 0.95, MAP ≈ 0.87) and verified by experts (κ = 0.896). KnowEdu operates at an impressive scale, but the published description provides limited information on runtime behavior or user-facing explanations. These gaps show up quickly when trying to apply similar systems in live courses. More broadly, these systems illustrate a common pattern in the literature: strong performance in isolated tasks (e.g., extraction or prediction), but limited support for connecting these outputs to explainable, pedagogically grounded feedback processes.

Learner-modelling work follows a different path. The OPKT model⁴ embeds knowledge and student ontologies inside an LSTM. On ASSIST09 and ASSIST12 it reports strong scores (AUC 0.848/0.897; ACC 0.775/0.827), beating older KT approaches like BKT, DKT and SAINT+. OPKT does include interpretability checks through ablations and embedding analysis, but those explanations mostly target researchers, not learners or instructors.

Earlier semantic web research, such as⁸, already pointed toward integrating ontologies and linked data to support more personalized and semantically grounded learning experiences. However, these approaches often stopped short of operationalizing reasoning for real-time feedback generation. More recent work, including¹⁴, demonstrates how OWL and SWRL can support structured, explainable reasoning over complex domains, reinforcing the feasibility of rule-based semantic inference as a backbone for decision-making systems.

Our own experience with ontology-augmented models pushed us toward a different approach. Instead of hiding ontology signals inside a neural model and explaining them afterwards, we chose to make each inference and each feedback message traceable directly through ontology relationships. This design explicitly shifts the focus from model-level interpretability toward end-to-end reasoning transparency, where intermediate inference steps are not hidden but directly accessible and linked to feedback generation. In practice, this made debugging much easier: when a learner was misclassified, we could usually follow the trail to a rule or a missing alignment. At the same time, this approach depends on manually specified semantic alignments (e.g., between assessments and learning outcomes), which serve to structure the reasoning process rather than to provide ground-truth annotation.

Automated and human-centered feedback

A systematic review by Cavalcanti et al.³ analyzed 63 feedback systems published between 2009 and 2018. Around 65% reported positive effects on performance. About half claimed to reduce instructor workload. Even so, many relied on brittle answer-matching or dashboards that offered little insight into how feedback was produced. We noticed a recurring issue across these systems: provenance was rarely addressed. Without it, the feedback rules may work, but you cannot see why a particular message fired. This limitation highlights the need for feedback systems where the generation process itself is transparent and inspectable, rather than inferred post hoc.

Human-centered studies underline the importance of that missing layer. Dawson et al.³⁴ surveyed 4,514 students and 406 staff. Both groups agreed that feedback should help students improve, but they disagreed on what “help” means. Staff focused on design and timing; students valued comments that were actionable (49%), detailed (20%) and personal. The study aligns with our own observations; in our pilot, several students reported dissatisfaction when feedback was perceived as too generic.

Engagement and affect research adds another dimension. Studies such as Wang et al.³⁵ show that motivational cues can keep learners engaged, but these cues are usually delivered using narrow heuristics or domain-specific rules. In our system, the cues are linked to explicit constructs such as mastery states and repeated low scores. These constructs are implemented as engineered proxies derived from interaction logs rather than direct measurements of affective states, and are not evaluated in isolation but combined through multi-hop reasoning across performance, engagement, and activity data, enabling more context-sensitive interventions. This integration helped reduce spurious alerts and made explanations easier to communicate to instructors.

Adoption and digital readiness

Technical accuracy alone does not guarantee that a system will actually be used. This point recurs in adoption studies. Yaseen et al.³⁶, for example, surveyed 500 university students and found that personalized feedback tools increased engagement only for students who felt digitally confident. For others, the tools sometimes had the opposite effect. The authors argue that digital literacy moderates how learners respond to AI systems.

Systematic reviews also flag persistent reproducibility issues³. Many feedback tools cannot be audited or replicated easily. When our own early prototypes became difficult to debug, we realized how important transparent pipelines were. This is why we ended up reporting runtime behavior, query latencies and ETL details more thoroughly than we originally planned. This reflects a broader design choice in our framework to prioritise transparency and reproducibility alongside performance, enabling both technical validation and practical deployment.

Finally, earlier work shows that students and staff value different things³⁴. This gap matters for adoption. You may build a model that scores well on benchmarks, yet neither group feels comfortable using it. The provenance layer in our system, showing how a message was formed, was partly motivated by these observations. To keep these adoption factors in view, we drew on several survey and review studies while designing the framework, and Table 7 pulls their main findings together.

Table 7.

Adoption, engagement, and perception evidence in learning analytics and educational technologies.

Domain/focus	Study	Dataset/context	Reported results	Relevance to our work
Feedback perspectives	34	4,514 students; 406 staff (Australia)	90% students/89% staff: feedback = improvement-oriented; divergence on effectiveness (students prioritize actionable comments, staff emphasize design)	Highlights need for feedback that is both usable and well-timed; our ontology models actionable, next-task-linked feedback
Automatic feedback (SLR)	3	63 studies (2009–2018)	~ 65% positive impact on student performance; limited evidence for instructor support; weak reproducibility	Our framework addresses gaps with traceable, explainable feedback and reproducible reporting
VR adoption & usability	37	1,320 professors (Latin America)	VR rated highly for didactic potential; low digital competence and discipline/gender gaps constrain adoption	Reinforces role of digital readiness; our system emphasizes usability + transparency
Digital literacy & engagement	36	500 university students (Jordan)	SEM β coefficients (p < 0.01): ALT = 0.484; PF = 0.367; DL = 0.428; AI tools β = − 0.466 (negative effect). Digital literacy moderates engagement effects.	Confirms that digital literacy strongly shapes engagement; our framework uses explainable feedback design to reduce the opacity often associated with AI-supported interventions

System benchmarks and comparative view

To situate our work among earlier systems, Table 8 brings together comparable performance metrics (F1, AUC, MAP, ACC). The picture is simple: KnowEdu excels at concept extraction and prerequisite detection; OPKT leads on learner prediction. Both focus heavily on accuracy. Our framework reaches similar accuracy (F1 = 0.77) but reports more about how the system behaves: runtime, scalability and provenance. These additional dimensions become particularly important when moving from controlled experiments to real course deployment.

Table 8.

System-level benchmarks with comparable ML metrics, situating our framework alongside prior baselines.

Domain	Study	Dataset/context	Reported results	Relevance to our work
KG construction	18	4,488 learners, 1.1 M records	F1 ≈ 0.75 (concept extraction, GRU/LSTM/CRF); AUC ≈ 0.95, MAP ≈ 0.87 (prereq. relations)	Demonstrates feasibility of accurate educational KG; we extend to semantic reasoning, feedback generation, and reproducibility
Learner prediction	4	ASSIST09/ASSIST12	AUC = 0.848, ACC = 0.775 (ASSIST09); AUC = 0.897, ACC = 0.827 (ASSIST12)	Ontology-augmented KT improves predictive accuracy; our framework adds semantic provenance, feedback generation, and transparent reproducibility
Integrated framework	Our framework	Multi-source ETL, ontology reasoning, reproducible academic workstation	F1 = 0.77 (struggling-learner detection); feedback evaluation reported in Table 4; end-to-end runtime and SPARQL latencies fully reported; a balanced evaluation across quantitative (accuracy, scalability) and qualitative (personalization, engagement, explainability) dimensions	Uniquely combines semantic reasoning, provenance-based explainability, affect-aware personalization, and reproducible performance reporting

Rather than introducing a new predictive model, the contribution of our framework lies in how these components are combined into a coherent and interpretable system that connects data representation, reasoning, and feedback generation. This distinction is important when comparing with systems that optimize accuracy in isolation but provide limited support for inspection or intervention. In particular, the framework operationalises the full assessment–feedback loop, ensuring that predictive signals are not treated as final outputs but are transformed into structured, explainable, and pedagogically aligned feedback.

A broader comparison is shown in Fig. 4. The values shown are normalized scores on a 1–5 scale derived from reported metrics and qualitative assessments in the literature, intended to provide a comparative, high-level view rather than a direct benchmark. The radar plot places the systems along five normalized axes: accuracy, personalization, scalability, engagement and explainability. These dimensions reflect those that recent adaptive learning tools also target when aiming to improve performance and satisfaction³⁸. KnowEdu and OPKT show strong spikes on accuracy but remain flatter on the other dimensions. Our framework shows a more balanced profile, particularly on transparency and engagement. This more balanced profile reflects the design choice of distributing reasoning across ontology, graph, and rule layers, rather than concentrating it within a single predictive component. These comparisons should be interpreted as indicative, as the systems are evaluated under different datasets and experimental settings.

Fig. 4.

Comparative performance of KnowEdu, OPKT, and our framework across five dimensions.

Discussion

This section brings the results together and considers what they mean for the field. We compare our system with earlier approaches, look at implications for practice, and point out where the current work still falls short. Several of these reflections came from issues we ran into while building and testing the prototype.

Position in the field

Earlier systems have shown that both large educational knowledge graphs and ontology-guided learner modelling are possible at scale. KnowEdu reached concept-extraction scores around 0.75 and AUC ≈ 0.95 for prerequisite detection. OPKT reported strong prediction performance, with AUC close to 0.90 across two standard datasets. What these systems do less well is explain how their predictions should be used or how the underlying decisions are made. KnowEdu is excellent at discovering concepts and relations, yet it does not describe runtime behavior or claim that its output can directly support feedback. OPKT provides some interpretability through embeddings and ablations, but those are aimed at researchers rather than instructors or students trying to make sense of a recommendation. When trying to compare our results to theirs, we sometimes found ourselves guessing about details they did not publish.

More broadly, earlier semantic web approaches highlighted the potential of combining ontologies and linked data to support personalized learning⁸, while more recent work demonstrated how rule-based semantic reasoning over OWL ontologies can enable interpretable, context-aware decision-making¹⁴. Our framework builds on these foundations but extends them toward an operational setting where reasoning directly supports feedback generation. Our framework moves the conversation forward in four areas:

1. Robust metrics: We report F1 = 0.77, but we also look at variation across runs. During development, this surfaced small inconsistencies we could fix early, something one would miss if reporting only “best” values.

2. Explanations built into the system: Each inference connects directly to ontology entries, SWRL rules and RDF triples. When a classification looked off, we could follow the chain and spot the cause. This explicit reasoning trace differs from post-hoc interpretability approaches by making the full decision process inspectable at runtime.

3. Scalability tested on everyday hardware: We ran all experiments on a mid-range workstation. It turned out to be a good decision: latency remained low even with millions of triples in our experiments.

4. Broader support for learners and staff: The ontology encodes personalization, feedback types and engagement cues directly, which helps respond to mismatches between what students want from feedback and what instructors think they provide³⁹.

Importantly, the reasoning process can combine multiple signals across the graph (performance, engagement, and activity traces), supporting more context-sensitive feedback. Figure 4 makes this contrast easy to see. KnowEdu and OPKT achieve strong accuracy but are less explicitly evaluated on scalability, engagement and explainability. Our system shows a more balanced profile across these dimensions, although such comparisons remain indicative given differences in datasets and evaluation settings.

Implications for e-learning research and practice

Four themes stood out during implementation, review sessions with instructors, and repeated debugging cycles:

1. Metrics that reflect actual system behavior: Reporting a single F1 value proved insufficient, as performance varied across runs. Including robustness checks appears important if learning-analytics tools are going to be trusted.

2. Explanations that people can read: Because reasoning is anchored in the ontology, the path from data to decision is easy to trace. In one early walkthrough, an instructor spotted a rule they disagreed with, not because the model was “wrong,” but because it clashed with how they teach that topic. That sort of feedback is only possible when the logic is inspectable.

3. Scalability demonstrated, not assumed: It was initially unclear whether the SPARQL queries would scale to a full institutional dataset. After tuning, however, the system handled approximately four million triples without performance issues in our evaluation setup.

4. Addressing differences in what users value: Students tend to want actionable, personalized feedback; staff often emphasize alignment and timing⁴⁰. Encoding feedback types and engagement signals directly into the ontology helped us meet both perspectives instead of prioritizing one over the other.

Collectively, these observations suggest that effective AI-supported feedback systems require not only accurate prediction, but also structured representations that make reasoning processes transparent and actionable in practice. Figure 4 provides a summary view of this balance. Earlier systems peak on accuracy. Ours distributes its strengths across accuracy, transparency, scalability and engagement.

Limitations and future work

Several limitations shape the direction of the next steps:

1. Evaluation scope and transferability. All experiments were conducted on OULAD, a widely used higher-education dataset that provides rich and realistic learner traces. While this setting allows for a robust evaluation under complex conditions, extending the framework to other contexts such as MOOCs, K-12 education, or workplace training represents an important direction for future work. In particular, such extensions would allow us to examine how well the ontology structure and reasoning rules generalize across different pedagogical settings.

2. Operationalization of learning outcomes and engagement constructs. Assessment-to-learning-outcome mappings were defined using an external mapping table, manually constructed by the authors based on course structure and assessment descriptions. These mappings were used to structure the reasoning process, were reviewed iteratively for consistency, but were not independently validated by domain experts. In addition, engagement-related constructs (e.g., engagementScore, motivationLevel, confidenceRating) are engineered proxy variables derived from interaction logs (e.g., activity frequency, time-on-task), and should be interpreted as indicative signals rather than direct measurements of affective states.

3. Limited feedback repertoire: Although the system supports hints, recommendations, remediation plans and encouragement, the underlying library is small. During early user sessions, instructors often asked for richer feedback types: peer review prompts, reflective questions, multi-step scaffolds, and media-based resources. Broadening this space is a clear priority.

4. No direct classroom trials yet: So far, evaluation relies on expert judgement and technical metrics. We have not yet placed the system in front of real learners. This was intentional: we wanted to establish that the reasoning pipeline works first. However, this still leaves open how students might react to the feedback, whether they trust it, and whether it influences behavior. Future work should include controlled pilots in live courses and longer-term studies that track achievement, engagement and trust.

5. Two promising development paths:

   • Adaptive sequencing. Extending the reasoning engine to recommend not just feedback but ordered learning paths: tasks, resources, and pace adjustments.
   • Instructor-in-the-loop tools. Several instructors asked for the ability to adjust rule weights or override alignments directly. Building these controls into the system would strengthen both agency and adoption.

Beyond these directions, recent advances suggest a further extension: integrating symbolic reasoning with LLMs, where the knowledge graph and rule engine provide structured, verifiable reasoning, and LLMs serve as a surface realization layer for generating natural language feedback. Such hybrid approaches could enhance expressiveness while preserving the transparency of the underlying reasoning process.

Three design principles emerge as particularly important for future work:

• Semantic alignment helps interpret scores and activities in terms of intended outcomes.

• Traceable reasoning turns explanations into concrete, checkable chains rather than decorative add-ons.

• Pedagogical grounding keeps interventions aligned with formative assessment theory rather than drifting toward convenient heuristics⁴¹.

These principles capture what we think this study contributes: a framework that aims to make predictive decisions more interpretable, inspectable and pedagogically grounded in the setting examined here.

Conclusion and future work

In this study, we presented a framework that uses an ontology-driven knowledge graph to support assessment and feedback in e-learning. Building on the Open University Learning Ontology, we introduced extensions for performance, engagement, and feedback, and instantiated this model using real learner activity data. On top of this structure, we implemented a reasoning layer based on SWRL rules and SPARQL queries, enabling feedback that is both adapted to the learner and traceable to explicit semantic relationships. The aim was to move beyond simple thresholds and opaque machine-learning models toward approaches that are more transparent and interpretable in practice.

The contribution of this work lies not in proposing new reasoning algorithms, but in showing how established semantic technologies can be applied in an operational setting to support explainable, pedagogically grounded feedback. By linking learner data, semantic representations, reasoning processes, and feedback generation within a single system, the framework enables decisions that remain open to inspection and interpretation.

The experimental results highlight three main aspects. First, the ontology-driven reasoning approach achieved higher observed performance than both a threshold baseline and a simple logistic regression model for identifying struggling learners, with gains primarily in precision. Second, expert evaluation indicates that the generated feedback is more relevant, actionable, and easier to interpret than baseline messages, particularly due to the availability of explicit reasoning traces. Third, scalability tests show that the framework can handle realistic data volumes, with query times remaining low even for graphs containing several million triples.

In the context of the OULAD dataset, these findings suggest that ontology-based reasoning can be a viable option in settings where transparency and interpretability are required. The results also indicate that structured semantic representations can support not only data integration but also the generation of feedback grounded in pedagogical concepts. At the same time, the approach depends on explicit modeling choices, including manually defined mappings between assessments and learning outcomes and the use of proxy engagement variables derived from interaction data.

Several directions for future work follow from these limitations. Extending the evaluation to other educational contexts, such as MOOCs, school-level education, or workplace training, would help assess the transferability of the ontology and reasoning design. Additional developments include expanding the range of feedback types, supporting adaptive sequencing of learning activities, and providing instructors with tools to adjust rules and ontology elements directly. From a technical perspective, future work may explore temporal reasoning over learning trajectories, uncertainty-aware representations, and graph-based analysis of cohort-level patterns.

A further extension is the integration of symbolic reasoning with LLMs, where the knowledge graph and rule-based system provide a structured and verifiable reasoning backbone, and LLMs support the generation of natural language feedback. Such hybrid approaches could improve the expressiveness of feedback while preserving transparency in the underlying decision process.

Overall, this work shows how ontology-driven knowledge graphs and rule-based reasoning can be applied to support explainable feedback in e-learning systems, while highlighting the importance of transparency, interpretability, and alignment with pedagogical principles in the design of learning analytics tools.

Appendices

Appendix 1: Ontology comparison and ablation results

To justify our design choices, we compared several representative educational ontologies and knowledge graph models. Table 9 summarizes their focus, strengths and limitations, and shows why we treated the Open University Learning Ontology (OULO) as the most suitable base to extend. This comparative analysis was conducted to ensure that the proposed framework builds on established semantic models while explicitly addressing gaps related to performance modeling, reasoning, and feedback generation.

In our own review, we found that LOM, curriculum ontologies and EduKG were excellent at structuring resources and content, but had little to say about performance or feedback. OULO, by contrast, already spoke the language of courses, outcomes and assessments, so extending it felt more natural than reinventing the wheel.

More importantly, OULO provides a semantically coherent backbone that supports alignment between assessment items and learning outcomes, which is a prerequisite for explainable reasoning in educational settings. In our framework, this alignment is operationalized through manually defined mappings between assessment items and learning outcomes, treated as part of the modeling layer rather than as ground-truth annotations.

We also ran an ablation study to see how key ontology components affect detection performance. Table 10 reports precision, recall and F1 averaged across outcomes. This analysis was designed to isolate the contribution of each semantic component to the overall reasoning performance, thereby providing empirical support for the architectural design choices described in Sect. 3.

Removing learning-outcome alignment had the strongest effect, dropping F1 from 0.77 to about 0.71. This matches our intuition from development: once assessments are no longer tied to explicit outcomes, the reasoning becomes noticeably less precise. This result also reinforces the importance of semantic alignment as a core mechanism for multi-hop reasoning, where performance signals must be interpreted in relation to intended learning objectives rather than in isolation.

Engagement metrics also contributed, especially for catching learners who were beginning to drift away rather than already failing. In practice, these signals enable the system to combine behavioral and performance indicators, supporting richer inference patterns than those based on scores alone. SHACL validation mainly improved data consistency. Its impact on F1 was small but it saved us time when debugging broken mappings. While its quantitative effect is limited, SHACL plays an important role in ensuring structural validity of the graph, which is critical for maintaining reliable reasoning at scale.

Appendix 2: Ontology extensions

This appendix looks more closely at the extensions we added to OULO. On its own, OULO does a good job of modelling core pedagogical elements (courses, modules, learning outcomes and assessments) but it does not cover what we needed for performance-based reasoning, adaptive feedback or detailed engagement signals. To close those gaps, we introduced a compact set of new classes and properties. These extensions were designed specifically to support rule-based inference and multi-hop reasoning across learner data, rather than to expand the ontology for its own sake.

We kept the extensions deliberately small. During modelling, we found that adding too many concepts too quickly made debugging harder, so we focused on pieces that directly support reasoning: performance metrics, feedback types and engagement indicators. In practice, this restraint made it easier to trace odd inferences back to a specific construct. This design choice reflects a trade-off between expressiveness and operational tractability, ensuring that the ontology remains both interpretable and computationally manageable.

Figure 5 presents the extended ontology as a simple taxonomy, with OULO’s original classes separated from the new ones. It also shows where evaluation links occur (e.g. from AssessmentItem to LearningOutcome, and from PerformanceMetric to AssessmentItem), which helped us keep the OWL serialisation internally consistent.

These links also define the core reasoning paths used by the inference layer, enabling traceable propagation from observed performance to outcome-level interpretation and feedback generation.

Table 11 summarizes the new elements and their roles.

Below is an excerpt of the OWL/Turtle serialisation:

Across Fig. 5; Table 11; Listing 1, the main message is that OULO has been extended in a small but targeted way. The new constructs:

• support fine-grained performance tracking (which appeared to help reduce false positives),

• enable feedback that stays tied to learning outcomes and concrete evidence, and

• bring engagement signals into the reasoning loop, which proved useful for earlier intervention.

These extensions operationalize the link between data representation and reasoning, allowing the system to move from isolated learner traces toward structured, explainable feedback, through explicit semantic relationships. For educational researchers, this offers a clear conceptual structure. For semantic-web practitioners, it provides a concrete OWL implementation they can adopt or adapt.

Author contributions

N.S. conceived the study, developed the methodology, conducted the experiments, and wrote the manuscript. W.J. contributed to the study design, interpretation of results, and manuscript revision. All authors reviewed and approved the final manuscript.

Funding

This work was supported by the Deanship of Scientific Research, Vice Presidency for Graduate Studies and Scientific Research, King Faisal University, Saudi Arabia (Project No. KFU253525).

Data availability

The primary dataset used in this study is the Open University Learning Analytics Dataset (OULAD), which is publicly available under a CC-BY 4.0 license for research purposes. It can be accessed at: https://analyse.kmi.open.ac.uk/open-dataset. OULAD contains anonymized demographic information, student assessments, and virtual learning environment (VLE) interaction logs from courses at The Open University and has been widely adopted as a benchmark dataset in learning analytics research. The ontology extension, SHACL validation shapes, SPARQL queries, and ETL template used in this study are available on GitHub at https://github.com/Baisou2025/ontology-driven-feedback.

Declarations

Competing interests

The authors declare no competing interests.