Decoding AI ethics from Users' lens in education: A systematic review

Abstract

In recent years, Artificial Intelligence (AI) has witnessed remarkable expansion, greatly benefiting the education sector. Nonetheless, this advancement brings forth several ethical dilemmas. The existing research on these ethical concerns within the educational framework is notably scarce, particularly when viewed from a user's standpoint. This research systematically reviewed 17 empirical articles from January 2018 to June 2023, sourced from peer-reviewed journals and conferences, to outlined existing ethical framework in Artificial Intelligence in Education (AIED), identify related concerns from user's perspectives, and construct Ethics Guideline for AIED. The finding revealed that certain ethical aspects, including the ethics of learning analytics and the ethics of algorithms in AIED, are often neglected in the existing ethical frameworks, principles, and standards for AIED. Based on the blank between existing ethical frameworks and ethic concerns from user's perspectives, the research proposes more inclusive and thoughtfully Ethics Guideline for AIED. The study also provides actionable recommendations for multiple stakeholders, emphasizing the need for guidelines that address user-centered concerns. In addition, How this Ethics Guideline for AIED could be developed is discussed, along with outlining potential avenues for future research.

1. Introduction

As a defining feature of the Fourth Industrial Revolution, Artificial Intelligence (AI) has become deeply embedded in various sectors of society, encompassing areas like the economy, education, healthcare, industry, and even daily human interactions [1,2] and facilitating the extraction and utilization of data to enable 'smart' governance through 'data-driven' approaches [3]. This integration signals a comprehensive embrace of AI across political, economic, educational, and social dimensions. However, it wasn't until the end of 2022, with the release of ChatGPT and its remarkable capabilities and accessibility, that an AI revolution truly seemed to sweep across every facet of society, particularly within the realm of higher education [4]. At the same time, the global outbreak of the COVID-19 pandemic propelled educational institutions into the virtual space, intensifying debates and efforts around how AI ecosystems could be leveraged to enhance global education in light of the shift to remote teaching and virtual learning [3].

The utilization of Artificial Intelligence in Education (AIED), defined as the harnessing of AI to provide personalized and automated feedback in educational contexts [5,6], is reshaping the educational domain [[7], [8], [9]]. This transformation includes diverse educational services such as automatic essay scoring, learning analytics, intelligent tutoring systems, smart assistive tech, natural language processing, and autonomous pedagogical agents [7]. These AI-enabled tools offer innovative strategies that can elevate teaching and learning [6,10,11], redefine stakeholder roles, streamline administrative tasks, and present solutions to prevailing educational challenges [7]. However, the extent of AI-enabled tools’ influence largely hinges on their integration and application within educational environments [8].

As a critical factor in the integration and application of AI-enabled tools, ethics continues to be a significant concern, particularly in both general AI and its educational applications. AI promises substantial advancements in social development and productivity, yet it also introduces concerns about transparency, data biases, security, privacy, and other ethical quandaries. AI ethics emphasizes human-centric values such as fairness, accountability, transparency, ethics, and safety [12,13] and concerns both public entities and private enterprises [14]. Grappling with these ethical and moral challenges that influence users, developers, and society at large becomes paramount [15]. While ethical concerns in AI have increasingly captured global attention, significant geographical divergence persists in how ethical principles are interpreted and should be applied. Proposed solutions to AI ethics challenges diverge considerably and global participation in the AI ethics debate is unequal. Developing regions such as Africa, South and Central America, and Central Asia are underrepresented, while more economically developed countries have a disproportionate influence in shaping this debate. This imbalance reveals a power disparity in international discourse and raises concerns about the neglect of local knowledge, cultural pluralism, and the imperatives of global fairness [14], which creates substantial barriers to realizing the human-centric values that ethical AIED seeks to uphold for users.

Embracing AI in education involves extensive data collection and analysis across various educational landscapes, bringing about serious human rights, ethical, and legal questions, and escalating concerns, particularly around personal data and learner autonomy [7]. Ethical AI is crucial for safeguarding the well-being of learners, teachers, and other stakeholders [16]. However, even with the accelerated expansion of AIED, particularly during the COVID-19 pandemic, the field still lacks robust ethical guidelines for crafting, developing, and implementing reliable and accountable AI within education [7,8]. Since the seminal work of Aiken and Epstein two decades ago, which introduced ethical guidelines for AIED [17], there's been a conspicuous scarcity of literature within the AIED community specifically discussing ethics [18,19]. Exceptions include the framework proposed by Nguyen et al. [20] for all relevant educational stakeholders through thematic analysis and the survey of questions conducted by Holmes et al. [21], which investigates of the AIED community's understanding of AIED ethics and emphasizes the significance of establishing a framework that combine a multidisciplinary approach and a set of robust guidelines.

While various frameworks, guidelines, and principles have been proposed by international or national institutions (see Table 1), most of these provide a relatively generic framework across disciplines and settings or fail to consider certain critical ethical issues about AIED. As such, they might not be appropriate, specific, and sufficient for education at all levels. There remains an absence of a consistent and comprehensive ethical approach to using AI in education.

Table 1.

Established international organizational and national institutional ethical frameworks.

Established international organizational and national institutional ethical frameworks
The Institute for Ethical AI & Machine Learning (2018) [22]	The Responsible Machine Learning Principles
House of Lords. (2018) [23]	AI in the UK
International Association of Privacy Professionals (IAPP, 2018) [24]	Ethics into Privacy Frameworks for Big Data and AI
The Public Voice (2018) [25]	Universal Guidelines for Artificial Intelligence
University of Montreal (2018) [26]	Montreal Declaration for Responsible Development of Artificial Intelligence
Natural Sciences and Engineering Research Council of Canada (NSERC, 2018) [27]	Canadian Field Robotics Network
NSERC Canadian Robotics Network (NCRN, 2018) [28]	Canadian Robotics Network (NCRN)
European Commission European Group on Ethics in Science and New. (2018) [29]	Artificial Intelligence, Robotics and ‘Autonomous’ Systems
British Educational Research Association (BERA, 2018) [30]	Ethical Guidelines for Educational Research
Monetary Authority of Singapore (MAS, 2018) [31]	FEAT principles
Institute of Electrical and Electronics Engineers (IEEE, 2019) [32]	Ethically Aligned Design (version 2)
European Union (2019) [33]	Ethics Guidelines for Trustworthy AI
United Nations Educational, Scientific and Cultural Organization (UNESCO, 2019) [34]	Consensus on AI & Education
World Economic Forum (Generation AI, 2019) [35]	Standards for Children and AI
The University of Edinburgh (2020) [36]	Data Governance Framework
United Nations Educational, Scientific and Cultural Organization (UNESCO, 2020) [37]	Draft of ethics of AI
Organization for Economic Cooperation and Development (OECD, 2021) [38]	Recommendation on Artificial Intelligence
United Nations Educational, Scientific and Cultural Organization (UNESCO, 2021) [39]	Ethics of AI
European Parliament (EP, 2021) [40]	AI in Education

Moreover, the ethical considerations of AIED have not been a priority for many educational technology companies and institutions. One contributing factor is the limited awareness among relevant stakeholders about the potential risks of AI in the educational context [16]. Coupled with the fact that establishing standard ethical guidelines for AIED is still relatively new for policymakers [41], concerns are growing among users about the lack of transparency and oversight. In this environment, understanding and approaching AI ethics in education from the user's perspective becomes both essential and urgent.

Overall, while AIED has been an area of academic research for over 30 years [42,43], experiencing significant growth in the last decade [44], the discourse on its ethics especially from the perspective of users hasn't kept pace. Emerging systematic literature reviews on AIED ethics concern a broad range of stakeholders, including users, policymakers, and developers, primarily focusing on ethical issues through meta-analyses [45] or exploring principles and their implications through thematic analysis [20]. However, they frequently lack a more focused exploration of the users' perspective. To fill the gap and respond to the need for a thorough understanding of the ethics of AIED from the perspective of users, this research used the Web of Science database to systematically review the exiting ethic framework of AI, ethical concerns of AI in education from the user's viewpoint, aiming to devise more inclusive and thoughtfully constructed ethical guidelines for AIED from the perspective of users, which can help raise awareness of ethics in AIED among users, address their concerns, encourage greater attention to ethics from ed-tech companies and educational institutions, and inform better policymaking. In doing so, it will ensure that human-centric values are upheld in the development and application of AI in education and promote the sustainable growth of AIED.

The Introduction provides an overview of the research problem, objectives, and the significance of the review. The Methods section details the search questions, research protocol, searching for relevant studies, inclusion and exclusion criteria, and data extraction, data analysis process. The Results section presents the findings from the included studies, organized in descriptively, synthetically and critically. The Discussion interprets the results in the context of existing literature, highlights implications, and suggests directions for future research. Finally, the Conclusion summarizes the key findings.

2. Literature review

Artificial intelligence (AI) has seen significant growth and adoption in the education sector over recent decades, owing to its capacity to replicate human-like functions [46] and automate routine tasks by harnessing vast amounts of data [47]. AI presents exciting possibilities for personalized and adaptive learning experiences for students, deepening teachers' insights into students' learning processes, streamlining administrative tasks for faculty, and offering on-demand, machine-supported assistance and immediate feedback—anytime, anywhere [7,48].

The global outbreak of the COVID-19 pandemic drastically shifted education towards online platforms, forcing institutions to rely heavily on ed-tech products to manage content, facilitate operations, and maintain communication with students. This influx of data, coupled with significant corporate investment, transformed the pandemic into a lucrative opportunity for educational technology, further accelerating the demand for AI-driven solutions that support remote learning. This trend is likely to fuel continued growth in AI-powered ed-tech products [16,49]. Among these innovations, ChatGPT—launched in 2022—has garnered particular attention for its unprecedented accessibility, and the quality and sophistication of its outputs, achieved through advanced deep learning techniques and cutting-edge algorithms. Its rise signals a profound potential to reshape both human-computer interaction and the educational paradigm.

While the integration of AI in education offers clear advantages, such as enhanced student engagement, collaboration, and greater accessibility, like all revolutionary technologies, certain AI applications can generate new ethical and legal challenges, especially around liability and potential biases in decision-making, raising concerns regarding ethics and equity, reliability, academic honesty and plagiarism, information privacy, transparency, algorithmic and content bias, and the possible reduction of human interaction [50,51], which poses great threat to human-centric values associated with different stakeholders including developers and users at all levels and presents significant challenges to future development of AI in education. Ethical concerns have undeniably evolved alongside the development and application of AIED, from its origins to its future advancements. As AI continues to advance and expand rapidly, its pervasive presence could expose various stakeholders to risks they may not even recognize. This situation may be further exacerbated by the effects of the COVID-19 pandemic, amplifying these challenges and uncertainties [52]. It is affirmed that, in addition to the ethical concerns related to the vast amounts of data collected and the computational approaches used, as with AI in general, AIED ethics must also address the ethics of education, the needs of teachers and students, and the ethical safeguards for AIED-enhanced pedagogical activities [21]. This highlights the importance of considering ethical AIED from the users' perspective.

The ethical landscape of artificial intelligence has been extensively explored by numerous international organizations, national institutions, and AI ethics initiatives (refer to Table 1). Prominent research includes Asimov's Three Laws of Robotics [53], Standardizing Ethical Design for AI [54], AI4people [55], Australia's Ethics Framework [56], and Ethical Principles and Values [57]. However, this study opted not to include these individual research frameworks, giving precedence to widely recognized frameworks from international and national authoritative bodies.

Table 1 illustrates that while numerous international organizations and technologically advanced national institutions are dedicated to crafting AI ethics models [58], the development of general guidelines for AIED platforms remains in its infancy for many stakeholders within the educational realm [41]. Overall, although numerous frameworks, guidelines, and principles have been proposed by international and national institutions (see Table 1), many of these are either too broad across disciplines and settings or overlook key ethical issues specific to AIED. Consequently, they may not be fully suitable, specific, or adequate for education at all levels. A consistent and comprehensive ethical approach to the use of AI in education is still lacking.

Nevertheless, it is noteworthy that UNESCO [34] pinpointed six challenges in achieving sustainable development of AIED [20], recommended actions to address the opportunities and challenges in education presented by AI, and affirmed AI development should be centered on people and AI in education must be designed ethically, without discrimination, ensuring equity, transparency, and auditability particularly in data collection, use and dissemination. The European Parliament [40] also emphasized that many users of AI in education are children and young people, a vulnerable group that requires special care and protection. This underscores the necessity and importance of addressing these challenges from the users' perspective.

The significant efforts by a wide range of stakeholders to establish AI principles and policies underscore not only the pressing need for ethical guidance but also reflect the vested interest these groups have in shaping AI ethics according to their unique priorities [59,60]. Veruggio emphasizes that the realm of ethics should envelop not just those who design and manufacture robots but also their users [61]. Ensuring user inclusion in the design journey is pivotal. Present-day trajectories in AI and computational science lean towards an AI approach that centers on the human experience, integrating personal attributes and contextual nuances to tackle ethical quandaries associated with algorithmic handling and execution [6,62].

Given that most AI in Education (AIED) platforms and tools are currently operated by private companies [63], and that the development of standard AIED guidelines is still a relatively new endeavor for many policymakers [41], users have growing concerns about the lack of transparency and oversight. In this context, understanding AI ethics from the user's perspective is critical. Such an understanding can not only inform better policy decisions but also address user concerns, thus contributing to the sustainable development of AIED.

Grasping AI ethics from the user's viewpoint is crucial for fostering its ethical usage in education. The willingness of educators to integrate AI into their practices hinges not just on the technology's accessibility and effectiveness, but also on their readiness to accept its role in the educational landscape [8]. Trust plays a pivotal role in technology adoption [64], and by addressing ethical considerations, we can enhance user confidence and subsequently their readiness to adopt AI-driven tools in education [65]. Hence, understanding AI ethics from a user standpoint can offer direction for the design, development, and policy-making processes, ensuring AI applications are ethically grounded and encouraging a proactive user approach to their optimal use. Furthermore, addressing AI ethics with user concerns in mind can offer solutions to their ethical reservations, motivating them to adopt AI in education in a conscientious manner.

Regarding emerging reviews on AI ethics, Jobin et al. [14] conducted a scoping review to examine global consensus and implementation of general ethical AI principles and guidelines. Yang et al. [66] focused specifically on radiology, while Mollmann et al. [67] conducted a systematic literature review centered on healthcare. However, all of these reviews gave limited attention to educational contexts. Recent systematic literature reviews on AIED ethics focus on ethical issues through meta-analyses [68] or explore principles and their implications through thematic analysis [20], considering a broad range of stakeholders, including users, policymakers, and developers. However, they often lack a more focused examination of the users' perspective, as seen in Hagendorf's [69] semi-systematic evaluation, which analyzes and compares relevant guidelines, highlighting overlaps, omissions, and the effectiveness of their implementation. Moreover, although many other reviews on AIED development touch on key ethical concerns regarding AI's role in education, they primarily focus on the use of specific AI-driven educational services or tools [70,71], affordance and challenges in applying AI in K-12 settings [72], as well as the impact of AI on student performance [73] educational process [74] and dental education in particular [75] there remains a noticeable gap in the comprehensive and thorough examination of ethical issues and frameworks within AIED.

A review of the existing literature reveals that the discourse on ethics in AIED remains underdeveloped, with consistent and comprehensive guidelines still lacking and insufficient attention given to users’ perspectives. This research addresses this gap by conducting a systematic literature review focused on ethical frameworks and issues in AIED from the perspective of users. By analyzing 17 peer-reviewed articles from 2019 to 2023 using content and thematic analysis, this study presents a conceptual framework that pinpoints three main areas of AIED ethics. The ultimate objective is to introduce an Ethics Guideline for AIED, designed to guide the design, development, and deployment of reliable and responsible AI within the educational realm.

3. Method

This research adopted the Tripartite model [76] as the framework for the systematic literature review. This model is informed by well-regarded systematic review methodologies such as the Cochrane Review Methods [77], which have significantly influenced the art and practice of systematic reviews and the methods to consolidate research evidence. The Tripartite model is structured around three core elements: Descriptive, which offers an overview of the literature; Synthesis, where research is grouped logically based on common themes, linkages, and underlying principles; and Critique, which involves a deep analysis of the literature to corroborate, refute, or introduce fresh perspectives.

3.1. Framing the review

The study is guided by the following research questions.

• (1)What theoretical frameworks have researchers employed to study AIED ethics from the standpoint of users?
• (2)What ethical concerns related to AIED have emerged from users' perspectives in these studies?
• (3)What Ethics Guideline of AIED be from users' perspective?

3.2. Developing a protocol

Initially, the study centered on collating existing ethical frameworks, principles, and guidelines in AIED and delineating AI ethical concerns within the educational realm. QA consulted an experienced education librarian and piloted the search strategies in major education databases in May 2023. Subsequently, based on the findings from this pilot search, we formulated a review protocol. This was succeeded by a synthesis of the literature, emphasizing the ethical frameworks and concers from the users' perspective in AIED. Subsequently, a critical evaluation of the research was undertaken, culminating in the creation of a conceptual framework that underscores three primary aspects of AIED ethics from the users’ perspective. The study concludes by presenting a Ethics Guideline for AIED.

Ethical deliberations surrounding AI are no longer confined to the realm of science fiction; they are tangible issues influencing the educational sector where AI is actively deployed. Stakeholders such as teachers, students, researchers, libraries, and administrators are still in the early stages of grasping the ethical ramifications associated with AI. Moreover, they are exploring how these frameworks can be harnessed to foster advancements in education. As such, examining the present research on ethical frameworks from the viewpoint of AIED users can offer actionable strategies for optimal AI integration in educational settings.

3.3. Searching for relevant studies

Articles were sourced from peer-reviewed database, Web of Science (Core Collection), including ProQuest Dissertations & Theses Citation Index, SciELO Citation Index, and Preprint Citation Index. To pinpoint studies that provide empirical evidence of AI ethics from the users' perspective in education, the following search terms were employed:

(educat∗ OR learn∗ OR train∗) AND (school OR institution OR college OR university) AND (user∗ OR applicant∗ OR student OR undergrad∗ OR postgrad∗ OR lectur∗ OR instruct∗ OR staff∗ OR academic∗ OR teacher∗ OR researcher∗ OR child∗ OR "young person" OR teenage∗ OR undergraduate∗ OR postgraduates∗ OR doctor∗) AND (AI OR "artificial intelligence" OR Chat∗) AND ethic∗.

3.4. Selecting the studies

The following were the inclusion and exclusion criteria.

• •Review Duration—Articles published from January 2018 to June 2023 were considered, beginning with the seminal paper "Ethics in AIED: Who Cares?" presented at the International Conference on Artificial Intelligence in Education (AIED) [78].
• •Language—Only articles written in English were included.
• •Article Type—To ensure the integrity and reliability of the findings, the study focused on peer-reviewed journal articles and conference proceedings. While literature reviews and books were consulted to enrich the research's conceptual depth, they were not included in the formal review process.
• •Focus of the Article—Articles were selected if they provided detailed descriptions and evaluations of AI ethics, or ethical applications of AI, from the perspective of users in the educational context. Articles that primarily discussed ethics that AI development should adhere to, ethical strategies of large organizations or national governments, purely theoretical framework discussions, or theoretical discussions related to curriculum design, were excluded.

3.5. Extraction data

The initial stage of the literature search yielded 310 papers published between January 2018 and June 2023, found through both keyword and manual searches. These articles were imported into EndNote for management, and duplicates were removed, leaving 298 papers. The articles were further filtered to ensure they included specific critical terms in the title, abstract, or keywords. QA reviewed the entire dataset, while JY and XX independently reviewed half of the dataset each, resulting in 67 articles being selected for more detailed title and abstract screening.

The next step involved verifying that the chosen articles were empirical rather than merely conceptual or theoretical. This verification process included analysis of the articles' discussions on frameworks (tools), data sources (from the user perspective), and research methods (quantitative, qualitative, or mixed). QA evaluated the full texts of these articles and discussed findings with the research team, leading to 17 articles being selected for comprehensive review. The risk of bias in each of these studies was independently evaluated by three reviewers using the Cochrane Collaboration's bias assessment tool. Any disagreements among the reviewers were resolved through discussion or consultation. Fig. 1 illustrates the steps taken to finalize the articles included in this review. This selection process was modeled after the PRISMA methodology [79].

Fig. 1.

Article screening with the PRISMA process.

3.6. Analyzing data

In the detailed examination phase, QA developed a spreadsheet for charting data after initially testing it on five manuscripts. This spreadsheet was subsequently reviewed and iteratively refined by JY and XX throughout the data charting stage. QA was responsible for charting the data, while JY and XX independently charted data from three articles each, selected at random, to verify the accuracy of the data extraction. All discrepancies were minor and were resolved quickly through discussion.

Qualitative methods were employed to analyze data. Descriptive statistics were analyzed to summarize key information of the reviewed literature, while content analysis [80] and thematic analysis [81] was utilized to synthesize existing frameworks, categorizing and comparing ethic concerns from users’ perspectives, and critical analysis helped thoroughly assess the quality, validity, and relevance of the outcomes and then construct Ethics Guideline for AIED. For the initial two research questions, the reviewers generated descriptive codes from the data extracted from the articles, such as “framework” and “guideline” “principle” and “standard” for RQ1, and “issue” “chanllenge” and “concern” for RQ2. Subsequently, the key findings related to RQ1 and RQ2 were identified, and a structured items map of Ethics Guideline for AIED was created to address RQ3. To ensure the accuracy and reliability of the findings, the reviewers regularly engaged in debriefing sessions to refine the Ethics Guideline for AIED items map. Additionally, consultations with Professor HZ, a seasoned expert in digital education and higher education leadership, were conducted to enhance the practical relevance of the review.

4. Findings

4.1. Descriptive characteristics

Most of the articles discussing ethics in AIED from the user perspective emerged in 2022, constituting over a third (35.29 %) of the total publications. This uptick may be attributed to the swift advancement of AIED and the escalating reliance on AI in education, particularly during the COVID-19 pandemic [82]. More than half (52.94 %) of these articles were centered on students' viewpoints, while approximately 17.65 % incorporated both students' and teachers' perspectives. Furthermore, 41.18 % of the articles sourced data from multiple regions rather than a single country.

In terms of educational levels, roughly half of the articles (52.94 %) were aimed at higher education, while 23.53 % catered to the high school or middle school demographic. Notably, most of the initially searched literature catered to general academic disciplines. Methodologically, a significant 58.82 % of the studies employed qualitative techniques—utilizing interviews, meetings, case studies, open-ended surveys, or observations—while 29.41 % used quantitative methods, primarily questionnaire surveys. However, only 17.65 % of these studies had sample sizes exceeding 300, just meeting the minimum threshold for enhanced generalizability [83].

Regarding theoretical underpinnings, 23.53 % of the articles turned to recognized international organizational and national institutional ethical frameworks. In contrast, some studies ventured without any predefined framework: 11.76 % relied on theories proposed by other researchers, and 35.29 % created their theoretical constructs. Although the methods across selected studies varied, they consistently honed in on the user group in the educational realm to ensure a user-centric view of AI ethics in education. Table 2 provides a concise overview of the articles' descriptive attributes.

Table 2.

Descriptive characteristics.

Characteristic	Number	% of total articles (N = 17)
Publication period
2019	3	17.65 %
2020	2	11.76 %
2021	3	17.65 %
2022	6	35.29 %
2023	3	17.65 %
Total	17	100.00 %
Site of study
United States of America	1	5.88 %
Serbia	1	5.88 %
Denmark	1	5.88 %
Brazil	1	5.88 %
Canada	1	5.88 %
Nigeria	1	5.88 %
Malaysia	1	5.88 %
Swiss	1	5.88 %
Russia	1	5.88 %
Spain	1	5.88 %
Multiple geographical locations	7	41.18 %
Total	17	100.00 %
User's perspective
students	9	52.94 %
teachers/researchers/experts	3	17.65 %
students & teachers/academics/dentists	5	29.41 %
libraries	1	5.88 %
Total	17	100.00 %
Educational level
higher education	9	52.94 %
k-12	1	5.88 %
high school/middle school	4	23.53 %
Elementary/primary schools and universities	1	5.88 %
(all levels)	1	5.88 %
not mention	1	5.88 %
Total	17	100.00 %
Discipline
general	12	70.59 %
specific-IT/science	2	11.76 %
specific-Dentistry/Medical	2	11.76 %
specific-Psychology	1	5.88 %
Total	17	100.00 %
Study methodology
Quantitative methods	5	29.41 %
Qualitative methods	10	58.82 %
Mixed methods	2	11.76 %
Total	17	100.00 %
Sample size
Below 50	9	52.94 %
51–149	3	17.65 %
150–300	2	11.76 %
300 above	3	17.65 %
Total	17	100.00 %
Frameworks usage
Established International organization's ethical frameworks None framework Total	4	23.53 %
	13	76.47 %
	17	100.00 %
Process of the study
Using established ethical framework with survey/observation to build a new one	2	11.76 %
Using established ethical framework to conduct a survey	2	11.76 %
Through survey/interview to build a framework	2	11.76 %
Using other researcher's framework to conduct a survey	3	17.65 %
Using self-developed framework to conduct a survey	5	29.41 %
Through interview/meeting/case study to investigate ethical issues	3	17.65 %
Total	17	100.00 %

4.2. Synthesis of reviewed literature

4.2.1. Summary of ethical principles or frameworks for AIED from user perspectives (RQ1)

A thematic analysis was conducted on the ethical principles and frameworks related to AIED mentioned in the selected literature, aiming to pinpoint common elements within these principles and frameworks. Among the referenced frameworks, "Ethics Guidelines for Trustworthy AI" emerged as an early foundational ethical principle for AIED. This guideline was frequently cited in articles that examined accessibility, biases, and ethics of chatbots and educational intelligent agents. Specifically, the fifth provision of these ethics guidelines, in tandem with the Web Content Accessibility Guidelines (WCAG) 2.1, was utilized to address accessibility concerns in chatbots, conversational agents, and virtual assistants [84]. In the same timeframe, this guideline, along with UNESCO's AI in Education [34] framework—which outlined six challenges for the sustainable development of AIED—was employed to investigate the influence of AIED's ethical impact assessments on primary teachers' capabilities and the benefits of AI on students' critical thinking [1].

Several prominent researchers in the AIED community have observed a lack of cohesive frameworks to address ethical concerns in the field. Responding to this gap, a robust framework called the 'strawman' draft framework for the ethics of AIED was created [18].

Furthermore, an adoption model for chatbots termed the extended Unified Theory of Acceptance and Use of Technology 2 (UTAUT2) Model, was introduced by incorporating four new constructs. Notably, two of these constructs focused on chatbot ethics, namely perceived trust and ethics. This model was employed to discern the factors influencing students' inclination to adopt chatbot technology. Even though it wasn't strictly an ethical framework, the model's expansion underscores the growing significance of AI ethics in upcoming models [64]. That same year, a model was fashioned based on UNESCO's 2020 principles. It zeroed in on the ethical ramifications of AI technologies within educational realms [35]. This model, which integrates both deontological and utilitarian ethical frameworks, aims to direct agent actions via ethical standards and to navigate ethical quandaries [85]. By 2023, to delve into the understanding of AI in medicine, a set of guideline-based expert discussions was initiated, culminating in a guideline. This guideline comprises three primary categories, with each integrating ethical contemplations [86].

4.2.2. Summary of ethical issues (RQ2)

Ethical concerns on the use of AI were anticipated findings and have been highlighted in prior scholarly work, while ethical issues from users' lens (teachers, students, administrators, libraries, and parents) in AIED have been neglected in the research [8,12,87,88], and public media e.g., Leprince-Ringuet [89]. While concerns of ethics was mentioned in some studies, it was interesting that it was not covered in detail and often reported in the limited sentences in the findings. Hence, this lack of emphasis could be filled in this review to find specific ethical issues raised from the user's point of view in AIED, including students, teachers, students & teachers/academics, researchers/experts, and librarians mentioned in the literature, which then were summarized and classified to highlight commonalities and variations in these viewpoints.

From the students' perspective, their concerns mainly include: 1) AI's perceived risks(81); 2) the relationsship between AI and humans, such as AI eventually supplanting humans and the potential adverse effects on an individual's social life, as well as safeguarding proper credit in collaboration with AI, grappling with matters of authorship, and broaching the boundaries of creative work copyrights (81, 77, 85); 3) trust in AI-derived quality of information and the potential for misinformation; 4) stakeholder responsibility and the absence of clarity on responsibility for AI-driven decisions (81, 86); 5) concerns of dehumanization and AI being devoid of emotions or causing physical harm (81, 86); 6) concerns of explainability, transparency, and fairness, such as seeking clarity of transparency in the data underpinning AI software and ensuring fairness (82, 87, 77); 7) concerns about maintaining data privacy and the potential misuse of data (83, 86, 87); 8) inherent biases in data sets toward women, visible minorities, individuals with disabilities, and indigenous populations (77, 86); 9) an undefined regulatory framework governing AI (87); 10) navigating issues of plagiarism and assessing the integrity of student writings (77); and 11) conflating AI with automation (86).

From the teachers' perspective, six concerns overlap with those mentioned by students: 1) transparency and fairness in AI decision-making processes(66,77), and 2) the relationship between AI and humans, with a potential scenario of increased AI presence and reduced teacher involvement(66), as well as safeguarding proper credit in collaboration with AI, grappling with matters of authorship, and broaching the boundaries of creative work copyrights(77); 3) potential privacy risks to both educators and learners due to data usage(66,77); 4) transparency and fairness in the data underpinning AI software(77); 5) inherent biases in data sets(77); and 6) issues of plagiarism and writing integrity(77). At the same time, teachers also consider six concerns different from students' concerns. They are the shift of autonomy and control from teachers to AI, discrepancies between AI evaluations and teacher assessments, the dearth of educational experience among AI developers, the AI algorithms' inability to grasp social, emotional, and motivational facets, the perceived supremacy of teachers' real-world expertise and instinct over a computer's logic, and the AI systems' ignorance of a student's external history(66).

From the librarians' perspective, their concerns are consistent with those raised by students, include existing discussions about data bias and 'filter bubbles'; transparency issues concerning collection decisions; concerns about privacy, data quality, and security regarding usage data; the potential replacement of current services with automation; and deliberations over whether libraries or other entities would drive information(80). Only one has not been mentioned, that is, the influence and methodology associated with data collection and its utilization for AI(80).

4.3. Critical analysis of AI ethical frameworks from user perspectives in education context (RQ3)

From a thorough analysis of the ethical frameworks, principles, guidelines, and concerns related to AIED drawn from the selected studies, it's evident that certain critical ethical issues about AIED have been overlooked in these documents. Among the selected studies, notable ethical guidelines include the Ethics Guidelines for Trustworthy AI [33], Consensus on AI & Education [34], the Draft of Ethics of AI [37], and the Unified Theory of Acceptance and Use of Technology 2 (UTAUT2) Model. To ensure a holistic and integrative review, five additional guidelines from four esteemed international organizations were also incorporated into the analysis. These include the Standards for Children and AI [36], Recommendation on Artificial Intelligence [38], Ethics of AI [39], and AI in Education [40]. Furthermore, Nguyen et al. [20] had previously scrutinized and summarized the ethical elements of six out of the eight aforementioned frameworks [82]. This prior work can serve as a valuable resource to facilitate a comprehensive evaluation of the existing ethical frameworks for AIED.

It is obvious that the ethical components of the eight pre-existing ethical frameworks, principles, and standards tailored for AIED have been centered around data ethics, computational methods, and education, specifically at the juncture where data ethics intersects with computational methods. This is commonly referred to as the general data ethics in AI. However, the ethical facets of learning analytics, a subject of significant research [90], often go unaddressed in many contemporary AIED ethical frameworks. Conversely, when it comes to the ethics of algorithms within the educational context, another area of substantial research [91], several critical areas are often overlooked in prevailing AIED ethical frameworks.

At the same time, ethical issues in AIED from users' perspectives, including students, teachers, students & teachers/academics, researchers/experts, and librarians, mentioned in the literature were summarized and classified to analyze whether these issues have been addressed by eight established international organizational and national institutional ethical frameworks for AIED. By comparing, there remain certain user-centric ethical issues highlighted in select literature that haven't been incorporated in these framework. Recognizing these user-driven ethical concerns can significantly aid in refining and expanding the ethical framework for AIED. Some of these concerns include the mutual respect between AI and humans [92], gearing AI to comprehend education better [1], acknowledging the rights and emotions of AI/robots [93], and reconciling differences between AI's diagnosis and a teacher's perspective. Other issues touched on the potential deficiencies of AI developers in the educational experience, AI algorithms' inability to grasp social, emotional, and motivational factors, and the perceived superiority of a teacher's real-world experience and intuition over AI. There's also the challenge that AI algorithms may be unaware of a student's history outside the system [65]. Moreover, challenges like diverse perceptions of AI's constraints on human understanding, matters of authorship, and copyright parameters for creative works also emerge [87].

Hence, it becomes apparent that certain ethical concerns from user's perspectives stemming from the research articles have not been adequately addressed in the aforementioned established ethical frameworks for AIED. These overlooked ethical facets fall primarily into two categories (the ethics of learning analytics and the ethics of Algorithm in AIED). The first pertains to the intersection of data ethics and education, which can be termed the ethics of learning analytics, as highlighted by Ref. [94], Drachsler and Greller [95], and Ferguson et al. [91]. The second category deals with the convergence of computational methods and education and is termed the ethics of Algorithm in AIED, based on the "strawman" draft framework presented by Holmes et al. [18]. This draft framework by Holmes and colleagues provides a visual representation of these intersections (see Fig. 2).

Fig. 2.

A ‘strawman’ draft framework for the ethics of AIED.

From the detailed analysis above, it becomes evident that there are discrepancies between prevailing ethical frameworks and the actual ethical challenges faced in AIED from users' perspectives. Drawing from Holmes's framework, we incorporated the ethical concerns in AIED from users' perspectives that haven't been included in pre-existing ethical frameworks into our framework system, in order to construct an Ethics Guideline for AIED from users' perspectives (refer to Table 3).

Table 3.

Ethics guideline of AIED.

Ethics of data used in AI	transparency/visibility and representation of data
	Explicability/intelligibility/interpretation/explanation of decision from AI system
	trust and reliability of the AI data
	assigned accountability/responsibility/liability for consequences or mistakes/errors of AI-based decision
	Auditability/inspectability of decision
	the data protection and privacy issues associated with AI use
	establishing and implementing procedures to guarantee individual/personal privacy
	confidentiality of the data
	Data ownership and control/governance
	Diversity, quality, and inclusiveness of data
	Non-discriminate/non-biased/non-prejudice assumption, data sets, and decision-making (e.g., about gender, age, race, social status, income inequality, religion, visible minorities, people with disabilities, and indigenous peoples)
Ethics of learning analytics	regulatory policies to surveil/Govern AI use involving all stakeholders/Multistakeholder including the data subjects as well as perspectives on data (e.g., institutional versus individual)
	Clarification of all stakeholders' responsibilities/obligations
	genuinely informed and freely given consent
	autonomy of humans (learners)
	management/governance, clarity, and legitimization of the collection, analysis, sharing using, acting, storage, dissemination, and exploitation of data
	power relations between teachers and their students
	open intentions, objectives, and purpose of education/learning
	guidance of determining added value and the rights of participants
	confidentiality and anonymity of data
Ethics of Algorithms in Education	equity and fairness: fairly and equitably access to education, equity and inclusive in AIED, treat research subjects ethically and fairly
	role of humans (teachers, library, researchers): human role and the human potential for learning and growth, the role of teachers (replace or augment by technology), give teachers new and creative roles
	the choice of pedagogy: embed pedagogies of learner and teacher empowerment into AIED systems, the entire sequence of AIED-enhanced pedagogical activity needs to be ethically warranted
	behavior and emotional impact: AIED interventions target behavioral and emotional change
	academic integrity: proper attribution and academic honesty in collaboration with AI, quality, and honesty of students' writing, prevention of plagiarism, ensuring academic honesty, rigor, excellence, and institutional reputation.
	requirements around human (students) agency: human agency in fully autonomous AI in Education systems and reflections on agency

5. Discussion

This systematic review outlined and analyzed exiting ethical principles or frameworks for AIED and the components in these frameworks. It was observed that the frameworks outlined and analyzed in the present study, 2 ethical principles have not been thoroughly examined before, although the other 6 ethical principles [34,33,[37], [38], [39], [40]] had been included in a prior review [82]. In addition, the ethical frameworks for AIED analyzed in this study is center in AIED, not on healthcare [67] and radiology [66].

This systematic review also outlined ethic concerns from users' perspectives, including students, teachers/academics/researchers/experts, and librarians. Simultaneously, the reviewed articles indicate that studies focusing on user perspectives primarily examine students' concerns. These studies explore students' perceptions, awareness, and the ethical implications of AIED, underscoring the significance of students as primary stakeholders. This emphasis is crucial given their vulnerability to AI-driven educational interventions and their direct exposure to the associated ethical challenges [13,64,84,85,96,97,98,99,100]. While previous reviews that focus specifically on AIED tend to address ethical issues for a broad range of stakeholders, including policymakers, developers and users [20,68], they often lack a focused exploration of the users' perspective. Thus, our review contributes a nuanced perspective of AI ethics in education as seen through the eyes of users, identifying specific ethical principles and issues relevant to AIED. Moreover, although many other reviews on AIED development touch on key ethical concerns regarding AI's role in education [[70], [71], [72], [73], [74], [75],101], they generally fail to provide an in-depth investigation of the essential elements that constitute ethical AI, its implication and the implementation for its realization, and fall short of offering a synthesized analysis of the existing frameworks and guidelines for AIED, particularly regarding their convergence, divergence, accomplishments and shortages.

This systematic review develops an Ethics Guideline for AIED from users' perspectives, aimed at guiding a wide array of users in AIED—including students, teachers/academics/researchers/experts, and librarians—in the ethical and reliable development and implementation of AIED systems; addressing overlooked ethical facets(the ethics of learning analytics and the ethics of Algorithm in AIED), based on the "strawman" draft framework presented by Holmes et al. [18]. Unlike previous principles that sought to establish a global consensus on ethical AIED, who explored global agreement and the implementation of general principles, they primarily catered to a wide range of stakeholders, including private companies, research institutions, and public sector organizations [14], this guideline tailored to the unique needs and concerns of AIED users.

In our study, we have aggregated fairly comprehensive Ethics Guideline of AIED, drawing from recognized international and national structures and focusing on the user's perspective. The research offers both theoretical and practical implications that are crucial for advancing the field of AIED ethics. By analyzing the theoretical frameworks researchers have used to study AIED ethics from a user's perspective, and identifying ethical concerns from users—students, teachers/academics/researchers/experts, and librarians, that have emerged from these studies, this study proposed an Ethics Guideline for AIED from a user-centered perspective. The research extends existing theories, paving the way for more comprehensive and inclusive ethical frameworks that incorporate the nuanced concerns of users. These theoretical advancements can influence future research directions, encouraging scholars to explore user-centric approaches in greater depth. Nevertheless, this constructed framework stands to benefit from further empirical investigations for validation and enhancement. On a practical level, highlighting user-centered ethical concerns and constructing inclusive and thoughtfully Ethics Guideline for AIED from a user's perspective can help educational institutions and ed-tech companies to create more user-focused and ethically responsible AI systems, can inform the development of policies, the design of AI tools, and the training of educators and students, ultimately leading to more ethical and user-aligned applications of AI in education. Additionally, this guideline can serve as a benchmark for policymakers aiming to regulate AI in educational settings, ensuring that policies are grounded in the real-world experiences and concerns of users.

5.1. Future research

5.1.1. Recommendations for the development of an AI ethics tool for evaluating

AIED ethics is a complex realm, spanning across six prominent areas: data, computational strategies, education, generalized data ethics within AI, ethics concerning data usage in education, and the ethical application of algorithms in educational scenarios. Establishing a thorough ethical guildline for AIED and identifying subtle ethical dilemmas intrinsic to AIED are merely the initial steps. The subsequent, equally vital phase involves strategizing the effective application of this guideline. This might involve devising specialized ethical tools for AIED, designed to cater to users, including educators, students, administrative personnel, parents, and the wider AIED audience.

User-driven surveys could be fashioned around elements extracted from the three focal points within the Ethics Guideline for AIED. For instance, to gauge students' consciousness or stance on AIED ethics, they could be prompted to identify the most pertinent ethical component within the Ethics Guideline for AIED. Furthermore, they could be posed an open-ended query: whether there exist any ethical considerations they deem crucial that aren't currently covered in the existing guidelines. Such an approach would cultivate a more comprehensive ethical framework for AIED, filtered through the students' lens.

Thematic-centered surveys could cherry-pick specific items from the Ethics Guideline for AIED to draft questionnaire prompts. This method can be deployed to assess a particular ethical dimension, like data ethics within education—a subject that has been the epicenter of numerous studies—across diverse demographics. This includes varied gender identities, age groups, ethnicities, educational backgrounds, disciplines, and geographies. Once these surveys are administered, the amassed data will provide a more balanced perspective, mitigating biases resulting from narrow data sourcing.

5.1.2. Recommendations for validating ethics guideline for AIED

The Ethics Guideline for AIED presented in this study should be validated through a longitudinal empirical studies. It would be particularly valuable in assessing the long-term impact of these guidelines on AIED practices, ensuring their relevance and effectiveness in diverse educational settings over time. Furthermore, theoretical analyses could help refine the ethical guideline by integrating insights from various ethical theories, particularly those that address emerging AI technologies. Research conducted from users from teachers, researchers, libraries, students and administrators is essential. This broader approach could address the current gap in diverse theoretical framings and ensure that the Ethics Guideline for AIED are robust, inclusive, and globally applicable.

6. Conclusion

The rapid development of AI, has brought about remarkable changes to human society, with significant implications for the field of education. In light of these advancements, we conducted a systematic review to consolidate current knowledge in this area. Our findings indicate that existing ethical frameworks, principles, and standards for AIED are inadequate in addressing the ethical issues from user's perspectives identified in the reviewed literature in the context of higher education. In addition, exiting ethical frameworks, principles, and standards for AIED predominantly focus on the general ethics of data in AI, often overlooking the ethics of learning analytics and algorithms within AIED. Therefore, we are poised to develop more inclusive and thoughtfully constructed Ethics Guideline for AIED from user's perspectives. For the future, higher education institutions should understand and develop Ethics Guideline for AIED from user's perspectives, ensuring the sustainable and responsible integration of AI in education.

CRediT authorship contribution statement

Qin An: Writing – original draft, Resources, Methodology, Data curation, Conceptualization. Jingmei Yang: Writing – original draft, Resources, Methodology, Formal analysis, Data curation. Xiaoshu Xu: Writing – review & editing, Validation, Supervision, Investigation, Formal analysis. Yunfeng Zhang: Writing – review & editing, Supervision, Project administration, Investigation. Huanhuan Zhang: Writing – review & editing, Visualization, Validation, Software.

Data availability statement

All data generated and analyzed during this study are included in this article.

Funding

This paper is supported by the Macao funding schemes granted by the Science and Technology Development Fund (0071/2023/RIB3).

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this article.