Dancing on the inside: A qualitative study on online dance learning with teacher-AI cooperation

Abstract

Recent technologies have extended opportunities for online dance learning by overcoming the limitations of space and time. However, dance teachers report that student–teacher interaction is more likely to be challenging in a distant and asynchronous learning environment than in a conventional dance class, such as a dance studio. To address this issue, we introduce DancingInside, an online dance learning system that encourages a beginner to learn dance by providing timely and sufficient feedback based on Teacher-AI cooperation. The proposed system incorporates an AI-based tutor agent (AI tutor, in short) that uses a 2D pose estimation approach to quantitatively estimate the similarity between a learner's and teacher's performance. We conducted a two-week user study with 11 students and 4 teachers. Our qualitative study results highlight that the AI tutor in DancingInside could support the reflection on a learner's practice and help the performance improvement with multimodal feedback resources. The interview results also reveal that the human teacher's role is essential in complementing the AI feedback. We discuss our design and suggest potential implications for future AI-supported cooperative dance learning systems.

Introduction

The online learning market has been rapidly expanding; in 2019, it was a USD 144 B industry, and by 2026, is expected to reach USD 374.3 B (Facts & Factors, 2020). The global pandemic due to coronavirus disease (COVID-19) accelerated this trend as schools and learning institutes have mostly closed due to lockdowns or social distancing. Students have started to learn at home through various online channels, such as Youtube and Zoom. Hence, understanding student experiences in the online environments (Zhou et al., 2021; Cumbo et al., 2021; Li, 2022; Lockee, 2021; Liu et al., 2023) has become increasingly important.

In the online environment, it is especially challenging to learn body activities, such as dancing or playing sports. For example, watching and following videos on YouTube provide convenient and easy opportunities to learn dance for beginners (Gratsiouni et al., 2016; Hong et al., 2020). However, the interactions between students and teachers are limited in this environment (DeWitt et al., 2013; Li, 2020, 2021; Liu et al., 2023). Unlike offline dance classes where teachers and students are co-located simultaneously, beginners may not easily grasp visual cues in tutorial videos, and compare their movements with those in the videos (Sööt & Leijen, 2012). In particular, they may not receive timely and proper feedback, which is crucial for the student’s learning experience (El Raheb et al., 2016a; Harbonnier-Topin & Barbier, 2012).

In recent years, Artificial Intelligence (AI) technologies have been growing rapidly in various domains, including dance (Lee et al., 2020; Zhou et al., 2021; Choi et al., 2021; Guo et al., 2022). Among these technologies, human pose estimation models for 2D video (Cao et al., 2019; Ma et al., 2022) allow us to recognize dance movements and offer feedback to learners based on the captured movements. In addition, AI-based pose estimation is more lightweight and practical than other technologies for motion recognition, such as motion capture (Camurri et al., 2016a, b; Qian et al., 2006) and Kinect (Alexiadis et al., 2011; Kim & Kim, 2018), which are expensive, and have limited accessibility to being generally applied for students. Hence, we employ a pose estimation model to provid timely feedback. In addition, we further consider the roles of human teachers, who while essential to learning dance, but have been paid little attention to in the design of AI-based dance learning systems (Lee et al., 2020). In particular, we focus on how human teachers cooperate with AI technology to learn dance.

In this paper, we propose a new attempt toward online learning, especially for dancing, which can provide timely and substantial feedback for dance novices to improve learning experiences. We design and develop DancingInside, an online dance learning system that supports dance learning through a ‘practice, feedback, reflection, and modification’ process, as illustrated in Fig. 1. We provide feedback through the cooperation of an AI tutor and a human teacher. Furthermore, we encourage reviewing the feedback and reflecting on a learner’s movements in remote learning environments to extend the roles of students from passive recipients of information to active learners.

Fig. 1.

Proposed process for online dance learning with human and AI tutors: Practice, Feedback, Reflection, and Modification

Based on components proposed in Slovák et al. (2017) to foster a scaffolded reflection learning process, DancingInside is designed to iteratively provide two types of feedback: (i) explicit feedback to correct a detailed movement, and (ii) social feedback to encourage a supportive learning environment. To this end, we introduce an AI tutor in DancingInside that estimates the similarity between the learner’s movements captured on a webcam and the teacher’s movements from a tutorial video. The movement of a learner or a teacher can be captured from a video using a pose estimation method (Cao et al., 2019). The AI tutor scores a learner’s movement by comparing it to a teacher’s, referred to as explicit feedback. In DancingInside, a teacher also provides feedback on what each learner has been doing over three to four days. The teacher feedback includes the overall assessment of a learner as explicit feedback and encouraging notes as social feedback.

To evaluate how the proposed online dance learning system helps the learning experience, we conduct a two-week field study with 11 students and four teachers. By qualitatively analyzing their experiences in the proposed system, we seek answers to the following questions: (Q1) Whether DancingInside helps students improve their dance skills? (Q2) How can the AI tutor support learning dance? (Q3) How can the human teacher cooperate with the AI tutor to learn dance?

Overall, we find that the movements of the students and the teacher become similar over practice, which demonstrates that the proposed system effectively improves the dance techniques of the learner in the online environment. We also find that the AI tutor and the teacher in DancingInside play complementary roles in providing feedback; e.g., timely and explicit feedback by the AI tutor vs. overall impression and social feedback by the teacher. Such collaboration between the AI tutor and the teacher is essential and effective in online dance learning. We suggest design implications for the future directions of AI-supported dance learning systems.

Background and literature review

In this section, we review the methods for dance learning and introduce the design goal of the proposed dance learning system, based on the literature. We further review the prior work on online dance education and computational tools to support learning dance.

Methods of dance learning

In the literature (Camurri et al., 2016a), there are three representative methods of dance learning and teaching, summarized as follows.

• Mimesis method: This approach mainly consists of specific dance sequences, i.e., a teacher demonstrates first, and then a learner follows the demonstrated sequence.

• Generative method: In this method, teacher sets a creative goal by providing the starting point of a goal. After the given point by the teacher, a student starts to generate their sequences.

• Reflexive method: In this method, a topic for movement is given to a student, such as “sprouting seeds” or “flying birds”, and the student then improvises to express the topic through body movements. In this way, a teacher provides feedback, rather than teaching a specific sequence.

Among the three representative methods, the mimesis method is usually applied to most online dance learning formats, especially for video-based dance learning in MOOC, YouTube, and TikTok videos (Putri, 2021; Trajkova & Cafaro, 2021). In the mimesis approach, beginners can easily learn dance by repeatedly watching and following a given dance sequence. We design the proposed system based on the mimesis method to support online dance learning for novices.

The learning process in the mimesis method relies on the student’s abilities, e.g., in recognizing and following movements. Hence, supporting learners to capture visual cues in a given video can be helpful. In addition, providing proper feedback for self-reflection can encourage students to actively engage in learning rather than passively copying the given sequences in the mimesis learning method (Harbonnier-Topin & Barbier, 2012; El Raheb et al., 2016a, WhoLoDancE project, 2016). El Raheb et al. (2016a) and the WhoLoDancE project (2016) described the learning cycle in the mimesis method by adopting Kolbe’s experimental learning theory (Kolb, 2014) in dance education as follows: (i) concrete experience, (ii) abstract conceptualization, (iii) reflective observation, and (iv) active experimentation. That is, educational experiences such as observing and imitating a teacher’s dance sequence (i.e., concrete experience) are initially provided to learners. Then, the learners try to digest each movement by naming the parts (i.e., abstract conceptualization) and reviewing the movements of a teacher, other students, or themselves (i.e., reflective observation). The learners can then apply those reflections to the movements by practicing again or interpreting the dance sequences in their own creative ways (i.e., active experimentation).

This paper proposes an online learning system that can support the above process of the mimesis method by designing a learning cycle of practice-feedback-reflection-modification. We design the proposed system to provide adequate feedback and help the understanding of visual cues, based on the framework proposed by Slovák et al. (2017) for effectively eliciting self-reflection.

Challenges of online dance learning

Learning experiences are often categorized based on space (distant vs. co-located) and time (synchronous vs. asynchronous) (Chen et al., 2021). While traditional classes rely on co-located and synchronous settings, live streaming can be regarded as a distant and synchronous learning format. MOOCs or YouTube training videos are both remote and asynchronous learning formats. In this study, just like MOOC and YouTube videos, our system targets an asynchronous online dance learning system that is independent of space and time.

Similar to other fields, online learning in dance education has received increasing attention in the last decade. A number of online videos contributes to attracting many learners to learn dance online (DeWitt et al., 2013; Gratsiouni et al., 2016; Hong et al., 2020; Trajkova & Cafaro, 2021). Various dance videos on Youtube and their ease of use contribute to YouTube being used as a learning channel for dance learners (Hong et al., 2020). In addition, it is also beneficial for dance teachers, as YouTube can be used as a supporting tool to provide students with information on dance (DeWitt et al., 2013).

However, learning from videos has the following limitations. The transition to the online learning environment can restrict student-teacher interaction because they are not in a face-to-face setting. A teacher cannot fully observe students and correct them immediately (DeWitt et al., 2013). Most beginners may not find it easy to understand visual cues in dance videos. The reflection through video feedback can also be limited without sufficient directions (Sööt & Leijen, 2012).

Dance learning systems are proposed to support better learning experiences in online environments that focus on offering sufficient feedback for self-assessment and reflection, provided by various sources. A remote teacher offered a visual and verbal feedback system in an interactive augmented environment, as proposed by Trajkova and Cafaro (2018). Peer feedback was encouraged by a video commenting system, designed by Hsia et al. (2016). In particular, there are many attempts to utilize technologies to capture dance movements for timely feedback (Anderson et al., 2013; Chan et al., 2010; Lee et al., 2020; Muneesawang et al., 2015; Zhou et al., 2021). For example, Lee et al. (2020) developed an AI-based smartphone application to evaluate pose accuracy in real-time for self-learning.

These studies are based on the technologies that support dance practice in remote environments through timely evaluation and feedback. However, little attention has been paid to the roles of the human teacher. The quality of dance movements consists of both quantifiable factors related to positional data, and unquantifiable factors related to non-verbal expression (Camurri et al., 2016b). While a dance teacher evaluates these two factors in an offline class, the existing literature (Anderson et al., 2013; Chan et al., 2010; Lee et al., 2020; Muneesawang et al., 2015; Zhou et al., 2021) provided feedback focusing on one side of the dance quality, technical viewpoints such as motion accuracy. Hence, our work seeks to address the above limitations, and explore new opportunities available in the online environment by considering human intervention in the loop of computational feedback. To this end, we introduce both an AI tutor and a human teacher into the learning process.

Technologies on computer-supported interactive dance learning systems

In traditional dance classes, mirrors are widely used as educational tools to support dance learning and teaching. A mirror is used as a visual tool to present the external image of the dance performance. Although a mirror provides bodily knowledge of a dancer’s moves in real-time, there is no specific indicator of whether the movement is correct, which can play an important role in dance education (Trajkova & Cafaro, 2018).

Recent advances in technologies and digital environments can improve the dance learning process. Technologies for capturing and quantifying dance moves, such as motion capture (Camurri et al., 2016a; Chen et al., 2005; Nakamura et al., 2005; Qian et al., 2006) and Kinect (Alexiadis et al., 2011; Kim & Kim, 2018; Kitsikidis et al., 2014; Raptis et al., 2011; Saha et al., 2013; Saha et al., 2016), are used as essential parts in computer-supported dance learning systems by providing partial feedback. These technologies allow abstract dance movements to be recognized and quantified with high accuracy. Therefore, students can look at their recorded movements from a distance, which can support deep understanding of the movement of muscles and bones, self-evaluation, and further reflection (Spector, 2005). However, the existing learning tools are impractical and expensive to easily integrate into distant and asynchronous online learning settings. For example, a user cannot access a Kinect-based learning system without a Kinect camera. The limited accessibility of equipment can be a barrier to participation for students in an online course (Li et al., 2021).

As mentioned above, we aim to utilize a computational tool that is applicable to the online learning environment. In this study, we apply a popular, lightweight, and accurate human pose estimation model for 2D video, OpenPose (Cao et al., 2019), to evaluate a learner’s movement captured in a given dance video. Compared to Kinect-based and marker-based methods, using deep learning on a 2D video for pose evaluation showed comparable performance (Kim & Kim, 2018). In addition, recent work on computer-supported dance learning systems has been extended to develop technologies for dance pose estimation with 2D video. With pose estimation for 2D video, SyncUp (Zhou et al., 2021) and Choi et al. (2021) have proposed a feedback system that provides pose similarity between the practice video and the reference video. Although these studies have contributed from the aspect of technologies, how to incorporate cooperation between AI feedback and the human teacher to support dance learning in a remote environment has not been addressed.

A preliminary study: Understanding the dance learning practice

Study design

Before designing the online learning system, we conducted a preliminary study (i) to understand the dance learning practice in a traditional setting (i.e., co-located and synchronous in a dance studio), and (ii) to confirm design considerations for the proposed system in addition to the related literature in the aforementioned Section, followed by a participatory design approach (Muller & Kuhn, 1993; Schuler et al., 1993; Lee et al., 2021) which is one of the system design methods where stakeholders participate in a system design procedure.

We recruited four dance teachers by contacting a contemporary dance education institute, with 10+ years of teaching experience. Table 1 provides the detailed information on the participants. These teacher participants are interested in teaching dance online and applying AI-supported tools to their classes. The four teachers have experience teaching students from diverse backgrounds, from beginners and amateurs to college majors. We conducted semi-structured 30-minute interviews via a virtual meeting software. We asked the participants about (i) the overall dance teaching process and their feedback methods during a class, and (ii) homework and follow-up dance practices after a class. The participants were asked to answer the questions based on beginners’ experiences as a target group. Note that these teachers also participated in the following process after the preliminary study, by creating dance sequence and tutorial videos, providing feedback as a human tutor, and participating in post-interviews.

Table 1.

Description of the teacher participants in a preliminary study

Label	Gender	Age	Education	Genre	Teaching Experiences (year)
T1	Woman	31	BA	Contemporary, Ballet	10
T2	Woman	38	MFA	Contemporary	8
T3	Man	40	MFA	Contemporary	13
T4	Woman	33	BA	Contemporary	13

Results

During class

A dance class generally takes 60–90 minutes and includes several students with one teacher. All teachers responded that the educational goal of the contemporary dance class is to develop more dynamic and creative body movements. For beginners, this goal is challenging because they may not be familiar with controlling their bodies. Therefore, teaching direction can vary according to the student’s skill level (i.e., beginner to non-beginner). The beginner classes tend to focus on students using and moving their bodies, whereas the other non-beginner classes emphasize creativity. More specifically, in beginner classes, the teacher guides the beginner dancer to imitate the teacher’s or other students’ movements. In addition, the teacher asks the beginners to achieve a specific goal to express something via body movements (e.g., writing their names by using only their chest), or remember the sequence of choreography. “Learning and imitating the given sequence is like a common promise in a class. That is, a student needs to understand before creating movement. Then, the learned movements are expanded into the student’s own moves.” (T3).

For a given student’s movement, most teachers tend to immediately provide both verbal and non-verbal feedback during a class. “I comment to my students by showing the suggested moves with an explanation.” (T1). However, the feedback can differ depending on students’ levels (e.g., beginners to non-beginners), educational goals (e.g., developing technical correctness, enhancing creativity or expressiveness), and the teacher’s teaching strategy. For example, one said, “I try to give more positive feedback for beginners to encourage them even if they improve a little bit, rather than provide negative feedback.” (T4). Another teacher answered that “I rarely give feedback individually. Instead, I catch overall problems first. Then, I notify all in the class about the problem and suggestions.” (T1). Some teachers answered that they encourage students to give feedback to each other to understand various viewpoints. However, for beginners, T2 said that the feedback from a dance beginner tends to be superficial, such as “difficult” or “interesting.”

After class

Similar to other educational fields, teachers give assignments at the end of the class. However, all teachers responded that they rarely provide an assignment to beginners. The teachers do not wish to make students feel burdened, because most beginners participate in dance class as a hobby and likely do not have sufficient time for practice. One teacher said, “The assignments usually are given to students who entered a dance school.” (T1).

T4 commented that most beginners do not know what practice is needed to improve dance skills and body expression. For example, students just repeat the sequence of dance performed in class. As they have difficulty in remembering the sequence of choreography learned in class, they usually practice together, not individually, to support each other in recalling the dance sequence.

Design considerations

Through the preliminary study, we extract three design considerations to apply to the proposed online dance learning system. First, in learning dance, dancing alone as a practice is essential (Harbonnier-Topin & Barbier, 2012). Unlike the offline environment, a dance learner can practice dance anytime, anywhere through online learning. Also, a computer-supported learning system can make timely and detailed feedback of a practice possible. Therefore, providing a function of computer-supported dance practice is essential in online dance learning.

Second, it is important to provide proper feedback to students in dance learning. One of the key characteristics of online dance learning is that the teacher and student can participate in dance learning in different places, at different times. That is, the teacher cannot immediately provide direction or feedback, according to the students’ current moves. Therefore, the online dance learning system should be designed to provide proper feedback functions by evaluating students’ dance, aligned with a teacher’s roles in traditional dance classes. Hence, we design a computer-supported tool that provides timely and accurate feedback. In addition, according to our interview results, the computer-supported tool should provide feedback in a multi-modal manner, e.g., explain incorrect moves in both textual and visual format.

Lastly, as students do not visit a dance studio in online settings, it is challenging to use mirrors, a traditional tool for reflecting body moves. Even if students can utilize their own mirror, the effectiveness of the feedback by mirror is likely limited. That is because beginners often fail to reflect and correct their body moves by looking in the mirror unless a teacher provides an appropriate guide. Moreover, those limitations also remain if the mirror is substituted by a recorded practice video, playing a similar role in presenting a learner’s external image. Therefore, the feedback in the proposed system should also be designed to support the students’ reflection process that can help them find and modify incorrect dance moves by themselves.

DancingInside: Online dance learning system

Theoretical background and system design

Considering the literature and the design considerations of the preliminary study, we design AI-teacher cooperated online dance learning system based on the theoretical background of reflective practice, proposed by Schöch (1987) and summarized by Slovák et al. (2017). From the reflective practice perspective, the student grasps knowledge through scaffolded reflection processes, rather than through a direct learning experience from the teacher’s teaching. Cooper (2013) incorporated the reflective practice for a novice who is the passive recipient of information due to limited bodily knowledge and self-feedback ability, and found that the reflective practice promotes engaged learning even for a beginner. Such a shift from passive learner to active learner can contribute to an interactive online learning environment by extending the roles of the student.

The reflection processes are generated by the interactions between various components in a learning program. Slovák et al. (2017) proposed three components: explicit, social, and personal. First, the explicit component directly provides a learning experience by offering specific tasks or tools to support reflection. Second, the social component offers a supportive learning environment and social resources, such as teacher or peer support feedback. Lastly, the personal component draws a student’s motivation and ability to reflect on the current learning state.

Inspired by Slovák et al. (2017), we propose DancingInside, an online dance learning system that can effectively drive the reflection process (i.e., reviewing feedback and modifying incorrect movement) in distant and asynchronous settings. Figure 2 presents a dance learning procedure with DancingInside. The proposed system sets a specific goal that completes 90 seconds of contemporary dance choreography. To make it easy for beginners, we divided the dance into three parts and organized the program to learn the dance cumulatively. The dance learning program consists of three chapters as follows: (1) the first chapter for 0–30 seconds, (2) the second chapter for 0–60 seconds, and (3) the third chapter for 0–90 seconds of the given dance sequences.

Fig. 2.

Dance learning modules in DancingInside

Each learning chapter consists of four modules: (1) warming-up videos and a dance tutorial video, (2) recording of practice, (3) an AI tutor report, and (4) a teacher report. First, the warm-up videos and dance tutorial videos are recorded by the dance experts, who are the four participants of the preliminary interview study in Section 3. Second, the student learns dance sequences through a tutorial video, including the teacher’s demonstration and explanation of the dance sequences. Third, the learner records a dance practice video as an assignment and immediately receives feedback from the AI tutor. The AI tutor includes a pose estimation method (Cao et al., 2019) to capture the similarities between the movements of the student and the teacher. The AI tutor feedback provides the overall assessment score and its detailed feedback, as explicit feedback, to provide information about the learner’s correct/incorrect movement. The overall score informs the student’s motion accuracy in numeric form. The detailed feedback evaluates the partial dance sequences in a multi-modal manner, along with a duet video and body parts evaluation. For the given sequence, the duet video visualizes the movement of the teacher and the learner together, and the body parts evaluation specifies which part is correct or incorrect. Finally, DancingInside provides teacher’s feedback on what each learner has been doing over three to four days. The teacher feedback offers an overall assessment of the learner’s dance moves as explicit feedback, and encourages notes in a positive tone as social feedback. Table 2 summarizes the types of feedback and the corresponding functions in the proposed system.

Table 2.

Feedback types in the proposed online dance learning system, inspired by Slovák et al. (2017)

Type	Role	Source	Method
Explicit feedback	Directly providing learning experience	AI tutor	Indicating correction of learner's movement
		Teacher	Commenting on learner's movement
Social feedback	Providing a supportive learning environment	Teacher	Reviewing in a positive tone with encouraging messages

Implementation

Figures 3 and 4 show the user interfaces of DancingInside, implemented as a web-based application. Its detailed implementation and user scenarios are presented as follows.

Fig. 3.

Example interfaces of dashboard and recording, translated into English. AI Tutor report Teacher report

Fig. 4.

Example interfaces of back pages, translated into English

Recording of practice

The student then practices the dance sequences alone as a daily assignment. Since it can be challenging for beginners to remember all the sequences, the system also displays the teacher’s moves next to the recording screen as shown in Fig. 3b. After the recording, the learner can decide to start analyzing the recorded video to receive AI feedback. If students want to practice more, they can do so again from the beginning without the analysis for feedback, which allows each student to practice at their individual learning pace. Note that due to computational costs, we set a maximum number (n = 3) for the submissions of the analysis.

AI tutor report

Once a student submits the recorded video for the daily assignment, the AI tutor immediately generates feedback, as shown in Fig. 5. First, each movement in the student video is synchronized with the teacher’s one in the tutorial video. Next, each frame in a movement video as input data passes through a pose estimation model, OpenPose (Cao et al., 2019), to obtain the 18 key-point vectors, which include the coordinates of body parts (e.g., eyes, nose, arms, and legs) in the given frame. To measure motion similarity between the teacher and student, we construct pose descriptor vectors from 18 key points for each frame by using 9 pre-defined pose descriptor vectors, which can represent the dynamics of the dance moves. Finally, we calculate cosine similarities between the key vectors of the frames for the student and the teacher. Since the cosine similarity score ranges from -1 to 1, we scale it from 0 to 1. Using the frame-level similarity scores, the AI tutor can assess the overall score and the correction for each body part.

Fig. 5.

The overall process of the pose estimation to compare the movement of a student and a teacher, provided by the AI tutor

The AI tutor report consists of the overall assessment and detailed feedback in Fig. 4a. The section for the overall assessment displays the overall score, the timing accuracy, and the summary of body parts. The overall score is the average similarity between the teacher and student’s movements. The timing accuracy measures whether the start timing of a dance movement is the same teacher and student. The function of summary body parts shows how and which body part was correct during dance practice.

The detailed feedback consists of the following sets: (i) the visual feedback by a duet video synchronizing the student’s video with the teacher’s, and (ii) parts evaluation indicating which parts (i.e., arm, legs, pelvis, and shoulder) need to be modified. The whole choreography is divided into partial segments with the sequence name tag, annotated by the dance teachers.

Teacher report

The teacher report in Fig. 4b is provided after repeatedly conducting the daily assignments and reflecting on the AI tutor reports, before moving to the next chapter. The student selects one of the videos among the submitted recordings of daily assignments as a chapter assignment. The teacher then reviews and comments on the submission of the student, which is delivered to the student as the chapter report.

Research method

We evaluate the proposed system by conducting experiments to answer the following questions.

• Q1. Does DancingInside help students improve their dance skills?

• Q2. How can the AI tutor support learning dance?

• Q3. How can the human teacher cooperate with the AI tutor to learn dance?

We first analyzed how the participants digested the given dance sequences during the experiments via usage logs involving the similarity scores for each assignment. We then investigated how the AI tutor feedback functions enabled students to support their practice by analyzing the students’ interview responses. Lastly, after investigating the review comments in the teacher report and interview responses from both the students and the teachers, we present the roles of the teacher cooperating with the AI tutor for learning and teaching dance.

Study design & participants

We aimed to understand the detailed usage and contexts of DancingInside based on focused interviews and qualitative analysis. In particular, we focused on beginners in dance. The proposed system aims to support the limited learning environment in asynchronous and remote settings, and compared to the higher levels of learning, beginners will likely have more trouble following dance videos and determining wrong points from their recorded practice videos. Therefore, we expect that as a potential user group, DancingInside will provide them with a better learning experience.

We recruited college students with little knowledge of dance background by uploading recruitment posts on the online boards at large universities in the Republic of Korea. As a result, 11 students joined the focused group for our study. The experiments were performed for 14 days, from February 15 to 28, 2021. Table 3 summarizes the information of the student participants. The student participants were guided to complete the dance learning course, which consists of three chapters, as described in Section 4. The first, second, and third chapters were conducted from (1) February 15 to 18^th, (2) February 19 to 23^rd, and (3) February 24 to 28^th, 2021, respectively. As there were plenty of assignments and an interview, we offered approximately 70 USD for each student as compensation to obtain sufficient participation for the analysis.

Table 3.

Description of student participants in the interview study

Label	Gender	Age	Background	Completing ratio of assignments (n = 17)
S1	Woman	25	Engineering	100.00%
S2	Man	23	Sport Science	64.71%
S3	Woman	21	Social Science	94.12%
S4	Woman	21	Engineering	58.82%
S5	Man	25	Engineering	100.00%
S6	Woman	27	Engineering	70.59%
S7	Man	25	Engineering	100.00%
S8	Man	20	Engineering	100.00%
S9	Man	23	Sport Science	47.06%
S10	Woman	25	Engineering	58.82%
S11	Woman	20	Engineering	41.18%

Among the four teachers who were interviewed in the preliminary study and created the tutorial video and the dance sequences, one dance teacher (T1 in Table 1) took the classes akin to an offline dance class, reviewing the submitted practice videos of the students and providing comments for the teacher report. Also, the other three teachers (T2–T4 in Table 1) participated in the user study by monitoring the experiment.

Data collection and analysis

During the experiment, the average completion ratio of the assignments was 75.94%. The average completion ratios were 80.00%, 75.76%, and 69.70% for each chapter, respectively. In addition, the usage logs, including the similarity scores for each assignment, were stored in the database of the proposed system. We obtained 206 usage logs made by the student participants.

After finishing the three learning chapters, semi-structured interviews were conducted for about 30–40 minutes through a virtual meeting software with the 11 students and four teachers. The interview questions were designed to seek out answers to our three experimental questions. The authors first created the detailed interview questionnaires, and then reviewed internally whether the questionnaires were properly designed to elicit answers for the experimental questions. The interview questions for the students were designed to tell us about (i) how they experience the feedback functions, (ii) whether and how such AI and teacher feedback functions are utilized during dance practice, and (iii) how they perceive feedback differently from the teacher, compared to from the AI tutor. The questions to the one teacher, who provided feedback, were designed to ask about the teaching experiences in the proposed system. The rest of the teachers were asked about system design.

To examine the participants’ responses, we employed an open coding approach, following Creswell (2013). First, two authors transcribed and reviewed the interview transcripts line-by-line, and then independently extracted themes by manually identifying frequently occurring answer patterns. Next, the authors integrated the extracted themes for each coder into the themes and the sub-themes, and then categorized the participants’ responses into these themes and sub-themes. The authors coded the answers if the participants’ answers were related to the (sub-)themes. For this coding process, two authors independently coded the interview answers. We then integrated and discussed the coding results with each other, and if we could reach agreement, excluded the coding results. Note that the teacher’s comments (described in Section 4.2.3.) are also coded, followed by the same coding process. All quotes in the paper have been translated into English from Korean.

Results

Changes in similarity score

We first investigate the changes in the overall scores across the dance learning chapters to analyze how much each student’s dance moves had improved in terms of the similarity to the teacher’s movement. Figure 6a and b show the average dance score of each learning chapter for the (a) (0 – 30)s and (b) (30 – 60)s sequences, respectively. The average dance score is calculated as follows:

$$\frac{1}{N}\frac{1}{T}\sum ^N\sum ^{T}similarity\_score$$

Fig. 6.

Changes in the average score (y-axis) of (a) (0 – 30)s sequences and (b) (30 – 60)s sequences for each learning chapter (x-axis). The points represent the average score for each participant. Note that the participant who did not submit any assignments for the corresponding chapter is not displayed

where N is the number of the students; T is the trial time of the AI tutor report that returns the similarity score.

As shown in Fig. 6a, the average scores of the 2^nd and 3^rd chapters are higher than that of the 1^st chapter, but there are slight differences between the 2^nd and 3^rd chapters. A repeated measures ANOVA determined that the average scores of the (0 – 30)s sequences differed statistically significantly between learning the chapters (F (2, 16) = 4.220, p < .05). Post hoc analysis with Bonferroni correction indicated that the average score of the (0 – 30)s sequence was significantly increased from the 1^st chapter to the 2^nd chapter, t(10) = −3.718, adj. p < .05 with a large effect size d = 0.695, but not from the 1^st chapter to the 3^rd chapter, t(10) = −2.036, adj. p = .207 with an effect size d = 0.613, or from the 2^nd chapter to the 3^rd chapter, t(10) = 0.484, adj. p > .999 with an effect size d = 0.175.

Figure 6b also shows similar trends to Fig. 6a; a paired samples t-test showed that the average scores of the (30 – 60)s sequences were significantly increased from the 2^nd chapter to 3^rd chapter, t(9) = −2.547, p < .05 with a large effect size d = 0.805.

The results show that as time went by, the participants became more accurate in performing the dance, especially compared to the initial period of learning.

Supporting from the AI tutor

Next, we qualitatively analyze the interview responses to understand how the feedback generated from the AI tutor helps the dance practice of the students in an online environment. Table 4 presents the themes and the codes from the students’ interviews: roles of AI assessment, reviewing the data, affection to the AI tutor, and limitations of AI feedback.

Table 4.

Themes and codes emerged from the students’ interview data representing the experience of the AI tutor

Theme	Sub-theme: Code	Students (n = 11)
Roles of Score	Overall Score: useful for self-assessment	7 (63.6%)
	Overall score: important for continued use	3 (27.3%)
	Overall score: not useful for self-assessment	4 (36.4%)
	Requires display of other students' scores for compassion	4 (36.4%)
Reviewing feedback	Detailed feedback—Annotated sequence: effective for reviewing	2 (18.2%)
	Detailed feedback—Duet video: intuitive for reviewing	4 (36.4%)
	Detailed feedback—AI feedback: providing visual cues	3 (27.3%)
	Duet video & AI feedback: complementary for reviewing	6 (54.5%)
	Reviewing the feedback: helpful for increasing the similarity	5 (45.5%)
Affection to AI tutor	Compared to teacher	2 (18.2%)
	Prefers objective tone	6 (54.5%)
	Prefers emotional tone	3 (27.3%)
Limitation of AI feedback	Requires more detailed evaluation and suggestion	3 (27.3%)
	Requires additional evaluation metrics	5 (45.5%)
	Requires more space	6 (54.5%)

Roles of AI assessment

During the experiment, the proposed system quantified the abstract learners’ moves as the ‘overall assessment score’ in the AI tutor report. The students perceived that this function was useful in informing their learning progress. “Though the overall score, I can be aware of how much I improved or worsened than before.” (S2). In addition, the overall score motivated the students to practice more. “Initially, my plan was to record my practice video in one take without any retakes. However, after receiving a lower dance score than I had anticipated, I made sure to take extra time to submit the video, in order to achieve a higher score.” (S10).

On the other hand, some participants responded that the overall score was not informative to know their performance levels because they could not agree with the given score by the AI tutor. “I felt that the overall score I received overrated my performance. I thought I would get a low score when recording the dance. However, I was surprised to receive an 87, higher than my expectations.” (S6). The students also wanted information about other learners’ scores (e.g., average or distribution) for comparison and self-assessment, in addition to their scores. For example, S11 responded that “As the dance sequences became more complex after the second chapter, I wanted to know how the peers did. I wanted to know if they were also experiencing difficulties like I was.”

Reviewing the data

The AI tutor report also provided detailed feedback for each dance move annotated by the dance teachers, which consists of the visual feedback tool (i.e., the duet video) and the textual feedback tool (i.e., the body parts evaluation). The participants said that each part of the detailed feedback was helpful to review and reflect on their practice video. First, one mentioned regarding the visualization of the annotated dance movement that “The feedback segments of the dance sequences were more detailed than the tutorial or demo videos. This allowed me to explore the entire dance sequence effectively.” (S1). Second, the participants answered that the duet video was intuitive for reviewing their moves. “If I reviewed only my video, it would be have been difficult to identify areas where I needed improvement. However, with my video and the demo settled side by side, I could easily compare my moves with the teacher’s one.” (S10). Lastly, the students responded that the body parts evaluation provided clear indicators of corrections. For example, S1 answered “The duet video was helpful in grasping the overall flows and forms of the dance. Also, the body parts feedback helped me know exactly which parts I needed to focuse on in the duet video.”

In summary, the learners stated that the components of the detailed feedback complement each other. They revisited their past movement videos to review comprehensive feedback resources, and then practiced again by considering the incorrect moves they found. For example, one answered, “In the beginning, when I reviewed the body parts feedback, I was confused about why I received negative feedback. But after watching the duet video, which clearly showed the differences between my moves and the teacher's moves, I began to understand the feedback results. In the next trial, I focused on correcting those errors, such as the timing.I tried to fix those wrong points, such as timing.” (S10). This process helps obtain a better score. “After reviewing the feedback, I decided to fix the wrong points. It took me three tries to fully digest the feedback and make the necessary changes” (S9).

Affection for AI tutor

The students stated that the AI tutor was perceived as more objective than the human teacher. “I find that the AI tutor provides an objective evaluation without any subjective bias. Since the AI tutor does not have emotions, it is more comfortable when performing the dance, and less painful when getting negative feedback” (S3).

In addition, we asked them which tone is preferred for the AI tutor. The participants prefer an objective tone for the AI feedback, which is applied in the proposed system, to an emotional tone. “I think that the AI tutor providing direct correction would be more helpful. It gave me more clearer and more confident understanding.” (S11). On the other hand, some learners mentioned that they needed a more emotional AI-tutor, like an encouraging human when presenting poor performance. For example, S1 said, “When I received a report that simply said 'not accurate,' it was discouraging and demotivating. If the AI tutor provided feedback in a more positive and encouraging tone, it would be easier for me to feel motivated to improve.”

Limitations of AI tutor

On the other hand, the participants required a more detailed evaluation of how inaccurate they were for each body part, and suggestions on how to fix the wrong movements. “I could identify my incorrect moves through the feedback tools, but I was not confident in my own perspective. It would be more useful if the AI tutor could provide more specific guidance, such as how much I need to stretch my right leg.” (S8).

The students also wanted the additional evaluation metrics such as the flow between the sequences, or individuals’ body differences. S4 explained, “Although the AI tutor evaluated the accuracy of my moves, it did not consider the groove or flow between sequences, which I felt was lacking. The teacher also pointed out the groove of my performance.”

We further found that spatial limitations often resulted in the AI tutor’s failure to offer accurate assessments. As most of the students used our system in their home, studio apartment, or dormitory, they had difficulties in practicing the dance because of a relatively small space as compared to a dance studio. “The AI tutor was not able to capture my full range of motion due to limited space, which resulted in a lower score than I anticipated.” (S5).

Cooperation of teacher and AI tutor

Finally, we investigate how the human teacher cooperated with the AI tutor in DacingInside by focusing on the experiences of the teacher feedback. We first analyzed the teacher’s comments to understand how the human teacher evaluates movements’ qualities, compared to the AI tutor’s evaluation metrics. Then, we investigated the students’ experiences of the teacher’s feedback and learning dance with the human teacher versus the AI tutor. In addition, we analyzed the interview responses from the teachers to integrate their viewpoints of teaching the students with the AI tutor.

Learning with teacher & AI

We categorize the teacher’s comments about which factors she points out, in Table 5. Overall, the teacher assessed the students by providing various evaluation criteria. Like the AI tutor, the human teacher (T1) evaluated ‘timing’ and ‘accuracy’, as well as ’equilibrium’. ‘Equilibrium’ indicates stable and balanced movement, which is indirectly quantified through motion accuracy and timing by the AI tutor, but cannot be fully evaluated through the particular metric in the AI report. Furthermore, the teacher reviewed ‘emotion’ such as ‘shyness’, or ‘confidence’, and ‘groove’, such as rhythmical movement, which is related to how the external observer feels about the movement.

Table 5.

Codes derived from the teacher’s comments (translated to English) representing the contents of the teacher feedback. The vocabularies of the codes are derived from Camurri, Volpe, et al. (2016)

Code	Negative Feedback	Positive Feedback	Can AI evaluate?	Example
Timing	2	5	O	“…Some of moves are slower than they should be, so my recommendation is to repeat them several times until you get them right.”
Emotion	3	0	X	“…You seem a little shy. Just try to be more confident. You're doing great!…”
Groove	7	2	X	“…I suggest that you focus on feeling the movement and music, and work on performing smoothly between sequences…”
Accuracy	11	11	O	“…When you're performing the last part, your right leg is staying too close to your left leg.…”
Equilibrium	5	5	△	“…Aside from the first part, you're doing a great job. Practicing standing on one foot to improve your balance could be really helpful…”

These reviews offered descriptions of what can be done to fix the students’ movements. S1 said, “The teacher's comments and guidances were more clear and specific, such as ‘you need to bend your right leg more’, which I could not realize how to perform accurately. That point was more helpful than the AI feedback.” The participants responded that this complementary relationship between the teacher and AI feedback is helpful to practice the dance sequences. “I think the collaboration between the teacher and the AI was effective. The AI corrected my moves, but the teacher pointed out what the AI could not capture, like fluidity and groove.”(S1).

Moreover, the students’ responses indicate that the teacher provided social support. “Even though my dance score was around 70 or 80, the teacher provided me with very supportive words, which helped to boost my self-esteem.” (S10). Table 6 summarizes the themes and the codes from the students’ interviews about the experiences on learning with the teacher and the AI tutor.

Table 6.

Themes and Codes emerged from the students’ interview data representing the experience of the teacher feedback and the cooperation of the Teacher-AI

Theme	Code	Students (n = 11)
Roles of Teacher Feedback	Provides more detailed evaluation and suggestion	6 (54.5%)
	Encourages me	3 (27.3%)
Teacher-AI cooperation	Both helpful	4 (36.4%)
	Teacher is only helpful	1 (9.1%)

Teaching with AI

From a teacher’s perspective, we analyze the experience of teaching with the AI in DancingInside. The teacher, T1, who led the learning chapters and provided the teacher report, answered that it is convenient to review the submitted videos because a single video is segmented by each dance sequence with the annotated name. “When giving feedback in a face-to-face class, I can simply demonstrate the specific sequence I want to address. But in an online class, that's not possible. The annotated motion name is useful to indicate the exact motion that needs attention.” Besides, the proposed system promotes dance practice. T1 explained, “It was good that the system provides particular tasks that a student could practice after the class on a continuous basis.”

However, T1 mentioned she did not use the feedback results from the AI as a reference. “I just skipped the body parts evaluation of the AI tutor and wrote a review instead.” In addition, she felt confused when the AI’s assessment differed from hers. “Some students did not perform correctly and their posture seemed unbalanced to me, but the AI tutor evaluated their movements as correct. This created a conflict for me when I had to write feedback.” Including T1, other teachers who participated in the content of the learning chapter and observed the experiment, suggested enhancing the benefits of teaching with the AI tutor. T1 said, “When providing feedback through the AI system, the quality of transitions between sequences should also be considered, not just focusing on timing and accuracy.” T2 also mentioned, “It would be more helpful to provide specific instructions, such as the exact degree to which the right leg should be turned, rather than just indicating whether a motion is correct or incorrect.” In summary, the teachers responded that the AI tutor is helpful to practice for the given sequences repeatedly but, also needs more enhancement to support students to fully digest their own movement with high quality.

Discussion

So far, we have proposed the AI-incorporated dance learning system for supporting distance and asynchronized learning environments, and analyzed the experiences of the students and the teachers in the proposed system. This section summarizes and discusses the evaluation results and suggests implications for the future directions of AI-supported dance learning systems.

Summary of findings

We found that the participants’ dance skills had been improved according to using DancingInside. In particular, our finding highlights that the AI tutor helps the students to be familiar with the given dance movements by providing immediate and substantial AI feedback reports. Also, the proposed system encouraged beginners, who cannot catch visual cues, to reflect on the learner’s own movement with multi-modal feedback resources, which is a critical ability in dance education (El Raheb et al., 2016a; Leijen et al., 2009a, b; Raal, 2009; WhoLoDancE project, 2016). This reviewing process resulted in the increase of the similarity scores as time went by. Hence, our approach supported interactive dance learning in a distant and asynchronous scenario to overcome its limitations (e.g., receiving timely feedback from a teacher is challenging in asynchronous settings), aligned with a highly interactive dance learning system, summarized in Raheb et al. (2019).

Our results also showed that the teacher’s feedback complements the limitations of the AI tutor feedback by offering further guidance, including evaluation for nonverbal communication of movement qualities (Camurri et al., 2016b), such as ‘emotion’ and ‘groove’ in the teacher’s comments. This suggests considering the roles of the teacher in the learning process in AI-based dance learning system, similar to the prior studies (Arisidou et al., 2015; Trajkova & Cafaro, 2021) of supporting teachers rather than replacing them.

Implications for AI-supported dance learning system

Beyond similarity

We found that the AI feedback has a limitation from a certain point of view. That is, T1 mentioned that “The ultimate goal of contemporary dance education is not just an accurate imitation. It aims to express their own movement.” Considering artistic viewpoints can provide advanced dance education by qualifying dance movements beyond similarities and correctness, which have been examined in the literature (Choi et al., 2021; Zhou et al., 2021) about the AI-based dance feedback technologies. For example, this limitation can be improved by considering the essential body parts in the desired dance movements in the evaluation process.

In addition, we found that there is a need for more detailed feedback and recommendations or additional evaluation from both the students and the teachers. Also, the changes in similarities did not significantly increase at the end of the learning chapters. This implies that the scale of the assessment score may be limited to capture fine-grained differences between the students’ improved moves. Hence, AI feedback can be improved by considering how to capture and display detailed information on incorrect moves.

We expect that these design implications can contribute to a better experience in learning and teaching dance, resulting in better teacher-AI cooperation.

Providing resource for better interpreting score

We found that the numerical score (i.e., the overall assessment score) is useful for self-assessment. However, as the numerical scale may not be easy to interpret, the AI-supported learning system should consider how to design a support for interpreting the score. We found that the participants perceived the score as informative when they agreed. Hence, providing visual resources, such as the duet video, helped explain how the AI tutor returns that score. We also found that the students wanted to share other peers’ scores to understand their scores by relative assessment.

It is usually challenging for a beginner to keep participating in learning and exercising dance. As a quantified-self approach (Hamari et al., 2018) that can increase the motivation for continual use, the interview results showed that the proper feedback can increase a learner’s motivation. For the enhanced design of the AI-supported dance learning system, we suggest that incorporating a proper visualization and tracking the changes of the AI-evaluated score can encourage learning motivation.

Segmenting dance sequences for communication

According to Rivière et al. (2018) that analyzed the learning process of professional dancers, they found that the dancers decompose the whole movement into temporal segments. However, most beginners have difficulty in distinguishing the composition and overall flow of the choreography. Therefore, annotation tools (de Lahunta & Jenett, 2017; El Raheb et al., 2016a, 2016b, 2017; Singh et al., 2011) have been proposed to accurately identify poses and explore dynamic movements. Similarly, we found that displaying partial dance sequences with the annotated name supported the students in understanding the whole sequences in segmented forms. It also helped the teacher communicate in the teacher report by offering the particular name of partial motions. Despite the usefulness of the annotation for movement, an annotation process requires the effort of the annotators (i.e., T1-T4 teachers in this system). Also, an annotator is required to have substantial experience in dance so that he/she can decompose the whole sequences into meaningful forms for easily imitating the dance. Hence, a designer of an AI-supported dance learning system should consider (i) how to segment dance sequences in feedback, and (ii) how to support the annotation process that can possibly be done by an AI tutor, which can help teachers.

Handling when AI is wrong

Our system was based on pose estimation with a 2D webcam, which has limitations in recognizing human motions when a camera is unable to fully capture the whole body, due to small spaces or dark lighting. We found these limitations in the students’ interview responses, which are commonly reported in physical training apps that utilize pose estimation. According to the human-AI interaction guideline proposed by Amershi et al. (2019), a design should consider how to handle the case when AI is wrong. Hence, AI-supported dance learning systems should consider how to recognize poor conditions in pose estimation, and how to guide or help a learner to mitigate the bad conditions; this is in line with the prior work (Garbett et al., 2021) that proposed the AI design considerations in fitness apps.

Furthermore, we observed the difficulty in providing feedback when the teacher disagrees with AI evaluation results. The teacher often experienced a conflict in giving feedback to the students in the above case. We suggest that a designer consider how a system handles and communicates with a teacher in a case where the AI tutor and the teacher have opposite opinions.

Possible extensions: Domain, learning format, peer

We found that DancingInside helped the participants with no background in dance to remember the given body movements by practicing alone with multiple feedback functions. The design components of the proposed system can be easily extended into other domains in physical education, which need whole-body interactions to acquire knowledge, such as other dance genres, sports, or home fitness. Furthermore, while we designed our system in distant and asynchronous environments, extending our system into other forms of learning environments can be possible. For example, the tutorial video can be substituted with live-streaming classes or offline dance classes, and a student can use our system at home after the class.

In addition, the design of our system focused on providing the three types of feedback generated by an AI tutor, a teacher, and a learner. We did not consider feedback from peers, which is another feedback source to support learning dance. For example, it has been reported that peer assessment is important in learning dance (Hsia & Hwang, 2021; Hsia & Sung, 2020; Hsia et al., 2016). We found that some students wondered how other students participating in the experiment were doing. For example, “I wondered how other students review my practice video.” (S7). Integrating the peer-provided feedback in the proposed system can be used in future work.

Limitations & ethics concerns

There are several limitations to this study. The validity of our field study may be limited because the experiment was conducted over a short amount of time, two weeks, and the number of participants (11 students and four teachers) was small. By analyzing multiple data sources (i.e., the usage logs, teacher’s comments, interviews for students and teachers), we attempted to enhance validity of the qualitative study results through triangulation. We plan to extend our field study on a large-scale as future work for more validation and generalization. In addition, as all of them were college students in Korea, and we offered a reward for participation, the results can differ from the different geographic and demographic backgrounds and attitudes towards learning.

We carefully and closely considered all possible ethical issues during the entire research process. We only collected participants’ gender, age, and study domain for reporting the information of the participants, with the approval of the participants. We stored the participants’ choreography videos on the server, but we discarded the videos after computing the similarity score based on the teacher’s moves. This study is for educational purposes to evaluate the effectiveness of online dance learning, and the acquired data cannot be identified directly or through identifiers linked to the participants. For these reasons, this study has been reviewed and exempted by the Institutional Review Board (No. blind). Also, the facial images in Figs. 1, 3, 4, and 5 have been approved and blurred.

Conclusion

This study proposed DancingInside, a new dance learning system for distant and asynchronous environments. DancingInside supported the dance learning processes, ‘practice, feedback, reflection, and modification’, by combining the AI-based and teacher-provided feedback functions. Our study highlighted that DancingInside provides effective and interactive learning experiences for learning the dance movements in online environments. The interview results further revealed the importance of the role of the human teacher in complementing the AI feedback. We suggested the design implications for the AI-supported dance learning system, including developing various dance evaluation metric beyond similarity, supporting the interpretation of quantified dance feedback, supporting segmentation of dance sequences for communication, and handling conflicts between AI-based and teacher-provided feedback.

Authorship Confirmation/Contribution Statement

Jiwon Kang: Conceptualization, Methodology, Investigation, Formal analysis, Writing - Original Draft. Chaewon Kang: Investigation, Writing - Original Draft. Jeewoo Yoon: Methodology, Software, Data Curation. Houggeun Ji: Software, Data Curation. Taihu Li: Software, Data Curation. Hyunmi Moon: Conceptualization. Minsam Ko: Methodology, Validation, Writing - Reviewing and Editing. Jinyoung Han: Conceptualization, Supervision, Writing - Reviewing and Editing.

Funding

This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2021S1A5B5A17056279) and the MSIT (Ministry of Science and ICT), Korea, under the ICAN (ICT Challenge and Advanced Network of HRD) program (IITP-2021–2020-0–01816) supervised by the IITP (Institute of Information & Communications Technology Planning & Evaluation).

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Declarations

Conflict of interest

The Authors declare that there is no conflict of interest.