A comparison of natural user interface and graphical user interface for narrative in HMD-based augmented reality

Abstract

Over the years, the various mediums available for storytelling have progressively expanded, from spoken to written word, then to film, and now to Virtual Reality (VR) and Augmented Reality (AR). In 2016, the cutting-edge Head-Mounted Display (HMD) AR Microsoft HoloLens was released. However, though it has been several years, the quality of the user experience with narration using HMD-based AR technology has been rarely discussed. The present study explored interactive narrative in HMD-based AR regarding different user interfaces and their influence on users’ presence, narrative engagement and reflection. Inspired by an existing exhibition at the National Holocaust Centre and Museum in the U.K., a HoloLens narrative application, entitled The AR Journey, was developed by the authors using two different interaction methods, Natural User Interface (NUI) and Graphical User Interface (GUI), which were used to perform an empirical study. As revealed from the results of the between-subject design experiment, NUI exhibited statistically significant advantages in creating presence for users without 3D Role Playing Game (RPG) experience, and GUI was superior in creating presence and increasing narrative engagement for users with 3D RPG experience. As indicated by the results of the interviews, the overall narrative experience in HMD-based AR was acceptable, and the branching narrative design was engaging. However, HoloLens hardware issues, as well as virtuality and reality mismatch, adversely affected user experience. Design guidelines were proposed according to the qualitative results.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11042-021-11723-0.

Introduction

A narrative or story is any account of a series of related events or experiences. Studies of interactive narrative have increased in number with the continued development of narrative media. Interactive narrative gives the user control over the story that is being told, chose-your-own-adventure style. A branching story structure is a story graph – a directed graph of nodes connected by arcs that represent user choices. Every possible path through the graph represents a story that can be told to the user. The market for interactive digital narratives that can be utilized for entertainment, education, tourism, training, and more has shown drastic growth. VR and AR are among the most impressive iterations of interactive media, which integrate the latest media technology. Since AR used to be subject to more technical constraints than VR in terms of display, tracking, and spatial understanding, high-quality research had been rarely conducted with AR until the invention of HoloLens, an HMD for AR. HMD-based AR technology like HoloLens1 showed significant improvements, particularly in terms of image display, as it utilizes an optical see-through approach and spatial understanding, through the use of 3D scanning; thus, it is capable of seamlessly integrating a vivid hologram into a real-world space. Due to these advancements, novel opportunities and challenges emerge. However, current research on AR with HMD-based AR has thus far been focused on industrial usage, such as architecture, automobile design, medical training, and tourism [24, 26], while studies on interactive storytelling using HMD-based AR technologies remain limited. Fragments, an adventure game released on the HoloLens platform, was recognized as an initial exploration of interactive narrative in HMD-based AR [59].

Since HMDs free users’ hands for interaction, it is possible to use tangible or Natural User Interface in the AR narrative. However, the impact of different interfaces on interactive narratives in the latest HMD-based AR has been rarely explored. Through the comparison of different user interfaces created for a HoloLens AR narrative, the goal of this study is to examine how different user interfaces can potentially affect narrative experience in HMD-based AR and to formulate guidelines for designing HMD-based AR narratives. An interactive narrative application for the National Holocaust Centre and Museum (NHCM) on HoloLens, The AR Journey, was developed with two different user interface designs. Afterwards, experiments and interviews were conducted to validate the hypotheses of this study.

Literature review

We will briefly review current research and trends in interactive narrative and AR technology, as well as discuss the common challenges in creating interactive narratives for different media, particularly in regard to HMD-based AR technology. We will also review the practices of narrative, interaction methods, narrative structure and the user interface in AR storytelling. Finally, we will present our focus questions and research objectives.

Interactive narrative

Interactive narratives refer to stories with unfolding, pace and outcome that can be impacted by some intervention by a spectator or an entire audience. In existing studies, interactive narrative can be achieved via autonomous agent, choice-based story graph (branching structure), narrative mediation [47], as well as environmental storytelling [29]. Branching structure storytelling was made famous by the Choose Your Own Adventure (CYOA)2 series and is still prominent as a popular format for narrative videogames [41]. Various works developed by the videogame developer Telltales Games, such as The Walking Dead and The Wolf Among Us, are recent examples of it [60]. In this format, players navigate a plot graph by making decisions at branching points in the narrative.

Apart from pure entertainment, interactive storytelling has also been incorporated into serious games focusing on topics such as history [57], STEM [16, 63], and bullying [1]. Given that branching structure interactive stories have been shown to promote empathy in the participants towards the protagonists [25, 52], an interactive storytelling approach can be effective for changing perspective and for moral education. However, the majority of research on educational interactive storytelling games has thus far been focused on interactivity [57, 66], narrative planning and generation [48, 68], or simply the game creation process itself [15]. As a result, there is surprisingly little work assessing the impact of the interactive narrative interface on learning outcomes.

As digital media technology advances, the mediums for interactive narrative have evolved from text, video, and video games to immersive media, like HMD-based VR and HMD-based AR. However, few studies on interactive narrative using immersive AR could be found.

Augmented reality

Definition and AR technology

Azuma proposed that AR can be defined as a system exhibiting three basic features: a combination of real and virtual worlds, real-time interaction, and accurate 3D registration of virtual and real objects [2]. Displays, input and tracking system, and computers are the major components of an augmented reality system [12].

Three major types of displays are used in AR: head-mounted, handheld and spatial. Handheld displays use a video see-through system to overlay virtual graphics onto live video feeds through smartphone or tablet cameras. On the other hand, Spatial augmented reality uses video-projectors, optical elements, holograms and tracking sensors to display virtual graphics directly onto physical objects without requiring the user to wear or carry the display. Compared with handheld displays, which currently have the advantage of greater accessibility, HMDs are a still more promising technology, with a larger Field of View (FOV) and the hands-free capacity. AR HMD is capable of either video see-through or optical see-through and can be equipped with a monocular or binocular display optic. HMD technology has advanced greatly over the past few years, as developers have taken advantage of improvements in optical see-through technology, with HoloLens and Magic Leap One at the forefront.

The tracking system acts as a core part of AR. There are four main types: image-based, model-based, area-based, and location-based (also known as GPS-based) systems. The area-based tracking system is the latest technology for environment tracking and reconstruction, as well as the augmentation of areas and spaces based on both preproduced and real-time 3D scanning [62]. By employing an area-based tracking system, augmentations, such as stationary and animated characters and objects, can be created in an interior real-world environment such as a museum, office, or the floor of a factory. HoloLens is regarded as the most advanced HMD-based AR device that supports area-based tracking [27].

Narrative in AR

The use and study of AR as a narrative medium is not new. According to the interaction paradigm, early AR narratives can be classified into three types: point of view-based exploration (POV), space-based exploration, and ontological interaction [55]. POV-based exploration gives the user control at the first-person level. For instance, in the AR games Mad Tea Party [38] and Three Angry Men [35], users are able to delve into the story from different spacial points of view as seen through the eyes of a character. For space-based exploration, users can examine either small props or prominent landmarks in a real-world space [42, 64]. For example, in Inbox [3], users can explore items with AR markers in a shipping container with a hand-held device and hear the story of the container itself, as well as the shipping container industry. Ontological interaction refers to interaction capable of altering the plot or the world of the augmented narrative. Examples include the aforementioned Mad Tea Party [38] and AR/Façade [17], in which users can alter the story via speech and hand gestures.

Early HMD-based AR, like Mad Tea Party and AR/Façade, could only overlay a flat 2D animation or live-action film onto a real-world environment. As a result, not only were users unable to move freely within the room, but it was difficult to mesh the augmentation with the real world. Fortunately, as area-based tracking technology has advanced, particularly in the case of HoloLens, 3D holograms can now be projected onto a floor-scale real-world space seamlessly, without any markers. In other words, HoloLens presents new possibilities for 3D character augmentation in a room-scale real-world space for AR narratives. The HoloLens applications Arnold [14] and Fragments [59] are initial explorations of interactive narrative in HMD-based AR. Arnold is a linear story, in which the dog protagonist can run within a physical room, seek the audience, and play with them. Fragments is a room-scale mixed reality experience that shows four virtual characters walking and talking in a real-world room and unfolds the story through puzzle solving. Still, practice and research on narrative with HMD-based AR is insufficient. Existing studies of AR narrative have focused on non-interactive linear stories and fragmented narratives, while AR narratives with a classic branching structure have scarcely been investigated. As an HMD AR device with head tracking and speech recognition features, HoloLens can provide a completely hands-free experience. Therefore, it also brings new challenges to the user interface design.

User interface for narrative in AR

One of the vital tasks of interactive narrative in AR is creating an appropriate interface to facilitate interaction between the user and the mixed reality environment. The main types of interfaces used in AR applications are Graphical User Interfaces (GUI), Tangible User Interface (TUI), and Natural User Interfaces (NUI).

GUI in AR, also known as 3D User Interface, allows direct interaction with virtual objects via controllers or gestures [8]. For instance, in Fragments, users can select, grasp, and manipulate virtual items and buttons via air tap gestures, which is also the standard input for HoloLens. A TUI uses physical objects to create virtual entities and convey information, bridging the gap between the physical and digital worlds. Tangible interfaces support direct interaction with the real world by drawing upon real objects and tools [28]. A classic example of tangible user interfaces is the VOMAR application developed by Kato et al. [3], which enables a person to select and rearrange the furniture in an AR living room design using a real, handheld paddle. A NUI, which is considered a multimodal user interface in AR, enables direct system control via natural human actions such as talking or grasping using speech and hand recognition [19]. Training to control a NUI is not required since manipulation with a controller or a physical object like a mouse or keyboard is not required [45].

Several studies have been conducted on GUI [18, 49] and NUI [13, 20, 67] with HoloLens. The GUI studies have primarily focused on the evaluation of the interface, while a NUI study placed significance on the design and prototype of the interface. Though GUI is more accessible and technologically easier to develop, NUI shows more potential, particularly in a narrative context [67]; several studies have shown that NUI functions as an ambient interface in an interactive narrative [6]. Since NUI is supposed to ensure that users have seamless and direct control over the mixed reality environment without intermediate interfaces, users are expected to have less distraction and greater immersion in the story. In other words, NUI should lead to better presence and narrative engagement within an HMD-based AR narrative. However, since there have been no formal evaluations of NUI with HoloLens, whether NUI positively impacts the narrative in AR remains unclear.

Research aims and objectives

The following empirical research aims to assess different interaction interfaces for immersive AR narrative in terms of presence, the narrative experience, and interactivity. This empirical study based on an immersive AR narrative application was conducted to compare the quality of interactive narratives presented with GUI and NUI. The research questions are:

1. Does a NUI impact users’ presence differently in comparison with a GUI?

2. Does a NUI affect users’ narrative engagement and reflection differently in comparison with a GUI?

HoloLens, the foremost representative of the HMD-based AR technology, provides the suitable technology with which to investigate the above questions. The aforementioned interactive narrative applications were conducted on the HoloLens platform with identical content based on different user interfaces (GUI vs. NUI). A between-subject design experiment was performed to compare GUI and NUI with respect to presence, narrative engagement, and reflection. Standard scales and semi-structured interviews were employed for data collection and analysis. The objectives of this study were to conduct a comparative analysis driven by Questions 1 and 2, and afterwards formulate a set of guidelines for interactive AR narrative according to qualitative and quantitative results as well as our reflections on the design and development of this AR narrative.

Design of the AR narrative in HoloLens

We developed an AR narrative application with both a GUI and NUI version and used the two versions in the following experiment. The AR Journey is designed as an HMD-based, single-user app. Though HoloLens supports multi-user mode, the narrative design is single-user based. On the whole, visitors of NHCM are group visitors, so it is possible the other visitors may interfere with the narrative experience of The AR Journey. For instance, the holographic virtual characters might collide with the other users, bystanders may embarrass the participant, and so on. When the app is deployed on site, it is recommended to divide the group visitors by allowing patrons to experience the AR narrative individually, while the other visitors explore other exhibitions of NHCM to avoid potential interference. In this section, we explain the narrative context and the narrative design, including the narrative structure and storyline. We then explain the user interface design, including the interaction model used in our app and the detailed interface in both the GUI and the NUI. We then briefly explain our development procedure at the end of this section.

Narrative context

The NHCM has several conventional exhibitions presenting collections and scenes from the Holocaust. The primary interactive experience is a Q&A with Holocaust survivors, however, the survivors have now aged, and their rich experience may be lost. In an attempt to create novel ways of sharing these stories with younger generations, an exhibit, called The Journey, was created that follows the story of one Leo, a young Jewish boy, before he was sent to Kindertransport3. In order to immerse the audience, the exhibit was created to set the scene of Leo’s life by recreating his living room, classroom, and family tailor shop, all while Leo’s monologue can be seen on TV in the living room. Leo is a fictional character, but Leo’s story is a typical for Jewish children before World War II. To further improve the immersion level of the exhibition, an interactive narrative approach using AR technology could be further developed with Virtual Reality (VR) or Augmented Virtuality (AV) in order to enrich the experience of museum visitors and to effectively convey the story to the younger generation. With the assistance of HoloLens, it is now possible to make the story “come alive” by mixing the virtual holographic characters and furniture into a physical space seamlessly, allowing the audience to walk around and interact with items in the physical and virtual worlds. This AR narrative in HoloLens was entitled The AR Journey.

Design of narrative

To ensure historical accuracy, the script and dialogue were rewritten by using an existing script from a parallel project in the Virtual Journey app [11] in collaboration with a historian at the NHCM. The dialogues and plots were designed based on survivors’ previous testimonies and facts. The interactive narrative consisted of two parts, the story fragment exploration and the branching main story (Fig. 1). After finding the key item, Leo’s diary, the branching main story was activated. Through HoloLens, audience could see the vivid holographic characters of Leo and his parents talking, arguing, and reassuring one another, all while integrated with the physical space of the room (Fig. 2). The story begins with a family debate between Leo’s father and mother about whether they should migrate to other countries or defend their right to stay in Germany. Through the debate, the audience could learn about hardships that Jewish people in Germany faced before World War II and afterwards would be prompted to decide which side to support in the debate between Leo’s parents. Different decisions send the story onto different paths. Afterwards, Leo gave a monologue about the injustice he had suffered at school during this time, and questioned whether or not people were truly equal. Then, the doorbell rang and the Hitler Youth broke in and threatened Leo’s family. Leo then learned more about the Hitler Youth and the wave of Jewish unemployment by asking his father questions. The three branches in the story were: (1) choosing which side to support in Leo’s parents’ argument (2) Leo’s reflections on whether everyone is equal or not (3) the decision to ask about the Hitler Youth or Jewish unemployment.

Fig. 1.

Flow diagram of narrative structure

Fig. 2.

Layout of the props (Left), concept image for the AR journey in living room (Right)

The other part was story fragment exploration. Several items were available to be explored in the room, including the telephone, radio, suitcase, newspaper, Iron Cross and gramophone (Fig. 2). Through interactions with the above items, audiences could find more fragments of the story. For example, Leo’s paternal grandfather was awarded The Iron Cross for his bravery in the battle fought for Germany, the newspapers, titled Der Sturmer, included examples of derogatory stereotypes of Jewish people, and an anecdote about Leo’s father having an interest in listening to classical music. Through the fragments of daily life discoverable within the items in the room, audience could more fully understand the situation Jewish people faced in Germany near the beginning of the war.

Based on Koenitz and Chen’s narrative model [33], another critical factor is the type of Point of View (POV) employed in the immersive AR narrative. Since the audience’s physical world and the protagonist’s narrative world overlap in AR, the audience could act as one of the characters in the story (first-person view), a visible observer (second-person view), or an invisible observer (third-person view). Specifically with the second-person view, the virtual character can “see and interact” with the audience. Thus, audience members can more easily empathize with the characters while still playing themselves instead of a particular character in the story. Still, according to research, the first-person view enables more accurate interactions in VR [22]. Despite this, Larsen and Mads argued that the second person view, where the viewer is the protagonist’s sidekick, exhibits more advantages since it requires less background knowledge and more effectively involves the narrative [34]. In addition, as demonstrated in existing studies, the second-person view stimulates a greater sense of presence and participation than third-person view [46, 61]. Thus, the second-person view was adopted for The AR Journey. In short, the audience is able to investigate different story branches by making choices thorough a second-person POV, indicating that the audience is a visible observer, and protagonist Leo can “see and find” the audience question them.

In summary, the narrative experience of The AR Journey is the most interactive drama. Audiences can discover the story fragments by examining different items, triggering the main story through interaction with a key item, and exploring the branched main story by making decisions.

Design of interaction interface

There are two types of interactions in the narrative of The AR Journey: selecting and manipulating props to access the story fragments and answering Leo’s question to make a choice to enter different story branches. To satisfy the requirements of the interactions, GUI and NUI were designed and developed as two separate versions, bot entitled The AR Journey. For the GUI design, a head-gaze and commit model using the HoloLens4 clicker was adopted as the interaction model based on our pilot study [30]. For the NUI design, direct hand manipulation with real-world objects and natural language command with virtual holographic character were adopted as the interaction model (Fig. 3).

Fig. 3.

Available interaction methods in HoloLens (Orange line represents the GUI design, and blue line represents the NUI design)

With the GUI, the user could select the virtual holographic gramophone with a head-gaze interaction and perform the “play music” action via the clicker. This was followed by using head-gaze to activate the choice and confirm the choice via the clicker. With the NUI, the user would need to directly interact with a physical object, in this case by physically turning the crank handle of the gramophone, in order to play music from HoloLens, and could directly talk to Leo to help him make a decision (Fig. 4).

Fig. 4.

Screen capture of different interface (1 refers to decision point interface in GUI, 2 is prop interaction in GUI, 3 is decision point interface in NUI, 4 is prop interaction in NUI)

The NUI version is currently difficult to implement on a technically level. Image recognition with Vuforia5 on HoloLens is subject to severe issues on latency, and its speech recognition of the Chinese language remains underdeveloped. As a result, the Wizard of Oz experiment was employed for the NUI. The Wizard of Oz method here refers to the process allowing a user to interact with a physical object and talk to Leo without knowing that the responses are being generated by a human, with someone hidden behind-the-scenes to control the response via a Bluetooth keyboard, rather than a computer.

Development

The programming was developed in the Unity3D game engine. HoloToolKit acts as the HoloLens SDK for Unity3D. Asset development in this study comprised three characters, 25 min full-body character animation for each character, six props and their associated animations, UI elements, sound effects and music. The visual asset development complies with the same rule as script development concerning historical accuracy. The NHCM and the online United States Holocaust Memorial Museum provided the most reference.

Character models and animation are considered the most significant part of asset development in this project. The model part complies with the latest physically based rendering pipeline in the game industry. Full body animation was captured with Vicon Mocap System in the Mocap Lab at Tongji University, where the layout of the rooms in NHCM was restored to ensure that the holographic virtual character can dovetail the real-world space (Fig. 5). The facial expression capture was implemented with PrimeSense 1.0 (a depth camera) and Faceshift software (Fig. 6). Students from the acting department of the university performed facial motion capture to match the existing body animation.

Fig. 5.

Motion capture for scene 1(left), concept composition of mocap animation and living room (right)

Fig. 6.

Using faceshift to perform facial expression capture

For spatial sound, HoloLens headphones can achieve spatial sound via Head-Related Transfer Function (HRTF) technology, making the position and direction of an audio source recognizable in a real-world space. Professional voice actors and sound designers performed voice acting and sound effects. When the development was conducted, it was tested in the NHCM (Fig. 7).

Fig. 7.

Screen captures of The AR Journey in NHCM

Experiment

To answer the research questions systematically and reliably, an experiment was performed to compare the two mentioned interaction strategies. Since the salient feature of AR technology involve real-time interaction and presence, this experiment was developed to answer whether different interaction strategies affect the presence and user engagement in AR narrative experience.

Hypotheses

H1

NUI design was expected to deliver more presence for users than GUI design in AR narrative. Since NUI can enable the audience to interact in a more natural and intuitive way, and the intuitive interaction is critical to the feeling of presence both in AR and VR environments [7, 31] ,.

H2

NUI design was expected enable users to achieve more learning outcomes in terms of narrative engagement and reflection. As the purpose of the application was moral education via narrative, narrative engagement and reflection were good indicators of the learning. Engagement is a critical aspect of the learning process [32], drastically influencing a learner’s motivation to continue interacting with a system and the educational content [43]. NUI, which was hypothesised as the design with better presence, were anticipated to make audience more focused and engaged in the narrative and thereby more let them reflect more.

To guide interactive narrative designing, depth interviews were employed, and participants’ answers would be analysed, coded and summarised to extract the consensus and insight into sensory experience, interaction experience and narrative experience.

Measures

Three constructs were adopted to assess the experimental results from different perspectives, i.e., presence, narrative engagement and reflection.

Witmer and Singer’s Presence Questionnaire (PQ) (Cronbach a = 0.88) [65], aiming at virtual environments, was taken to measure presence, consisting of 20 Likert Scale affirmations with the answers ranging from 1 (totally disagree) to 7 (totally agree). It contained factors of control, sensory, distraction and realism. Two relevant subscales of PQ referred to Involvement/Control and natural.

The Narrative Engagement Scale [10] (Cronbach a > 0.8) was exploited to assess participants’ subjective experience of engaging with a narrative and enjoyment. 11 items were included from 4 dimensions of narrative, i.e., narrative understanding, attentional focus, emotional engagement, and narrative presence.

Since this AR narrative aims at moral development for young people, the educational gain in this case is the attitude and reflection, instead of knowledge. A scale was developed to assess the extent to which participants felt empathetic and connected to the protagonist and were provoked to consider meaningful subjects. The connectedness part was adapted from Bartsch’s emotional engagement with the characters scale [5], and the reflective thinking part was modified from a contemplativeness scale developed in existing studies [4, 5]. 10 scale items were selected in total and the wording of the scale items was changed to be relevant to AR experience rather than applying to film and television as they have been doing originally. For instance, the general statement ‘… since I identify with the characters’ outlook on life’ was adapted to ‘… I was able to put myself “in the shoes” of those depicted in the Hololens’. All measures complied with a high-reliability rating (Cronbach a > 0.8).

• I had a connection to the people presented in the AR experience;

• I understood how the people in the story were feeling;

• I was able to put myself ‘in the shoes’ of those depicted in the HoloLens;

• I was able to relate to those depicted in the story;

• I cared about what happened to those shown in the story;

• I was inspired to consider meaningful issues (e.g., equality, wars and race);

• I was inspired to gain novel insights;

• I thought about meaningful or relevant events in today’s society;

• I thought about myself in relation to others;

• I found the task thought-provoking.

The scales were translated into Chinese to cater to the participants.

Participants

As impacted by the current pandemic of COVID-19, the AR narrative app failed to be deployed at the NHCM. Accordingly, the authors decided to deploy it remotely in China for experiments. Eighty-one university students from Tongji University in Shanghai, P.R. China, were recruited and completed the pre-survey for the experiments. Since the intended audience for this narrative experience are young generations, university students can represent part of the generations. Lastly, 60 students from various majors (e.g., film making, computer science, design, engineering, journalism and telecommunications) were selected randomly regarding balance of gender, major and pre-existing 3D Role-Playing Video Game (RPG) experience. All the students volunteered to participate with a compensation of 10 dollars for each.

As revealed from existing studies, gamers and non-gamers may have a different perception of the virtual environment in terms of immersion, emotions of fear, adaptation to interaction methods [21, 53, 56]. As a result, 3D RPG experience acted as an independent variable to filter the potential influence of existing game experience. The participants were then divided into two groups, and the gender and pre-existing 3D RPG experience of both groups were balanced.

Two students quitted for equipment failure; both were from the group 2, i.e., natural interaction design group. Lastly, 58 participants completed the equipment successfully. Table 1 lists the information of their demographics and educational background.

Table 1.

Demographic data and character-based game experience of the participants

Gender		Age			3D RPG experience
male	female	min	max	avg.	Yes	No
29	29	19	29	22.07	29	29

Materials and apparatus

The natural interface and the virtual interface applications presented in Section 3 were exploited as test materials. Both applications implemented on Microsoft HoloLens 1st, offering an FOV of 35˚ and mass of 579 g, were adopted as the HMD equipment. For the NUI version, props were brought and then arranged in the room, which included a gramophone, a hand-crank telephone, a newspaper, an old radio, a suitcase and a diary book.

Procedure

The experiment was performed in the rehearsal room in the College of Arts and Media, Tongji University following the procedure described in Fig. 8. Participants fell to two groups randomly, keeping balance of the gender and RPG experience within each group. Participants were asked to enter the room one by one and go through the experimental procedure individually. Participants of group A experienced the 25-minutes narrative in HoloLens with a virtual interface design, while participants of group B experienced the same narrative in HoloLens with a natural interface (Fig. 9). After the AR narrative intervention, participants were required to complete the post-questionnaire and have an in-depth semi-structured interview.

Fig. 8.

Experimental procedure, N_RPG refers to participants with RPG experience before, N_none refers to participants without any RPG experience

Fig. 9.

Participant was taking part in the experiment in China (left); the hologram mixed into the real-world space participants can be observed with HoloLens (right)

Results

We used two-way ANOVA to quantitatively analyse the dependent variables, including the presence, the narrative engagement and the reflection. Then we further analysed our interview results on the sensory experience, the interaction experience, the narrative experience and the suggestions.

Quantitative results

Participants rated the statements on the questionnaires with seven-point Likert Scales with 7 indicating total agreement. The final score of a scale or subscale was the total of all the ratings in the scale. Shapiro-Wilk test and Levene’s test were performed to check normality and homogeneity of variance. As indicated from the result, presence, narrative engagement and reflection data were well-modelled by normal distribution and exhibited the identical variance. Moreover, the subscale control/involvement and natural data of presence were normally distributed on the Q-Q plot. Since there were two categorical independent variables in this study, two-way analysis of variance (two-way ANOVA) was conducted for the statistical test.

Presence

First, a 2 × 2 ANOVA was conducted to assess the presence effects of interaction strategy and pre-existing RPG experience. The analysis yielded several results: a statistically significant main effect of interaction strategy, F(1, 54) = 5.06, p = .029, η2 = 0.086, a statistically significant main effect of pre-existing RPG experience, F(1, 54) = 4.30, p = .043, η2 = 0.074, as well as a statistically significant interaction between interaction strategy and pre-existing RPG experience, F(1, 54) = 12.78, p = .001, η2 = 0.191. As suggested from the examination of Fig. 10, the NUI design invoked better presence for participants without pre-existing RPG experience (P_none) than virtual GUI design, whereas NUI reduced presence for participants with pre-existing RPG experience (P_RPG). P_none achieved a mean presence of 93.6 under GUI condition and a mean of 110.79 under NUI condition, representing an unstandardized difference of 17.19 and a Cohen’s d of 1.49, respectively. Such a difference was of statistical significance (p < .001). P_RPG had a mean presence of 110.27 in the GUI and a mean of 106.36 under NUI condition, indicating an unstandardized difference of 3.91 and a Cohen’s d of 0.33, respectively. This difference was of no statistical significance (p = .36). This study inferred that the NUI worked for P_none, instead of for P_RPG regarding presence. Besides, P_RPG gained a mean presence of 110.27 under NUI condition and P_none had a mean of 93.6 under the identical condition, representing an unstandardized difference of 16.67 and a Cohen’s d of 1.54, respectively. This difference was also of statistical significance (p < .001). A conclusion was drawn that the participants with pre-existing RPG experience could have better presence under GUI condition than those without such experience.

Fig. 10.

Interaction plot of interaction strategy and pre-existing RPG experience for presence

Further, 2 × 2 ANOVA was conducted to analyse the two subscales of presence, i.e., control/involvement(C/INV) and natural. Specific to control/involvement, the analysis yielded a statistically significant main effect of pre-existing RPG experience, F(1, 54) = 4.41, p = .041, η2 = 0.075, as well as a statistically significant interaction between interaction strategy and pre-existing RPG experience, F(1, 54) = 8.43, p = .005, η2 = 0.135. The main effect of interaction strategy was slight, p = .35. As indicated from the examination of Fig. 11, P_RPG achieved better sense of control/involvement in GUI than P_none, whereas it had less control/involvement feelings in NUI. P_RPG had a mean C/INV of 36.73 in the GUI condition, and P_none achieved a mean of 31.4 under the identical condition, representing an unstandardized difference of 5.33 and a Cohen’s d of 1.42. This difference was of statistical significance (p = .002). P_RPG had a mean C/INV of 34.64 under NUI condition, and P_none achieved a mean of 35.5 under the identical condition, representing an unstandardized difference of 0.86 and a Cohen’s d of 0.23. This difference was of no statistical significance (p = .56). A conclusion was drawn that the participants with pre-existing RPG experience gained more sense of control/involvement than those without the experience under GUI condition, instead of under NUI condition.

Fig. 11.

Interaction plot of interaction strategy and pre-existing RPG experience for feelings of control/involvement

Specific to natural feeling, the analysis yielded a statistically significant main effect of pre-existing RPG experience, F(1, 54) = 4.28, p = .043, η2 = 0.073, a statistically significant main effect of interaction strategy, F(1, 54) = 3.90, p = .05, η2 = 0.067, as well as a statistically significant interaction between interaction strategy and pre-existing RPG experience, F(1, 54) = 8.48, p = .005, η2 = 0.136. As indicated from the examination of Fig. 12, P_none had better natural interaction feelings under NUI condition, whereas P_RPG had better natural interaction feelings under GUI condition. P_none had a mean natural feeling of 7.8 under GUI condition and a mean of 10.29 under NUI condition, representing an unstandardized difference of 2.49 and a Cohen’s d of 1.34. This difference was of statistical significance (p =.005). Besides, P_RPG gained a mean natural feeling of 10.33 under NUI condition, and P_none had a mean of 7.8 under the identical condition, representing an unstandardized difference of 2.53 and a Cohen’s d of 1.55. This difference was also of statistical significance (p = .002). Thus, this study concluded that the participants without pre-existing RPG experience had better natural interaction feelings under NUI condition than GUI, and participants with pre-existing RPG experience had better natural interaction feelings in GUI than NUI.

Fig. 12.

Interaction plot of interaction strategy and pre-existing RPG experience for feelings of natural interaction

Narrative engagement and reflection

Moreover, 2 × 2 ANOVA was conducted to assess the narrative engagement and reflection of interaction strategy and pre-existing RPG experience. For narrative engagement, the analysis yielded a statistically significant interaction between interaction strategy and pre-existing RPG experience, F(1, 54) = 6.87, p = .011, η2 = 0.113. The main effect of both independent variables was slight, p > .05. As indicated from the examination of Fig. 13, the NUI made the narrative more engaging than GUI for participants without pre-existing PRG experience, whereas participants with the experience achieved more narrative engagement under GUI condition than those under NUI condition. P_RPG had a mean narrative engagement of 65.93 under NUI condition, and P_none had a mean of 58.8 under the identical condition, representing an unstandardized difference of 7.13 and a Cohen’s d of 0.93. This difference was statistically significant (p = .027). A conclusion was drawn that the participants with pre-existing RPG experience could have better narrative engagement under GUI condition than those without such experience.

Fig. 13.

Interaction plot of interaction strategy and pre-existing RPG experience for narrative engagement

Specific to reflection, the analysis also yielded a statistically significant interaction between interaction strategy and pre-existing RPG experience, F(1, 54) = 5.17, p = .027, η2 = 0.087. The main effect of both independent variables was slight, p > .05. As suggested from the examination of Fig. 14, the NUI generated the more reflection than GUI for participants without pre-existing PRG experience; as opposed to such a result, participants with the experience had more reflection under GUI condition than those under NUI condition. However, after the comparison of the mean reflection of four sub-groups, no statistically significant difference was identified (p > .05). Accordingly, the conclusion of the relationship between reflection, interaction strategy and pre-existing RPG experience could not be drawn.

Fig. 14.

Interaction plot of interaction strategy and pre-existing RPG experience for reflection

Interview results

In this section, a descriptive analysis of the interview results is presented, which is also expected to underpin the later discussion of design guidelines for AR narrative. Interview questions covering three aspects, i.e., sensory experience, interaction experience and narrative experience, are elucidated below.

• Q1: Was the overall sensory experience acceptable during this activity?

• Q2: What do you like or dislike the sensory experience? Please describe in detail.

• Q3: Are the interaction methods easy to use?

• Q4: What do you like or dislike the interaction experience? Please give details.

• Q5: Are you curious to explore the interactive story?

• Q6: Please describe the narrative experience you like and those you dislike.

• Q7: Did AR keep you engaged in the narrative experience? Will you recommend it to your friends?

• Q8: Do you have any other suggestions for improving the interactive AR narrative?

As Q1, Q3, Q5 and Q7 were generally yes or no questions, the interview results could be coded according to participants’ choices. Figure 15 illustrates participants’ coded responses.

Fig. 15.

Results for experience of sensation, interaction, interactive narrative design and overall feelings

A rough consensus was reached among participants that the AR narrative experience was relatively positive for sensation, interaction, interactive narrative design and overall satisfaction. 48 participants (82.8 %) agreed that the overall sensory experience was acceptable, 52 (89.7 %) considered the interaction methods were easy to use, 50 (86.2 %) reported that they were curious about other branches of the story, and 51 (87.9 %) felt that they were engaged in the experience, and would recommend it to their friends (Fig. 15). In fact, three participants did recommend it to their friends, who later contacted the authors and requested to participate in the experiment.

However, the in-depth data show that only 11 participants (19 %) agreed the sensory experience was attractive, 8 (13.8 %) considered the interaction methods to be natural and interesting, 17 (29.3 %) generated high interest in exploring other branches of the story, and 15 (25.9 %) would highly recommend the AR narrative for both the new medium and content while 36 (62.1 %) recommended it mainly for experiencing HoloLens (Fig. 15). As revealed from the mentioned finding, the experience of AR narrative in HoloLens was acceptable but not satisfactory.

The remaining four interview questions (Q2,Q4,Q6,Q8) aimed to uncover the problems and merit of the AR narrative, forming a guideline for future designers and discovering the possible reasons behind the previous findings. To analyse the responses, we applied a two-cycle, simultaneous coding procedure. The first cycle established an initial, in vivo coding. The second cycle used a holistic pattern coding method to provide both a summary of the content and a point of reference for the discussion of specific quotes [51].

Sensory experience

In Q2, the participants were asked to describe the positive and negative sensory experience during the experiment. The second cycle codes and results were analysed according to group type and RPG experience. There was no finding in group type, and Fig. 16 presents the analysis based on RPG experience.

Fig. 16.

Codes of Question 2. Horizontal axis shows percentage of participants who mentioned these in their responses to the total number of all participants

The Code 1 rate of 72.42 % indicated that 42 participants reported HoloLens were too heavy or uncomfortable to wear, and 32 (55.17 %) reported Code 2, indicating that the FOV was to narrow. Figure 16 also indicates that participants with RPG experience had less trouble to wear HoloLens and less tolerance for narrow FOV issue than those without such experience. Some comments are presented below:

Participant # 23: I couldn’t wear HoloLens in a comfortable way, I had to hold it by my hand during the experiment.

Participant # 14: The FOV was far too narrow. I could only see Leo’s head when he was in front of me. I had to put my head down to see Leo’s legs.

The statements for Code 3 (17.24 % of participants) indicated that the proper visual cue design could potentially alleviate the issue of narrow FOV.

For Code 4, Code 5 and Code 6, 28 participants (48.28 %) noticed the Computer Generated (CG) virtual objects did not match the physical world perfectly, and over the half (51.72 %) reported the virtual CG character were artificial and unnatural. 11 participants (18.97 %) considered the CG characters to be natural and alive, and 8 of them had no RPG experience. Code 7 further demonstrated that 12 participants (20.69 %) felt the voice acting or sound FX were disturbing, and 11 (18.97 %) highlighted that the CG props and furniture were arranged or looked strange (Code 8). Figure 16 suggests that participants with RPG experience were more sensitive and have higher expectation for the authenticity of CG characters and the mixture of virtual and physical realities. Examples of comments include:

Participant # 22: The virtual objects and characters were too bright which did not match the light in the physical room.

Participant # 18: The model of the character was not satisfactory. The stiff expression distracted me most.

Participant # 6: The number of the props and furniture was too little. It did not look like a real living room.

Though many participants reported the unnatural feelings of the CG content, Code 10 revealed that over the half participants (55.17 %) claimed the unnatural CG content only slightly affected them during the experiment. Only 7 participants (12 %) agreed that the unnatural CG content had a clearly negative impact (Code 11), and 6 of them had RPG experience. This is consistent with our finding that participants with RPG experience tend to have higher expectation for CG content in AR. Some comments are presented below:

Participant # 6: It affected me slightly, as my focus was on the plot.

Participant # 58: It clearly affected me, because my attention was always distracted by the unnatural characters.

Lastly, Code 9 indicated that 24 participants (41.38 %) reported feeling immersive.

Participant # 52: It is hard to describe, but I had an urge to approach the characters to watch. It made me feel immersive.

Participant # 14: Compared with movie, the experience was clearly more immersive while I am just a bystander when I watch movies.

In summary, the analysis for code of questions 2 inferred that the main issues of sensation aspect were HoloLens related, e.g., weight, narrow FOV, and the mismatch between virtual and physical realities. Over the half of participants noticed the unnatural characters. However, majority of them claimed the artificialness only slightly affected them. Moreover, participants with RPG experience tend to be more fastidious about the details of CG characters. Less than half of the participants (41.38 %) had positive impression of immersion.

Interaction experience

Q4 asked user interaction experience. Through a similar vivo coding procedure in Q2, we investigated the detailed process of the interaction. The second cycle codes and results were analysed by complying with the group types (Fig. 17).

Fig. 17.

Codes of Question 4. Horizontal axis shows percentage of participants who mentioned these in their responses to the number of participants in their own group

Code 1 and Code 2 indicated a mixed result for interaction method of GUI. Code 1 demonstrated that 14 participants (50 %) reported that interaction methods was natural, funny and novel in group B (NUI), and 10 participants were fond of interacting with physical object while 5 participants preferred talking to virtual characters. However, Code 2 showed that 9 participants stated that they either felt awkward to talk to Leo or uncomfortable to interact with physical objects, and 8 participants denied the design of talking to virtual characters while 3 participants rejected interacting with physical objects. As revealed from the mentioned finding, NUI could act as a controversial interaction method, probably causing mixed attitude of users. It is noteworthy that NUI was criticized mainly for natural language design to talk to virtual characters, and interaction through physical object was more welcome other than disapproving. It is also noteworthy that participants in group A (GUI) neither admired nor complained about the interaction methods. Some statements are detailed below:

Participant # 52: “…I like the experience of talking with Leo, besides, it was also funny and fresh to interact with physical objects …”.

Participant # 55: “…It is a very engaging methods, especially for opening the suitcase, picking up the telephone…”.

Participant # 40: “…I felt distracted to talk with Leo, as you could feel that you were not really talking but answering multiple choice questions…”.

For Code 3 and Code 4, 8 participants (26.7 %) of group A and 2 (7.1 %) of group B reported Code 3, indicating that the interaction feedback was interesting; 6 participants (20 %) of group A and 6 of group B (21.4 %) reported Code 4, suggesting that the interaction feedback was either slow or tedious. Some comments are presented below:

Participant # 12: “…I liked to explore the props. The unfolding animation of newspaper was amazing…”.

Participant # 42: “…It is interesting that the physical objects are capable of generating sound and music…”.

Participant # 37: “…Leo’s response was too slow…”.

Participant # 56: “…The props were extremely simple. Secondary interaction design should be considered and applied in this case. For instance, let audience open the drawer to find the diary, other than putting the diary directly on the ground…”.

Next, only participants (26.7 %) of group A reported Code 5, indicating that the number of interactable objects was small and insufficient. Some statements are detailed below:

Participant # 57: “…I would like to interact with more objects…”.

For Code 6, as stated by 9 participants (28.6 %) of group B and 3 (10 %) of group A, they wanted more visual or audio cues for interactable objects. As revealed from the mentioned finding, visual or audio cues were important and necessary for NUI. Some comments are presented below:

Participant # 48: “…the visual hint was just a line of text floating on the prop. It was too simple…”.

Lastly, 4 participants (13.3 %) of group A hoped to interact with physical object or directly use one’s hand (Code 7), and 6 participants (21 %) of group B wished to interact with virtual objects (Code 8). Some remarks included:

Participant # 37: “…I assumed the virtual cabinet could be opened to find a hidden clue…”.

Participant # 18: “…If I could interact with physical objects, the experience could be even better…”.

Then an analysis based on RPG experience are shown in Fig. 18. Code 2 revealed participants with RPG experience (20.7 %) tended to resist NUI more than those without such experience (10.3 %). As indicated from Code 4, participants with RPG experience (34.5 %) were obviously keener for slow and tedious feedback than those without such experience (6.9 %). Besides, according to Code 5 and Code 7, the number of participants without RPG experience request for more interaction hints and more interactive items were three times those such experience. Moreover, participants without RPG experience generated more interest in a mixture of virtuality interaction and physical object interaction than those such experience.

Fig. 18.

Codes of Question 4. Horizontal axis shows percentage of participants who mentioned these in their responses to the number of participants in their own group

In brief, current GUI can be concluded as a reliable interaction method not causing either unpleasant experience or enjoyable emotion. Besides, the number of interactable objects may impact participants’ experience, especially under GUI condition (Code 5). In addition, NUI is a controversial interaction method that frustrates one third of the participants for its natural language interaction with a virtual character, whereas it is also praised by half of the participants for its natural and novel experience. Moreover, participants with RPG experience were inclined to oppose NUI more. Besides, good visual or audio cue design might be especially important for NUI and participants without RPG experience (Code 6). Slow interaction response should be avoided, and a range of interaction feedback should be designed (e.g., animation, audio and characters’ actions), especially for participants with RPG experience. Lastly, though small number of participants requested, a mixed interaction experience inquiry that users can interact with both virtuality and reality should not be neglected, especially for those without RPG experience.

Narrative experience

In Q6, the participants were asked to describe the positive and negative narrative experience during the experiment. The second cycle codes and results were analysed by complying with group type and RPG experience, and no finding was achieved. Thus, the codes results were presented as a whole instead of into groups (Fig. 19).

Fig. 19.

Codes of Question 6. Horizontal axis shows percentage of participants who mentioned these in their responses to the total number of all participants

50 participants (86.2 %) reported that they generated interest in branched interactive narrative (Code 1), and 10 (17.2 %) mentioned that they were fond of the first branch since it was a dilemma to support Leo’s mother or father (Code 2), e.g. Participant # 22: “…especially the disagreement between mum and dad. It asked you to make a decision. I was gaining insights at that moment…”. Besides, only 8 participants (13.8 %) stated that the branched story was not attractive, e.g. Participant # 11: “…I was not curious. No conflict was suggested between different choices for me…”. This finding basically showed that branched narrative design was conducive, and dilemma choice was potential effective regarding branches designing.

Moreover, 35 participants (60.3 %) stated that they remembered some pieces of the story (Code 4), and 21 (36.2 %) remembered the dispute between mom and dad, e.g. Participant # 22: “…I could basically recall all the plot. The quarrelling between mom and dad was more clearly remembered by me…”. As indicated from this finding, this AR interactive narrative was relatively positive for storytelling.

Furthermore, as demonstrated by Code 5, 36 participants (62.1 %) had a sense of communicating with Leo when Leo was walking towards and looking at them, e.g. Participant # 22: “… As Leo looked at me and he could find and walk towards me, I gained a strong sense that Leo was communicating with me…”. As opposed to the mentioned, Code 6 presented that 12 participants (20.7 %) did not have a feeling of communicating with Leo, e.g. Participant # 21: “…Leo’s response was slow. I also tried to move a step, Leo did not follow me but stood in place…”. As implied from this finding, eye contact and finding behaviour of virtual characters were generally valid to improve virtual character’s presence.

Next, Code 7 and Code 8 suggested the merit of AR interactive narrative. 14 participants (24.1 %) expressed that exploration in the space was attractive, e.g. Participant # 26: “…I thought it was fascinating to explore the props and find clues…”, and 17 participants (29.3 %) mentioned that walking in the scene and the freedom of focus were helpful to improve engagement and immersion, e.g. Participant # 17: “…Since I could walk around and interact with Leo, I got the feeling that the story was happening just in front of me…”.

Lastly, Code 9 and Code 10 implied the defect of the narrative in this study. 14 participants (24.1 %) claimed that the plot was weak and flat, and 8 participants (13.8 %) complained about the interminable dialogue, which made them absent-minded sometimes.

In summary, branched narrative design was attractive, and eye contact and finding behaviour of virtual characters were proved as effective design strategies to generate certain character presence.

The narrative experience was positive regarding short-term memory of the plot. As the evidence was insufficiently strong, exploration, walking around in the scene and the freedom of focus might lead to positive feelings of the participants. Moreover, flat plot and long dialogue could cause negative feelings of the participants.

User suggestions and guidelines for AR

The first part of this section was the summary of users’ suggestions from Q8, and the rest of this section was our proposed guidelines from qualitative feedback of the survey and interviews, as well as our reflections on the design and development of this AR narrative.

Q8 asked additional suggestions for improving the AR experience. The responses were analysed and repeated patterns were extracted and listed below.

• Virtual character could call participants by their names so that participants could feel more connected to the virtual character.

• Virtual character could assign actions for participants to make them feel more involved (e.g., follow me, please help me to pick an item and give it to me).

• To make participants more engaged into the story, let participants play a role in the narrative and sit next to virtual characters to talk.

• Place participants into a dilemma at the branches of the narrative.

• Design internal links between different branches of the narrative to make participants believe the branches are dependent.

• Put more virtual or real-world furniture or decorations into the environment to make participants more immersive.

• Virtual characters should be designed with actions and performance, other than simply talking.

• Design the story with more dramatic conflicts to build the tension.

Combing the above suggestions of the participants and conclusions of Sections 5.2.1, 5.2.2, 5.2.3, as well as our reflections on the design and development of this AR narrative, the three authors had a group discussion about guidelines and reached a consensus. The guidelines for narrative in HMD-based AR were to propose below:

1. Branched narrative design can feasibly make the storytelling more attractive. Over 80 % of participants showed curiosity to see what would happen in other branches. Notably, it is adequate to let the audience face a dilemma when making decisions at a branch. Approximately 20 % of participants were particularly impressed by the first branch since they struggled to decide to support Leo’s mother or father.

2. Based on eye contact and face-to-face talk with the audience, virtual characters can be more visually appealing. The virtual character should be made intelligent when required to talk to the audience, so they can seek, walk towards, talk to and gaze at the audience in real-time.

3. Gaze and commit with a clicker is a reliable interaction method for graphical interface in HMD-based AR. Compared with direct hand manipulation or hand gestures, this method is easy to learn. All participants using this method in the experiment didn’t report any problems or negative feedback, but they also didn’t show surprise or praise either.

4. Tangible interface design for HMD-based AR can achieve positive feedback, and some audiences may report novel and funny feelings. However, tangible interface design also requires more visual or audio cues and hints for interaction; otherwise, the audience may feel confused. For example, it’s better to highlight the gramophone crank with a rotating icon floating above, rather than just floating the text “Please turn on the gramophone”.

5. Natural language interface for talking to virtual character in HMD-based AR should be employed rigorously. Audience could feel embarrassed to talk loudly in a public space like a museum and may feel strange to speak to a virtual hologram character as a real person.

6. Slow interaction response should be avoided. Virtual characters are required to response quickly to the audience’s action or words as a real person. Numerous participants noticed that virtual character would delay one second before his response.

7. Visual or audio cue design can be conducive to alleviating negative feelings of narrow FOV. The on-screen arrow or spatial sound can help guide the attention of the audience when they lost their target on the small screen.

8. Interaction feedback should be diverse (e.g., animation, audio and character’s actions). For instance, when a virtual newspaper is triggered, the newspaper is capable of unfolding itself with an offscreen voice introduction, and then the protagonist walks to and stops in front of the newspaper, describing the influence of this newspaper on his life.

9. Designers may consider not to put emphasis on refining on the appearance of CG characters. Producing a realistic CG character is difficult and time-consuming, and the uncanny valley effect could make the user experience even worse if the CG character looks close to a real person but still fails to be realistic. On the other hand, the majority of users claimed artificialness of CG characters only slightly affected them in an HMD-based AR narrative. In our study, most participants stated that character animation was more important than model, texture and material.

10. Designers may consider place stress on whether target users have 3D RPG experience or not. Users with 3D RPG experience are inclined to be more fastidious about the authenticity of CG characters, response speed of interaction and diversity of interaction feedback. Users without RPG experience need more guidance or visual/audio cues for interaction as opposed to the mentioned.

Discussion

One of the goals of this study is to identify how different interaction designs in AR narrative affect the presence and narrative engagement of the audience. Subsequently we assessed the quantitative results and the finding showed that AR narrative with NUI led more presence for audience without RPG experience, other than audience with RPG experience. The mentioned finding complies with existing studies[9, 54] and the qualitative results that half of the participants hold positive attitude towards NUI. One possible explanation could be that the tangible objects of NUI can act as a shared items between the audience and protagonist and blur the boundary between the audience and the narrative world [23] .

However, an unexpected result was identified that audience with RPG experience showed less presence in NUI design than those in GUI design. Since RPG primarily used GUI for interaction, and few games used NUI, audience with RPG experience might have a stronger preference for GUI inherently. Moreover, quantitative results further evidenced that audience with RPG experience had better experience in GUI than those without such experience in terms of presence. Moreover, as indicated from qualitative result, participants with RPG experience were more significantly inclined to have fixed responses and negative feelings of talking to virtual character, which is an unusual design for most RPG. One research revealed a similar finding that people who had the one type interaction experience (desktop) before performing the identical task with a new interaction system (VR) worse than novice without any experience [44].

Another goal of this study was to identify the design guidance for AR narrative. The findings from qualitative are listed in Section 5.2.4. The interview result suggested branching narrative design was effective, and this finding is in accord with the previous studies[39, 40]. As this study initially investigated interactive branching narrative in AR, and the previous conclusion was expanded to interactive narrative in AR. Besides, virtual character’s eye contact with audience and finding behaviour for audience were proved valid to optimize the narrative experience. Furthermore, a similar conclusion was drawn from other researchers[36]. This study also summarised several useful suggestions from the in-depth interviews in Section 5.2.4.

Among the mentioned suggestions, we found that majority of users claimed the artificialness of CG characters had little negative influence on them which was different from previous research [50]. In addition, we found possible explanations in cognitive theory of multimedia learning proposed by Mayer. Verbal information and visual information were processed in two separate channels, each of which had limited capacity for information processing, and people would allocate and adjust the cognitive resources, as suggested from Mayer [37]. Since the AR narrative applied here was dialog-based and narrative-centred, audience had to allocate more cognitive recourse for verbal information processing. Besides, as the AR experience of HoloLens offered a rich sensory visual mixed with a physical space, audience would be subject to more cognitive load than a usual movie on a flat monitor. Thus, audience were inclined to have less sensitivity of CG characters’ artificialness. Indeed, several participants asserted they became relatively tolerant for the unnatural characters either as impacted by the HoloLens’s novel experience or the concentration on the story. Given the mentioned finding, a conclusion was drawn that excess attention on authenticity of CG characters is unnecessary for such an AR narrative.

Furthermore, slow response of virtual character was reported by about one third of the participants, and participants of the virtual GUI group mentioned that the number of interactable items was inadequate. This finding confirmed Steuer’s research on interactivity influencing presence[58]. Steuer highlighted that response speed and range, which refers to the number of possibilities to alter or interact with, were key variables of interactivity. In the case of this study, the slow response of virtual character could be even more harmful, as it could broke the suspension of disbelief. To sum up, speed and range issues of interactivity should be considered for an AR narrative.

Last but not the least, as HoloLens is a representative of HMD-based AR device, the findings in this study is applicable to another HMD AR.

Limitation and conclusion

Overall, there were several limitations to this study. The NUI was not fully implemented, and instead a Wizard-Of-Oz approach was employed, which could have polarized the results. Since the number of participants just satisfied the minimum requirements, this could increase the risk of relevant defects or factors going undetected, while at the same time increasing the risk that the findings of this study are not fully representative of the participants’ experience. Given that many participants had experience with RPGs, subsequent research should include more participants to validate the findings of this study with users who have a wider range of RPG experience. Another limitation is the demography of the sample of participants used. The participants were all university students aged 19 to 29, whereas the target audience of the AR narrative in museums is typically younger. Therefore, future studies should include a wider age-range of participants that includes primary school students, middle-school students, and younger. Most importantly, controlled experiments should be conducted at the museum site using the fully implemented NUI to achieve a more reliable result. In addition, a Structural Equation Modelling (SEM) approach could be used to investigate the relationship between the quality of user interfaces, presence, enjoyment, and learning outcomes in AR narratives.

Hardware issues, such as the heaviness of the HMD, narrow FOV, and uncomfortable ergonomic design, are also a limitation of this study. Since AR technologies are improving at a rapid rate, further research should take into account hardware improvements in the near future. For instance, the recently released HoloLens 2 has a double sized FOV and is considerably lighter than the HoloLens 1.

This study has made several contributions to the theory and practice of interactive AR narrative. In our study, over 80 % of participants agreed that the overall experience of HMD-based AR narrative was acceptable in its sensory experience, interaction methods and narrative experience. Specific to sensory experience, the main issues are the HoloLens hardware issues and mismatch of virtuality and reality. As for interaction interface design, the NUI design has been proven to have better performance than virtual GUI design in presence and for users without RPG experience. Conversely, the virtual GUI design exhibits better performance than the NUI design in terms of presence and narrative engagement for users with RPG experience. Furthermore, for narrative experience, branching narrative design, as well as the virtual character’s eye contact and seeking behaviour, have been verified as an effective strategy to enhance engagement. The guidelines were summarised in compliance with the qualitative and quantitative results in Section 5.2.4.

In brief, the research and analysis here present an initial attempt to enhance the interactive narrative in immersive AR. Further explorations in this field to increase understanding and develop further guidelines will be even more fruitful in the future as the technology in the field continues to advance.