Storytelling and user experience in the cultural metaverse

Abstract

Enthusiasm for the metaverse is intensifying in academia and industries. The metaverse is a complex concept, combining many technologies to create many different types of user experiences (UX), depending on the intended use. The cultural metaverse was first introduced in this study.

This study is an initial attempt to fill the gaps in the practical research and storytelling research in the metaverse. Augmented reality (AR) technology is an applicative tool in cultural experience, which displays computer – generated virtual information on a real-world scene. AR displays digital information realistically, making it appear to be part of the actual environment, deepening or expanding the user's understanding of “reality”. This study constructed a cultural metaverse using the innovative AR storytelling. The cultural metaverse is a new cultural ecology in which advanced information technologies are deeply integrated with cultural spaces and exhibits. It combines digital technologies and cultural industries, mixing virtual space and physical space to facilitate the UX in cultural experiences. In this study, the existing AR e-book and the innovative AR version were compared while measuring multiple aspects of UX, including presence, flow, enjoyment, education, and engagement.

By analyzing questionnaire data from two groups with a total of 368 participants, the results indicated that the innovative AR storytelling produced a better UX across all variables compared to the AR e-book application. Overall, innovative AR storytelling allows visitors to transition between real and virtual spaces, enriches their interactive experience, and improves user engagement with the metaverse exhibition of cultural experience. Therefore, practitioners can construct a primary cultural metaverse through innovative AR storytelling.

Highlights

• •This study proposes cultural metaverse for the first time.
• •Cultural metaverse creates a new space between virtual and physical spaces.
• •AR storytelling improves user experience compared to the AR e-book application.

1. Introduction

The metaverse is currently one of the most talked about topics in the world. In 2021, the metaverse entered into users’ field of view again. The word “metaverse” originally came from the American writer, Neal Stephenson’s novel Snow Crash (1992). In the novel, the main character lives in a virtual world through digital virtual avatars. The virtual space is called the “metaverse” [1].

The metaverse does not yet have an accepted definition. As explained by Meta, the metaverse is the digital parallel universe that represents the evolution of social connection, which will allow us to socialize, learn, collaborate, and play in ways that surpass our imagination [2]. Some scholars argue that metaverse is an enduring virtual space where users can enjoy a variety of social, economic and leisure activities as an extension of their offline lives [3].

The metaverse is a complex web of integrated technologies, including 5G, block chain, AI, and immersive technologies [4]. The metaverse is immersive and has a strong sense of connection to the real – world objects, human lives and bodies [5]. Using diverse technologies, users can access and interact with immersive technologies, which are the foundation of the metaverse. Although there is no universally agreed definition of the metaverse. Much of the existing research has described visions of the future, no single example that can represent the metaverse has been provided so far. The application scenarios of the metaverse are still unclear. However, existing research and industry studies have reached a consensus that the metaverse can create new experiences in traditional industries as well as new business and marketing opportunities [6,7].

Microsoft put forward an industrial metaverse at the end of 2021 through its “industry + metaverse” model. Nishant Batra, the Chief strategy and technology officer at Nokia indicated that “The metaverse (of the future) is one where physical space, human and digital realities are conjoined” [8]. Instead of keeping the metaverse in our homes and offices, we may take it with us everywhere we go. This metaverse will be equally at home in the consumer, business, and industrial realms. Therefore, this paper proposes the concept of “cultural metaverse” for the first time. The cultural metaverse goes beyond the scope of the current digital museum and is the future form of the cultural industry in the cultural space.

Recently, Zuckerberg proposed a vision for the next stage in the digital age [9]. Although bold statements should be viewed with skepticism, augmented reality (AR) and the stimulation of the internal and external elements were the foundational axis in the metaverse [10]. Metaverse technology deals with maximum user interaction, which is supported by several emerging technologies such as immersive technologies, information and communication technologies, and human-computer interaction technologies [11]. The metaverse allows visitors to move freely between the virtual and real worlds and constitutes a hybrid world that enhances and expands visitors’ user experience (UX) [12]. To solve the aforementioned limitations concerning space, AR was chosen as the tool in the study. AR has the characteristic ability to link the real world to the virtual world and enable interaction between the two [13]. In the current study, AR was used in the cultural venue where augmented objects are used to provide visitors with a UX in a cultural metaverse.

It is unclear how the metaverse, as a cutting-edge manifestation, influences cultural experience in the cultural venues. This study attempts to design an experimental metaverse with immersive technologies in cultural experience in order to promote user engagement. In this study, the use of AR in virtual storytelling embedded cultural venues to facilitate stimulating visitor experiences. We found that these experiences ultimately facilitate visitors’ engagement. As a tool for transforming cultural knowledge into content that enhances visitors' experiences, immersive technologies fundamentally facilitate the formation of a cultural metaverse. AR can also be effectively implemented for virtual storytelling; for example, virtual scenes interacting with real scenes and realistic digital objects. As a visitors’ engagement strategy, cultural experience with virtual storytelling can facilitate the connection of participants to the content of the visit and thus promote engagement. The current research contributes to the literature on cultural metaverse by examining the impact of AR interactions on UX, which in turn facilitates the improvisation of UX and cultural metaverse.

2. Literature review

The development of immersive technologies has allowed users to interact with a content environment through unique technical features and vivid presentations, such as virtual reality (VR) and AR. VR and AR have been widely used. Existing research has revealed that the use of VR and AR in cultural institutions has economic, social, experiential, cognitive, cultural, historical, and educational value [14].

Both VR and AR are already being used for cultural experiences in specific scenarios, showing their respective advantages. VR is defined as the digitally constructed representation of a natural or artificial environment [[15], [16], [17]]. AR is a technology that links the real and virtual worlds [18,19], enabling real-time interactions by combining computer-generated objects with the natural environment [20].

The use of VR in cultural experience reduces the distance barrier between the visitors and the exhibits in museums [21], through advanced and realistic virtual simulation, thus providing the visitors with a unique virtual experience by creating infinite possibilities in metaverse museum exhibitions. VR also provides a completely immersive virtual environment (IVE) where visitors can be fully immersed [22], thus enhancing their experience by facilitating interaction with the museum’s exhibits [23,24]. VR has been heavily used to reconstruct historical environments, digital virtual museums, and galleries [25,26].

Some scholars emphasized the two sides of VR in cultural experiences. On the one hand, VR works as a tool to substitute for an in-person experience [27]; it offers the real-world exhibition space as a virtual space with detailed visuals for scenes in places that cannot be physically visited [28]. Thus, it enables visitors to feel as though they have visited the actual environments represented by the exhibitions [29]. On the other hand, VR makes sense only for virtual spaces that provide a limited form of experiential content.

Cultural venues are real spaces for displaying, appreciating, and learning about culture. Modern cultural venues are becoming integrated social learning spaces [13] by expanding into experiential spaces. AR has proved its potential to create an interactive and enjoyable cultural heritage experience [30]. Additionally, AR compensates for the limitations of VR—which changes or replaces one's real world—by augmenting the real world through information such as text, images, videos, among others. In cultural experiences, AR can facilitate visitors’ collaborative participation and active learning. For example, one study compared the impact of different VR and AR devices in an immersive gallery experience at the National Gallery in London, by examining UX, which included enjoyment, presence, cognitive, emotional, and behavioral engagement [31]. The study confirmed that equipment characteristics produced differences in the measured UX variables. Although in the study, VR and two different AR groups had high UX, Verhulst et al. (2021) mainly discussed the variation in immersion between devices and did not focus much on the other factors that could have affected UX [31]. Another study examined the impact of AR and VR experiences on museum services and users’ satisfaction [32] at the Ara Pacis Museum in Rome. AR and VR tools was explored to enhance multi-experiential value in cultural heritage by integrating traditional cultural content with other experiences, such as socialization, entertainment, and learning, allowing visitors to engage in a new type of value co-creation. However, the study was hypothetical and conducted via questionnaire, and there have been no practical applications of AR and VR.

Cultural experiences have evolved in the metaverse system in cultural venues with two limitations. First, a clear boundary between the virtual and real worlds has been developed. Second, storytelling has not been well utilized in cultural experiences; instead, information is typically narrated, which is a one-way method of delivery. Therefore, cultural space should attain the meaning from the real-world space, with storytelling bridging the gap between virtual and real spaces.

Multi-sensory stimulation, realism, interactivity, and immersion have been extended by AR storytelling. AR enables an augmented experience by using the camera function on smartphones or wearable devices. It exists as a hybrid space that combines the characteristics of real and virtual spaces to structure an overlapping space. AR is increasingly used in museums, galleries, and other cultural venues. Specific themes for special exhibitions are explored using AR storytelling to enhance cultural learning. Users can encounter unique stimuli that affect their UX while using AR because of the range of manipulation between the virtual and real worlds [33]. AR could replace VR as a more suitable digital tool to serve museums, where users can continue being connected to a real context in the physical world. However, there have been no controlled studies that compare differences in AR storytelling. The existing research has compared the impact of VR and AR technologies on UX but has not discussed and investigated the role of AR storytelling. The artifacts, functions, and storytelling produce an AR cultural experience in a hybrid space. For example, an exhibition can provide different story guides and reward mechanisms when visitors enter the museum. When visitors reach each step in the museum, they can get a hint of the next step; when all the goals are reached, the visitors can get the final reward. Visitors can choose their own tour maps and receive rewards.

3. Theoretical background and hypotheses development

3.1. Experience economy theory

The most important function of a cultural venue is to provide visitors with cultural experiences. A cultural product is defined by three factors: satisfying consumers’ intrinsic needs, eliciting good feelings, and engagement in the experience. Visitors in the cultural venues seek to create memorable images, arouse the senses, gain knowledge, and inspire thoughtful experiences [34]. Thus, the cultural metaverse can be thought of as a cultural experience product. Virtual tours, digital museums, and cultural experiences that integrate technology into physical space have all been attempted in the past. However, visitors now look for distinctive and unforgettable experiences and for high-quality experiential products and services.

Pine and Gilmore [35] delineated four dimensions of consumer experience, including education, entertainment, aesthetics, and escapism. These four characteristics are related to their level of engagement (passive versus active participation), as well as the connection that ties them to a certain performance or event (i.e., absorption and immersion).

Aesthetic experiences, in which users immerse themselves in a scene or event, are critical to creating a positive and memorable UX. In the experience economy, aesthetics are defined as consumers’ perception of their immediate glamorous physical environment [36] and their complete and enjoyable engagement in real-world AR experience. Researchers have discovered that the aesthetic motive precedes the other three motives in the setting of immersive experiences [37]. Since immersive cultural metaverse experiences must first pull in clients in order to supply them with immersive cultural experiences that encourage feelings of entertainment, education, and escapism, aesthetics is an essential component of these encounters. Thus, aesthetics are associated with immersion, as participants immerse themselves in the allure of events or performances [35]. A research of Hosany and Witham examined data from 500 visitors to heritage/cultural attractions and discovered that the aesthetic features of such tourist attractions are essential factors influencing revisit and public praise intentions. In other words, when people are drawn to an immersive, aesthetically beautiful experience, other incentives follow. Consequently, aesthetics is critical in driving the experience economy [36].

The most fundamental requirement for an experience is entertainment. The previous study looked into the potential of AR to generate entertainment value [38,39]. The need for amusement was the primary motivation for using immersive technologies, which has been demonstrated to boost the perceived enjoyment of UX. Meanwhile, entertainment encourages a deeper engagement in AR [40,41]. Offerings for high-quality entertainment provide tourists with unforgettable memories [42].

People look for entertainment to escape reality and the business or monotony of everyday life, to find thrilling and interesting activities, to relax and to enrich the mental world in ways that go beyond entertainment. Escapism is therefore what enables individuals to forget reality by immersing themselves in the experience. Freud [43] believed that “intrusive foreign bodies” could result in “diverting emotional impulse” (p.105-117). In that vein, escapism is a defense mechanism against reality, driven by the desire to maximize the enjoyable experience. People are actively engaged in the experience rather than just being onlookers. Escapism is likely satisfied by the cultural metaverse experience.

Education in a cultural context refers to the desire to learn something new and to participate actively. Participants must have a subjective enthusiasm for an event and believe that participating in this experience will improve their knowledge and abilities. Participants absorb what they want to know and learn from the experience, and the outcome of a communication experience depends on the participants’ attitudes. Compared to traditional communication, interaction and immersion lead to a higher level of engagement with digital content [44]. The desire to gain knowledge and skills as the primary educational motivation drives the participation in the experience [45].

The four dimensions of experience economy theory have already received widespread acceptance and validation from earlier investigations. This study, founded on experience economy theory, evaluates UX from the viewpoint of the cultural metaverse as an experienced good.

3.2. User experience in immersive environment

The metaverse is a two-dimensional to multi-dimensional extension of experiential communication [46], meant to appreciate, learn, entertain, and spread culture. By expanding into the space of experience, the modern technological scene is becoming an integrated socio-cultural communication space. The subject of experience is the “human” who enters the metaverse. Within the context of augmented technological, how is UX influenced?

Immersion has an impact on aesthetics and escapist experiences. The perception of being physically present in a virtual world is referred to as immersion, which is the central concept used to measure IVE. When people are completely immersed in an activity, they enter a state of complete isolation in the virtual world and have an immersive experience [47]. As technology advances and research expands, scientists are beginning to investigate the impact of UX in various contexts [[48], [49], [50]].

An existing study by Kim mentioned the immersion in AR as context immersion [51]; AR logically interacts with the user’s real physical environment. A framework of context immersion that combines three dimensions was structured in IVEs of AR. Time and location-based context, object-based context, and user-based context represent mobility, relationship, and communication or interaction, respectively.

Presence and flow are main variables to measure the concept of immersion UX in an IVE [52]. Here, presence is a psychological state related to the human response to immersion [53]. In IVE, presence is defined as experiencing the virtual objects as real in either sensory or non-sensory ways [54,55]. In IVEs of AR, augmented technology provides valuable contextual information. A combination space of both virtual objects and real physical environment is structured in a metaverse space. Presence reflects a specific feeling whereby users experience “being there” and physically located in the metaverse space. In IVEs of AR, users can physically interact with the objects and environment. A study confirmed that presence had positive impact on theme park visitors’ IVE [56]. Previous studies indicated that flow is also related to immersion and presence [57,58]. Flow is defined as the holistic sensation that people experience when they are engaged in action [59], which is something that most individuals have experienced, either through games, reading, watching movies, surfing the internet, visiting galleries, or even doing sports. The immersion in the metaverse is triggered and facilitated by stimuli in the IVEs of AR. AR E-book has been widely used in cultural experience, which is a mobile device application of AR as a one-way method of delivery. Users allows to experience visual and audible information on museum exhibits. AR technology can enhance UX through multi-sensory augmentation. An innovative AR that uses AR storytelling enabling two-way delivery, should boost more immersion and interaction than the existing E-book AR in IVEs complex, thereby leading to better UX.

Shernoff suggested that engagement may enhance through immersive technology [60]. Engagement is defined as the interaction between a person and an environment, and it includes participation, focus, and persistence within a task [61].

Enjoyment or entertainment is an effect of media. A sense of enjoyment comes from the continuous use of cultural and media sources. Previous studies have explored the value of immersive technologies to enhance UX [[62], [63], [64]]. For example, cultural venues support visitors seeking an enjoyable and entertaining experience in addition to information and learning [13]. In one study, people who watched 360-degree news videos in VR had a higher presence and enjoyment, and accepted the credibility of the news [65,66]. Similarly, in a study of shoppers, AR positively influenced flow, enjoyment, and other shopping experience outcomes [67].

A recent and comprehensive approach reconsidered the UX model by identifying the need to incorporate hedonistic and enjoyment attributes to explain how the attractiveness and aesthetics of technologies impact visitors [68]. Perceived enjoyment is one of the advantages of engaging with content via immersive technology. When compared to traditional cultural experiences, IVEs provide a higher level of enjoyment. This is supported by evidence that suggests interactivity is a key indicator of enjoyment [69]. In this study, virtual storytelling will be incorporated to improve interactivity. The enjoyment gained from interacting with AR technology has a positive impact on engagement. Thus, users who interact with AR content should have a higher level of enjoyment, which should lead to increased consumer engagement.

Therefore, the special exhibition of Shanghai Museum of Glass was chosen in this study. The theme was “It is the fate of glass to break.” Taking this well-known proverb as its starting point, it explores how things shattered, fractured, and cracked can speak to heartbreak and hope, and everything in between. With glass as their core medium - a material whose inherent fragility renders it a potent and recurring metaphor in countless languages - as well as music, 21 installations explore the connotations, implications, and possibilities of brokenness. The informational content of the museum has no interactive storytelling to structure a cultural metaverse. Instead, the museum only uses AR to convey the information of the exhibits. Thus, this background provided a suitable opportunity to test a more innovative AR technique.

Two AR applications were used to run an experiment. An AR e-book application was used for the control group, which was similar to techniques being used by the museum, which allowed visitors to scan a QR code scan to view a brief overview of the exhibits. The second application was an innovative AR storytelling application. Depending on the map selected by the visitors, new hints were subsequently sent to them. The hints prompted the visitors to find the answer for each step of the game based on their understanding and guesses. Finally, a reward was provided upon completion of the entire exhibition. In this process, information and the overview of the exhibits were obtained in a game-like manner. This study examined the effects of AR storytelling on engagement through possible psychological and UX elements: presence, flow, education, information utility, and enjoyment to build a cultural metaverse.

This study aims to fill two gaps in the existing research. First, it aims to contribute towards the lack of research on cultural metaverse by AR storytelling in cultural experience. Second, it aims to contribute towards the lack of practical research on the real-world study. The cultural metaverse is investigated through the effects of AR storytelling on UX in an augmented technology-supported cultural experience and metaverse environment.

3.3. Research hypotheses

This study focuses on designing a primary cultural metaverse by innovative AR storytelling that is different from the existing AR applications. Existing AR applications in cultural venues consist mostly of AR-books. Users can view exhibits’ audiovisual information in 3D [13]. E-book AR is a digital tour guidance, which has become increasingly popular in the museum and cultural heritage sector as it offers visitors a unique and engaging way to explore and interact with exhibits. It can notify users of content or autoplay audio through three basic method, tracking based on their physical location within the museum; inputting the code of exhibits; simply hitting the next button to see the multimedia content for each asset. While the basic functionalities of E-Book AR can provide useful information and enhance the visitors’ experiences, it do not extend the UX beyond enhanced 3D information and multimedia messages. In contrast, innovative AR storytelling makes cultural content more informative, emotional, memorable, and attractive [31]. Innovative AR storytelling allows visitors to actively participate in the experience by interacting with virtual objects, exploring hidden layers of information. In a cultural metaverse, visitors can access a hybrid space that combines the virtual and real worlds, allowing for an even more immersive experience. Innovative AR storytelling enables visitors to explore exhibits in a more interactive and engaging way, creating a deeper understanding and appreciation of the cultural content. It can be used to tell the stories of underrepresented communities and cultures, fostering empathy and understanding among visitors, allowing visitors to connect with the cultural content in a meaningful way.

Therefore, a cultural experience results from a co-created activity between the visitor and the provider. The goal of the experience is to provide value to the visitor by facilitating interaction between the visitor, content, and space [31,70]. Based on the review of the existing literature, this study proposed the following hypothesis.

H1

The innovative AR storytelling application provides greater presence (H1a), flow (H1b), education (H1c), information utility (H1d), enjoyment (H1e), and engagement (H1f) than the AR e-book application.

Drawing on previous findings, combining presence, flow, engagement, and enjoyment, creates an integral UX that facilitates interaction between the visitor and an immersive environment that is enjoyable, informative, and educational [67]. Visitors' needs are met by the cultural metaverse, which allows them to quickly absorb new knowledge, foster positive emotions, create amazing memories, and engage in cultural experiences. Users can participate in an experience either actively or passively. Specifically, users engage in experience through active participation when they have an impact on the experience, as opposed to passive participation, when they do not have an impact on the experience [37,57]. Users can either immerse themselves in the event or absorb it in terms of connectedness. Based on this, the following hypothesis was proposed.

H2

Presence (H2a), flow (H2b), education (H2c), information utility (H2d) and enjoyment (H2e) influence engagement.

4. Materials and methods

4.1. Study design

This is a polit study. The Shanghai Museum of Glass is a young museum that opened in 2011 with a glass theme. The AR applications for the museum were created using unity engine. One of the AR applications was an AR e-book, which allowed participants to experience the guide of the exhibits in 3D view.

The other AR application, the AR storytelling application, incorporated a storyline that facilitated two-way communication between the participants and the exhibits. Participants were prompted on their smartphones to find six AR exhibit presentation panels in the museum's real space, each containing icons that would help them, trigger the appearance of “crystal fairies” in the real space of the museum using AR tools. The icon corresponding to the exhibit area where the participant was located could be selected from the menu bar of the mobile app. Next, the smartphone screen presented a positioning box and overlaid it with the corresponding object in real space. Finally, it triggered a crystal fairy. Each completed part provided clues for locating the next exhibit. The button on the smartphone screen performed additional interactive functions such as sharing, screenshotting, and smashing, as shown in Fig. 1 (a - e). Both the AR e-book and the AR storytelling application were based on local network departures. This app with embedded story clues adds to the object recognition. Overall, the use of AR storytelling application to enhance the museum experience for visitors, making it more interactive and engaging. By combining AR technology with storytelling, participants can explore the exhibits in a new and exciting way, with added features such as sharing and screenshotting to encourage social media engagement.

Fig. 1.

(a–e). Screenshots of the main pages of the AR storytelling application.

4.2. Procedure

All visitors to Shanghai Museum of Glass were invited to participate in the AR cultural experience. However, participants could not choose if they wanted to be in the control group (i.e., using the AR e-book application) or the experimental group (i.e., using the AR storytelling application) because the different applications they were shown depended on when they visited the museum to eliminate bias in self-preference selection. They alternate between the AR e-book application and the AR storytelling application depending on the order of the visit. A staff member introduced the experiment to the participants before they entered the museum, and the participants voluntarily chose to participate by completing an anonymous questionnaire on our mobile devices at the end of their visit (Fig. 2).

Fig. 2.

Sketch map of the museum AR experience. The orange color dots represent the museum staff; the small blue dots represent clues; the blue triangles represent “cultural treasures”; the green rectangles indicate the introduction display area and photo area with the crystal fairies. (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)

The questionnaire was available in English and Chinese, and it did not request personal identifying information. No incentives or payments were provided to the participants in this study. The experiment was provided using 30 devices, which allowed for 30 participants to visit the museum simultaneously. During the entire experiment, the staff explained the purpose and procedures of the experiment to potential participants for approximately 10 min. The participants determined the amount of time spent visiting the museum, and the questionnaire, which took approximately 8–12 min to complete, was completed at the end of their visit (Fig. 3). This study did not include vulnerable minors or persons incapable of judgment, and informed consent to participate was obtained from all participants. The study was not ethically sensitive and conformed to the Ethics Committee’s rules and guidelines.

Fig. 3.

Study’s framework.

4.3. Participants

The participants were randomly selected from the museum visitors. A total of 368 individuals were randomly divided into two groups to complete the experiment. Both female and male participants were 184 by chance. This study included a wide range of people of various ages, which reflects the characteristics of the museum visitors. Participants were at least 18 years old, which complied with the ethics committee's recommendations. The majority of participants held a bachelor's degree or higher. In addition, participants were briefly introduced on the methodology, purpose of the experiment, and the willingness to participate in the AR experiment. Participants were reasonably new to museum immersive technologies, with 87% indicating that they had never or only used an AR application in a cultural venue once or twice before.

5. Measures

The questions were divided into two main sections: user demographic variables and UX of the six aspects, presence, flow, education, information utility, enjoyment and engagement. In the first part, respondents were asked to give some general demographic information, including gender, age and level of education. The second part, variables were measured by having participants rate their overall experience using an 11-point Likert scale (0–10) [71]. The questionnaire design incorporated questionnaires that have been used in existing studies and drew on existing research on the reliability and validity measures of UX questionnaires in IVEs [72].

The quality of the items and scales of the questionnaire was analyzed through reliability and validity to avoid the risk of misinterpreting the items’ meaning and providing a stereotyped answer. The R package “semTools” was used to test the Cronbach's alpha coefficient, the composite reliability (CR), and the average variance extracted (AVE) [73]. To examine collinearity, the variance inflation factor (VIF) was tested using the R package “car”. The t-test for independent samples and Levene's test for equality of variances were applied using the R package “car”. The regression model was statistically examined when it was discovered via ANOVA that the regression coefficients of the independent variables were not equal to 0.

5.1. Presence

The variables were assessed using multi-item scales adapted from those validated in previous studies. Presence was measured using questionnaires based on the MEC Spatial Presence Questionnaire and other existing studies [74,75]. Some changes were made to accommodate the AR cultural experience scenario and to keep the questionnaire as concise as possible. The resulting scale included the following items, all of which were measured on an 11-point Likert scale ranging from 0 (strongly disagree) to 10 (strongly agree): “I felt like I was not only in the museum's physical space but also in the virtual space,” “I had the feeling that I was in the middle of the action rather than merely observing,” “It was as though my true location had shifted into the environment of not only the physical museum but also in the task,” and “The boundaries between the physical and virtual spaces of the museum are increasingly blurred.” This scale had the highest reliability (Cronbach’s α = 0.946) among all scales used in this study.

5.2. Flow

We constructed a measure of flow using the items proposed by Bachen et al. [76]. Four items were included in the scale: “I was deeply engrossed,” “I was absorbed in the experience,” “I felt that I was trying my best to understand the information,” and “I felt as though I was moving through the task.” The scale had good reliability (Cronbach’s α = 0.864).

5.3. Education

For the education scale, the items were taken from two existing studies [77,78], and included the following four items: “I learned something from the experience that I did not know before,” “The experience made me more knowledgeable,” “It stimulated my curiosity to learn new things,” and “It was a real learning experience.” These four items had comprehensive reliability (Cronbach’s α = 0.848).

5.4. Information utility

Information utility was measured using items based on a study that explored the influence of processes on information technology acceptance [79]. Four items were adopted to measure information utility in the context of AR application: “The AR provided was informative,” “The information provided in AR was helpful,” “The information provided in AR was valuable,” and “The information provided during the AR application was persuasive.” The reliability of these four items was comprehensive (Cronbach’s α = 0.826).

5.5. Enjoyment

Enjoyment was assessed using questions from scales originally proposed by Schmitt and Kleemans et al. [80,81]. The item descriptions were modified to make them suitable for the study of experimental environments. The three items adapted to the cultural experience were as follows: “I found the exhibition attractive,” “I was so involved that I lost track of time,” and “I would like to have this viewing experience again.” The reliability of the enjoyment scale was high (Cronbach’s α = 0.899).

5.6. Engagement

To explore a reliable measure of engagement, items were included existing research were used [14,82]. The three items were as follows: “The experience has motivated me to find out more about the information of this cultural experience.” “The experience has motivated me to find out more about this area of the exhibition”; “The experience has motivated me to participate in this area of the exhibition.” The reliability of the engagement scale was comprehensive (Cronbach’s α = 0.855).

6. Results

6.1. Randomization check and preliminary analyses

Because the impact of technology and material stimuli on users’ cognitive and emotional responses can be shaped by individual differences [83], differences in participant characteristics in the two experimental groups were investigated. Gender, age, and education were tested for significance, and most participant characteristics did not differ significantly between the two groups, as shown in Table 1 (p > .05). These results indicated that the randomization was successful.

Table 1.

Participants’ characteristics.

Characteristics	Conditions				Significance Test
	Group1		Group2
	Frequency	Percent	Frequency	Percent
Gender					x2(1) = .011, p = .917
Female	85	49.4	99	50.5
Male	87	50.6	97	49.5
Age					x2(5) = .863, p = .973
18–20	8	4.7	12	6.1
20–25	48	27.9	57	29.1
26–30	69	40.1	74	37.8
31–35	28	16.3	33	16.8
36–40	11	6.4	11	5.6
>40	8	4.7	9	4.6
Education					x2(3) = 2.533, p = .469
No Formal Qualification	24	14	27	13.8
Undergraduate Degree	127	73.8	135	68.9
Postgraduate Degree	14	8.1	26	13.3
Doctoral Degree	7	4.1	8	4.1
Total	172		196		368

Notes. Group1: AR e-book; Group 2: Innovative AR storytelling.

A portion of the tested samples was analyzed for reliability and validity. After removing the items that failed the test, the results indicated that the reliability and validity of the questionnaire structure were high. After pre-testing, the final sample was used for data processing. The reliability assessed the consistency of a measure using the Cronbach’s alpha coefficient, which was higher than 0.8 for all items. The values for CR were also higher than 0.8, indicating statistical significance. The AVE for all items was above 0.5, which was higher than all squared correlations between constructs, supporting discriminant validity. This evaluation provided strong support for reliability and validity, as each dimension variable had a higher variance than the observed items (Table 2). The R package “car” was used to examine the variance inflation factor (VIF) to avoid the influence of collinearity [84]. The results (presence = 3.0614, flow = 2.154; enjoyment = 3.485, education = 1.283, information utility = 1.393) indicated no collinearity in this model.

Table 2.

Convergent validity and internal consistency reliability.

Variables	Cronbach's α	CR	AVE
Presence	0.946	0.949	0.785
Flow	0.864	0.858	0.616
Enjoyment	0.899	0.902	0.664
Education	0.848	0.839	0.581
Information	0.826	0.828	0.612
Engagement	0.855	0.822	0.577

6.2. Results related to different groups

Levene’s test for equality of variances and independent-samples t-test were performed. All the hypotheses are supported at relatively high levels of significance. The finding regarding the presence, flow, education, information utility, enjoyment and engagement of UX showed significant differences between the two groups. The mean and standard deviation results for all the factors of UX in the two groups were showed in Fig. 4. In accordance with H1a, participants in Group 2 reported more presence (M = 6.041; SD = 2.464; t (366) = −7.699) than participants in Group 1 (M = 4.093; SD = 2.373). Flow (H1b) had higher scores in Group 2 (M = 6.383; SD = 1.699; t (366) = −7.452) than in Group 1 (M = 4.791; SD = 2.379). The results showed that education (H1c) was higher in Group 2 (M = 7.985; SD = 1.465; t (366) = −5.747) than in Group 1 (M = 7.076; SD = 1.567). Information utility (H1d) was also higher in Group 2 (M = 7.010; SD = 1.433; t (366) = −4.700) than in Group 1 (M = 6.194; SD = 1.890). Participants in Group 2 felt more enjoyment (H1e) (M = 5.492; SD = 2.403; t (366) = −6.953) than participants in Group 1 (M = 3.762; SD = 2.359). Regarding H1f, engagement was reported to have higher scores in Group 2 (M = 7.126; SD = 1.684; t (366) = −7.571) than in Group 1 (M = 5.651; SD = 2.010). In other words, Group 2 scored significantly higher than Group 1 for all factors. Thus, H1a, H1b, H1c, H1d, H1e, and H1f are supported (Fig. 4).

Fig. 4.

The score distribution for presence, flow, education, information utility, enjoyment and engagement in Groups 1 and 2.

6.3. Overall hypothesis testing

Multiple regression analysis was used to test H2. Among all the potential predictors, only presence was not significant (p = .563, 95%CI: [-0.056,102]). The predictors were examined through stepwise regression analyses, which were entered at the 0.05 significance level and removed at 0.01. An updated regression model was tested using four significant predictors: flow, education, information utility, and enjoyment. The results indicated that the regression relationship between all four independent variables (flow, education, information utility, and enjoyment) on engagement was relatively close (R = .795). The total variance in engagement was 63.3% (R² = 0.633), which was explained by four independent variables in the model. The corrected coefficient of determination, R², increased, and the remaining standard deviation decreased, indicating an increasingly good fit. The change in R² with new variables was also statistically significant. Using ANOVA, the regression coefficients of the independent variables were found to not be 0, and the regression model was considered statistically significant. There was no collinearity among flow (VIF = 1.922), education (VIF = 1.250), information utility (VIF = 1.354), and enjoyment (VIF = 1.855). Therefore, there was sufficient evidence to support H2b (β = 0.388, p < .0001), H2c (β = 0.324, p < .0001), H2d (β = 0.149, p < .0001), H2e (β = 0.144, p < .001), and to reject H2a (β = 0.031, p = .563).

Regarding the measurement of other experimental data, 142 (72.5%) of the visitors in Group 2 completed the tasks prompted in the AR. Moreover, at the end of the experiment, researchers approached visitors at the exit and asked if they were willing to register for other special exhibitions. Group 2 registered 23% more people for special exhibitions than Group 1. Furthermore, participants in Group 2 were willing to help their friends register for special exhibitions 37% more than those in Group 1.

7. Discussion

Major findings are described in this section, with their theoretical and practical contributions and implications for further research, while also taking into account the limitations of this study. The primary contributions of this study's exploration are summarized, which demonstrate how AR storytelling can increase visitor engagement, and design concepts and foundational frameworks for practitioners of the cultural metaverse are discussed.

This study contributed to the literature of AR storytelling in cultural experiences and explored the psychological, behavioral, and technological effects of AR storytelling on participants in AR cultural experiences, and shows a pilot project of cultural metaverse. This is one of the few explorations of AR storytelling that discusses the relationship between different UX factors and different influences on engagement; thus, it represents a new extension of previous work in the field. Despite the differences in the results between the two applications, the items generally scored high, which supports earlier studies that found that AR produced greater immersion and UX than the average immersion obtained when AR was not employed [20,24], suggesting that AR technology creates an overall positive UX in cultural experiences. This is consistent with earlier findings [85]. On top of that, through the data analysis of the pilot experiment, the study supported that AR is a worth promoting as an effective immersive storytelling tool in cultural metaverse. The study revealed that the cultural metaverse creates new opportunities for innovative and personalized services, allowing users to interact with the digital and physical worlds in new and meaningful ways.

One of the key challenges that arise with the emergence of the metaverse is the need to develop new storytelling design. Unlike traditional media formats, the metaverse offers users a highly interactive and customizable experience, which demands a different kind of storytelling. The creators must explore new methods of storytelling that are optimized for two -way interactive communication in IVEs. This includes exploring the potential for user-generated content, real-time storytelling, and dynamic environments hat respond to user input.

Overall, the emergence of the metaverse introduces new and interaction between the virtual and physical worlds, resulting in new opportunities and potential models in cultural experiences and beyond. To fully realize the potential, more research is needed to research and explore new methods and means to actively contribute to storytelling design in cultural metaverse, exploring our understanding and optimization of two-way interactive communication content for UX in IVEs, in order to create truly compelling and engaging cultural content.

7.1. Theoretical contributions

This study contributes to AR storytelling in cultural experience and to the construction of cultural metaverse literature by exploring visitors' UX in cultural venues. The results of our two AR applications show that highly interactive AR applications can lead to increased visitor engagement and thus, positive behavioral intentions in technologically enhanced cultural experiences. As expected, innovative AR storytelling can improve visitors’ high-level presence, flow, enjoyment, education, information utility, and engagement. This is consistent with previous studies of UX in IVEs based on the experience economy theory [37,78].

The results demonstrated the importance of AR storytelling, suggesting that it offers absorption, entertainment, challenges, and rewards to receive a heightened state of flow, thus obtaining high engagement in UX. The study design was based on prior research that found that flow was the main factor contributing to engagement. In addition, the design is supported by, who proposed that “perceived skills and challenges,” “a clear goal,” and “immediate feedback” are significant flow triggers, which were the main reasons why the innovative AR storytelling application had higher flow than the AR e-book [67].

Furthermore, the findings demonstrated that AR storytelling mostly depends on flow and instruction to improve engagement outcomes in cultural experiences. The flow is influenced by a highly immersive and interactive environment [86]. In the present study, an AR e-book group was chosen to control the technology effects, instead of a physical visiting group that controlled different immersive technologies to influence the elements’ changes. Therefore, the impact of outcomes mainly came from different AR storytelling. The UX and engagement processes are reflected in the UX regression model.

Education is another important factor that influences engagement. This study’s results provide empirical support for past research on UX [87,88] and experience economy theory studies in IVEs [42]. The results confirmed that cultural experiences have a strong educational attribute, which is the primary purpose of visiting a museum [89].

In addition, this study found that presence is a critical factor in the cultural metaverse experience. However, that was not significant in the regression model. There are two probable reasons for this result. First, both experimental groups visited the museum, which is a real physical space, and thus, there was little difference in the experience of actually “being there.” Second, the results of the part and partial correlations were somewhat different, suggesting that the correlation between the variables may be the primary reason for this result. Under the same technical environment, future research should explore the influence of presence and other variables by designing different interaction contents.

This study appears to open up new opportunities for the use of AR, particularly storytelling in cultural metaverse. Consistent with our expectations, the AR applications with embedded story clues resulted cultural metaverse a great experience for museum visitors. This study was an attempt at a cultural metaverse. By allowing the visitors to choose their own tour route and providing them with diverse stories, a two-way interactive AR storytelling and an experimental cultural metaverse could be realized.

7.2. Practical contributions

Practitioners are developing applied technology strategies to address the advent of the age of metaverse cultural experiences. This study provides much needed guidance for practitioners working on the cultural metaverse by revealing how the implementation of AR technology solutions can facilitate UX. Furthermore, this study highlights important points for practitioners planning to implement immersive technology applications for cultural experiences to improve UX and engagement.

7.3. Limitations and future research direction

While this study provides valuable insights into the impact of AR technology on cultural metaverse, this study has several limitations related to the population sample that need to be addressed in future research to ensure the validity and generalizability of the results.

Firstly, the museum exhibition focused on technology and digitization; therefore, visitors tended to be highly educated people who were interested in digital culture and technology, limiting the generalizability of the findings to other visitor profiles. This has the potential to lead to high overall results for the questionnaire. Owing to the geographical limitations of this venue, the population was relatively homogeneous in terms of cultural background, which may not be representative of other cultural institutions or events. It is essential to replicate the study in other cultural venues to confirm the results. Future research should explore cultural experiences that display different types of cultural venues to gain a more comprehensive understanding.

Furthermore, the pilot study discusses the differences between the AR E-book group and the AR storytelling group. While these groups were found to be significantly different in terms of UX, there may be other groups that were not studied that could yield important insights. Due to the small sample size and less-than-rich demographics, the relationship between the factors was not discussed in this study.

Finally, the study also acknowledges that the creation and selection of measurement instruments were also influenced by the sample size. Future studies should extend the population to include cultural metaverse users from other cultural backgrounds to obtain more accurate and generalizable results.

8. Conclusions

The cultural metaverse can be thought of as a multiple experience. Combining the cultural experience of physical space with the experience of computer – generated information allows for education, entertainment, aesthetics, and escapism, with varying levels of visitor engagement improves the process of interaction between users and cultural spaces by activating opportunities for facilitating the emergence of new experiential forms.

The study reveals that AR is a suitable tool for constructing a primary cultural metaverse. Interactive AR storytelling can provide immersive experiences that effectively boost visitors' engagement. Together, flow, education, information utility, and enjoyment influence user engagement. In the cultural experience, flow and education are the two influential factors that contribute most to engagement. This study contributes to AR storytelling in cultural experience and to the construction of cultural metaverse literature. It demonstrates how the use of AR technology solutions can improve UX, offering much-needed direction for practitioners on the cultural metaverse.

The industrial metaverse opens up avenues for personalized and immersive UX and influences user behavior in a variety of ways. Practitioners can apply the metaverse's own strengths to make the UX more immersive, enjoyable, learner-friendly, and knowledge-enhancing, thereby increasing user engagement with the experience. It is, therefore, recommended that practitioners consider investing in AR storytelling to promote interactive user engagement and provide more exciting cultural experiences for a cultural metaverse with real value.

Author contribution statement

Shuran Yang: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Data availability statement

Data will be made available on request.