QoE modeling for audiovisual associated with MOBA game using subjective approach

Abstract

With the advancement of multimedia communications and information technologies, online video games have become very popular recreational activities worldwide, while Multiplayer Online Battle Arena (MOBA) games have gained immense popularity from game players in recent times. Therefore, based on quality of experience, audiovisual quality evaluation models for MOBA games have been proposed in this study. The subjective data from 210 game players (both males and females) about audio quality during conversation, the video quality, the overall quality, and the weighted percentage of video and voice have been collected from small competition events of the Garena ROV (Arena of Valor) in Hua Hin, Thailand. The data from 160 game players have been used to create the models, while the data from 50 game players have been applied for model performance evaluation. After developing the models, Mean Absolute Percentage Error (MAPE) technique has been utilized for model evaluation. On completion of this study, it was found that two thirds of the proposed models provide better performance than the other previous works. All of them, called MOS_{Audiovisual-MOBA1} model, MOS_{Audiovisual-MOBA2} model and MOS_{Audiovisual-MOBA3} model, show excellent performances with MAPE values of 4.95%, 5.92% and 4.75% respectively. Therefore, excellent performance of these proposed models becomes one of the contributions of this study. Also, this evidence of excellent model performance makes the proposed models ideal for utilization for assessment of audiovisual quality of the MOBA games. All audiovisual quality evaluation models provide excellent efficiencies.

Introduction

Internet has been utilized widely for various purposes; for example, multimedia communications, information searching, business (e.g., e-commerce and Internet banking), education (e.g., e-learning), entertainment (e.g., Netflix and YouTube), social media networking (e.g., Facebook and Twitter) and Gaming. Furthermore, with the advancement of mobile communications, people can also use mobile Internet via mobile communication networks (e.g., 4G/LTE and 5G). It has been forecasted that mobile communications networks will support several kinds of data traffic including video traffic that covers online video game [1].

Because of the availability of mobile Internet at present, it has changed entertainment medium significantly, including video game industry [3]. A lot of video games, which has become online video games, face a constant increase in popularity. Similar to any other media, video game industry is now considered as one which is continuously growing and has one of the largest marketshare [38]. Particularly, in 2020–2021, Coronavirus (COVID-19) has spread to every country in the world. The virus changes human behavior from offline activity to online activity. It is one of the key factors driving the game industry. It increases interest in video gaming due to COVID-19- related lockdown measures. Especially, mobile gaming saw the most significant growth. Mobile games generated revenues of 77.2 billion USD approximately in total in 2020 (growing +13.3% compared to 2019) [14]. In Thailand, the expected growth value of the game industry will be increased from about 33.7 million USD in 2021 to 39 million in 2022 approximately. This information is consistent with the expected global growth values that will be about 174.2 billion USD and 187.8 billion USD in 2021–2022 respectively [14]. Nevertheless, it has been reported in 2019 that Thailand was the 19th biggest market in the world with about 18.3 million video game players [5]. That means video games are very popular in this country, particularly online video games that are played by not only regular teenagers but also Buddhist novice monks [40]. One of several reasons for gaming popularity is the support and promotion from the Tourism Authority of Thailand [43].

For playing online games, particularly real-time multiplayer games, mobile phone’s rendering efficiency (e.g., frame rate) and a high level of synchronism are required [30, 33]. Furthermore, it requires good network performance, otherwise network parameters (e.g., packet loss, latency or delay and jitter) impact the perception and response of game players. Consequently, perceived audio quality and video quality, called audiovisual quality, might be degraded, while playing online games with poor network quality. Therefore, this is very important for online game operators and/or service providers to understand how those factors affect gaming experience of game players [30]. One of the methods to understand the game players from each of their ‘point of view’ is using the Quality of Experience (QoE) metric, which can be applied for audiovisual quality measurement. Therefore, this research proposes a method to evaluate audiovisual quality of online video games provided by video game operators and/or service providers at the end-users point. However, this study is limited to a Multiplayer Online Battle Arena (MOBA) game that is very popular in Thailand, called ROV that stands for Realm of Valor [3, 12].

This article, which is the extended version of [12], has been arranged into seven sections. After the introduction, background information is described in Section 2. It covers three sub-sections, including Esports, MOBA and RoV, influence factors for gaming quality and audiovisual quality assessment methods, and prior research works. Section 3, describes the methodology, including data gathering from the event of online game competitions, proposed models that is the highlight of this article, and model evaluation and analysis. As for section 4 these discussions are presented. The conclusion with future work is eventually expressed in Section 5, the last section.

Background and related work

Esports, MOBA and RoV

As described in [10], the term ‘esports’ stands for ‘electronic sports’, which is considered to be a variant of online gaming and/or video gaming. ‘Esports’ can be defined as a multiplayer video game played competitively for spectators, usually by professional game players [10]. The esports association in South Korea defines it as a leisure activity within cyberspace in which participants matched their electronic game skills to compete with each other for winning [27]. In other words, this is a sort of sports within and through the cyberspace medium where a competitive game is played typically within accepted rules of tournaments and leagues on Internet [26, 48]. It can be a form of sports where the primary aspect of the sport is supported and/or facilitated by computer systems/networks and/or Internet and the input of the system are players and teams, while the output is mediated by human computer interfaces [18, 27]. It has been stated in [6] that esports is a type of sporting activity in which game players train and develop their hand-eye co-ordination skills and mental skills while using game-based multimedia communication and information technology, and where game players are virtually represented in an electronic sporting world [20, 46]. Esports involve a number of different video game genres including action games, racing games, sports games, first person shooters, fighting and strategy games. Furthermore, this is now accepted worldwide, given the popularity and growth of the esport industry. It has been estimated that these numbers will be grow to 234 million esports enthusiasts and more than 240 million occasional viewers, in total 474 million people approximately, in 2021 [41]. Furthermore, it was reported that the 2019 SEA Games hosted the tournaments for six games across mobile phones, consoles and personal computers, including Arena of Valor (AoV or called ROV in Thailand) and other five popular video games, in December 2019 [16]. Recently, The International Olympic Committee (IOC) announced that the Olympic Virtual Series organized competitions in five esports (e.g., cycling, auto racing, baseball and sailing) and staged ahead of the Summer Games 2021 in Japan [21].

At present, while there are many genres of online video games (e.g., action-adventure, shooter, role-playing, sport, fighting and strategy games), one of the most popular game genres is MOBA. This is one of the largest subgenres of strategy video games [17, 34]. According to the game player motivation model [13, 51], mastery is one of the major objectives for game players. It is also one of the most popular strategy game genres in eSports at present [2]. There are several MOBA games in the video game industry, for example, League of Legends (LoL), Defense of the Ancients (DotA) and Heroes of the Storms [3]. For ROV, it stands for Realm of Valor as it is called in Thailand, while it is called Arena of Valor and Honor of Kings for global and China respectively [12, 43]. This is one of the most popular and attractive MOBA games on smart phones in Asia since its gamification tool is applied to explore and analyze the main elements in ROV’s competitive and entertainment experience [3]. Its popularity is evident from the 100 million downloads from the public [45].

ROV is a Free-to-Play (F2P) online video game that can be downloaded, installed and played by game players. It provides the scenario that two teams of players can fight with each other to become the winner. It has a shorter game period, about 15–20 min. Also, it has easy controller for playing on smart phones; thus, it can provide entertainment to the game players [12]. Furthermore, this is a game of 5 versus 5 played by the Red and the Blue team [3]. This game supports both iOS and Android operating system [3]. The goal of each team, it is to destroy the opponent’s tower located at the opponent’s base, for winning [3]. While playing the game, each one in each team can talk or communicate to other team members via Voice over IP (VoIP) that is an ROV’s characteristic. Nevertheless, game players may encounter technical problems (e.g., low video quality and unclear voice quality) if their network provides them poor Quality of Service (QoS). ROV is very popular in Thailand. It was reported that there were more than 31 million registered accounts in this country, 11 million watch hours and 55 million views of Season 3 videos in Q2/2019, while the Thai population was 69.6 million people approximately in the same year [5, 44]. Recently, the ROV team from Thailand has become the winner in the Arena of Valor World Cup 2021 (AWC 2021) that was organized with the theme ‘Fight together rise together’ (see Fig. 1) [4].

Fig. 1.

a The background screen presented in the AWC 2021 tournament b Well-known ROV advertisement

Influence factors for gaming quality and audiovisual quality assessment methods

ITU-T describes that there are several influence factors that can impact on the gaming QoE perceived by users or game players who play networked games or online games [22]. Those influence factors have been grouped into three main factors which are comprised of:

1. Human factors: these factors are described as variant or invariant properties or characteristics of game players. The characteristics can describe the demographic and backgrounds, the emotional state and/or the physical and mental constitution. These factors cover experience with gaming in general, experiences with a specific game, extrinsic and intrinsic motivation, dynamic, and static human factors, and human vision.

2. Context factors: they are things that embrace any situational property to describe the user’s surrounding in terms of physical, task, social, and technical characteristics, for example. Context influence factors consist of physical environment factors, service factors, novelty, and social context that is one of the essential factors for online multiplayer games.

3. System factors: these factors are characteristics and properties that determine the produced quality of a service or an application in technical terms. They cover the factors and their parameters or components as shown in Table 1 [22].

Table 1.

System factors and their parameters or components [22]

System factors	Parameters or components
Game (Game content)	- Video game genre - Video game rules and mechanics - Temporal and spatial video complexity - Temporal and spatial accuracy - Visual perspective of the player - Design characteristics and aesthetics - Learning difficulty - Pace
Playing devices	- Portability - Size of a handheld device - Input modalities - Output modalities - Display (e.g., size and monitor refresh rate)
Network transmission	- Packet delay - Packet loss - Jitter - Bandwidth
Compression	- Frame rate - Resolution - Rate controller modes - Group of Pictures - Motion range search - Audio compression

For Gaming QoE, it is a multidimensional construct related to the quality features, which were studied and proposed by [31]. Then, they have been applied to the ITU-T P. 809 standard [23]. For quality features, there are aesthetics and appeal, interaction quality, playing quality, positive and negative effect, player experience acceptability, and engagement which consist of involvement, immersion, presence, flow, absorption, and relationships between engagement concepts [23].

Audiovisual quality evaluation methods can be categorized into subjective and objective methods [9]. They are referred to as QoE measurement methods. QoE directly focuses on user perception of services and/or applications. It has been described in [28] that QoE is the key determining factor in the success of multimedia services and/or applications. QoE covers a wider range of aspects and includes more factors than QoS. Furthermore, it has been described in [25] that QoE directly reflect users’ acceptance of the services, this is therefore appropriate for the assessment of service quality in applications, networks and/or systems.

For QoE, Mean Opinion Score (MOS) is the key quality factor that presents the QoE level directly [7]. It is usually a five-point scale to quantify level of multimedia quality perception (5 = excellent whereas 1 = bad). MOS is widely utilized as it expresses the QoE level provided by real-time services and/or applications. Therefore, from the users’ point of view, this is very useful to evaluate applications, services, systems and networks [50].

Previous related research works

In 2018, Mora-Cantallops and Sicilia [32] presented a literature review by focusing on 23 MOBA game papers. They found that one of the main lines in research on MOBA games is based on modeling and prediction, covering strategy modelling. However, they did not mention about QoE modeling based on MOBA games. From other previous research works, there are several interesting works associated with QoE models, including objective-online game QoE models [37, 39, 42, 47], but only the subjective QoE models are presented in Table 2 that can be briefly described as follows:

1. Dwyer & Finn [15] performed a study to assess MOS of gameplay under the controlled conditions based on latency. They recruited 32 volunteers to play the game ‘Team Fortress 2’, before the analysis and modeling two MOS models associated with delay and jitter as shown in (1)–(2).

2. Based on video telephony, ITU-T proposed the QoE model called ‘Opinion model for video telephony applications as presented in (3) in Table 2 [24]. This mathematical model has the four coefficients derived from the study based on two different video display sizes of 2.1 and 4.2 in. only, since this QoE model was first developed and released in 2000s. Hence, the display sizes are quite small compared to the display sizes of modern smart phones.

3. Hands [19] conducted a study based on audio and video (no games) to obtain MOS scores from the experiments with more than 36 participants within controlled environment (e.g., about 15 in. flat-screen and a sound insulated cabinet). He found that audio quality is weighted slightly higher than video quality for normal-motion content, whereas video quality is weighted significantly higher than audio quality for high-motion content. In addition, he performed the regression analysis and then proposed two QoE models as shown in (4)–(5).

4. Similar to [19], Winkler and Faller [49] performed the tests with 17 in. LCD screen following ITU-T standards subjectively to investigate about perceived audiovisual quality based on low-bitrate multimedia content (e.g., high motion video of American football and the sounds of female commentator and crowd cheering). After conducting the tests with 24 participants and analysis. Then, two models as shown in (6)–(7) were proposed.

5. Belmudez and Moller [8] conducted the study with 21 and 48 participants in the passive and interaction experiments respectively. They studied with various scenarios before performing the analysis, and then proposed several mathematical models, including two models for the short conversation test as in (8) and the audiovisual short conversation test as shown in (8)–(9) respectively.

6. Lastly, Pornpongtechavanich and Daengsi [35] conducted the study based on video telephony. This work was conducted with about 200 subjects who participated the subjective tests with short conversation scenario via two applications, LINE and Skype. After obtaining the MOS results for audio quality and video quality, the analysis was performed before proposing a simple mathematical model for video telephony quality evaluation as shown in (10).

Table 2.

List of previous research works

Authors	Mathematical model	Coefficient						Video Telephony		Online game
		a	b	c	d	e	f	Objective	Subjective	Objective	Subjective
Dwyer & Finn [15]	MOSTF2 G-model1 = a·Delay + b·Jitter + c (1)	−1.2 × 10−3	−1.3 × 10−2	4.15	–	–	–	–	✓	✓	–
	MOSTF2 G-model2 = a·Delay+b·(Jitter)2 + c (2)	−1.3 × 10−3	−1.3 × 10−4	3.98	–	–	–	–	✓	✓	–
ITU-T [24]	MOSAdiovisual-G.1070 = a∙MOSAudio + b∙MOSVideo + c∙MOSAudio∙MOSVideo + d (3)	−32.55 × 10−2	33.09 × 10−2	14.94 × 10−2	54.57 × 10−2	–	–	–	✓	–	–
Hands [19]	MOSAudiovisual-H1 = a∙MOSAudio∙MOSVideo + b (4)	0.17	1.15	–	–	–	–	–	✓	–	–
	MOSAudiovisual-H2 = a∙MOSVideo + b∙MOSAudio∙MOSVideo + c (5)	0.25	0.15	0.95	–	–	–	–	✓	–	–
Winkler & Faller [49]	MOSAudiovisual-WF1 = a∙MOSAudio∙MOSVideo + b (6)	1.03 × 10−1	1.98	–	–	–	–	–	✓	–	–
	MOSAudiovisual-WF2 = a∙MOSAudio + b∙MOSVideo + c (7)	4.56 × 10−1	0.77	−1.51	–	–	–	–	✓	–	–
Belmudez & Moller [8]	MOSAudiovisual-BM1 = a∙MOSAudio + b∙MOSVideo + c∙MOSAudio∙MOSVideo + d (8)	5.84 × 10−1	3.57 × 10−1	1.30 × 10−3	1.27 × 10−1	–	–	–	✓	–	–
	MOSAudiovisual-BM2 = a∙MOSAudio + b∙MOSVideo + c∙MOSAudio∙MOSVideo + d (9)	0.03	7.90 × 10−2	1.23 × 10−1	13.74 × 10−1	–	–	–	✓	–	–
Pornpongtechavanich & Daengsi [35]	MOSAudiovisual-PD = a∙MOSAudio + b∙MOSVideo (10)	0.65	0.35	–	–	–	–	–	✓	–	–

One can see in Table 2 that the QoE models as in (1)–(2) are associated with QoS parameters (e.g., latency or delay, jitter and packet loss), unlike the approach in this article that is based on subjective approach. While the models in (3)–(10) are mainly based on video telephony utilizing subjective approach but they are not designed or proposed for online game quality evaluation like the proposal in Section 3. Furthermore, it has been found in [36] that (3)–(5) are unreliable, while (6)–(7) provide lower performance than (8)–(10). Therefore, the branch of research line based on QoE modeling and prediction for MOBA games covering audiovisual quality becomes a new challenge and a gap of proposing the approach as in this article.

Methodology

This section presents the methodology that covered, game competition design and data gathering, proposed models, and model evaluation and analysis, following the processes as shown in Fig. 2. More details are described in each sub-section.

Fig. 2.

Overview of the research methodology

Game competitions and data gathering

This study focused on calling or conversation of game players while playing the online video game via smart phones or mobile phones. The online video game in this study was based on the Garena AOV or called ROV for short in Thailand. This online game is one of the very popular MOBA games in Thailand.

For creating the mathematical model, the subjective data from the game players is necessary. In order to gather a lot of game players at the same period, therefore the ROV competitions were designed by organizing as two competitive events in 2019 in Hua Hin, Prachuap Khiri Khan, Thailand. All game players were the students of Rajamangala University of Technology Rattanakosin (RMUTR), Wang Klai Kangwon Campus in Thailand, who applied to join the ROV competitions.

Two competition events or tournaments were organized in September and October 2019 for the subjective data gathering, Further information can be described as follows:

• There were four modes, consisting of 2v2, 3v3, 4v4 and 5v5 (see Fig. 3), that means there were 2, 3, 4, and 5 game players per team respectively.

• For the first competition event, there were 16 teams of 2v2, 12 teams of 3v3, 8 teams of 4v4 and 8 teams of 5v5, altogether consisting of 140 game players.

• For the second competition event, there were 8, 6, 4 and 4 teams of 2v2, 3v3, 4v4 and 5v5 respectively, consisting of 70 game players.

• Totally, there were 210 game players of which there were 137 males and 73 females with normal vision and hearing.

• All participants were Thai native speakers communicating with each other using Thai language.

• Before starting the competition, all game players received explanation about filling in the forms while playing or after completion of the competition.

• They had to fill in their basic information and vote or give feedback on their perception about video quality, audio quality and the overall audiovisual quality.

• Similar to [63], all of them had to vote their opinion in weighted percentage about the audio quality and video quality (e.g., audio/video = 90% / 10%, 80% / 20%, 70% / 30%, 60% / 40%, … or 10% / 90%).

Fig. 3.

The example of ROV competition scenario with 5v5 mode

After finishing the events, the gathered data from the game players were split into two parts. The first part was used for creating the audiovisual quality evaluation model, consisting of 103 male and 57 female participants, 40 participants per mode or 160 subjects in total, as presented in Table 3. The average age of them was 19.71 ± 1.35 years old at that period. While, the second part, consisting of 34 male and 16 female participants or 50 subjects in total, was applied for the model performance evaluation in Sub-section 3.3.

Table 3.

The information of 160 participants in the first competition event

		Game mode
		2v2	3v3	4v4	5v5	All modes (Average)
Participants	No. of Male	30	24	22	28	103
	No. of Female	10	16	18	12	57
	Average age (years)	20.98 ± 1.14	19.43 ± 1.01	19.80 ± 1.16	19.15 ± 1.01	19.71 ± 1.35

For the subjective results, MOS values (with the standard deviation values (SD)) for audio, video and audiovisual called MOS_Audio, MOS_Video and MOS_Audiovisual respectively are presented in Table 4. One can see that the values of MOS_Video are higher than the values of MOS_Audio in all conditions of the game mode. For overall, the average values of MOS_Audio, MOS_Video and MOS_Audiovisual are 3.94 ± 0.59, 4.24 ± 0.51 and 4.00 ± 0.51 respectively. In every game mode, except 4v4 mode, MOS_Audiovisual value is between MOS_Audio value and MOS_Video value.

Table 4.

The MOS scores and the standard deviation (SD) from the first competition event

		Game mode
		2v2	3v3	4v4	5v5	All modes (Average)
MOS ± SD	Audio	3.88 ± 0.54	3.80 ± 0.56	4.05 ± 0.50	4.05 ± 0.68	3.94 ± 0.59*
	Video	4.25 ± 0.54	4.28 ± 0.45	4.28 ± 0.45	4.13 ± 0.56	4.24 ± 0.51*
	Audiovisual	3.98 ± 0.48	3.93 ± 0.53	4.00 ± 0.45	4.05 ± 0.55	4.00 ± 0.51*

Remark: ^* These values will be used to explain the proposed model in Section IV

For the weighted percentage of audio and video quality, after obtaining the votes about the ratio between audio quality and video quality from all participants, the weighted percentage for audio and video for each mode of the gameplay was calculated and presented in Table 5. One can see that the weighted percentage of video quality is slightly higher than the percentage of audio quality. Furthermore, for the average from all modes of the gameplay, the weighted percentages of video quality and audio quality were 46.56% and 53.44% respectively. Nevertheless, the second part of the subjective data will be described in details in Sub-section 3.3 for model performance evaluation.

Table 5.

The weighted percentages of audio and video from the first competition event

		Game mode
		2v2	3v3	4v4	5v5	All modes (Average)
Weighted percentage (%)	Audio	45.50	44.50	47.25	49.25	46.56
	Video	54.50	55.50	52.75	50.75	53.44

Proposed models

This sub-section presents the highlight and the major contribution of this article. In order to create models, the proposed mathematical models for audiovisual - QoE model associated with MOBA game can be simply created using the concept of the Simple Additive Weighting (SAW) that is presented as follows [29]:

$${\mathrm{M}\mathrm{O}\mathrm{S}}_{\mathrm{S}\mathrm{A}\mathrm{W}}={\sum }_{j=1}^n{w}_j\cdot {MOS}_j$$ 11

Where MOS_SAW is the overall MOS based on SAW, while w_j and MOS_j are is the weight and the MOS of criteria j respectively. Furthermore, this proposed model also followed (10) in Table 2, which is the QoE model for video telephony [35]. Nevertheless, there is advancement beyond [35], since the gathered data of audiovisual quality have been also applied to the QoE model.

As shown in Table 5 in the previous section, the weighted percentage of video quality and audio quality are 53.44% and 46.56% respectively. The calculation of them with MOS_Video of 4.24 and MOS_Audio of 3.94 as in the last column in Table 4 can be shown as follows:

$${\mathrm{MOS}}_{\text{Audiovisual}}=\frac{\left(4.24\ast 53.44\right)+\left(3.94\ast 46.56\right)}{100}=\frac{409.78}{100}\approx 4.10$$

One can see that the calculated MOS_Audiovisual of about 4.1 is higher than the average MOS_Audiovisual of 4.00 shown at the last column and row in Table 4. That means the difference in value of 4.10–4.0 = 0.10 between those MOS_Audiovisual can be applied to create new models as well. Therefore, the weighted percentage of video quality and audio quality of 53.44% and 46.56% respectively, and the difference value of 0.1 can be utilized to the new model using a simple technique.

However, this article aims to propose simple QoE models for MOBA games, thus the weighted coefficients should be simple values as well. Therefore, the coefficient a and b should be 0.47 and 0.53 instead of 46.56% and 53.44% respectively, while the coefficient c should be 0.10 (from the difference between the calculated MOS_Audiovisual and the average MOS_Audiovisual as the above mentioned. These three coefficients can be utilized in the proposed QoE models for MOBA game as shown in (12)–(14).

$${\mathrm{MOS}}_{\text{Audiovisual}-\text{MOBA}1}=\mathrm{a}·{\mathrm{MOS}}_{\text{Audio}}+\mathrm{b}·{\mathrm{MOS}}_{\text{Video}}$$ 12

$${\mathrm{MOS}}_{\text{Audiovisual}-\text{MOBA}2}=\mathrm{a}·{\mathrm{MOS}}_{\text{Audio}}+\mathrm{b}·{\mathrm{MOS}}_{\text{Video}}+\mathrm{c}$$ 13

$${\mathrm{M}\mathrm{O}\mathrm{S}}_{\mathrm{A}\mathrm{u}\mathrm{d}\mathrm{i}\mathrm{o}\mathrm{v}\mathrm{i}\mathrm{s}\mathrm{u}\mathrm{a}\mathrm{l}-\mathrm{M}\mathrm{O}\mathrm{B}\mathrm{A}3}=\left\{\begin{array}{c}\mathrm{a}\cdot {\mathrm{M}\mathrm{O}\mathrm{S}}_{\mathrm{A}\mathrm{u}\mathrm{d}\mathrm{i}\mathrm{o}}+\mathrm{b}\cdot {\mathrm{M}\mathrm{O}\mathrm{S}}_{\mathrm{V}\mathrm{i}\mathrm{d}\mathrm{e}\mathrm{o}};{\mathrm{M}\mathrm{O}\mathrm{S}}_{\mathrm{V}\mathrm{i}\mathrm{d}\mathrm{e}\mathrm{o}}={\mathrm{M}\mathrm{O}\mathrm{S}}_{\mathrm{A}\mathrm{u}\mathrm{d}\mathrm{i}\mathrm{o}}\\ \mathrm{a}\cdot {\mathrm{M}\mathrm{O}\mathrm{S}}_{\mathrm{A}\mathrm{u}\mathrm{d}\mathrm{i}\mathrm{o}}+\mathrm{b}\cdot {\mathrm{M}\mathrm{O}\mathrm{S}}_{\mathrm{V}\mathrm{i}\mathrm{d}\mathrm{e}\mathrm{o}}+\mathrm{c};{\mathrm{M}\mathrm{O}\mathrm{S}}_{\mathrm{V}\mathrm{i}\mathrm{d}\mathrm{o}\mathrm{e}}\ne {\mathrm{M}\mathrm{O}\mathrm{S}}_{\mathrm{A}\mathrm{u}\mathrm{d}\mathrm{i}\mathrm{o}}\end{array}\right\}$$ 14

Model evaluation and analysis

After obtaining three equations of the proposed QoE models as in (12)–(14), they were evaluated and compared with the previous QoE models. However, as mentioned in Sub-section 2, (1)–(7) are not appropriate for model evaluation Moreover, after considering (10), it was found that the weighted coefficient of 0.65 for MOS_Audio is significantly higher than the coefficient of 0.35 for MOS_Video. This is inconsistent with the weighted coefficients in (12)–(14) where the coefficient of 0.53 for MOS_Video is higher than the coefficient of 0.47 for MOS_Audio. Thus, (10) has been also discarded from model evaluation because the principal criteria of the proposed model is predominantly audio quality, and not video quality as in (12)–(14). Therefore, only two models as in (8) and (9) that have been used for model evaluation for comparison with (12)–(14).

For model evaluation, Mean Absolute Percentage Error (MAPE) that is a popular technique with high reliability has been selected for this article. It can be calculated using (15), where MAPE is the mean absolute percentage error (%), S_cal means the calculated score from the mathematical model, and S_ref is the reference score from the test set [11, 35]. The MAPE value can be interpreted as follows:

• 0% < MAPE <10% presents an excellent forecast

• 10% < MAPE <20% presents a good forecast

• 20% < MAPE <50% presents a fair forecast

$$\mathit{\text{MAPE}}=\frac{\sum _{i=1}^n\left|\frac{S_{{\mathit{cal}}_i}-S_{{\mathit{ref}}_i}}{S_{{\mathit{ref}}_i}}\right|}{n}$$ 15

For MAPE calculation in this article, test sets must be applied. In this article, three test sets were created from the 2nd part of the gathered data as in Table 6, randomly. Each test set consisted of 6 records from each game mode. Thus, each test set is comprised of 24 records in total from 24 game players (both female and male). Those were 18–20 years old at that period. The information about those test sets, called T1-T3 for short, is presented in Table 7. After evaluating with T1-T3, one can see in Table 8 that MOS_{Audiovisual-MOBA3} as in (14) provides the lowest MAPE values when compared with other MOS_Audiovisual models, while MOS_{Audiovisual-BM2} as in (12) provides the higher MAPE values.

Table 6.

The second part of results (the test sets) from 50 participants

Group of participants	Game Mode	N		Age	MOS ± SD			Weighted percentage
		Male	Female		Audio	Video	Audiovisual	Audio	Video
2nd part	2v2	6	2	20.25 ± 1.16	3.75 ± 0.54	4.38 ± 0.52	3.88 ± 0.35	41.25	58.75
	3v3	10	4	18.08 ± 0.28	4.00 ± 0.41	4.15 ± 0.38	4.08 ± 0.28	47.86	52.14
	4v4	5	3	19.25 ± 0.46	4.00 ± 0.53	4.13 ± 0.35	4.13 ± 0.35	50.00	50.00
	5v5	13	7	20.65 ± 1.53	4.10 ± 0.45	4.45 ± 0.51	4.15 ± 0.37	45.50	54.50

Table 7.

Information of the test sets

Testset	N		Age	MOS ± SD
	Male	Female		Audio	Video	Audiovisual
T1	20	4	19.79 ± 1.50	3.92 ± 0.50	4.29 ± 0.46	4.00 ± 0.29
T2	15	9	19.63 ± 1.24	4.00 ± 0.51	4.33 ± 0.48	4.08 ± 0.41
T3	16	8	19.50 ± 1.34	3.92 ± 0.41	4.25 ± 0.44	4.04 ± 0.36

Table 8.

Evaluation results using the MAPE technique

Model	MAPE (%)			Remark
	TS1	TS2	TS3
MOSAudiovisual-BM1 (8)	5.94	5.24	4.53
MOSAudiovisual-BM2 (9)	8.07	7.57	6.78	The lowest performance
MOSAudiovisual-MOBA1 (12)	6.02	5.48	3.34
MOSAudiovisual-MOBA2 (13)	6.67	6.30	4.80
MOSAudiovisual-MOBA3 (14)	5.74	4.97	3.05	The highest performance

One can obviously see from Fig. 4, which shows the average MAPE values from Table 8, (8) or the MOS_{Audiovisual-BM1} model and (9) or the MOS_{Audiovisual-BM2} model provide average MAPE of 5.24% and 7.47% respectively. Meanwhile (12)–(14) or MOS_{Audiovisual-MOBA1} - MOS_{Audiovisual-MOBA3} models provide average MAPE values of 4.95%, 5.92% and 4.79% respectively. That means MOS_{Audiovisual-BM2} model as in (9) is the worst model when compared to the other models, whereas MOS_{Audiovisual-MOBA3} as in (14) is the best model because it provides the highest performance with the lowest MAPE value (only 4.79%). Therefore, this can be used as the criteria for the best model selection.

Fig. 4.

The average MAPE from each QoE model for gaming

Nonetheless, all models under evaluation, as in (8)–(9) and (12)–(14) provide excellent forecasts since these MAPE values are less than 10%. The average MAPE values from all mathematical models don’t seem different. Therefore, this is necessary to verify this by using a statistical technique called Analysis of Variance (ANOVA) as in [11]. For analysis using this technique, the raw data of MOS_Audiovisual values from calculation with T1-T3 using (8)–(9) and (12)–(14) have been utilized based on the hypothesis as follows:

• H₀: The audiovisual quality calculated from (8)–(9) and (12)–(14) is the same.

• H₁: The audiovisual quality calculated from (8)–(9) and (9)–(14) is not the same.

After performing the hypothesis tests within the confidence interval of 95%, the results are presented in Table 9. The analysis results show that the p values from hypothesis testing with T1-T3 are 0.172, 0.295 and 0.156 respectively. Thus, the null hypothesis (H₀) can be accepted, while the alternative hypothesis (H₁) has been rejected, because those p values are less than the significant level of 0.05. That means there is no significant difference in every case with each test set among the model (8)–(9) and (12)–(14), although (14) or the MOS_{Audiovisual-MOBA3} model provides the best performance when compared to the other models.

Table 9.

ANOVA analysis results

Hypotheses	Test set	p value
H0: MOSAudiovisual-BM1 = MOSAudiovisual-BM2 = MOSAudiovisual-MOBA1 = MOSAudiovisual-MOBA2 = MOSAudiovisual-MOBA3 H1: MOSAudiovisual-BM1 ≠ MOSAudiovisual-BM2 ≠ MOSAudiovisual-MOBA1 ≠ MOSAudiovisual-MOBA2 ≠ MOSAudiovisual-MOBA3	T1	0.172
	T2	0.295
	T3	0.156

Remark: Significant at p value <0.05 within 95% Confidence Interval

Discussion

From this study with the MOBA game and subjective approach, there are several issues to be discussed before concluding in the last section, as follows:

1. Based on an approach with simplicity, this study has applied the concept following ITU-T G.1070 by using only two qualities, consisting of video quality and audio quality, for modeling the proposed QoE model for MOBA game, instead of taking interaction quality and playing quality as mentioned in ITU-T P.809 into account [23].

2. In this article, simple audiovisual – QoE models for MOBA game based on subjective data that are highly accurate, reliable and authentic have been proposed with only 2–3 coefficients, see (12)–(14) in the previous section. They have been created with a simple concept but it has been proven in the previous section that their performances are excellent. Of course, this issue is the major contribution of this article.

3. This is one of the audiovisual studies that has been conducted with a lot of participants. With 210 subjects, this number is higher than other prior works [8, 19, 24, 49], except [35]. Therefore, the proposed models can be applied to measure audiovisual quality for MOBA game with confidence. Moreover, this article is the first study based on MOBA game that was conducted with more than 200 subjects.

4. The first part of MOS_Audiovisual from the game players as in Table 4, called the training set, shows average MOS_Audiovisual of 4.00 ± 0.51, while the MOS_Audiovisual for the test set T1-T3, as in Table 7, shows MOS_Audiovisual of 4.00 ± 0.29, 4.08 ± 0.41 and 4.04 ± 0.36 respectively. There are slight differences between the training set and the test set (T1-T3). However, this issue is acceptable because both sets of data have been gathered from the students with the same criteria.

5. From the coefficient of 0.53 for MOS_Video and the coefficient of 0.47 for MOS_Audio, one can see that video quality slightly seems more important than audio quality for the game players. These coefficients are not conformed with the coefficients of the video telephony QoE as shown in [35], since in that QoE model, audio quality is more important than video quality.

6. One of the proposed models, MOS_{Audiovisual-MOBA3} provides the lowest MAPE value of 4.79%, whereas MOS_{Audiovisual-MOBA1}, MOS_BM1, and MOS_{Audiovisual-MOBA2} provides the MAPE values of 4.95%, 5.24%, and 5.92% respectively. That means the MOS_{Audiovisual-MOBA3} model is the most reliable compared with the previous work [8]. Therefore, it should be the best option for subjective QoE assessment for gaming, while MOS_{Audiovisual-MOBA1} should be the second option. Furthermore, video game developers may apply these new models to help improving the audiovisual quality of their games, in order to increase users’ perception and satisfaction.

7. The subjective test method as presented in this study might be utilized to study other game genres as described in Section 2 (e.g., FPS) or other games (e.g., Free Fire) on both PCs and mobile phones. Moreover, the proposed models in this article might be utilized to evaluate other MOBA games with high reliability.

8. Lastly, other techniques, such as Analytic Hierarchy Process (AHP) may be applied for finding the coefficients of the new QoE models for audiovisual or new gaming QoE models, in order to expand and enhance the discipline and related disciplines.

Conclusion and future works

This article presents three audiovisual quality evaluation models with reference to MOBA games using subjective tests with ROV that is a popular game in Thailand. After obtaining the data from the study with 210 game players who joined the ROV game competition events, three QoE models based on MOBA game were proposed. Then, the model performance evaluation was performed. The evaluation results show that the proposed QoE models, consisting of MOS_{Audiovisual-MOBA1}-MOS_{Audiovisual-MOBA3} for MOBA game provide excellent performances, particularly MOS_{Audiovisual-MOBA3} that provides the highest performance when compared to MAPE values from pervious QoE models with MAPE value of 4.79%. Although, this study with subjective approach on mobile phones covers all modes in a MOBA game (e.g., 2v2, 3v3, 4v4 and 5v5), similar studies on PCs, other game genres and various other games should be conducted in future works.

Declarations

Conflict of interest

All authors declare that there is no any conflict of interest.