E-Sports Training System Based on Intelligent Gesture Recognition

Abstract

In order to improve the effect of e-sports training, this paper combines the intelligent gesture recognition technology to construct an e-sports training system and judges the training effect of players through the recognition of players' gestures. Moreover, this paper studies the commonly used feature extraction algorithms and proposes an improved SLC-Harris feature extraction algorithm, and the feasibility of this algorithm is verified by the experimental results on the EUROC data set. In addition, this paper uses the KLT optical flow algorithm to track the extracted feature points and calculates the pure visual pose through epipolar geometry, triangulation, and PnP algorithms. The experimental research results show that the electronic economic training system based on intelligent gesture recognition proposed in this paper has certain effects.

1. Introduction

The reason why e-sports can become a sports competition is that it is closely related to the progress of society, the development of science and technology, and the spiritual and cultural needs of the people. Although there are countless people who enjoy this high-tech intellectual sports event, in fact, public opinion instills the harmful opinion of e-sports in people intentionally or unintentionally. Some media reported extensively that some students were addicted to games and could not extricate themselves, wasting their youth and studies, which made e-sports become “electronic heroin” that everyone shouted. The huge pressure of public opinion makes e-sports face severe survival pressure, and it is difficult for enterprises to enter this market justifiably. Moreover, athletes can only be called “players,” and their treatment cannot be compared with that of ordinary athletes. At the same time, the majority of fans can only engage in e-sports secretly. In addition, in the face of huge pressure from public opinion, it is difficult for the government to guide and supervise confidently, and sometimes, it has to prohibit escrow. The ban on television broadcasting of e-sports competitions can be described as a huge obstacle to the normal development of the current social discrimination against the e-sports sports industry.

Generally speaking, the development of e-sports is not yet mature, and the development of e-sports is still in its infancy [1], which is manifested in many aspects: the public recognition is not enough, there are few related large-scale events, there is no professional-scale operation, there is less research in this area, and so on [2]. Especially on college campuses, although students have more time for self-discipline than before, the school does not pay enough attention to e-sports, and there is no relatively formal organization and management of participants, which has led to many human resources problems waste [3].

In order to cater to the trend of e-sports development, vigorously develop e-sports business, improve the overall level of e-sports, and enable e-sports activities to develop well in colleges and universities, the current primary task is to deepen the characteristics of students participating in e-sports activities [4]. Among them, the analysis and research on the current situation, development trend, and participation significance of e-sports in colleges and universities are particularly important in order to discover the problems existing in the development of campus e-sports and put forward reasonable suggestions for the development [5].

As emerging sports, e-sports are mainly participated by the younger generation, which has the characteristics of younger and younger age. E-sports can exercise people's thinking ability, psychological pressure resistance, unity and cooperation, hand-eye coordination, and so on. It can also make the younger generation have the awareness of abiding by the rules in the process of participating in e-sports [6]; trained participants have a fair and open, never admit defeat, pursue a stronger competitive spirit, and have a positive impact on the lives of participants. Many colleges and universities have successively opened related majors in e-sports. Although e-sports is popular in the world, the related research and guiding theories on how to cultivate e-sports talents are rare [7].

Different scholars have different views on the attributes and characteristics of e-sports. Literature [8] proposed that “e-sports include three basic characteristics: one is electronics, the other is competitive sports, and the third is a confrontation between people. At the same time, e-sports sports are divided into virtualized e-sports sports and fictionalized sports.” Literature [9] pointed out that “the most fundamental characteristics of video games that distinguish them from other artificial games are: virtual environment, absence of the body and artificial intelligence,” emphasizing the main position of electronic communication technology in e-sports. Scholar Yang Fang believes that “e-sports should return to the essence of games, and games to competitive sports are based on the evolution trend of play-game-competitive sports” and based on the development process of traditional competitive sports, puts forward a plan for the development of e-sports. Jia Peng and Yao Jiaxin believe that e-sports has great characteristics: the diversity of functional structure requirements, the full expansion of self-awareness, the complexity of sports information pattern recognition, the agility of information processing efficiency, and the accuracy of intuitive thinking and decision-making. Sex analyzes and clarifies the various attributes of e-sports from many aspects [10].

The discussion on the attributes of e-sports is still going on. Based on the current research, it can be determined that the two essential attributes of e-sports are electronic interaction and confrontational competition. Without electronic interaction, it becomes traditional competitive sports; it becomes a video game, so the two are interdependent and indispensable. With the development of electronic interaction technology, various forms of e-sports have emerged [11].

Event services are mainly engaged in e-sports referees, coaches, club operation and management, game commentary, data and tactical analysis, and so on. Practitioners need to have data analysis capabilities, management capabilities, and commentary capabilities. The production and broadcast of the event include content production and external dissemination, mainly involving the design of live content and promotion plans, venue layout, equipment debugging, video data collection, postprocessing, background data analysis, and so on. The practitioners should have journalism, communication, broadcasting, TV technology, and other related professional abilities [12].

Since the e-sports industry is an emerging industry, most employees are not from e-sports majors and have not received a complete and systematic e-sports theoretical education, but nearly 90% of the employees believe that the e-sports industry needs prejob training [13]. Judging from the current situation of the development of the entire industry, it is undoubtedly the most attractive option to work for game manufacturers, but it is difficult for game manufacturers to absorb more human resources without major business adjustments. Therefore, the need to train practitioners in support organizations around e-sports events becomes more obvious [14]. For example, training content production capabilities (reporters, screenwriters, copywriters, and anchors) requires a professional background in journalism and communication; training event support capabilities (coaches, data analysts, nutritionists, and brokers) requires sports and information technology. Professional background: training public relations, marketing capabilities (products, business, brand marketing, and media), requires a professional background in marketing and management [15].

E-sports self-media is still a media, and you must have the ability to report news, or you can dig deep into a vertical field, such as specializing in video commentary of games, specializing in game clearance strategies, specializing in sharing game skills, and so on. After all, hot spots can bring traffic. WeMedia is a personalized media with social attributes; it communicates with users; and it has its own distinct character orientation [16]. To be a self-media, you should also have strong analytical skills and be able to interpret a topic or special event from a unique or professional perspective. Current e-sports professional ability training pathways.

1.1. Current E-Sports Professional Ability Training Pathways

Most training institutions in society position themselves as training professional players but basically lack training resources. Training institutions do not have coaches, data analysts, or club managers, and it is difficult for the trained people to find a suitable position in the e-sports circle. Rather than cultivating professional skills, it is better to make money from e-sports hot spots. Money has no intention or inability to contribute to the development of the e-sports industry [17].

At present, the main e-sports talents are cultivated by e-sports companies and e-sports clubs. The club mainly trains professional players, coaches, and data analysts in order to achieve better results in the league. Game companies train referees, game developers, commentators, and other related talents to ensure the healthy development of the e-sports industry [18]. An analysis of the revenue structure of the e-sports industry can help us see the e-sports industry more transparently. The truly profitable institutions are still game manufacturers, which continuously create market value through development and operation. In the context of the continuous development and popularization of the video game industry, competition has become a starting point for expanding influence and creating new commercial value. The comprehensive development of competitive value is inseparable from the promotion of surrounding formats, and new jobs such as video, live broadcast, and commentary emerge in an endless stream [19].

This paper combines the intelligent gesture recognition technology to construct an e-sports training system and uses the player's gesture recognition to judge the player's training effect to improve the e-sports training effect.

2. Intelligent Gesture Recognition

2.1. Gesture Intelligent Positioning

The structural framework of the gesture autonomous localization algorithm is shown in Figure 1.

Figure 1.

Gesture positioning algorithm structural framework.

Monocular visual-inertial odometry uses a pure camera in the front end for motion estimation. The algorithm firstly extracts the features of the image information collected by the camera, then uses the optical flow method to track the feature points, and finally uses PnP (Perspective-n-Point) to perform motion estimation on the tracked feature points. Then, the algorithm eliminates the mismatched point pairs through random sampling consistency (RANSAC) and uses nonlinear optimization to optimize the pose. The front-end process is shown in Figure 2.

Figure 2.

Front-end flowchart.

2.2. SLC-Harris Feature Extraction

The feature is the digital expression of the object in the image, and the image can be quantitatively analyzed by extracting the feature. Commonly used feature extraction methods mainly include SIFT algorithm, SURF algorithm, and ORB algorithm.

The traditional Harris algorithm calculates the angular responsivity as shown below. It is mainly based on the weighted summation of the squared and multiplied gradients of all pixels in the window.

$$\begin{array}{c}R=\left|C\right|-k\times {\left(\text{trace}\left(C\right)\right)}^2.\end{array}$$ (1)

Among them, there are

$$\begin{array}{c}\left|C\right|={\lambda }_1\times {\lambda }_2,\end{array}$$ (2)

$$\begin{array}{c}\text{trace}\left(C\right)={\lambda }_1+{\lambda }_2.\end{array}$$ (3)

In formula (1), k is a constant ranging from 0.04 to 0.06, and both λ₁ and λ₂ in formula (2) represent eigenvalues.

For a grayscale image, the value of any point (x, y) in the integral image ii (x, y) refers to the sum of all grayscale values from the upper left corner of the image to the area where this point is located, as shown in Figure 3.

Figure 3.

Rectangular window pixel calculation.

The calculation formula of pixels in the rectangular window is as follows:

$$\begin{array}{c}ii\left(x,y\right)={\displaystyle \sum _{x^{\prime }\le x,y^{\prime }\le y}I\left(x^{\prime },y^{\prime }\right)}.\end{array}$$ (4)

The most complex calculation in the Harris algorithm is the calculation of diagonal responsivity. The original calculation method causes the calculation overlap between each pixel in the integration window, resulting in high computational complexity. For this, the gradient values in g_x², g_y² and g_xg_y are used to integrate the image to speed up the calculation of the angular responsivity. The calculation formula is as follows:

$$\begin{array}{c}i{i}_{xx}\left(x,y\right)={\displaystyle \sum _{x^{\prime }\le x,y^{\prime }\le y}{g}_x^2\left(x^{\prime },y^{\prime }\right)},\\ i{i}_{yy}\left(x,y\right)={\displaystyle \sum _{x^{\prime }\le x,y^{\prime }\le y}{g}_y^2\left(x^{\prime },y^{\prime }\right)},\\ i{i}_{xy}\left(x,y\right)={\displaystyle \sum _{x^{\prime }\le x,y^{\prime }\le y}{g}_x\left(x^{\prime },y^{\prime }\right)g_y\left(x^{\prime },y^{\prime }\right)}.\end{array}$$ (5)

Efficient nonmaximum suppression (E-NMS) is used to efficiently extract unique feature locations for each corner region, and the region thresholds are compared using image patches instead of pixels. The principle is shown in Figure 4.

Figure 4.

Efficient nonmaximum suppression.

2.3. KLT Optical Flow Tracking

After the key points are extracted, the optical flow method is used to calculate the minimum photometric error by establishing an error model. This method does not need to calculate descriptors or feature point matching, which will greatly save the amount of calculation.

The basic idea of LK optical flow is to assume that the optical flow in the local neighborhood of a pixel is invariant, and based on this assumption, construct a least-squares problem about the optical flow of the neighborhood pixels.

First, it is assumed that the light intensity of the pixel in each frame of the image is constant. According to this, for the pixel located at (x, y) at time t, moving to (x + dx, y + dy) at time t + dt, there are

$$\begin{array}{c}I\left(x,y,t\right)=I\left(x+dx,y+dy,t+dt\right).\end{array}$$ (6)

Then, according to another basic assumption of LK optical flow, the displacement of pixels in adjacent images is small; the Taylor expansion of formula (6) is

$$\begin{array}{c}I\left(x+dx,y+dy,t+dt\right)\approx I\left(x,y,t\right)+\frac{\partial I}{\partial x}dx+\frac{\partial I}{\partial y}dy+\frac{\partial I}{\partial t}dt.\end{array}$$ (7)

Combining the above formulas and dividing by dt into both sides of the formula, we get:

$$\begin{array}{c}\frac{\partial I}{\partial x}\frac{\text{d}x}{\text{d}t}+\frac{\partial I}{\partial y}\frac{\text{d}y}{\text{d}t}=-\frac{\partial I}{\partial t},\end{array}$$ (8)

where dx/dt represents the motion speed of the pixel on the x-axis, dy/dt represents the motion speed of the pixel on the y-axis, and the two speeds are recorded as u and v, respectively. At the same time, ∂I/∂x represents the gradient value of the image in the x-axis direction at the pixel point; ∂I/∂y represents the gradient value in the y-axis direction at the pixel point; and ∂I/∂t represents the derivative value of the image in the t direction, which are denoted as I_x, I_y, and I_t, respectively. Therefore, formula (8) can be written in matrix form as follows:

$$\begin{array}{c}\left[\begin{array}{cc}{I}_x\hfill & I_y\hfill \end{array}\right]\left[\begin{array}{c}u\\ v\end{array}\right]=-I_t.\end{array}$$ (9)

Finally, according to the third basic assumption of the LK optical flow method, adjacent pixels in the same image plane have similar motion; a w × w size window is defined. According to the same motion of all pixels in the window, w² formulas can be listed; the overdetermined formulas can be constructed; and the motion parameters of the center point can be obtained by the least square method. Accordingly, its formula can be expressed as follows:

$$\begin{array}{c}\left(\begin{array}{cc}{\displaystyle \sum I_x}{I}_x\hfill & {\displaystyle \sum I_x}{I}_y\hfill \\ {\displaystyle \sum I_x}{I}_y\hfill & {\displaystyle \sum I_y}{I}_y\hfill \end{array}\right)\left(\begin{array}{c}u\\ v\end{array}\right)=-\left(\begin{array}{c}{\displaystyle \sum I_x}{I}_t\\ {\displaystyle \sum I_y}{I}_t\end{array}\right).\end{array}$$ (10)

Each image frame is downsampled by pyramid layering, and multilevel pyramids are established.

$$\begin{array}{c}{I}^L\left(x,y\right)=\frac{1}{4}{I}^{L-1}\left(2x,2y\right)+\frac{1}{8}\left(\begin{array}{c}{I}^{L-1}\left(2x-\mathrm{1,2}y\right)+I^{L-1}\left(2x+\mathrm{1,2}y\right)\\ +I^{L-1}\left(2x,2y-1\right)+I^{L-1}\left(2x,2y+1\right)\end{array}\right)\\ +\frac{1}{16}\left(\begin{array}{c}{I}^{L-1}\left(2x+\mathrm{1,2}y+1\right)+I^{L-1}\left(2x+\mathrm{1,2}y+1\right)+\\ I^{L-1}\left(2x+\mathrm{1,2}y+1\right)+I^{L-1}\left(2x+\mathrm{1,2}y+1\right)\end{array}\right),\end{array}$$ (11)

where L represents the Lth layer image.

The algorithm calculates the value of the bottom layer from top to bottom according to the Gaussian pyramid and calculates the pixel value near the edge of the image based on the following formulas:

$$\begin{array}{c}{I}^{L-1}\left(-1,y\right)\approx I^{L-1}\left(-1,y\right),\\ I^{L-1}\left(x,-1\right)\approx I^{L-1}\left(x,0\right),\\ I^{L-1}\left(n_x^{L-1},y\right)\approx I^{L-1}\left(n_x^{L-1}-1,y\right),\\ I^{L-1}\left(x,n_y^{L-1}\right)\approx I^{L-1}\left(x,n_y^{L-1}-1\right),\\ I^{L-1}\left(n_x^{L-1},n_y^{L-1}\right)\approx I^{L-1}\left(n_x^{L-1}-1,n_y^{L-1}-1\right).\end{array}$$ (12)

The camera motion pose is estimated using SFM in the vision front end. For a monocular camera, the camera pose can be estimated by the geometric relationship between two points in different locations in real space and their projected points on their respective imaging planes. As shown in Figure 5, P is any point in the three-dimensional space, and its coordinates are [X, Y, Z]^T; O₁ and O₂ are the optical centers of the two camera positions. p₁ and p₂ are the projection points of point P on the imaging plane I₁ and the imaging plane I₂, respectively. According to the pixel positions of the two matched point pairs p₁ and p₂, the essential matrix E and the fundamental matrix F can be obtained.

Figure 5.

Epipolar geometric constraints.

According to the camera imaging model, we assume that K is the camera internal parameter matrix, and R and t represent the rotation matrix and translation vector from plane I₁ to plane I₂, and the following formula can be obtained:

$$\begin{array}{c}{s}_1{p}_1=KP,s_2{p}_2=K\left(RP+t\right).\end{array}$$ (13)

Homogeneous coordinate transformation and normalization between 2D and 3D, we can get

$$\begin{array}{c}{x}_1=K^{-1}{p}_1,x_2=K^{-1}{p}_2,\end{array}$$ (14)

where x₁ and x₂ represent the coordinates of pixels p₁ and p₂ in the normalized plane, respectively. The algorithm combines formuls (13) and (14) and multiplies by x₂^Tt^∧ to obtain the essential matrix E and the fundamental matrix F, which can be sorted out as follows:

$$\begin{array}{c}{p}_2^T{K}^{-T}{t}^{\wedge }R{K}^{-1}{p}_1=0,\\ E=t^{\wedge }R,F=K^{-T}E{K}^{-1},\end{array}$$ (15)

where t^∧ represents the antisymmetric matrix.

When there are more than eight sets of point pairs such as p₁ and p₂, the eight-point method can be used to construct a linear formula system for the simplified formula, and then the unique solution of R and t can be obtained.

$$\begin{array}{c}{x}_2^{T}E{x}_1=p_2^{T}F{p}_1=0.\end{array}$$ (16)

When the monocular camera recovers the pose through the epipolar geometric relationship, the obtained translation is the normalized value, which has no practical significance. In order to obtain the depth information on feature points, triangulation needs to be introduced. We assume that s₁ and s₂ represent the depth of the two feature points; we can get

$$\begin{array}{c}{s}_2{x}_2=s_{1}R{x}_1+t.\end{array}$$ (17)

The feature point depth values x₂^∧ and x₁∧ can be obtained by left-multiplying formula (17) by s₁ or s₂, respectively, as follows:

$$\begin{array}{c}{s}_1{x}_2^{\wedge }R{x}_1+x_2^{\wedge }t=0,\\ s_2{x}_1^{\wedge }R{x}_2-x_1^{\wedge }t=0.\end{array}$$ (18)

When the positions of multiple points in space are known, the camera pose can be estimated by the PnP algorithm. Common PnP algorithms include P3P, DLT, and BA optimization. Among them, the P3P algorithm is the most common method. The algorithm needs to know at least three points and their projection points on the camera imaging plane. Then, the camera pose can be estimated by solving the relationship between point pairs according to the similar triangle principle and the cosine theorem. A schematic diagram of the P3P relationship is shown in Figure 6.

Figure 6.

Schematic diagram of P3P relationship.

The coordinate system convention is as follows: the world coordinate system is represented by (·)^w, and (·)^b and (·)^c represent the IMU coordinate system and the camera coordinate system, respectively. The relationship between the coordinate systems is shown in Figure 7. (.)^v represents the visual reference frame in the sliding window, which is independent of the IMU measurement and can represent any frame in the visual structure. (.)_b^w represents the transformation from the IMU coordinate system to the world coordinate system; b_k represents the IMU frame of the kth image; (·)_c^v represents the transformation from the camera coordinate system to the visual reference system; and c_k represents the camera frame of the kth image. $\widehat{\left(·\right)}$ represents the measured value and parameter estimation value of the sensor; $\overline{\left(·\right)}$ represents the latest scale parameter of the sliding window; and the rotation can be represented by the rotation matrix R and the quaternion q. g^w=[0,0, g]^T represents the gravity vector in the world coordinate system, and g^v represents the gravity vector in the visual reference coordinate system.

Figure 7.

Coordinate conversion relationship.

2.4. IMU Preintegration

The sampling frequency of the camera used in this paper is 20 Hz, and the sampling frequency of the IMU is 200 Hz. It can be seen that the frequency of the IMU is much higher than that of the image. In order to avoid the repeated integration phenomenon caused by the frequency change of the visual frame optimization state caused by the high sampling rate of the IMU, a preintegration technique is used for all IMU sampling data between two image key frames. Furthermore, inertial measurements between adjacent image key frames are aggregated into a relative motion constraint through a preintegration technique. The principle of preintegration is shown in Figure 8.

Figure 8.

Principle of preintegration.

In Figure 8, from top to bottom are the time scale line, the number of image frames generated, the number of image key frames generated, the number of IMU samples, and the IMU preintegration value.

The measurement error of the system is mainly affected by bias random walk b and white noise η, and other errors such as the Markov process are ignored. Then, the measurement model of the accelerometer and gyroscope in the IMU can be expressed by the following formula:

$$\begin{array}{c}{\widehat{\omega }}^b={\omega }^b\left(t\right)+b_{\omega }+{\eta }_{\omega }\frac{1}{2},\\ {\widehat{a}}^b=q_b^{w^T}\left(a^w-{\mathbf{g}}^w\right)+b_a+{\eta }_a,\end{array}$$ (19)

where ${\widehat{\omega }}^b,{\widehat{a}}^b\left(t\right),{\omega }^b\left(t\right)$, and a^w(t) represent the measured value and real value of angular velocity and acceleration, respectively; b_ω, b_a, η_ω, and η_a represent the random walk noise and measurement white noise of angular velocity and acceleration, respectively; and q_b^wT is the rotation matrix transformed from the world coordinate system to the IMU coordinate system.

White noise obeys a Gaussian distribution, that is, η_a ~ N(0, σ_a²), η_ω ~ N(0, σ_ω²). The derivative of random walk noise also obeys the Gaussian distribution, that is, η_ba ~ N(0, σ_ba²), η_bω ~ N(0, σ_bω²).

The differential kinematic formulas for P, V, Q (representing the position, velocity, and rotation expressed in quaternions, respectively) versus time can be written as follows:

$$\begin{array}{c}{\dot{p}}_{b_t}^w=v_t^w,\\ {\dot{v}}_t^w=a_t^w,\\ {\dot{q}}_{b_t}^w=q_{b_t}^w\otimes \left[\begin{array}{c}0\\ \frac{1}{2}{\omega }^{b_t}\end{array}\right],\end{array}$$ (20)

where ⊗ represents quaternion multiplication.

Through the above derivative relationship, the position, velocity, and rotation at time k + 1 can be obtained from the position, velocity, and rotation at time k and by integrating the measured values of the IMU over time Δt_k. The continuous integration formula for PVQ is as follows:

$$\begin{array}{c}{p}_{b_{k+1}}^w=p_{b_k}^w+v_{b_k}^w\mathrm{\Delta }{t}_k+{\displaystyle \int {\displaystyle {\int }_{t\in \left[k,k+1\right]}\left[R_t^w\left({\widehat{a}}_t-b_{a_t}\right)-g^w\right]\text{d}{t}^2}}\text{,}\\ v_{b_{k+1}}^w=v_{b_k}^w+{\displaystyle {\int }_{t\in \left[k,k+1\right]}\left[R_t^w\left({\widehat{a}}_t-b_{a_t}\right)-g^w\right]}\text{d}t,\\ q_{b_{k+1}}^w=q_{b_k}^w\otimes {\displaystyle {\int }_{t\in \left[k,k+1\right]1/2}\Omega \left({\widehat{\omega }}_t-b_{w_t}\right)q_t^{b_k}\text{d}t},\end{array}$$ (21)

where ${\widehat{a}}_t$ and ${\widehat{\omega }}_t$ represent the acceleration and angular velocity measured in the IMU coordinate system, respectively. Δt_k represents the time difference from the kth frame to the k + 1 frame. R_t^w represents the rotation matrix from the world coordinate system to the IMU coordinate system. Because the measured ${\widehat{a}}_t$ and ${\widehat{\omega }}_t$ belong to the IMU coordinate system, in order to transform the IMU measured value to the world coordinate system, the rotation matrix needs to be left-multiplied. Ω(ω) means quaternion right multiplication; ω_x means antisymmetric matrix in quaternion multiplication (ω means the imaginary part value of quaternion). We assume that the quaternion is $q=\left[\begin{array}{cccc}x\hfill & y\hfill & z\hfill & s\hfill \end{array}\right]=\left[\begin{array}{cc}\omega \hfill & s\hfill \end{array}\right]$; then we have

$$\begin{array}{c}\Omega \left(\omega \right)=\left[\begin{array}{cc}-{\omega }_{\times }& \omega \\ -{\omega }^T& 0\end{array}\right],\\ {\omega }_{\times }=\left[\begin{array}{ccc}0& {\omega }_z& {\omega }_y\\ {\omega }_z& 0& -{\omega }_x\\ -{\omega }_y& {\omega }_x& 0\end{array}\right].\end{array}$$ (22)

By observing the continuous integral formula of PVQ, it can be seen that the current state is recursively obtained from the state of the previous time, and the estimated value is constantly changing. This will cause the IMU measurements to be repropagated, causing the velocity and rotation to be reintegrated after each nonlinear optimization iteration, resulting in a higher computational cost. Therefore, the optimization variables are separated from the IMU preintegration terms of the two key frames, and the rotation matrix R_w^b_k of the world coordinate system to the IMU coordinate system can be obtained by simultaneously left-multiplying the left and right sides of the continuous integration formula of PVQ:

$$\begin{array}{c}{R}_w^{b_k}{p}_{b_{k+1}}^w=R_w^{b_k}\left(p_{b_k}^w+v_{b_k}^w\mathrm{\Delta }{t}_k-\frac{1}{2}{g}^w\mathrm{\Delta }{t}_k^2\right)+{\alpha }_{b_{k+1}}^{b_k},\\ R_w^{b_k}{v}_{b_{k+1}}^w=R_w^{b_k}\left(v_{b_k}^w-g^w\right)+{\beta }_{b_{k+1}}^{b_k},\\ q_w^{b_k}\otimes q_{b_{k+1}}^w={\gamma }_{b_{k+1}}^{b_k}.\end{array}$$ (23)

The image frames b_k and b_k+1 of two consecutive moments are given, and the linear acceleration and angular velocity are preintegrated in the local coordinate system b_k to obtain

$$\begin{array}{c}{\alpha }_{b_{k+1}}^{b_k}={\displaystyle \int {\displaystyle {\int }_{t\in \left[k,k+1\right]}{R}_t^{b_k}\left({\widehat{a}}_t-b_{a_t}\right)\text{d}{t}^2}},\\ {\beta }_{b_{k+1}}^{b_k}={\displaystyle {\int }_{t\in \left[k,k+1\right]}\left[R_t^{b_k}\left({\widehat{a}}_t-b_{a_t}\right)\right]\text{d}t},\\ {\gamma }_{b_{k+1}}^{b_k}={\displaystyle {\int }_{t\in \left[k,k+1\right]2}\left(1/2\right)\Omega \left({\widehat{\omega }}_t-b_{{\omega }_t}\right){\gamma }_t^{b_k}\text{d}t},\end{array}$$ (24)

where α_bk+1^b_k, β_bk+1^b_k, γ_bk+1^b_k represent the relative pose, velocity, and rotation constraints, respectively, and are also the relative motion of b_k+1 to b_k. It can be seen that they are only related to ${\widehat{a}}_t$ and ${\widehat{\omega }}_t$ in b_k and b_k+1, and they have nothing to do with the initial position and velocity of coordinate system b_k.

Therefore, the preintegration formula is rediscussed, in terms of α_bk+1^b_k; it is related to ${\widehat{a}}_t$ and ${\widehat{\omega }}_t$ of the IMU; and ${\widehat{a}}_t$ and ${\widehat{\omega }}_t$ are also variables that need to be optimized. When the bias change is small, α_bk+1^b_k, β_bk+1^b_k, γ_bk+1^b_k are adjusted according to their first-order approximations to the bias.

$$\begin{array}{c}{\alpha }_{b_{k+1}}^{b_k}\approx {\widehat{\alpha }}_{b_{k+1}}^{b_k}+J_{b_a}^{\alpha }\delta b_a+J_{b_{\omega }}^{\alpha }\delta b_{\omega },\\ {\beta }_{b_{k+1}}^{b_k}\approx {\widehat{\beta }}_{b_{k+1}}^{b_k}+J_{b_a}^{\beta }\delta b_a+J_{b_{\omega }}^{\beta }\delta b_{\omega },\\ {\gamma }_{b_{k+1}}^{b_k}\approx {\gamma }_{b_{k+1}}^{b_k}\otimes \left[\begin{array}{c}0\\ \frac{1}{2}{J}_{b_a}^{\gamma }\delta b_{\omega }\end{array}\right],\end{array}$$ (25)

where J_ba^α and J_bω^α are the block matrices in J_bk+1^α and J_ba^β and J_bω^β are the block matrices in J_bk+1^β.

There are errors in the integral values of the IMU at different times, and the errors at time t are mainly related to the measured values of α_t^b_k, β_t^b_k, θ_t^b_k, b_at, and b_wt at time t. The following definition is given to represent the error vector:

$$\begin{array}{c}\delta z_t^{b_k}=\left[\begin{array}{c}\delta {\alpha }_t^{b_k}\hfill \\ \delta {\beta }_t^{b_k}\hfill \\ \delta {\theta }_t^{b_k}\hfill \\ \delta b_{a_t}\hfill \\ \delta b_{{\omega }_t}\hfill \end{array}\right].\end{array}$$ (26)

The derivation is based on the derivative of the error term kinetic formula. First, two concepts are introduced: true and nominal, where true represents the real measurement value containing noise and nominal represents the theoretical value without noise, and δ represents the measurement error; there are

$$\begin{array}{c}\delta \dot{\alpha }={\dot{\alpha }}_{\text{true}}-{\dot{\alpha }}_{\text{nominal}},\\ \delta \dot{\beta }={\dot{\beta }}_{\text{true}}-{\dot{\beta }}_{\text{nominal}}.\end{array}$$ (27)

Among them, there are:

$$\begin{array}{c}{\dot{\beta }}_{\text{true}}=R_t^{b_k}\left[{\widehat{a}}_t-{\eta }_a-b_{a_t}-\delta b_{a_t}-\left[{{\widehat{a}}_t-b_{a_t}}_{\times }\delta \theta \right.\right],\\ {\dot{\beta }}_{\text{nominal}}=R_t^{b_k}\left({\widehat{a}}_t-b_{a_t}\right).\end{array}$$ (28)

Combining the above formulas, we can obtain

$$\begin{array}{c}\delta \dot{\beta }=-R_t^{b_k}{{\widehat{a}}_t-b_{a_t}}_{\times }\delta \theta -R_t^{b_k}\delta b_{a_t}-R_t^{b_k}{\eta }_a.\end{array}$$ (29)

The derivation of $\delta \dot{\theta }$ is as follows, and according to the formula in the literature, it can be known that

$$\begin{array}{c}{q}_{\text{true}}=q_{\text{nominal}}\otimes \delta q.\end{array}$$ (30)

In this paper, according to the noise model and bias, we can get

$$\begin{array}{c}\delta \dot{\theta }\approx -{{\widehat{\omega }}_t-b_{{\omega }_t}}_{\times }\delta \theta -{\eta }_{\omega }-\delta b_{{\omega }_t}.\end{array}$$ (31)

In summary, the derivative of the IMU measurement error term at time t can be as follows:

$$\begin{array}{c}\left[\begin{array}{c}\delta {\dot{\alpha }}_t^{b_k}\hfill \\ \delta {\dot{\beta }}_t^{b_k}\hfill \\ \delta {\dot{\theta }}_t^{b_k}\hfill \\ \delta {\dot{b}}_{a_t}\hfill \\ \delta {\dot{b}}_{{\omega }_t}\hfill \end{array}\right]=\left[\begin{array}{ccccc}0& I& 0& 0& 0\\ 0& 0& -R_t^{b_k}{{\widehat{a}}_t-b_{a_t}}_{\times }& -R_t^{b_k}& 0\\ 0& 0& -{{\widehat{\omega }}_t-b_{{\omega }_t}}_{\times }& 0& -I\\ 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\end{array}\right]\left[\begin{array}{c}\delta {\alpha }_t^{b_k}\\ \delta {\beta }_t^{b_k}\\ \delta {\theta }_t^{b_k}\\ \delta b_{a_t}\\ \delta b_{{\omega }_t}\end{array}\right]+c\left[\begin{array}{c}{\eta }_a\\ {\eta }_{\omega }\\ {\eta }_{b_a}\\ {\eta }_{b_{\omega }}\end{array}\right].\end{array}$$ (32)

We set $F_t=\left[\begin{array}{ccccc}0& I& 0& 0& 0\\ 0& 0& -R_t^{b_k}{{\widehat{a}}_t-b_{a_t}}_{\times }& -R_t^{b_k}& 0\\ 0& 0& -{{\widehat{\omega }}_t-b_{{\omega }_t}}_{\times }& 0& -I\\ 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\end{array}\right],G_t=\left[\begin{array}{cccc}0& 0& 0& 0\\ -R_t^{b_k}& 0& 0& 0\\ 0& I& 0& 0\\ 0& 0& I& 0\\ 0& 0& 0& I\end{array}\right]$. The above formula can be simplified to

$$\begin{array}{c}\delta {\dot{z}}_t^{b_k}=F_t\delta z_t^{b_k}+G_t{n}_t.\end{array}$$ (33)

According to the definition of the derivative, the prediction formula of the mean is as follows:

$$\begin{array}{c}\delta {\dot{z}}_t^{b_k}=\underset{{\delta }_t⟶0}{\lim }\frac{\delta z_{t+\delta t}^{b_k}-\delta z_t^{b_k}}{{\delta }_t},\\ \delta z_{t+{\delta }_t}^{b_k}=\left(1+F_t{\delta }_t\right)\delta z_t^{b_k}+\left(G_t{\delta }_t\right)n_t.\end{array}$$ (34)

According to the error value at the current moment, the mean and covariance at the next moment can be predicted. The prediction formula for covariance is as follows:

$$\begin{array}{c}{P}_{t+\delta t}^{b_k}=\left(1+F_t{\delta }_t\right)P_t^{b_k}{\left(1+F_t\right)}^T+\left(G_t{\delta }_t\right)Q{\left(G_t{\delta }_t\right)}^T,\end{array}$$ (35)

where P_t^b_k represents the initial value of the iteration and its value is zero and Q represents the diagonal covariance matrix of the noise term as follows:

According to formula (35), the iterative formula of the error term Jacobian can be obtained as follows:

$$\begin{array}{c}{J}_{t+\delta t}=\left(1+F_t\right)J_t,\end{array}$$ (36)

where the iterative initial value of the Jacobian matrix J_t is I.

2.5. Sliding Window Initialization

When the camera extrinsic parameter matrix $\left({\overline{p}}_{b_k}^v,{\overline{p}}_{c_k}^v\right)$ is known, the pose obtained by the initialization of the monocular camera is transformed from the visual coordinate system to the IMU coordinate system to obtain the following formula:

$$\begin{array}{c}{q}_{b_k}^v=q_{c_k}^v\otimes q_b^v,\\ s{\overline{p}}_{b_k}^v+q_{c_k}^v{p}_b^c=s{\overline{p}}_{c_k}^v,\end{array}$$ (37)

where s is the translation factor obtained by visual initialization, which has no real information.

The pure visual initialization method lacks absolute scale information. Therefore, the value estimated by the visual SFM is aligned with the IMU after preintegration to estimate the true scale. Visual-inertial alignment initialization is mainly to solve the following problems, including the initialization of gyroscope bias, the initialization of velocity, gravitational acceleration, and scale.

The first is to initialize the gyroscope. The gyroscope bias can be obtained from two consecutive key frames with known orientations, considering two consecutive frames b_k and b_k+1 in the sliding window, and q_bk^v and q_bk+1^v represent the rotations obtained from the pure visual sliding window optimization, respectively. Linearize the IMU preintegration term for the gyroscope bias and minimize the following function:

$$\begin{array}{c}{\min }_{\delta b_w}{\displaystyle \sum _{k\in \mathrm{ℬ}}{‖q_{b_{k+1}}^{v-1}\otimes q_{b_k}^v\otimes {\gamma }_{b_{k+1}}^{b_k}‖}^2}.\end{array}$$ (38)

Among them, there are:

$$\begin{array}{c}{\gamma }_{b_{k+1}}^{b_k}\approx {\widehat{\gamma }}_{b_{k+1}}^{b_k}\otimes \left[J_{b_w}^{\gamma }\begin{array}{c}1\\ \delta b_w\end{array}\right].\end{array}$$ (39)

In formula (42), ℬ represents all the frames in the window, and the product of the two quaternions indicates that the camera rotates from the kth frame to the k + 1th frame, and the gyroscope rotates from the k + 1th frame to the kth frame, and the optimized objective function is

$$\begin{array}{c}{q}_{b_{k+1}}^{v-1}\otimes q_{b_k}^v\otimes {\gamma }_{b_{k+1}}^{b_k}=\left[\begin{array}{c}1\\ 0\end{array}\right].\end{array}$$ (40)

The algorithm takes $\widehat{\gamma }$ into the formula and multiplies the inverse moment ordering of the relative constraints obtained from the preintegration to the left on both sides of formula (40) and obtains by Cholesky decomposition (multiplying the transpose of J_bw^γ on both sides of the formula):

$$\begin{array}{c}{J}_{b_w}^{\gamma T}{J}_{b_w}^{\gamma }\delta b_w=2J_{b_w}^{\gamma T}2{\left({\widehat{\gamma }}_{b_{k+1}}^{b_k-1}\otimes q_{b_k}^v\otimes q_{b_{k+1}}^{v-1}\right)}_{vec}.\end{array}$$ (41)

In this way, the initial calibration value of the gyroscope bias b_w can be estimated, and then the IMU preintegration terms ${\widehat{\alpha }}_{b_{k+1}}^{b_k},{\widehat{\beta }}_{b_{k+1}}^{b_k},{\widehat{\gamma }}_{b_{k+1}}^{b_k}$ are corrected with the new gyroscope bias.

The second is the initialization of velocity, gravitational acceleration, and scale. The initialized state vector is as follows:

$$\begin{array}{c}{\chi }_I=\left[v_{b_0}^v,v_{b_1}^v,\dots v_{b_n}^v,{\mathbf{g}}^v,s\right],\end{array}$$ (42)

where the state vector v_bk^v represents the speed of the visual coordinate system of the kth frame image, g^v represents the gravity vector in the visual coordinate system, and s is the estimated scale parameter. To sum up, the dimension of χ_I is 3(n + 1) + 3 + 1. The constraint relationship between the scale parameter and the speed of the visual SFM is as follows:

$$\begin{array}{c}{\widehat{z}}_{b_{k+1}}^{b_k}=\left[\begin{array}{c}{\widehat{\alpha }}_{b_{k+1}}^{b_k}\\ {\widehat{\beta }}_{b_{k+1}}^{b_k}\end{array}\right]={\mathbf{H}}_{b_{k+1}}^{b_k}{\chi }_I+{\mathbf{n}}_{b_{k+1}}^{b_k}\approx ,\\ \left[\begin{array}{cccc}-q_v^{b_k}\mathrm{\Delta }{t}_k& 0& \frac{1}{2}{q}_v^{b_k}\mathrm{\Delta }{t}^2& q_v^{b_k}\left({\overline{p}}_{b_{k+1}}^v-{\overline{p}}_{b_k}^v\right)\\ -q_v^{b_k}& q_v^{b_k}& q_v^{b_k}\mathrm{\Delta }{t}_k& 0\end{array}\right]\left[\begin{array}{c}{v}_{b_k}^v\\ v_{v_{k+1}}^{g^v}\\ g^v\end{array}\right],\end{array}$$ (43)

where q_bk^V, q_bk^V, q_bk^V are all obtained from visual SFM. q_V^b_k and q_bk^V are mutually inverse matrices. The following linear least squares problem is constructed to complete the initialization of velocity, gravitational acceleration, and scale:

$$\begin{array}{c}{\min }_{{\chi }_I}{\displaystyle \sum _{k\in \mathrm{ℬ}}{‖{\widehat{z}}_{b_{k+1}}^{b_k}-{\mathbf{H}}_{b_{k+1}}^{b_k}{\chi }_I‖}^2}.\end{array}$$ (44)

2.6. Monocular Visual Inertial Coupling Nonlinear Optimization

When coupling the visual constraint value and the IMU constraint value, the data of the inertial sensor should be introduced first, and the constraint value of the IMU on the state should be added to the optimized state vector. Then, nonlinear optimization is performed within a sliding window, and all state vectors of the sliding window are as follows:

$$\begin{array}{c}\chi =\left[{\mathbf{x}}_0,{\mathbf{x}}_1,\dots {\mathbf{x}}_n,{\mathbf{x}}_c^b,{\lambda }_0,{\lambda }_1,\dots {\lambda }_m\right],\\ {\mathbf{x}}_k=\left[{\mathbf{p}}_{b_k}^w,{\mathbf{v}}_{b_k}^w,{\mathbf{q}}_{b_k}^w,{\mathbf{b}}_a,{\mathbf{b}}_g\right],k\mathrm{ϵ}\left[0,n\right],\\ {\mathbf{x}}_c^b=\left[{\mathbf{p}}_c^b,{\mathbf{q}}_c^b\right],\end{array}$$ (45)

where x_k represents the state of the IMU when the kth image is captured. There are n + 1 states in the sliding window, and each state contains the position, velocity, and rotation in the world coordinate system, and the IMU offsets in the IMU coordinate system. λ_m represents the inverse depth information of the mth 3D point, and x_c^b represents the external parameter from the camera to the IMU. The objective function is

$$\begin{array}{c}{\min }_{\chi }\left\{{‖r_p-{\mathbf{H}}_p\chi ‖}^2+{\displaystyle \sum _{k\in \mathrm{ℬ}}{‖r_{\mathrm{ℬ}}\left({\widehat{\mathbf{z}}}_{b_{k+1}}^{b_k},\chi \right)‖}_{{\mathbf{p}}_{b_{k+1}}^{b_k}}^2}+{\displaystyle \sum _{\left(l,j\right)\text{ó}C}\rho \left({‖r_C\left({\widehat{z}}_l^{c_j}\right)‖}_{{\mathbf{p}}_l}^2{c}_j\right)}\right\},\end{array}$$ (46)

where H_p is the Huber norm, which is defined as follows:

$$\begin{array}{c}\rho \left(s\right)=\left\{\begin{array}{c}1,s\ge 1,\hfill \\ 2\sqrt{s}-1,s<1.\hfill \end{array}\right.\end{array}$$ (47)

In formula (52), ‖·‖_P represents the Mahalanobis distance weighted by the covariance matrix P, and $\left\{r_p,{\mathbf{H}}_p\right\},r_B\left({\widehat{z}}_{b_{k+1}}^{b_k},X\right)$, and $r_C\left({\widehat{z}}_l^c,X\right)$ represent the marginalized prior information, the IMU measurement residual, and the visual reprojection error, respectively. ℬ is the set of all IMU measurement frames, and 𝒞 is the set of visual features in the sliding window.

According to the Gauss–Newton method, the incremental method can be used to calculate the minimum value of the Gaussian objective function, as follows:

$$\begin{array}{c}{\min }_{\delta X}{‖r_{\mathrm{ℬ}}\left({\widehat{\mathbf{z}}}_{b_{k+1}}^{b_k},\chi +\mathrm{\Delta }X\right)‖}_{{\mathbf{p}}_{b_{k+1}}^{b_k}}^2={\min }_{\delta X}{‖r_{\mathrm{ℬ}}\left({\widehat{\mathbf{z}}}_{b_{k+1}}^{b_k},\chi \right)+J_{b_{k+1}}^{b_k}\mathrm{\Delta }X‖}_{{\mathbf{p}}_{b_{k+1}}^{b_k}}^2,\end{array}$$ (48)

where J_bk+1^b_k is the Jacobian matrix of the error term r_ℬ with respect to all state vectors χ.

The algorithm differentiates the above formula from ΔX and then sets its derivative to 0, and the formula for the increment ΔX can be calculated as follows:

$$\begin{array}{c}{J}_{b_{k+1}}^{b_{k}T}{P}_{b_{k+1}}^{b_k-1}{J}_{b_{k+1}}^{b_k}\mathrm{\Delta }X=-J_{b_{k+1}}^{b_{k}T}{P}_{b_{k+1}}^{b_k-1}{J}_{b_{k+1}}^{b_k}{r}_{\mathrm{ℬ}}.\end{array}$$ (49)

The overall incremental equation of the objective function is as follows:

$$\begin{array}{c}\left({\mathbf{H}}_p+{\displaystyle \sum J_{b_{k+1}}^{b_{k}T}{P}_{b_{k+1}}^{b_k-1}{J}_{b_{k+1}}^{b_k}}+{\displaystyle \sum J_l^{c_j^T}{P}_l^{c_j-1}{J}_l^{c_j}}\right)\mathrm{\Delta }X=b_p+{\displaystyle \sum J_{b_{k+1}}^{b_{k}T}{P}_{b_{k+1}}^{b_k-1}{r}_{\mathrm{ℬ}}}+{\displaystyle \sum J_l^{C_j^T}{P}_l^{C_j^{-1}}{r}_C},\end{array}$$ (50)

where P_bk+1^b_k represents the covariance of the IMU preintegrated noise term and P_l^C_j represents the visually observed noise covariance. When the noise covariance P_bk+1^b_k of the IMU is larger, the inverse of P_bk+1^b_k, that is, its information matrix is smaller, which means that the IMU observations are not as reliable as the visual observations. Formula (50) can be simplified to

$$\begin{array}{c}\left({\mathrm{\Lambda }}_p+{\mathrm{\Lambda }}_B+{\mathrm{\Lambda }}_C\right)\mathrm{\Delta }X=b_p+b_B+b_C,\end{array}$$ (51)

where Λ_p, Λ_B, and Λ_C represent the Hessian matrix. Using the perturbation method to calculate, we can get

$$\begin{array}{c}J=\frac{\partial r}{\partial X}\\ =\underset{\delta X⟶0}{\lim }\frac{r\left(X\oplus \delta X\right)-r\left(X\right)}{\delta X},\end{array}$$ (52)

where δX represents the small disturbance of the state vector instead of the incremental ΔX, ⊕ represents the disturbance of the state vector.

The continuous preintegration formula is derived in the IMU preintegration, and the IMU residuals are as follows:

$$\begin{array}{c}{r}_{\mathrm{ℬ}}\left({\widehat{z}}_{b_{k+1}}^{b_k},\chi \right)=\left[\begin{array}{c}\delta {\alpha }_{b_{k+1}}^{b_k}\\ \delta {\beta }_{b_{k+1}}^{b_k}\\ \delta {\theta }_{b_{k+1}}^{b_k}\\ \delta b_a\\ \delta b_g\end{array}\right]\\ =\left[\begin{array}{c}{R}_w^{b_k}\left(p_{b_{k+1}}^w-p_{b_k}^w-v_{b_k}^w\mathrm{\Delta }{t}_k+\frac{1}{2}{g}^w\mathrm{\Delta }{t}_k^2\right)-{\alpha }_{b_{k+1}}^{b_k}\\ R_w^{b_k}\left(v_{b_{k+1}}^w\mathrm{\Delta }{t}_k-v_{b_k}^w\mathrm{\Delta }{t}_k+g^w\mathrm{\Delta }{t}_k\right)-{\beta }_{b_{k+1}}^{b_k}\\ 2{\left[{\gamma }_{b_{k+1}}^{b_k}\otimes q_{b_k}^w\otimes q_{b_{k+1}}^w\right]}_{xyz}\\ b_{a_{b_{k+1}}}-b_{a_{b_k}}\\ b_{{\omega }_{b_{k+1}}}-b_{{\omega }_{b_k}}\end{array}\right].\end{array}$$ (53)

According to the above formula, it can be seen that the optimization variables of the IMU residual mainly include the position, rotation, speed, and inertia bias at the i and j times:

$$\begin{array}{c}\left[p_{b_k}^w,q_{b_k}^w\right],\left[v_{b_k}^w,b_{a_k},b_{{\omega }_k}\right],\left[p_{b_{k+1}}^w,q_{b_{k+1}}^w\right],\left[v_{b_{k+1}}^w,b_{a_{k+1}},b_{{\omega }_{k+1}}\right].\end{array}$$ (54)

To calculate the Jacobian matrix, perturbation is added to each optimization variable to obtain

$$\begin{array}{c}\left[\delta p_{b_k}^w,\delta {\theta }_{b_k}^w\right],\left[\delta v_{b_k}^w,\delta b_{a_k},\delta b_{{\omega }_k}\right],\left[\delta p_{b_{k+1}}^w,\delta {\theta }_{b_{k+1}}^w\right],\left[\delta v_{b_{k+1}}^w,\delta b_{a_{k+1}},\delta b_{{\omega }_{k+1}}\right].\end{array}$$ (55)

Taking the partial derivatives for the above disturbance variables, respectively, we can get

$$\begin{array}{c}J{\left[0\right]}^{15\times 7}=\left[\frac{\partial r_B}{\delta p_{b_k}^w},\frac{\partial r_B}{\delta {\theta }_{b_k}^w}\right],\\ J{\left[1\right]}^{15\times 7}=\left[\frac{\partial r_B}{\delta v_{b_k}^w},\frac{\partial r_B}{\delta b_{a_k}},\frac{\partial r_B}{\delta b_{{\omega }_k}}\right],\\ J{\left[2\right]}^{15\times 7}=\left[\frac{\partial r_B}{\delta p_{b_{k+1}}^w},\frac{\partial r_B}{\delta {\theta }_{b_{k+1}}^w}\right],\\ J{\left[3\right]}^{15\times 7}=\left[\frac{\partial r_B}{\delta v_{b_{k+1}}^W},\frac{\partial r_B}{\delta b_{a_{k+1}}},\frac{\partial r_B}{\delta b_{{\omega }_{k+1}}}\right].\end{array}$$ (56)

The visual residual is a reprojection error, which represents the difference between the estimated value and the observed value of the feature point in the normalized camera coordinate system. The small receiver camera used in this paper belongs to the fisheye camera model and belongs to the fisheye with a large viewing angle, so its projection on the unit sphere needs to be considered, as shown in Figure 9.

Figure 9.

Unit spherical projection model.

Through the unit spherical projection model illustrated in the figure above, the value of the visual residual is decomposed into two directions. The final visual residual model looks like this:

$$\begin{array}{c}{r}_c\left({\widehat{z}}_l^{c_j},\chi \right)={\left[b_1,b_2\right]}^T·\left({\widehat{p}}_l^{c_j}-\frac{p_l^{c_j}}{‖p_l^{c_j}‖}\right),\\ {\widehat{p}}_l^{c_j}={\pi }_c^{-1}\left(\left[\begin{array}{c}{\widehat{u}}_l^{c_j}\\ {\widehat{v}}_l^{c_j}\end{array}\right]\right),\\ p_l^{c_j}=R_c^b\left\{R_w^{b_j}\left[R_{b_i}^w\left(R_c^b\frac{1}{{\lambda }_l}{\overline{P}}_l^{c_i}+p_c^b\right)+p_{b_i}^w-p_{b_j}^w\right]-p_c^b\right\},\end{array}$$ (57)

where ${\widehat{\overline{p}}}_l^{c_j}$ and p_l^c_j represent the estimated and observed coordinates of the landmark point 1 in the j-th frame image under the normalized camera coordinate system, respectively. The optimization variables of the visual residual are as follows:

$$\begin{array}{c}\left[p_{b_i}^w,q_{b_i}^w\right],\left[p_{b_j}^w,q_{b_j}^w\right],\left[p_c^b,q_c^b\right],{\lambda }_l,\end{array}$$ (58)

where λ_l represents the inverse depth value when the landmark point 1 is first observed by the j-th image. The inverse depth is used as the optimization variable because the inverse depth satisfies the Gaussian distribution, and it can reduce the parameter variables in the actual optimization process. According to the above formula, by adding disturbance to each optimization variable, the following Jacobian is obtained:

$$\begin{array}{c}J{\left[0\right]}^{3\times 7}=\left[\frac{\partial r_C}{\partial p_{b_i}^w},\frac{\partial r_C}{\partial q_{b_i}^w}\right],\\ \\ J{\left[2\right]}^{3\times 7}=\left[\frac{\partial r_C}{\partial p_c^b},\frac{\partial r_C}{\partial q_c^b}\right],\end{array}$$ (59)

$$\begin{array}{c}J{\left[3\right]}^{3\times 1}=\frac{\partial r_C}{\partial {\lambda }_l}.\end{array}$$ (60)

3. E-Sports Training System Based on Intelligent Gesture Recognition

This paper combines the finger joints and the sensor in the data glove to demarcate the movement of the finger joints. This paper mainly considers the distal phalanx of the thumb (TDP) and the proximal joint proximal phalanx of the thumb (TPP) of the thumb as shown in Figure 10 and the changes in the intermediate joints middle phalanges (MP) and proximal phalanges (PP) of the remaining four fingers.

Figure 10.

Demarcation boundaries of hand movements.

This paper combines the algorithm part of the second part to construct the e-sports training system, and the overall framework of the system is shown in Figure 11.

Figure 11.

Overall system framework.

The simulation of the system proposed in this paper is carried out through the MATLAB platform, and the gesture recognition effect of the system and the application effect in the e-sports training system are evaluated, and the obtained results are shown in Tables 1 and 2.

Table 1.

Gesture recognition effect of the system.

Number	Gesture recognition	Number	Gesture recognition	Number	Gesture recognition
1	86.095	13	87.363	25	86.633
2	88.782	14	89.124	26	90.414
3	90.695	15	90.055	27	86.697
4	86.325	16	89.968	28	86.916
5	86.705	17	88.482	29	89.003
6	89.557	18	91.278	30	88.446
7	86.623	19	89.215	31	89.911
8	91.281	20	86.272	32	90.202
9	88.050	21	91.253	33	89.507
10	87.621	22	86.770	34	88.130
11	88.498	23	86.843	35	91.326
12	86.304	24	90.614	36	86.846

Table 2.

The application effect of the method proposed in this paper in the e-sports training system.

Number	Training effect	Number	Training effect	Number	Training effect
1	82.685	13	82.234	25	79.365
2	80.315	14	78.537	26	81.356
3	79.700	15	81.913	27	78.790
4	78.179	16	82.938	28	79.700
5	82.942	17	78.167	29	78.194
6	80.821	18	78.250	30	78.253
7	81.176	19	81.512	31	80.133
8	78.183	20	80.076	32	82.668
9	82.636	21	81.925	33	83.747
10	81.613	22	82.857	34	82.354
11	80.514	23	82.951	35	78.416
12	80.913	24	79.066	36	83.128

It can be seen from the above research that the electronic economic training system based on intelligent gesture recognition proposed in this paper has certain effects.

4. Conclusion

As emerging sports, e-sports are mainly participated by the younger generation, which has the characteristics of younger and younger age. E-sports can exercise people's thinking ability, psychological pressure resistance, unity and cooperation, hand-eye coordination, and so on. Moreover, in the process of participating in e-sports, it can also make the younger generation have the awareness of abiding by the rules, cultivate the participants to have a fair and open, never admit defeat, pursue a stronger competitive spirit, and have a positive impact on the participants' lives. This paper combines the intelligent gesture recognition technology and the construction of the performance e-sports training system and judges the training effect of the players through the player gesture recognition. The research shows that the electronic economic training system based on intelligent gesture recognition proposed in this paper has certain effects.

Data Availability

The labeled data set used to support the findings of this study is available from the corresponding author upon request.

Conflicts of Interest

The authors declare that there are no conflicts of interest.