Detecting Gait Phases from RGB-D Images Based on Hidden Markov Model

Abstract

Gait contains important information about the status of the human body and physiological signs. In many medical applications, it is important to monitor and accurately analyze the gait of the patient. Since walking shows the reproducibility signs in several phases, separating these phases can be used for the gait analysis. In this study, a method based on image processing for extracting phases of human gait from RGB-Depth images is presented. The sequence of depth images from the front view has been processed to extract the lower body depth profile and distance features. Feature vector extracted from image is the same as observation vector of hidden Markov model, and the phases of gait are considered as hidden states of the model. After training the model using the images which are randomly selected as training samples, the phase estimation of gait becomes possible using the model. The results confirm the rate of 60–40% of two major phases of the gait and also the mid-stance phase is recognized with 85% precision.

INTRODUCTION

Nowadays, human gait is a popular subject for several research projects. The studies of movement analysis began in the 19^th century for the first time. The recent studies focus on the achievement of objective and quantitative measurement of human movement parameters. Analysis and studies of the gait mechanism are carried out with different incentives. In a general classification, the existing incentives for the human movement analysis are classified into two general categories: individual identification for security goals and hiring in different fields including sport and medical analyses. Bouchrika et al.[1] present a solution based on car vision to people tracking in images using the gait information. Focusing on medical field, the changes of gait present key information about people's quality of life.

In clinical conditions, the gait analysis is traditionally conducted by specialists through semi-subjective approaches and patients’ gait observation. In this method, the specialists specify the gait quality by walking observation. Therefore, sometimes patients are asked to present a subjective assessment of their gait quality. The advantage of this method is no need for specific equipment and only a trained specialist is required for the test. However, the disadvantage is the negative impact on diagnosis, tracking, and treatment of damages due to subjective measurements particularly in terms of accuracy and precision. Besides, it influences subjective nature, precision assessment, accuracy, repeatability, and reproducibility of measurements.

Recent advances in new technologies have developed some equipment and techniques allow objective assessment of the gait. These methods present more precise data for the assessment and make this possible to obtain information which is not readily obtained by the gait observation. Consequently, patients are provided with more efficient measurement and authoritative information and the error due to subjective techniques are decreased. Movement analysis has attracted attentions since centuries age. Aristotle is one of the pioneers in scientific movement analysis (animals).[2] In 1680, Alfonso Borelli offered a qualitative analysis of human movement. In 1890, a German anatomist “Wilhelm Braune,” in partnership with his colleagues, published their researches, resulted in human movement biomechanics in loaded and loadless conditions as a papers collection.[3]

With advances in imaging and filming technology, it has been possible to provide consecutive images from human movement process. This led to discovering some hidden aspects of biomechanics of human movement, which could not be realized by naked eyes. Muybridge and Marey are two pioneers of using this technique in the movement analysis.[4]

Studies of human gait include qualitative and quantitative assessment of different factors which make walking possible. According to the research field, the noteworthy factors for studies are different.[5] For example, for the sake of security goals, people detection and identification may be emphasized based on general specifications, the person's silhouette, and motion of different organs while walking. However, in sport fields, studies may focus on the analysis of various forces applied on each muscle through electromyogram signals.

In medical and clinical applications, lots of gait phases are required to be analyzed and assessed. Therefore, to distinguish natural waking from unnatural one, study of gait phases and their duration is necessary.[6] Figure 1 shows different gait phases.

Figure 1.

Different gait phases. q1 – HeelStrike; q2 – FootFlat; q3 – Mid-stance; q4 – Pushoff; q5 – Acceleration; q6 – Mid-Swing; q7 – Deceleration phase

Nowadays, the advances of new technologies have developed some equipment and techniques which allow objective assessment of diverse parameters of the gait. As a result, more efficient measurements and authoritative information are provided for patients, and the error due to subjective techniques is decreased. Technical equipment which are used for subjective study of human gait are classified according to two different approaches as follows:[7]

1. Based on nonwearable sensors (NWSs)

2. Based on WSs.

NWS systems use controlled research facilities. In these systems, after mounting sensors, the data related to the gait are logged while walking in a specific pathway. In contrast, WS systems allow analyzing data outside the laboratory environment and saving the information associated with the human gait during routine activities. Furthermore, there is a third group of hybrid systems which use a combination of both methods.

Signal modeling is one of the methods used in signal processing. Features and specification of a signal can be modeled and described by several approaches. In a general classification, signals can be categorized into two models including deterministic models and statistical models. The former[8] employs some specific and known attributes of a signal in the model. This model's advantage is simplicity and simple relationship between using features to describe a signal while its disadvantage is that it cannot describe complicated signals with hidden relationship among signal features. In the latter, a model is tried to be addressed that can describe statistical attributes of the signal with a reasonable approximation. Methods based on the deterministic model have limited use. However, methods based on statistical models are some multivariate concepts which make a mapping from input to output with no restriction in terms of inputs. Machine learning is a field of computational intelligence including statistical rules to design and develop algorithms which let a computer learn statistical models and experimental data.[9] This kind of learning can be useful in a set where wide data exist and thereby the pattern perception and model making is difficult to forecast them. Furthermore, this concept provides a method for learning through data and divides data into two categories including training data and experimental data. The advantages are summarized as follows.

This concept can make single and specific model per person. Since the patterns associated with the gait is variable for different people, the use of distinct models has better compatibility in the gait analysis compared to making a general model for all people. There are some machine-learning methods which are used to analyze the gait including feed-forward neural network, perceptron neural network, K-means, neuro-fuzzy, hidden Markov model (HMM), and support vector machine. Table 1 reviews some papers associated with the gait analysis where machine-learning methods combined with image processing are employed.

Table 1.

Some studies in gait analysis using computer learning approaches

PROPOSED SYSTEM

To extract the gait phases, primarily, it is needed to select a technique among the introduced ones to log the gait. Here, the main approach is based on image processing using RGB-Depth (RGB-D) images and the corresponding processing to extract the human's gait phases. Use of RGB-D images is preferred compared to colored imaging due to have different and supplementary information of a scene. Hence, in the present study, a Kinect sensor is chosen as imaging mean. In general, it can be argued that selection of RGB-D image processing technique for this purpose has its specific advantages and disadvantages. On the one hand, we can refer to low-cost basic equipment and lack of need to install equipment or marker on patient's body. On the other hand, in contrast to WSs, low operating range can be pointed as disadvantage of the techniques based on image processing.

Predetermined space for subjects’ walking was planned in 4 m long and 2 m width. Hereafter, the predetermined space is called pathway. The length of pathway was selected according to acceptable precision of depth imaging by Kinect sensor; however, the width of pathway can be longer. Figure 2 shows subjects’ trajectories in the proposed system.

Figure 2.

Subjects’ trajectories in proposed system

In this section, an algorithm is introduced base on phases of the gait which can be extracted through the logged video sequences. Figure 3 shows the general block diagram of the proposed algorithm separated by constituents.

Figure 3.

Block diagram of different stages of system

Loading Frames and Preprocess

After obtaining the video sequences by the Kinect sensor which was used in gait physics logging system, the obtained video will enter the processed algorithm and will be processed in offline form. At the next step, the unsuitable area should be removed from the image. The suitable area is the predetermined pathway for the subjects’ gait. The depth images obtained by Kinect sensor are noisy. Noise reduction in an image can play an important role in improving the efficiency of the proposed method. Therefore, the next step in preprocess is reducing the noise existing in the depth images. Noise reduction is carried out through two single operations. The first operation includes filling holes of the depth image using median filter of 8 × 8 pixel. The second one is performed by applying Gaussian filter at the desired size for image. Although the addressed noise reduction methods cannot completely eliminate the noises of depth image at long distance from the sensor, they can significantly reduce such noises.

Feature Extraction

In this part, we used the improved depth images at the preprocessing stage to obtain the appropriate feature.

Although the extracted feature vector is simple, it shows good resolution at different phases of gait. N sampling points of the skeleton of two feet affect the size of feature vector. Size of feature vector is 120 in this study. Size of the vector can be less or more than this size. Size of feature vector is impressive in describing the image and the observation vector and therefore, it makes the model of the problem bulkier. To clear this, Figure 4 displays the vector extracted at different phases of gait.

Figure 4.

Sampled points on legs in depth image and corresponding extracted feature vectors in different gait phases (a) HeelStrike, (b) FootFlat, (c) Mid-stance and (d) Pushoff

At the first step, we will segment the moving person image by applying a threshold level on the depth image and morphological operation. At the next step, with the help of human joints information that is obtained from Kinect sensors, the lower body part of the person's image will be cut. Area of image that includes the lower body of the person has been segmented by the left hip and right hip, and four lower points on the skeletal information of body. The connection between three mentioned points for each leg brings the skeleton of each leg. The existence of two corresponding points on the obtained skeleton of two legs in the horizontal direction of the image is enough to calculate the distance feature.

Consequently, the lines connecting three points of each leg, consisting hip, knee, and ankle, are used for calculating distance vector. Obtained lines or equivalent feet's skeletons in-depth image are examined. Every point on the feet skeleton has a specific distance to the camera in depth image. Approximate distance of all points on feet skeleton is determined using depth image of Kinect sensor. However, the number of points, constructing the lines of leg's skeleton, are too many and having redundant information. Therefore, to extract the proper feature, we select N points, equally on each leg. Figure 5 shows different stages of system and the corresponding result of each stage. Stage c-f shows feature extraction process.

Figure 5.

Different stages of system. (a) Input RGB-Depth images. (b) Preprocessed image. (c-e) Feature extraction process. (f) Extracted feature vector

The number of sampling points is equal on both feet. In addition, they are sampled in the distance between hip and ankle. Therefore, they are representative of the same points on both legs. Using distance of representative points from each other, the “distance feature” is determined. Considering this feature for all of the points provides a feature vector with the length of N. In this study, the right foot is used as the basis for calculating the distance at all stages of feature extraction. In other words, L̄R̄ vector contains the depth of N sampling points of the right leg's skeleton and L̄L̄vector contains the depth of N sampling points of the left leg's skeleton and vector distance is called V̄_i. V̄_i is obtained for each frame of the image using Eq. 1:

Figure 4 sampled points on legs in-depth image in different gait phases.

Classification

Here, HMM was employed to separate different gait phases. HMM is a powerful statistical solution to model time series. Human gait cycle is also an iterative time series. HMM due to relation between plans available classes and simplicity in the calculation to identify human movement phases, will be more proper in comparison with other methods.

Since subjects’ gait process is carried out in controlled conditions, the first gait phase is clear. Therefore, the probability distribution of the initial state for the start mode (phase) is set to one, and other probabilities are set to zero.

According to the above-mentioned contents, an HMM can be specified by discrete probability distribution by means of three parameters. Equation (2) shows an HMM with three described parameters:

λ=(Λ, B, π)      (2)

where Λ is the probability distribution matrix of the state transitions:

Λ={a_ij}_Q×Q a_ij = P(X(t_n)=S_i|X(t_n+1)=S_j)      (3)

B is the probability distribution matrix related with the M-size set of the Y observations at the state S_i:

B={b_ij}_Q×N b_ij = P (Y(t_n)=Y_j|X(t_n)=S_i)      (4)

with switching to a continuous HMM (cHMM), the preceding probability can be computed assuming a normal distribution:

where w_jk is, for each state j, a mixture coefficient, weighting K multivariate normal distributions N, with mean µ_jk.

π is the initial state vector distribution:

π = {π_i}_Q×1 π_i = P(Y(t₀) = S_i)      (6)

In this paper, HMM is used in two stages. The first stage is to collect training data to calculate λ parameters. To calculate the HMM parameters, the well-known Baum–Welch algorithm is employed.[20] The second stage is based on the model parameters obtained from the first stage, and it results in final classification. This stage is also known as decoding problem. Therefore, the probability q_i shall be maximized if the observations vector (the features extracted from the image depth) is observed in per image frame Eq. 3.

S_i =arg max_qi P (qi|{o₁,o₂,…,o_M})1≤i≤N      (7)

Actually, the most likely obtained q_i through maximizing Eq. 7 is the estimation of the gait phase (S_i) by HMM. Viterbi algorithm applies an effective way for mode estimation base on observations. Therefore, the second stage can estimate one of the possible modes in the gait phase base on extracted features in each frame of video. In common, the algorithm includes three steps that are given as follows:

Step of initialization:

Step of recursion:

γ_tn (i)=max[γ_tn-1(i)a_ij]b_i(Y(t_n))      (9)

Step of estimating likely sequence:

l_tn=arg max[γ_tn (i)]      (10)

Gait sequence is generally divided into several gait phases. Figure 1 shows the former phases of Gait cycle, which signify the hidden states of here implemented cHMM. In normal gait, walking phases occur following the above-reported sequence. State transitions follow a left to right model and therefore, transition matrix A, implemented in this study is:

Since transitions are very quick with respect to the gait cycle, they are less frequent in the current state sequence; thus, diagonal elements assume higher values than the others,[13] and Figure 6 shows specifications of the selected HMM and possible transitions among gait phases are reported.

Figure 6.

Specifications of the selected hidden Markov model in order to distinguish the gait phases

Because the initial state of the model at time t₀ is not identified, its phase can be selected as:

This shows that each state has the similar probability of being the first in a state sequence.

DISCUSSION

To assess proposed algorithm, the front view of RGB-D images of gait is required. There are several databases of gait mostly possessed by only colored images from different views.[12,22] Since our desired images for process are colored-depth ones, the images database RGB-D was assessed with more focus. The mentioned database is composed of concurrent colored-depth images of 12 people, consist of 10 men and 2 women walking toward the camera with two speed rates including normal and quick. Imaging and predefined pathway disciplines were compatible with roles in Figure 2.

After training HMM by the corresponding data for a leg, test images were evaluated following extraction of selected features and applied to Markov classifier. In this stage, using HMM and according to obtained parameters through training stage and also Viterbi algorithm, each input frame will be mapped into a mode (gait phases). In fact, the diagnosed mode by classifier estimates the gait phase for a leg. Figure 7 illustrates block diagram for the phase estimates stages by HMM. Each HMM mode addresses a gait phase. By calling the depth image frames and applying extracted features into HMM, one of the model modes will be activated, and other modes will be kept passive. Becoming active or passive by the model modes acts like a logical switch.

Figure 7.

Block diagram for estimation of the gait phase stages

Per frame, the pretrained model will present an estimation of existing mode. This process proceeds until image frames finish and it will follow the limited cycle between gait phases.

The model modes switches become active/passive during captured frames. The comparison between the model output in a subject's gait sequence and its base output shows the estimation accuracy. The base output or ground truth is available for all frames, and the success of model is calculated based on correct/incorrect distinguishes of the gait phases toward the base. To assess the results of separation of the gait phases more accurately using Markov model, the following criteria are applied:

Accuracy rate is introduced by “Pre” parameter and obtained from Eq. 13:

Sensitivity or Recall

Sensitivity or recall is the ratio of the number of image frames which phase diagnosed correctly to the total number of diagnosed for that phase. The recall calculates based on Eq. 14.

P value: To determine the significance of obtained values statistically, t-tests were employed for each phase of gait. The P value indicates the probability of accidentally extracted data. Table 2 reports obtained mean and standard deviation for total “Pre” and “Rec.”

Table 2.

Obtained values for total accuracy rate and Recall

In a comprehensive classification, two primary phases are considered for the gait phases including stance and swing. If the total number of identified frames by classifier is calculated for each primary phase and all subjects, the ratio of frames number of two primary phases in a complete gait cycle can be obtained. In References [2,14] and other references, the natural ratio of primary gait phases is addressed as 40% or 60% for a complete cycle. In this regard, the comparison of ratios of two primary phases (the results of classifier and base) is presented in Table 3.

Table 3.

The comparison on ratios for two primary gait phases

CONCLUSION

The present paper tried to assess gait measurement and analysis tools. In addition, the new and emerging technology “Kinect” was employed to analyze the gait; this technology is one of the hot topics of researches related to machine vision. In the proposed system for detecting the gait phases, depth data were used to extract the feature vector. Furthermore, the size of extracted feature vector was N. Ultimately, the statistical HMM allowed gait phases classification after training. The simulation results indicate that the proposed algorithms can be effective for human's gait analysis.

In the gait physics system and the regarding proposed algorithm, fault rate of the phase distinguishing is high at initial frames. This fault is impacted by the depth imaging quality during long distances. The depth images in long distances contain higher noises compared to the images obtained close to the sensor. The closer subject to the sensor, the lower fault probability at the classifier output. The results indicate that Kinect sensor and/or time-of-flight cameras can be useful for this application. The advantage of depth imaging systems compared to stereo imaging is their low calculation processing load. With the technology advances, imaging cameras’ cost is going to be decreased, and such an equipment can considerably reduce the cost, and thus, enhance results quality compared to similar systems.

Financial Support and Sponsorship

Nil.

Conflicts of Interest

There are no conflicts of interest.