OptiGait: Gait Monitoring Using An Ankle-Worn Stereo Camera System

Abstract

We developed an ankle-worn gait monitoring system for tracking gait parameters, including length, width, and height. The system utilizes ankle bracelets equipped with wide-angle infrared (IR) stereo cameras tasked with monitoring a marker on the opposing ankle. A computer vision algorithm we have also developed processes the imaged marker positions to estimate the length, width, and height of the person’s gait. Through testing on multiple participants, the prototype of the proposed gait monitoring system exhibited notable performance, achieving an average accuracy of 96.52%, 94.46%, and 95.29% for gait length, width, and height measurements, respectively, despite distorted wide-angle images. The OptiGait system offers a cost-effective and user-friendly alternative compared to existing gait parameter sensing systems, delivering comparable accuracy in measuring gait length and width. Notably, the system demonstrates a novel capability in measuring gait height, a feature not previously reported in the literature.

Graphical Abstract

I. Introduction

Gait can be viewed as an automatic movement [1] that requires an intact musculoskeletal function controlled by multiple neural pathways and brain areas. The abnormality of gait [2] is a common disorder that could cause severe injuries and reduce an individual’s quality of life and ability to perform activities of everyday living owing to a reduction in mobility and an increased dependence on external care [3]. Researchers are paying increasing attention to gait abnormalities since a rising number of the population worldwide are facing them due to aging or medical conditions, such as multiple sclerosis [4], cerebral palsy [5], stroke [6], Parkinson’s disease (PD) [7], and other age-related diseases. For example, it has been reported that, by 2030, the number of people older than 45 years with PD in the US will increase to approximately 1,238,000 [8]. Therefore, the early detection of gait abnormalities could benefit a large portion of the population.

The ability to measure and monitor important gait parameters, such as gait length, width, and height [9], over different phases of gait is an important clinical tool for diagnosing an individual’s gait abnormalities. Gait parameters allow clinicians to assess the cause of the displayed gait abnormalities and provide reliable metrics for intervention strategies for long-term conditions. Clinicians typically use simple diagnostic methods in a hospital setting to evaluate gait abnormalities such as visual inspection, patient self-assessment surveys [10], or simple descriptors such as using a specific walked distance divided over the total walking time to obtain gait speed. These approaches largely depend on the clinicians’ personal experience and knowledge. Subtle changes in gait parameters are difficult and nearly impossible to detect since they are usually not sensitive to human eyes. With the current advancement in technology, it is now possible to enable a more accurate and data-driven approach for monitoring gait parameters to assist with expert analysis.

Current commercial technologies for gait monitoring in clinical settings, such as inertial measurement units (IMUs), pressure walkways or walkways with photoelectric cell bars, and optical motion capture systems, suffer from many disadvantages [11][12]. These systems are prohibitively expensive, have substantial hardware complexity, and are often not user-friendly. Commercial IMUs [13] range in cost from 600 USD [14] to 3,900 USD [15] and are only able to detect gait length. Moreover, manual analysis of a large number of observed IMU signals is needed since it is difficult to directly correlate the signals to a known gait characteristic [9][16][17]. Other problems include periodically repeated drift or change, which introduces bias or offset in readings [13]. Similarly, floor sensing products, such as gait pressure measurement mats [18] or walkway systems with photoelectric cell bars [19], priced at around 25,000 USD [20], are cumbersome and limited in their spatial resolutions on gait length and width. Besides, the analysis of gait is limited to a few steps in a restricted space, resulting in missed gait patterns. Optical motion capture systems require the use of multiple camera systems, complex machine learning algorithms, and powerful computational resources [9]. These kinds of optical systems [21]–[24] are usually limited to highly structured research environments. It requires manual intervention to fix mismatched feature points to accurately describe the human skeleton [25] and requires participants to walk perpendicularly at a fixed depth relative to the camera [12]. It is also challenging to apply such systems because of the insufficient estimation of subtle gait movement [26]. As a result, it is difficult to measure gait width and height using these methods.

In general, existing works cannot simultaneously measure a) temporal-spatial properties of the gait, such as gait length, width, height, and time; b) gait events, such as initial contact with the floor or toe-off; c) and the gait cycle, including the standing and swing time. They do not meet the need of clinical assessments of gait abnormality, which require a simple and low-cost process to provide quantitative information on the patient’s status.

We propose a new method for the measurement of gait length, width, and height that allows practitioners to supplement the existing measurement setups in a convenient manner, and demonstrate a prototype. As shown in Fig. 1, our system uses ankle-worn stereo cameras for infrared (IR) imaging and on-body computer vision processing of the captured IR images to calculate the gait parameters. Combined with the ArUco markers [27] (an open-source library for camera pose estimation) and image undistortion algorithm, the large field-of-view (FoV) cameras are capable of catching the ArUco markers in a wide scene and enabling the measurement of gait height even while the person is walking up a flight of stairs. Using undistorted IR images keeps the high accuracy while significantly reducing the amount of captured visual information, allowing for the preservation of privacy. The novel device we propose is low-cost (less than 200 USD), efficient during measurement, easy to use, suitable for real-time measurement, and enables an on-body computer vision process, all while providing competitive accuracy compared with state-of-art technology.

Fig. 1.

(a) Image of the ankle-worn device with all the components. (b) Schematic of the device, showing the positions of the Raspberry Pis, large FoV IR cameras with IR filters, ArUco markers, and force sensitive resistors.

This paper is organized as follows. Section II describes the hardware and camera calibration settings. Section III illustrates the algorithm in detail. The experiments and results are demonstrated in Section IV. Finally, the discussion and conclusion are presented in Sections V and VI, respectively.

II. System Overview

Fig. 2 shows a schematic of the overall device operation, including stereo camera calibration, gait event detection, IR image taking and processing, gait parameters calculations, and wireless data transmission. The hardware design utilizes this schematic as guidance.

Fig. 2.

Flowchart showing the overall operation of the gait monitoring system.

A. Design

There are two wearables in the system, one for each ankle. Each 3D-printed wearable is outfitted with a pair of Raspberry Pi Zero 2 W, large-FoV IR cameras, 780nm IR filters, and 1200mAh PiSugar3 portable batteries to form the stereo camera system. The 120° FoV cameras are triggered by two polymer force sensing resistors (FSR 402 from Interlink Electronics) placed at the first metatarsal and the heel [28]. During the toe-off and heel strike phases, the cameras take images of the ArUco marker on the other wearable. The 780nm IR filter limits the camera’s wavelength response to the near-IR range. While the IR cameras limit background visual information, they provide sufficient resolution alongside the high contrast of the ArUco markers, affording an element of privacy during the data capture. As demonstrated later in Section III, the use of the IR cameras along with the ArUco markers greatly reduces the computational complexity of object detection, as the marker segmentation is significantly simplified.

The PCB design focuses on capturing gait events and the communication between Raspberry Pis. The force sensing circuit uses the same concept as a low-pass filter circuit, where the FSR is the resistive element and a 100nF multilayer ceramic capacitor is the output load. The time count for the capacitor to reach the high state indicates the approximate magnitude of the FSR impedance, which suggests the force level applied to the FSR. The preset hardware interrupt triggers IR imaging and records the time stamp when the time count is less than the pre-defined threshold, implying that a gait event has occurred. The system transmits the core data instead of raw images for higher power efficiency. The Universal Asynchronous Receiver-Transmitter (UART) communication protocol is used for the wired transfer of the extracted markers’ pixel positions between the Pi pair on the same wearable. The system utilizes the User Datagram Protocol (UDP) to wirelessly collect the computed gait length, width, and height from the two wearables and save them to the server.

The portable batteries can run the devices for 4-5 hours. The 3D printed model can be adjusted to wrap around the ankle with Velcro tapes and cushions to fit most people comfortably. The dimensions of the whole device are 62 × 156 × 81 mm. The entire device weighs 215.45 grams.

B. Camera Calibration

120°-FoV (Horizontal: 95°; Vertical: 75°) cameras were utilized in this work, which replaced the 72°-FoV Pi cameras used in our previous report [29]. The larger FoV enables more space than the prior one for capturing the marker while walking. The camera is rotated by 90° to have an extensive vertical FoV to obtain gait height even while the person is walking up stairs. Meanwhile, the smaller dimension of the 7.9 × 7.9 mm camera improves the measurement accuracy because the extra physical gap between the two cameras benefits the stereo camera algorithms.

Fig. 3 shows the stereo camera calibration setting and one of the calibration reconstruction results. The stereo camera, sitting on the center line, captures a series of unique checkboard image pairs. The checkboard is set at different positions and tilt angles. The calibration process extracts the camera’s intrinsic and extrinsic matrix through the image pairs. The intrinsic matrix is related to the internal characteristics of the camera, including focal length, principal points, distortion coefficients, etc. The extrinsic matrix represents the positional relationship between the two cameras using the rotation and translation matrices. The camera matrices are applied to rectify the stereo images, especially the distorted ones. The system only needs a one-time calibration procedure as long as the cameras’ relative position does not change. No manual interventions are required since there are no periodic sensing drifts. The camera calibration utilizes the MATLAB Computer Vision Toolbox because of its straightforward GUI.

Fig. 3.

(a) Camera calibration setting; (b) Calibration reconstruction, showing the stereo camera and pattern with various poses.

Camera calibration is crucial for the accuracy of the gait parameter measurement. The reprojection error is used to indicate the calibration performance, which is the pixel distance between a pattern keypoint detected in a calibration image and the corresponding world point projected into the same image using the camera parameters. Ideally, the reprojection error should be smaller than 1 pixel. To reduce the reprojection error, the calibration pattern should be at a distance that is roughly equal to the actual distance between the camera and the object of interest. Additionally, the intercept angle between the camera plane and the pattern plane should be less than 45°. During the calibration, we used twenty image pairs for calibration and one image pair for a quick performance assessment. For the following experiments, the calibration pattern was set at three locations, 30/45/65 cm, respectively, with various poses as shown in Fig. 3. The reprojection error is around 0.45 pixels.

III. Algorithm

This section discusses the operation of the algorithm behind the gait monitoring system. The calibration process determines camera parameters, including the distortion coefficients, then rectifies the captured IR images. The images first go through image processing for better feature extraction to obtain the markers’ pixel positions. After that, the pixel positions from both cameras in the stereo camera system are combined with the cameras’ matrix to calculate the 3-axis positions of the markers and the corresponding gait parameters. The Pseudocode is shown at the end of this section.

A. Distortion Coefficients

As shown in Fig. 4 (a) and (b), the larger FoV camera obtains more content compared with the regular Raspberry Pi camera because of the wide-angle lens. However, the corresponding image has stronger geometric distortions, which increase closer to the edges of the image. The straight lines on the edge are curved inwards owing to the barrel distortion. This is because the FoV of the lens is wider than the size of the CMOS sensors, and the scenes are “shrunk” to fit the canvas. The distortion effect on images may increase the error in measuring the gait parameters.

Fig. 4.

Image vision comparison (without IR filter) between (a) 72°-FoV camera; (b) 120°-FoV camera; (c) rectified image using only radial distortion coefficients; (d) and rectified image using all three distortion coefficients.

Considering the device is used for wearable purposes with limited computational resources, traditional solutions to the distortion, such as using additional compensating optical elements or adding extra computation processes to rectify each image, would drastically increase the complexity, weight, and cost of the system. To counter these, the distortion correction is handled during the camera calibration. The one-time calibration includes the calculation of the distortion coefficients. The camera parameters with these coefficients relieve the distortion problem during the estimation of the gait parameters.

The main contributor to the distortion seen in modern cameras is radial distortion, with some caused by tangential distortion and skewness. Radial distortion is common in low-cost, wide-angle lenses when the light rays bend more on the edge of the lens than the optical center. Tangential distortion happens when the camera lens is not perfectly aligned parallel to the CMOS sensor and causes the image to look tilted. Meanwhile, skewness occurs when the axis of images is not perpendicular and makes square content in a pixel appear like a parallelogram.

There are many models to describe radial distortion, for example, the even-order polynomial model [30] or Fitzgibbon’s division model [31]. The calibration applies the commonly used polynomial model to represent the radial distortion because it performs well on 120°-FoV distortion cases. The radial distortion has the following form,

$$\begin{gathered} x^{\prime }=x\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6} \\ {y}^{\prime }=y\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6} \end{gathered}$$ (3.1)

where, the $x$ , $y$ are the undistorted pixel locations, the $x^{\prime }$, $y^{\prime }$ are the distorted pixel locations, $k_n$ are the radial distortion coefficients of the lens, $r$ is the Euclidean distance of the distorted point to the distortion center. In the polynomial model, the $k_n$ values should be positive because the images represent a barrel distortion. The higher-order coefficients are not considered since six coefficients are sufficient to describe radial distortion. The radial distortion coefficient $k_n$ can be estimated from the parameters of circular arcs identified in image sets during camera calibration [32].

The tangential distortion has a form as shown below [33],

$$\begin{gathered} x^{\prime }=x+[2p_{1}xy+p_2(r^2+2x^2)] \\ {y}^{\prime }=y+[p_1(r^2+2y^2)+2p_{2}xy] \end{gathered}$$ (3.2)

where $p_n$ are the tangential distortion coefficients of the lens, the $x$, $y$ are the undistorted pixel locations, the $x^{\prime }$, $y^{\prime }$ are the distorted pixel locations, and $r^2=x^2+y^2$.

In addition, the skewness of the image is non-zero if the image axes are not perpendicular [34]. Using basic linear equations or changing the basis, the effects of skewness have the following format,

$$\begin{gathered} x^{\prime }=f_{x}x+x_0=f_{x}x-f_x\text{cot}\theta y+x_0^{\prime } \\ {y}^{\prime }=f_{y}x+y_0=\frac{f_y}{\text{sin}\theta }y+y_0^{\prime } \end{gathered}$$ (3.3)

where $f_x=F\ast s_x$ and $f_y=F\ast s_y$, $F$ is the focal length, $s_x$ and $s_y$ are expressed in pixel × m⁻¹, $\theta$ is the angle between the two axes, $x_0^{\prime }$ and $y_0^{\prime }$ are the offset of the image center from the origin. The definitions of $x$ , $y$ , $x^{\prime }$ , and $y^{\prime }$ are the same as above.

An image distortion effect combines the equations from the above three parts. Fig. 4(c) shows an undistorted image rectified from Fig. 4(b) using only the radial distortion coefficient. Fig. 4(d) displays the image using all three distortion coefficients. Compared with Fig. 4(c), Fig. 4(d) shows no significant changes. Because of the improved precision and optimized optical design in solid-state manufacturing, the tangential coefficients and skewness in modern cameras are negligible [35][36]. Therefore, we apply approximate computing by using only the radial distortion coefficient rather than all three coefficients to reduce the computation load while achieving similar accuracy.

B. Image Processing

As shown in Fig. 5(b), the IR filter limits the background visual information but also makes it harder to distinguish the ArUco markers. Image processing is thus first performed on the captured IR camera images to obtain the corresponding grayscale images and then the binary images for segmentation. The feature extraction method recognizes the ArUco markers and their pixel positions with less computation because of the binary image.

Fig. 5.

(a) Processed image after cross-correlation between the pixel’s neighbor area and the Gaussian kernel; (b) Original IR camera image displays the extracted ArUco marker location and ID.

Fig. 5(b) containing the markers were converted to binary images by applying a threshold. Common methods for thresholding, such as the simple constant threshold, cannot handle complex environments during walk, including various light conditions or complicated dynamic background information. Here, our system applies adaptive thresholding to generate multiple thresholds. The threshold can be either a pixel’s neighbor area average or a weighted sum of a KernelSize × KernelSize neighbor area of a pixel minus an offset, where the weighted sum is a cross-correlation between the neighbor area and the same size Gaussian window.

The 3.3 × 3.3 cm markers with high specificity were chosen from the default ArUco dictionary for feature extraction to obtain the marker pixel positions. The marker has a 5×5 grid for quick detection of the pattern and 25 bits for storing the information. The printed pattern uses a simple graphic with a large dimension, and contains data ‘5’. This helps the system accelerate the detection and error correction process. In general, the marker detection process consists of contour catching, shape filtering, perspective transforming, black/white areas separating, bits determining, and comparison with the dictionary. The above detections return the locations of the four corners of the marker and its contents if there is a match in the dictionary. We denote the middle points of the four corners as the location of the marker.

Fig. 5(a) displays the processed image with Gaussian Kernel, and Fig. 5(b) shows the originally captured IR image displaying the middle point of the detected ArUco marker.

C. Stereo Cameras Algorithm

The ArUco marker pixel positions from both images (from both cameras) were combined to compute the 3-axis marker positions and corresponding gait parameters through the stereo camera algorithm.

A binocular stereo camera can be used to estimate the depth of the 3D structure, where one camera is chosen to be a reference camera. The mapping from the world coordinate 3D points to 2D points in an image is described by the camera projection matrix $P$ ,

$$x=PX=K[R|t]X$$ (3.4)

where $\mathbf{X}\in {ℝ}^4$ and $\mathbf{x}\in {ℝ}^3$ are the reference camera homogenous coordinates of the 3D points and its 2D image points. The camera projection matrix $P$ can be decomposed into a camera intrinsic matrix $K\in {ℝ}^{3\times 3}$ and a camera extrinsic matrix $[R\mid t]\in {ℝ}^{3\times 4}$.

The camera intrinsic matrix $K$ has the form,

$$K=\left[\begin{array}{ccc}{f}_x& -f_x\text{cot}\theta & x_0\\ 0& f_y& y_0\\ 0& 0& 1\end{array}\right]$$ (3.5)

where $f_x$ and $f_y$ are the focal lengths in pixel units along the x- and y-directions respectively, $x_0$ and $y_0$ are the principal point offsets. The camera extrinsic matrix describes the pose relations between two cameras, where $R\in {ℝ}^{3\times 3}$ is the camera rotation matrix and $t\in {ℝ}^{3\times 1}$ is the camera translation matrix. As shown in Section III. A, when the skewness in the modern camera is negligible, the axis is perpendicular, and $\mathrm{c}\mathrm{o}\mathrm{t}\theta$ is then 0.

Since one camera in the binocular stereo system is chosen as a reference, its projection matrix is $P_1=K_1[I\mid \mathbf{0}]$, where $I$ is the identity matrix and $\mathbf{0}$ is the zero vector. The intrinsic camera matrix $K_1$ and the projection matrix for the second camera $P_2=K_2\left[R_2\mid t_2\right]$ can be found from the calibration procedure [37].

A point in the world coordinate is transformed into the camera coordinate using the extrinsic matrix. The camera coordinates are mapped into the image plane using the intrinsic matrix. On the other hand, the 3D coordinates of any point in the scene can be found given the relative camera poses and its image projections onto both cameras. If ${\mathbf{x}}_1={\left(x_1,y_1,1\right)}^T$ and ${\mathbf{x}}_2={\left(x_2,y_2,1\right)}^T$ are the image projections of the same $3\mathrm{D}$ scene point $\mathbf{X}={\left(x_w,y_w,z_w,1\right)}^T$, then $\mathbf{X}$ can be reconstructed given $P_1$, $P_2$, ${\mathbf{x}}_1$, and ${\mathbf{x}}_2$.

From the camera projection equations, we know that ${\mathbf{x}}_1=P_1\mathbf{X}$ and ${\mathbf{x}}_2=P_2\mathbf{X}$. This system of linear equations is equivalent to solving $A\mathbf{X}=0$ by taking the cross product ${\mathbf{x}}_i\times \left(P_i\mathbf{X}\right)=0$, $i=1,2$ to eliminate the homogeneous scale factor with

$$A=\left[\begin{array}{c}{x}_1{P}_1^{3T}-P_1^{1T}\\ y_1{P}_1^{3T}-P_1^{2T}\\ x_2{P}_2^{3T}-P_2^{1T}\\ y_2{P}_2^{3T}-P_2^{2T}\end{array}\right]\in {ℝ}^{4\times 4}$$ (3.6)

where $P_j^{iT}$ are the $i$th row of $P_j$, $i=1,2,3$, $j=1,2$.

The Least-squares solution of $A\mathbf{X}=0$ subject to $\parallel \mathbf{X}\parallel =1$, is the last column of $V$, where $A=U\sum V^T$ is the singular value decomposition of A [35].

D. Calculation of Gait Parameters

As shown in Fig. 6, the gait length and width are the vertical and horizontal distances between the camera on one foot and the marker on the opposite foot along the walking direction ( $-y$ direction); the gait height is the maximum vertical distance traveled by the foot during a leg swing.

Fig. 6.

(a) Top view: computation of gait length and width from marker positions; (b) Side view: computation of gait height from marker positions.

The 3D position values (solution to $A\mathbf{X}=0$ in Section III. C) are under the reference camera coordinate. However, natural gait length, width, and height should be intuitively in a world coordinate system, specifically, the ground coordinate. Considering that the wearables are nearly parallel to the ground, calculating the gait length and width from the 3D positions output of the stereo camera algorithm can be viewed as a coordinate transformation problem. Therefore, a point $v^{\prime }$ in the reference camera coordinate ( $x_{\mathit{\text{cam}}}{y}_{\mathit{\text{cam}}}$ coordinate) can be transferred to the world coordinate ( $xy$ coordinate) through the transformation matrix $Q$, as shown below,

$$v^{\prime }=Q\cdot v=\left[\begin{array}{cc}\text{cos}\theta & -\text{sin}\theta \\ \text{sin}\theta & \text{cos}\theta \end{array}\right]\cdot v$$ (3.7)

where $v$ is a matrix that includes gait length and width, and angle $\theta$ is the line of sight intercepted with the walking direction, which can be measured at device installment.

Gait height is the relative height difference between two wearables while walking, the same height difference as markers travel. The initial difference in height between the two wearables when both legs are at the foot-flat periods is measured in advance. The subsequential gait height computation utilizes the calculated height values and subtracts the initial height difference.

E. Pseudocode

The Pseudocode shown in Fig. 7 briefly demonstrates the overall deployment of the above gait monitoring devices. The Pseudocode has three major parts: 1. gait event detections; 2. image processing and marker detections; 3. position calculations. Note that this Pseudocode is applicable to the wearable that connects to the FSR on the gait heel. Small changes are made for the device that connects to the FSR at the top of the sole, in the hardware interrupt design and marker pixel position transfer through UART. The hardware interrupt triggers the cameras on the heel to take images simultaneously.

Fig. 7.

Pseudocode for device operation.

Camera calibration, calibration accuracy evaluation, and corresponding hardware adjustment are done separately. Furthermore, many other peripheral codes were used depending on the application, for example, adjustment of camera parameters such as ISO, shutter speed, and exposure gain; failure detection/error correction; power-saving sleep mode with auto wake/running script, etc.

IV. Experiment and Result

In this section, two experiments are described to test the performance of the gait monitoring system. The accuracy and precision of the gait length, width, and height are counted for the main metrics of the performance. We also compare the device’s performance with state-of-the-art technologies.

A. Methodology

Three coauthors (healthy, aged 28-35) volunteered to test the gait monitoring system to verify its sensing ability and repeatability. The experiments were conducted at different times over two months. The physical stability of the device was quite crucial for the two experiments, especially during walking. A piece of towel was used as cushion to keep the device tight on the ankle while also providing comfort for the participants during experiments.

The first experiment, shown in Fig. 8(a), assessed the ability of the devices to monitor the gait length and width. We designed a five-meter long, one-meter wide printed grid with a 5 mm resolution to record the ground truth for the gait length and width. Side cameras and tape markers were utilized to mark the gait pattern to achieve better ground truth accuracy. Three participants walked five times on the grid. The range of gait length for the participants varied between 11.5 cm to 48.5 cm, and the range of gait width for these participants was 10.5 cm to 22.5 cm. The number of steps for each round varied from 4 to 8, depending on the person. A total of 75 meters was recorded.

Fig. 8.

Experiment setup. A participant walks (a) through the printed grid to measure the gait length and width; (b) up the stairs to estimate the gait height.

The second experiment explored the feasibility of the monitoring system to measure gait height. The three participants lifted their feet to five predetermined heights (3, 6, 10, 15, and 21 cm) and one natural gait height. We also tested the devices’ performance on stairs, which could be a valuable feature for future disease classification. The stair height was around 18 cm. The gait height experiment was repeated five times on each participant.

B. Accuracy and Absolute Error

Fig. 9(a)–(c) demonstrates the gait length, width, and height measurement results, respectively. Each figure has three objects representing the data from the three participants. The first columns of Fig. 9 are the gait parameters’ average accuracy with their standard deviation (SD). The second column depicts the average absolute error with the SD of gait parameters. For gait length and width, each round of experiment was divided into left to right and right to left to better classify the data.

Fig. 9.

Results for (a) Gait length; (b) Gait width; (c) Gait height. Columns from left to right 1) Gait length/width/height average accuracy/SD; 2) Gait length/width/height absolute error average/SD; 3) Box chart of gait length/width/height data points over three persons.

The average accuracy for the gait length, width, and height was found to be 96.52%, 94.46%, and 95.29%, respectively. The definition of average accuracy is absolute errors divided by the ground truth values. The average absolute errors for the above three parameters were 1.0 cm, 0.9 cm, and 0.6 cm, respectively. The errors were converted to absolute values because part of the measured data was positive while others were negative.

From the above figures, person 2 has a significant absolute error and a relatively large SD on the gait length compared with the other two persons. A possible reason is that person 2 has a large step length and narrow step width, which caused the markers to appear on the edges of the images, increasing the error. Furthermore, a considerable step length means fewer step data points, making the SD relatively large. Another observation is that the gait width accuracy is less than the length because, with a similar absolute error, the ground truth of width is smaller than the length, amplifying its impact.

C. Precision

The last column of Fig. 9 shows a box chart of the data points for the gait length, width, and height from three persons in the two experiments. Five horizontal lines from bottom to top of the box represent the 5th, 25th, median point, 75th, and 95th percentiles of data. The white dot inside the box is the average value point. The box size indicates the interquartile range, which illustrates the statistical dispersion of the data set. In the meantime, the sharpness of the convex curve also intuitively pictures the variance of the data. From the figures, the gait monitoring system has high precision since most of the data are clustered.

As reasoned above, the gait length data from person 2 was relatively scattered, which lowers the precision compared with the other two persons. We also see that 90% of the gait height data points are within the ±1 cm error range. However, several data points from the 21-cm gait-height test are far from the box range. This is understandable, since a 21-cm gait height does not reflect a natural walk and exceeds the normal calibration range of the camera system. Furthermore, a larger distance also causes an increase in the error due to difficulty in marker capture.

D. Performance Comparison

We did a survey on the gait monitoring technologies developed in the past five years. None can simultaneously measure the gait length, width, and height, especially the height. Some only measure the gait length. As shown in Table 1, the performance of our ankle-wore stereo camera system on gait length and width exceeds or is on par with these competing technologies, with a relatively low cost. In the meantime, our system provides the gait height measurement with high accuracy.

TABLE I.

Accuracy Performance Comparison

Works	Gait Length	Gait Width	Gait Height
OptiGait	96.52%	94.46%	95.29%
Z. Yang et al. [38]	93.06%	N/A	N/A
S. An et al. [39]	94.51%	N/A	N/A
C. Tjhai et al. [40]	> 95%	N/A	N/A
P. Arens et al. [41]	96.4%	N/A	N/A
C. Ferraris et al. [42]	94.69%	86.22%	N/A
B. Anderson et al. [43]	95.25%	95.15%	N/A

V. Discussions

There are some possible explanations for the deviations in the calculated results from the ground truth data. First, the stereo camera algorithms rely on the assumption of the pinhole model. However, large lenses in low-cost cameras disobey the assumption, specifically at the edge of images. This can be corrected by extra compensations in the algorithm. Second, the low-resolution images used in stereo camera systems provide fewer pixels, resulting in uncertainty errors in feature point correspondence between markers on the two images. This can be fixed by designing markers better suited for gait monitoring instead of the standard ArUco marker. Lastly, fitting the device to the ankle might lead the stereo cameras and markers to deviate from the original positions, introducing errors. This can be improved by the careful and iterative design of a new type of model with a more stable and ergonomic shape.

Future work involves solving the above problems. Furthermore, more gait parameters, such as gait speed, acceleration, standing time, swing time, cadence, foot pose angle, step frequency, step time, etc., can be measured using the time stamp recorded from gait event detection and better-designed markers. A model-based object detection method will be developed to detect the self-designed marker. We will recruit more participants from different races, genders, ages, health status, etc., to establish a standard balanced gait parameters database based on our iterative monitoring system. With more gait features and data points involved, it is possible to classify and predict gait diseases through machine learning.

VI. Conclusions

A new type of ankle-worn gait monitoring system based on computer vision has been developed for monitoring the gait. A wide-angle infrared stereo camera is mounted for tracking gait parameters. The system has demonstrated high accuracy and precision in gait length, width, and height measurements. The measurement accuracy is 96.52%, 94.46%, and 95.29% for gait length, width, and height, respectively. The average absolute errors for the above three parameters are 1.0 cm, 0.9 cm, and 0.6 cm, respectively. The proposed gait monitoring system is low cost, easy to use, and suitable for real-life measurement, all while delivering comparable sensing performance.

Biographies

Jiangang Chen received the B.E. degree in Electronic Science and Technology from Heilongjiang University, Harbin, Heilongjiang, China, in 2018 and the M.S. degrees in both Electrical and Computer Engineering, and Computer Science from the University of Wisconsin–Madison, Madison, WI, USA, in 2023. He is currently a Ph.D. candidate in the Department of Electrical and Computer Engineering at the University of Wisconsin–Madison, Madison, WI, USA. His research interests include vision-based sensors, flexible sensors, computer vision, machine learning, and MEMS.

Jayer Fernandes received the B.E degree in Electronics and Communications Engineering from the Visvesvaraya Technological University, Karnataka, India, in 2009, and the M.S. and Ph.D. degree in Electrical and Computer Engineering from the University of Wisconsin–Madison, Madison, WI, USA in 2014 and 2021 respectively. He is currently a Post Doctoral Research Associate with the Department of Electrical and Computer Engineering at the University of Wisconsin–Madison, Madison, WI, USA. His research focuses on the development of microfluidic optical actuators, flexible capacitive and photonic sensors for different biomedical applications. His other research interests include flexible optical sensors, micro-optics and flexible electronics incorporating a wide range of sensing modalities.

Jianwei Ke received a B.S. Degree in Electrical Engineering from the University of Delaware with the highest distinction in 2015. He is a Ph.D. candidate in the Electrical and Computer Engineering department at the University of Wisconsin–Madison since 2016. In 2022, he completed M.S. degrees in Math, Computer Science, and Electrical and Computer Engineering at the University of Wisconsin–Madison. His research interests span general computer vision, deep learning, and signal/image processing. He is currently working on image-based 3D reconstruction and real-time 3D reconstruction and rendering.

Francis Lu received the B.S. degree in Electrical Engineering from University of Wisconsin–Madison, Madison, WI, USA in 2023. He is currently a master’s student in the Department of Electrical, Computer, and Energy Engineering at Arizona State University, Tempe, AZ, USA.

Dr. King is a professor of Nursing at the University of Wisconsin-Madison, School of Nursing. Her area of research centers on improving outcomes for hospitalized older adults. Her studies include creating a pragmatic intervention to improve older adult patient walking during their hospital stay and studying older patient ambulation recovery after hospital discharge.

Yu Hen Hu received BSEE from National Taiwan University, Taiwan ROC in 1976, and MSEE and Ph.D. degrees from the University of Southern California, Los Angeles, CA, USA in 1982. He was a member of faculty in the Electrical Engineering Department of Southern Methodist University, Dallas, Texas from 1983 to 1987. Since 1987, he has been with the Department of Electrical and Computer Engineering, University of Wisconsin, Madison where he is currently a professor. In 1999, Dr. Hu is a visiting researcher at Bell Laboratories, Holmdel NJ, and at Microsoft Research – China, Beijing, China. He has been a visiting professor at National Taiwan University, Graduate Institute of Electronics, Taipei, Taiwan in 2007 and 2015. Dr. Hu has broad research interests ranging from design and implementation of signal processing algorithms, computer-aided design and physical design of VLSI, pattern classification and machine learning algorithms, and image and signal processing in general. He has published more than 170 journal papers, 280 conference proceedings papers, edited, and co-authored four books. He also holds 10 US patents. Dr. Hu is a Life fellow of IEEE.

Hongrui Jiang (Fellow, IEEE) received the B.S. degree in physics from Peking University, Beijing, China, and the M.S. and Ph.D. degrees in electrical engineering from Cornell University, Ithaca, NY, USA, in 1999 and 2001, respectively.

From 2001 to 2002, he was a Postdoctoral Researcher at the Berkeley Sensor and Actuator Center, University of California at Berkeley. He is currently the Vilas Distinguished Achievement Professor and the Lynn H. Matthias Professor of engineering with the Department of Electrical and Computer Engineering, a Faculty Affiliate with the Department of Biomedical Engineering, the Department of Materials Science and Engineering, and the Department of Ophthalmology and Visual Sciences, and a member of the McPherson Eye Research Institute with the University of Wisconsin–Madison. His research interests include micro-/nano-fabrication technology, biological and chemical micro-/nano-sensors, microactuators, optical microelectromechanical systems, smart materials and micro-/nano-structures, lab-on-chip, and biomimetics and bioinspiration. He is a fellow of the Institute of Physics, the Royal Society of Chemistry, the American Institute for Medical and Biological Engineering, and the Institute of Electrical and Electronics Engineers. He was a recipient of the National Science Foundation CAREER Award and the Defense Advanced Research Projects Agency Young Faculty Award in 2008, the H. I. Romnes Faculty Fellowship of the University of Wisconsin–Madison in 2011, the National Institutes of Health Directors New Innovator Award in 2011, the Vilas Associate Award of the University of Wisconsin in 2013, and the Research to Prevent Blindness Stein Innovation Award in 2016.