A 3D Computer Vision-Guided Robotic Companion for Non-Contact Human Assistance and Rehabilitation

Abstract

With the rapid aging of the U.S. population, the mobility impairment is becoming a more and more challenging issue that affects a large number of individuals. The research presented in this paper aims at helping the mobility-challenged individuals with a novel robotic companion, which is a walker-type mobile robot capable of accompanying the human user and keeping user at the center for protection and possible power assistance. The robotic companion is equipped with a 3D computer vision system, which provides a unique capability of sensing the human-robot relative position/orientation without physical contact or the need for wearable sensors. As such, the robotic companion enables the user to walk freely with minimum disturbance to his/her normal gait, relieving the user from the physical and cognitive loads associated with the use of traditional assistive devices. For the development of the robotic companion, the authors designed and fabricated a low-cost, differentially steered mobile robotic platform, and also developed a unique image processing system to extract the position/orientation information from the 3D camera-captured images. Furthermore, an advanced motion control system was developed for the robotic companion, which provides novel solutions to the unique challenges such as sway reduction and noise reduction in digital differentiation. To quantify the performance, component and system-level experimentation was conducted, and the results demonstrated that robotic companion and its key components function as desired and the system is expected to reduce the user load and improve the user mobility in real-world scenarios.

1. INTRODUCTION

The rapid aging of the U.S. population nowadays is unprecedented in history: the number of older adults will double in the next 25 years. By 2030, there will 72 million older adults, accounting for 20% of the general population [1]. Under this long-term trend, the impairment of mobility is becoming an increasingly challenging problem affecting a large number of individuals. Older adults gradually lose their physical work capacity due to age-related musculoskeletal degeneration (25% to 30% decline between the ages of 30 and 70 [2]). Individuals with neuromuscular diseases (such as stroke) also suffer from partial or complete loss of their capability of conducting physical activities. With the impaired mobility, these individuals are more likely to live in a sedentary lifestyle, and suffer from the various problems associated with the sedentary lifestyle, e.g. high blood pressure, depression, obesity, etc [3].

To help the mobility-impaired individuals, a wide variety of assistive tools have been developed and used. Conventional walking aids, such as canes and walker, enhance the stability of walking and reduce the risk of fall [4, 5]. However, the passive devices require the users to lift and advance the devices for each step. Such action not only increases energy expenditure, but also places heavy cognitive burden on the user [6]. Wheelchair is another major type of mobility tool in extensive use. Unlike the walking aids, wheelchairs allow users to take a seated position in movement, which minimizes the risk of fall. Additionally, the metabolic energy consumption of the users is minimal when powered wheelchairs are used. However, a wheelchair largely precludes its user’s lower-limb muscle activity and bone load-carrying, and thus accelerates his/her musculoskeletal degeneration [7].

With the recent advances in robotic technologies, numerous wearable robots (exoskeletons) have been developed for the rehabilitation and assistance purposes. Some are integrated with treadmills to form complete rehabilitation training systems, for example, Lokomat [8], LOPES [9], ALEX [10], and Walkbot [11]. These systems are usually bulky and expensive, primarily used in clinics for physical therapist-guided rehabilitation training. In recent years, portable exoskeletons capable of untethered operation started to become more common, and typical examples include Ekso [12], ReWalk [13], and Vanderbilt Exoskeleton/Indego [14]. These portable devices can potentially expand the motion assistance beyond the traditional clinic setting, and improve the users’ mobility in daily life. However, an exoskeleton is closely coupled to its human user in operation, and the additional load tends to affect his/her normal walking gait. Furthermore, when wearing an exoskeleton, a human user usually needs an additional walking aid (such as canes) to improve the stability in walking and reduce the risk of fall, which exacerbates the deviation from his/her normal walking gait.

Motivated by the aforementioned issues with existing assistive devices, the authors developed a novel walker-type robotic companion as an inexpensive and easy-to-use assistive robot to help the individuals with mobility challenges. Unlike the aforementioned mobility-assistive devices, the robotic companion does not rely on the physical interaction with the human user to provide assistance. Instead, the main purpose of the device is to accompany the user in walking and provide assistance and protection when needed. As such, the interference to the user’s walking gait is minimized, which encourages the user to take an active role in his/her own physical exercise or rehabilitation training. To realize such functionality, the proposed robotic companion is essentially a 3D computer vision-guided robotic walker capable of tracking the human user’s motion in real-time and maintaining the user at the center of the robot in walking.

Note that smart robotic walkers with enhance functionality over traditional walker/rollator have been a topic of active research in recent years. For example, robotic walkers have been developed to provide navigation assistance to frail older adults and individuals with cognitive and vision impairments [15–17]; robotic mobility aids have also been developed to provide assistance to mobility-challenged individuals in walking and other daily activities (e.g. [18]). Similar to the robotic companion in this paper, some robotic platforms are capable of automatically controlling the robot motion when accompanying the user in walking, with or without physical contact with the user. To measure the human motion for the control purpose, some robots require the human users to wear body sensors (e.g. [19]), which causes inconvenience to the users. Further, mounting a sensor to the human body reliably and consistently over repeated use is difficult, especially when the user is constantly moving during ambulation. This issue affects the accuracy of human motion measurement in the real-world application of these robotic platforms. Alternatively, some robotic platforms use robot-mounted infrared (IR) or laser sensors to scan the human body at a certain height. For example, Ragaja et al. used an IR sensor array (5 sensors) to measure the user’s waist position and orientation for the control of a posterior active walker [20]; Valadao et al. [21] and Shin et al. [22] both used laser range scanners to detect the user’s legs for the calculation of the user’s distance to the robot. With the measurement conducted on a single horizontal plane, these approaches are vulnerable to disturbances and irregularities in human walking (e.g., irregular leg movements or user postures). Additionally, gesture-based control interface has also been attempted in the work of Shin et al., but the users’ elevated cognitive load makes such approach impractical [22].

Compared with the aforementioned robotic walkers, the advantages of the vision-guided robotic companion in this paper are obvious. The use of a 3D computer vision system eliminates the need for body sensors, making the robotic companion more convenient to use. Compared with the IR/laser sensor-based approaches [20–22], the vision system measures the human motion based on the 3D point cloud data obtained over the entire upper body, and thus provides a more reliable, robust, and accurate measurement of the human position relative to the robot. Based on the preliminary results in [23] and [24], the authors present the comprehensive details on the design, instrumentation, control, and experimentation of the novel robotic companion system. The paper is organized as follows: Section 2 presents the design of the robotic companion, Section 3 presents the 3D computer vision-based human measurement system, Section 4 presents the 3D computer vision-guided robot controller, Section 5 presents the experimental results that demonstrate the effectiveness of the robotic companion, and Section 6 contains the conclusions of the paper.

2. DESIGN OF THE ROBOTIC COMPANION

The robotic companion in this paper is essentially a smart mobile robot that is able to regulate its own movement in response to its human user. To minimize the mechanical complexity while providing the desired mobility, the classic differential steering configuration was chosen, such that no complex wheel-turning mechanism is needed. Based on this configuration, two independently powered hub motors are mounted to the robot chassis to obtain the desired maneuverability, i.e., independently controlled fore-aft movement and left-right turning. Additionally, two passive castors are mounted at the rear of the chassis for balance and stability. Figure 1 depicts the robotic companion when a human user is situated at the center, and Table 1 lists the major components of the robotic companion.

Figure 1.

System layout of the robotic companion.

Table 1:

Major Components of the robotic companion.

Component	Model	Manufacturer
Hub motor	280–1342M	Hub Motors, Rogers, AR
Frame	Aluminum square tube	80/20 Inc., Columbia City, IN
Servo drive	AZBDC20A8	Advanced Motion Controls, Camarillo, CA
Microcontroller	dsPIC33fj128mc802	Microchip Technology, Chandler, AZ
Incremental encoder	AS5311	AMS AG, Premstaetten, Austria
3D camera	Realsense R200	Intel, Santa Clara, CA
Single-board computer	UP board	Intel, Santa Clara, CA

As shown in Fig. 1, the chassis of the current robotic companion prototype was constructed with a lightweight framing system. Additionally, a pair of parallel bars are mounted on the chassis for the support and stability enhancement for the user when necessary. The robotic companion is fully self-contained, with all sensing, control, and power supply components integrated. The power output of each hub motor is regulated by a servo drive, which interfaces with a microcontroller through pulse-width modulation (PWM) connection. For the feedback control, each wheel is fitted with a rotary incremental encoder for the real-time measurement of wheel rotation. The entire robot is powered with two auto batteries, which are connected in series to form a 24V power supply.

For the measurement of the human-robot relative position/orientation, a compact 3D camera is mounted on a pole at the front end of the robot. The mounting height of the camera can be adjusted such that the camera faces the user at approximately the chest height. Due to the significant computation required by the 3D image processing, a single-board computer is dedicated to this task, with the results (human position/orientation relative to the robot) transmitted to the aforementioned microcontroller for robot motion control. The subsequent section presents the image processing algorithm implemented in the single-board computer, which processes the 3D image to obtain the desired position/orientation signals in real time.

3. 3D COMPUTER VISION SYSTEM

In the literature, many computer vision-based approaches have been developed for 3D human pose estimation [25–32]. Seoud et al. proposed a convolutional neural-networks-based method to segment the range image of human body and determine the 3D spatial information of the human body parts related to the segmented images [25]. The method is robust to missing data and artifacts induced by the structure-light system. Mehrizi et al. proposed a two-step method to predict the 3D human pose in lifting [26]. The method first employs a deep neural network to extract the whole human body from the camera images and then uses the anthropometric information to calculate the positions of the subject’s skeleton joints in 3D space. In addition to the neural-networks-based methods, anthropometry-based skeleton fitting approaches have been proposed to estimate the posture of the upper limbs [27] and lower limbs [28] for rehabilitation exercises. A comparative evaluation of three state-of-the-art methods [29–31] for the estimation of the position, orientation and articulation of the human body from RGB-D data is given in [32]. The existing methods, however, incur a high computational cost and are thus unsuitable for a mobile platform with limited computing power.

To address the issue, a simple yet effective approach was developed in this work to compute the parameters of human torso body by processing the RGB-D data of a 3D camera. The method can be implemented on a credit card-sized computer (Up Board computer) to compute these parameters in real time. The specific camera used in this work provides both color image and depth data at a resolution of 640×480 and a refresh rate of 60 fps. Such performance has been demonstrated to be sufficiently strong for robot control purpose. Figure 2 shows a typical image of human upper body taken by this camera.

Figure 2.

A typical image of human upper body taken by the 3D camera in the robotic companion.

The processing of the data obtained from the 3D camera consists of four steps as shown in Fig. 3. First, color and depth imaging data (Fig. 2) are acquired from the 3D camera. Second, the imaging data related to human upper body is located within a predefined bounding box. Third, a histogram-based filtering scheme is used to remove noise and extract human torso from the upper body data. Fourth, human pose parameter including the center point of the torso and the user’s heading direction are computed from the torso data.

Figure 3.

Computer vision data processing pipeline.

Each depth data frame provides a 3D point cloud P of the user’s upper body (red and green points as shown in Fig. 4). Based on the parameters, location and field-of-view of the camera, the point cloud data of the user’s body are located inside a bounding box given by:

$$\begin{gathered} B=\left[x_{min}, x_{max}, y_{min}, y_{max}, z_{min}, {\text{z}}_{\text{max}}\right] \\ \hspace{1em}=\left[-1, 1, -1, 1, 0.4, 1.5\right] \end{gathered}$$ (1)

The parameters in Eq. (1) ensure that the bonding box encloses all data points of the entire upper body when the distance between the user and the 3D camera is within a permitted range. As shown in Figure 4, $\overline{p}$ is the centroid of the user’s upper body data points, ${\theta }_h$ is the heading angle, which is defined as the angle between the user’s heading and the Z axis.

Figure 4.

Extraction of the upper body data points and computation of the position and orientation.

To identify the user’s upper body, all 3D data points in P are projected onto XZ-plane, which is discretized into 40×22 grid cells (grid-size: 0.05 m). The number of data points falling into cell $c_{ij}$, for $i=1,\cdots ,40$, j = 1,⋯,22, is recorded and denoted by $h_{ij}$. $\mathit{H}=\left\{h_{ij}\right\}$ forms a histogram. It is then converted into a grey image U by scaling $h_{ij}$ into the range [0, 255] to obtain the intensity value $u_{ij}$. As the torso casts more data points on the related grid cells, these cells have larger h values and the related image pixels are brighter. This cluster of image pixels ${\mathit{U}}^{\prime }$ can be extracted from U by the following region growing method:

Step 1: Select the pixel with the maximal intensity (255) as a seed point. Push the pixel’s data (intensity and coordinate values) into a FIFO queue Q. Create an empty set ${\mathit{U}}^{\prime }$.

Step 2: Pop the data of a pixel $q_k$ from Q and add it into ${\mathit{U}}^{\prime }$.

Step 3: Find all neighboring pixels of $q_k$ in U. For each of the pixels that have not been visited before, compute its intensity difference from $q_k$. If the value is below a threshold (40 in this paper), push the pixel’s data into Q.

Step 4: Repeat steps 2 and 3 until Q is empty.

The 3D data points related to ${\mathit{U}}^{\prime }$ are the extracted torso data points, denoted $\mathit{P}{ }^{\prime }$. Taking the point cloud in Fig. 4 as an example, the grey image $\mathit{U}$ for the upper body data is depicted in Fig. 5a, and the image (in binary) for the extracted torso data is shown in Fig. 5b. The torso data points ${\mathit{P}}^{\prime }$ related to ${\mathit{U}}^{\prime }$ are labeled in green in Fig. 4.

Figure 5.

Torso data extraction: (a) the grey image $\mathit{U}$ related to the upper body data points as shown in Figure 4; (b) the extracted torso image ${\mathit{U}}^{\prime }$ (binary image).

To estimate the user’s heading angle, the torso points ${\mathit{P}}^{\prime }$ of the user are first mapped onto an image plane under the bird’s-eye view (i.e., XZ plane). A quadratic curve is then fitted to the projected points to estimate the profile of the torso. Taking the torso points ${\mathit{P}}^{\prime }$ of Figure 4 as an example, the projected points are rendered as an image as shown in Fig. 6, where the fitted curve represents the torso’s shape. For a point $p_m$ (m = 1, ⋯, M) on the curve, the heading angle ${\theta }_h$ (the angle between the normal of $p_m$ and -Z axis) is computed and the averaged value of ${\theta }_h$ over the M points (M=100 in this paper) is taken as the estimated heading of the user. Quadratic curve fitting is used in this work because our experiments showed that the use of a higher-order curve tends to cause over-fitting.

Figure 6.

Quadratic curve fitting for the estimation of the torso’s shape and orientation. The yellow line stands for the fitted curve

4. ROBOTIC COMPANION MOTION CONTROL

The human-robot relative position-orientation measured from the 3D computer vision system can be used as a feedback signal to enable the robotic companion to track the human movement. The motion control of the robotic companion, however, is a challenging research topic due to the noise in the position/orientation measurement, as well as the oscillation in the orientation signal resulting from the upper body sway in human walking. To address these issues, the motion controller in this works was developed by incorporating an orientation signal preprocessing module for sway suppression and a hybrid PID controller with lower-pass filtered differentiation. The schematic of the robot companion control system is shown in Fig. 7.

Figure 7.

Control system of the robotic companion.

4.1. Sway Suppression

Sway of the upper body is an inherent component of the full-body movement in human walking [33, 34]. As a result of such sway motion, the orientation of a human’s upper body oscillates slightly across the direction of walking, and such oscillation is clearly visible in the orientation data generated by the 3D computer vision system. From the control perspective, such oscillation is categorized as a special type of disturbance, as the robot is expected to move smoothly in the direction of walking (instead of oscillating left and right in response to the user’s upper body sway). How to reject such disturbance, therefore, constitutes a major challenge in the controller development.

To gain more insight into the problem, the authors conducted a simple experiment of healthy subjects walking in straight lines, having their upper body orientation trajectories recorded. A typical trajectory is shown in Fig. 8. It can be clearly observed from this figure that the sway motion is a somewhat irregular oscillation at approximately the same frequency as the walking cadence, and its magnitude varies significantly from step to step. As such, it is nearly impossible to completely isolate such sway motion, as the onset of a turning motion would be very similar in shape.

Figure 8.

A typical trajectory of upper body orientation in straight-line walking.

Based on such observation, the approach developed in this work is sway suppression (instead of sway elimination), in which the negative effects of sway motion are minimized while the robot is still responsive to the user’s turning motion. Specifically, a suppression band is defined based on the angular range of the sway motion:

$$-\Theta \le \theta \le \Theta$$ (2)

where $\theta$ is the measured upper body orientation and $\Theta$ is the width of the suppression band, which can be determined through observation or tuning. Subsequently, a suppression function is defined, which meets the following requirements within the suppression band: 1) continuous (to avoid discontinuity in the control command); 2) symmetric with respect to the origin (to make the control actions symmetric); 3) derivative close to zero around the origin (to make the controller insensitive to small-magnitude body sway); and 4) equal to the independent variable $\left(\theta \right)$ at the boundary (to ensure the overall continuity and smooth transition to the no-suppression regions). A simple solution adopted in this work is a 3^rd-order polynomial, which provides sufficient flexibility in tuning. The overall output function is defined as:

$${\theta }^{\prime }=\{\begin{array}{cc}{a}_3{\theta }^3+\mathrm{sgn}\left(\theta \right)\cdot a_2{\theta }^2+a_1\theta & when\hspace{0.17em}-\Theta \le \theta \le \Theta \\ \theta & else\end{array}$$ (3)

where ${\theta }^{\prime }$ is the output orientation angle (post-suppression), and a₁, a₂, and a₃ are the tunable parameters. After repeated tuning, the width of the suppression band was set as $\Theta =8^{\circ }$, and the parameters were set as a₁ = 0.01, a₂ = 0.03, a₃ = 0.12. The comparison of the orientation angles pre-suppression versus post-suppression is shown in Fig. 9. As can be observed from this figure, the post-sway signal is largely compressed within the suppression band, and remains unchanged beyond the suppression band.

Figure 9.

Comparison of the orientation angles pre-suppression versus post-suppression.

4.2. Inverse Kinematics

The post-suppression orientation signal ${\theta }^{\prime }$, combined with the relative position in the longitude direction d, is used to calculate the desired wheel rotation angles through inverse kinematic calculation. The robotic companion is a typical differentially steered vehicle, and thus the typical differential steering model [35] is applied (Fig. 10):

$$\left[\begin{array}{c}{v}_x\\ v_y\\ \omega \end{array}\right]=\left[\begin{array}{cc}cos\alpha & 0\\ sin\alpha & 0\\ 0& 1\end{array}\right]\left[\begin{array}{c}\left(v_R+v_L\right)/2\\ \left(v_R-v_L\right)/L\end{array}\right]$$ (4)

where $v_x$ and $v_y$ are the linear velocity x and y components of the robot center, which can also be defined as $v_x=vcos\alpha$, $v_y=vsin\alpha$; $\omega$ is the angular velocity of the robot; α is the robot turning angle (i.e., change in the orientation); S is the linear displacement of the robot center, approximated by the arc length of the robot center trajectory; υ_L. and υ_R are the center velocities of the left and right wheels; and L is the distance between the wheels.

Figure 10.

Differential steering model of the robotic companion.

The velocity equation (4) can be integrated over a measurement cycle (Δt) of the computer vision system (40 ms corresponding to the 25 Hz refresh rate in this work). Assuming that the robotic displacement over a measurement cycle is very small, the wheel trajectories as well as the robot center trajectory can be approximated by straight lines, and the following equations can be obtained based on the geometry in Fig. 10:

$$v_x\times \Delta t=S\cdot cos\alpha$$ (5)

$$v_y\times \Delta t=S\cdot sin\alpha$$ (6)

$$\omega \times \Delta t=\alpha$$ (7)

The center displacements of the powered wheels ΔS_R and ΔS_L can be calculated in a similar way:

$$\Delta S_R=v_R\times \Delta t$$ (8)

$$\Delta S_L=v_L\times \Delta t$$ (9)

Substituting (5)~(9) into (4), the wheel center displacement can be calculated as

$$\Delta S_R=S+\frac{\alpha L}{2}$$ (10)

$$\Delta S_L=S-\frac{\alpha L}{2}$$ (11)

Finally, the wheel rotation angles are calculated as the set points of the local (wheel position) controllers. Here the desired turning angle and longitude displacement are set as the measured human-robot orientation (post-suppression) and longitude position, as the robot should track the human movement and keep him/her at the center. As such, the wheel rotation angles are expressed as

$$\Delta {\varnothing }_R= \frac{2d+{\theta }^{\prime }L}{2\pi r}$$ (12)

$$\Delta {\varnothing }_L= \frac{2d-{\theta }^{\prime }L}{2\pi r}$$ (13)

where r is the wheel radius.

4.3. Wheel Motion Control

On the lower level, a wheel motion controller was developed to regulate the wheel rotation. The traditional proportional-integral-derivative (PID) control approach was applied [36], which can be expressed as:

$$u\left(k\right)=k_p\cdot e_k+k_i \cdot {\int }_0^t{e}_{k}dt+k_d\cdot \frac{e_k-e_{k-1}}{T_s}$$ (14)

where $u\left(k\right)$ is the controller input in Iteration k; k_p, k_i, k_d are the gains for the PID controller; T_s is the period of one the iteration loop; e_k is the angular error between the desired versus the measured wheel angles in Iteration k. Note that the wheel angular velocities are not directly measured in the robotic companion. As such, the derivative term in the PID controller has to be obtained through digital differentiation. To minimize the impact of measurement noise, a practical differentiator was developed, which takes the following form in the Laplace domain:

$$\frac{D\left(s\right)}{E \left(s\right)}=\frac{S}{T_{c}S+1}$$ (15)

where D(s) is the filtered derivative term of the control error E(s), and T_c is the cut-off frequency of the low-pass filter. Subsequently, a Tustin transformation is conducted on (15):

$$\frac{D\left(z\right)}{E \left(z\right)}=\frac{\begin{array}{cc}\frac{2}{T_s}& \frac{z-1}{z+1}\end{array}}{\begin{array}{ccc}{T}_c& \begin{array}{cc}\frac{2}{T_s}& \frac{z-1}{z+1}\end{array}& \begin{array}{cc}+& 1\end{array}\end{array}}$$ (16)

which can be rewritten as

$$\left(1-\frac{2T_c}{T_s}\right)D\left(z\right)=\frac{2Z}{T_S}E\left(z\right)-\frac{2}{T_s}E\left(z\right)-\left(\frac{2T_c}{T_s}+1\right)zD\left(z\right)$$ (17)

This equation is then converted to the following difference equation:

$$\left(1-\frac{2T_c}{T_s}\right)d\left(k\right)=\frac{2}{T_S}e\left(k-1\right)-\frac{2}{T_s}e\left(k\right)-\left(\frac{2T_c}{T_s}+1\right)d\left(k-1\right)$$ (18)

which can be converted to the following equation for d(k) to be implemented in the PID controller:

$$\begin{array}{c}d\left(k\right)=\frac{2}{2T_c-T_S}\left(e_k-e_{k-1}\right)\end{array}+\frac{2T_c+T_S}{2T_c-T_S}\begin{array}{c}d\left(k-1\right)\end{array}$$ (19)

After implementing the PID controller for the wheel motion control, the controller gains were tuned with the Ziegler-Nichols approach [36], followed by minor manual adjustments. The parameters of the PID controller are summarized in Table 2.

Table 2:

Parameters of the PID Controller.

Parameters	Value	Unit
$k_p$	0.24
$k_i$	0.012
$k_d$	0.01
$T_c$	0.03	s
$T_s$	0.002	s

5. EXPERIMENTAL RESULTS

After the robotic companion was developed, the authors conducted a series of experiments to demonstrate its performance. As the key component in the robot, the computer vision system was validated through a comparison with the measurement results obtained with the standard marker-based motion capture system. Subsequently, the performance of the entire system was quantified through the experiments of simple human-initiated movements and navigation of an indoor environment tracking human movement.

5.1. Validation of the Computer Vision System

To evaluate the performance of the computer vision system, an OptiTrack motion capture system (MCS) was used to obtain the ground-truth measurement of the user’s orientation and position, leveraging the high accuracy of its position measurement (sub-millimeter level) and rotation measurement (<1° error). The MCS consists of 8 Prime 17W motion capture cameras that use 850 nm IR illuminations (the same wavelength as that of the RealSense R200). To avoid interference between the MCS and the RealSense R200, the MCS was set to operate in its passive mode (with the IR illuminator off). We constructed two active targets respectively with four IR LEDs arranged in a T-shape holder and with four IR LEDs arranged at the corners of a flat board. As shown in Fig. 11, the T-shape holder is attached to the camera, and the flat board is tightly attached to the user’s chest. The MCS can detect the IR LEDs as 3D points in the MCS’s coordinate system. By employing the approach by Umeyama [37], we used the four points on the T-shape holder to estimate the transformation matrix from the MCS’s coordinate system to the camera’s coordinate system. Then we transformed the 3D points into the camera’s coordinate system (labeled as XYZ in Fig. 11). Next, we used the four 3D points on the flat board to compute its central point $p_o$ (which is also the center of the torso) and its normal vector ${\overline{n}}_o=\left\{n_x,n_y,n_z\right\}$. Following the method in [23], we computed the user’s heading angle as

$${\theta }_o={\text{sin}}^{-1}\left(n_x/\sqrt[2]{n_x^2+n_z^2}\right)$$ (20)

The heading angle ${\theta }_o$ and the central point $p_o$ were used as ground truth estimations for the user’s orientation and position in the experiments, respectively.

Figure 11.

The experimental setup for computer vision system testing.

The first experiment was conducted to investigate the accuracy of the image processing system for orientation detection. A human subject stood in front of the camera, and rotated from right to left and then turned back to right side, with the velocity at approximately 12 degree/s. The heading angles estimated by the computer vision system and by the MCS are plotted in Fig. 12. The estimation error is shown as the dashed red line. As can be observed in this figure, the measurement of the computer vision system matches the MCS measurement very closely: the root mean square (RMS) of the error is 2.16 degree and the maximum error is 4.14 degree. The result supports that the computer vision system is capable of computing the human orientation with small errors.

Figure 12.

Measurement of the user’s orientation: ground truth vs computer vision system.

The second experiment was conducted to evaluate the accuracy of the computer vision system for position detection. A human subject stood in the front of the camera about 1.3 m, then moved forward about 0.5 m and moved backward to the starting point. His walking speed was about 0.13 m/s. In this case, since the vertical and lateral displacements are negligible, we only compared the position estimation along the fore-aft axis. By setting the start point as the base point, the displacements of the human center estimated by the MCS and by the image processing system were plotted in Fig. 13. It shows that the human subject moves about 1m in 8 seconds. The RMS of for the human center displacement is 0.012 m and the maximum error is 0.024 m. The result supports that the image processing system is capable of computing the human position with small errors.

Figure 13.

Measurement of the user’s fore-aft displacement: ground truth vs computer vision system.

5.2. Performance of the Robot Controller

Based on the validation results presented in the section above, it is reasonable to assume that the relative position and orientation captured by the computer vision system is sufficiently accurate for robot control purpose. Based on this assumption, we used a CAN bus logging to collect the data obtained by the computer vision system, and analyzed these data to quantify the performance of the robot controller.

First, a step-response experiment was conducted to evaluate the controller’s performance in response to the user’s sudden movement in the fore-aft direction. In the experiment, the human user was asked to move backward swiftly for a certain distance (approximately 160 mm) and stop. The relative linear displacement between the robot and the user with respected to the time is recorded as shown in Fig. 14.

Figure 14.

The robot controller’s response to a translational step input.

Next, a similar step-response experiment was conducted to evaluate the controller’s performance in response to the user’s sudden turning movement. The human user was asked to rotate swiftly for a certain degree and stop. The orientation error between the robot and the human user with respect to the time is as shown in Fig. 15.

Figure 15.

The robot controller’s response to a sudden turning movement.

As can be observed in Figs. 14 and 15, the robot was able to respond to the step inputs rapidly with no visible overshoot. The translational motion error reduced from 158 mm to approximately zero in 1 second, and the rotation error reduced from 25.5 degree to approximately zero in 0.8 second. The minimal steady-state error and short settling time demonstrate that the robot controller is able to track the human user motion in both fore-aft movements and turning actions.

After the aforementioned step-response experiment, we also conducted experiments to evaluate the robot’s performance in tracking a human user’s continuous walking and turning movements. First, the human user walked forward in a straight line at a speed of approximately 1.0 m/s (a typical walking speed for older adults [38]). The human-robot relative position (i.e., fore-aft movement tracking error) was recorded for 30 seconds, as shown in Fig. 16.

Figure 16.

The human-robot relative position (i.e., fore-aft movement tracking error) in the straight-line walking experiment.

As can be seen in Fig. 16, the tracking error is very small compared with the overall movement, with the maximum at 0.03 m and the average at 0.01 m. The subject’s feedback indicates that the controller was able to create a comfortable experience of the robot accompanying the user, despite the fact that the subject was constantly moving. The frequency of the oscillation in the error is about 2 Hz, very close to the test subject’s walking cadence. It suggests that the peaks of the error were most likely to be caused by the intermittent movement of the upper body, and the robot was able to compensate for such intermittent movement through its own movement.

A similar experiment was conducted for the robot to track a human’s continuous turning movement. During the experiment, a human user made continuous turning movement by following a circle at a speed of approximately 60 degree/s. The human-robot relative orientation (i.e. turning tracking error) is shown in Fig. 17. Similar to the straight-line walking, the robot was able to change its orientation and effectively tracking the user’s turning movement.

Figure 17.

The human-robot relative orientation (i.e. turning tracking error) in the continuous turning experiment.

5.3. Demonstration of the Sway Suppression

Sway suppression, as described in Section 4.1, is an important component of the robot controller. As such, we conducted a set of comparison experiments to evaluate its effectiveness in robot control. In the experiments, the test subject walked in a straight line at a speed of approximately 1 m/s, and the robot’s sway movement is measured with and without the sway suppression algorithm implemented. For such measurement, an inertia measurement unit (MPU6050, InvenSense, San Jose, CA) was attached to the robot frame to measure its absolute orientation change. Figure 18 shows the comparison of the robot orientation trajectories with and without sway suppression. It can be clearly observed that the magnitude of the robot sway reduced significantly after the sway suppression algorithm was applied. To further quantify its effectiveness, the average value of the amplitude (AVA) and the root mean square (RMS) of the magnitude were calculated to facilitate the comparison, as summarized in Table 3. As indicated by these key metrics, the robot sway was reduced by over 50%, demonstrating the effectiveness of the sway suppression algorithm.

Figure 18.

Comparison of the robot orientation trajectories with and without sway suppression.

Table 3:

Comparison of robot sway with and without sway suppression

	Without sway suppression	With sway suppression
AVA (degree)	2.43	1.22
RMS (degree)	2.48	1.17

5.4. System Testing in a Typical Indoor Environment

The final part of the testing was conducted to evaluate the system-level performance of the robotic companion. This experiment was conducted in a typical indoor environment, with long and narrow hallways and right-angle turns, as shown in the map in Fig. 19. The total length of the route was approximately 60 meters. In the experiment, the test subject walked along the designated route, with no physical contact with the robot in the entire process.

Figure 19.

Map of the indoor environment for robot testing.

Figure 20 shows two typical photos of the robot-accompanied human walking, and more details can be found in the attached video. It can be clearly observed that the test subject was able to walk with ease, without expending extra effort to coordinate with the robot. In comparison, when a mobility-challenged individual uses a walking aid (such as walker and rollator) in such environment, the long walkways and multiple turns in the course would require the user to push and maneuver the device, causing significant physical and cognitive loads to the user. As such, the powered and worry-free operation enables the robotic companion to relieve its user from such burden, which is expected to improve his/her mobility significantly in the daily life. Additionally, to demonstrate the performance of the robotic companion when physically interacting with the human user, we also conducted experiments with a user holding the handles in walking. As shown in the second video attached to this paper, the performance of the robotic companion in such experiments remained largely unchanged (compared with the no-contact experiments), fully demonstrating the robustness of the robot control system in different conditions of interaction with the human user.

Figure 20.

Typical scenarios in the system testing: (a) straight walking; (b) turning.

6. CONCLUSIONS

In this paper the authors present the design and control of a novel robotic companion to assist the individuals suffering from mobility challenges. Unlike existing walker-type assistive devices, the robotic companion is able to track user movement by using a novel 3D computer vision system, which provides accurate and robust measurement of the user position and orientation relative to the robot without requiring the users to wear body-mounted sensors. The robotic companions was designed and fabricated as a low-cost mobile robot with two hub motor-powered wheels, operating in the differential steering mode for independent fore-aft movement and left-right turning. The computer vision system was developed to extract the human-robot relative position/orientation by processing the images obtained through an on-board 3D camera, and such information functions as the feedback signal to the robot control system to construct a closed-loop motion controller. In the controller development, a novel sway suppression algorithm was developed to reduce the robot heading oscillation caused by the human upper-body sway, and a discrete PID controller with integrated digital practical differentiator was developed to provide high control performance in the presence of measurement noises. The robotic companion system was experimentally tested to quantify the performance of key individual components and the entire system. The computer vision system was validated through a comparison with the standard marker-based motion capture system, in which the RMS of the orientation measurement error was only 2.16° and the RMS of the measurement error of the torso displacement was only 0.012 m; the robot motion control was tested through step response and continuous movement experiments, in which the average tracking error was approximately 0.01 m; the sway suppression algorithm was validated through the measurement of robot oscillation with and without the algorithm, and the suppression algorithm reduced the robot sway by over 50%; and finally the system performance was demonstrated in an experiment conducted in a typical indoor environment, and the human participant was able to walk through long walkways and make multiple turns without pushing or manipulating the robotic companion over the entire process. The results from the experiments support the conclusion that the robotic companion functions in the desired way in accompanying the user in his/her daily movement, and the worry-free operation is expected to reduce the user load and improve the user mobility in the daily life.