Tracking Exercise Motions of Older Adults Using Contours

Abstract

In this paper we describe the development of a novel markerless motion capture system and explore its use in documenting elder exercise routines in a health club. This system uses image contour tracking and swarm intelligence methods to track the location of the spine and shoulders during three exercises — treadmill, exercise bike, and overhead lateral pull-down. Validation results show that our method has a mean error of approximately 2 degrees when measuring the angle of the spine or shoulders relative to the horizontal. Qualitative study results demonstrate that our system is capable of providing important feedback about the posture and stability of elders while they are performing exercises. Study participants indicated that feedback from our system would add value to their exercise routines.

1 Introduction

Everyone can benefit from some type of exercise including, and perhaps most importantly, older adults. However, it has been shown that less than 25% of older adults aged 65 and older include a regular exercise routine in their daily activity [23]. Sedentary elders who begin an exercise program ultimately benefit from improved quality of life and reduced health care expenditures [7]. Additional benefits of a daily exercise routine for elders include prevention of falls, alleviation of depression, improved cognitive function, improved bone density, improved cardiovascular function — the list of benefits is virtually endless [6, 7, 12, 15, 19, 20, 23]. In [12], the authors discovered that exercise is an under prescribed therapeutic intervention due to misconceptions by elders, their caregivers, and their health care providers about exercise safety. For the reasons stated above, we aim to improve the exercise safety for older adults, which could have a significant impact on the overall health, and subsequent independence, of elders.

We conducted a pilot study that examined the human factors issues of a custom technology interface designed to capture range of motion and provide feedback to elderly people using exercise equipment. In health care, human factors researchers attempt to understand the interrelationships between humans and the tools they use, the environments in which they live and work, and the tasks they perform [21, 22, 27]. Thus, the goal of a human factors approach is to optimize the interactions between technology and the human in order to minimize human error and maximize human-system efficiency, human well-being, and quality of life [19]. This paper describes the research and development of the technology used in our human factors project. This technology is a novel exercise-feedback computer interface that combines image segmentation, contour tracking, motion capture, and swarm intelligence.

Section 2 introduces previous work in using contour tracking for human motion analysis. Section 3 gives a detailed description of our system while Section 4 outlines some results from our study as well as the validation of our methods. We summarize in Section 5.

2 Related Work

Human motion analysis is a well researched topic and is pertinent to many fields including sports medicine, nursing, physical therapy and rehabilitation, and surveillance. There have been special issues of journals and tracks in computer vision conferences completely dedicated to human motion analysis in video.

References [1, 5, 16, 25] provide a good background on human motion analysis techniques. The approach from [3] uses silhouette-based features to recognize falls in monocular video of elders in the home environment. In [29], the author proposes a method to analyze human pose during exercise. This method has the strength that it is generalizable to any pose; however, as the author points out, it is very error prone. Also, the assumption is made that the background subtraction (silhouette extraction) is near ideal. Achieving an ideal silhouette in a gym environment is virtually impossible as we will provide examples for in Section 4. Another assumption that is made in [29] is that the subject is facing the camera and is upright. We wish to measure the angle of the spine, as seen from the side view in both upright (treadmill) and sitting (overhead pull-down) poses; hence, this assumption makes this method undesirable for use in our research.

Active contours, called snakes [17], have been used to track face features (e.g. eyebrows and mouth) and humans in video. Although active contour methods are effective, they do not give us the direct capability of measuring the posture information we require. Hence, we use a contour tracking method based on the edge distance transform [24]. Consider this supporting example. Snakes can be severely affected by poor silhouette extraction. Because we are performing our research in a real environment, poor silhouette extraction is a reality we must consider in the design of our algorithm. Our proposed method is less affected by the poor silhouette and the angle of the spine relative to the ground is easily determined. To deduce the spine angle from the active contour in this example would require a more complex (and, perhaps, less accurate) method. Additionally, one could imagine that our method is, in essence, an active contour method where the contour is restricted to zero curvature.

3 Exercise-Feedback System

Our method tracks body contours in the video of exercising humans. The two contours we are interested in are the edge of the back (spine) as seen from the side view and the shoulders as seen from the rear or front view. Fig. 1 shows these two contours on example video frames of a research participant walking on a treadmill. Fig. 2 illustrates our approach in a block diagram. We designer our approach to be both robust and flexible.

Figure 1.

Silhouette extraction examples show spine and shoulder tracking of treadmill exercise with contour templates shown in gray.

Figure 2.

Block diagram of exercise-feedback system components — spine tracking on side view of treadmill exercise.

The environment in which we are performing this research study is a public gym; hence, our ability to control experimental conditions, such as lighting conditions, background environment, and subject clothing, is very limited. As a result, we chose simple and proven methods to perform the operations in our algorithm.

First, the silhouette of the human in each video frame was computed. We used a statistics-based background subtraction algorithm that is adapted from [28]. Second, the chamfer distance transform of each silhouette frame was computed, as in [18]. The chamfer distance transform provides an error surface upon which we can fit a contour template. We created a novel optimization algorithm called Roach Infestation Optimization (RIO) [11] to find the best position of the contour template, which, ideally, is located on the body contour of interest, either the back or spine. The best position of the contour template is defined by a temporal fitness function that accounts for exercise dynamics and template translation and rotation. We now describe in more detail each element shown in the block diagram in Fig. 2.

3.1 Human Silhouette Extraction

Silhouette extraction or background subtraction is a problem that is very pertinent to many fields of research, such as surveillance, activity recognition, and computer vision. However, this problem has many difficult facets including dynamic lighting conditions and backgrounds, poor scene illumination, inferior cameras, and highly variable foregrounds. It is beyond the scope of this paper to address these matters; however, we emphasize that extracting “good enough” silhouettes is essential to our algorithm.

The silhouette extraction algorithm we use is adapted from [28]. We, first, record the video of the scene without the participant — we denote these video frames as I_bg. Then we record the video of the participant performing the exercise — denoted I_bg. It is important to keep the camera parameters — such as pointing, contrast, and brightness — consistent for the recording of both I_bg and I_fg. The red-green-blue (RGB) digital images are then converted to a huesaturation-value (HSV) color space [8]. Then a statistical background representation is formed from approximately 100 frames of the background video I_bg (no human is in view). This statistical background representation is then used to classify each pixel in I_fg as either background or foreground. The participant is the foreground; hence, the background subtraction leaves only the image pixels that correspond to the image of the participant. Figs. 1(c,d) show the silhouettes computed from the corresponding video frames shown in Figs. 1(a,b). We denote the silhouette image of video frame f as S(i,j,f), where S(i,j,f), = 1 indicates the ith row, jth column pixel is a foreground pixel and S(i,j,f), = 0 indicates a background pixel.

3.2 Contour Tracking

We adapt the chamfer distance transform, described in [18, 24], and define

$$C(r,c,f)=max\left\{\underset{\forall i,\forall j:S(i,j,f)=0}{min}[{(r-i)}^2+{(c-j)}^2]\underset{\forall i,\forall j:S(i,j,f)=1}{min}[{(r-i)}^2+{(c-j)}^2]\right\}$$ (1)

Essentially, (1) calculates the minimum squared distance between each pixel location and the edge of the human silhouette. We compute (1) for each pixel in the image and this distance transform map can be used to determine the best location for a contour template. Let T be a template, i.e. T is a set of coordinates describing a shape. The template is a discrete list of pixel coordinates, which define the template shape. For example, a linear (line) template, such as that used to track the spine, could be defined as

$$\mathbf{T}=\{{[00]}^T,{[01]}^T,{[02]}^T\},$$

where, in this example, T is a vertical line, three pixels long. This template formulation is very general and can represent any types of shapes, including lines, curves, and broken shapes.

The template error score is

$$M({\overrightarrow{x}}_f,f)=\sum _{\{(r,c)\in T\}}C(r,c,f),$$ (2)

where x⃗_f = (x_f,y_f,θ_f) are the contour template parameters, and f is the video frame. The parameters x_f and y_f represent the translation and θ_f represents the rotation of the template. We then map (2) onto the interval [0,1] with a simple spline function of the form

$$M_{[0,1]}({\overrightarrow{x}}_f,f)=min\{M({\overrightarrow{x}}_f,f)/1000,1\}.$$ (3)

The reason we perform this mapping is so that in Section 3.3 we can use standard fuzzy operators to combine this objective function with a membership function that limits the search space.

The best contour template location is defined as

$${\overrightarrow{x}}_{\mathit{best},f}={\mathit{arg}}_{{\overrightarrow{x}}_f}{\mathit{minM}}_{[0,1]}({\overrightarrow{x}}_f,f)$$ (4)

We use a straight line to model the contour of the spine, see Fig. 3(a), and two sloping lines with attached vertical lines to model the contour of the shoulders, see Fig. 3(b). As Fig. 3 shows, the templates we used for this study are customizable for each participant. However, if one wished to use our technique to track other body contours, a template can easily be designed. Equation (2) is an error score of the fit of the contour template, for a given parameter vector x⃗_f, to the edge of the silhouette in frame f. The coordinates (r,c) over which the summation in (2) is computed, are found by the linear transformation

Figure 3.

Templates used to track (a) spine and (b) shoulders.

$${\left[\begin{array}{c}c\\ r\end{array}\right]}_i=\left[\begin{array}{cc}cos{\theta }_f& -sin{\theta }_f\\ sin{\theta }_f& cos{\theta }_f\end{array}\right][T_i]+\left[\begin{array}{c}{x}_f\\ y_f\end{array}\right]$$ (5)

where $[T_i]={[t_c{t}_r]}_i^T$ is the coordinates of the ith pixel in the contour template T. In our algorithm we define the center of the template as the origin, but this is arbitrary.

Fig. 4 illustrates the value of the template error scores M and M_[0,1] for two examples. Both examples use the same silhouette and contour template, only the parameter vector x⃗_f is changed. As Fig. 4(a) shows, the best location of the contour template — on the spine — results in the lower error score. Assuming that the contour template is defined properly and the silhouette image is ideal, the human contours can be tracked in video by solving (4) for each successive video frame.

Figure 4.

Example values of contour scoring function (2) for spine contour tracking of treadmill exercise

3.3 Temporal RIO-Based Contour Search

Under ideal circumstances where an ideal silhouette image can be computed, solving (4) would be sufficient for tracking the contours on the human. However, we conducted this research in a gym-environment; hence, ideal silhouettes were not always achieved. We added a temporal term to (4) that limited the candidate contour locations x⃗ to those that only changed slightly from the previous frame. In other words, because we are tracking human motion, we can assume that the spine or shoulder contours only move a small amount between video frames (video was taken at 7.5 frames-per-second). For each video frame, the error function that must be minimized is the fuzzy union of M_[0,1] and R

$$E({\overrightarrow{x}}_f,{\overrightarrow{x}}_{f-1}^{\ast },f)=max\{R({\theta }_f,{\theta }_{f-1}^{\ast }),M_{[0,1]}({\overrightarrow{x}}_f,f)\}$$ (6)

where ${\theta }_{f-1}^{\ast }$ is the previous frame’s best rotation parameter solution $R({\theta }_f,{\theta }_{f-1}^{\ast })$, is the membership in rotated more than expected for one video frame. The temporal damping function is designed such that large changes in the template parameters produce high membership. We use the following formulation for the membership R

$$R({\theta }_f,{\theta }_{f-1}^{\ast })=\{\begin{array}{ll}0,\hfill & \overline{\theta }\le a\hfill \\ 2{\left(\frac{\theta -a}{b-a}\right)}^2,\hfill & a\le \overline{\theta }\le \frac{a+b}{2}\hfill \\ 1-2{\left(b-\frac{\theta }{b-a}\right)}^2,\hfill & \frac{a+b}{2}\le \overline{\theta }\le b\hfill \\ 1,\hfill & b\le \overline{\theta }\hfill \end{array}$$ (7)

where $\overline{\theta }=\mid {\theta }_f-{\theta }_{f-1}^{\ast }\mid$ and, a and b set the inflection points of the spline. In essence, a sets the maximum expected change in rotation between frames, while b sets the point at which candidate solutions are severely punished. Values that we found effective for our study are a=5 degrees and b=10 degrees. Hence, it is clear, by comparing Eqs. (3) and (7), that R will dominate the value of E in (6) for changes in rotation angle greater than a. E reduces to M_[0,1] for θ̄ ≤ a

RIO [11] is a swarm intelligence method that attempts to find the global optima of objective functions, such as (6). In [11] we showed that RIO is more effective than Particle Swarm Optimization (PSO) [4, 14] at finding the global optima of several test functions. The RIO algorithm is based on the social behavior of cockroaches described in [9, 13, 26]. The cockroaches behaviors that we assimilated into the RIO algorithm are:

• Cockroaches search for the darkest location in the search space. The level of darkness at a location x⃗ ∈ ℝ^D is directly proportional to the value of the fitness function at that location F(x⃗) (in this paper F(r⃗) is E in (6, which also includes the temporal portion of the fitness);

• Cockroaches enjoy the company of friends and socialize with nearby cockroaches;

• Cockroaches periodically become hungry and leave the comfort of darkness or friendship to search for food.

Algorithm 1 shows the specific steps of RIO as described by the above behaviors. In the context of our systems for tracking contours on the human body, RIO works in the following way:

Algorithm 1.

Roach Infestation Optimization (RIO)

1. The roaches are initialized randomly within a pre-defined bounding box around the contour of interest — the spine or the shoulders — and within a predefined parameter space;

2. The RIO algorithm searches for the best set of parameters x⃗_f that minimize E;

3. Advance the video frame and return to step 1.

3.4 Interface

The interface we developed provides feedback to the study participants. The layout of our interface is shown in Fig. 6. The upper left shows the silhouette extraction result, the upper right shows the tracking contour on the silhouette. The lower left view is a zoomed-in view of the tracking area. This view provides the participant with a more detailed view on how their spine or shoulders look as compared to the contour tracking reference. Finally, the angle of the spine or shoulders is graphed on the lower right. The solid black line shows the angle at each video frame, while the dotted line is a running average of the angle. Thus, information on both the movement (solid line) and overall posture (dotted line) is shown on the graph. Section 4.2 discusses the participants’ views on the exercise feedback interface.

Figure 6.

Layout of contour tracking interface — upper left shows silhouettes, upper right shows tracking result on silhouettes, lower right shows exploded view of tracking area, and lower right is a plot of the angle versus video frame.

4 Results

4.1 Motion Capture Validation

We validated our contour tracking method with a Vicon motion capture system (http://www.vicon.com). The Vicon system is a three-dimensional tracking system that tracks the location of reflective markers that are placed on the body and is one of the most accurate methods of tracking 3D position of the human body. In contrast to our system, which consists of an $800 PC and two $100 webcams, the Vicon system we used in this study cost >$160,000 and uses seven high-resolution cameras.

Figures 7(a,b) shows the placement of the markers on the body that we used in the validation experiment. Figure 7(a) shows the three markers on the spine — one marker was placed at the base of the C7 cervical vertebrae (the base of the neck), one marker was placed between the T5-T7 thoracic vertebra, and one marker was placed between the L5 lumbar and S1 sacral vertebra (around the belt line). These marker locations were used to validate our spine tracking method. For the shoulder tracking validation experiment, one marker was placed on the tip of each of the acromion (the top of the shoulder) and one marker was placed on the C7 cervical vertebrae, as shown in Fig. 7(b).

Figure 7.

Vicon reflective marker locations shown as circles.

We validated our spine tracking method by having the subject perform a series of bends forward from the waist. Each experiment consisted of 3–4 bends, starting from an upright posture and bending forward. Figure 8(a) plots the results of this validation study — 0 degrees is vertical and positive angle indicates leaning forward. The dotted black line indicates the spine angle measured by the Vicon system, the solid black line indicates the angle measured by our contour tracking method, and the solid gray line indicates the absolute error, in degrees, of each measurement. The mean error for the spine tracking validation study shown in Fig. 8(a) was 2 degrees.

Figure 8.

Validation results of contour tracking methods.

The shoulder tracking was validated in a process similar to the spine tracking; however, this time the subject was asked to sway side to side. Figure 8(b) shows the results of the shoulder tracking validation study. Again, the dotted black line indicates the angle of the shoulders — positive angle indicates the subject is leaning to the right and negative angle indicates leaning to the left — as measured by the Vicon system, the solid black line indicates the angle measured by the contour tracking method, and the solid gray line indicates the absolute error, in degrees, at each measurement. The mean error for the shoulder tracking validation study shown in Fig. 8(b) was 6 degrees. Note that this error includes the portion of the validation where the contour tracking method lost track (at about frame 600). The tracking degraded at the end of the validation run because of poor silhouette extraction. Fig. 9 shows the frames near the end of the shoulder tracking validation where the tracking fails. If we only consider the portion of the validation where the silhouette extraction was accurate (up to about frame 600), the mean error was 2 degrees.

Figure 9.

Silhouette frames of shoulder tracking validation that show tracking failure.

4.2 Study Results

Our pilot study consisted of a qualitative study of key informant interviews of 35 older adult participants aged 65 years and older. We recorded images of each participant doing each of the three exercises and then showed each participant the results of the exercise-feedback system. Structured key informant interviews were then conducted to gain feedback about how the interface could be developed further to support the participants during their exercise routines. These interviews were recorded and transcribed and phenomena that emerged and reappeared across all interviews and observations were identified. Preliminary results of the qualitative study are described in [10] and a thorough analysis of the participant interviews is in [2].

Preliminary results show that 100% of research participants were interested in seeing their images after they performed the exercises. All participants were most interested in how their posture appeared during the period of exercise. Participants discussed that processed images assisted them to visualize how they interacted with the exercise equipment, if they were using a good technique to perform the activities, and if they had any unusual movements while performing the desired tasks. For example, one participant stated,

Well, it seems to me that [the images tells you how to] use the body the way you’re supposed to use it to maintain good leg support and arm support. I do sway back and forth, but I don’t think you can do anything other than that when your body is moving like it is below the trunk.

Many of the key informants interviewed discussed their fear of losing their balance and falling while walking; they indicated that the images provided them the ability to see if they were maintaining good balance over the core of their body, which is important in preventing falls. Some participants indicated the images would provide some added value to their exercise safety: making them feel safer, less likely to be injured, and less likely to fall. Other participants indicated that the images were useful but could not really take the place of a trainer who could help interpret what they need to do to be the most successful in reaching their exercise goals. Only two participants stated that they always felt safe on the exercise equipment and, thus, did not see any benefit to the interface.

Figures 10(a,b) provide a comparative example of two study participants. The participant shown in Fig. 1 had issues both with posture, hunching, and gait, which manifested as a significant limp. In contrast to the participant shown in Fig. 1, the participant shown in Fig. 11 had good mobility and little to no afflictions that affected gait and posture. Figure 10(a) shows the spine angle plot of the participant shown in Fig. 1 and Fig. 10(b) shows the spine angle plot of the participant shown in Fig. 11. As these plots show, the contour tracking method is able to detect not only the over difference in the participants’ postures — as represented by the overall deviation from 0 degrees — but the differences in their gaits are shown by the differences in the patterns of the two plots.

Figure 10.

Spine angle plots that show how contour tracking captures posture information

Figure 11.

Sample video frames of participant with good mobility on treadmill.

5 Conclusions and Future Work

Our exercise feedback interface has broad application in fields where measuring human body movement is important — e.g. physical therapy, sports medicine, and nursing. We applied our methods to eldercare, specifically to improve the safety and effectiveness of exercise. Our study included key informant interviews of 35 older adult participants and these interviews indicated that our interface was both effective in showing older adults information on how they move while they exercise, but, also, showing older adults areas in which they could improve their exercise.

The markerless system has potential uses for the clinician as well. Quantitative information can be derived from the kinematic data to use as baseline measurements for comparisons throughout a rehabilitation program; e.g. following an invasive surgery such as joint replacement. Periodic feedback using the system could be used to track progress and help with patient education regarding body mechanics that prevent further postural abnormalities and consequent adaptations.

The clinician will be able to collect objective data concerning posture during different functional activities and assess symmetry of movement between left and right sides. Immediate postural feedback is available to the patient and the health-care provider to facilitate correction of faulty positions and movements during exercise. The ability to look at specific contours on the video will help emphasize potential problem areas specific to the patient resulting in earlier therapeutic interventions.

In conclusion, the contour tracking method we present in this paper has direct and pertinent benefits. We validated our method against a gold-standard motion capture system, the Vicon 3D marker-based system, and showed that our system is accurate in measuring the angle of the spine and shoulders relative to the horizontal floor plane. Initial validation experiments showed that our technique is quite accurate with only a 2 degree error in measuring the angle of the spine and shoulders. Additionally, our technique is generalizable, such that health professionals could choose to track other parts of the body that can be represented by a rigid contour template.

A common theme among the key informant interviews was that participants were not provided with a goal on the interface. The participants desired an expert-guided goal or demonstration that would help them achieve a proper and safe exercise form. In the future we hope to use the results of the contour tracking to provide a synchronized video representation of an expert performing the same exercise as the participant. Additionally, feedback could be provided in the form of a linguistic goal — “stand up straight” — or quantitative goal — green and red regions on the graph that represent a “good” and “bad” exercise form.

Currently, we are adapting the contour tracking methods for use in a home environment. A large part of our ongoing eldercare research is the use of technology to help older adults maintain independence. We are investigating the use of silhouette-based techniques to detect falls, assess mobility, and perform activity analysis. These techniques are important to effectively and inexpensively address the needs of our aging population.

Figure 5.

Plot of $R({\theta }_f,{\theta }_{f-1}^{\ast })$ for inflection points a=5 and b=10.