Prediction of sustained harmonic walking in the free-living environment using raw accelerometry data

Abstract

Objective

Using raw, sub-second level, accelerometry data, we propose and validate a method for identifying and characterizing walking in the free-living environment. We focus on the sustained harmonic walking (SHW), which we define as walking for at least 10 seconds with low variability of step frequency.

Approach

We utilize the harmonic nature of SHW and quantify local periodicity of the tri-axial raw accelerometry data. We also estimate fundamental frequency of observed signals and link it to the instantaneous walking (step-to-step) frequency (IWF). Next, we report total time spent in SHW, number and durations of SHW bouts, time of the day when SHW occurred and IWF for 49 healthy, elderly individuals.

Main results

Sensitivity of the proposed classification method was found to be 97%, while specificity ranged between 87% and 97% and prediction accuracy between 94% and 97%. We report total time in SHW between 140 and 10 minutes-per-day distributed between 340 and 50 bouts. We estimate the average IWF to be 1.7 steps-per-second.

Significance

We propose a simple approach for detection of SHW and estimation of IWF, based on Fourier decomposition.

1 Introduction

Accelerometers are now widely used to monitor physical activity in large observational studies where thousands of subjects are observed for weeks at a time. Walking patterns are very often the main focus of such studies as they have been shown to be associated with major health and aging outcomes, including: survival (Studenski et al 2011), Parkinson disease (Din et al 2016), obesity (Browning 2006) and overall physical capability (Godfrey et al 2015). Here, we propose method for estimating when and how people walk based on high-frequency data obtained from wearable accelerometers. These data exhibit extraordinary levels of heterogeneity due to the natural within- and between-person variability, as well as to measuring devices that are prone to batch effects, rotations and random artifacts. Heterogeneity makes walking prediction in the natural environment much more difficult than for “in-the-lab” experiments (Ermes et al 2008, Grant et al 2006a, Francois et al 2009). The differences between data collected “in-the-lab” under strict protocols and data collected in large observational studies is dramatic. Thus made us coin the term data collected “in-the-wild” to emphasize the highly unstructured nature of the data obtained during free-living human activity.

There are a number of accelerometry-based wearable devices designed to recognize and quantify walking “in-the-wild”. These include ankle-worn step counters (Coleman et al 1999) that provide well-validated information on occurrence and duration of walking as well as the number of steps per minute. Such devices have been widely used in large epidemiological studies, allowing researchers to study walking habits of individuals (Dall et al 2015, Orendurff et al 2008). Another popular device that has been used to detect walking and posture of individuals is a thigh-worn activPAL monitor (PAL Technologies Ltd, Glasgow, UK). Grant et al. (Grant et al 2006) discuss the validity of these devices in posture and activity recognition for “in-the-lab” experiments, whereas in Dall et al 2015 and Klenk et al 2012 authors use activPAL to collect data on walking patterns in a large group of participants “in-the-wild”. While the utility of such devices is undisputable, researchers might often want to address overall physical activity levels rather than the ambulation exclusively (Weuve et al 2004). In this paper we propose a method for detection of walking on data collected by hip-worn, accelerometry-based physical activity monitors.

Our work was preceded by a number of approaches. For example papers by Dijkstra et al 2008 and Weiss et al 2013 both proposed walking-recognition procedure for tri-axial, hip-worn accelerometers. These methods use pre-defined threshold of recorded signal to classify walking activities. The manuscript of Lugade et al 2014 introduced an algorithm for classification of postural orientation and movement using data collected by four custom-build activity monitors, while their follow-up work Fortune et al 2014 discussed the problem of gait features extraction. The paper by Maurer et al 2006 presented a range of classifiers, for different body location of the sensor, including hip-worn devices. Most recently, Hickey et al 2017 introduced prediction algorithm dedicated for data collected in free-living environment and validated it against video recordings of daily activities.

Definition of walking based on the accelerometry measurements is quite ambiguous. For example, the accelerometry signal might have a fundamentally different structure for a 6 minute relaxed walk versus multiple bouts of a few seconds of walking interspersed with other activities. Moreover, laboratory studies typically collect labeled data that is consistent with the 6 minute walking (e.g. 400 meter walking (Chang et al 2004) or simply, 6 minute walk task (Enright 2003)). Thus, in practice it is often difficult to validate walking periods that are not consistent with the laboratory-standardized definition without proper gold-standard labels (e.g. video recordings). For these reasons we focus here on sustained harmonic walking (SHW), defined as walking for at least 10 seconds with low variability of step frequency. Based on our observation, we assume that such type of walking is similar in nature to the well-controlled walking, therefore can be identified with methods trained on data collected “in-the-lab” and applied in “in-the-wild” settings.

2 Methods

2.1 Data collection

The data were collected in the Developmental Epidemiologic Cohort Study (DECOS), a study of healthy older individuals. Participants wore tri-axial ActiGraph GT3X+ device collecting the raw accelerometry signal with a sampling frequency of 80 Hz. The devices were placed on the left hip via an elastic strap. The weight of the devices was 18 grams and the dimensions were 4.6cm × 3.3cm × 1.5cm.

We concentrate on the data from N = 49 individuals (24 females and 25 males) who have both “in-the-lab” and “in-the-wild” accelerometry measurements together with associated demographic and clinical covariates. Median age of participants was equal to 78.0 years (Q1=74.0, Q3=82.25).

During the “in-the-lab” phase of the experiment all participants were asked to perform a series of physical tasks including 400-meter fast walk, simulated dressing, simulated shopping and chair stands. During 400-meter fast walk participants were instructed to walk 10 laps, 40 meters each, at fast pace. For simulated dressing, participants were instructed to unfold lab jacket, put jacket on (no zipping or buttoning), then remove it, place on a hanger, and put the hanger on a nearby hook. For simulated shopping participants were instructed to walk few steps along the wall, stop by target items, remove them from the upper shelf and place them on the lower shelf. When completed, participants walked back the other direction, moving target items back to the top shelf.

Median time to finish 400-meter walk was 5 minutes 49 seconds (Q1=4 min. 40 s, Q3=6 min.). Dressing activity was set up to last exactly 3 minutes, median time to finish shopping activity was 2 minutes 38 seconds (Q1=1 min. 56 s, Q3=3 min.), median time of standing up from the chair task was 19 seconds (Q1=13 s, Q3=24 s).

“In-the-wild” accelerometry data activities were collected over a one-week period in the natural living environment with no activity labels.

2.2 Modeling approach

2.2.1 Definitions and notation

We define SHW as a steady-pace walking performed for at least 10 seconds with a roughly constant step-to-step frequency. Here we propose a method for automatic recognition of SHW intervals in raw accelerometry data together with estimation of instantaneous walking (step-to-step) frequency. The reported frequency of human steps “in-the-wild” ranges between 1.4 - 2.5 Hz (Pachi and Ji 2005). However, we will conservatively focus on the range 1.2 - 4.0 Hz to include slow walking of older individuals and running.

Let x(t) = {x₁(t), x₂(t), x₃(t)} denote the measured signal, where x_k(t) (expressed in g-units) is the measurement at time index t = 1, … T along the axis k = 1,2,3. For notational simplicity we drop the subject index. We denote the sampling frequency of the data expressed as number of observations per second by f₀.

Our goal is to estimate the walking indicator y(t), where y(t) = 1 corresponds to SHW and y(t) = 0 corresponds to non-SHW. This is achieved by employing a short-time Fourier transform (STFT) for each axis separately. The discrete Fourier transform (DFT) for axis k is $X_k\left(f\right)={\displaystyle {\sum }_{t=0}^{T-1}{x}_k}\left(t\right)e^{-i2\pi f\frac{t}{T}}$, where f is the frequency index, t is the time index and T is the total number of observations. We define the short time Fourier transform (STFT) at time t for axis k of the acceleration signal x(t) as

$$X_k\left(t,f;\tau \right)={\displaystyle {\sum }_{u=[t-\tau /2]}^{[t+\tau /2]}{x}_k}(u)h(u)e^{-i2\pi f\frac{u}{\tau }},$$ (1)

where τ is a parameter specifying the number of observations in the interval centered at t. We use the Hann window defined as, h(u;τ) = 0.5[1 − cos{2πu/(τ − 1)}]. The spectrum is defined as S_k(t,f;τ) = |X_k(t,f;τ)|, where |·| denotes the absolute value of a complex number. Next, we introduce the comb function, a function that “combs” the spectrum for those frequencies that are likely to be related to walking (see Figure 1). A comb function is defined by a fundamental frequency s, and the thickness of the comb “teeth” with the same width for all frequencies and the total number of teeth equal to n_m. For any frequency s, we define a neighborhood, $N_s=\{s-\frac{1}{\mathrm{U}},s,s+\frac{1}{U}\}$, where the local FFT is applied, where U is the duration of the window expressed in seconds. Thus, N_s is the shortest frequency interval centered at s and consisting of three subsequent frequency components. The comb family of functions, C(f;s), indexed by s, is defined as

$$C(f;s)=\{\begin{array}{cc}1& \text{for}f\in {\cup }_{l=2}^{n_m}{N}_{ls}\\ 0\end{array}$$ (2)

Spectra shown in the top panel of Figure 1 correspond to a sample walking interval. The bottom panel displays the comb function corresponding to the fundamental frequency of walking acceleration signal, equal to s = 1.034 Hz. The tooth centered at 2s = 2.067 Hz also contains the frequencies between 1.967 and 2.167 Hz. There are additional “comb teeth” centered at 3s = 3.102Hz and so on. Figure 1 also displays an example of another comb shown in red corresponding to s = 1.40 Hz, a frequency unrelated to walking acceleration signal. Note that integrating the spectrum in areas corresponding to the “black” comb will result in a higher value than for the spectrum integrating the areas corresponding to the “red” comb, which does not match the spectral peaks. The comb function idea was inspired by the widely used comb filter of harmonic components (Deller et al 2000). However, in contrast to a filter that uses the entire range of spectral components, our comb uses only frequencies between 2s and n_ms to limit the number of high and low frequency components and reduce both: high-frequency random noise and low-frequency oscillation resulting from other non-harmonic body movements.

Figure 1.

Top figure shows Fourier spectra of tri-axial acceleration signal of walking. Bottom figure shows “Comb” function C(f; s) for s = w(t) = 1.034 Hz (black) and for s = 1.40 Hz (red). Black lines correspond to the spectral lines representing walking signal components while red lines miss those spectral lines completely.

2.2.2 Prediction of walking periods and its characteristics

We now provide the technical description of the SHW prediction approach. Specifically, for each axis k, we define the area under the full spectrum, S_k(·), as $I{S}_k(t)={\displaystyle {\int }_{f=0}^{f_0/2}{\mathrm{S}}_k}(f,t;\tau )$ df and the area under the spectrum corresponding to the comb function C(f,s) as $I{S}_k(t,s)={\displaystyle \int { }_{f=0}^{f_s/2}}{S}_k(f,t;\tau )C(f;s)$ df. All the functions under the integral are positive and IS_k(t) > IS_k(t, s) for every s and t,. Next, we define:

$$Y_k(t,s)=\frac{I{S}_k(t,s)}{I{S}_k(t)-I{S}_k(t,s)}$$ (3)

which is a measure of the size of the periodic content of the component of accelerometry signal corresponding to the fundamental frequency s, along axis k.

We use Y(t,s) = max_k{Y_k(t,s)}, which is the maximum of the fraction of the signal explained by the frequency s along the three axes k=1,2,3 to estimate the SHW periods. For a threshold δ of Y(t, s), we estimate that a person performs SHW by

$${\widehat{y}}_{\delta }(u)=\{\begin{array}{c}\text{if} \underset{s\in D_f}{max}Y(t,s)>\delta ,\\ 0,\end{array}$$ (4)

for every u ∈ [t − τ/2,t + τ/2], where δ is a threshold δ of Y(t,s), and D_f is the family of frequencies corresponding to walking (here, D_f = 1.2, …, 4Hz). The estimation is repeated in 10-second windows shifted by 1 second. The duration of each walking bout is defined as the number of consecutive time windows where walking was estimated to occur plus the window length multiplied by the overlap. For example, if walking was detected in 8 consecutive windows, the walking bout was classified as lasting for 8 + 0.9 × 10 = 17 seconds. It is important to note that we cannot estimate duration of the walking bout that is shorter than the length of the window τ. Therefore, if walking was detected for only one window it will be classified as a bout lasting for 10 seconds.

For SHW periods, we estimate IWF, denoted as $\widehat{w}(t)$, as the double of frequency s, that maximizes Y(t, s). More precisely,

$$\widehat{w}(t)=\{\begin{array}{c}2\mathit{ar}\mathit{gma}{x}_{s\in D_f}\left\{Y(t,s)\right\}\mathit{for}{\widehat{y}}_{\delta }(t)=1,\\ \mathit{Nan},\end{array}$$ (5)

The complete description of the approach is summarized in Algorithm 1.

Algorithm 1.

Input: x(t) - accelerometry signal, fs - sampling frequency, T - observation time, τ - time window, δ - threshold, smin = 0.6Hz, smax = 2.0Hz.

Output: y(t) - walking indicator, v(t) - vector magnitude, w(t) - IWF.

Step 1. Compute the value of the “comb” function C(f; s) for each value of s ∈ Df.

Step 2. Compute the spectrum, Sk(t,f,τ), for each axis k = 1,2,3 and the area under the spectrum ISk(t) for each t.

Step 3. Compute the partial area under the spectrum ISk(t,s) for each s and each k.

Step 4. Calculate the periodic content of the signal $Y_k(t,s)=\frac{I{S}_k(t,s)}{I{S}_k(t)-I{S}_k(t,s)}$ for each k.

Step 5. Estimate yδ(t) with ${\max }_{s\in D_f}{Y}_k(t,s)$.

Step 6. For the times t with ${\widehat{y}}_{\delta }(t)=1$ estimate w(t) by finding the s that maximizes Y(t,s).

Step 7. Identify walking if any of the estimators ${\widehat{y}}_{\delta }(t)$ is 1.

2.3 Validation using “in-the-lab” data

To validate the method, we used data collected during “in-the-lab” phase of the DECOS study.

2.3.1 Selection of the tuning parameters

In this section, we discuss the tuning parameter choices: the window length τ, the threshold δ of Y(t,s) and the number of harmonics (comb teeth) n_m.

There are trade-offs in choosing τ as: 1) longer windows result in more precise spectrum estimation while 2) shorter time windows are less likely to capture changes in walking frequency or changes in activity type. Based on the empirical evaluation of “in-the-lab” data, we have found that a τ = 10 second interval is long enough for people to maintain their IWF yet short enough to not be sensitive to smooth changes of IWF, resulting in clean spectral signatures.

SHW is estimated in time increments equal to 1 second, so consecutive windows overlap by 90% and a one second interval is declared to be SHW if any of the intervals containing it is estimated to be a SHW. The duration of each walking bout is equal to the number of consecutive time windows where walking was estimated to occur plus the window duration multiplied by the overlap. For example, if walking was detected in 8 consecutive windows, the walking bout was determined to have lasted 8 + 0.9 × 10 = 17 seconds. It is important to note that we cannot estimate a length of the walking bout that is shorter than the length of the window τ. Therefore, if walking was detected for only one window it will be classified as a bout lasting for 10 seconds.

Choice of the threshold δ and the number of harmonics n_m are inter-related. The proportion of the variability $({max}_{s\in D_f}Y(t,s))$ explained is an increasing function of the number of harmonics n_m. We studied δ as a function of n_m, for n_m = 2, …, 17, where the upper limit 17 is determined by the sampling frequency of the raw accelerometry data. We estimate the density function of ${max}_{s\in D_f}Y(t,s)$ for all SHW and non-SHW periods for all subjects. The parameter δ was then estimated for each subject separately as the intersection between the subject-specific SHW and non-SHW density functions. Figure 2 displays the boxplot of the estimated δ values across subjects as a function of the number of harmonics, n_m. While some between-subject variability exists in the estimation of δ at every value of of n_m, having a population level value simplifies the procedure considerably. For example, when n_m = 6 harmonics are used an estimated median value of δ at the population level is equal to 0.115.

Figure 2.

Boxplot representing distributions of subject-specific δ vs. n_m.

The selection of the number of harmonics, n_m, is important for the estimation of IWF, w(t) (equation 5). In principle, we would like to utilize as many frequencies as possible without degrading IWF estimation. To study the choice of n_m, we have calculated the IWF, w(t), for every subject at each time point during their 400-meter walk. For every subject, we then calculated the coefficient of variation of IWF during this period. Given that this is a well-controlled experiment, the coefficient of variation of the IWF is expected to be small. Figure 3 displays the coefficient of variation as a function of number of harmonics for every subject. The average coefficient of variation is relatively stable for n_m = 2, …, 6 and it starts to increase for larger values of n_m. Thus, to use as much information as possible and still keep the coefficient of variation small, we selected n_m = 6.

Figure 3.

Boxplot representing coefficient of variation as a function of number of harmonics.

2.3.2 Performance of the algorithm

For further processing we used τ = 10 and δ = 0.115 and a range of possible IWF between 1.2 and 4.0 steps/second. The upper boundary for possible IWF was purposefully set to be relatively high to account for running for active individuals.

Next, we calculated specificity, sensitivity and prediction accuracy of the algorithm in three separate classification tasks. Namely, shopping vs. walking, chair stands vs. walking and dressing vs. walking. Shopping, chair stands and dressing were defined as non-walking activities, whereas 400-meter walk was defined as a walking activity. For shopping and chair standing activities specificity was equal to 95% and 97% respectively. Specificity was the lowest for dressing activity and was equal to 87%. Sensitivity of the classifier was equal to 97%. Prediction accuracy was equal to 97% for shopping vs. walking and chair stands vs. walking and 94% for dressing vs. walking.

3 Analysis of “in-the-wild” data

We applied the approach described in Section 2.3 to data collected during “in-the-wild” portion of the experiment, to estimate when SHW occurs together with the corresponding IWF. We used the same tuning parameters as in section 2.3.2 (τ = 10 and δ = 0.115 and a range of possible IWF between 1.2 and 4.0 steps/second, with 90% time-window overlap).

Figure 4 displays the total SHW time (top panel) for each of the 49 participants together with the corresponding total number of walking bouts (bottom panel) as per-day averages from the 7 days of activity. Results are sorted in decreasing order according to the estimated average total walking time. Color shading corresponds to different length of walking bouts. For example, the width of the lightest blue bars in the top panel of Figure 4 corresponds to the total walking time from walking bouts equal to 10 seconds. The yellow bars in the bottom panel of Figure 4 display the total number of bouts equal to 10 seconds. Results indicate that long SHW time does not necessarily indicate a large number of SHW bouts. For example, subject 1 had the longest SHW time per day (140 minutes), spread in 210 SHW bouts, which is about the third quartile of the number of SHW bouts for this group. Results indicate that the majority of daily walking bouts for all subjects are between 10 and 30 seconds.

Figure 4.

Bar plot presenting the total time of walking and corresponding number of bouts.

Figure 5 provides the lasagna plot (Swihart et al 2010) for daily walking patterns for all subjects. The shades of blue correspond to the number of minutes of walking within a one-hour window. Each row corresponds to one day and dashed red lines separate data for individual subjects. Results were sorted using the same ordering as in Figure 4, with subjects with the highest average daily walking time shown at the top.

Figure 5.

Lasagna plots presenting number of minutes of walking per one hour. Rows represent consecutive days. Red lines separate different subjects.

Figure 6 displays the boxplots of IWF for each subject sorted according to total walking time. The width of boxes is proportional to the number of walking bouts.

Figure 6.

Box-plot presenting IWF for each participant observed “in-the-wild”. Width of boxes is proportional to the number of observations.

4 Summary and discussion

We have introduced the definition of SHW together with the classification algorithm based on the quantification of the “local degree of periodicity” of all tri-axial time series. The performance of our method depends strongly on the assumption that SHW is a repetitive and sustained process in a particular time window. For example, it is possible for a window of 10 seconds to contain 3 to 4 seconds of walking which are not recognized as walking. Thus, our method will tend to work well at recognizing sustained walking for 10 and more seconds. This can be seen in the results obtained “in-the-lab” where specificity for shopping vs. walking was 95% even though shopping task consists of some non-sustained walking. Performance of proposed method is comparable to the previously published results. For example manuscript by Dijkstra et al 2008 reported sensitivity of 89.5%, whereas in the paper by Lugade et al 2014 authors obtained median sensitivity ranging between 84% to 95% depending on gait velocity. Also, the manuscript by Maurer et al 2006 reported prediction accuracy equal to 87% for hip worn accelerometers. That compares favorably to our approach with prediction accuracy between 94% and 97%.

Based on our observations of the data we assumed that SHW is similar in both “in-the-lab” and “in-the-wild” settings and that methods developed using “in-the-lab” data can be scaled-up to the free-living experiment. In reality, however, it is difficult to talk about classifiers performance “in-the-wild” without proper gold standard labels. We believe that our method performs well based on the comparison of obtained results to previously published studies that used accelerometry. For example, paper of Klenk et al 2012 reported similar daily walking duration in elderly population, equal to 104.4±50.7 minutes and 102.9±47.8 minutes for males and females respectively. In the manuscript by Orendurff et al 2008 authors presented comparable distribution of durations of walking bouts, where 60% of all walking bouts lasted 30 seconds or less, with 20.1% of walking bouts lasting for 10 seconds or less. That suggests significant number of walking bouts that do not fall into a definition of SHW. Reference by Dall et al 2015 reported an average value of step frequency around 2.0 steps/second, therefore slightly higher than indicated by our results. Possible explanation of this discrepancy is that the studied population was much younger (mean age 46 years), therefore characterized by higher gait speed (Studenski et al 2011) and consequently higher step frequency. Although the arguments given above cannot replace proper validation of our algorithm’s performance, they suggest that our method is promising when analyzing high-density accelerometry data collected “in-the-wild”.

In future studies, we will explore alternative validation methods for movement recognition “in-the-wild” using multiple wearable sensors as well as investigate the associations between the SHW features and health outcomes.