Accuracy of a novel multi-sensor board for measuring physical activity and energy expenditure

Abstract

The ability to relate physical activity to health depends on accurate measurement. Yet, none of the available methods are fully satisfactory due to several factors. This study examined the accuracy of a multi-sensor board (MSB) that infers activity types (sitting, standing, walking, stair climbing, and running) and estimates energy expenditure in 57 adults (32 females) 39.2 ± 13.5 years. In the laboratory, subjects walked and ran on a treadmill over a select range of speeds and grades for 3 min each (six stages in random order) while connected to a stationary calorimeter, preceded and followed by brief sitting and standing. On a different day, subjects completed scripted activities in the field connected to a portable calorimeter. The MSB was attached to a strap at the right hip. Subjects repeated one condition (randomly selected) on the third day. Accuracy of inferred activities compared with recorded activities (correctly identified activities/total activities × 100) was 97 and 84% in the laboratory and field, respectively. Absolute accuracy of energy expenditure [100 – absolute value (kilocalories MSB – kilocalories calorimeter/kilocalories calorimeter) × 100] was 89 and 76% in the laboratory and field, the later being different (P < 0.05) from the calorimeter. Test–retest reliability for energy expenditure was significant in both settings (P < 0.0001; r = 0.97). In general, the MSB provides accurate measures of activity type in laboratory and field settings and energy expenditure during treadmill walking and running although the device underestimates energy expenditure in the field.

Introduction

The ability to relate physical activity to health depends on its accurate measurement. In clinical and population-based research, measuring physical activity is necessary because it is a behavioral factor that plays a critical role in energy balance and in the etiology and prevention of chronic disease (Helmrich et al. 1991; Pate et al. 1995; Leon et al. 1997; Wannamethee and Shaper 2001; Knowler et al. 2002; Bianchini et al. 2002). Precise information about physical activity is also needed to estimate the energy expenditure of measured activities.

Research to refine and improve methods for measuring physical activity has been ongoing for years (Chen and Bassett 2005; Warren et al. 2010). Although substantial progress has been realized, none of the available methods are fully satisfactory because of problems related to cost, convenience, and measurement error. The major challenge in establishing the accuracy of any one method under free-living conditions has been the lack of a suitable gold standard for comparison. Furthermore, methods must be evaluated not only for their efficacy in measuring individual activity levels, but also for feasibility in population-based studies (U.S. Department of Health and Human Services 1996).

To address limitations of existing methods, we developed a portable multi-sensor board (MSB) that captures diverse cues from ongoing human movements to infer specific types of activity (Lester et al. 2005, 2006; Brunette et al. 2005). The device uses novel pattern-recognition approaches to identify activity type and speed of movement. The MSB also provides contextual information not provided by current devices, for example, walking indoors compared to outdoors, or walking in the proximal neighborhood compared to a distal location because it measures activity in a real time and space continuum. The purpose of this study was to evaluate our previously established methods for estimating physical activity type in a larger group of volunteers under two experimental settings and to determine if we could accurately estimate the energy expenditure of the identified activities. We hypothesized that the MSB provides valid and reliable measures of major physical activity types, including sitting, standing, walking, stair climbing, and running, and the associated energy expenditure under both laboratory and field conditions.

Methods

Participants

Volunteers were recruited from a university and surrounding metropolitan area using print media (e.g., fliers and newspapers). Interested individuals contacted study staff and were screened for conditions limiting mobility using a questionnaire (ACSM 2006). Demographics including age, sex, and height and weight were obtained by self-report. We screened volunteers so that the sample would consist of a broad range of ages and body sizes to ensure that our device would be applicable to a broader population. Potentially eligible participants were then scheduled for an initial test at which time responses to the activity questionnaire were reviewed, and eligible participants signed an informed consent form reviewed and approved by the local Institutional Review Board.

Procedures

Subjects completed three tests on separate occasions in the following order: laboratory, field, and a repeat of either the laboratory or field test chosen at random as a measure of test–retest reliability. Before the first test, with no shoes and wearing light (exercise) clothing, subjects were weighed on a self-zeroing digital scale while height was obtained using a wall-mounted stadiometer. Body mass index (BMI) was expressed as kg/m². During all tests, subjects wore a single MSB on the right hip, clipped to a fanny pack secured around the waist, an Actical^® monitor (Actical Software v2.1, Mini Mitter Company, Inc., Bend, OR, USA) on the left hip secured to a Velcro strap, and a heart rate monitor (model T34, Polar Electro Oy, Kempele, Finland).

During the laboratory test, subjects were connected to a metabolic cart (ParvoMedics TrueMax 2400, Sandy, UT, USA), used as the criterion measure of energy expenditure in the laboratory. The system was calibrated before each test and sampled gas exchange every 30 s. Data collection forms were annotated with the time the devices were initialized and the test was started, and a description of each stage in terms of time started and completed with any explanatory notes (e.g., “nose clip came off”) to establish the actual activity performed (“ground truth”), used as the criterion for activity type. Subjects walked or ran on a treadmill over a select range of speeds and grades for 3 min each (see Table 1) while connected to the calorimeter. The initial work stage was chosen at random. However, the four walking and two running stages were always preceded and followed by brief periods of sitting/standing. Any stage in which the subject’s heart rate exceeded 85% of the age-predicted maximum was terminated, and the work rate was set to the next lowest stage for six possible work stages.

Table 1.

The laboratory protocol for determining accuracy of the multi-sensor board

Stage	1	2	3	4	5	6	7	8	9
MPH	Sit	Stand	1.8	2.5	3.0	3.5	4.5	4.5	Sit
Grade (%)	N/A	N/A	0	5	0	7.5	0	2.5	N/A
Duration (min)	3	2	3	3	3	3	3	3	5
			Walking				Running

Subjects always started and ended with a sitting stage. The initial work stage of the four walking and two running stages was always executed in a random order, and a given stage was terminated if the subject exceeded 85% of the age-predicted maximum heart rate

In the field, subjects were connected to the Cosmed portable metabolic system (K4b², Cosmed Srl, Pavona di Albano, Rome, Italy), used as the criterion measure of energy expenditure for field experiments. The MSB, Actical, and heart rate monitor were worn in the same manner as the laboratory test. The field test lasted 10–15 min and consisted of a scripted set of activities. Direct visual observation was used as the gold standard for activity classification. Two experimenters conducted each data collection; one would verbally instruct the subject using the script while the other recorded ground truth activities, times, and any explanatory notes on a tablet PC. The script included the following activities executed in order: (1) start data collection on the fifth floor of the building while sitting down for 3 min; (2) stand, then walk to the elevator and descend to the main floor; (3) walk across an atrium and then outside; (4) walk down a flight of stairs and then walk around an inclined area outdoors; (5) walk back up stairs; (6) pick up a heavy object (~20 lb) and walk around with it for a short distance; (7) place the object down and sweep for approximately 30 s; (8) finish sweeping and then walk back inside to the elevator; and (9) ascend back to the fifth floor and walk back to the starting position and sit for 3 min.

Sensor board and data processing

The MSB platform consists of a small sensing unit (2 3/8″ × 3 1/8″ × 7/8″), Bluetooth sensor node, and USB rechargeable battery board, weighing about 25 g total. The unit includes a suite of multiple sensing (3-axis accelerometry, barometric pressure, humidity, temperature, light, audio, and GPS), data storage, communication, and local computation abilities. The main components of the system include: (1) an activity inference engine which classifies activities into different types (sitting, standing, walking, running, cycling, and stepping); (2) a step counter algorithm which uses signal analysis to detect steps and infer speed of movement; and (3) prediction formulas (ACSM 2006) for converting the activity types listed above, body weight, and horizontal and vertical velocity components into an estimate of oxygen consumption. Energy expenditure is then computed using the estimate of oxygen consumption and the caloric equivalent per liter of oxygen consumed (i.e., 1 L of O₂ liberates 5 kcal of energy) (ACSM 2006). This approach is similar to others (Zhang et al. 2004) where metabolic equivalent values (METs) (Ainsworth et al. 2000) were used to estimate energy expenditure for detected movements.

The technical details of the MSB and approach for classifying activity type (Lester et al. 2005, 2006) and estimating energy expenditure (Lester et al. 2009) have been published. Briefly, magnitude of acceleration is measured to avoid orientation-dependent effects and used to compute six features consisting of fast Fourier transform (FFT) bands, magnitude variance and standard deviation, and step speeds from a step counter four times per second; these features are used as inputs into a naïve Bayes classifier that infers activity as one of the major types listed above. Feature computation and classification is done inside the MSB LIRA (library for the inference of real time activities). The current study validates models that underwent machine learning using leave-one-out training as described in our previous studies (Lester et al. 2005, 2006, 2009).

Statistical analysis

Descriptive information was computed as the mean and standard deviation (mean ± SD) or percentage where appropriate. The criterion measure of energy expenditure was computed using the average oxygen consumption and respiratory exchange ratio (RER) values determined from the respective calorimeters over the entire test period, including all transitions between activities, as described by others (Zhang et al. 2004). The average RER was converted to a thermal (caloric) equivalent using published data (McArdle et al. 1996). We calculated accuracy measures for activity inference using Eq. 1 and energy expenditure (denoted as “C”) using Eqs. 2 and 3. Finally, we calculated sensitivity (number of true positives divided by number of true positives + number of false negatives) and specificity (number of true negatives divided by number of true negatives + number of false positives).

$$\text{Inference}=\frac{N_{\text{correctly}\text{identified}\text{activities}}}{N_{\text{all}\text{activities}}}\times 100\%$$ (1)

$$\text{Accuracy}=\frac{C_{\text{device}}}{C_{\text{calorimetry}}}\times 100\%$$ (2)

$$\text{Absolute}\text{accuracy}=100\%-\left|\frac{C_{\text{device}}-C_{\text{calorimetry}}}{C_{\text{calorimetry}}}\right|\times 100\%$$ (3)

Accuracy measures are presented as mean percent ± SD with range. Differences in energy expenditure accuracy for the MSB, Actical, and criterion were determined using a mixed linear model. An unpaired t test was used to examine differences in energy expenditure accuracy for the MSB by sex and an one-way ANOVA with means comparison using Tukey’s studentized range test to examine differences in energy expenditure accuracy by BMI category. Pearson’s correlation coefficient was calculated to examine the strength of the relation between MSB energy expenditure and the criterion for each testing condition and MSB energy expenditure between repeat tests for each testing condition as a measure of test–retest reliability. Finally, agreement between the MSB and criterion was assessed using the method of Bland and Altman (1986). Statistical analyses were performed using SAS version 9.2 (Cary, NC, USA), with significance defined a priori as P < 0.05.

Results

The study enrolled 57 subjects with 20–64 years of age. Data were unavailable from four subjects because (1) the MSB malfunctioned on the first lab or field test in three subjects, and each failed to return for subsequent testing and (2) in one subject, data were removed because the nose clip came off during on the treadmill, and we were unable to form an adequate seal between the subject’s face and the K4b² mask during the field tests. This left a final sample of 53 subjects. The total number of individual tests was 159 (53 subjects × 3 tests/subject). Our results are based on 148 total tests (93% of all possible tests), including 53 initial and 22 repeat laboratory tests, and 51 initial and 22 repeat field tests. The discrepancy is due to five pre-data processing errors (e.g., files failed to record on the MSB or were corrupted), from discovering two post-data processing errors (e.g., nose clip came off during one additional treadmill test while the K4b² unit was out of calibration during one field test), and the remainder because some subjects failed to return for all test visits. Descriptive data for the 53 subjects include age 40.0 ± 13.4 years, 55% female, 86% White, not Hispanic, BMI 27.2 ± 4.3 kg/m², and 20 normal weight, 19 overweight, and 14 obese subjects.

Accuracy measures for activity inference and energy expenditure are provided in Table 2. Overall, accuracy of inferred activities compared with the ground truth was 97% under laboratory and 84% under field conditions. In the laboratory, inference accuracy was lowest for running at 91% (91% sensitivity and 100% specific) and highest for walking at 99% (99% sensitive and 96% specific), with about 97% accuracy (97% sensitive and 100% specific) for sitting and standing, and less than 1% of activities that were unclassified (returned as “null”). In the field, sitting and walking were correctly classified 97% of the time with identical sensitivity and specificity for both activities. However, the lower overall accuracy of activity inference in the field (84%) was due to 13% of activities being unclassified.

Table 2.

Accuracy of the multi-sensor board and Actical for laboratory and field tests

Device	Inference (%)	Accuracy (%)	Absolute accuracy (%)	Energy expenditure difference (kcal)
Laboratory
MSB	97.2 ± 2.9	92.1 ± 9.7	89.4 ± 6.6	−10.0 ± 12.4
	86.0–99.2	66.6–124.6	66.6–98.5	−45.6 to 28.4
Actical	N/A	60.6 ± 10.5	60.6 ± 10.5	−44.5 ± 16.1*,†
	N/A	36.6–86.7	36.6–86.7	−4.9 to −84.8
Field
MSB	83.8 ± 3.7	76.0 ± 10.1	76.0 ± 10.1	−12.8 ± 6.5*
	70.1–91.2	58.1–96.5	58.1–96.5	−1.2 to −26.0
Actical	N/A	65.5 ± 9.0	65.5 ± 9.0	−18.4 ± 6.9*,†
	N/A	44.7–88.2	44.7–88.2	−4.6 to −33.0

Data for inference and accuracy are presented as the percent and standard deviation (SD) with the range indicated below, while the energy expenditure difference is presented as the mean and standard deviation difference between the calorimeter and device in kilocalories (kcal). Inference refers to measured activities from the multi-sensor board (MSB) compared to the recorded activity or “ground truth” (total number of correctly identified activities/total number of activities performed × 100). Accuracy of the energy expenditure estimates are expressed as the accuracy (total kilocalories computed from device/total kilocalories from the calorimeter × 100) and absolute accuracy {100 – absolute percentage error, where absolute percentage error = [(total kilocalories computed from device – total kilocalories from the calorimeter)/total kilocalories from the calorimeter] × 100}

^*Significant difference versus calorimeter at P < 0.001

^†Significant difference versus other device at P < 0.001

The MSB energy expenditure was not different (P > 0.05) from the criterion measure in the laboratory. However, Actical energy expenditure was significantly lower than both the MSB and criterion in the laboratory test (both P < 0.001). In the field, MSB energy expenditure was significantly lower than the criterion, and the Actical was significantly lower than both the MSB and criterion (all P < 0.001). There were no differences in MSB energy expenditure accuracy by sex or BMI category for either of test conditions (both P > 0.05).

Figure 1 shows the correlation (both P < 0.0001) between computed kilocalories from the MSB and calorimeter in the laboratory (Fig. 1a) and field test (Fig. 1b). MSB energy expenditure estimates between tests 1 and 2 were significantly correlated with each other in both testing conditions (both P < 0.0001 and r = 0.97).

Fig. 1.

Relationship between average estimated energy expenditure (kilocalories) from the multi-sensor board and measured energy expenditure from the calorimeter in the laboratory (a) and field test (b). Values are provided as the total energy expenditure calculated during the entire test period

The bias plots shown in Fig. 2 for the laboratory (Fig. 2a) and field test (Fig. 2b) present the degree of agreement between the MSB and criterion measure. The mean difference between the MSB and criterion in the laboratory was −10.0 ± 12.4 kcal (dashed line), with upper bound 14.2 and lower bound −34.2 kcal (long dash dot lines). In the field, mean difference between the MSB and criterion was −12.8 ± 6.5 kcal (dashed line), with upper bound −0.2 and lower bound −25.4 kcal (long dash dot lines).

Fig. 2.

Bland–Altman bias plots for the laboratory (a) and field test (b). Kilocalorie average denotes average of kilocalories from the multi-sensor board, and calorimeter and kilocalorie difference is kilocalories from the multi-sensor board –kilocalories from the calorimeter. The mean difference in energy expenditure between the MSB and criterion is represented by the dashed line, with the 95% prediction intervals represented by long dash dot lines

Discussion

The MSB provides accurate estimates of physical activity type and energy expenditure under laboratory and field conditions. Activity-type inference was substantially better for laboratory than field activities, with 97 and 84% accuracy, respectively. Although these values indicate that our approach performed quite well under both conditions, the difference in accuracy between settings is not surprising because the field test consisted of a greater range of activity types and less constrained activities that are more difficult to measure.

There are few methods currently in use to estimate activity type. For example, the intelligent device for energy expenditure and activity (IDEEA) was shown to correctly identify and quantify 32 types of human movements with a high degree of accuracy in a sample of 76 subjects (Zhang et al. 2003). Similarly, the DynaPort MoveMonitor objectively evaluates gait and postures and was found to detect time spent walking and lying accurately but much less so for sitting, standing, and stair stepping (Dijkstra et al. 2010). Using data from an accelerometer in six subjects performing level walking, walking uphill, vacuuming, and working at a computer in a laboratory setting, Pober et al. (2006) used quadratic discriminant analysis and a Hidden-Markov model (HMM) to infer activity type. These methods correctly identified activity type for 71 and 81% of the time points, respectively. Troped et al. (2008) employed discriminant function analysis to identify the combination of accelerometer and GPS variables that optimally predicted activity mode in a convenience sample of ten adults who performed a prescribed set of activities in an outdoor (walking, jogging, bicycling, and inline skating) and sedentary setting (driving an automobile). The combination of accelerometer counts, steps, and GPS speed were able to correctly classify activity type in 91% of the observations. We are unaware of other devices that measure activity type, particularly for the kinds of activities we included in the field test; so, the MSB is certainly among a handful of devices that are innovative in this regard. Nonetheless, there are inherent limitations in all of these devices and methods, including ours, as will be discussed in a subsequent paragraph. In general, the body of research is limited by methodological studies with very small convenience samples, protocols employing a restricted array of activities and settings, and cumbersome devices that require individualized calibration.

Our device also performed quite well for estimating energy expenditure under both testing conditions, although similar to activity inference, the MSB performed better in the laboratory than in the less constrained field condition. In particular, accuracy in the field suffered because subjects exerted an increased amount of energy sweeping and walking while carrying a heavy load that the MSB was not able to detect. Instead, the MSB merely detected walking and estimated energy expenditure based on the speed of walking alone.

Accuracy in the laboratory was slightly better when calculated using the average as in Eq. 2 (92 vs. 89%), although this approach is less robust because a device with equally large positive and negative deviations from the standard will have an average difference that approaches the standard, even though each individual measurement may have a large magnitude of absolute error. This level of accuracy is comparable to that obtained by IDEEA under similar laboratory testing (Zhang et al. 2004). Furthermore, the MSB was superior to the Actical for estimating energy expenditure under both testing conditions. MSB energy expenditure in the laboratory was, on average, only 10 kcal lower than computed using the calorimeter, while the difference in the field was only about 13 kcal lower. In the laboratory, the Bland–Altman plot (Fig. 2a) demonstrates that the direction of error is to under rather than overestimate energy expenditure. In the field, the plot (Fig. 2b) demonstrates that the MSB never overestimated energy expenditure. This may have clinical implications because a device that overestimates energy expenditure could potentially lead the user to consume excess energy, believing they had expended more energy than they actually had.

Although the results presented in this paper are from offline analysis, the MSB runs in real time. Offline analyses were used because we needed to properly align MSB data to the recorded calorimetry data and ground truth annotations to ensure appropriate validation. This was done using a data-merging process to match the various data streams using their respective timestamps. We compared a random sample of data using both methods, and the differences between the offline/desktop and real-time versions were negligible. This is important because, unlike other devices, data from the MSB can be provided to the user in real time. The practical implication is that potentially valuable information on activity and energy expenditure can be provided to the user as a form of instant feedback that could, in turn, stimulate behavior change.

The low failure rate of the MSB, large sample size, variety of activities in the field, and comparison to a “gold-standard” are particular strengths of the present study. However, there are also several important limitations worth noting. First, accuracy for activity recognition may be artificially high in the laboratory because we input the actual grade performed due to the fact that we were unable to detect elevation changes on the treadmill. Likewise, accuracy for activity recognition may be artificially high in the field because a limited range of activity types were performed compared to a “real-world” situation. Second, we included a commercially available monitor (Actical) during testing for comparison to the MSB. However, device choice was merely out of convenience rather than scientific merit, so it is unknown how well our MSB compares to other devices, such as the more widely used Actigraph accelerometer. Finally, energy expenditure estimates were influenced by a number of factors, as described below.

Resting energy expenditure was estimated using a MET value (3.5 ml O₂ kg^−1.min⁻¹). Although we could potentially improve energy expenditure estimates using subject-specific measures of resting metabolic rate, the degree of improvement is probably small relative to the burden of precisely measuring this component. Similarly, we used the caloric equivalent per liter of oxygen consumed (5 kcal) to convert estimates of oxygen consumption to energy expenditure. This value is indicative of carbohydrate oxidation and can lead to imprecision in the estimate. However, we are not able to measure carbon dioxide production, and thus, the precise fuel mixed used, though as noted above, the degree of improvement is probably small. Likewise, use of prediction equations for energy expenditure likely led to error because they are not applicable to all speeds and types of activities (e.g., the sweeping motion in the field test). Similarly, accuracy of energy expenditure in the field was reduced by our inability to distinguish between level and graded walking (i.e., the “walk around inclined area outdoors” task). This was also the case for the ascending/descending step task; the steps were located on an outdoor promenade and consisted of five steps on a gentle grade that the MSB likely detected as walking instead of stair stepping. Although we recently developed a method for extracting grade information from the barometric pressure signal and GPS/GIS data sources (Lester et al. 2009), we were not able to fully implement this approach in the present study. In the future, we could potentially improve the accuracy by using a classifier method, as used for predicting activity type. Finally, we estimated the energy expenditure using the entire period of activity including transitions when subjects were not in steady state. Although this is a major source of error (see Lester et al. 2009 for discussion), transition is a natural part of physical activity, particularly in regularly active persons who frequently change from one activity to another. Future studies could extend the findings presented here by using the doubly labeled water technique as the gold standard to address the issue of steady state because the method provides a measure of total energy expenditure over an extended period.

Conclusion and future directions

The MSB provides acceptable measures of validity and reliability for estimating activity type and energy expenditure under laboratory and field conditions. However, the device did underestimate energy expenditure in the field. Our future work will be directed at improving activity recognition and energy expenditure estimates under field conditions, and those experiments will include a wider variety of activities. In some of our ongoing studies, we have already witnessed how the system can use GPS, GIS, and barometric pressure data to enhance the performance of the energy expenditure estimates by 5–10% during more natural free-living conditions. Finally, our intent is to do away with the MSB as a stand-alone device and instead integrate it directly into a mobile phone platform. Ideally, specific elements of our platform and the analytic approach used to quantify activity type and energy expenditure can be integrated into a mobile phone to create an “all-in-one” device that can be worn discreetly to measure prolonged periods of physical activity and energy expenditure. Ultimately, this device could have numerous practical and research applications in the future for a variety of end users, including use for weight control and improving upon national surveillance of physical activity.