Validation of Consumer-Based Hip and Wrist Activity Monitors in Older Adults With Varied Ambulatory Abilities

Abstract

Background:

The accuracy of step detection in consumer-based wearable activity monitors in older adults with varied ambulatory abilities is not known.

Methods:

We assessed the validity of two hip-worn (Fitbit One and Omron HJ-112) and two wrist-worn (Fitbit Flex and Jawbone UP) activity monitors in 99 older adults of varying ambulatory abilities and also included the validity results from the ankle-worn StepWatch as a comparison device. Nonimpaired, impaired (Short Physical Performance Battery Score < 9), cane-using, or walker-using older adults (62 and older) ambulated at a self-selected pace for 100 m wearing all activity monitors simultaneously. The criterion measure was directly observed steps. Intraclass correlation coefficients (ICC), mean percent error and mean absolute percent error, equivalency, and Bland–Altman plots were used to assess accuracy.

Results:

Nonimpaired adults steps were underestimated by 4.4% for StepWatch (ICC = 0.87), 2.6% for Fitbit One (ICC = 0.80), 4.5% for Omron HJ-112 (ICC = 0.72), 26.9% for Fitbit Flex (ICC = 0.15), and 2.9% for Jawbone UP (ICC = 0.55). Impaired adults steps were underestimated by 3.5% for StepWatch (ICC = 0.91), 1.7% for Fitbit One (ICC = 0.96), 3.2% for Omron HJ-112 (ICC = 0.89), 16.3% for Fitbit Flex (ICC = 0.25), and 8.4% for Jawbone UP (ICC = 0.50). Cane-user and walker-user steps were underestimated by StepWatch by 1.8% (ICC = 0.98) and 1.3% (ICC = 0.99), respectively, where all other monitors underestimated steps by >11.5% (ICCs < 0.05).

Conclusions:

StepWatch, Omron HJ-112, Fitbit One, and Jawbone UP appeared accurate at measuring steps in older adults with nonimpaired and impaired ambulation during a self-paced walking test. StepWatch also appeared accurate at measuring steps in cane-users.

Regular engagement in physical activity promotes health and reduces risk factors for many chronic diseases in older adults (1,2). However, two-thirds of older adults in the United States do not meet the physical activity guidelines of 150 minutes of moderate intensity physical activity a week and an additional 20% do not engage in any daily leisure-time physical activity (3). Two-thirds of older adults also cite walking as their most preferred method of physical activity (4,5); therefore, tools and strategies to increase walking are a critical target for this population. Wearable activity monitors may be an effective tool in aiding older adults to manage their walking activities and to encourage them to walk more.

The consumer wearable activity monitor market is growing at an incredible rate with an expected 20 million units being sold in 2015.(6) Many of the activity monitors responsible for this growth have been validated in the general population (ie, children, healthy adults, obese individuals) (7–10). Use of activity monitors by older adults is also increasing (11); however, there are very few validation studies for this population. One study assessed the accuracy of a smartphone device to track steps in middle- and older-aged adults and found strong correlations with a research-grade accelerometer (ActiGraph, Fort Walton, FL) (12). Even less is known about activity monitors in measuring step activity in older adults with altered ambulation abilities. Older adults often have a slower or altered gait compared to younger individuals and may use assistive devices (eg, cane, walker) that are likely to have an impact on monitor accuracy. For this population, monitor location (eg, wrist, hip, ankle) and the sensitivity of monitor algorithm technology to detect true stepping must be considered. The limited testing of activity monitors in older adults is only beginning to identify some of the difficulties encountered when assessing step activity in individuals who may have altered walking patterns. For example, Laurtizen and coworkers (13) reported low accuracy and precision in two consumer-based monitors when measuring steps in older adults with an altered gait pattern or very slow gait.

The purpose of this study was to test the accuracy of four consumer-based activity monitors worn at different locations on the body to measure steps during ambulatory activity. We added the StepWatch activity monitor to the study to serve as a field-based reference to aid in interpretation of the other devices. Additionally, as a consumer-based ankle-mounted model was not available at the time of the study, these results may show how valid and reliable a future consumer model could be on the ankle. The activity monitors were selected based on varied locations: StepWatch (ankle), Fitbit One (hip), Omron HJ-112 (hip), Fitbit Flex (wrist), and Jawbone UP (wrist). Accuracy was assessed in four groups of older adults with varying levels of ambulatory ability: nonimpaired walkers, impaired walkers, cane-users, and walker-users.

Methods

Participants

Older adults from two suburban community centers and a senior mixed-living community in a large metropolitan area (through researcher presentations or flyers) were invited to participate in the study. Participants were eligible if they (a) were 62 years or older; (b) met the health conditions on a modified Physical Activity Readiness Questionnaire-Revised (14) such as stable blood pressure for 3 months, no reports of chest pain with activity or joint problems limiting physical activity; (c) able to walk 100 m without stopping with or without an assistive device; (d) able to speak and understand English; and (e) willing to wear all activity monitors during the study. Approval was obtained prior to the study from the University Institutional Review Board.

Participants were informed of study procedures and provided consent prior to participation. If participants stated they were currently using a cane or walker as an assistive device for most walking activities, they were classified into that specific testing group. If they did not indicate an ambulatory assistive device, we administered the Short Physical Performance Battery test to classify them as a “nonimpaired walker” (Short Physical Performance Battery ≥ 9) or “impaired walker” (Short Physical Performance Battery < 9). The Short Physical Performance Battery scores highest in reliability, validity, and responsiveness, and has been extensively investigated in different populations of older adults ranging from highly active to frail (15).

Instruments

We tested the accuracy of measured steps of four widely available consumer-based activity monitors. The Fitbit One and Omron HJ-112 were mounted on the nondominant hip. The Fitbit Flex and Jawbone UP were mounted on the nondominant wrist. We also included accuracy of measured steps for the StepWatch activity monitor mounted to the nondominant ankle to serve as a previously validated monitor reference. Specifications for each monitor are found in Table 1.

Table 1.

Description of Activity Monitors

Instrument	Manufacturer	Placement	Measures and Programming
StepWatch	OrthoCare Innovations, Mountlake Terrace, WA	Ankle	Dual-axial accelerometer that measures step counts and step rates per unit time; programmed and downloaded via software interface (Mac/PC); records for up to 2 months
Fitbit One	Fitbit, San Francisco, CA	Hip	Triaxial accelerometry monitor assesses physical activity patterns; syncs wired (via USB) to computer or wireless (Bluetooth) to iOS, Android monitor; 7-day charge from USB charging cable
Omron (HJ-112)	Omron Healthcare, Bannockburn, IL	Hip	Dual-axial accelerometer that measures steps and estimated activity level based on stride; on-screen display; battery- powered with 7-day memory
Fitbit Flex	Fitbit, San Francisco, CA	Wrist	Triaxial accelerometry monitor assesses physical activity patterns; syncs wirelessly to tablets, computers, iOS, Android, and Windows smartphones using Bluetooth 4.0 wireless technology; 5-day charge from USB charging cable
Jawbone UP	JAWBONE, San Francisco, CA	Wrist	Triaxial accelerometry monitor assesses sleep and physical activity patterns; syncs data with iPhone or Android using attached cable connection; 10-day charge

Procedures

Data were collected between October 2013 and April 2014. Participants were instructed to walk at a self-selected pace along an unobstructed 100-m predetermined, flat marked route at their respective community center location. The 100-m distance was selected to accommodate all ambulation categories of the study population as many older adults with impaired ambulation or who use assistive devices may not have the ability to ambulate continuously for longer distances (16). Participants were encouraged to walk at their normal walking pace and use their assistive device as was comfortable.

Measures

Demographic and physical characteristics reported by the participant included age, gender, height and weight, level of ambulatory ability, and dominant hand use. Participants were fitted with the five activity monitors on the nondominant side of the body according to manufacturer guidelines in a fixed placement order. The criterion measurement of steps was obtained by direct observation through continuous videography. Steps were counted by two independent observers with very high interrater reliability (intraclass correlation [ICC] = 0.99). Steps recorded by the StepWatch were downloaded to a computer and marked based upon recorded start and stop times for the activity. Steps recorded by the Fitbit One, Omron (HJ-112), Fitbit Flex, and Jawbone UP were captured directly from the monitor display or synced smartphone immediately prior and following the walking activity while the participant was motionless.

Statistical Analysis

Data were cleaned, and Bland–Altman plots were produced using SAS 9.4. All other statistical analyses were performed using SPSS version 21. Descriptive statistics were computed for each walking category. Accuracy relative to the criterion of direct observation was assessed first with the ICC. Specifically, we used ICCs (2,1) with absolute agreement, which are used to assess the agreements between two or more measurements. The ICCs (2,1) account for both agreement of performances between the two measures in the same individual (within-subject differences) and change in average performance of participants as a group between the two measures (ie, between-subject differences). ICC values >0.75 were interpreted as excellent, 0.40–0.75 were fair to good, and <0.40 were poor (17). We also assessed accuracy by calculating mean percent error ([activity monitor recorded steps − direct observation/direct observation] × 100). Mean percent error allows us to obtain the direction of the error of measurement by each device. Positive values indicate overestimates of steps and negative values indicate underestimates relative to the criterion. Additionally, we calculated mean absolute percent error ([|activity monitor recorded steps − direct observation|/direct observation] × 100) to obtain the magnitude of the error. Larger mean absolute percent error values indicate more error in step estimates whereas smaller values indicate less error relative to the criterion. Equivalency testing was used to examine whether each one of the activity monitors under assessment was statistically equivalent to the criterion. Equivalency testing is an alternative approach to testing for significant difference (in which the evidence of a difference does not necessarily indicate equivalency) (18). When conducting equivalency testing it is necessary to identify a clinically meaningful equivalence range also known as equivalence zone. Comparisons between the activity monitors and the equivalence zone can then be performed, and, if the full 90% confidence interval of a monitor lies within the equivalence zone, it can be concluded with 95% confidence that the test sensor is equivalent to the criterion. An equivalence zone of ±10%, while arbitrary, is consistent with other activity monitor validation studies (8,19). To assess systematic variation between activity monitor and observation, Bland–Altman plots (20) were derived for each monitor against the criterion. In addition to the traditional display on the Bland–Altman plot, we included a regression in which the difference score between methods was regressed upon the average of the two scores. This regression line provides information on whether the activity monitor value becomes more or less accurate at varying levels of the criterion; thus, a slope close to zero (flat regression) line indicates that the step estimate of the activity monitor varies in the same manner as the criterion value. If the slope is not close to zero, the activity monitor is positively or negatively biased when compared to the criterion. In all statistical tests, p < .05 was considered significant.

Results

A total of 99 older adults completed the study protocol. Participants were 71% female, 78.9±8.6 years of age (range: 62–101 years), and body mass index = 28.0±6.5kg/m². Nonimpaired (N = 25, 72% female) participants were 74.3±6.3 years. Impaired ambulators’ (N = 25, 72% female) age was 79.2±7.4 years, and cane-users’ (N = 25, 64% female) age was 77.6±9.8 years. Walker-users (N = 24, 75% female) were significantly older than their nonimpaired peers with an age of 84.8±8.7 years (p < .001). Gender distribution and body mass index were similar across ambulatory ability levels.

Table 2 displays ICCs, mean percent error, and mean absolute percent error with 95% confidence intervals for the five activity monitors by ambulatory ability levels. StepWatch had high accuracy in all walking categories (ICCs for all walking categories >0.87). Fitbit One, Omron, and Jawbone UP were more accurate when measuring nonimpaired and impaired walking groups; however, accuracy was poor for these three monitors when measuring steps in cane- and walker-user groups. Fitbit Flex had very poor ICC values in all groups. All of the tested activity monitors underestimated steps to a different extent across the four walking categories. Overall, StepWatch had the lowest underestimation across all walking categories. Fitbit One, Omron, and Jawbone UP had <10% underestimation in the nonimpaired and impaired ambulatory categories; however, underestimation increased substantially in the cane- and walker-user groups. Fitbit Flex underestimated steps by markedly high levels. Mean percent error and mean absolute percent error results were similar across devices, suggesting the direction of error (underestimation) was consistent within each device and across all devices.

Table 2.

ICC (95% CI), Mean Percent Error (95% CI), and Absolute Percent Error (95% CI) by Ambulatory Category

	ICC	MPE	MAPE
Nonimpaired (N = 25)
StepWatch	0.87 (0.68, 0.95)	−4.42 (−0.41, −8.43)	5.25 (1.45, 9.05)
Fitbit One	0.80 (0.60, 0.91)	−2.59 (0.04, −5.23)	2.69 (0.07, 5.31)
Omron	0.72 (0.42, 0.87)	−4.48 (−1.20, −7.76)	4.47 (1.19, 7.76)
Fitbit Flex	0.15 (−0.12, 0.45)	−26.94 (−14.26, −39.61)	27.65 (15.27, 40.03)
Jawbone UP	0.55 (0.21, 0.78)	−2.86 (4.32, −10.04)	10.87 (5.20, 16.53)
Impaired (N = 25)
StepWatch	0.91 (0.74, 0.97)	−3.45 (0.06, −6.96)	3.76 (0.34, 7.18)
Fitbit One	0.96 (0.88, 0.99)	−1.71 (−0.69, −2.73)	1.81 (0.82, 2.79)
Omron	0.89 (0.63, 0.96)	−3.15 (−1.37, −4.93)	3.21 (1.45, 4.96)
Fitbit Flex	0.25 (−0.10, 0.69)	−16.31 (−10.90, −21.71)	16.30 (10.90, 21,70)
Jawbone UP	0.50 (0.10, 0.75)	−8.43 (−2.93, −13.92)	9.50 (4.34, 14.66)
Cane-user (N = 25)
StepWatch	0.98 (0.95, 0.99)	−1.78 (0.06, −3.61)	2.29 (0.57, 4.01)
Fitbit One	−0.16 (−0.19, 0.50)	−11.48 (−1.84, −21.11)	11.47 (1.84, 21.11)
Omron	−0.61 (−0.37, 0.32)	−17.64 (−3.23, −32.06)	18.56 (4.37, 32.75)
Fitbit Flex	0.05 (−0.27, 0.41)	−22.88 (−4.57, −41.19)	35.89 (21.77, 50.01)
Jawbone UP	−0.04 (−0.31, 0.29)	−34.10 (−13.44, −54.75)	46.03 (29.77, 62.29)
Walker-user (N = 24)
StepWatch	0.99 (0.98, 0.99)	−1.33 (0.57, −3.22)	1.75 (−0.05, 3.56)
Fitbit One	−0.38 (−0.58, 0.13)	−23.25 (−8.99, −37.51)	23.25 (8.99, 37.51)
Omron	−0.16 (−0.37, 0.19)	−33.72 (−17.84, −49.61)	33.72 (17.83, 49.61)
Fitbit Flex	−0.03 (−0.13, 0.16)	−57.40 (−44.36, −70.43)	57.39 (44.35, 70.43)
Jawbone UP	−0.01 (−0.05, 0.07)	−94.27 (−83.71, −104.84)	95.36 (87.04, 103.67)

Notes: CI = confidence interval; ICC = intraclass correlation coefficients; MAPE = mean absolute percent error; MPE = mean percent error.

Figure 1 displays results from equivalency testing. None of the activity monitors were within the equivalence zone across all walking groups. Fitbit One was the only device that was equivalent to the criterion for the nonimpaired walking groups with marginal equivalence for StepWatch, Omron, and Jawbone UP. Three activity monitors (StepWatch, Fitbit One, and Omron) were equivalent to the criterion for the impaired walking group. StepWatch was the only activity monitor equivalent to the criterion for the cane-user group. None of the activity monitors were equivalent to the criterion for walker-user group, although StepWatch bordered equivalence.

Figure 1.

Equivalency plots by ambulatory category.

Bland–Altman plots are shown in Figures 2–5 for nonimpaired, impaired, cane-users, and walker-users, respectively. Limits of agreement for StepWatch were narrow across all walking groups without important variation. Wide limits of agreement with larger scatter were observed in the cane-user (see Figure 4) and walker-user (see Figure 5) groups for Fitbit One, Omron, Fitbit Flex, and Jawbone UP. Significant variation was observed in Omron, Fitbit Flex, and Jawbone UP for the cane-user group (see Figure 4).

Figure 2.

Bland–Altman plots for nonimpaired ambulators (N = 25).

Figure 5.

Bland–Altman plots for walker-users (N = 24).

Figure 4.

Bland–Altman plots for cane-users (N = 25).

Figure 3.

Bland–Altman plots for impaired ambulators (N = 25).

Discussion

Activity monitors are increasingly popular among older adults (11). Monitors that provide accurate information are needed for this population. Our study is one of the first to compare various consumer-based monitors in the older adult population across varying ambulatory abilities. These results show that monitor location may be an important consideration for older adults with various gait patterns and/or who use assistive devices. The best results were observed for the ankle-mounted device, with acceptable values also present at the hip, and in some walking groups acceptable on the wrist (eg, Jawbone UP with nonimpaired and impaired walkers). However, algorithm differences clearly also contribute to accuracy even when considering different devices at the same location. This was evident in our study based upon the measurement differences observed between the two wrist-worn monitors, Fitbit Flex and Jawbone UP.

For older adults with nonimpaired and impaired ambulation, placement of activity monitors at all three locations—wrist, waist, or ankle—seemed to yield relatively accurate results at normal walking speeds. Older adults may prefer use of monitors easily managed on the wrist or waist. Older adults using a cane may require an ankle-mounted monitor or a waist worn monitor sensitive enough to capture slight changes in gait or balance. An ankle-mounted activity monitor may be the only option for accurate step counting in older adults who use a walker to assist with ambulation. Further investigation is needed with consumer-based ankle monitors to test for accuracy, as the StepWatch is not typically used outside of clinical contexts. It has a significantly higher cost and requires more extensive software/computer equipment than most consumer monitors on the market. At the time of the study, however, other ankle-worn monitors were not feasible. It may be more useful to apply a monitor to roller-walkers that measures distance to estimate steps in that population. Before validation in free-living conditions, studies should address the acceptability of use of these monitors by the older adult population as they may have different learning needs regarding digital monitor interaction (21,22). Devices should be easily manageable for a population that may have reduced vision and muscle dexterity, for example.

Consumer-based activity monitors may be a valuable tool to support older adults to track and manage their physical activity levels, along with other health outcomes such as sleep and dietary intake. Additionally, they may provide valuable health data for older adults to share with their clinicians. The ability to accurately assess activity patterns across days and weeks—with ease of use and at a relatively low cost—may support physical activity promotion strategies that are tailored to the needs of older adults. These monitors may be easily adopted in clinical settings as an added assessment tool.

Strengths

There are several strengths of this study. First, the very high interrater reliability (ICC = 0.99) for the criterion measure assures very little measurement error. Second, to our knowledge, this is the first study to test the accuracy of several monitors in different body locations across a wide range of older adults with varied ambulatory abilities. The use of multiple monitors in three separate locations highlights the potential variations in algorithm technology that should be addressed for use in the older adult population. These findings are in line with earlier results from Tudor-Locke and coworkers (23) who identified the need for appropriate algorithm development for individual monitors in specific ambulatory groups.

Limitations

There are a few limitations of this study. First, we tested activity monitor measurement in only one prescribed walking activity. This limits generalizability to other free-living conditions in this population. We chose this single activity considering the safety for this population. Second, it is possible that monitor placement—on the dominant or nondominant side—may cause an error in estimation of steps due to changes in arm swing or gait related to their disability. It is also unknown if there is a hand dominance effect for cane-users. We could not test the difference in this study as our cane-user sample population was relatively small, and among those only four participants used their cane with the nondominant hand. However, we did not observe any differences in accuracy among these four individuals. Lastly, the StepWatch is not a typical consumer-based activity monitor; therefore, results may not translate to other ankle-worn consumer monitors on the market.

Conclusion

The use of consumer-based, objective activity monitors may be a realistic option to assist older adults to manage or improve their physical activity levels. However, this study shows that one device does not “fit all.” Consideration of the individual’s ambulatory pattern and/or limitation is needed to ensure accurate step detection. Placement of a device at the wrist, hip, or ankle should be investigated based on the older adult’s walking pattern and use of assistive equipment. Additionally, monitor sensitivity to detect true steps in older adults with varied gait patterns such as shuffling should be considered when selecting a device.

Funding

This study was supported in part by Aliphcom, and data were collected as a subset of a larger study. Aliphcom is the parent company of Jawbone and did not have any input in the study design, analysis, or interpretation of data. Manuscript development was supported by the National Institutes of Nursing Research Grant T32 Interventions for Preventing and Managing Chronic Illness (5T32NR007091).