The Ability of Monitor-Independent Movement Summary Units, Euclidean Norm Minus One, and Mean Amplitude Deviation to Harmonize Accelerometry Data Across Research-Grade and Consumer Wearable Devices During Simulated Free-Living Physical Activity in Children

Abstract

Background:

Epoch-level accelerometry summary metrics have the potential to be device-agnostic, meaning that similar estimates should be obtained regardless of the device used. The objective of this study was to identify which metric (ENMO, MAD, MIMS) best harmonizes data across devices in an applied setting measuring children’s activity.

Methods:

Children (n=239; 9.3 ± 2.1 years, 47% female, 30% Black) wore ActiGraph GT9X (+/−8g, 50Hz) accelerometers and were randomized to wear two of three consumer wearables including Apple Watch Series 7 (+/−16g, 50Hz), Garmin Vivoactive 4S (+/−8g, 25Hz), and Fitbit Sense (+/−4g, 50Hz) on their non-dominant wrist, while participating in 60 minutes of simulated free-living activities (i.e., walking, running, soccer). The standard deviation (SD) across z-scores for ENMO, MAD and MIMS of each device was calculated at the second-level to quantify variability across devices. Lin’s Concordance Correlation Coefficient (LCCC) was calculated for each combination of devices by metric to determine agreement. Multi-level intra-class correlation coefficients (ICC) were additionally used to quantify harmonization.

Results:

The SD of z-score across devices was the lowest, indicating better harmonization, for MAD (0.13±0.23), followed by ENMO (0.24±0.50), and then MIMS (0.26±0.39). LCCC was strongest for MAD, with coefficients ranging from 0.89 to 0.96. LCCC for MIMS ranged from 0.70 to 0.83, and was lowest for ENMO, ranging from 0.62 to 0.76. Overall ICC was highest for MAD (0.88), followed by MIMS (0.73), and ENMO (0.62).

Conclusions:

MAD appears to perform best at harmonizing across all the sampled research-grade and consumer devices in children.

1. Introduction

Accelerometry is a valuable method for monitoring physical activity, offering the capability to unobtrusively measure free-living physical activity over extended periods of time.¹ A challenge within the field of device-based physical activity measurement is the lack of standardization in how accelerometry data is processed and summarized.² Historically, early versions of accelerometers stored data in proprietary counts due to data storage and battery constraints.³ Technological advancements have improved the ability of accelerometers to capture and record raw acceleration at higher frequencies. As a result, open-source summary metrics have been proposed to process raw accelerometry data.^4,5 While this is beneficial for promoting open-source data processing methods, as opposed to relying on proprietary count methods, this advancement also has implications for measurement uniformity. For instance, since accelerometry processing decisions are not standardized across the field, this provides researchers with more autonomy in their decision-making for how they process raw accelerometry.⁶ A growing body of evidence indicates that accelerometry processing decisions that researchers make can have a substantial impact on their physical activity outcomes,^7,8 which has been further highlighted by experts in the field calling for more standardization in accelerometry processing methods.⁹

This uniformity challenge is compounded by the abundance of accelerometer-based devices that are available to researchers.⁹ Although ActiGraph continues to be the most highly used accelerometer brand for research,² there are a variety of other research-grade accelerometer brands (i.e., ActivInsights, Axivity), in addition to consumer wearables (i.e., devices targeted at consumers) that have become of interest for research applications. Although consumer wearables were previously limited by offering exclusively proprietary physical activity estimates, many now provide access to raw accelerometry data by using application programming interfaces (APIs).¹⁰ While, in theory, raw data should be comparable across different devices, there is variability in the hardware and software of accelerometer-based devices, given that these devices come from different device manufacturers. Three key differences in devices from different manufacturers include (1) accelerometer dynamic range, (2) resolution of the sensor, and (3) sampling rate.^11,12 Accelerometer dynamic range is the range of accelerations a device can capture (e.g., ±2 g, ±4 g), representing the accelerometer’s sensitivity to motion across intensity. Devices with more narrow dynamic ranges could saturate the acceleration signal, and may not accurately capture more vigorous intensity activity.^11,13 The resolution of the sensor is the smallest change in acceleration that the accelerometer can capture (e.g., 8-bit, 12-bit). When an accelerometer has higher resolution (e.g., higher bit depths), it is more sensitive to low-intensity movements. The sampling frequency is the rate at which the accelerometer captures acceleration (e.g., 25 Hz, 50 Hz). An accelerometer sampling at a higher sampling rate can more precisely capture when there are rapid changes in acceleration.

These differences among devices from different manufacturers could limit the comparability of data produced across different devices, even when raw data are used. A number of different epoch-level summary metrics have been proposed to enable harmonization across devices. Metrics for summarizing raw acceleration, include Euclidean Norm Minus One (ENMO)⁵, Mean Amplitude Deviation (MAD),⁴ and, more recently, Monitor-Independent Movement Summary units (MIMS).¹¹ These metrics enable the same processing to be applied to the raw data. Therefore, these metrics have the potential to be device-agnostic,¹⁴ meaning that similar estimates should be obtained for a given dynamic range, resolution, and sampling frequency. MIMS in particular has been pitched as a metric to enable harmonization across devices with different sampling frequencies and dynamic ranges.¹¹ However, the relative strengths and limitations of the various methods have not been fully evaluated. Previous research is limited by only comparing between two devices,^15,16 comparing across generations from the same device manufacturer (ActiGraph),^15,16 or evaluating only a few indicators (ENMO and MAD, in particular).^15–17 There is only one study to our knowledge that has compared across both research-grade and consumer devices, however that study is limited by only comparing MIMS and ENMO, and using data collected from a shaker table.¹¹ Therefore, the purpose of this study is to evaluate which metric harmonizes accelerometry data most optimally across devices with different hardware and software during a simulated free-living physical activity protocol in children. As a secondary goal, this study also aims to examine whether harmonization across devices varies across intensity.

2. Methods

2.1. Setting and Participants

This study was conducted in one southeastern metropolitan United States city from May of 2022 to April of 2024. All study procedures were approved by the Institutional Review Board of the lead author’s institution. Participants (n=239) were recruited through afterschool programs, summer day camps, newsletters, social media, and participant referrals. Children were eligible for inclusion if they were between 5 and 12 years of age, could be physically active without an assistive device, and did not have a neuro-muscular disease.

2.2. Protocol

Data for the current study were collected as part of the Wearables for Kids (W4K) study, which has been described in full detail elsewhere.¹⁰ The W4K study aims to develop and validate algorithms that predict child physical activity energy expenditure using the underlying accelerometry and heart rate from consumer wearables. The study protocol was designed to engage children in a series of simulated free-living activities ranging from sedentary to vigorous, based on the Youth Compendium of Physical Activities.¹⁸ Children participated in the study protocol while attending their summer day camp, afterschool program, or on-site at the university. Following parent consent, child assent, and anthropometric measurements, trained research assistants initialized and placed all equipment on the child. A key feature of the W4K study is that children wore three devices simultaneously in a counterbalanced design (See 2.3 Research-Grade and Consumer Wearable Devices). This feature allowed for the direct comparison of harmonization performance by accelerometry summary metrics across three devices worn at the same time.

Supplementary Table 1 presents the full protocol. Children began the protocol with a rest session by laying down for 10 minutes and self-selecting a video to view or game to play on an iPad. After the 10-minute rest session, there were two blocks of activities that the child completed. The first block was consistent across all children. This block had 4 activities that were each 5 minutes in length and consisted of activities that progressed in intensity from sedentary to vigorous. The second block was randomly assigned to the child from one of three possible blocks. This block also had four activities that were each 5 minutes in length and included activities ranging from sedentary to vigorous. However, the order of the activities was different across blocks (i.e., one block started with a vigorous intensity activity while another block started with a moderate intensity activity). This was done to simulate free-living activity, as children commonly alternate between intensities when engaging in physical activity.¹⁹ All protocol activities were selected because children commonly engage in them during their daily lives.

2.3. Measures

Demographics and Anthropometrics.

Parents completed an online survey to provide the child’s age, sex, and race. Child height and weight were measured prior to each protocol. Trained research assistants measured height to the nearest tenth of a centimeter with a portable stadiometer (Model S100, Ayrton Corp., Prior Lake, Minn.) and weight to the nearest 0.1 kg with a digital scale (Healthometer model 500KL, Health o meter, McCook, Ill.). Each measurement was taken twice, with the average used for analysis. If the two measurements were different by more than 1 cm (height), or 0.1 kg (weight), a third measurement was taken, and the closest two measurements were averaged. BMI z-score was calculated using CDC growth charts.

Research-Grade and Consumer Wearable Devices.

In alignment with the aims of the original study,¹⁰ this study used several popular consumer wearable devices, and one research-grade accelerometer. Prior to the protocol, trained research assistants fitted each participant with three devices on their non-dominant wrist: two of three consumer wearable devices and one research-grade device. Participant handedness was determined through one of two methods: (1) they were asked to draw on a sheet of paper to identify which hand they wrote with (i.e., the dominant hand) or (2) researchers gave the child a high-five and indicated the hand that they high-fived with (i.e., the dominant hand). Consumer wearable devices included: Apple Watch Series 7 (Apple; Apple Technology Company, Cupertino, CA), Garmin Vivoactive 4 (Garmin; Garmin Ltd., Olathe, KS), and Fitbit Sense (Fitbit; Fitbit Inc, San Francisco, CA). The research-grade accelerometer was the ActiGraph GT9X Link (v6.14.3, ActiGraph; ActiGraph LLC, Pensacola, FL). For the consumer wearables, the device order was counterbalanced. The ActiGraph was always placed most proximally on the arm, and the counterbalanced consumer wearables were placed most distally on the arm. The ActiGraph was placed most proximally due to the aims of the parent study, which compared the performance of the research-grade accelerometer to consumer wearables. Therefore, the research-grade accelerometer (ActiGraph) was standardized in placement relative to the consumer wearables to reduce noise.

The ActiGraph GT9X Link is a triaxial accelerometer with a dynamic range of +/− 8 g and sampling resolution of 12-bit. The dynamic ranges of the consumer wearables’ accelerometers are not publicly available, however, previous research indicates that Apple’s dynamic range is likely +/− 16 g, Fitbit dynamic range is likely +/− 4 g, and Garmin’s dynamic range is likely +/− 8 g.¹⁰ The sampling resolutions of the consumer wearables are not publicly available.

Underlying accelerometry data were collected from the Apple Watch, Garmin, and Fitbit via associated open-source applications: SensorLog (version 5.2), RawLogger (version 1.0.20211201a), and Slogger, respectively. SensorLog is an iOS sensor tracking application that can access and record data streams (i.e., raw accelerometer, gyroscope) on iOS devices at a sampling rate of up to 100 Hz. For the current study, data was logged and exported from SensorLog using the long-term acceleration feature, which can capture accelerometer data up to a sampling frequency of 50 Hz. RawLogger is a sensor tracking application available on Garmin Connect that can capture accelerometer data at a sampling rate of 25 Hz. Slogger is a sensor tracking application that our team developed to access and record underlying accelerometry and heart rate data from Fitbit devices. Slogger is available for download on the Google Play store and can collect accelerometry and gyroscope data up to 100 Hz. All sensor logging applications collected accelerometry data directly on the watch, independently of their associated phone(s). Accelerometry data was extracted from the ActiGraph GT9X Link through Actilife software from .gt3x files (v6.13.4, ActiGraph LLC, Pensacola, FL).

The consumer wearable devices used in this study were selected due to their widespread use and relatively large share of the wearables market.²⁰ Additionally, in accordance with the aims of the current study, these devices are also different brands, have different dynamic ranges, and can log acceleration at different sampling frequencies. For this study, ActiGraph, Apple, and Fitbit were set to log acceleration at 50 Hz, while Garmin was set to log acceleration at 25 Hz. All wearables (consumer, research-grade) were affixed to the wrist with manufacturer-provided bands. For the Apple watch, a third-party strap made of a flexible, stretchable material, was also used, allowing for a secure and adjustable fit.

During this study, the Apple Watch started on firmware version 8.5 and had 5 firmware updates (8.5.1, 8.6, 8.7, 9, and 9.0.2). The Garmin Vivoactive 4 started on firmware version 6.8 and had 5 firmware updates (6.84. 6.9, 7.01, 7.2, 7.3) and the Fitbit Sense started on firmware version 128.6.16 and had one firmware update (128.6.17). The ActiGraph GT9X Link was on firmware version 1.7.2 and did not update during the study. According to manufacturers, none of these updates impacted accelerometry data, as these updates were aiming to improve functionality and increase security on these devices.

For both the research-grade and consumer wearable devices, the orientation of the X,Y and Z axes is consistent. The X axis is mediolateral; parallel with the screen of the device, aligned with the top and bottom edges, in the left to right direction. The Y axis is vertical; parallel with the screen of the device, aligned with the left and right edges, in the top to bottom direction. Lastly, the Z axis is anteroposterior; perpendicular to the device’s screen, pointing up.²¹

In the current study, 20 different ActiGraph, 10 different Apple watches, 11 different Garmin watches, and 19 different Fitbit watches were used. Previous work from our research group has highlighted strong intra-device reliability, with ICCs of 0.99, 0.97, 0.88, and 0.88 across Apple, ActiGraph, Garmin, and Fitbit, respectively.²² Therefore, we do not expect that differences between unique devices of the same type of device would markedly impact harmonization.

2.4. Data Processing

Pre-Processing.

Following the protocol, accelerometry data were exported from the consumer wearables as a CSV file. Accelerometry data were not calibrated. The entire protocol was used, including all activity blocks.

Vector Magnitude.

VM was calculated as the square root of the sum of the squared acceleration in each axis at the one-second level using a custom Python script developed for this study. VM was used as a benchmark for comparison of the other epoch-level summary metrics, as VM is the closest to the raw acceleration signal and we would therefore expect this metric to be more variable across devices.²³

$$\mathit{\text{Vector}}\mathit{\text{Magnitude}}(VM)=r_i=\sqrt{x^2+y^2+z^2}$$

Mean Amplitude Deviation.

MAD was calculated as the average distance from the mean over a given window of time (5 seconds)^4,24 using a custom Python script developed for this study.

$$\mathit{\text{Mean}}\mathit{\text{Amplitude}}\mathit{\text{Deviation}}(\mathit{\text{MAD}})=\frac{1}{n}\times \sum _{i=1}^n|r_i-\overline{r}|$$

where;

$$r_i=\sqrt{x^2+y^2+z^2}$$

$$\overline{r}=\mathit{\text{mean}}\mathit{\text{vector}}\mathit{\text{magnitude}}\mathit{\text{within}}\mathit{\text{the}}\mathit{\text{time}}\mathit{\text{period}}\mathit{\text{of}}\mathit{\text{interest}}$$

Euclidean Norm Minus One.

Consistent with previous research, ENMO was calculated as the square root of the sum of the squared acceleration in each axis, subtracting one, with negative values rounded to zero.^5,25 ENMO was calculated across each accelerometer observation (i.e., at the Hz level) and aggregated to the one-second level by taking the mean. ENMO was calculated using a custom Python script developed for this study.

$$\mathit{\text{Euclidean}}\mathit{\text{Norm}}\mathit{\text{Minus}}\mathit{\text{One}}(\mathit{\text{ENMO}})=\text{max}(r_i-1,0)$$

where;

$$r_i=\sqrt{x^2+y^2+z^2}$$

Monitor-Independent Movement Summary Units.

MIMS units were generated using the “MIMSunit” package on R (GNU Project, Auckland, New Zealand) at the one-second level. This conversion from raw acceleration data to MIMS units occurs over a 4-step process including interpolation, extrapolation, bandpass filtering and aggregation of data for each axis.¹¹

2.5. Data Analyses

The first two harmonization analyses (Standard Deviation of Z-score, Lin’s Concordance Correlation Coefficient) and all sensitivity analyses were completed in Stata (version 17, StataCorp LLC, College Station, TX). The third harmonization analysis (multi-level intra-class correlation coefficient) was conducted in R Studio (version 4.5, GNU Project, Auckland, New Zealand). Descriptive means and standard deviations were calculated for sample characteristics.

Harmonization of Data.

The primary aim of this analysis was to evaluate which accelerometry metric harmonized most optimally across all devices, including both research-grade and consumer wearable devices. Harmonization of accelerometry data across devices was evaluated through three methods. For the first method, hereafter referred to as “Standard Deviation (SD) of z-score”, z-scores were calculated separately for each metric of each device (i.e., Apple MIMS, Garmin MAD) at the observation level (1-second). This generated a total of 16 z-distributions (4 metrics * 4 devices across all participants): Apple MIMS, Apple MAD, Apple ENMO, Apple VM, Garmin MIMS, Garmin MAD, Garmin ENMO, Garmin VM, Fitbit MIMS, Fitbit MAD, Fitbit ENMO, Fitbit VM, ActiGraph MIMS, ActiGraph MAD, ActiGraph ENMO, ActiGraph VM. Then, the SD across the three z-scores (i.e., one for each of the three devices worn simultaneously) was calculated at the observation level for each metric. For instance, for a child wearing ActiGraph, Apple, and Garmin, twelve z-scores would be calculated (based on the z-distributions across all participants for ActiGraph MIMS, ActiGraph MAD, ActiGraph ENMO, ActiGraph VM, Apple MIMS, ActiGraph MIMS, ActiGraph ENMO, ActiGraph VM, Garmin MIMS, Garmin MIMS, Garmin ENMO, Garmin VM). To calculate the SD of z-score for MIMS, the standard deviation across ActiGraph MIMS, Apple MIMS, and Garmin MIMS at the 1-second level would be calculated. Put simply, the SD of z-score method standardized each observation per metric of each device relative to the mean and standard deviation of that metric/device distribution and then estimated the variability among those standardized scores. Supplementary Table 8 shows a worked example of calculating SD of z-score for MIMS.

For the second method, Lin’s Concordance Correlation Coefficient (LCCC) was calculated for each combination of the devices by metric (e.g., Apple MIMS and Garmin MIMS, Apple MAD and Garmin MAD). Both methods were selected because they provide a measure of variability independent of accelerometry metric units and because they overcome limitations of other methods that are relative to the mean (i.e., coefficient of variation),²⁶ particularly when there are a large number of small values present. LCCC quantified the precise agreement of accelerometry metric estimates between devices, however, this coefficient can only compare two devices. Therefore, SD of z-score was used to provide a metric of harmonization across the three devices. Conversely, the SD of z-score may be subject to limitations if estimates are systematically different from one device to another. This is because each z-score was calculated using the distribution of each metric and device combination (i.e., Apple MIMS, Garmin ENMO, ActiGraph MAD), for a total of sixteen distributions (4 devices/4 metrics). This could be problematic in situations where one distribution is consistently different from another distribution. For example, if Apple MIMS is always 3 units higher than Garmin MIMS, the standardized scores could appear more harmonized in terms of SD of z-score, hence, if there was low agreement, it may not be apparent in that metric. In other words, turning these estimates into z-scores may be artificially improving harmonization, as this method would not detect poor agreement between two metrics that have systematically different estimates. Therefore, both methods (i.e., SD of z-score and LCCC) provide valuable information on the harmonization and agreement across devices.

For the third method, multi-level intra-class correlation coefficients (ICC) were calculated for each metric.²⁷ Multi-level ICCs were used to account for the nested structure of our data (i.e., rated observations nested within children). The approach proposed by ten Hove and colleagues applies a generalizability theory framework to decompose the variance of multilevel observational data into subject-, cluster- and rater-related components.²⁷ For our current analysis, the ‘subject’ refers to the observation, the ‘cluster’ refers to the participant and the ‘rater’ refers to the device. Specifically, we used Design 1, in which raters are crossed with raters within clusters (i.e., each device evaluates all observations within a given participant but does not evaluate observations across all participants). This is due to the planned missing design of our data, in which children only wore three of four possible devices. In line with this proposed approach, for this analysis only complete cases were used, which relies on the assumption that the data is missing-at-random. Given our planned missing design, this assumption holds true in our data. We used the ICC_s(A,1) coefficient, which is the subject-level single absolute ICC. We specifically used the maximum likelihood estimation approach which facilitates the estimation of reliability coefficients in the presence of missing-at-random data.²⁷ We used open-source R code and packages for calculating multi-level ICCs available via: https://osf.io/bwk5t/.²⁷

Harmonization of Data Across Intensity.

To explore whether intensity was related to harmonization performance we calculated the SD of z-score and LCCC across quartiles for each metric. For each metric, we generated quartiles of intensity: 0-25^th, >25^th-50^th, >50-75^th, >75^th. Therefore, the intensity was metric-specific (MIMS, ENMO, MAD, VM). We then evaluated SD of z-score and LCCC across each quartile for each metric.

Sensitivity Analyses.

Four sensitivity analyses were also conducted to explore relevant factors that may impact harmonization across devices.

Device Order.

For the first sensitivity analysis, we examined the impact of device order (i.e., placement of devices on the child’s arm) on device harmonization (Supplementary Figure 1). Specifically, we evaluated the pattern of device harmonization across metrics for each of the six device orders (Supplementary Table 3).

Timing.

For the second sensitivity analysis, we evaluated the impact of the timing of the metric. Specifically, MIMS, ENMO, and VM were calculated at the second level, while MAD was calculated at the 5-second level. In an effort to maintain the practical usefulness of our findings, we opted to process these metrics in the most standard time window that they are calculated in.^28–33 However, given that there was this difference in the timing of how these metrics were processed, we evaluated the pattern of device harmonization across metrics in two different ways to address this timing discrepancy. First, we compared ENMO and MIMS to MAD only on seconds where there was an observation of MAD (every 5 seconds). Second, we averaged ENMO and MIMS over the 5-second window that MAD was generated from so that ENMO and MIMS were also summary metrics at the 5-second level.

Activity Type.

For the third sensitivity analysis, we examined the impact of protocol activity on device harmonization. As described in Section 2.2 Protocol, all children engaged in the first block of activities that progressed in intensity from sedentary to vigorous, and children were randomized to their second block of activities (Supplementary Table 1). We compared SD of z-score for each metric across all activities to determine whether protocol activity impacted device harmonization.

Clustering.

Lastly, for the fourth sensitivity analysis, we explored whether there was clustering at the individual and activity level. Intra-class correlation coefficients (ICC) were calculated to quantify variation in the standard deviation of z-score due to differences in the individual and the activity.

3. Results

Demographic characteristics of the sample are presented in Table 1. Of the 239 participating children, 53% were male and 30% were Black. Supplementary Table 6 presents descriptives (mean, standard deviation, range) across all devices for each metric. There does not appear to be differences in epoch-level summary metrics across devices.

Table 1.

Demographics of Participating Children (n=239)

Sex	n (%)
Female	112 (47%)
Male	127 (53%)
Race
Asian	3 (1%)
Black or African American	71 (30%)
White	143 (60%)
Other; More than 1 race	18 (8%)
Unknown	3 (1%)
Ethnicity
Not Hispanic or Latinx	230 (96%)
Hispanic or Latinx	9 (4%)
BMI category
Underweight	9 (4%)
Normal weight	141 (59%)
Overweight	45 (19%)
Obese	42 (18%)
	Mean (SD)
Age (yr)	9.3 (2.1)
BMI z-score	0.50 (1.2)

SD = standard deviation; yr = years

Device Harmonization.

Figure 1 displays the overall SD of z-score across devices with a box plot for each metric. The mean SD of z-score across devices was lowest, indicating better harmonization, for MAD (0.13 ± 0.23), followed by ENMO (0.24 ± 0.50), MIMS (0.26 ± 0.39) and then VM (0.27 ± 0.53). Figure 1 indicates that the median SD of z-score of VM is lower than the median SD of z-score of MIMS (0.08 vs. 0.12), while the mean SD of z-score of VM is higher than the mean SD of z-score of MIMS. This indicates that SD of z-score is skewed for VM (i.e., high SD of z-score outliers driving an increased mean relative to the distribution). Although all metrics appear to be right-skewed in Figure 1, VM and ENMO appear to be the most heavily skewed with MAD the least skewed. While median is typically the central tendency measure recommended for skewed distributions,³⁴ it is important to contextualize this finding of VM outperforming MIMS in light of the large number of outliers driving this skewed distribution for VM. Figure 2a–d displays SD of z-score across devices by quartile of intensity. The SD of z-score across devices for MAD by quartile of MAD intensity is presented in Figure 2a. The SD of z-score across devices for ENMO by quartile of ENMO intensity is shown in Figure 2b. Figure 2c displays the SD of z-score across devices for MIMS by quartile of MIMS intensity. Figure 2d presents the SD of z-score across devices for VM by quartile of VM intensity. For all four metrics, there is a consistent pattern such that as intensity increases, the harmonization across devices worsens (i.e., SD of z-score gets larger). When comparing the metrics (Figures 2a–d), MAD consistently had the lowest SD of z-score across all intensity quartiles. This performance gap between MAD and the other metrics appears to widen at higher intensities, with a median SD of z-score for MAD of 0.23 at the 100^th intensity quartile, compared to ENMO (0.51), MIMS (0.52), and VM (0.55).

Figure 1.

Standard Deviation of Z-Score for MAD, ENMO, MIMS, and VM

MAD: Mean Amplitude Deviation, ENMO: Euclidean Norm Minus One, MIMS: Monitor-Independent Movement Summary units, VM: Vector magnitude. VM is shown as a benchmark for comparison to the other metrics (MAD, ENMO, MIMS).

Whiskers represent 1.5 times the interquartile range (IQR)

Red X represents mean value

Figure 2a-d:

MAD, ENMO, MIMS, VM Standard Deviation of Z-Score by Metric Intensity

MAD: Mean Amplitude Deviation, ENMO: Euclidean Norm Minus One, MIMS: Monitor-Independent Movement Summary units.

Whiskers represent 1.5 times the interquartile range (IQR)

Table 2 presents the overall multi-level ICC and Lin’s CCC for each combination of the six devices. MAD had the highest ICC (0.88), followed by MIMS (0.73), ENMO (0.62) and then VM (0.56). Across all device combinations, MAD consistently had the strongest performance in terms of LCCC, with an average LCCC of 0.93. MIMS performed second best (average LCCC: 0.77), followed by ENMO (average LCCC: 0.68) and then VM (average LCCC: 0.58). Across all metrics, the Apple and ActiGraph device combination had the best harmonization. For MAD and MIMS, the Apple and Fitbit device combination had the worst harmonization, while for ENMO and VM, the Garmin and ActiGraph device combination had the worst harmonization, followed by Apple and Fitbit.

Table 2.

Multi-Level Intra-Class Correlation Coefficients (ICC) and Lin’s Concordance Correlation Coefficients (LCCC)

Multi-Level ICC	MAD	ENMO	MIMS	VM
Overall	0.88	0.62	0.73	0.56
LCCC: Device Combination	MAD	ENMO	MIMS	VM
Apple & ActiGraph	0.96	0.76	0.83	0.71
Garmin & ActiGraph	0.91	0.62	0.75	0.56
Fitbit & ActiGraph	0.93	0.71	0.80	0.65
Apple & Fitbit	0.89	0.64	0.70	0.58
Apple & Garmin	0.95	0.70	0.80	0.65
Fitbit & Garmin	0.91	0.64	0.75	0.58
Mean (SD) LCCC	0.93 (0.02)	0.68 (0.05)	0.77 (0.04)	0.62 (0.05)

VM = Vector Magnitude, MIMS = Monitor-Independent Movement Summary units, MAD = Mean Amplitude Deviation, ENMO = Euclidean Norm Minus One, SD = Standard Deviation

The highest performance (indicated by the highest LCCC, ICC) is bolded in each row.

Sensitivity Analyses.

Supplementary Table 2 displays the SD of z-score when adjusting for the metric time discrepancy, compared to the original method. The same trend of MAD performing the best in terms of device harmonization is consistent across both methods of adjusting for the time discrepancy. The SD of z-score across device orders is presented in Supplementary Table 3. Device order did not appear to meaningfully impact the results and the same pattern of MAD performing the best was apparent in this sensitivity analysis as well. The SD of z-scores across activities is presented in Supplementary Table 4. This table highlights that while there are differences in SD of Z-score across intensity, it appears that these differences are due to the intensity level of those activities. ICC results are presented in Supplementary Table 5. A small proportion of total variance is due to person-level differences (ICC range: 2.1% to 5.9%). There was a larger proportion of total variance due to activity level differences (ICC range: 36.6% to 38.8%). As indicated by Supplementary Table 4, these differences at the activity-level appear to be driven by differences in the intensity associated with each activity.

4. Discussion

This study examined between-device harmonization using widely used epoch-level summary metrics, including MIMS, ENMO, and MAD, to determine which metric best harmonizes across a combination of different research-grade and consumer devices. A unique aspect of the design is that the three devices were worn simultaneously in a counterbalanced design. This provides a useful evaluation of harmonization performance since the monitors were worn simultaneously on the same wrist. There is a lack of empirical head-to-head comparisons of harmonization across devices across metrics. In this study, MAD outperformed both ENMO and MIMS in harmonizing across devices, as evidenced by SD of z-score, Lin’s CCC, and ICC. This pattern of MAD harmonizing the best across devices was consistent across all randomized combinations of devices (i.e., device order) and when adjusting for the timing discrepancy of the metrics. These findings highlight the strengths of MAD in distilling accelerometry data into similar estimates across devices with different hardware and software. While this study provides preliminary evidence that MAD may perform better than MIMS and ENMO in harmonizing data across devices, more work is needed to compare outcomes of interest (i.e., physical activity energy expenditure) produced with these metrics across different devices.

Previous literature comparing accelerometry metric performance in terms of agreement across devices is limited by (1) largely focusing on ENMO and MAD, (2) only comparing two devices, and (3) only comparing research-grade devices (i.e., different generations of ActiGraph accelerometers). Two studies compared VM, ENMO, and MAD across two hip-worn ActiGraph monitors, with one study on children and one study on adults.^15,35 Consistent with our findings, MAD performed the best in terms of inter-monitor comparability in both studies. However, these studies are limited by comparing one research-grade accelerometer brand, where the hardware (i.e., resolution, dynamic range) is likely consistent across models.

MAD may be the best at harmonizing across devices due to the way that it is calculated, making it the least affected by inter-device differences. More specifically, MAD is calculated from the average distance from the mean over a given window, in this case 5 seconds. Since it is relative to the mean of that epoch, inter-device differences may not affect MAD as much as the other metrics, which use an absolute reference point (i.e., 1g for ENMO). Interestingly, MIMS had poorer harmonization performance than MAD. In terms of median SD of z-score, MIMS even performed worse than VM, a metric that is closest to the raw acceleration signal (i.e., the benchmark for comparison). This is in contrast to what we would expect, as MIMS has been designed to harmonize across devices with different sampling frequencies and dynamic ranges.¹¹ Another interesting finding of the current study was the similar performance of ENMO and VM, with mean SD of z-scores of 0.24 (±0.50) and 0.27 (±0.53), respectively. Although ENMO slightly out-performed VM in harmonizing across devices, this relatively similar performance is likely due to the way that ENMO and VM are calculated, as ENMO is VM subtracting 1 and rounding negative values to 0.

Consumer wearables are appealing for use in research for a number of reasons, including their relatively lower price, comfortable fit, and ability to capture both accelerometry and heart rate. While consumer wearables offer potential for measuring physical activity, they introduce additional challenges in ensuring consistent measurement across studies. First, consumer wearables use proprietary algorithms to predict physical activity estimates, meaning that estimates are incomparable over time and across different consumer wearable devices. However, this can be overcome by using a device-agnostic approach by leveraging application programming interfaces (APIs) to access and record the underlying accelerometry data.¹⁰ Second, consumer wearables introduce variability in terms of the sensors being used. More specifically, since these are different devices, consumer wearable devices have different accelerometer sensors with varying dynamic ranges and sampling frequency capacities. Additionally, there may be different accelerometer sensors within different iterations of the same device from a brand. Therefore, summary accelerometry metrics that can achieve device-agnostic accelerometry estimates are critical if researchers want to compare estimates across devices with different dynamic ranges and sampling frequencies.

There is a paucity of research examining the performance of MIMS, in addition to ENMO and MAD, across multiple devices with different hardware and software designed for both research and consumer purposes. Only one study, to our knowledge, has evaluated the variability of estimates across both research and consumer wearables when using ENMO, MIMS, and ActiGraph counts.¹¹ MIMS achieved the lowest coefficient of variation, indicating better harmonization, when comparing across eight different acceleration sensors (two consumer, six research-grade) on a mechanical shaker table. While that study did not evaluate the MAD metric, their findings contrast with the current study’s mixed findings on whether ENMO or MIMS performs better following MAD. In the current study, LCCC and ICC support that MIMS out-performs ENMO, however the SD of z-score supports that ENMO outperforms MIMS. However, the current study evaluated harmonization across two (LCCC) to three (SD of z-score, ICC) devices, whereas John et al.¹¹ compared across eight different accelerometers. It is possible that MIMS’ relative performance increases when there are more devices (i.e., more variability), contributing to a widened performance gap between MIMS and ENMO when they are summarizing across more devices. Additionally, the present study evaluated harmonization during a simulated free-living protocol, while their study used a shaker table, which may be contributing to differences in findings.

Another key finding of this study is that harmonization across devices decreases as intensity increases for all metrics (ENMO, MAD, MIMS). Within each metric, there was a clear, consistent pattern of less harmonization (i.e., greater variability between devices) with increasing intensity. When comparing across metrics, MAD consistently outperformed the other metrics across all intensity quartiles. This performance gap was especially evident at the highest intensity quartile, where MAD had a considerably lower SD of z-score compared to the other metrics. This may also be driven by the way that MAD is calculated, as this metric is likely the least impacted by inter-device difference because it focuses on the variability in acceleration and not the magnitude. At higher intensities, other metrics may have greater variability across devices due to the larger magnitude of movement (i.e., there is more room for greater variability at higher intensities). Differences between devices may become greater at increased intensity, however, since MAD is relative to the epoch mean, it may be less impacted. Although this comparison of between-device harmonization across intensity has not been previously done with epoch-level summary metrics, a shaker table study explored comparability of raw acceleration signals across different accelerometer brands and versions, including Axivity AX3, ActiGraph GT3X, ActiGraph wGT3X+, ActiGraph wGT3X-BT, GENEActiv, MOX Logger, and activPAL micro3.³⁶ While that study did not evaluate any epoch-level summary metrics (i.e., ENMO, MAD), they found a similar trend, such that brands are more comparable at lower frequencies (i.e., slower movements). This finding suggests that the issue may not be with device-agnostic metrics being less effective at summarizing acceleration consistently across devices at higher intensities. Instead, it may be due to differences in the raw accelerometry data itself at these intensities, perhaps due to onboard preprocessing of the raw data.⁶ This has important practical implications, and researchers should be cautious when comparing data from multiple devices at higher intensity levels.

Experts in the field have advocated for a single optimal accelerometry metric to simplify analyses and interpretations across studies.⁹ Choosing this metric involves considering two key factors. First, given the adoption of a range of devices (both research and consumer) with different hardware and software, the metric should harmonize across devices. Second, the metric should predict outcomes of interest (i.e., time spent in 24-hour movement behaviors, physical activity energy expenditure) optimally. Therefore, the findings of the current study should be interpreted in the context that this is one element contributing to a broader understanding of which accelerometry metric to use. As these metrics summarize acceleration differently, they rely on both different and similar intrinsic properties,²⁹ thus potentially impacting how they predict outcomes, including physical activity energy expenditure and time spent in 24 hour movement behaviors. For instance, MAD and ENMO do not use a filter, while MIMS uses a band-pass filter to remove motion artifacts, vibration, and gravity.²⁹ MAD and ENMO account for gravity differently, as MAD subtracts the epoch mean to adjust for gravity, while ENMO consistently subtracts 1G.^4,5

This study provides a more granular assessment of harmonization across devices (i.e., at the epoch-level), in contrast to another line of research which has compared the performance of metrics in predicting outcomes such as physical activity energy expenditure and time spent in different 24 hour movement behaviors.⁸ Such studies have different research objectives and differ from the present study by comparing accelerometry data from one device processed with different methods. For instance, one study compared estimates of time spent in 24 hour movement behaviors when data was processed using ENMO, MAD, and counts per minute (CPM) in adults, finding that ENMO represented the most sedentary movement profile and CPM represented the most active profile.⁸ For physical activity energy expenditure prediction, MAD has been shown to outperform other metrics,³⁷ but it can be activity-dependent, with one study finding that the MAD metric predicts most optimally during walking as opposed to running.³⁸ In contrast, another study found that the ActiGraph intermittent metric (an extension of ActiGraph counts) performed better in predicting physical activity energy expenditure compared to MAD.³⁹ Although these studies have different research objectives and are not directly testing the device-agnostic properties of these metrics, they highlight that there are strengths and weaknesses of these metrics. Key outcomes of interest, including time spent in 24-hour movement behaviors and physical activity energy expenditure, are greatly influenced by decisions made by researchers on how to process their accelerometry data. Therefore, in addition to considering which metric harmonizes the best across devices, researchers should also consider what metric performs best in predicting their outcomes of interest.

This study has several strengths that address important gaps in the literature. First, this study used popular consumer wearables with different hardware and software, including Apple Watch, Fitbit Sense, and Garmin Vivoactive. These specific consumer wearables, amongst others, have been widely adopted for research purposes. Previous work evaluating inter-monitor comparability is limited by using different generations of devices from the same brand (i.e., ActiGraph). Secondly, this study compared across three devices being worn simultaneously. Previous work is limited by its reliance on pairwise comparisons only comparing two devices worn simultaneously. Third, this study improved upon previous work by evaluating device harmonization during a simulated free-living physical activity protocol, as opposed to data collected by a shaker table.

This study should also be interpreted in the context of its limitations. First, while the physical activity protocol used was designed to simulate free-living activity in children, the protocol was structured with prescribed activities, therefore these findings would not necessarily generalize to free-living physical activity in children. Second, this study is limited by focusing solely on children, therefore future research should examine if there are consistent findings in adult populations. Third, although this study improved upon previous research by using three devices, future studies should evaluate harmonization across more than three devices and use different research-grade and consumer wearable devices. Fourth, this study is limited by not calibrating accelerometry data. Accelerometers are typically calibrated during the manufacturing process under local gravitational acceleration.⁴⁰ Calibration post-manufacturing can adjust the raw sensor output to account for factors like sensor drift and temperature effects. In the current study, calibration differences among local gravitational acceleration conditions where the wearables were manufactured could impact comparability. MAD may be performing better than ENMO and MIMS because of its reliance on relative deviations rather than an absolute reference point. The way that MAD is calculated could make it less sensitive to the lack of calibration of these devices. Additionally, the way that ENMO is calculated could make it more sensitive to the lack of calibration. Future studies exploring harmonization across devices may consider calibrating accelerometry data as part of accelerometry processing to address this limitation.⁴⁰ Fifth, the metrics were calculated across different time windows, with MIMS, ENMO, and VM calculated across a 1-s epoch, and MAD calculated across a 5-s epoch. Although there is potential for this timing discrepancy to impact findings, we conducted a sensitivity analysis to explore whether device harmonization was different when adjusting for this timing discrepancy and found consistent results (i.e., MAD performing the best). Lastly, this study used three analytic approaches which yielded different conclusions and should therefore be interpreted cautiously. Although MAD consistently performed best, LCCC and ICC supported that MIMS performed better than ENMO, and SD of z-score supported that ENMO performed better than MIMS. While using the combination of these analyses was necessary due to the limitations inherent in each independent analysis (see 2.5 Device Harmonization above), a single unified analytic method that can compare across metrics with different units is needed to address these challenges.

In summary, the findings highlight the strength of the MAD metric in harmonizing accelerometry data across devices with different hardware and software. Across all metrics, between-device harmonization was impacted by intensity, such that as the intensity increased, harmonization worsened. Across all intensities, MAD outperformed the other metrics, with a greater performance gap at the highest intensity quartile. While the current study suggests that MAD may be better than ENMO and MIMS in providing an agnostic metric, more research is needed to evaluate device harmonization in free-living contexts and across more devices.