Skip to main content
NCBI home page
European Journal of Sport Science logoLink to European Journal of Sport Science
. 2024 Mar 18;24(5):614–622. doi: 10.1002/ejsc.12080

Validity of Apple Watch, Garmin Forerunner® 935 and GENEActiv for estimating energy expenditure during close quarter battle training in Special Forces soldiers

Angela Uphill 1,2,✉, Kristina L Kendall 1, Alison Fogarty 3, Stuart Guppy 1, Hannah Brown 1,2, Travis Zomer 2, Simon Parker 2, G Gregory Haff 1
PMCID: PMC11235614

Abstract

The purpose of this study was to investigate the validity of three wrist‐worn devices for estimating energy expenditure (EE) and heart rate (HR) during close “Close Quarter Battle” (CQB). Fifty male soldiers (mean ± SD: age 30.9 ± 4.6 years, height: 1.81 ± 0.64 m and body mass 87.3 ± 7.7 kg) wore three activity monitors (Apple Watch 5, Garmin Forerunner® 935 and GENEActiv accelerometer), a Metamax 3B metabolic cart and a Polar chest strap, whilst conducting a CQB training activity (duration: 26.6 ± 5.0 min). EE and HR data from each test device were compared against criterion measures using ordinary least products regression, 95% limits of agreement, equivalence testing and Mean Absolute Percentage Error (MAPE). Based upon the criterion measure the mean EE for the activity was 372.2 ± 57.6 kcal. All of the devices tested demonstrated fixed and/or proportional bias for EE and a MAPE of >10% (Apple 11.3%, Garmin 15.3%, GENEActiv 57.7%) and therefore did not agree with the criterion. The Apple Watch was a valid method for measuring HR, with a MAPE of 0.6%, and differences with the criterion falling within acceptable limits (≤1 bpm = 83.7%; ≤3 bpm = 97.5% and ≤5 bpm = 97.5%), whereas the Garmin Forerunner® 935 was not valid for measuring HR due to an unacceptable difference compared to the criterion (≤1 bpm = 19.1%; ≤3 bpm = 33.3% and ≤5 bpm = 45.2%). Overall, the Apple Watch 5 can be recommended for measuring HR, but none of the devices are recommended for estimating EE, during CQB.

Keywords: assessment, exercise, measurement, technology, testing

Highlights

  • The Apple Watch 5, Garmin Forerunner 935 and GENEActiv did not agree with the criterion for estimating energy expenditure during close quarter battle military training.

  • The Garmin Forerunner 935 did not agree with the criterion Polar chest strap for measuring heart rate during close quarter battle military training.

  • The Apple Watch 5 accurately measured heart rate when compared to the Polar chest strap and was found to be a valid method for monitoring heart rate during CQB training.

1. INTRODUCTION

The ability to accurately measure energy expenditure (EE) in military populations is important when designing training that adequately prepares soldiers for the metabolic demands of operations (Horner et al., 2013), and for informing nutrition, recovery and injury prevention strategies (Siddall et al., 2019). This is especially true for Special Forces (SF) soldiers, who are more likely to be involved in higher intensity activities, such as “Close Quarter Battle” (CQB), which can include demands such as jumping, sprinting, roping, navigating obstacles and engaging the enemy whilst weighed down with combat equipment (Farina et al., 2017). In order to monitor EE in the military, methods are required that are not only accurate, but also robust, portable, non‐obstructive and able to be utilized during a wide variety of activities (Redmond et al., 2013). There are several techniques for estimating EE, with the most commonly presented methods being doubly labeled water (DLW), indirect calorimetry (IC) via stationary or portable metabolic carts and wearable activity monitors such as heart rate monitors and motion sensors (Ainslie et al., 2003). Whilst DLW and IC are highly valid and reliable methods (Shephard et al., 2012; Westerterp, 2017), they are not practical for ongoing monitoring outside of research due to their high cost and the requirement for specialized procedures and equipment (Ainslie et al., 2003). On the other hand, wearable activity monitors offer a relatively low‐cost, portable method for measuring heart rate (HR) and estimating EE. Studies validating wearable activity monitors have demonstrated varied results in terms of their ability to accurately estimate EE and HR, with a variety of monitors being tested over a wide range of protocols including running, cycling, resistance training and various activities of daily living. For example, Wahl et al. (2017) assessed multiple consumer devices against indirect calorimetry during walking and running. They reported the Mean Absolute Percentage Error (MAPE) of EE estimates for the Garmin Vivoactive watch ranged from an underestimation of 1.3% during interval sprints, to an overestimation of 37% during jogging, whereas the MAPE of the Polar Loop ranged from a 5.6% overestimation during interval sprints to a 56.4% overestimation during slow jogging. Although there are no standard thresholds for MAPE, several authors suggest a threshold of no more than 10% be used to indicate good validity of wearable activity monitors for estimating EE (Boudreaux et al., 2018; Nelson et al., 2016; Passler et al., 2019).

In a study validating wearable activity monitors during resistance exercise, EE from seven different wearable devices had weak correlations (R = 0.02–0.18) and high MAPE values (42%–57%) when compared to IC (Boudreaux et al., 2018). The authors suggested that the accuracy of wrist‐based measures during resistance training may be impacted by changes in skin tension due to repeated contractions of forearm musculature. This potential cause of error could be relevant if wearable activity monitors are used to assess EE in the military, where training regularly involves the carriage and handling of weapons. Further demands unique to the military that may interfere with accuracy of activity monitoring devices include heavy load carriage from body armor or packs and lack of usual arm swing due to weapon carriage.

There have been fewer studies assessing the accuracy of wearable activity monitors for estimating EE in military environments. Horner et al. (2013) reported the 3DNX tri‐axial accelerometer poorly estimated EE for a range of military cohorts when compared to DLW, whereas Siddall et al. (2019) reported that the GENEActiv accelerometer produced statistically equivalent estimates of EE during a 10‐day training activity when compared to DLW. Recently, Kloss et al. (2023) reported poor to excellent correlation between the Polar X Pro and indirect calorimetry for estimating EE in a military cohort; however, the protocol used was treadmill running and did not involve combat relative activities. While there is some evidence supporting the use of wearable activity monitors for estimating EE during military training periods of a week or more, the validity of such devices for determining the metabolic cost of short duration combat training activities has not been comprehensively examined. Specifically, there is a lack of research regarding the validity of using wearable activity monitors to estimate EE during discrete activities involving weapon handling and load carriage, which may challenge the accuracy of these devices. There is also a lack of military research regarding the validity of devices that combine both HR sensors and accelerometry, which may have a better potential to respond to the increased energy demands of load carriage compared to accelerometer‐only devices.

Therefore, the primary aim of this study was to investigate the validity of three wearable activity monitors for estimating EE during CQB training in SF soldiers. The secondary aim was to investigate the validity of wrist‐based HR estimates from wearable activity monitors during dynamic military training involving weapon handling.

2. METHODS

2.1. Participants

A purposive sample of 50 male soldiers (mean ± SD: age 30.9 ± 4.6 years, height: 1.81 ± 0.64 m and body mass 87.3 ± 7.7 kg) were recruited and volunteered to take part in the present study. Inclusion criteria required participants to be qualified SF soldiers, or soldiers who were undergoing SF training. Exclusion criteria included any injury that would prevent participants from fully participating in the activity protocol. All participants were required to refrain from strenuous activity for 24 h, fast for 2 hours and avoid caffeine and nicotine 4 hours prior to conducting the activity protocol. The study was approved by the Department of Defence and Veterans Affairs Human Research Ethics Committee (304‐20) and Edith Cowan University (ECU) Human Research Ethics Committee (2019‐00935). All participants voluntarily read and signed a participant information and consent form prior to participation in the study.

2.2. Procedure

This study utilized a cross sectional, within‐subject design to compare the estimated EE determined from the Apple Watch (Series 5), Garmin Forerunner® 935 and GENEActiv (original) watches, and average HR from the Apple and Garmin Forerunner® 935 watches, with criterion measures of EE (Metamax 3B) and HR (Polar Team2 pro chest strap) during a CQB training activity. Height and body mass were collected the day prior to the CQB protocol. Height was assessed in standing, without shoes, using a stadiometer (Seca 222, Seca). Body mass was assessed using a digital platform scale (Jac 929, NUWEIGH) with participants dressed in light clothing, without shoes. Immediately prior to conducting the protocol, participants were re‐weighed wearing their full combat load in order to establish the total load carried.

2.3. Test devices

The Apple Watch (Series 5, Apple Inc.) is a multi‐sensor watch containing optical heart rate sensor (photoplethysmography), electrical heart rate sensor, accelerometer, GPS, compass and gyroscope. The EE is based on custom algorithms that are not publicly available. Each Apple Watch was connected to a separate iPhone XR (running apple iOS 13.3.1), and all EE and average HR data was exported using the Apple health app. In cases where the activity data needed to be trimmed, full health data was exported via the Apple health app, with the raw data CSV file imported into Microsoft Excel (version 16.36, Office 2020, Microsoft) for further analysis and cleaning.

The Garmin Forerunner® 935 (Garmin Inc., Olathe, Kansas, USA) is a multi‐sensor watch, containing GPS, optical heart rate sensor (photoplethysmography), accelerometer, gyroscope and thermometer. Garmin watches estimate EE based upon an algorithm incorporating age, height, weight, sex, activity and HR (Garmin Support Centre, 2023). EE and HR were exported via the Garmin Connect app (version 4.29.2.1, Garmin). Where data needed to be trimmed, this was completed using the inbuilt options within the Garmin Connect app.

The GENEActiv Original (Activinsights Ltd) is a wrist‐worn tri‐axial accelerometer that offers continuous monitoring of raw accelerometry data for up to 45 days depending on sampling frequency. The device was set to record activity at a frequency of 100 Hz, and data was extracted utilizing GENEActiv software version 3.3 (Activinsights Ltd). The raw sum of vector magnitude (SVM) data was converted to 5 s epochs and analyzed using an Excel macro spreadsheet from Activinsights. The spreadsheet converted SVMs to Metabolic Equivalents (METs) utilizing conversion threshold cut‐offs of <483 = sedentary, 484–678 = light, 679–2264 = moderate and >2264 = vigorous activity, applying a linear scale within each intensity zone. These cut‐offs were adapted from a previous study which validated GENEActiv recording at 80 Hz (Esliger et al., 2011). MET minutes were totaled for each participant and converted from METs to kcal using the equation: Energy (kcal) = (MET.mins x 3.5 x bodyweight)/200. The method used for conversion of raw GENEActiv data to EE was based upon the methods described by Siddall et al. (2019) in their study validating the use of the GENEActiv accelerometer for estimating total daily EE during military training.

Participant height, unloaded weight, date of birth and wrist orientation were entered into each device prior to the activity protocol. Wrist orientation for the Garmin and Apple watches was randomized; however, the Garmin and Apple watches were always worn on opposite wrists to allow the photoplethysmography sensors on each device to be located just above the wrist as per manufacturer instructions. The GENEActiv was worn on the left or the right wrist, just proximal to the Apple or Garmin watch.

Apple and Garmin watches were set to record the activity protocol as a workout, with watches being started and stopped at the commencement and completion of the activity. The Apple Watch was set to the “High intensity interval training” activity. This activity was chosen due to the relative input from each of the device sensors, considering an activity of inconsistent movement, and in which normal arm swing was not expected due to weapon carriage. The Garmin watch was set to record “outdoor run”, as this was the most relevant option available on the watch.

2.4. Criterion devices

The Metamax 3B (Cortex Biophysik) portable metabolic cart was used as the criterion method for estimating EE. This device is considered a valid method for estimating EE (Vogler et al., 2010), and has commonly been used as a criterion method in similar studies (Düking et al., 2020; Gastin et al., 2018; Hajj‐Boutros et al., 2022). The Metamax was fitted into custom pouches affixed to the rear of each soldier's body armor to ensure that the system did not impede weapons handling. Device set‐up, calibration and data extraction were conducted via MetaSoft Studio Software (Cortex Biophysik).

The Polar Team2 Pro (Polar Electro Oy) chest strap served as the criterion method for measuring average HR as it has previously been validated against ECG (Macleod et al., 2012; Schönfelder et al., 2011). Each participant was fitted with the Polar chest strap, set to collect continuous HR data throughout the protocol. On completion of the activity, data was transferred from each Polar transmitter to a central laptop to be viewed and analyzed using Polar Team 2 software (version 1.4.5, Polar Electro Oy). Each participant's HR data from the full activity was averaged to provide a single HR for comparison against Apple and Garmin data.

2.5. Close quarter battle activity

Participants were instructed on the CQB training scenario by senior military staff. Briefly, the scenario involved groups of five or 6 participants wearing body armor and carrying rifles, fast roping from a mock helicopter, walking and running through an outdoor range and engaging targets as required. Participants then ascended stairs to the top of a two‐story range building and systematically cleared a series of rooms throughout the two building floors. The scenario finished with the soldiers climbing up and down a tower (i.e., 15 flights of stairs). This protocol was typical of CQB training and simulated a possible enemy engagement or hostage rescue situation. The activity was designed to take at least 20 min to complete.

2.6. Statistical analysis

All data ware presented as means and standard deviations. Normality was assessed using the Shapiro–Wilk test (p < 0.05) and visual inspection of Q‐Q plots. Agreement between wearable devices and the criterion Metamax for estimates of energy expenditure was assessed using ordinary least products regression, 95% limits of agreement and equivalence testing (Dixon et al., 2018; Ludbrook, 2002; Martin Bland et al., 1986). Proportional bias was considered present if the 95% confidence interval for the slope of the least products regression did not include one, while fixed bias was considered present if the 95% confidence interval for the intercept did not include zero (Ludbrook, 2012). Where proportional bias was present, the 95% limits of agreement were adjusted accordingly (Ludbrook, 2010). Agreement was also assessed using two one‐sided t‐tests, with the null hypothesis rejected if the 90% CI was not completely within the equivalence bounds of ±10% of the mean of the criterion (Bai et al., 2018; Dixon et al., 2018). Due to the presence of non‐normal distributions, agreement between wearable devices and the Polar HR strap for estimates of HR was assessed using a non‐parametric approach. Agreement was considered acceptable if 60%, 85% and 95% of the differences between the criterion and novel devices was less than ±1, ±3 and ± 5 bpm, respectively (Bland et al., 1999). Statistical equivalence for the HR measures was assessed using a bootstrap resampling approach, with the null hypothesis rejected if the 90% CI was not fully contained within equivalence bounds of −5 to 5 bpm. Mean absolute percentage error (MAPE) was calculated as |actual—predicted|/actual × 100, with acceptable error defined as a MAPE of ≤10% as per previous wearable activity monitor validation studies (Boudreaux et al., 2018; Nelson et al., 2016; Passler et al., 2019). Additionally, standard error of the estimate (SEE = SD1−r2) and constant error (CE) were calculated. Statistical analyses were performed in custom R scripts (version 4.2). Bias corrected and accelerated 95% CIs for the least products regression estimates were calculated using bootstrap resampling (Canty et al., 2021). Equivalence tests were performed using the TOSTER package (version 0.8.0) (Caldwell, 2022; Lakens, 2017). Non‐parametric limits of agreement were calculated in a custom Excel spreadsheet (Microsoft Corp).

3. RESULTS

A total of 50 male soldiers completed the CQB training protocol. Participants carried a combat load of 25.2 ± 2.5 kg, inclusive of body armor, helmet and weapons, in addition to the research equipment (i.e., Metamax 3 B). The total time to complete the protocol ranged between 20.8 and 36.7 min (average 26.6 ± 5.0 min), depending upon the tactics applied during the activity and experience level of the participants.

There was a considerable rate of data loss due to technical issues throughout the activity, resulting in varying numbers of data sets available for analysis. After applying an exclusion criterion of <75% activity duration recorded, the data remaining for analysis were as follows: EE: Apple versus Metamax N = 34, Garmin versus Metamax N = 30 and GENEActiv versus Metamax N = 36; HR: Garmin versus Polar N = 42 and Apple versus Polar N = 49.

3.1. Energy expenditure

Using the data from the Metamax 3B, the average reference value for EE for the CQB activity was 372.2 ± 57.6 kcal (range: 265.0–470.6 kcal). The EE and estimate error data for each device is presented in Table 1.

TABLE 1.

Energy expenditure estimates and comparisons with criterion (Metamax).

Device N Mean ± SD (kcal) CE (kcal) SEE (kcal) MAPE (%)
Metamax 36 372.2 ± 57.6#
Apple 34 350.4 ± 73.9 −25.2 22.9 11.3
Garmin 30 369.1 ± 102.2 −5.2 67.3 15.3
GENEActiv 36 156.2 ± 23.9 −216.0 31.1 57.7
Open in a new tab

Note: #Value shown refers to Metamax values for the data set of N = 36 used for comparison with GENEActiv. Samples used for comparison with Apple and Garmin had corresponding Metamax values of 375.6 ± 56.1 kcal for comparison with Apple (N = 34) and 374.3 ± 60.1 kcal for comparison with Garmin (N = 30).

Abbreviations: CE, Constant Error; MAPE, Mean Absolute Percentage Error; SEE, Standard Error of the Estimate.

When estimates of EE were compared using ordinary least products regression to the criterion Metamax portable metabolic cart, both fixed and proportional bias were present for the Apple and Garmin devices (Figure 1). Only proportional bias was present for the GENEActive device. The significant relationship between the differences and means for each comparison between criterion and wearable device, along with the relatively large heteroscedasticity of errors (R 2 = 0.21–0.62), further confirmed the presence of proportional bias. The Apple and Garmin devices were however statistically equivalent to the Metamax for estimates of EE as the 90% CI was entirely within the defined equivalence bounds (Apple: −37.16, −13.16; Garmin: −30.93, 20.50). The GENEActiv was not however statistically equivalent to the Metamax (90% CI: −229.09, −202.89). When the MAPE was examined, none of the devices tested met the ≤10% threshold for estimating EE (Table 1).

FIGURE 1.

FIGURE 1

Open in a new tab

Ordinary least products regression comparisons and Bland–Altman plots of energy expenditure. For (A–C), the solid line represents the ordinary least products regression line, and the dashed line represents identity. For (D–F), the horizontal line represents the bias between energy expenditure estimates from criterion (Metamax) and Apple (D), Garmin (E) and GENEActive (F), and the dashed lines represent the adjusted 95% limits of agreement. The solid line represents the relationship between the criterion and the wearable device.

3.2. Heart rate

The mean HR for the activity according to the reference Polar chest strap was 158.0 ± 10.8 bpm. Mean HR estimates and corresponding errors and associations between the Apple and Garmin devices compared to the Polar HR monitor are presented in Table 2.

TABLE 2.

Heart rate outputs and comparisons with criterion (Polar).

Device N Mean ± SD (bpm) CE (bpm) SEE (bpm) MAPE (%)
Polar 49 158.0 ± 10.8#
Apple 49 157.4 ± 10.9 −0.62 1.33 0.6
Garmin 42 146.8 ± 16.1 −11.60 13.45 7.3
Open in a new tab

Note: #Value shown refers to sample of N = 49 for comparison with Apple. The sample of N = 42 for comparison with Garmin had value 158.4 ± 10.8 bpm.

Abbreviations: CE, Constant Error; MAPE, Mean Absolute Percentage Error; SEE, Standard Error of the Estimate.

Based on the proportion of differences in heart rate between the Apple Watch and Polar that fell within acceptable limits (≤1 bpm = 83.7%; ≤3 bpm = 97.5% and ≤5 bpm = 97.5%) (Figure 2), the Apple Watch agreed with the criterion. The Garmin watch however did not demonstrate acceptable agreement, with only 19.1% of differences being ≤1 bpm, 33.3% of differences being ≤3 bpm and 45.2% of differences being ≤5 bpm. When compared to the criterion measure of HR from the Polar strap, the Apple Watch was considered statistically equivalent (90% CI = −0.96, −0.35), while the Garmin was not (90% CI: −15.17, −8.41).

FIGURE 2.

FIGURE 2

Open in a new tab

Bland–Altman plots of average heart rate. The dotted dash and dashed lines in A represent ±1 and ± 3 beat per minute difference between the criterion (Polar) and Apple, respectively. Dashed lines in B represent ±5 beat per minute difference between the criterion and Garmin.

4. DISCUSSION

We investigated the validity of the Apple Watch (Series 5), Garmin Forerunner® 935 and GENEActiv for estimating EE and HR in soldiers during CQB training. The primary finding of this study was that the wearable devices displayed some group‐level agreement with the criterion Metamax 3B, but the presence of unacceptably high MAPE (>10%) and both fixed and proportional bias means that these devices should not be used interchangeably to estimate EE during a high‐intensity CQB task. Similarly, there was a lack of agreement between the HR determined with the Garmin Forerunner® 935 and the Polar HR monitor; therefore, the Garmin did not provide valid measures of average HR. However, there was agreement between the Apple Watch and the Polar HR monitor, and therefore, this device can be used to validly monitor heart rate during CQB training.

Despite statistical equivalence with the criterion Metamax 3B, the Apple Watch demonstrated an overall underestimation of EE for the CQB activity, with a CE of −25.2 kcal (Table 1). Further, both fixed and proportional bias, as well as unacceptably high MAPE were found, suggesting a lack of agreement between the two devices on an individual level, similar to previous findings relating to the accuracy of EE estimates from wearable devices (Shei et al., 2022). This divergence in the assessment of agreement is not unexpected as the statistical methods used assess different components of agreement (group‐level vs. individual‐level) (Adamakis, 2019), but does highlight the need to interpret the results of studies investigating the validity of wearable devices carefully. Of note in this study, the MAPE of 11.3% for the Apple Watch, which was the lowest of the devices tested in this study, was similar to the error found in other studies using protocols of fast walking or running, and substantially lower than the MAPE reported in studies estimating EE at rest. For example, Dooley et al. (2017) reported a MAPE of 14.1% during brisk treadmill walking, but a MAPE of 210.8% during sitting for the Apple Watch (Series 1). Additionally, Hajj‐Boutros et al. (2022) reported a MAPE of 14.9% during running and 47.8% during sitting for the Apple Watch (Series 6). It is unclear why the device is more accurate at high compared to low intensity activities; however, it is likely that the algorithms used for calculating EE are better suited to vigorous activity rather than the rest, given the workout function used to record the activities is intended for use during exercise.

Similar results were found when EE from the Garmin Forerunner® 935 was examined, with the device found to be statistically equivalent to the Metamax 3B. Moreover, the Garmin watch achieved the lowest CE (−5.2 kcal). However, the MAPE for this device (15.3%) was higher than the defined acceptable error. Further, both fixed and proportional bias was found when compared to the Metamax 3B, again indicating that while a degree of group‐level agreement may be present between the EE estimates from both devices as indicated by the results of the equivalence tests; substantial errors are present on the individual level, which is arguably more important to practitioners. Although we are unaware of previous studies validating the Garmin Forerunner® 935, a similar device comparison comes from a study by Wahl et al. (2017) who reported MAPEs ranging from 9.2% during intermittent running to 26.6% during slow treadmill walking for the Garmin Forerunner® 920XT. In contrast to the present study, Wahl et al. (2017) reported that the Garmin Forerunner® 920XT consistently underestimated EE. Given that the EE algorithm incorporates HR, it is possible that some of the error of EE in the present study may have been due to the error in measured HR (Table 2 and Figure 2). Nevertheless, Garmin watches are a popular choice among military members due to their robust design and integration of GPS and sport‐tracking functionality. However, based on the results of this study, this device is not recommended for estimating EE during CQB training.

The GENEActiv also demonstrated proportional bias when compared to the criterion measure, and therefore, the estimate of EE during the CQB task did not agree with the Metamax 3B. Specifically, this device underestimated the mean EE by 58%, which was substantially greater than the MAPE of the other devices tested. Furthermore, unlike the other two wearable devices tested in this study, the estimates of EE from the GENEActiv were not statistically equivalent to the Metamax 3B. An important consideration in the use of accelerometer‐only devices in the military is that accelerometers do not have a capacity to recognize that the soldier is wearing load. Our findings were in contrast to those of Siddall et al. (2019), who reported that the GENEActiv was statistically equivalent to DLW for estimating total daily EE during 10 days of military training. These differences are likely due to the very different activities and durations involved in each study, with the present study involving a short duration acute bout of high intensity activity completed while carrying a combat load (i.e., 25.2 ± 2.5 kg). The study by Siddall et al. (2019), however, was conducted over 10 days and comprised “usual training”, which included classroom‐based learning, physical training sessions, military drill and a single day of field based combat during which the participants wore 25 kg load. The MET conversions utilized to convert the raw data into EE in both studies were originally based on non‐military activities (Esliger et al., 2011) and proved insufficient for the demands of the CQB activity. Therefore, based upon the results of the present study, it is recommended that future studies create military‐specific MET conversions from SVM data that accounts for load carriage and the intensity of military training.

The HR output from the Garmin Forerunner® 935 did not demonstrate acceptable agreement with the criterion measure. The cause of this lack of agreement was unclear; however, it is possible that the accuracy of the photoplethysmography sensors was reduced due to changes in skin tension and movement artifact (Boudreaux et al., 2018) caused by carrying and using weapons. Despite the apparent challenge that weapon handling and CQB may have presented for the sensors, this study demonstrates that highly accurate wrist‐based HR is possible, as evidenced by the results from the Apple Watch. The Apple Watch demonstrated very good accuracy for measuring average HR in this study. Specifically, for 83.7% of the participants, there was 0 or 1 beat per minute difference compared to the Polar chest strap, and in 97.5% of participants, there was 3 or less beats per minute difference compared to the criterion. These results are comparable to other studies that have reported that the Apple Watch is an accurate wrist‐based HR measurement device (Apple Watch series 1, 2 and 6) (Boudreaux et al., 2018; Düking et al., 2020; Shcherbina et al., 2017). In contrast to the present study, Khushhal et al. (2017) reported that the Apple Watch (series 0) demonstrated an accurate estimate of heart rate during walking (bias ± 95% LoA: 0 ± 2 bpm and r = 0.97); however, there was significantly greater error as intensity increased, which was reflected by wider LoA and reduced correlation during running at 10 km.hr−1 (2 ± 23 bpm, r = 0.8). It is likely that the differences between the present study and the study by Khushhal et al. (2017) are related to upgrades to the photoplethysmography sensors on the Apple Watch Series 5 compared to those contained within the Apple Watch Series 0. In addition to this, smart watch software is regularly updated for the Apple and Garmin devices utilized in this study, which would presumably include updates to algorithms used when quantifying heart rate from the various sensors contained within each watch. This constantly evolving technology makes it difficult to directly compare against studies that used previous versions of devices or software.

While the present study provides valuable insights into the utility of the investigated devices within the military environment, there were some limitations that must be noted. Firstly, there were no female soldiers involved. It is assumed that the Apple and Garmin devices would use different algorithms for females compared to males; thus, the results of this study may not apply to female soldiers. Additionally, the activity duration was relatively short, averaging 26.6 min. While this study provides an indication of the expected accuracy or error from each device, it is possible that longer duration activities may give different results. Therefore, future studies should investigate the accuracy of these devices during a variety of military training activities and durations. Finally, while the participants wore a common standardized fighting load (25.5 ± 2.5 kg) (Nindl et al., 2013), the results of the present study may not be generalizable to activities that require heavier load carriage.

Overall, the Apple Watch 5, Garmin Forerunner® 935 and GENEActiv accelerometer did not prove to be valid methods for estimating EE during CQB training in SF soldiers. While none of the devices provided a valid method for monitoring EE in this unique military environment, the results of the present study are encouraging and provide preliminary evidence that continued improvement to device sensors and algorithms may ultimately lead to wearable activity monitors that may provide a valid method of estimating EE during CQB training. The Apple Watch was the only wrist‐worn device tested that provided a valid estimate of average HR during CQB training, with <1% error compared to the criterion. As such, this device can be used with confidence to provide accurate HR feedback during CQB. Conversely, the Garmin Forerunner® 935 demonstrated inconsistent HR results with an overall underestimation of average HR and is not recommended as a method for monitoring HR during CQB training.

CONFLICT OF INTEREST STATEMENT

The authors report there are no competing interests to declare.

ACKNOWLEDGMENTS

This work was supported by the Defence Science Center WA under a Research Higher Degree Student Grant (G1004741), Edith Cowan University and Wanderers Education Programme (G1004466) and the Australian Defence Force. The authors wish to thank the soldiers from the Australian Defence Force who so generously gave their time to participate in this study.

Open access publishing facilitated by Edith Cowan University, as part of the Wiley ‐ Edith Cowan University agreement via the Council of Australian University Librarians.

REFERENCES

  1. Adamakis, Manolis . 2019. “An Opinion Regarding Equivalence Testing for Evaluating Measurement Agreement.” Journal of Scientific Research and Reports 24(5): 1–4. 10.9734/jsrr/2019/v24i530163. [DOI] [Google Scholar]
  2. Ainslie, Philip N. , Reilly Thomas, and Westerterp Klass R.. 2003. “Estimating Human Energy Expenditure: A Review of Techniques with Particular Reference to Doubly Labelled Water.” Sports Medicine 33(9): 683–698. 10.2165/00007256-200333090-00004. [DOI] [PubMed] [Google Scholar]
  3. Bai, Yang , Hibbing Paul, Mantis Constantine, and Welk Gregory J.. 2018. “Comparative Evaluation of Heart Rate‐Based Monitors: Apple Watch vs Fitbit Charge HR.” Journal of Sports Sciences 36(15): 1734–1741. 10.1080/02640414.2017.1412235. [DOI] [PubMed] [Google Scholar]
  4. Bland, J. Martin , and Altman Douglas G.. 1999. “Measuring Agreement in Method Comparison Studies.” Statistical Methods in Medical Research 8(2): 135–160: Epub 1999/09/29. 10.1177/096228029900800204. [DOI] [PubMed] [Google Scholar]
  5. Boudreaux, Benjamin D. , Hebert Edward P., Hollander Daniel B., Williams Brian M., Cormier Corinne L., Naquin Mildred R., Gillan Wynn W., Gusew Emily E., and Kraemer Robert R.. 2018. “Validity of Wearable Activity Monitors during Cycling and Resistance Exercise.” Medicine & Science in Sports & Exercise 50(3): 624–633: Epub 2017/12/01. 10.1249/mss.0000000000001471. [DOI] [PubMed] [Google Scholar]
  6. Caldwell, Aaron R. 2022. “Exploring Equivalence Testing with the Updated TOSTER R Package.” PsyArXiv. 10.31234/osf.io/ty8de. [DOI] [Google Scholar]
  7. Canty, A. , and Ripley B.. 2021. Boot: Bootstrap R (S‐Plus) Functions.
  8. Dixon, Philip M. , Saint‐Maurice Pedro F., Kim Youngwon, Hibbing Paul, Bai Yang, and Welk Gregory J.. 2018. “A Primer on the Use of Equivalence Testing for Evaluating Measurement Agreement.” Medicine & Science in Sports & Exercise 50(4): 837–845: Epub 2017/11/15. 10.1249/MSS.0000000000001481. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Dooley, Erin E. , Golaszewski Natalie M., and Bartholomew John B.. 2017. “Estimating Accuracy at Exercise Intensities: A Comparative Study of Self‐Monitoring Heart Rate and Physical Activity Wearable Devices.” JMIR Mhealth Uhealth 5(3): e34‐e. 10.2196/mhealth.7043. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Düking, Peter , Giessing Laura, Frenkel Marie Ottilie, Koehler Karsten, Holmberg H.‐Christer, and Sperlich Billy. 2020. “Wrist‐Worn Wearables for Monitoring Heart Rate and Energy Expenditure while Sitting or Performing Light‐To‐Vigorous Physical Activity: Validation Study.” JMIR Mhealth Uhealth 8(5): e16716: Epub 2020/05/07. 10.2196/16716. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Esliger, Dale W. , Rowlands Ann V., Hurst Tina L., Catt Michael, Murray Peter, and Eston Roger G.. 2011. “Validation of the GENEA Accelerometer.” Medicine & Science in Sports & Exercise 43(6): 1085–1093: Epub 2010/11/23. 10.1249/MSS.0b013e31820513be. [DOI] [PubMed] [Google Scholar]
  12. Farina, Emily K. , Taylor Jonathan C., Means Gary E., Williams Kelly W., Murphy Nancy E., Margolis Lee M., Pasiakos Stefan M., Lieberman Harris R., and McClung James P.. 2017. “Effects of Combat Deployment on Anthropometrics and Physiological Status of U.S. Army Special Operations Forces Soldiers.” Military Medicine 182(3): e1659–e1668: Epub 2017/03/16. 10.7205/MILMED-D-16-00022. [DOI] [PubMed] [Google Scholar]
  13. Garmin Support Centre . 2023. [Internet]. NSW: Garmin Ltd; c1996–2023. Forerunner 935; Calorie; Article; Calorie Terminology: [about 1 screen]. Available from: https://support.garmin.com/en‐AU/?faq=lkl4cwCLlK7ox362uGQEV7&productID=564291&searchQuery=Calorie&tab=topics.
  14. Gastin, Paul B. , Cayzer Cassy, Dwyer Dan, and Robertson Sam. 2018. “Validity of the ActiGraph GT3X+ and BodyMedia SenseWear Armband to Estimate Energy Expenditure during Physical Activity and Sport.” Journal of Science and Medicine in Sport 21(3): 291–295. 10.1016/j.jsams.2017.07.022. [DOI] [PubMed] [Google Scholar]
  15. Hajj‐Boutros, Guy , Landry‐Duval M.‐Anne, Comtois Alain Steve, Gouspillou Gilles, and Karelis Antony D.. 2022. “Wrist‐worn Devices for the Measurement of Heart Rate and Energy Expenditure: A Validation Study for the Apple Watch 6, Polar Vantage V and Fitbit Sense.” European Journal of Sport Science 23(2): 1–13: Epub 2021/12/28. 10.1080/17461391.2021.2023656. [DOI] [PubMed] [Google Scholar]
  16. Horner, Fleur , Bilzon James L., Rayson Mark, Blacker Sam, Richmond Victoria, Carter James, Wright Anthony, and Nevill Alan. 2013. “Development of an Accelerometer‐Based Multivariate Model to Predict Free‐Living Energy Expenditure in a Large Military Cohort.” Journal of Sports Sciences 31(4): 354–360. 10.1080/02640414.2012.734632. [DOI] [PubMed] [Google Scholar]
  17. Khushhal, Alaa , Nichols Simon, Evans Will, Gleadall‐Siddall Damien, Page Richard, O'Doherty Alasdair, Carroll Sean, Ingle Lee, and Abt Grant. 2017. “Validity and Reliability of the Apple Watch for Measuring Heart Rate during Exercise.” Sports Medicine International Open 1(6): E206–E211: Epub 2017/10/18. 10.1055/s-0043-120195. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Kloss, Emily B. , Givens Andrea, Palombo Laura, Bernards Jake, Niederberger Brenda, Bennett Daniel W., and Kelly Karen R.. 2023. “Validation of Polar Grit Pro for Estimating Energy Expenditure during Military Field Training: A Pilot Study.” Journal of Sports Science and Medicine 22: 658–666. 10.52082/jssm.2023.658. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Lakens, Daniël . 2017. “Equivalence Tests: A Practical Primer for T Tests, Correlations, and Meta‐Analyses.” Social Psychological and Personality Science 8(4): 355–362. 10.1177/1948550617697177. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Ludbrook, John . 2002. “Statistical Techniques for Comparing Measurers and Methods of Measurement: a Critical Review.” Clinical and Experimental Pharmacology and Physiology 29(7): 527–536: Epub 2002/06/13. 10.1046/j.1440-1681.2002.03686.x. [DOI] [PubMed] [Google Scholar]
  21. Ludbrook, John . 2010. “Confidence in Altman‐Bland Plots: A Critical Review of the Method of Differences.” Clinical and Experimental Pharmacology and Physiology 37(2): 143–149. 10.1111/j.1440-1681.2009.05288.x. [DOI] [PubMed] [Google Scholar]
  22. Ludbrook, John . 2012. “A Primer for Biomedical Scientists on How to Execute Model II Linear Regression Analysis.” Clinical and Experimental Pharmacology and Physiology 39(4): 329–335: Epub 2011/11/15. 10.1111/j.1440-1681.2011.05643.x. [DOI] [PubMed] [Google Scholar]
  23. Macleod, H. , and Sunderland C.. 2012. “Reliability of the Physiological and Metabolic Responses to a Field Hockey Specific Treadmill Protocol for Elite Female Players.” The Journal of Sports Medicine and Physical Fitness 52(4): 351–358: Epub 2012/07/26. [PubMed] [Google Scholar]
  24. Martin Bland, J. , and Altman DouglasG.. 1986. “Statistical Methods for Assessing Agreement between Two Methods of Clinical Measurement.” Lancet 327(8476): 307–310. 10.1016/s0140-6736(86)90837-8. [DOI] [PubMed] [Google Scholar]
  25. Nelson, M. Benjamin , Kaminsky Leonard A., Dickin D. Clark, and Montoye Alexander H. K.. 2016. “Validity of Consumer‐Based Physical Activity Monitors for Specific Activity Types.” Medicine & Science in Sports & Exercise 48(8): 1619–1628: Epub 2016/03/26. 10.1249/MSS.0000000000000933. [DOI] [PubMed] [Google Scholar]
  26. Nindl, Bradley C. , Castellani John W., Warr Bradley J., Sharp Marilyn A., Henning Paul C., Spiering Barry A., and Scofield Dennis E.. 2013. “Physiological Employment Standards III: Physiological Challenges and Consequences Encountered during International Military Deployments.” European Journal of Applied Physiology 113(11): 2655–2672: Epub 2013/02/23. 10.1007/s00421-013-2591-1. [DOI] [PubMed] [Google Scholar]
  27. Passler, Stefanie , Bohrer Julian, Blöchinger Lukas, and Senner Veit. 2019. “Validity of Wrist‐Worn Activity Trackers for Estimating VO2max and Energy Expenditure.” International Journal of Environmental Research and Public Health 16(17): 3037. 10.3390/ijerph16173037. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Redmond, J. E. , Cohen B. S., Simpson K., Spiering B. A., and Sharp M. A.. 2013. Measuring Physical Activity during US Army Basic Combat Training: A Comparison of 3 Methods, 48–54. US Army Med Dep J. [PubMed] [Google Scholar]
  29. Schönfelder, Martin , Hinterseher Georg, Peter Philipp, and Spitzenpfeil Peter. 2011. “Scientific Comparison of Different Online Heart Rate Monitoring Systems.” International Journal of Telemedicine and Applications 2011: 631848–631856: Epub 2011/07/16. 10.1155/2011/631848. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Shcherbina, Anna , Mattsson C., Waggott Daryl, Salisbury Heidi, Christle Jeffrey, Hastie Trevor, Wheeler Matthew, and Ashley Euan. 2017. “Accuracy in Wrist‐Worn, Sensor‐Based Measurements of Heart Rate and Energy Expenditure in a Diverse Cohort.” Journal of Personalized Medicine 7(2): 3–15. 10.3390/jpm7020003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Shei, R.‐Jay , Holder Ian G., Oumsang Alicia S., Paris Brittni A., and Paris Hunter L.. 2022. “Wearable Activity Trackers‐Advanced Technology or Advanced Marketing?” European Journal of Applied Physiology 122(9): 1975–1990: Epub 2022/04/22. 10.1007/s00421-022-04951-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Shephard, Roy J. , and Aoyagi Yukitoshi. 2012. “Measurement of Human Energy Expenditure, with Particular Reference to Field Studies: an Historical Perspective.” European Journal of Applied Physiology 112(8): 2785–2815. 10.1007/s00421-011-2268-6. [DOI] [PubMed] [Google Scholar]
  33. Siddall, Andrew G. , Powell Steven D., Needham‐Beck Sarah C., Edwards Victoria C., Thompson Jane E. S., Kefyalew Sarah S., Singh Priya A., et al. 2019. “Validity of Energy Expenditure Estimation Methods during 10 Days of Military Training.” Scandinavian Journal of Medicine & Science in Sports 29(9): 1313–1321: Epub 2019/05/29. 10.1111/sms.13488. [DOI] [PubMed] [Google Scholar]
  34. Vogler, Andrew J. , Rice Anthony J., and Gore Christopher J.. 2010. “Validity and Reliability of the Cortex MetaMax3B Portable Metabolic System.” Journal of Sports Sciences 28(7): 733–742: Epub 2010/04/27. 10.1080/02640410903582776. [DOI] [PubMed] [Google Scholar]
  35. Wahl, Yvonne , Düking Peter, Droszez Anna, Wahl Patrick, and Mester Joachim. 2017. “Criterion‐validity of Commercially Available Physical Activity Tracker to Estimate Step Count, Covered Distance and Energy Expenditure during Sports Conditions.” Frontiers in Physiology 8: 725: Epub 2017/10/12. 10.3389/fphys.2017.00725. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Westerterp, Klaas R. 2017. “Doubly Labelled Water Assessment of Energy Expenditure: Principle, Practice, and Promise.” European Journal of Applied Physiology 117(7): 1277–1285. 10.1007/s00421-017-3641-x. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from European Journal of Sport Science are provided here courtesy of Wiley

RESOURCES