Assessing the Validity of Several Heart Rate Monitors in Wearable Technology While Mountain Biking

Abstract

Purpose

This study sought to assess the validity of several heart rate (HR) monitors in wearable technology during mountain biking (MTB), compared to the Polar H7® HR monitor, used as the criterion device.

Methods

A total of 20 participants completed two MTB trials while wearing six HR monitors (5 test devices, 1 criterion). HR was recorded on a second-by-second basis for all devices analyzed. After data processing, validity measures were calculated, including 1. error analysis: mean absolute percentage errors (MAPE), mean absolute error (MAE), and mean error (ME), and 2. Correlation analysis: Lin’s concordance correlation coefficient (CCC) and Pearson’s correlation coefficient (r). Thresholds for validity were set at MAPE < 10% and CCC > 0.7.

Results

The only device that was found to be valid during mountain biking was the Suunto Spartan Sport watch with accompanying HR monitor, with a MAPE of 0.66% and a CCC of 0.99 for the overall, combined data.

Conclusion

If a person would like to track their HR during mountain biking, for pacing, training, or other reasons, the devices best able to produce valid results are chest-based, wireless electrocardiogram (ECG) monitors, secured by elastic straps to minimize the movement of the device, such as the Suunto chest-based HR monitor.

INTRODUCTION

The development of fitness trackers and other wearable technology designed for health and fitness purposes is growing in popularity and sophistication every day. Wearable devices have been the top fitness trend in five out of the last seven years (it was number three in 2018 and number two in 2021), as determined by health and fitness professionals throughout the world (43–49). Wearable devices can be used in a range of exercise formats, from running, cycling, swimming, rowing, weightlifting, and mountain biking, to name a few. They can measure or estimate a variety of physiological and physical variables, such as step count, energy expenditure, VO₂max, lactate threshold, heart rate, stride length, vertical oscillation, ground contact time, blood oxygen saturation (via pulse oximetry), and many others. As these devices have gained acceptance among the general population, they have also caught the interest of athletes, sports scientists, and researchers (1, 24, 37). These wearable devices have the potential to revolutionize physiological research, due to their prevalence and constant monitoring of the user’s physiology (51). However, in order to properly use these devices, independent validation needs to take place to determine the device’s validity and reliability in measuring or estimating each variable (4, 6).

Each physiological or physical variable tracked by these devices ranges in the precision of the measurements or estimates, with aspects such as step-count, run cadence, stride length and VO₂max generally being accurate (6, 7, 13, 35), and energy expenditure and vertical oscillation being less accurate (4, 6, 7). One of the most common measurements for devices to record is heart rate (HR), and its performance in recording HR during exercise has had mixed results, with wireless, chest-based ECG monitors showing high levels of validity and reliability (18, 28, 30) and wrist-based sensors showing less accuracy (30, 38, 39). Wearable technology designed to measure HR comes in an array of different devices that can include chest straps, wrist-based watches and sensors, smart bras, earbuds, rings, and forearm or bicep-based devices. These devices are designed to be used in all environments, measuring HR throughout the day. There have been an abundance of studies utilizing the laboratory to validate the ability of wearable technology to measure/estimate variables like HR, energy expenditure, and step-count while performing common exercise modalities like running or biking (16, 34). However, there is a lack of both validation studies and reliability studies that take place in field, outdoor, or applied settings for wearable technology (6).

There are two types of technology currently utilized by wearable technology to measure or estimate HR, photoplethysmography (PPG) and electrocardiography (ECG). PPG devices utilize light and optical sensors to determine changes in blood flow, and therefore, HR. ECG devices measure the electrical signals produced by the polarization and depolarization of the heart to determine HR. A more detailed analysis of the technology can be found in the discussion, but PPG devices can be placed virtually anywhere on the body, as long as light can pass through and be reflected back by the blood vessels. This makes them more versatile in terms of placement and are generally more comfortable. However, they are more susceptible to motion artifacts and other noise in the signal processing compared to ECG devices. ECG devices, as used by wearable technology companies, are usually in one of two forms. The first (and the type used for this study) is a chest strap with two leads (sensors), placed on either side of the anterior chest to measure the difference in electrical signals. This type of device allows for continuous monitoring, and therefore is the only type to be used to track HR during exercise (thus far). The second form is built into the watch or device that requires two points of contact, usually a finger from the other hand, to act as the second lead. As exercise would be awkward with the opposite hand always needing to be in contact with the watch, this is impractical for continuous monitoring.

Mountain biking is a popular and growing sport, enjoyed recreationally and professionally (14, 41). It was conceived back in the late 1970s and became an Olympic sport in 1996 (15). It involves both uphill and downhill biking on dirt roads and can be physiologically demanding, especially on the uphill segments (5, 21). Mountain biking athletes can use wearable devices to track physiological and physical variables and make training decisions. They may use it to determine HR, energy expenditure, altitude gain, distance traveled, and many other variables, as stated earlier. According to a recent systematic review, it is important to test these devices in many different environments and exercise formats, to better understand their limitations and use cases (6). Because of these two aspects, 1) the growing popularity of mountain biking, and 2) the need to validate wearable devices in a range of exercises in applied settings, mountain biking was a satisfactory exercise modality to choose for this study. Therefore, the purpose of this study was to determine the validity of several HR monitors while mountain biking outdoors. We hypothesized that all devices would meet predefined thresholds for accuracy and validity.

METHODS

Participants

Twenty apparently healthy participants (no areas of concern after answering the PAR-Q) volunteered for this study (10 male, 10 female, age = 26.3 ± 6.6 years, height = 171.8 ± 8.0cm, mass = 73.9 ± 19.0kg, reported as mean ± standard deviation [SD]) (see Table 1). A sample size of 20 is a common amount for wearable technology validation studies, and as we would be collecting data on a second-by-second basis, even with a very low effect size, such as 0.1, this study would be sufficiently powered. An a-priori power analysis determined that to reach a power of at least 0.8 (with an effect size of 0.1), we would need 614 data points. Participants met at a predetermined destination (McCullough Hills Trailhead, Henderson, NV, USA) and were asked to sign a written informed consent document that was previously approved by the University of Nevada, Las Vegas Biomedical Sciences Institutional Review Board. This study was conducted in accordance with the ethical standards of the International Journal of Exercise Science (33). Body mass was then taken via digital scale (Omron HBF-516b, OMRON Corp., Kyoto, Japan), and self-reported height and mountain biking experience was recorded. Researchers then explained to the participants that they would be expected to perform two self-paced 3.22 km (two mile) mountain biking trials while donning the fitness trackers and HR monitors. There was a total of six devices worn by each participant (5 test devices and 1 criterion), two wrist worn devices (fēnix 5, Polar A360), one forearm device (Rhythm+), one earbud device (Jabra), and two chest strap devices placed as close to the xiphoid process as possible while allowing for two devices (Polar H7, Suunto) (see Table 2). Participants could ride their own mountain bike if they owned one, if not, they were provided a bike and helmet.

Table 1.

Demographic Data

Total Subjects	21
Mean Age (yrs)	26.3 ± 6.6
Mean Height (ft)	5.6 ± 0.3
Mean Height (in)	67.7 ± 3.2
Mean Height (cm)	171.8 ± 8.0
Mean Weight (lbs)	162.67 ± 41.8
Mean Weight (kg)	73.9 ± 19.0
Total Participants (Males)	10
Total Participants (Females)	10
Mean Activity Level (min/week)	351.0 ± 183.3
Total MTB Experience (Low)	17
Total MTB Experience (Moderate)	3
Total MTB Experience (High)	0

Table 2.

Device and Company Information

Brand	Device	Company Information
Garmin	fēnix® 5	Garmin Ltd., Schaffhausen, Switzerland
Jabra	Elite Sport Earbuds	Jabra, Copenhagen, Denmark
Suunto	Spartan Sport Watch + Chest HRM	Suunto Oy, Vantaa, Finland
Scosche	Rhythm+	Scosche Industries Inc., Oxnard, CA, USA
Polar	H7 Heart Rate Monitor	Polar Electro Inc., Woodbury, NY, USA
Polar	A360 Fitness Tracker	Polar Electro Inc., Woodbury, NY, USA

Company information of each device used in the current study.

The trail that was used for this study had a 48m elevation change and desert terrain consisting of dirt and rock. Participants performed the same route, twice, with 10 minutes of rest between trials. The criterion device used for this study was the Polar H7 heart rate monitor, which contains a single, flexible plastic sensor (2.4x27.9cm) worn at the level of the xiphoid process, with the strap being wrapped around the torso by an elastic band. The trail was marked with small orange yard flags for the majority of participants, however, due to operational convenience, not all participants had the trail marked. Due to this, some participants departed from the set pathway. This was anticipated, and they were instructed that they needed to go one mile, as recorded by the Garmin fēnix® 5 on their wrist, then turn around. For the second trial, they were instructed to use the same route completed previously. Approximately 3 of the 20 participants used an alternate route than the set path, but all were able to use the same route for both trials. There was a technical error with the data recording, and the data for the criterion device was not collected for four participants, leaving the available data for analysis at 16 participants and 32 trials.

Statistical Analysis

The data processing and statistical analyses performed for the current investigation was previously described (8). The data was merged via the date:time stamp. Overall, there were 35,774 data points (lines/seconds with HR) from the criterion device for possible comparison between the other devices. Granular calculations were performed in Google Sheets (Google LLC, Mountain View CA, USA), while statistical tests were done with SPSS (Version 24.0, International Business Machines Corp. [IBM], Armonk, NY, USA), and jamovi (The jamovi project [2021]. jamovi Version 1.6 [Computer Software]. Retrieved from https://www.jamovi.org). Validity was determined for each analysis via multiple statistical tests: 1. Error analysis, mean absolute percentage errors (MAPE), mean absolute error (MAE), and mean error (ME); 2. Correlation analysis, Lin’s concordance correlation coefficient (CCC) and Pearson’s correlation coefficient (r); and 3. Equivalence Testing (two one-sided t-tests [TOST-test]). Pre-established validity thresholds were: MAPE > 10%, CCC < 0.7.

RESULTS

Of the 16 participants (32 trials) with available data, the mean elapsed time for trial 1 was 20.29 ± 6.49 minutes (all values given as mean ± SD), which was 1.37 minutes slower than trial 2. The only device that had a MAPE below 10% was the Suunto Spartan Sport Watch with accompanying chest-based HR monitor. This was also the only device that had a correlation value of above 0.7. Further results, including validity measures, can be found in tables 3 and 4.

Table 3.

Time and Heart Rate Data by Trial

	Trial 1	Trial 2	Combined Mean Trial	Summed Trial
Mean Time (min)	20.29	18.92	19.61	39.21
Standard Deviation (min)	6.49	5.69	5.85	11.70
Minimum Time Elapsed (min)	11.60	9.90	10.75	21.50
Maximum Time Elapsed (min)	34.67	30.78	31.82	63.65
Mean Heart Rate (bpm)	159.63*	161.87*	160.57
Standard Deviation (bpm)	11.53	11.08	11.16

Time and mean HR data by trial with accompanying one-tailed, paired t-tests for all participants that data analysis was able to be completed on (n = 16, 32 trials). Significance was set at an alpha of 0.05. * indicates statistically significant results based on the t-test.

Table 4.

Validity Measures – Combined Data

	Polar H7	Suunto	Rhythm+	fēnix 5x	Polar A360	Jabra
Mean HR (bpm)	Chest 161.79 HRM	Chest 162.11 HRM	HR 144.50 Monitor	Watch 143.94	Watch 142.14	Earbuds 140.12
Standard Deviation	19.43	19.51	43.62	37.00	30.23	41.15
Total Data Points (bpm)	35864	35845	34851	34571	33238	7967
MAPE (%)		0.66	10.90	11.12	13.20	26.56
MAE (bpm)		1.03	18.32	18.60	21.75	43.62
Pearson Correlation		0.99	0.29	0.31	0.41	−0.32
Lin’s Concordance		0.99	0.19	0.22	0.29	−0.20
r 2		0.98	0.09	0.10	0.17	0.10

Validity measures for all data and all participants that data analysis was able to be completed on (n = 16, 32 trials). All comparisons were made against the Polar H7 HRM that was used as the criterion. Bolded values represent results that met the pre-established validity thresholds. MAPE = mean absolute percentage error. MAE = mean absolute error.

DISCUSSION

The purpose of this study was to determine the validity of several HR monitors during mountain biking. We hypothesized that all the devices would be considered valid during bouts of mountain biking. Our findings reveal that only the Suunto Spartan Sport watch with accompanying chest strap HRM displayed acceptable overall agreement with the criterion measure.

Heart rate is an important physiological variable, allowing athletes and exercise scientists the ability to measure and track intensity. It is uniquely valuable for pacing, as it can be used independent of the course. Rather than a pace given in terms of min/mi (or similar units) that will change depending on the grade or altitude of the course. Pacing according to HR allows the athletes to maintain a pace at a cardiovascular intensity that is sustainable for them (22). Having an accurate HR measurement is important for mountain biking as it contains many hills and altitude changes. It can also be used for determining overall intensity or zones of the ride for training purposes. Therefore, the data produced by the current study will be valuable for any mountain bikers, coaches, researchers, etc. who want to use HR for race, training, or other purposes.

Sensor Technology Validity and Reliability

The devices used in this study utilize two different types of technology to measure HR, photoplethysmography (PPG) and electrocardiography (ECG). Both technologies have been around for decades and have been important innovations for biosensors, fitness trackers, and wearable technology. Both technologies have been investigated to determine their validity and reliability during exercise. While ECG technology continues to out-perform PPG technology, the rapid rise in popularity of PPG devices warrants a more in-depth look at the current state of validity, possible advantages or disadvantages, and appropriate use-cases for each technology.

The Polar H7 device (criterion) and the Suunto Chest HRM both utilize ECG technology, which measures the electrical signals of the heart. Due to the nature of the technology, these sensors must be worn on the chest, and are often used as an accessory device, paired with a watch or similar device. Wireless ECG monitors have existed since the early 1980’s (39), and the technology has been shown to have high agreement with 2–12 medical grade, ambulatory ECGs in a variety of different exercise modalities and intensities (11, 17, 25, 29, 31). They have clearly shown themselves to be capable criterion devices when HR is the level of resolution needed, such as in the current study and in many other validation studies. The present investigation found that the Suunto could be considered for a criterion device as the HRM performed well.

All devices used in the present study except for the Suunto and the Polar H7 use PPG sensor technology, which uses light-based optical sensors to determine the rate of blood flow, and thereby HR. PPG technology has been around since the 1930’s and has been integrated into modern fitness-based wearable devices almost as soon as they were developed (10). Unlike the wireless chest based HRMs that utilize ECG technology, PPG technology can be placed virtually anywhere on the body, though the most common locations are the wrist, forearm, or bicep, with other devices being placed in the ear or on the head, hands, etc. (30, 38, 42). The current study utilized multiple locations on the body to place PPG devices, including both wrists, the forearm, and in the ear. This allows for greater resolution as to how the location and the means of attachment of the device impacts accuracy.

The accuracy of the PPG devices has been studied in a range of exercise modalities and intensities, similar to wireless ECG devices (27, 52). These PPG devices generally have acceptable agreement at rest (9, 38) and low intensity exercise (9, 36, 42), but tend to decrease in accuracy and performance at higher intensities (9, 19, 42). The exercise modality can also influence the performance of the device greatly (3, 9, 19, 31), as the PPG sensors are more susceptible to motion artifacts during movement when measuring the blood flow via the optical sensors than ECG sensors (12, 26). PPG devices have been tested in a range of modalities, including rest (3, 9), walking (2, 9, 19), running (3, 9, 19, 30, 42), cycling (9, 23), yoga (36), resistance training (3, 20) and many more. These findings elicit mixed results for the performance of PPG sensors in measuring HR. It appears certain brands tend to do better than others, and devices at higher price-points tend to do better as well (32) likely due to the use of higher quality sensors. As evidenced in the present study and others, the mechanics of securing the device to the body will have an important influence on the stability, and therefore validity of the device (19, 28). Devices that are secured via elastic bands have improved mechanical optimization and tend to do better, especially during high intensity exercise and exercise modalities that involve lots of movement. This is most likely due to the improved mechanical optimization that reduces motion artifacts that limit the performance of the PPG devices (26). While some remedies to the challenges of reading HR through PPG sensors have been suggested (40), it appears that until we can develop or utilize better sensors, mechanical optimization, or algorithms, these devices will continue to be outperformed by ECG monitors.

In terms of advantages, the Suunto and other wireless ECG monitors have the clear advantage of producing more accurate HR measurements compared to PPG devices. This extends to accuracy across exercise modalities and intensities. As the ECG technology reads the electrical signals of the heart, it is limited to chest-based monitors. This may be a disadvantage in certain circumstances where the lack of location placement options may influence comfort and compliance with the user. Additionally, wireless ECG sensors are often used as an accessory device to complement the fitness watch or other wearable device. This means that it will likely be more expensive and more work for the user and more work to use both devices, thus leading to lower utilization and compliance. This inconvenience in the need to use multiple devices, and decreased comfort level when compared to PPG-based HR monitors are potential disadvantages athletes, coaches, researchers and others should be aware of when deciding which devices to use.

The PPG sensors in wearable technology can be small and placed virtually anywhere on the body. This represents a major advantage of PPG sensors. They are often incorporated into other devices, such as smart watches, earbuds, phones, etc. that improve compliance. However, as has been shown previously, they have many limitations as to their potential use-cases. Accuracy of PPG sensors is not sufficient for many exercise modalities, and during high intensity exercise the accuracy tends to fall as well. This represents a major disadvantage of PPG sensors, as accuracy is likely the most important factor to consider when choosing a device. If the participant is mountain biking, no PPG monitor tested would be an appropriate choice, as none were classified as valid during mountain biking. Thus, when identifying which HRM to use, the user should consider several variables, including body placement, exercise intensity, exercise modality, comfort, cost, required accuracy (and therefore thresholds of validity), and perhaps others to identify which device would be most appropriate for the specific situation.

Measuring and Determining Validity

There are many aspects that a person seeking to use any of these devices to measure HR should consider. However, as suggested earlier, the required accuracy of the device is a crucial factor in the determination of which device to use and is dependent on the proposed use-case. Depending on the potential use-case of the device, the user may require differing levels of accuracy to measure HR. A recreational athlete may need less accuracy than a professional athlete, who may need even less than a researcher. Distinguishing between valid devices and non-valid devices may not provide enough resolution for certain cases, as there are no set validity thresholds. In fact, currently, there is no consensus upon criteria to measure accuracy and validity, and accepted thresholds to determine validity have even less consensus. While some analytical techniques have been proposed, and common tests have begun to emerge (6, 50), there is a need to standardize validity thresholds for these devices. There will likely need to be multiple thresholds for differing use cases (recreation, athletics, research, etc.). After all, if the foundation of validity studies in the field of wearable technology is to be able to determine whether a device is, in fact, valid, then without the establishment of widely accepted thresholds, the question of validity will remain largely unanswered for these devices. We have used the relatively liberal thresholds of a MAPE < 10% and CCC > 0.7 for the current study, but as has been established, this is not a universally agreed upon threshold for validity (9, 31).

Limitations

While not strictly a limitation, caution when applying this study to newer technology should be considered. As the devices used in the current study are now many years old, they do not represent any possible advancements in PPG technology that have been made recently, and validity cannot be assumed across differing models and devices. There may also be a possible limitation in the study methodology pertaining to the ability of the PPG devices in reading the HR in a timely manner (compared to the ECG devices). As the ECG devices read the electrical signals of the heart, there is no “lag” in the reading of heart rate for those devices. However, there may be a lag in the PPG devices compared to the ECG, as the change in fluid velocity within the vascular system may take a bit to register by the PPG sensors. A potential lag of even a couple of seconds could be enough to significantly alter the validity status of the PPG devices when comparing to an ECG criterion device, as could be the case in the current investigation.

Conclusion

This study assessed the validity of several HR monitors during mountain biking. Participants were asked to wear six devices (5 test devices, 1 criterion) and perform two trials of mountain biking. There was only one device that met the pre-established validity criteria, which was the Suunto Spartan Sport Watch with Chest HRM. This device may be considered valid in producing measures of HR while mountain biking. Forearm-based devices (and likely bicep-based devices) secured through elastic straps would be a better alternative to wrist-based devices if chest monitors are not available, though they were not considered valid, according to the data obtained in this investigation. Therefore, if a person would like to track their HR during mountain biking, for pacing, training, or other reasons, the devices best able to produce valid results are chest-based, wireless ECG monitors, secured by elastic straps to minimize the movement of the device, such as the Suunto chest-based HRM.