Commercial smartwatch with pulse oximeter detects short-time hypoxemia as well as standard medical-grade device: Validation study

Abstract

Objective

We investigated how a commercially available smartwatch that measures peripheral blood oxygen saturation (SpO₂) can detect hypoxemia compared to a medical-grade pulse oximeter.

Methods

We recruited 24 healthy participants. Each participant wore a smartwatch (Apple Watch Series 6) on the left wrist and a pulse oximeter sensor (Masimo Radical-7) on the left middle finger. The participants breathed via a breathing circuit with a three-way non-rebreathing valve in three phases. First, in the 2-minute initial stabilization phase, the participants inhaled the ambient air. Then in the 5-minute desaturation phase, the participants breathed the oxygen-reduced gas mixture (12% O₂), which temporarily reduced their blood oxygen saturation. In the final stabilization phase, the participants inhaled the ambient air again until SpO₂ returned to normal values. Measurements of SpO₂ were taken from the smartwatch and the pulse oximeter simultaneously in 30-s intervals.

Results

There were 642 individual pairs of SpO₂ measurements. The bias in SpO₂ between the smartwatch and the oximeter was 0.0% for all the data points. The bias for SpO₂ less than 90% was 1.2%. The differences in individual measurements between the smartwatch and oximeter within 6% SpO₂ can be expected for SpO₂ readings 90%–100% and up to 8% for SpO₂ readings less than 90%.

Conclusions

Apple Watch Series 6 can reliably detect states of reduced blood oxygen saturation with SpO₂ below 90% when compared to a medical-grade pulse oximeter. The technology used in this smartwatch is sufficiently advanced for the indicative measurement of SpO₂ outside the clinic.

Trial Registration

ClinicalTrials.gov NCT04780724

Introduction

Recently, consumer wearables have created the vision of new possibilities for personal care.^1–5 Routine monitoring of biological signals such as heart rate or sleep pattern using wearable devices is an emerging trend in health monitoring outside the clinic and in-home care with a multi-billion dollar potential.^6,7 The COVID-19 pandemic and its aftermath will only emphasize this trend.^8,9 Nevertheless, the clinical applicability of wearables must be separated from consumer curiosity.^10–14 Currently, the role of smartwatches in health care is investigated and discussed. Earlier feasibility studies focused on activity monitoring and chronic disease self-management.^15,16 Recent prospective studies have looked at the use of smartwatch technology in a range of medical applications such as the detection of atrial fibrillation,^17,18 sleep monitoring,¹⁹ post-admission recovery in pediatric patients with respiratory diseases,²⁰ monitoring women during pregnancy²¹ or pre-habilitation prior to abdominal cancer surgery.²² Several studies were also interested in using smartwatch data in the detection of viral infections such as COVID-19.^23,24 However, a previous study warned that a smartwatch did not have sufficient accuracy in measuring blood pressure or pulse oximetry compared to clinical standards.¹³

Pulse oximetry as a method of indirect measurement of peripheral blood oxygen saturation (SpO₂) is a relatively new metric in smartwatches, but it is becoming routinely available in new models,²⁵ allowing convenient SpO₂ monitoring at home or, with some restrictions due to movement, outdoors without the need for a dedicated pulse oximeter. In addition, the smartwatch's SpO₂ sensor does not need to be attached to a finger to complicate daily activities. This might be useful not only to athletes in training or mountaineers in high altitudes but more importantly to patients suffering from cardiovascular diseases, lung diseases such as chronic obstructive pulmonary disease (COPD), or dealing with the consequences or concerns of COVID-19.^26,27 In particular, the ability of smartwatches to measure SpO₂ without conscious user intervention might help to detect intermittent hypoxemia associated with sleep apnea, a chronic health disorder that results in neurocognitive dysfunction and cardiovascular problems.^28–30

Pulse oximetry is an optical method that evaluates changes in light absorption at multiple frequencies due to the oxygen content in arterial blood. Levels of SpO₂ 95% or higher are considered normal, whereas SpO₂ below 90%, even if transient, is considered clinically relevant.³¹ Standard medical pulse oximeters, including portable oximeters, use transmission pulse oximetry, in which the light sources and the photodetector are positioned on the opposite sides of the measurement site (usually a thin place such as a fingertip or an earlobe) and the light passing through the site is evaluated. Smartwatches, and other wrist-worn devices, for practical reasons, utilize reflectance pulse oximetry, in which the light sources and the photodetector are positioned on the same side of the measurement site and the light reflected into the photodetector from the tissue is evaluated. Reflectance pulse oximeters face less light absorption and thus have less power consumption, can be placed at diverse measurement locations, and the absence of moving parts increases their resistance to motion artifacts.^32,33 However, in practice, the reflectance mode can exhibit a low signal-to-noise ratio and be sensitive to ambient light sources.³⁴ At the wrist, the performance of the reflectance pulse oximeter depends on the exact placement of the sensor.^34,35 In a study with an experimental reflectance pulse oximeter system, SpO₂ measurement at the wrist showed an unacceptably large error.³⁶ Similarly, a study with a wrist-worn reflectance pulse oximeter under development found its performance was worse than finger-based oximeters and it was not able to detect hypoxemia.³⁷ Also, Hermand et al. reported a commercial smartwatch failed to provide trustworthy SpO₂ values, especially during induced oxygen desaturation.³⁸ On the other hand, a study by Lauterbach et al. tested a commercial smartwatch in a normobaric hypoxia chamber and found only minimal differences in SpO₂ measured by the smartwatch compared to a standard pulse oximeter,²⁶ with the largest difference for the lowest inspiratory oxygen fraction. Other recent studies have also reported positive results on the accuracy of wrist SpO₂ measurements by commercial devices, but most of them did not focus on hypoxia.^39–42

Thus, there are currently a few studies available that evaluate wrist SpO₂ measurement with mixed results. Concerns about measurement accuracy remain and, as new smartwatch models are launched, further studies are desirable.²⁷ A question persists whether wrist-worn devices, and smartwatches in particular, can monitor SpO₂ even in low blood oxygen levels well enough to provide early warning of desaturation episodes.

This study aims to compare the measurement of peripheral blood oxygen saturation using a very popular smartwatch to a medical-grade pulse oximeter at normal and potentially hypoxic levels.

Methods

The prospective single-arm interventional study was approved by the Ethical Review Board of the Faculty of Biomedical Engineering, Czech Technical University in Prague (No. B1/2021). The study was registered with ClinicalTrials.gov (identifier NCT04780724).

Recruitment

Twenty-four healthy student volunteers (mean ± SD: age 24 ± 2 years, height 181 ± 8 cm, mass 77 ± 11 kg) were recruited for the study. They were only included if they did not suffer from any disease of the cardiovascular system and had no injury to the upper limbs or hands that could affect the peripheral perfusion. In addition, participants were excluded for pregnancy, diabetes, hypotension, hypertension, acute asthma or any other acute respiratory disease. None of the participants used nail polish or had false nails at the time of the measurement. Participants were required to stay at least 30 min at rest before entering the laboratory. All participants provided written informed consent before their enrollment into the study.

Experiment setup and protocol

Upon arrival at the laboratory, Apple Watch Series 6 (Apple Inc., Cupertino, CA, USA)—further referred to as the smartwatch—was placed on a participant's left wrist and the sensor of a medical-grade pulse oximeter Radical-7 (Masimo Corp., Irvine, CA, USA)—further referred to as the oximeter—was attached to the left middle finger of the participant. During the experimental procedure, SpO₂ readings were taken by hand from the smartwatch and oximeter simultaneously. Participants were sitting at rest throughout the experiment, and they were asked to keep their hands still on the table with their wrist and palm down and flat and avoid any movement according to the instructions of the smartwatch manufacturer.

A simple breathing circuit with a three-way non-rebreathing valve was assembled for the experiment. It allowed the participant to inhale the hypoxic gas mixture (12% O₂) from a polyethylene Douglas bag or the ambient air and to exhale into the ambient air outside the Douglas bag. The gas composition was monitored continuously by a Datex Ohmeda S/5 patient monitor (Datex-Ohmeda Inc., Madison, WI, USA) with a sensor placed between the three-way valve and the participant. A disposable antibacterial filter separated the participant from the breathing circuit.

There were three phases of the experimental procedure. During the first 2 min, in the initial stabilization phase, participants inhaled the ambient air via the breathing circuit. Two SpO₂ readings were taken (times 0:45 min and 1:15 min of the experiment). Then, in the 5-minute desaturation phase, participants inhaled the hypoxic gas mixture from the Douglas bag. Readings of SpO₂ were taken every 30 s (from time 2:45 min to time 6:45 min of the experiment). The final stabilization phase followed when the participants inhaled the ambient air and SpO₂ was recorded every 30 s (from time 7:30 min) until SpO₂ returned to normal values. Typically, three or four readings were taken in the final stabilization phase. Each participant underwent the experimental procedure twice. There was a delay of a minimum of 1 h between the two iterations of the experimental procedure to address possible slow washout of test gas.

Data processing and analysis

We concluded that the number of participants enrolled in the study and the number of paired SpO₂ observations would meet the basic recommendations of the Food and Drug Administration and the International Organization for Standardization (ISO 80601-2-61) for study design for in vivo accuracy testing of pulse oximeters (10 or more healthy subjects, 200 or more paired measurements).^43,44

We used the Bland–Altman analysis to compare the agreement between simultaneous smartwatch and oximeter SpO₂ measurements. The Bland–Altman analysis looks at two parameters, the bias and 95% limits of agreement. The bias is quantified as the mean difference in the paired measurements. The 95% limits of agreement, calculated as the mean difference ± 1.96 standard deviations, determine the range of expected difference in future simultaneous smartwatch and oximeter measurements. Uncertainties in the estimates of the bias and 95% limits of agreement are expressed as 95% confidence intervals. The standard deviation was calculated using the modified Bland–Altman method for multiple observations per individual when the measured quantity changes over the period of observation.⁴⁵ In addition, we evaluated the root mean square difference between smartwatch and oximeter paired measurements as

$$A_{\mathrm{rms}}=\sqrt{{\displaystyle \frac{\sum {(\mathrm{Sp}{\mathrm{O}}_{2,\mathrm{smartwatch}}-\mathrm{Sp}{\mathrm{O}}_{2,\mathrm{oximeter}})}^2}{n}}}$$

where n is the number of evaluated pairs of SpO₂ measurements.⁴⁴

Further, the differences in the smartwatch and oximeter measurements were evaluated with respect to study time, that is, to evaluate the relative response rate of the two devices. To do this we averaged the measurements of all participants at each study time for the smartwatch and for the oximeter. The mean SpO₂ values across all participants and iterations of the experimental procedure were used to graphically compare the average time courses of the pooled smartwatch data and the pooled oximeter data. A two-tailed paired t test was used to evaluate the statistical difference between the smartwatch data and the oximeter data at each measurement time. P value less than 0.05 was considered statistically significant. Only the observations, where simultaneous readings from both devices were available, were included in the analysis. All data were analyzed in Matlab 2021a (MathWorks, Natick, MA, USA) after transcription from the log.

Results

Agreement between devices

The study was conducted in the Laboratory of special equipment for ICU of the Czech Technical University in Prague, Department of Biomedical Engineering, Kladno, Czech Republic, during February and March 2021 at an altitude of 405 m (1330 ft). All 24 volunteers (five women and nineteen men, all Caucasian, aged 20–28 years) completed the experiment with two iterations of the experimental procedure and two measuring devices, so there were 48 series of paired measurements available. As in some cases, one of the devices did not provide a valid reading, there were 1284 valid paired readings in total out of a possible number of 1364. The SpO₂ readings ranged between 76% and 100%. Most (75%) were between 90% and 100%, 24% between 80% and 89% and 1% below 80%.

The presented Bland–Altman plot is based on 642 individual data points calculated from all complete pairs of pooled SpO₂ readings (Figure 1). The bias (mean difference) in SpO₂ between the smartwatch and oximeter was 0.0% for all the data points. The 95% confidence limits of the bias were −0.2% and 0.3%, indicating that there was no statistically significant bias between the measuring devices. The 95% limits of agreement were estimated to be −5.8% and 5.9%. The most extreme individual differences between the smartwatch and oximeter SpO₂ measurements were −9% and 17%. The A_rms evaluated across the pooled SpO₂ readings was 3.0%. The same approach was used to analyze the data after splitting into SpO₂ 90%–100% and SpO₂ less than 90%. The results are summarized in Table 1. As shown, the absolute bias was greater for SpO₂ measurements under 90%.

Figure 1.

Differences between simultaneous SpO₂ readings of the smartwatch (Apple Watch 6) and oximeter (Masimo Radical-7) across different ranges of oxyhemoglobin saturation. Pooled SpO₂ measurements were analyzed for all participants grouped. The solid line is the mean difference of the measurements (bias). Dashed lines are the 95% limits of agreement. The area of markers is proportional to the number of measurements.

Table 1.

Comparison of measurement bias and agreement.

SpO2a, %	Biasb (95% CI), %	Lower LOA (95% CI), %	Upper LOA (95% CI), %	Arms, %
Entire range	0.0 (−0.2 to 0.3)	−5.8 (−6.2 to −5.4)	5.9 (5.5–6.3)	3.0
<90	1.2 (0.7 to 1.7)	−5.3 (−6.1 to −4.4)	7.6 (6.7–8.4)	3.4
90–100	−0.3 (−0.6 to 0.1)	−5.8 (−6.2 to −5.4)	5.1 (4.7–5.5)	2.8

^a [(smartwatch + oximeter)/2].

^b [smartwatch − oximeter].

LOA: 95% limits of agreement.

Average response of devices

The time series of average smartwatch and oximeter measurements show the absolute differences between the means of SpO₂ measurements were small (Figure 2). The difference between the means of the smartwatch and oximeter ranged from −0.64% (study time 195 s) to 0.74% (study time 480 s) with the minimum absolute difference of the means 0.22% (study time 450 s). None of the differences between paired smartwatch and oximeter measurements at any study time reached a statistically significant difference.

Figure 2.

The time courses of the mean of all smartwatch SpO₂ measurements (Apple Watch 6) and the mean of all oximeter SpO₂ measurements (Masimo Radical-7) across all 24 participants. Data are mean ± SEM.

Discussion

Principal results

The main finding of our study is that SpO₂ measurement by Apple Watch Series 6, a consumer product, did not differ on average from SpO₂ measurement by Masimo Radical-7 pulse oximeter, a medical device. The average absolute difference or bias between smartwatch and oximeter SpO₂ measurements, evaluated for all pooled data, in two ranges and at the individual study times, was less than 1% SpO₂. This is the resolution in which the SpO₂ values are displayed on both devices.

At low-oxygen levels, the smartwatch tended to measure higher SpO₂ values than the oximeter, and this difference averaged approximately 1% SpO₂ for readings less than 90%. The time chart (Figure 2) illustrates a very similar response of both devices for the “average patient,” with the average difference between SpO₂ reported by the smartwatch and oximeter at the end of the desaturation phase being only 0.26%, and −0.23% upon recovery. The time series in Figure 2 also suggests that the response of the smartwatch to sudden desaturation may be slower than the response of the oximeter. The smartwatch required a 15-s period for a single SpO₂ measurement compared to the 2–4-s averaging time of the oximeter, so the smartwatch reading lagged behind the oximeter readings during the continuous SpO₂ decrease. This may have contributed to the higher average SpO₂ measured by the smartwatch during induced desaturation. Generally, there are differences between the reaction times of pulse oximeters to sudden hypoxia.⁴⁶ During the experiments, we also observed a faster return of smartwatch values than oximeter in the final stabilization phase after the desaturation phase, but not being the primary concern of our study, there were not enough data to evaluate for this.

Comparison with prior work

Several studies have evaluated smartwatches in hypoxemia. In their analysis, Lauterbach et al. compared a different smartwatch Garmin fēnix® 5X Plus (Garmin, Olathe, KS, USA) with a medical-grade pulse oximeter Model 7500 (Nonin Medical BV, Amsterdam, the Netherlands) in a customized chamber that allowed to change and maintain the inspiratory oxygen fraction. Twenty-three volunteers breathed a gas mixture under normobaric conditions with inspiratory oxygen fractions between 14% and 21%. The study reported SpO₂ bias (smartwatch−oximeter) only 0.7%–0.8% for higher values of the inspiratory oxygen fraction, but 3% for the smallest inspiratory oxygen fraction. Two explanations were offered for the bias increase by the authors of the study; first, elevated PaCO₂ levels resulting in increased other hemoglobin derivatives in the bloodstream, and second, hypoxia-mediated vasoconstriction that altered blood flow in fingers compared to the wrist.²⁶ Hermand et al. compared a smartwatch from the same manufacturer (Garmin Forerunner 245) with a medical-grade oximeter on 10 healthy participants during normoxia and normobaric hypoxia when the inspiratory oxygen fraction was gradually reduced to 10.5%. The total observed bias of the smartwatch was 5.4%, and the bias for the lowest oxygen fraction was even 13.2%. The authors concluded the smartwatch was not a reliable alternative to medical-grade oximeters.³⁸ A study with another smartwatch (Withings ScanWatch) by Kirszenblat and Edouard reached opposite findings. Measurements of SpO₂ in 14 healthy participants were compared with arterial blood oxygen saturation (SaO₂) determined with a co-oximeter at various stable levels of oxygen saturation. The total bias found was 0.98% (right wrist) and 1.56% (left wrist), and overall accuracy was adequate to medical-grade oximeters.⁴⁰ Our results, i.e., the negligible bias at higher saturation and the small bias with decreased saturation, generally correspond to those of Lauterbach et al. and Kirszenblat and Edouard although we detected a smaller bias for lower inspiratory oxygen fraction (12% in our study vs. 14%) and somewhat lower measured SpO₂ values than Lauterbach et al. We also suggest that the differences reflect different devices used in the studies.

Two recent studies examined the SpO₂ measurement using Apple Watch Series 6 compared to medical-grade pulse oximeters.^41,42 The studies on subjects at rest included both healthy participants and diseased participants with lung or cardiovascular diseases. Both studies reported a bias (smartwatch−oximeter) of less than 1% and no significant differences between subject groups (healthy or diseased). However, neither of the two studies induced hypoxemia in the subjects, and they contained very few SpO₂ measurements below 90%.

The differences between Apple Watch Series 6 and Masimo Radical-7 within 6% SpO₂ can be expected for individual measurements for SpO₂ readings 90%–100% and up to 8% for SpO₂ readings less than 90%. This again is consistent with Lauterbach et al. and Kirszenblat and Edouard who reported 95% limits of the agreement up to 8.6% and 6.6%, respectively. The differences in individual SpO₂ measurements between the smartwatch and oximeter are also similar to what was reported as differences in individual SpO₂ measurements against direct measurements of SaO₂ by co-oximetry under progressive normobaric hypoxia. The 95% limits of agreement reported by Kolb et al. were (−6.5%, 5.6%) and (−7.6%, 9.8%) for SpO₂ finger measurements when SaO₂ was above 85% and under 85%, respectively.⁴⁷ Others also reported individual readings may differ as much as 6%.⁴⁸ In a more recent study, narrower 95% limits of agreement (−1.8%, 1.8%) were reported by Louie et al. for a nonmotion SpO₂ measurement when SaO₂ was above 90%.⁴⁹

The root mean square difference is the standard metric for assessing accuracy in pulse oximetry that combines bias and precision of the SpO₂ measurement when compared to co-oximetry. Accuracy better or equal to 4.0% SpO₂ is required in general.⁴⁴ Typically, A_rms ≤ 3.0 and A_rms ≤ 3.5 are expected for transmittance and reflectance sensors, respectively.⁴³ Medical-grade oximeters have an accuracy of 2%–3% according to manufacturers or 3%–4% according to what was reported in clinical studies.^32,50 Numerous studies however reported that the accuracy of pulse oximeters deteriorates as blood oxygen saturation decreases.^47,49,51,52 The A_rms metric has also been utilized when comparing SpO₂ measurements. Verkruysse et al. compared contactless photoplethysmography with a median of measurements taken by standard pulse oximeters in healthy adults under normoxic conditions and also hypoxic conditions where the inspiratory oxygen fraction was about 15%.⁵³ They estimated A_rms ≤ 2.5% for short-time segments and even A_rms ≤ 1.7% when discarding short-time errors. Hahnen and her colleagues investigated the accuracy of a handheld portable device for vital sign measurements on 85 participants and reported A_rms was 3.1% for SpO₂ when compared to a medical-grade vital signs monitor.¹³ In this context our results, A_rms < 3.0% for saturation of 90% and greater and A_rms < 3.5% for saturation under 90%, seem within the expected range with some of the 6%–8% span likely attributable to the Masimo device. Even with the large uncertainty between paired measurements, the smartwatch seems reliable in detecting relevant drops in SpO₂ below 90% even of short duration.

Limitations

Our study has numerous limitations. The study included only healthy young volunteers and short-time desaturation induced by the low-oxygen level of the inhaled gas mixture. The results could be different in the case of chronic elderly patients with very long or extreme desaturations. However, the contribution of wearables for such patients in real-world situations will not be a detailed analysis of the severity of the condition, but rather a warning of an aggravated trend in the chronic problem or a sudden major change. Our results suggest that SpO₂ monitoring using wearables could be, due to its ability to detect the magnitude and speed of desaturation, a useful tool in self-care outside the clinic.

We did not evaluate SaO₂ in our study as this would require arterial blood sampling and greatly complicate the experiment. It was demonstrated that SpO₂ overestimates saturation compared to SaO₂.^52,54,55 Due to the inaccessibility of actual SaO₂ values, we chose 12% O₂ and the 5-minute duration of the desaturation phase as the limit to avoid a frequent decrease of SpO₂ below 80% and prevent transient cognitive effects that may be associated with deep hypoxia.⁵⁶ The reduced oxygen fraction we used under normobaric conditions corresponds approximately to the partial pressure of oxygen at an altitude of 4400 m and the results of our study may therefore not be applicable to areas of higher altitude or to SpO₂ below 80% in general.

The steady decline of the SpO₂ levels at the end of the desaturation phase (Figure 2) suggests that the desaturation phase needed to be extended to reach the plateau. This may have better explained whether there was some time delay in the smartwatch readings compared to the oximeter readings. Nevertheless, our focus was primarily on whether the smartwatch can provide an alert of the same quality as repeated SpO₂ measurements with a medical-grade pulse oximeter and thus be a useful screening method for detecting hypoxia.

Finally, in our study, we used one type of smartwatch from a single manufacturer. This must be considered when generalizing our observations to other smartwatches in the rapidly evolving market. Smartwatches from other manufacturers may show differences in performance, even if they use the same principle of reflectance pulse oximetry, as several hardware and software factors can affect the PPG signal, including the geometry of the light emitter and light detector or denoising.⁵⁷ Smartwatch performance may also vary between users at rest and while active. We measured participants at rest, as required by the manufacturer. The results may not correspond to measurements during or just after sporting activities due to motion artifacts, which could be the subject of further study.

Future perspectives

The availability and convenience of measuring biological signals using wearable devices such as smartwatches offer the potential to expand patient care options in chronic disease management. The clinical standard so far has been isolated measurements under the supervision of health professionals, which are taken with a relatively large time lag and then compared with the prevalence of the clinically relevant events in the population. Wearables allow long-term and continuous monitoring of trends or, on the contrary, detection of abnormal fluctuations in individuals^9,11,58 and thus more quickly assess the change in their health status over time. Wearables are not intended to replace medical devices, but they need sufficient accuracy to provide an approximate assessment of an individual's condition.⁵⁹ The risk is both overreacting to clinically irrelevant fluctuations in monitored signals and neglecting serious changes related to real health complications.¹⁴ In particular, while portable pulse oximeters with transmission technology have been shown to be comparable to patient monitors,⁶⁰ data contradict SpO₂ measurement with commercial smartwatches as this feature is relatively new. The results of our study are intended to help fill this gap. They suggest that smartwatch technology for measuring SpO₂ has matured enough to be considered part of patient care. This can help detect hidden, but potentially serious problems such as sleep apnea, which is a growing problem with possible cognitive impacts,⁵⁶ or in the early detection of acute exacerbations of chronic conditions such as COPD.¹⁴ We further suggest that the exact requirement of each of these potential health care applications need to be articulated and wearable devices evaluated against those requirements.

Conclusions

Apple Watch Series 6, as a representative of wearables, provides reliable SpO₂ values as compared to a medical-grade pulse oximeter, at both normal oxygen levels and induced desaturation with SpO₂ below 90%. The SpO₂ monitoring technology used in this smartwatch is sufficiently advanced for the indicative measurement of SpO₂ outside the clinic and can detect states of reduced blood oxygen saturation.