Assessment of Noninvasive Oxygen Saturation in Patients With COPD During Pulmonary Rehabilitation: Smartwatch versus Pulse Oximeter

Abstract

BACKGROUND:

Patients with COPD can have hypoxemia; hence, monitoring peripheral S_pO2 during pulmonary rehabilitation is recommended. This study aimed to examine the accuracy of S_pO2 readings in patients with COPD as measured by wearable devices at rest and after physical exercise.

METHODS:

Thirty-six participants with COPD (20 women), ages 52–89 years, participated in this cross-sectional study. Oxygen saturation was concurrently measured by using the Contec Pulse Oximeter CMS50D as a comparator, and the Apple Watch Series 7 and the Garmin Vivosmart 4 at rest and immediately after the 30-s sit-to-stand test and the 6-min walk test (6MWT).

RESULTS:

For the Apple Watch, the root mean squared error showed a deviation of 3.5% at rest, 4.1% after the 30-s sit-to-stand test, and 3.9% after the 6MWT. The level of agreement was 2.8 ± 2.4 (7.6, –1.9) at rest, 3.1 ± 2.8 (8.6, –2.3) after the 30-s sit-to-stand test, and 2.8 ± 2.9 (8.6, –2.9) after the 6MWT. For the Garmin Vivosmart, the root mean squared error showed a deviation of 3.3% at rest, 6.1% after the 30-s sit-to-stand test, and 5.4% after the 6MWT. Level of agreement was 1.9 ± 2.7 (7.2, –3.3) at rest, 2.9 ± 5.4 (13.5, –7.7) after the 30-s sit-to-stand test, and 2.3 ± 5.0 (12.1, –7.4) after the 6MWT. The limits of agreement showed considerable measurement variance and a tendency for the devices to be less accurate at lower saturation levels.

CONCLUSIONS:

The Apple Watch Series 7 and Garmin Vivosmart 4 overestimated S_pO2 in participants with COPD when S_pO2 was < 95% and underestimated oxygen saturation when saturation was > 95%. These findings suggest that wearable devices should not be used to monitor oxygen saturation during pulmonary rehabilitation.

Introduction

COPD is a heterogeneous lung condition characterized by chronic respiratory symptoms (dyspnea, cough, sputum production, exacerbations) due to abnormalities of the airways (bronchitis, bronchiolitis) and/or alveoli (emphysema) that cause persistent, often progressive, air-flow obstruction.^1,2 COPD is a progressive disease characterized by a gradual health decline and is a common cause of mortality and morbidity.³ The symptoms of COPD make engagement in physical activity unpleasant, hence people with COPD are generally less physically active compared with the general population.⁴ This may cause a vicious circle whereby physical activities are avoided, which causes muscle de-conditioning, which further reduces the capacity to engage in physical activity.⁵ There is evidence that fear of breathlessness contributes to reduced physical activity levels and reluctance to participate in physical rehabilitation.⁶

Pulmonary rehabilitation is an essential component of the management of COPD. Pulmonary rehabilitation improves health-related quality of life and may reduce mortality.⁷ Pulmonary rehabilitation has proved effective at all stages of COPD. However, the efficacy of pulmonary rehabilitation is challenged by poor uptake, and low adherence is a considerable barrier to effective rehabilitation.⁸ Exercise at moderate intensity (a perceived exertion on the 10-point Borg scale of 4 to 6) is needed to induce physiologic changes, but this is often not possible in patients with COPD due to dyspnea or fear of breathlessness.⁹ Exercise-induced desaturation is common, and close to half of the patients with COPD referred to pulmonary rehabilitation will desaturate during exercise (peripheral S_pO2 < 90%).¹⁰ It is generally recommended not to let the S_pO2 fall < 88–90% during pulmonary rehabilitation to minimize the risk of exercise-induced hypoxemia.^9,11 It, therefore, is recommended to monitor S_pO2 during pulmonary rehabilitation to guide exercise intensity.⁷ It has become standard practice to spot check S_pO2 by using noninvasive finger pulse oximetry during pulmonary rehabilitation.¹²

Wearable devices are increasingly used for monitoring digital health information, and can benefit by increasing motivation and engagement in physical activities in both patient and healthy populations.^13,14 Wearable devices show moderate-to-high agreement with comparator devices for biological data such as heart rate, step count, and sleep monitoring.^15,16 There is emerging evidence that health-care professionals increasingly use health information collected by wearable devices to aid diagnosis and treatment.¹⁷ The latest wearable devices can now perform on-demand S_pO2 measures added to continuous background S_pO2 measures during the day and night. However, there is a paucity of evidence on the validity of S_pO2 measures made by wearable devices for patients with COPD. This study aimed to examine the accuracy of pulse oximetry measured by wearable devices in patients with COPD at rest and immediately after physical activity. We hypothesized that the wearable devices would show high agreement with comparator measures at rest and moderate-to-high agreement after physical activity.

QUICK LOOK.

Current knowledge

Wearable devices are increasingly used to monitor health information and to guide treatment and rehabilitation. There is emerging evidence that wearable devices may accurately measure S_pO2 in patients during rest. However, the accuracy of S_pO2 measures performed by wearable devices in patients with COPD during physical activity has not been assessed.

What this article contributes to our knowledge

Wearable devices did not accurately measure peripheral oxygen saturation in participants with COPD. Due to considerable measurement variability, wearable devices should not be used to measure peripheral oxygen saturation in patients with COPD.

Methods

In this cross-sectional study, S_pO2 was assessed in a single session for each participant to examine the accuracy of S_pO2 measured by 2 wearable devices in patients with COPD when at rest and immediately after physical activity. Assessments were conducted in a controlled clinical setting in 2 municipal rehabilitation centers during regular pulmonary rehabilitation (Aalborg and Vesthimmerland Municipalities).

Participants

Patients > 18 years of age diagnosed with COPD who were attending pulmonary rehabilitation in the 2 municipal health centers were invited to participate in this study. The clinical staff (eg, physiotherapists at the municipal health centers) conducted the recruitment process. The staff conducted face-to-face recruitment with all patients eligible for inclusion. Exclusion criteria were as follows: (1) missing an upper extremity, such as a hand or a finger; (2) clinical instability or known co-morbidity that would prevent the participants from safely performing the study protocol; and (3) cognitive deficits that might have interfered with the activities in this study.

The required sample size was not selected a priori. However, a sample size of 30 - 45 participants was considered sufficient.^18,19 Therefore, as many patients as possible were recruited from the 2 municipal health centers. Recruitment was terminated when the research pool closed (eg, when rehabilitation activities were concluded). Informed consent was obtained from all the participants included in the study. The study complied with all the relevant national regulations and institutional policies, and was performed by following the tenets of the Helsinki Declaration. The protocol was reviewed by the North Denmark Regional Committee on Health Research Ethics and deemed exempt from ethics approval.

Procedures

Participants performed a standardized exercise protocol in a controlled clinical setting where S_pO2 was measured during seated rest and immediately after the 30-s sit-to-stand test and 6-min-walk test (6MWT). The activities were selected to represent activities commonly used in pulmonary rehabilitation to assess physical capacity.²⁰ The Medical Research Council dyspnea scale was used to grade participants' breathlessness according to the level of exertion required.²¹ Each participant's skin type was assessed by using the Fitzpatrick skin type scale (1–6) on the dorsal aspects of the forearms and wrists.²² Type 1 is ivory, whereas type 6 is very dark brown.

The activities, presented in experimental order, with a standardized transition time of 3 min between activities, were as follows: (1) rest: the participants were seated on a chair, resting their arms on their lap; they were instructed to sit upright, with their wrists and palms facing down, and to be still during measurements; (2) the participants transitioned to the 30-s sit-to-stand test; they were instructed to stand up from the chair, straighten their knees and sit back down as many times as possible for 30 s by following previous procedures²³; and (3) the participants then transitioned to the 6MWT; they were instructed to walk as far as possible for 6 min by following previous procedures.²⁴

Instrumentation

The Contec Pulse Oximeter CMS50D (Contec Medical Systems, Qinhuangdao, Hebei, China), a standard finger pulse oximeter, was used as a comparator measure. The oximeter uses photoplethysmography technology, whereby green light emitting diode sensors measure the amount of light refracted in the blood vessels to calculate S_pO2. It is accurate to within less than ±2% in the range of values from 70% to 99% oxygen saturation.^25,26 Several wearable devices that could provide the relevant S_pO2 outcome metrics were available at the time of the study. The following selection criteria were used to choose the most appropriate devices: (1) ability to perform on-demand S_pO2 measures; (2) performance accuracy in related biomedical data, such as heart rate and step count; (3) usability; and (4) different price ranges.

The Apple Watch Series 7, OS 8.51 (Apple, Cupertino, California) was chosen due to its high agreement with comparator devices for biological data, such as heart rate, step count, and sleep monitoring.^15,16 The Garmin Vivosmart 4 (Garmin International, Olathe, Kansas) was chosen for its usability, acceptable accuracy in related biological data, such as heart rate and step count,²⁷ and low cost. Both devices use green light emitting diode photoplethysmography technology to calculate S_pO2. The participants were fitted with the wearable devices and the pulse oximeter according to the manufacturer's guidelines. The devices were individualized for age, sex, and anthropometrical data because all the participants wore the devices. The Apple Watch and Garmin Vivosmart were placed on either the right or the left wrist by using a block randomized configuration by following previous procedures.^28,29 Before the measurements, the participants were familiarized with the testing procedures. Correct placement of all 3 devices was ensured by performing a test measurement.

Movement artifacts can affect the validity of measurements when using wearable devices and according to the manufacturer’s specifications, the S_pO2 measurements must be performed when the patient is stationary. The S_pO2 measurements were made once during rest and immediately after the 30-s sit-to-stand test, and 6MWT, and were performed in the same seated position. The participants sat comfortably on a chair with their hands placed on their lap. The participants were instructed to avoid physical movements during the measurements. The resting S_pO2 measurement was conducted after 5 min of seated rest. S_pO2 from the wearable devices and the corresponding S_pO2 measured by the oximeter were noted by 2 researchers. One researcher (A. H. Graversen) noted the S_pO2 from the Garmin Vivofit, and one (S. H. Pettersen) noted the S_pO2 from the Apple Watch. The participants were blinded to the results of the measurements.

This study aimed to examine the accuracy of ${\text{S}}_{{\text{pO}}_{\text{2}}}$ measured by wearable devices in participants with COPD when at rest and immediately after physical activity. However, a decrease in ${\text{S}}_{{\text{pO}}_{\text{2}}}$ values during physical exercise will gradually normalize when the patient rests. Recent evidence shows that erroneous ${\text{S}}_{{\text{pO}}_{\text{2}}}$ measures are widespread.³⁰ Therefore, a maximum of 3 measurement attempts or a total of 90 s from each activity's completion allowed a measurement to be accepted for analysis.

Statistical Analyses

The data were analyzed by using SPSS 27 (SPSS, Chicago, Illinois). The normality of the data were assessed by using the Shapiro-Wilks test. Mean absolute percentage error values were calculated as the average absolute value of the errors of each device relative to the comparator measures. Limits of agreement (Bland-Altman plot) were calculated to determine the agreement of measurements between the devices and the comparator measure. Bias assessment parameters included the root mean square error (an overall performance measure used by the FDA), which was calculated as the square root of the mean difference between the Apple Watch and Garmin Vivismart and the comparator Contec Pulse Oximeter CMS50D squared. The following formula was used for calculation³¹:

$$\text{root mean square error}=\sqrt{\frac{{{\displaystyle \sum }}_{\text{i}=1}^{\text{n}}*{({\text{S}}_{{\text{pO}}_2}-{\text{S}}_{{\text{pO}}_2})}^2}{n}}$$

Values were calculated and expressed as a percentage to assess the accuracy of the Garmin Vivosmart and the Apple Watch compared with the comparator Contec Pulse Oximeter CMS50D. An alpha level of .05 was defined for the statistical significance of all the tests.

Results

A total of 48 patients were screened for enrollment in this study. One patient had a co-morbidity that prevented participation. Two patients had cognitive deficits that interfered with participation, and 9 were not interested in participating. After screening, 36 patients (20 women) were recruited for this study. Among the participants, age ranged from 52 to 89 years (mean 72.3 ± 7.9 years), height ranged from 1.52 to 1.87 m (mean 1.7 ± 9.6 m), weight ranged from 47 to 138 kg (mean 81.1 ± 23.6 kg), and body mass index ranged from 17.3 to 45.9 kg/m² (mean 28 ± 6.7 kg/m²).

The participants’ Medical Research Council (MRC) dyspnea scale grades ranged from 2 to 4, with 13 participants graded at MRC dyspnea scale score of 2; n = 12 participants were graded at MRC dyspnea scale score of 3; and 11 participants were graded at MRC dyspnea scale score of 4. Two participants used supplemental oxygen during the study procedures. All the participants were white, and the participants’ skin types were distributed from phototypes 1 to 3 on the Fitzpatrick scale. Before the definitive analyses, the data were checked for statistical outliers, and scatter plots were derived to illustrate the closeness between the methods. Mean ± standard deviation ${\text{S}}_{{\text{pO}}_{\text{2}}}$ (minimum – maximum) for the Apple Watch, the Garmin Vivosmart, and the comparator for all activities are summarized in Table 1.

Table 1.

${\text{S}}_{{\text{pO}}_{\text{2}}}$ Values as Measured by the Apple Watch, the Garmin Vivosmart and the Comparator Measure With Participants at Rest and Immediately After the 30-s Sit-to-Stand Test and the 6MWT

Limits of agreement and mean absolute percentage errors for the Apple Watch and the Garmin Vivosmart compared with the comparator measure are summarized in Table 2. For the Apple Watch, the root mean square error was of 3.5% at rest, 4.1% after the 30-s sit-to-stand test, and 3.9% after the 6MWT. For the Garmin Vivosmart, it was 3.3% at rest, 6.1% after the 30-s sit-to-stand test, and 5.4% after the 6MWT. Limits of agreements and mean absolute percentage errors indicated that both devices overestimated ${\text{S}}_{{\text{pO}}_{\text{2}}}$ during all activities compared with the comparator measure (Table 2). For the Apple Watch, we found biases of 2.8% during rest, 3.8% after the 30-s sit-to-stand test, and 2.8% after the 6MWT. For the Garmin Vivosmart, we found biases of 1.9% during rest, 2.9% after the 30-s sit-to-stand test, and 2.3% after the 6MWT (Table 2).

Table 2.

Limits of Agreement (Bland-Altman outcomes) and Mean Absolute Percentage Error for ${\text{S}}_{{\text{pO}}_{\text{2}}}$ Measures at Rest and Immediately After the 30-s Sit-to-Stand Test and 6MWT

However, visual inspection of the Bland-Altman plots revealed that the Apple Watch tended to overestimate ${\text{S}}_{{\text{pO}}_{\text{2}}}$ after the 30-s sit-to-stand test and the 6MWT when ${\text{S}}_{{\text{pO}}_{\text{2}}}$ was < 95%. The Bland-Altmann plots revealed that this discrepancy increased when ${\text{S}}_{{\text{pO}}_{\text{2}}}$ declined. However, the Apple Watch tended to underestimate ${\text{S}}_{{\text{pO}}_{\text{2}}}$ after the 30-s sit-to-stand test and the 6MWT when ${\text{S}}_{{\text{pO}}_{\text{2}}}$ was > 95%. Again, the Bland-Altman plots showed that this discrepancy increased when ${\text{S}}_{{\text{pO}}_{\text{2}}}$ rose. During rest, the Apple Watch estimated ${\text{S}}_{{\text{pO}}_{\text{2}}}$ with no apparent tendency (Fig. 1). The Garmin Vivosmart consistently overestimated ${\text{S}}_{{\text{pO}}_{\text{2}}}$ when ${\text{S}}_{{\text{pO}}_{\text{2}}}$ was < 95%. The Bland-Altmann plots revealed that this discrepancy increased when ${\text{S}}_{{\text{pO}}_{\text{2}}}$ declined. However, the Garmin Vivosmart also underestimated ${\text{S}}_{{\text{pO}}_{\text{2}}}$ when ${\text{S}}_{{\text{pO}}_{\text{2}}}$ was > 95%. Again, the Bland-Altman plots showed that this discrepancy increased when ${\text{S}}_{{\text{pO}}_{\text{2}}}$ rose (Fig. 1).

Fig. 1.

Bland-Altman plots. Limits of agreement for the Apple Watch Series 7 (A, C, and E) and the Garmin Vivosmart 4 (B, D, and F) with the participant at rest (A and B) and after the 30-s sit-to-stand test (C and D), and the 6-min walk test (E and F).

Of the 2 devices, the Apple Watch demonstrated the highest level of erroneous measurements during the study (Table 3). The results showed that, after the 30-s sit-to-stand test and the 6MWT, the Apple Watch performed erroneous measurements in almost 50% of the participants. The Garmin Vivosmart performed fewer erroneous measurements than the Apple Watch (Table 3). However, we observed a considerable measurement latency for the Garmin Vivosmart after the 30-s sit-to-stand test and the 6MWT, with measurement times that ranged from 15 to 90 s.

Table 3.

Distribution of Erroneous Measurements Made During the Protocol (N = 36)

Discussion

This study examined the accuracy of pulse oximetry measured by the Apple Watch Series 7 and the Garmin Vivosmart 4 in the participants with COPD when at rest and immediately after physical activity. To our knowledge, this is the first study to examine the ${\text{S}}_{{\text{pO}}_{\text{2}}}$ accuracy of 2 wearable devices to an approved oximeter during rest and after physical exercise in patients with COPD in a clinical setting. In contrast to our expectations, the main findings showed low-to-moderate agreement between the Apple Watch Series 7 and the Garmin Vivosmart 4 compared with the comparator measure during rest or after physical exercise. This finding contradicts previous studies, which suggested that the Apple Watch Series 6 has a high agreement with pulse oximeters at rest in healthy populations and patients with lung disease.^30,32 This unexpected result, however, may be related to several factors. Spaccarotella et al³⁰ and Pipek et al³² based their findings on a combined analysis of healthy populations and patients with lung and cardiovascular disease. Although Pipek et al³² performed a subgroup Bland-Altman analysis (interstitial lung disease, COPD, and healthy participants), they did not observe significant differences between the Apple Watch and a pulse oximeter. Still, visual inspection of their Bland-Altman plots shows notable differences in measurement variability and limits of agreement between healthy and patient populations, that are similar in proportions to the finding in the present study.

In the present study, 2 researchers (AHG and SHP) were tasked with making the ${\text{S}}_{{\text{pO}}_{\text{2}}}$ measures to ensure that the ${\text{S}}_{{\text{pO}}_{\text{2}}}$ values from each wearable device and the pulse oximeter were taken simultaneously. This is in contrast to previous procedures used by Spaccarotella et al,³⁰ when measurements with the Apple Watch and the pulse oximeter were taken within 1 min of each other. Another possible explanation may be that, in contrast to the present study, previous studies made several ${\text{S}}_{{\text{pO}}_{\text{2}}}$ measurements and used the average values for analysis.^30,32 In contrast to earlier findings, we experienced a multitude of erroneous measures and extensive measurement latency after physical activity. Consequently, this led to increased time between the task completion and measurement time, which thus allowed for the ${\text{S}}_{{\text{pO}}_{\text{2}}}$ to increase. Therefore, our results may present the performance of the Apple Watch and Garmin Vivosmart more favorably, given that wearable devices perform best when ${\text{S}}_{{\text{pO}}_{\text{2}}}$ values are high.³³

Erroneous ${\text{S}}_{{\text{pO}}_{\text{2}}}$ measures were previously reported when using wearable devices.³⁰ However, the present study showed erroneous measures to an extent that well exceeds what has been previously reported. We experienced erroneous measures with the Apple Watch for result was exactly 50% of the participants and substantial measurement latency with the Garmin Vivosmart. What is intriguing is that these erroneous measures occurred even though the performance of each device was tested and confirmed with a participant when in the same seated position just before starting the protocol. It is difficult to explain this phenomenon, but it might be related to the fact that low perfusion on the wrist makes the devices more susceptible to motion artifacts causing a higher number of erroneous ${\text{S}}_{{\text{pO}}_{\text{2}}}$ measures.

Several factors, such as motion artifacts, variations in skin color, and poor tissue perfusion on the back of the wrist, render photoplethysmography data collection by wearable devices less accurate.³⁴ The Apple Watch and the Garmin Vivosmart use reflective photoplethysmography sensors to measure both ${\text{S}}_{{\text{pO}}_{\text{2}}}$ and heart rate. However, there is emerging evidence that reflective photoplethysmography technology has a low signal-to-noise ratio when measuring on the wrist. This makes wearable devices more susceptible to motion artifacts, thereby increasing the risk of erroneous ${\text{S}}_{{\text{pO}}_{\text{2}}}$ measures.³⁵ Wearable devices can significantly increase motivation and physical activity,^13,14 and health-care professionals increasingly use this technology to aid diagnosis and treatment.¹⁷ Wearable devices may therefore be valuable in minimizing the fear of breathlessness during physical activity and guide individual exercise intensity, thus increasing the efficacy of pulmonary rehabilitation. However, the present results show that current wearable devices are not yet suitable for measuring ${\text{S}}_{{\text{pO}}_{\text{2}}}$ in patients with COPD.

Limitations

The sample size in the present study was relatively small. However, a sample of 30 - 45 participants is generally considered sufficient to provide information that concerns the accuracy of wearable devices.^18,36 Measurements were conducted at a single time point, with the participant positioned correctly with the help of the researchers. Therefore, the transfer of results to conditions in which patients would be responsible for conducting their own measurements should be made cautiously. Skin pigmentation is known to affect the accuracy of photoplethysmography technology.³⁷ In this study, however, all the participants were white and had no tattoos in the areas used for measurement; hence, generalizations cannot be made for individuals with darker skin tones or tattoos in the measurement area.

This study used a standard finger pulse oximeter as a comparator measure. Therefore, this is a comparison of devices, not a comparison to the reference standard measure of blood oxygen saturation levels (arterial oxygen saturation). The true accuracy of the Garmin Vivosmart 4 and Apple Watch Series 7 would require a comparison with a measure of blood oxygen saturation levels (arterial oxygen saturation). However, this study aimed to assess the accuracy of pulse oximetry measures performed by wearable devices during pulmonary rehabilitation when blood oximetry was not feasible. Although this study was not designed to measure the effects of co-morbidities on the accuracy of wearable devices, it is likely that cardiovascular co-morbidities among patients with COPD, of which tobacco exposure is a common contributor, are likely to influence these results.

Conclusions

The results of this study indicate that the Apple Watch Series 7 and Garmin Vivosmart 4 overestimated oxygen saturation compared to a standard oximeter in participants with COPD when saturation was < 95% and underestimated oxygen saturation when saturation was > 95%. These findings suggest that, due to substantial measurement variability, wearable devices should not be used to monitor ${\text{S}}_{{\text{pO}}_{\text{2}}}$ during pulmonary rehabilitation.