The Accuracy of Readily Available Consumer-Grade Oxygen Saturation Monitors in Pediatric Patients

Abstract

BACKGROUND:

Pulse oximetry measurement is ubiquitous in acute health care settings in high-income countries and is familiar to any parent whose child has been treated in such a setting. Oximeters for home use are readily available online and are incorporated in several smartphones and smartwatches.

METHODS:

We wished to determine how accurate are oximeters available online that are designated for adult and pediatric use, and the saturation monitor integrated in a smartphone, when used in children, compared to reference, hospital-grade oximeters. We evaluated a fingertip oximeter marketed for children purchased online; an adult fingertip oximeter purchased online; the oximeter integrated in a smartphone; and reference, hospital-grade oximeters. Participants were < 18 y of age. Bland-Altman charts were generated, and the estimated root mean square error (EA_RMS) was calculated. Rates of failure to obtain a measurement, relationship between device and time to successful measurement, relationship between age and time to successful measurement, and relationship between error (vs the reference device) and age were evaluated for each consumer-grade device.

RESULTS:

We measured S_pO2 in 74 children between 0.1–17.0 y of age. Subjects weighing < 30 kg had a median (interquartile range [IQR]) age of 2 (1.0 month–1.4 y) months, and subjects weighing ≥ 30 kg had a median (IQR) age of 14.3 (11.9–16.2) y. Readings could not be obtained in 7.5, 0, and 38.8% of subjects using the pediatric, adult, and smartphone oximeters, respectively. The time to successful reading had a modest negative correlation with age with the inexpensive adult and pediatric oximeters. The inexpensive pediatric oximeter had an overall negative bias, with a mean difference from the reference device of −4.5% (SD 7.9%) and an error that ranged from > 8% to < 33% the reference device. The EA_RMS was 7.92%. The inexpensive adult oximeter demonstrated no obvious trend in error in the limited saturation range evaluated of 87–99%. The overall mean difference was −0.7% (SD 2.5%). EA_RMS was 2.5%. The smartphone oximeter underestimated S_pO2 at saturations < 94% and overestimated S_pO2 for saturations > 94%. Saturations could read as much as > 4%, or < 17%, than the reference oximeter. The mean difference was −2.9% (SD 5.2%). EA_RMS was 5.1%.

CONCLUSIONS:

Our findings suggest that the performance of consumer-grade devices varies considerably by both subject age and device. The pediatric fingertip device and smartphone application we tested are poorly suited for use in infants. The adult fingertip device we tested performed quite well in larger children with relatively normal oxygen saturations, and the pediatric fingertip device performed moderately well in subjects > 1 y of age who weighed < 30 kg. Given the vast number of devices available online and ever-changing technology, research to evaluate nonclinical oximeters will continue to be required.

Introduction

S_pO2 measurement is ubiquitous in most acute health care settings in high-income countries and is familiar to any parent whose child has been treated in such a setting. Oximeters for home use have become increasingly available; a brief search on Amazon Canada for “saturation monitor” found over 3,000 results (Amazon.ca: https://www.amazon.ca/s?k=saturation+monitor&crid=235UQ96R8CEFJ&sprefix=saturation+monitor%2Caps%2C575&ref=nb_sb_noss_1. Accessed December 13, 2023) and over 17,000 results on eBay (eBay: https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2334524.m570.l1313&_nkw=oximeter&_sacat=0&LH_TitleDesc=0&_odkw=saturation+monitor&_osacat=0. Accessed December 13, 2023) for “oximeter.” Most of these devices are fingertip oximeters, use transmission technology, and cost around $25 Canadian. Some devices use a reflectance technology with a light source built into a smartphone or smartwatch; some Android smartphones utilize a dedicated biosensor for this purpose.^1-3 Families may use these machines to help determine the severity of acute respiratory illnesses, including to decide when to seek emergent care, and for monitoring children with chronic lung disease.⁴ Nearly all published studies that have evaluated commercially available, non-FDA approved oximeters were performed in adults (Supplementary table, see related supplementary materials at http://www.rcjournal.com); few participants had a clinically important S_pO2 < 90%, and some studies immobilized the participants’ extremity.^2,5-9 Published studies that examined whether devices correctly displayed normal or abnormal values generally reported disappointing results, as even a small amount of bias could move a subject into a different category.^6,10 Given the reliance of many families on these devices, there is an urgent need to assess the accuracy of consumer-grade oximeters in children in real-life settings.

We examined 3 readily available consumer-grade devices consisting of an adult and a pediatric fingertip oximeter and an app built into a Samsung smartphone and compared them to hospital-grade oxygen saturation monitors. We hypothesized that the non–medical grade oximeters would be less accurate than hospital-grade oximeters.

QUICK LOOK.

Current knowledge

A vast number of inexpensive pulse oximeters are available online and pulse oximetry has been integrated into some smartphones and smartwatches. Few studies have evaluated how accurate consumer-grade devices are, particularly at lower S_pO2 levels and in children. Available studies suggest considerable bias, particularly at lower S_pO2.

What this paper contributes to our knowledge

An inexpensive pediatric pulse oximeter had an overall negative bias; a wide error range; occasionally failed to obtain a reading; and, as with its adult counterpart, took longer to obtain a measurement in younger children. An inexpensive adult oximeter obtained a reading in all subjects and had minimal error in relatively normoxic children and youth. An oximeter integrated into an Android smartphone had increased bias at lower and higher S_pO2 and often failed to obtain a reading in children.

Methods

Design and Sampling

We used a cross-sectional design. A convenience sample of subjects were recruited from the Children’s Hospital of Eastern Ontario (CHEO), Ottawa, Canada, pediatric respirology and cardiology clinics. Inclusion criteria were children < 18 y of age and ability of the subject and/or family to provide informed consent and/or assent. Planned exclusion criteria were use of supplemental oxygen at the time of S_pO2 measurement, presence of nail polish or artificial nails on all digits, and/or clinical instability or hypoperfusion. However, in a few subjects who were admitted and were on supplemental oxygen, measurements were made while the child remained on oxygen. We prioritized children known to have low oxygen saturations (< 90%) to evaluate the performance of these devices when children had clearly abnormal S_pO2. Due to limited numbers of subjects with S_pO2 < 90%, we recruited additional participants (typically more ill children who were desaturated) on hospital wards, the ICU, and in the emergency department. Data collection started in the fall of 2019. As in-person research data collection was not permitted during the COVID-19 pandemic peak, data collection was halted in 2020–2021, and the final 11 subjects were recruited in the summer of 2022.

Measurements

The research team chose an inexpensive adult and pediatric fingertip oximeter from Amazon Canada that seemed representative of available products. The pediatric device was chosen on the basis of the product description indicating it was marketed for pediatric use. Both devices cost less than $40 Canadian before shipping. After obtaining informed consent, the participant’s age, weight, sex, and location were recorded. Small subjects were considered those weighing < 30 kg, and larger subjects were classified as those weighing ≥ 30 kg based on the recommended weight ranges for adult Medtronic Nellcor Oximax reusable probes (Medtronic, Minneapolis, Minnesota).¹¹

S_pO2 was measured on room air in each subject using a reference, hospital-grade oximeter. Our plan was to use a portable Nellcor PM10N oximeter (Medtronic) as the reference device. Unfortunately, due to miscommunication within the research team, for a minority of subjects other hospital-grade oximeters were used, including the Nellcor OxiMax N-65 portable oximeter (Medtronic), the Welch Allyn Connex 6999 Series portable stand-mounted monitor (Hillrom, Chicago, Illinois), and ward wall-mounted units. In the majority of participants, a Nellcor Oximax MAXN single-use self-adhesive probe (Medtronic) was used; but for a few subjects weighing > 30 kg, an adult Nellcor OxiMax clip-on reusable sensor was used. For children who weighed ≥ 30 kg, S_pO2 was also measured using a Concord Emerald fingertip pulse oximeter (Concord Health Supply, Skokie, Illinois) and a Samsung Galaxy S9+ smartphone (Samsung, Suwon, South Korea) using the native Samsung Health app (version 5.12.025). For children who weighed < 30 kg, S_pO2 was measured using the reference oximeter, the Samsung Galaxy smartphone, and a “pediatric” “Finger Pulse Oximeter Pediatric S_pO2 Blood Oxygen Saturation Meter Rechargeable” (marketed by MCP Healthcare, Model A1).

Protocol

The oximeters were applied in random order using a random number table, and the right hand was used for measurement, with measurements performed in accordance with usual practice. Devices were applied to the digit that seemed most appropriate for the size of the device. We encouraged children to remain still while measurements were taken with each device. Measurements were taken once a stable heart rate and plethysmographic tracing was displayed, and the S_pO2 and heart rate were then recorded. In a subset of children, measurements were performed on both hands to evaluate reproducibility (n = 7; data available on request). The time needed to acquire a stable reading was also recorded, and measurements were abandoned if a stable reading was not obtained after 3 min. Approval for the study was obtained from the CHEO Research Ethics Board.

Analysis

The cohort was collected through an opportunistic sample at CHEO, aiming for a sample size of 50–100 subjects based on a literature review of previous studies. As the precision of the selected devices was unknown, we did not base the sample size on a power calculation for sample size. The mean difference and limits of agreement (95% CI) were presented for each consumer-grade pulse oximeter and the hospital-grade oximeters as the accepted standard, and Bland-Altman plots were used to visualize these differences. The proportion of successful readings, with 95% CI calculated using the Wilson score method, was our secondary evaluation measure. In some children, measurements were done on both or the left hand (depending, for example, on location of intravenous lines); statistics were based on the number of measurements done (rather than the number of children who had a measurement), so the total number of measurements analyzed was sometimes greater than the number of subjects in each size category. The United States FDA requires Bland-Altman plots and standards for root mean square error (A_RMS = square root [mean square error or deviation]) < 3% for S_pO2 70–100%, determined by arterial blood gas analysis, for oximeter FDA 501(k) certification.^9,13,14 We estimated A_RMS (EA_RMS) relative to the reference oximeter.¹² We also performed an exploratory analysis into whether child age impacts the measurement accuracy of consumer-grade pulse oximeters using linear regression. Statistical analyses were carried out using R (R Core Team 2022, R Foundation for Statistical Computing, Vienna, Austria). For time to successful reading by the oximeter, results are presented as median (interquartile range).

Results

Seventy-four children were included in the study and were grouped according to weight. The median age of the smaller children (n = 54) was 2 (1.0 month–1.2 y) months, and the median age of the larger children (n = 20) was 14.3 (11.9–16.2) y. Forty subjects (54.1%) were female, and 32.4% (24/74) of subjects had a history of cyanotic congenital heart disease (22/74, 29.7%) or lung disease with cyanosis (2/74, 2.7% of the study population). The remainder of the children were acyanotic. Among the smaller children, the median weight was 4.3 (3.5–10.7) kg; and among the larger children, the median weight was 56.6 (48.9–68.9) kg.

Readings could not be obtained in 7.7% (4/52) of attempts using the inexpensive pediatric oximeter; in 2 subjects, no measurement could be performed or attempts failed with all tested devices. All 23 attempts with the inexpensive adult oximeter were successful. Readings were unsuccessful in 38.8% (31/80) and 1.2% (1/82) of attempts using the smartphone and reference oximeters, respectively (Table 1). In small children, the probability of a successful read was lower with the smartphone than the inexpensive pediatric oximeter or the reference devices (Wilson score 0.49 [95% CI 0.37–0.62] vs 0.92 [95% CI 0.82–0.97] and 0.98 [95% CI 0.91–1.00], respectively). In larger children, the probability of a successful reading was also lower with the smartphone (0.91 [95% CI 0.73–0.98]) compared with the inexpensive adult oximeter (1.00 [95% CI 0.86–1.00]) or the reference devices (1.00 [95% CI 0.86–1.00]). There was no evidence of a difference between the other devices.

Table 1.

Successful and Unsuccessful Oximeter Reading Attempts by Size Group

Time to successful reading ranged from 5.0 s–3.0 min (57.5 s, 24.0 s, 99.8 s) with the pediatric oximeter, 10.0 s–1.0 min (19.0 s, 16.5 s, 25.0 s) with the adult oximeter, 10.0 s–3.0 min with the smartphone (133.0 s, 49.0 s, 180.0 s), and 2.0 s–3.0 min with the reference oximeters (28.0 s, 12.0 s, 40.0 s). The time to successful reading had a modest negative correlation with age (Supplemental Figs. 1–4, see related supplementary materials at http://www.rcjournal.com); Spearman correlation coefficient (r_s) was −0.33, −0.39, and −0.28 for the pediatric, adult, and smartphone devices, respectively. Time to successful reading did not vary significantly with age for the reference oximeters (r_s −0.08).

Instrument error of the inexpensive pediatric pulse oximeter systematically underestimated oxygen saturations for small children with low S_pO2 (Figure 1A). It could read as much as > 8%, or < 33%, than the reference oximeters. The mean difference was –4.5% (SD 7.9%). Limits of agreement with the reference device were −20 to 11%, and EA_RMS was 7.9%. Readings correlated moderately with the reference oximeters, with an r_s of 0.58. The degree of error between the inexpensive pediatric oximeter and the reference oximeters did not vary by age (r_s 0.02; Supplemental Fig. 5, see related supplementary materials at http://www.rcjournal.com). As the inexpensive pediatric oximeter was likely not intended for use in very small infants, we performed a post hoc analysis comparing this device to the reference oximeters in small children who were > 1 y of age. The consumer-grade device had greater accuracy in this subpopulation, with a mean difference of −1.5% (SD 9.2%); and EA_RMS was 4.0%, although the fingertip could still underestimate S_pO2 as much as 10%.

Fig. 1.

Bland-Altman plots for inexpensive oximeters. Solid line = bias; black dashed line = 95% limits of agreement. A: Inexpensive pediatric oximeter vs reference oximeter (n = 48). B: Inexpensive adult oximeter vs reference oximeter (n = 23). C: Smartphone oximeter vs reference oximeter (n = 49). Middle dash line indicates mean difference.

For larger subjects, the inexpensive adult oximeter demonstrated no obvious trend in error in the limited saturation range evaluated of 87–99% (Figure 1B). Instrument error overestimated oxygen saturations in 35% of readings. The overall mean difference was −0.7% (SD 2.5%). EA_RMS was 2.5%. Limits of agreement with the reference device were −5.6 to 4.1%. Readings were strongly correlated with the reference oximeters over this limited range, with an r_s of 0.77. There was a weak negative correlation between the degree of error between the inexpensive adult oximeter and the reference oximeters and age (r_s −0.15; Supplemental Fig. 6, see related supplementary materials at http://www.rcjournal.com).

For the smartphone, the Bland-Altman plot demonstrated that for the smartphone (Figure 1C) there was a linear relationship to the reference S_pO2, with an underestimation of S_pO2 at saturations < 94% and an overestimation of S_pO2 for saturations > 94%. Saturations could read as much as > 4%, or < 17%, than the reference oximeter. The mean difference was −2.9% (SD 5.2%). Limits of agreement with the reference oximeters were −13.2 to 7.4%, and EA_RMS was 5.1%. Readings correlated well with the reference oximeter, with an r_s of 0.81. There was a modest negative correlation between the degree of error between the smartphone and the reference oximeters and age (r_s 0.40; Supplemental Fig. 7, see related supplementary materials at http://www.rcjournal.com). As the smartphone was intended for adults, we performed a post hoc analysis in subjects 30 kg and older (data available on request). The smartphone performed better in this population: The mean difference was −1.1% (SD 3.6%), and EA_RMS was 3.2%.

Discussion

We report one of the few, and among the largest, studies done in a pediatric population and the first using an oximeter marketed for pediatric use. It is also the first pediatric study evaluating a smartphone oximeter. Our findings suggest that the performance of consumer-grade devices varies considerably by both subject age and device. We found the performance of the pediatric fingertip oximeter and an Android smartphone, used in infants, was inadequate. A pediatric fingertip device, when examined only in small children > 1 y of age, functioned somewhat better, as did an adult fingertip oximeter tested in larger subjects who had relatively normal S_pO2 values.

The adult fingertip oximeter, when used in larger and older children who also had more normal S_pO2 values, had a relatively small instrument error, as previously observed in analogous adult studies, where the absolute value ranged from 0.1–2.1%. Prior studies found only a minority of devices had an acceptable A_RMS performance at lower S_pO2 values, and some devices had large errors in some subjects.^10,14-16 In our study, EA_RMS was acceptable, at 2.4%, although the limits of agreement were still broad: −5.6 to 4.1%. The similarity of our results to comparable adult studies of fingertip devices likely reflects similar patient populations, with few desaturated subjects and probably similar devices.^10,14-16 To our knowledge, no information is available on how many companies manufacture the hardware used in consumer-grade oximeters, despite the plethora of brands being marketed. Importantly, in our study, a relatively small number of larger participants were evaluated, and we were unable to recruit older subjects with abnormal saturations. The performance of our adult fingertip device could not be evaluated in hypoxemic subjects or in a large group of subjects. We, therefore, cannot comment on the ability of this device to detect desaturation, which has been a common issue in similar adult studies.

The pediatric fingertip oximeter we used had a mean bias of −4.5%, which was larger than had been previously reported in adults studies examining fingertip oximeters.^10,15 EA_RMS was higher, and the limits of agreement were also much broader than reported in those studies (see related supplementary materials at http://www.rcjournal.com).^10,14,15 Bias with the pediatric fingertip device was also much more pronounced at S_pO2 values < 90% and in younger children. The worse performance of the pediatric fingertip device compared to adult fingertip monitors could reflect multiple device factors, including inadequacies of the device algorithm to manage motion artifact and lower blood flows in children, and worse performance at low S_pO2 (which is also the case with many medical-grade devices) as well as protocol-related issues: Our study included small infants where S_pO2 is difficult to measure and children with abnormal S_pO2 values, whereas the adult studies had few subjects who were desaturated and/or were kept motionless.^{6,10,14,15,17,18} Only 2 pediatric studies have been published to date. One only included children with relatively normal S_pO2.⁷ The other evaluated an FDA-cleared monitor with a probe attached by cable (Nonin WristOx2) in 24 infants with cyanotic congenital heart disease. They observed a bias of 4% and like us noted worsening performance at lower S_pO2 values.⁹

Several studies of older generations of Apple iPhone and the native camera flash and photograph sensor that used reflectance technology reported unsatisfactory results.^19,20 A number of Android smartphones, including recent generations of Samsung Galaxy smartphone, employ reflectance technology using a dedicated Maxim biosensor with a proprietary infrared light emitter and receiver. This was shown to have very good performance in a laboratory setting where subject motion was prevented but with wider limits of agreement during subject activity.³ We found that the Samsung smartphone performed more poorly in children than had been previously reported in adult studies.² Performance deterioration was marked in younger (or smaller) children, presumably related to smaller digits or extremities and more motion artifact. EA_RMS was somewhat above acceptable standards, but the limits of agreement were very broad: −13 to 7%. In contrast, the absolute value for reported instrument error in adults ranged from 0.2–2.4%.^3,21 The lower performance of the Samsung smartphone in our setting was likely in part due to our study design, as we tested the device in infants much smaller than the intended use. Supporting this assertion, error decreased with age; and, in a post hoc analysis, measured performance was considerably more comparable to other studies when we limited our sample to older children and youth.^2,3,13,21 Some very recent studies reported low levels of bias and fairly narrow limits of agreement using the Apple smartwatch and reflectance technology, although few subjects had a lower S_pO2 reading.^5,8,22

Inability to obtain a reading was a particular problem with the smartphone. We were unable to obtain a reading in nearly 40% of subjects, which was very similar to the 36% rate reported by Modi et al.³ Time to successful reading was problematic with both the pediatric device and the smartphone, which is a particular concern in children who often have a limited ability to stay still. Time to successful reading was acceptable in older subjects using the adult fingertip device. In summary, performance of the Samsung smartphone, despite its dedicated sensor, appeared to be unsatisfactory in the pediatric population.

Our study has a number of limitations. We had a moderate sample size, though our sample size was relatively large compared to published studies (see related supplementary materials at http://www.rcjournal.com). We tested only 3 devices; though given the thousands of devices available online and likely ongoing changes in manufacturers and design, testing even a significant fraction of available devices is unrealistic. Unfortunately, due to miscommunication within the research team, we did not use the same reference oximeter and saturation probe in every subject. Differences between different reference oximeters can be expected to have reduced the precision of our findings. Lack of documentation of the reference oximeter has been noted previously.⁷ In addition, to address the limitations of previous studies, we attempted to maximize the recruitment of subjects with low S_pO2. Despite this, < 40% of our measurements were done in participants with a clinically important S_pO2 of < 90%; and despite our attempts to prioritize this group, we were unable to find larger children who were desaturated. Many of our subjects in the < 30-kg category were infants (particularly the desaturated subjects) where it is fairly unlikely that any available non-clinical devices will work well given the size of infants’ extremities and perhaps higher likelihood of motion artifact. Post hoc analysis confirmed that this appeared to have exaggerated the reported imprecision of consumer-grade devices in young children, and there was better precision in small children who were > 1 y of age. The smartphone could have been expected to work particularly poorly in infants, and it might have been preferable to have planned ahead of time to not test this device in this subpopulation. In post hoc analysis, we confirmed that this device performed better in older children who more closely resembled adults in size. Hospital-grade oximeters usually use self-adhesive “Band-Aid” style probes in this population, where small probes designed for infants allow the light emitter and sensor to be fairly close to each other, and adhesion of the band reduces motion artifact. Some of the subjects were in critical care areas, and while very sick children generally move less, we did not track this nor use an objective method of quantifying movement. Measurements were made consecutively rather than concurrently. This may have increased or decreased subject motion, as subjects became more restless or more acclimatized to the measurements and/or the researcher adjusted his approach to the subject. We felt that not having multiple devices attached to the subject concurrently made the study less intrusive, and random placement of the monitors should minimize any such effects. Our design could have increased real error as saturations changed during the measurement period. However, the total duration of individual subject evaluations was short. We did not evaluate our devices’ performance against the accepted standard of arterial blood gases, although these are rarely done in children outside a critical care setting. Our methodology paralleled most other studies of non-clinical oximeters and provides data more representative of home use. A few subjects were evaluated while on supplemental oxygen. This did not affect our analyses. Unfortunately, we did not record whether the subject was receiving oxygen at the time the measurements were taken. Importantly, we did not document skin color or use standardized measures to evaluate skin tone.¹³ There is growing evidence that skin color significantly affects S_pO2, with increased bias at lower S_pO2 values.²³

Strengths of our study included near-simultaneous measurement of the study and reference oximeters, inclusion of subjects with abnormal S_pO2, a large pediatric sample size, and measurement in real-world settings. We also documented the proportion of instances when devices were completely unable to obtain a reading, the time required to obtain a reading (when successful), and we related measurement acquisition time to age.

Given the widespread home use of oximeters, it will be important to carry out further research documenting the factors prompting families to purchase of non–clinical grade devices, family perceptions of the accuracy of their readouts, the training they have received on using their device, and how the results are applied. One study in a group of seniors with chronic lung disease, many of whom were on home oxygen therapy, found that most purchases were self-initiated; nearly all devices were bought online or at a local pharmacy, and participants felt that having an oximeter increased confidence and reduced anxiety.²⁴ An unresolved issue, which would require a qualitative study with subjects, is whether families are more concerned about a threshold S_pO2 that would prompt further action or the accuracy of the actual reading displayed on the device.²⁵ Reliance on threshold values was highlighted during the COVID-19 pandemic, when thresholds were used for clinical decision making, including admission to hospital.^26,27 Given the broad limits of agreement of the devices we evaluated, relying on a home device to accurately determine whether a S_pO2 value is above or below a threshold may not be realistic. Furthermore, with the probable variability of readings with these devices, using a device to follow a trend may not actually be reliable during home use. Of note, a Cochrane review assessing the use of oximeters as part of an asthma action plan found no relevant studies.²⁵

Conclusions

In summary, our findings suggest that the pediatric fingertip device and smartphone we tested are not well suited for use in infants. The adult and pediatric fingertip devices tested performed reasonably well in larger children with oxygen saturations > 90% and fairly well in small children > 1 y of age, respectively. Given the ongoing evolution in technology and vast number of devices available, particularly online, research to validate the accuracy of consumer-grade oximeters will continue to be needed.