The Utility of iPhone Oximetry Apps: A Comparison with Standard Pulse Oximetry Measurement in the Emergency Department

Introduction

Pulse oximetry (pulse ox) is an expedient and accurate tool to non-invasively measure the oxygenation status of any patient in whom this might be a clinical concern. It was developed by Glenn Allan Millikan, an American physiologist and mountaineer during World War II¹. The term “oximetry” is attributable to him. Pulse oximeters measure saturation of peripheral oxygen (SpO2) by measuring the difference in absorption of oxygenated vs deoxygenated blood at two different wavelengths (typically red light at 660 mn and infrared at 940mn). Oxygenated hemoglobin absorbs more infrared and deoxygenated hemoglobin absorbs more red light. This difference is measured by the diodes on the device and is used to calculate the SpO2². This tool can be placed on multiple spots on the body to non-invasively obtain an accurate measure of blood oxygenation and detect hypoxemia^{3, 4, 5, 6}. The finger probe is commonly used for measurements, but multiple other sights can be used with varying accuracy^{7, 8}. The use of pulse oximetry lias been shown to reduce the need for more invasive measurements, such as an arterial blood gas⁹. More recently, portable finger probes have been developed, allowing one to measure oxygen saturation in a variety of enviromnents and situations.^10,11 Small, portable pulse ox devices allow measurement of oxygen saturation in resource limited enviromnents, however even these may not be immediately available.

In the past decade, smart phone technology lias become nearly ubiquitous, with usage growing rapidly around the world. Ninety-five percent of Americans own cell some type of cell phone and 77% own a smart phone¹². In recent years there have been applications have been developed claiming to measure vital signs such as heart rate, respiratory rate, and pulse oximetry with nothing but the device’s camera lens and light¹³. Overall, little research lias been done to verily the validity of these applications’ claims. Recent studies have found validity in the HR measuring capabilities of these applications when compared to control^{13, 14, 15}, yet when analyzing their ability to measure pulse ox it was found that the SpO2 did not clinically correlate to the monitor.¹⁵ Studies among both pediatric and adult assessing the accuracy of these devices have used healthy volunteers with presmnably normal SpO2s¹⁶. There have been no studies of these applications in an undifferentiated or potentially hypoxic patient population.

The objectives of this study are to determine the correlation of three iPhone application pulse ox measurements to that of standard pulse oximetry. We also wanted to test their ability to detect hypoxia as compared to control measurements in an undifferentiated emergency department (ED) patient population who had both nonnal and abnormal SpO2 readings.

METHODS

Study Design

This study was approved by the University of Alabama at Birmingham Institutional Review Board. This was a correlational study in which three iPhone pulse ox applications pulse ox and pulse measurements were compared to measurements from a standard pulse oximeter in the ED. The three applications used were: “Oximeter” (Ox) (produced by digiDoc technologies, Egersund, Norway) “Heart Rate & Pulse Oximeter,” (POx) (produced by LIJUN LIU), both of which use the iPhone’s onboard camera and light to record measurements, and “iOx” (produced by Safe Heart USA, Atlanta, GA) which uses an external probe that can be purchased from the Safe Heart company and which connects to the iphone. This device uses red light similar to a more conventional pulse ox monitor. The ED TRAM 451 (General Electrics, New York City, New York) pulse ox monitors were used as control measurements. The applications were purchased and downloaded onto a single decommissioned iPhone 5S.

Patients

The study population was a convenience sample of patients 18 years of age and older presenting to a large university medical center ED. Participants were recruited from the triage area or main ED patient rooms. Inclusion criteria were presentation with a cardio/pulmonary chief complaint or an initial sPO2 of ≤ 94%, both of which could be assessed via the ED’s FirstNet electronic medical record (EMR). Exclusion Criteria included the presence of peripheral artery disease (PAD) and anemia of severity that might affect pulse ox, or inability to give consent. In order to include a reasonable number of patients with hypoxia, an effort was made to include as many participants as possible with an initial SpO2 of ≤ 94%.

Measurements

The authors enrolled all patients. After verbal consent and collection of demographic data, experimental measurements were taken from the right index finger using all three iPhone applications. The control was recorded first and then the experimental measures in the predetermined individualized random order. The order was randomized to minimize any effect of second to second variations in continuous pulse ox readings. If patients were on supplemental oxygen, a ventilator, or a CPAP, tins was noted.

Data analysis

Descriptive statistics for the study population are reported as means and standard deviations for continuous variables and percentages for categorical variables. To assess differences in the distributions of SpO2s for each of the devices, we reported means, medians, 25^th to 75^th percentiles, minimum to maximum, and differences relative to the control measures. The level of agreement between each of the devices and control measures was assessed using several measures included in the user-written “concord” Stata package (Pearson’s correlation coefficient, bias correction factor, intercept/slope, average difference with 95% limits of agreement, and the concordance correlation coefficient) and the alpha coefficient for reliability^17,18,19. Data was analyzed for the entire population and the subpopulation of patients with SpO2 ≤94%. We also produced a pairplot of the differences by the mean of the measurements (i.e., a Bland-Altman plot) for the entire population²⁰. We also reported measures of validity (sensitivity, specificity, positive predictive value, and negative predictive value) for detection of hypoxia, defined as an SpO2 ≤94%. All analyses were performed using Stata 13.1 (Stata Corp, LLC, College Station, TX).

Results

Overall 191 patients were evaluated, population characteristics are shown in Table 1. The majority of patients were white (55%) and male (56%). Among these patients, 23.0% required supplemental oxygen at the time of data collection.

TABLE 1.

Population characteristics

Characteristic	Mean (SD) or N (%)
Age (years) (Mean / SD)	58.7 (15.7)
Race (N / %)
White	106 (55.5%)
Black	80 (41.9%)
Other	5 (2.6%)
Gender (N / %)
Female	84 (44.0%)
Male	107 (56.0%)
Anemia or PAD not severe(N / %)
Yes	13 (6.8%)
No	178 (93.2%)
Supplemental Oxygen (N / %)
Yes	44 (23.0%)
No	147 (77.0%)

Mean and median SpO2 readings were generally higher for Ox and Pox relative to control measurements, while iOx readings were lower (Table 2). The minimum readings were 80%, 94%, 95%, and 75% for control. Ox, Pox, and iOx respectively. Control readings indicated that 35.6% of patients were hypoxic (SpO2 ≤94%). For Ox, only 9.4% of patients were classified as hypoxic. No patients were classified as hypoxic by Pox. The proportion classified as hypoxic was higher than controls for iOx (36.1%).

TABLE 2.

Distribution of oxygen saturation and measured differences by instrument

Measure	Measurement Instrument
	Control	Ox	Pox	iOx
Total Patients (N)	191	191	191	191
Oxygen Saturation (%)
Mean (SD)	96.0 (3.2)	98.2 (2.0)	97.2 (1.2)	94.7 (4.5)
Median (25th , 75th Percentile)	97 (94, 99)	99 (97, 100)	97.2 (96.2, 98.1)	95 (93, 98)
Minimum , Maximum	80, 100	94, 100	95, 100	75, 100
Hypoxia (<=94%) (N / %)	68 (35.6%)	18 (9.4%)	0 (0%)	69 (36.1%)
Difference (Measured Oxygen Saturation - Control) (%)
Mean (SD)	---	2.19 (3.78)	1.14 (3.33)	−1.35 (3.56)
Median (25th , 75th Percentile)	---	2 (0, 5)	0.8 (−1.4, 3.1)	−1 (−2, 0)
Minimum , Maximum	---	−6, 14	−4.9, 15.3	−19, 7

For the full population of patients, iOx readings showed the highest concordance and reliability with control measurements (Table 3). The iOx device also perfonned best in the subgroup of patients with hypoxia. Average differences indicated that Ox and Pox measurements were frequently higher than control readings, whereas iOx was slightly lower (Figure 1). Differences relative to control readings were greater for Ox and Pox when limited to hypoxic patients. For the whole population, as well as for participants with hypoxia, we found little to no agreement with the control measures for Ox and Pox.

TABLE 3.

Concordance and reliability of instruments compared with control measurements

Statistic	Measurement Instrument
	Ox	Pox	iOx
All Patients (N=191)
Concordance
Pearson’s Correlation Coefficient	0.015	0.112	0.613
Bias Correction Factor	0.666	0.578	0.898
Intercept / Slope	0.61 / 39.73	0.36 / 62.38	1.37 / −36.95
Average Difference (95% Limits of Agreement)	2.19 (−5.21, 9.60)	1.14 (−5.39, 7.66)	−1.35 (−8.32, 5.63)
Concordance Correlation Coefficient (95% CI)	0.01 (−0.09, 0.11)	0.07 (−0.02, 0.15)	0.55 (0.46, 0.63)
Reliability (Alpha Coefficient)	0.03	0.13	0.74
Hypoxia Patients (Control <=94%) (N=68)
Concordance
Pearson’s Correlation Coefficient	0.195	−0.058	0.497
Bias Correction Factor	0.178	0.182	0.730
Intercept / Slope	0.88 / 17.2	−0.54 / 146.9	2.26 / −116.9
Average Difference (95% Limits of Agreement)	6.06 (1.03, 11.09)	4.59 (−0.31, 9.49)	−0.63 (−8.89, 7.63)
Concordance Correlation Coefficient (95% CI)	0.04 (−0.01, 0.08)	−0.01 (−0.06, 0.03)	0.36 (0.22, 0.51)
Reliability (Alpha Coefficient)	0.32	0.09	0.54

FIGURE 1.

Pairplots showing differences in instrument compared with control measurements

Of the three instruments examined, iOx demonstrated the highest sensitivity, positive predictive value, and negative predictive value (69.1%, 68.1%, and 82.8%, respectively) (Table 4). Specificity was lower for iOx relative to the other instruments. However, sensitivity and positive predictive value were extremely low for Ox and Pox.

Table 4.

Measures of validity for identifying hypoxia

Statistic	Measurement Instrument
	Ox	Pox	iOx
Hypoxia(<=94%)
Control No / Measured No (N)	110	123	101
Control Yes / Measured No (N)	63	68	21
Control No / Measured Yes (N)	13	0	22
Control Yes / Measured Yes (N)	5	0	47
Sensitivity (%)	7.4%	0%	69.1%
Specificity (%)	89.4%	100%	82.1%
PPV (%)	27.8%	Not Defined	68.1%
NPV (%)	63.6%	64.4%	82.8%

Discussion

Of the three approaches for measuring SpO2, iOx performed better than Ox and Pox in tenns of agreement with control measures. While there was a moderate agreement between the external device (iOx) and the ED pulse oximeters used as controls, it is not strong enough to recoimnend to patients or physicians, even in austere enviromnents. Until the technology is capable of obtaining reliable and valid measurements, it should be recoimnended that patients and care providers use portable devices that have been shown to be accurate.

Alexander, et al.¹⁵ found poor correlation of SpO2, blood pressure, and heart rate among a variety of iphone applications compared to clinical monitors when assessed in a group of healthy volunteers. Our study also utilized iphone only applications, but we included an app with an attached external finger probe that purported to measure pulse ox. We also studied a population of real patients in a clinical enviromnent and included a moderately sized subgroup who were hypoxic. We recruited this sample because we wanted to test these apps not only in persons with normal SpO2, but also in those with low oxygen saturation. Similar to the prior study, we found limited agreement with measures taken on a standard ED pulse oximetry device.

Our findings do not suggest that these resources should be completely disregarded. As technology continues to advance, it is possible that accuracy will improve and portable devices will provide valid measures that are clinically actionable. These instruments should continue to be examined by peer reviewed research for their possible use moving forward. Other similar avenues of research are opening every day as more companies are releasing technologies to the public for vitals measurement and health maintenance. Wearable technology and devices, such as handheld electrocardiograms, are now widely available to the public. It should also be of vital importance to ensure that these technologies are safe and reliable, and only then should they be made available to providers in resource-limited settings.

Limitations

There are several limitations that should be considered in the interpretation of this data. First, oxygen was not withheld from patients in hypoxic states and in some situations this altered the control measurement as data collection progressed. Researchers in the future should be made aware of this difficulty and informed to rapidly respond to hypoxic patients. Secondly, there are a number of possible conditions that can reduce the accuracy of pulse oximetry devices. Differences in the calibration of light source, anemia, optical interference with endogenous and exogenous substances in the blood, and even skin color have been documented to potentially reduce the accuracy of an oximetry measurement. The same iPhone was used for all of the applications throughout the study, the variance in calibration of the light source was deemed not to be significant. However, a second external adaptor had to be purchased for the iOx application because the first one stopped functioning midway through data collection. This may have added to some of the variance in light calibration. It was decided not to exclude patients based on their skin tone because patients with both light and dark skin tones should be equally as likely to use these pulse ox apps.

Conclusions

While iOx had modest agreement with the control measures, Ox and POx showed almost none. The iOx device also showed the best ability to correctly identify patients in low-oxygen states, but almost one-fourth of patients were incorrectly classified. Overall, the three apps provided inaccurate SpO2 measurements and had limited ability to accurately detect hypoxia. In their current state, these devices should not be recommended for use in situations where there is a need to assess oxygenation status. Additionally, patients and healthcare providers alike should be informed that these devices are not equivalent to gold-standard pulse oximetry devices. Should anyone want to assess oxygenation status in austere environments, they should be directed to seek out portable fingertip pulse oximetry monitors, which have found to be accurate in prior studies³.