Racial Bias in Pulse Oximetry Measurement Among Patients About to Undergo Extracorporeal Membrane Oxygenation in 2019-2020

Abstract

Background

Pulse oximeters may produce less accurate results in non-White patients.

Research Question

Do pulse oximeters detect arterial hypoxemia less effectively in Black, Hispanic, and/or Asian patients than in White patients in respiratory failure and about to undergo extracorporeal membrane oxygenation (ECMO)?

Study Design and Methods

Data on adult patients with respiratory failure readings 6 h before ECMO were provided by the Extracorporeal Life Support Organization registry. Data was collected from 324 centers between January 2019 and July 2020. Our primary analysis was of rates of occult hypoxemia—low arterial oxygen saturation (Sao₂ ≤ 88%) on arterial blood gas measurement despite a pulse oximetry reading in the range of 92% to 96%.

Results

The rate of pre-ECMO occult hypoxemia, that is, arterial oxygen saturation (Sao₂) ≤ 88%, was 10.2% (95% CI, 6.2%-15.3%) for 186 White patients with peripheral oxygen saturation (Spo₂) of 92% to 96%; 21.5% (95% CI, 11.3%-35.3%) for 51 Black patients (P = .031 vs White); 8.6% (95% CI, 3.2%-17.7%) for 70 Hispanic patients (P = .693 vs White); and 9.2% (95% CI, 3.5%-19.0%) for 65 Asian patients (P = .820 vs White). Black patients with respiratory failure had a statistically significantly higher risk of occult hypoxemia with an OR of 2.57 (95% CI, 1.12-5.92) compared with White patients (P = .026). The risk of occult hypoxemia for Hispanic and Asian patients was equivalent to that of White patients. In a secondary analysis of patients with Sao₂ ≤ 88% despite Spo₂ > 96%, Black patients had more than three times the risk compared with White patients (OR, 3.52; 95% CI, 1.12-11.10; P = .032).

Interpretation

Compared with White patients, the prevalence of occult hypoxemia was higher in Black patients than in White patients about to undergo ECMO for respiratory failure, but it was comparable in Hispanic and Asian patients compared with White patients.

Graphical Abstract

Take-home Points.

Study Question: Is there race and ethnicity variation on pulse oximetry sensitivity in a cohort of respiratory failure patients requiring extracorporeal membrane oxygenation (ECMO) therapy?

Results: Black patients with respiratory failure had a statistically significantly higher risk of occult hypoxemia when defined as Sao₂ ≤ 88% despite Spo₂ 92% to 96% with an OR of 2.57 (95% CI, 1.12-5.92) compared with White patients (P = .026)

Interpretation: Prevalence of occult hypoxemia was higher in Black patients compared with White patients when assessed by Sao₂. The prevalence of occult hypoxemia in Hispanic and Asian patients was comparable to that of White patients.

A combination of intermittent arterial blood gas measurements and continuous pulse oximetry is used to assess for possible hypoxemia in ventilated patients.¹ Although arterial blood gas (ABG) measurements of oxygen saturation (Sao₂) are the reference standard, continuous pulse oximetry measurement of peripheral oxygen saturation (Spo₂) is less invasive and used to make minute-to-minute decisions in the ICU.^2,3 Titration in response to pulse oximetry alone may be common in some institutions.⁴

Variation in pulse oximetry sensitivity among healthy adult patients with different skin colors has been reported.^5,6 A 1990 report found increased rates of hypoxemia (Po₂ as low as 49 mm Hg) despite normal-range pulse oximetry in critically ill Black patients admitted to medical ICUs and receiving mechanical ventilation.⁷ More recently, matched pulse oximetry Spo₂ and ABG saturation (Sao₂) pairs from a single-center cohort and a multicenter cohort of patients receiving supplemental oxygen therapy were tested for occult hypoxemia (Sao₂ < 88% despite Spo₂ between 92% and 96%).⁸ Black patients were three times more likely to have occult hypoxemia compared with White patients. These findings were present both in unadjusted and adjusted analyses for age, sex, and SOFA (Sequential Organ Failure Assessment) score. There are at least four limitations to the data presented in the most recent investigation: (1) race and ethnicities other than Black and White were not compared; (2) patients in the cohort were being treated for a range of critical illnesses, raising questions about the generalizability of the findings to patients in extreme respiratory failure; (3) occult hypoxemia rates when Spo₂ > 96% were not reported; and (4) matching between arterial blood gas and pulse oximetry was done using time-stamped electronic data, rather than data confirmed by a physician to be obtained simultaneously.

Because of the potential importance of disparate accuracy of pulse oximetry by race and ethnicity, we sought to address the first three of these limitations in the current study. We tested the separate hypotheses that pulse oximetry would be less effective in Black, Hispanic, and Asian patients than in White patients in the detection of arterial hypoxemia. We specifically focused on the magnitude of the discrepancy between Spo₂ and Sao₂-defined hypoxemia in a cohort of ventilated patients with respiratory failure so severe that they received extracorporeal membrane oxygenation (ECMO) therapy within 6 h of data collection. The analysis was performed in a matched data set of Spo₂ and Sao₂ measurements hand-curated for accuracy by Extracorporeal Life Support Organization (ELSO) data managers with full access to bedside information.⁹

Study Design and Methods

Deidentified data from the ELSO registry⁹ were used for this analysis. This investigation was deemed exempt from regulation by the University of Michigan Institutional Review Board Core Committee (HUM00191710). The ELSO registry was queried for patients age 18 years or older who were diagnosed with either (1) ARDS or (2) COVID-19. Patients with ARDS were identified by International Classification of Diseases, 10th Revision (ICD-10) code J80. COVID-19 was defined by the presence of severe acute respiratory syndrome coronavirus 2 on laboratory testing. These patients were placed on ECMO for treatment of respiratory failure between January 1, 2019 and July 13, 2020.

Data were collected retrospectively to represent the situation 6 h before the start of extracorporeal life support (ECLS). The pre-ECLS arterial blood gas values evaluated for each patient corresponded to arterial blood gas samples that met the following criteria: (1) were drawn before the ECLS start time; (2) were drawn no more than 6 h before the ECLS start time; and (3) if multiple arterial blood gases were collected in this 6-h period, the one closest to and before the ECLS start time was reported. Spo₂ and Sao₂ data were hand extracted by an ELSO data manager as part of the ELSO registry method.⁹ We included all patients with relevant data for any given analysis; case-wise deletion was used. Only first ECMO runs within a hospitalization were included.

The data contained a single entry for race and ethnicity as defined at the ECMO center: Asian, Black, Hispanic, Middle Eastern or North African, Native American, Native Pacific Islander, Multiple, Other, Unknown, and White. Using an event per variable framework¹⁰ and based on the reported unadjusted rates of occult hypoxemia event in Black and White patients receiving supplemental oxygen, 11.7% and 3.6%, respectively,⁸ we projected that the sample size necessary to capture 10 cases of this low-frequency outcome would be approximately n = 400 before accounting for anticipated missing data and patients with Spo₂ out of our range of interest, and used that as the minimum number of observations needed for inclusion of a racial group in the analysis. We selected racial and ethnic groups before exclusion of observations for missing ABG data; hence the final numbers of observations for each group are lower than the total number of patients from each group present in the data set. Bias (mean difference, Spo₂ – Sao₂), precision (SD of the differences), limits of agreement (bias ± 1.96 × SD), and accuracy root mean square error $\left(A_{\text{RMS}}={\left[\left(\mathrm{b}\mathrm{i}\mathrm{a}{\mathrm{s}}^2+\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}{\mathrm{n}}^2\right)\right]}^{1/2}\right)$ data by race and ethnicity for the entire dataset were calculated according to previously described methods.^11,12 Lower values for root mean square error indicate a higher degree of correlation between the two measurement modalities evaluated.^13,14 A root mean square error of ≤ 2% to 3% is required by the US Food and Drug Administration (FDA) for pulse oximetry devices.¹¹ The range of clinical interest for Spo₂ (92%-96%) was selected for the analysis of occult hypoxemia because within this range bedside physicians would be unlikely to make changes to the level of oxygen provided. Objective data on skin tone were not available.

New ECMO initiations with Spo₂ values between 92% and 96% were the focus of the primary analysis. The pulse oximetry readings (Spo₂) and matching Sao₂ values were intended to be collected at the same time, but no external validation of the exact synchrony of the measurements was available. We obtained descriptive statistics for patient characteristics. The primary outcome of interest was the unadjusted rate of occult hypoxemia across different races and ethnicities. Occult hypoxemia was defined as Sao₂ ≤ 88% despite Spo₂ between 92% and 96%. χ² tests were used to compare the rates of pre-ECMO occult hypoxemia in White patients compared with each of the other races and ethnicities. This specific hypothesis with White patients as the reference comparator was derived from past epidemiologic and historical work showing that White patients were the population in which current pulse oximeters were developed and calibrated.^15,16 A P value < .05 was considered to be significant.

Multivariable analyses were performed by logistic regression for each race and ethnicity group compared with White patients to examine the relationship between these variables with the odds of occult hypoxemia. The regression was adjusted for sex and measured Spo₂. A secondary analysis examining the rates of occult hypoxemia in patients of different races and ethnicities for Spo₂ measurements > 96% was conducted in a similar fashion. Statistical analyses were performed with Stata version 16.1 (release 16; StataCorp).

Results

Data for 3,569 individual ECMO runs from 324 centers were provided. No information on the specific make or model of pulse oximeter was available. There were 3,515 first ECMO runs, 52 second runs, one third run, and one fourth run. Bias, precision, limits of agreement, and root mean square error were calculated, using the 1,562 available Spo₂-Sao₂ pairs without Spo₂ range restriction. The mean difference between Spo₂ and Sao₂ (bias) was 0.6% with an SD of 7.5 (precision). The limits of agreement between Spo₂ and Sao₂ were 15.5% for the upper limit and –14.3% for the lower limit. The root mean square error was 7.5%.

Four racial and ethnic groups had sufficient observations (n ≥ 400) for initial inclusion in the occult hypoxemia analysis: White, Black, Hispanic, and Asian. A total of 1,644 records were excluded because of incomplete Spo₂-Sao₂ pairs. The final cohort had 372 ECMO initiations with Spo₂-Sao₂ in the 92% to 96% range of clinical interest (Fig 1). There were 253 men and 119 women in this cohort. For each racial and ethnic group analyzed, there was a larger proportion of men than women receiving ECMO therapy. COVID-19 status was available for n = 219 patients (58.9%) in the cohort. COVID-19 was confirmed by testing for n = 160 patients (73.1%), there was no clinical suspicion or testing information for n = 48 patients (21.9%), and n = 4 (5.0%) had a confirmed negative test result. The proportions of patients in each racial or ethnic group are detailed in Table 1. The mean pre-ECMO arterial blood gas results for the study cohort were as follows: pH, 7.3 (95% CI, 7.2-7.3); pCO₂, 60.9 (95% CI, 59.8-61.9); pO₂, 77.3 (95% CI, 74.5-80.1).

Figure 1.

Flow diagram for the creation of study cohort. Racial and ethnic categories (as defined by the Extracorporeal Life Support Organization) with fewer than 400 observations were excluded. The excluded groups were as follows: Middle Eastern or North African (n = 131), Native American (n = 55), Native Pacific Islander (n = 10), Multiple (n = 149), Other (50), and Unknown (n = 125). ECMO = extracorporeal membrane oxygenation; Sao₂ = arterial oxygen saturation; Spo₂ = peripheral oxygen saturation.

Table 1.

Patient Demographics of Study Cohort

Category	All		Included in Analysis
	No.	%	No.	%
Sex
Male	2,300	65.4	253	67.5
Female	1,215	34.6	119	32.5
Race
White	1,575	44.8	186	50
Black	446	12.7	51	13.7
Hispanic	515	14.7	70	18.8
Asian	459	13.1	65	17.5
Unknown or Other (excluded from analyses)	550	14.7	0	0

Table 2 and Figure 2 summarize the results of the bias, precision, and root mean square error analyses for the races and ethnicities included. For White patients, measurement bias was 0.3% with precision of 7.3, and root mean square error of 7.3%. For Black patients, measurement bias was worse at 1.7%, with precision worse at 10.2, and root mean square error worse at 10.4%.

Table 2.

Bias, Precision, Limits of Agreement, and Root Mean Square Error Data by Race and Ethnicity^a

	All Available Spo2-Sao2 Pairs
	All (N = 1,562)	White (n = 675)	Black (n = 195)	Hispanic (n = 238)	Asian (n = 243)
Bias, %	0.6	0.3	1.7	0.8	–0.3
Precision, SD	7.5	7.3	10.2	6.4	5.8
Upper limit of agreement, %	15.5	14.9	22.2	13.7	11.3
Lower limit of agreement, %	–14.3	–14.2	–18.7	–12.1	–11.8
Root mean square error, %	7.5	7.3	10.4	6.5	5.8

Sao₂ = arterial oxygen saturation; Spo₂ = peripheral oxygen saturation.

^aLower values for bias, precision, and accuracy indicate a higher degree of correlation between the two measurement modalities presented. Note that these are calculated for all available Spo₂-Sao₂ pairs, regardless of Spo₂ value.

Figure 2.

Difference between Spo₂ and Sao₂ against the mean of Sao₂ across different races and ethnicities. Bland-Altman plots were produced using the following command in Stata version 16.1 [blandaltman Spo₂ Sao₂]. See e-Appendix 1 for complete analytic code. Sao₂ = arterial oxygen saturation; Spo₂ = peripheral oxygen saturation.

Occult Hypoxemia Rates at Spo₂ Between 92% and 96%

We analyzed 372 Spo₂-Sao₂ pairs representing 186 White, 51 Black, 70 Hispanic, and 65 Asian patients. Among pre-ECMO Spo₂ measurements between 92% and 96%, 19 of 186 (10.2%; 95% CI, 6.2%-15.3%) measurements in White patients had Sao₂ less than 88%. Compared with this group, 11 of 51 (21.5%; 95% CI, 11.3%-35.3%) measurements in Black patients (P = .031 vs White by χ²), 6 of 70 (8.6%; 95% CI, 3.2%-17.7%) measurements in Hispanic patients (P = .693 vs White), and 6 of 65 (9.2%; 95% CI, 3.5%-19.0%) measurements in Asian patients (P = .820 vs White) had Sao₂ less than 88% (Fig 3). Logistic regression with White race as the reference category demonstrated that acutely ill Black patients with respiratory failure and about to receive ECMO therapy, with an Spo₂ of 92% to 96%, had a statistically significantly higher risk of Sao₂-defined occult hypoxemia (OR, 2.57; 95% CI, 1.12-5.92) compared with White patients (P = .026).

Figure 3.

Prevalence of Sao₂ of ≤ 88% despite Spo₂ of 92% to 96% across different races and ethnicities. Sao₂ = arterial oxygen saturation; Spo₂ = peripheral oxygen saturation.

Occult Hypoxemia Rates at Spo₂ > 96%

Among pre-ECMO Spo₂ measurements > 96%, 7 of 101 measurements in White patients (6.9%; 95% CI, 2.8%-13.8%) had Sao₂ less than 88%. Using the same definitions, 7 of 34 measurements in Black patients (20.6%; 95% CI, 8.7% to 37.9%), 3 of 37 measurements in Hispanic patients (8.1%; 95% CI, 0.6%-18.2%), and 1 of 24 measurements in Asian patients (4.2%; 95% CI, 0.1%-21.1%) had Sao₂ ≤ 88%. Logistic regression with “White” as the reference category (adjusting for Spo₂ and sex) demonstrated that Black patients with Spo₂ > 96% had a statistically significant higher risk of Sao₂ ≤ 88% (OR, 3.52; 95% CI, 1.12-11.10) compared with White patients (P = .032). The risk of Sao₂ ≤ 88% for Hispanic and Asian patients was similar to that of White patients for Spo₂ > 96%.

Discussion

This study in patients so severely ill they would shortly be placed on ECMO for respiratory failure, demonstrates significant differences in pulse oximetry ability to detect occult hypoxemia between critically ill White and Black patients. Differences in terms of Sao₂—the parameter to which pulse oximeters are, in principle, calibrated—were large and estimated precisely. Compared to White patients, critically ill Black patients receiving mechanical ventilatory support had increased risk of occult hypoxemia defined as low Sao₂ despite a pulse oximetry reading within ranges of 92% to 96%. Similarly, higher odds of occult hypoxemia amongst Black patients were noted even at higher Spo₂ levels. Bland-Altman plots similarly demonstrated racial differences in bias, precision, and (consequently) root mean square error—all worse in Black patients. There was no evidence of such diminished detection of occult hypoxemia in Hispanic and Asian patients compared with White patients.

Earlier publications demonstrated similar increased odds of occult hypoxemia in patients receiving supplemental oxygen therapy,5, 6, 7 which prompted investigators to question the reliability of pulse oximetry in the assessment of a Black patient’s oxygen requirement.⁸ We demonstrate a similar measurement discordance in acutely ill Black patients with extreme respiratory failure, with pulse oximetry routinely underestimating the severity of hypoxemia in Black patients as measured by Sao₂. This investigation expands on these findings by demonstrating even higher odds of occult hypoxemia for Black patients at Spo₂ levels > 96%. These findings have implications for minute-to-minute treatment of mechanically ventilated patients.

Mechanical ventilatory support adjustments are often triggered by changes in pulse oximetry. Although arterial blood gas measurements are frequently obtained in this setting, delayed intervention given occult hypoxemia on pulse oximetry is possible. Our findings join the body of evidence suggesting that the use of Spo₂ to rule out acute hypoxia, particularly in Black patients with other concerning signs or symptoms, should be reevaluated given its diminished reliability. Also concerning is the limitation of the technology to identify clinically significant hypoxemia for Black patients at higher Spo₂ levels. At an Spo₂ range of > 96%, most physicians would be reassured about the oxygenation status of even critically ill patients with respiratory failure, but our findings indicate that reassurance may be unwarranted. Given the widespread use of pulse oximetry, a higher degree of suspicion of hypoxemia for patients of color should be considered, with consideration of more frequent use of arterial blood gas examination for patients at risk until a technological fix is implemented.

Although there is no available evidence reporting low pulse oximetry accuracy in Hispanic patients, previous observational investigations have noted variations in pulse oximetry sensitivity in a cohort of patients of Asian descent.¹⁷ We did not identify a statistically significant difference in the occult hypoxemia rates of Hispanic and Asian patients in this study. Bias, precision, limits of agreement, and accuracy analyses for the entire study cohort demonstrated a similar accuracy between the two measurement methods (Spo₂ and Sao₂) for these groups as compared with White patients. There are a few possible explanations for these results. Hispanic and Asian patients belong to heterogeneous groups, in which individuals frequently identify with multiple ethnicities and races and could have a wider range of skin colors,^18,19 albeit frequently less pigmented compared with Black individuals—or pigmentation may not be the essential difference in skin that drives these findings. Despite the suspected connection between lower skin pigmentation in these groups and the negative findings in the Hispanic and Asian members of this cohort, we must consider of how the mechanism of pulse oximetry miscalibration might play a role in the care of darker pigmented Asian and Hispanic patients. Although not captured by these data, self-identified Asian and Hispanic patients with darker skin, Black ancestry, mixed race, or skin color other than white could be affected by pulse oximetry calibration as well.

It is worth restating that the increased rates of occult hypoxemia in Black patients is a phenomenon that has been intermittently recognized by the medical community for the past three decades. Universal calibration of pulse oximeters, using data from patients of different skin tones, was first suggested by Jubran and Tobin in 1990,⁷ but this has yet to be accomplished or prioritized. Systematic underdiagnosis of hypoxemia in Black patients is likely attributed to technical design issues, but the decision to tolerate the miscalibration for Black patients has been collective despite the available evidence. Steps toward addressing this disparity must be taken. On January 25, 2021, Senators Elizabeth Warren (D-Mass.), Cory Booker (D-N.J.), and Ron Wyden (D-Ore.) sent a letter to the commissioner of the FDA requesting an expedited review of device sensitivity in patients of different races.²⁰ Although initial statements from the FDA in response to the findings of Sjoding et al²¹ reported confidence in the accuracy of hospital-based oximeters,²² a follow-up safety communication reminded physicians that skin tone can contribute to pulse oximetry inaccuracy,²³ and others have called for changes in devices.^24,25

This investigation has several limitations. ELSO data are submitted voluntarily, and ECMO centers are often well-resourced hospitals in metropolitan areas. Because objective data on skin tone were not available, surrogate use of race limits our conclusions. The observed data represent what bedside physicians considered simultaneous collection, but time stamps were not available for the Spo₂ and Sao₂ data used in this analysis, which is an important limitation given that oxygen levels may fluctuate significantly in a matter of minutes.²⁶ In general, however, delays between the recorded values of Sao₂ and Spo₂ would add nondifferential measurement error, which would bias the data toward findings of no difference in rates of occult hypoxemia between racial and ethnic groups. For measurement error from time delays alone to account for the observed and consistent racial differences in this study and many others, there would need to be a consistently greater delay in arterial blood gas measurement after pulse oximeter recording for Black patients than for White patients. This measurement delay would have to be present in both our data and in published results from contemporary investigations using both single-center and multiinstitutional datasets,⁸ studies from the last two decades,^6,27 and those dating back as early as the 1990s.⁷ We are unaware of any data demonstrating such differentials. Nonetheless, prospectively collected synchronous measures will be required to draw final conclusions on this phenomenon.

There are additional limitations to this analysis. Risk adjustment for severity of illness was not possible with the available data. Because low circulation during critical illness is theorized as a potential mechanism of pulse oximetry disruption,²⁸ this is an important limitation. Similarly, smoking status or diabetes can also affect pulse oximetry measurements, and the data used in this study did not contain patient comorbidity information. Of note, the most recent previous investigation on this topic conducted two subgroup analyses to test whether heavy smoking or diabetes influenced the occult hypoxemia rates reported in Black and White patients, and the investigators found no differences.⁸ The pulse oximetry device in use for each of these patients was not recorded. It is unknown if variation in device performance was considered to be of sufficient clinical importance to practicing physicians to be taken into consideration. Likewise, these data are hand-abstracted to represent matching Spo₂ and arterial blood gas measurements, but abstraction errors are possible—although it is not clear why they would be systematically different by race or ethnicity.

Interpretation

Pulse oximetry has limited and racially differential usefulness for the assessment of hypoxemia in acutely ill patients. Compared with White patients, the prevalence of occult hypoxemia was comparable for Hispanic and Asian patients but significantly higher in Black patients when assessed by Sao₂ on arterial blood gas measurement.