Estimated Pao₂: A Continuous and Noninvasive Method to Estimate Pao₂ and Oxygenation Index

Supplemental Digital Content is available in the text.

BACKGROUND:

Pao₂ is the gold standard to assess acute hypoxic respiratory failure, but it is only routinely available by intermittent spot checks, precluding any automatic continuous analysis for bedside tools.

OBJECTIVE:

To validate a continuous and noninvasive method to estimate hypoxemia severity for all Spo₂ values.

DERIVATION COHORT:

All patients who had an arterial blood gas and simultaneous continuous noninvasive monitoring from 2011 to 2019 at Boston Children’s Hospital (Boston, MA) PICU.

VALIDATION COHORT:

External cohort at Sainte-Justine Hospital PICU (Montreal, QC, Canada) from 2017 to 2020.

PREDICTION MODEL:

We estimated the Pao₂ using three kinds of neural networks and an empirically optimized mathematical model derived from known physiologic equations.

RESULTS:

We included 52,879 Pao₂ (3,252 patients) in the derivation dataset and 12,047 Pao₂ (926 patients) in the validation dataset. The mean function on the last minute before the arterial blood gas had the lowest bias (bias –0.1% validation cohort). A difference greater than or equal to 3% between pulse rate and electrical heart rate decreased the intraclass correlation coefficients (0.75 vs 0.44; p < 0.001) implying measurement noise. Our estimated Pao₂ equation had the highest intraclass correlation coefficient (0.38; 95% CI, 0.36–0.39; validation cohort) and outperformed neural networks and existing equations. Using the estimated Pao₂ to estimate the oxygenation index showed a significantly better hypoxemia classification (kappa) than oxygenation saturation index for both Spo₂ less than or equal to 97% (0.79 vs 0.60; p < 0.001) and Spo₂ greater than 97% (0.58 vs 0.52; p < 0.001).

CONCLUSION:

The estimated Pao₂ using pulse rate and electrical heart rate Spo₂ validation allows a continuous and noninvasive estimation of the oxygenation index that is valid for Spo₂ less than or equal to 97% and for Spo₂ greater than 97%. Display of continuous analysis of estimated Pao₂ and estimated oxygenation index may provide decision support to assist with hypoxemia diagnosis and oxygen titration in critically ill patients.

Acute hypoxic respiratory failure is a common reason of admission in PICUs and is associated with a high mortality and morbidity (1–3). The gold standard to assess hypoxemia severity in pediatric requires an arterial blood gas (ABG) to measure the oxygen partial pressure (Pao₂) and to calculate the oxygenation index (OI = [Fio₂ × 100 × mean airway pressure]/Pao₂) (4). Although Pao₂ is the reference, it is an invasive method available only for selected patients. Furthermore, only intermittent spot checks are routinely possible, precluding accurate continuous evaluation. Oxygen is most a rapidly and widely changing analyte. Clinically, this may lead to periods of undetected hypoxemia (5). Pulse oximetry (Spo₂) is the most common continuous surrogate, but the relationship among Spo₂, the Sao₂, and Pao₂ is complex, especially when Spo₂ is greater than 97% (6, 7). Therefore, pediatric and adult acute respiratory distress syndrome (ARDS) guidelines recommend to use only Spo₂ less than or equal to 97% to assess hypoxemia severity (4, 8). This is an important limitation for a continuous estimation in the ICU as a majority of patients have Spo₂ above 95% (3, 9–13).

First, equations describing oxyhemoglobin dissociation curve had important limitations (14–16). They cannot estimate the Pao₂ when Sao₂ is close to 100% (asymptotic relationship) and are significantly biased for high Sao₂ values (17). Gadrey et al (17) modified equation by Hill (15) to address that limitation and showed improved performance in hospitalized non-ICU patients. Furthermore, all existing equations used a value of P₅₀ (Pao₂ value when Sao₂ = 50%) that was measured on a few healthy adults (15), but this parameter is known to be different in a critically ill population (18). Brockway showed that Pao₂ can be estimated from Spo₂ in neonates (19). Other equations estimating the Pao₂/Fio₂ ratio from the Spo₂/Fio₂ ratio have been developed (20–24). Although Pao₂ can be mathematically solved, fit was generally poor (17, 20, 25).

Only few studies have used automatic continuously collected streams of data (6). This rich data source is helpful to precisely pair ABGs with other continuous monitoring data. Because no human is involved in the process, they require accurate automatic detection mechanisms to exclude potentially erroneous data (26). Different time windows and aggregation functions have been described in the literature from –60 minutes to the near-exact time (6, 17, 23, 27), but no comparative methods have been published in the literature to our knowledge.

This study aimed to validate a method to continuously estimate the hypoxemia severity, valid for all Spo₂ values, by estimating the Pao₂ using noninvasive monitoring information. Such a method would facilitate early detection and monitoring of hypoxemia.

MATERIAL AND METHODS

We collected two independent datasets at Boston Children’s Hospital (BCH, Boston, MA) ICU (excluding the cardiac unit) and at Sainte-Justine Hospital (SJH, Montreal, QC, Canada) PICU. Both are academic referral centers that treat a broad range of highly specialized and common pediatric critical conditions, and they are approximately at the same altitude (< 50 m above sea level). We used the BCH cohort to develop new models and the SJH cohort to validate them independently. We included any patients admitted in these PICUs who had an ABG with concomitant continuous Spo₂ measures. ABG were excluded if more than 20% of the Spo₂ were missing before the ABG. This study was approved by both BCH (reference IRB-P00021911) and SJH (reference 2018-1587) institutional review boards and waived the need for informed consent. We followed the 2020 standards for prediction models in critical care (28) and the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis guidelines for development and validation of predictive models (29).

Data Collection

Data were extracted from secured servers located in each hospital. At BCH, data were retrieved at a 5-second frequency from the T3 database (Etiometry, Boston, MA) from October 2011 to December 2019. At SJH, data were collected from January 2017 to January 2020 and retrieved at 1-second rate from a software developed in-house (30). We collected Spo₂, pulse rate, and heart rate from –20 to +10 minutes around the ABG. SJH data were resampled (closest 5-s time) to have a simulated 5-second frequency. In both centers, ABGs were analyzed using Radiometer ABL90 FLEX (Radiometer Medical ApS, Copenhagen, Denmark). In both centers, Spo₂ was measured using Masimo (Masimo, Irvine, CA) probes (RD SET Neo and RD SET Inf). Body locations of the probe were not available as continuous data. Once extracted, data were stored and managed on a secured local PostgreSQL 12.1 server (The PostgreSQL Global Development Group) and were analyzed using Python 3.7.7 (Python Software Foundation, Wilmington, DE) and R 3.6.3 (R Core Team, Vienna, Austria) scripts. Neural networks models were built with Keras 2.3.0 on TensorFlow 2.1.0 (Google, Mountain View, CA). In the models, we only used Spo₂, heart rate, and pulse rate.

Statistical Analyses and Aggregation Function

Categorical data were described as count and percentage, and continuous data as median and interquartile range. CIs and p values were calculated using 10,000 bootstrap repetitions (31). We used the intraclass correlation coefficient (ICC) as a global metrics for agreement, accuracy, and precision (32). We used the two-way mixed effects ICC, also known as “ICC” (3,1) in the Shrout and Fleiss (33) classification. The accuracy was estimated using the bias defined as the mean of the differences (mean biased error or fixed bias) (34). Precision was estimated with the mean absolute error (MAE) and the limits of agreements (95% CI on the differences) (34). Agreement was shown with Bland-Altman plots grouped by a range of Spo₂. We also calculated the proportional bias, defined by the slope on the Bland-Altman plot. Discrimination was estimated using the area under the receiving operating characteristic curve (AUROC) for mild (OI ≥ 4) and severe (OI ≥ 16) hypoxemia.

We used the direct relationship between Spo₂ and Sao₂ to find the best Spo₂ aggregation. We compared mean and median function and different time windows from an exact time match to a 20-minute period before the Pao₂. We chose the aggregation method that minimized the bias.

Spo₂ Quality Assessment

In ICU, heart rate is continuously monitored using an electrocardiogram monitor and pulse oximeters. As both measurements should be identical in most clinical situations, the similarity is a way to validate the quality of the Spo₂ measures as already used by our team (35). We investigated if a higher difference in percentage between the pulse rate and the electrical heart rate (PR-HR) is associated with a greater MAE and lower ICC.

Empirical Optimization Model

As general equation by Hill (15) is complex to optimize by traditional mathematical techniques (Fig. 1, equation 1), we used an empirical method that calculated nearly all the combinations in the plausible ranges for Spo₂ (80–100%), P₅₀ (20.0–40.0 mm Hg), m (0.950–0.999), and Hill (15) number n_H (2.50–3.50). Then, we identified the optimal combination that minimized the mean squared error (MSE) in the derivation dataset. This approach, also called exhaustive search, brute-force search, or grid search, is often used in machine learning hyperparameter optimization to solve a multidimensional problem (36). However, this optimization method requires a significant amount of time and computing resources.

Figure 1.

Known equations to estimate Pao₂ using Sao₂ or Spo₂.

Neural Network Models

In order to train the neural network model, we randomly split at the patient level the derivation cohort (BCH) into a train (85%) and a validation (15%) dataset. Once finalized, models were tested on the external distinct cohort (SJH). We used the last 20 minutes before Pao₂ to build a standardized multidimensional matrix of Spo₂, PR-HR (241 × 3, chronologically ordered). Data were normalized relatively to the minimum and maximum values of the training cohort. Missing values were imputed with the last value available. We compared three common types of networks able to assess serial or repeated measurements: multilayer perceptron, convolutional neural networks, and long short-term memory (LSTM) networks (37–39). Architecture, optimizer, and hyperparameters were adjusted to minimize the MSE.

Hypoxemia Severity Validation

Following the Pediatric Acute Lung Injury Consensus Conference (PALICC) (4), hypoxemia severity was defined using the OI: mild (values between 4 and 7.9), moderate (8–15.9), and severe (OI ≥ 16). If no arterial sample is available, PALICC guidelines recommend using the oxygen saturation index (OSI) where Pao₂ is replaced by Spo₂. OSI has different thresholds (mild 5–7.4, moderate 7.5–12.2, and severe ≥ 12.3) extrapolated from equations that estimate the OI (7). Mean airway pressures and Fio₂ were extracted at the exact same time the ABG was drawn and were the same for all models. We selected the best Pao₂ model to estimate the OI and compared its performance with the OSI on severity classification (kappa) and hypoxemia (mild and severe) discrimination (AUROC). Results were separated into two Spo₂ categories, below or equal to 97% and above 97%, even if OSI has not been validated in that upper range.

RESULTS

We included 64,926 Pao₂ (4,178 patients) with 46 million concomitant Spo₂ values. Patients and ABG characteristics in each cohort are described in Table 1. In Supplemental Digital Content (SDC 1, http://links.lww.com/CCX/A807), we showed that P₅₀ varied from 20 to 40 mm Hg but was similar between age groups.

TABLE 1.

Patients and Arterial Blood Gases Characteristics

Variables	Boston Children’s Hospital	Sainte-Justine Hospital
	Derivation Cohort	Validation Cohort
	n (%) or median (IQR)	n (%) or median (IQR)
Arterial blood gases, n	52,879	12,047
Patients (total), n	3,252	926
Adult patients (≥18), n	308 (9.5)	15 (1.6)
Arterial blood gases per patient	5 (2–17)	6 (2–13)
Age (yr)	2.4 (0.3–11.2)	1.4 (0.1–7.7)
Gender, female	23,545 (44.5%)	5,420 (45%)
pH	7.4 (7.3–7.4)	7.4 (7.4–7.5)
Pao2 (mm Hg)	97 (75.4–128)	94.6 (72.6–129)
Spo2 (%)	98 (96–100)	98.1 (95.5–100)
Lactate (mmol/L)	1.2 (0.8–1.8)	1.2 (0.8–2)
Gases under invasive ventilation	14,378 (27.2%)	7,547 (62.6%)
Mean airway pressure (cm H2O)	11.4 (9–14.4)	10.5 (8.5–13)
Inspired oxygen fraction	0.4 (0.3–0.6)	0.4 (0.3–0.6)
Oxygenation index	4.5 (2.8–8.1)	4.3 (2.6–7.9)

IQR = interquartile range.

Aggregation Methods and Data Quality

We compared different time windows around the Sao₂ measurement (SDC 2, http://links.lww.com/CCX/A807) from –20 minutes to the exact time of the ABG. Although the differences were small, closer results to the exact time had a lower bias in both derivation and validation cohorts. Median function provided very slightly more biased results than the mean in both datasets.

Our results also showed that any difference between the pulse rate read by the pulse oximeter and the electrical heart rate was associated with a lower correlation and higher bias in the derivation cohort (SDC 3, http://links.lww.com/CCX/A807). The impact was more significant when the difference was greater than or equal to 3%. We confirmed the results in the validation cohort when the difference was less than 3% (ICC, 0.75; 95% CI, 0.74–0.75), greater than or equal to 3% (ICC, 0.44; 95% CI, 0.38–0.49), or unable to read a pulse (ICC, 0.49; 95% CI, 0.39–0.57). MAE increased from 2.2 (difference < 3%) to 3.4 (difference ≥ 3%) and 4.2 (no pulse read). ICC improved from 0.73 (95% CI, 0.72–0.73) to 0.75 (95% CI, 0.74–0.75; p < 0.01) after excluding those values, even if these results represented a minority of the measurements (51,777, 866, and 236, respectively, in the derivation cohort and 11,271, 582, and 194 in the validation cohort).

Empirical Optimization Model

Over the 1.015 million combinations of parameters using the equation structure by Hill (15), we found that results that minimized the MSE in the derivation cohort were P₅₀ = 27.8 mm Hg, m = 0.99, and n_H = 2.8 (Fig. 1, equation 5). Performance in the derivation cohort is presented in SDC 4 (http://links.lww.com/CCX/A807). Results for the validation cohort compared with the other models are shown in SDC 5 (http://links.lww.com/CCX/A807) and illustrated in Figures 2 and 3. The _Sauthierestimated Pao₂ (ePao2) equation outperformed all existing models for ICC (0.38 vs 0.35 and lower, p = 0.004) and slopes (1.01 vs 1.14 and further from one; p < 0.001).

Figure 2.

Grouped Bland-Altman plots showing models bias (mean of the errors, black dots) and limits of agreement (boxplot extremes) for different Spo₂ categories. Severinghaus (15) and Brockway and Hay (19) were not validated for Spo₂ > 97%. CNN = convolutional neural network, LSTM = long short-term memory network, MLP = multilayer perceptron.

Figure 3.

Pao₂ estimation on the derivation dataset using different models.

Neural Networks

The three networks had approximately similar performance (Fig. 2; and SDC 5, http://links.lww.com/CCX/A807), but the LSTM model was slightly less correlated with the observed Pao₂ than the others (ICC, 0.32 vs 0.35; p < 0.01). Bias was significantly lower (p < 0.01) for the neural networks compared with the mathematical equations (Fig. 2). The neural network structures are detailed in SDC (http://links.lww.com/CCX/A807).

Hypoxemia Severity

Pediatric hypoxemia is determined by the OI or the OSI if no arterial sample is available. We compared the estimated OI based on _SauthierePao₂ with the OSI (Table 2). We used the same filtered Spo₂ by pulse rate and heart rate differences for both models. We found that eOI had better hypoxemia severity classification than the OSI for both Spo₂ less than or equal to 97% (Kappa 0.79 vs 0.60; p < 0.001, respectively) and Spo₂ greater than 97% (Kappa 0.58 vs 0.52; p < 0.001, respectively). We also found that the eOI had a higher global agreement (ICC), lower fixed bias, higher correlation (slope), and better precision (limits of agreements difference and coefficient of determination) than OSI for both Spo₂ less than or equal to 97% and greater than 97%. Furthermore, the _SauthierePao₂ had a similar classification performance (Kappa) for Spo₂ greater than 97% than OSI has for Spo₂ less than or equal to 97% (0.58 vs 0.60; p = 0.06). Discrimination for mild hypoxemia or more severe (OI ≥ 4) and for severe hypoxemia (OI ≥ 16) was similar for ePao₂ and OSI. Bland-Altman plots and AUROC plots are shown in Figure 4 and SDC 6 (http://links.lww.com/CCX/A807), respectively.

TABLE 2.

Hypoxemia Severity Assessment

Metric (95% CI)	Spo2 ≤ 97%			Spo2 > 97%
	OI Using OSI	OI Using ePao2	p	OI Using OSI	OI Using ePao2	p
Weighted kappa	0.60 (0.58–0.62)	0.79 (0.77–0.81)	< 0.001	0.52 (0.50–0.54)	0.58 (0.56–0.59)	< 0.001
Intraclass correlation coefficient	0.91 (0.89–0.92)	0.93 (0.92–0.94)	0.029	0.62 (0.60–0.64)	0.72 (0.69–0.75)	< 0.001
Mean absolute error	3.68 (3.57–3.79)	1.90 (1.80–2.01)	< 0.001	3.58 (3.47–3.68)	1.60 (1.54–1.67)	< 0.001
Coefficient of determination (R2)	0.84 (0.82–0.86)	0.87 (0.85–0.89)	0.043	0.51 (0.48–0.55)	0.55 (0.51–0.59)	0.170
Slope	0.81 (0.78–0.84)	1.02 (0.98–1.06)	< 0.001	0.42 (0.39–0.44)	0.93 (0.88–0.99)	< 0.001
Fixed bias	2.01 (1.85–2.17)	0.15 (0.02–0.28)	< 0.001	–0.48 (–0.63 to –0.34)	0.13 (0.05–0.21)	< 0.001
Proportional bias	–0.12 (–0.16 to –0.09)	0.09 (0.06–0.13)	< 0.001	–0.61 (–0.67 to –0.55)	0.26 (0.20–0.34)	< 0.001
Upper and lower limits of agreements difference	0.32 (0.30–0.34)	0.25 (0.23–0.28)	< 0.001	0.29 (0.28–0.31)	0.16 (0.15–0.18)	< 0.001
AUROC for mild hypoxemia (OI ≥ 4)	0.92 (0.91–0.93)	0.93 (0.92–0.94)	0.122	0.87 (0.86–0.88)	0.88 (0.87–0.89)	0.052
AUROC for severe hypoxemia (OI ≥ 16)	0.98 (0.97–0.98)	0.98 (0.98–0.99)	0.055	0.96 (0.95–0.97)	0.96 (0.95–0.97)	0.986

AUROC = area under the receiving operating characteristic curve, ePao₂ = estimated Pao₂, OI = oxygenation index, OSI = oxygenation saturation index.

Figure 4.

Bland-Altman plots for estimated oxygenation index (OI) using ePao₂ and oxygenation saturation index (OSI). ePao₂ = estimated Pao₂.

DISCUSSION

We have developed and validated a predictive model that can automatically and continuously estimate the Pao₂ using only noninvasive data that are commonly measured in ICU (Spo₂ and PR-HR) across two independent cohorts. A difference between the pulse read using the oximeter and the electrical heart rate was a good indicator of erroneous values, showing lower Sao₂-Spo₂ agreement. Spo₂ is optimally aggregated with a mean function on a 1-minute time window. A new mathematical equation (_SauthierePao₂) outperformed three existing mathematical equations for estimating Pao_2, and application of deep learning algorithms was unable to further improve performance. At Spo₂ values 99% and above, estimation of Pao₂ was still unbiased, but imprecise across all models. When ePao₂ is used to estimate OI, a better hypoxemia estimation is found than that with OSI. Although the improvement was modest, those findings are clinically relevant as they will allow building real-time clinical decision support systems that aim to automatically monitor degree of hypoxemia in patients on life support. Furthermore, OI and eOI will have the same thresholds, facilitating the interpretation. They will also allow continuous quantitative analysis of hypoxemia, such as duration and severity of each hypoxemia episode. Those results may also decrease blood draws and resource needs in a vulnerable population to anemia.

This study confirmed the contribution by Gadrey et al (17) to the equation by Severinghaus (15) and Hill (15) (SDC 5, http://links.lww.com/CCX/A807) that changing m = 1 to m = 0.99 improved most of the bias seen for Spo₂ values above 95%. However, we found no studies in the literature that tried to optimize Severinghaus (15) P₅₀ value and n_H for a pediatric critical care population. In our study, these new parameters improved the agreement (ICC), the bias, and the slope, for all the clinically significant ranges of Spo₂. As we observed no clear association between age and P₅₀, it was unlikely that age would improve the model. However, precision remained low as a single value of Spo₂ can correspond to a broad range of Pao₂ (6). Future models may integrate patients’ previous ABGs to improve continuous Pao₂ estimation precision. To improve reading in higher oxygenation values, other solutions such as the oxygen reserve index (40) used Svo₂ changes when Spo₂ is saturated. However, it requires specific proprietary material. To our knowledge, this study is the first to show different ways to retrospectively improve precision using continuous data: selecting a short time window aggregated by a mean function and exclude erroneous values when PR-HR is different.

In this study, we showed that three types of deep learning models were able to capture information from the continuous noninvasive input data resulting in the estimations with the lowest bias. However, the difference with the _SauthierePao₂ equation was small (between 2 and 4 mm Hg) and is probably not clinically significant. Nonetheless, considering that a simple and an explicit mathematical equation can perform similarly, or better, neural networks do not seem to be worth the complexity they imposed, especially when ICC is low.

Our study has strengths. To our knowledge, this is the first and largest study to explore the Pao₂-Spo₂ relationship in the critical care field using only automatic continuously collected data. The ability to continuously assess hypoxemia severity is essential to improve early identification. Second, our model (Fig. 1, equation 5) is simple and explicit, and has been validated in a distinct external cohort. Even though most of the cohorts are children, the results did include adult patients (8%), and the _SauthierePao₂ parameters were close to the ones validated on adults, suggesting a good generalizability. Third, this study suggests that Spo₂ greater than 97% can still provide significant information regarding the hypoxemia severity, as also shown in severity scoring assessment (13). The _SauthierePao₂ had a similar classification performance (Kappa) for Spo₂ greater than 97% than OSI has for Spo₂ less than or equal to 97%. As precision is still lower for high Spo₂, it still supports the recommendation to minimize Fio₂ for appropriate Spo₂ targets (4).

However, our study has limitations. First, the retrospective nature of the study precludes the validation of the metadata (especially patient identification, timestamp, arterial labeling, and body location of the probe). However, those ABGs represented real-life situations, and both hospitals are committed to the higher standard of care including the validation of research databases (41). Second, measuring arterial oxygenation is a clinical decision and is not necessarily indicated for all patients. Thus, the patients included in this study are expected to be more severe than those who did not get any arterial measurement. However, overcoming this limitation in prospective trial would likely require invasive procedures in patients that are not warranted clinically. The large number of patients and ABG included in this study should mitigate this limitation. Third, we used a broad PICU population to derive and to validate the models. One would prefer to revalidate the model in a specific ARDS cohort before using it in a clinical decision support system dedicated to this entity.

CONCLUSIONS

In this two-center study with over 4,000 patients and nearly 65,000 ABGs paired with continuous data, we developed and validated a simple mathematical equation and methods to filter Spo₂ streams accurately and automatically in order to estimate the Pao₂ for patients admitted in critical care units using only continuous and noninvasive data. Despite a limited precision for high Spo₂, those results allow a precise OI estimation even for Spo₂ greater than 97% and are important for all future bedside clinical decision tools aiming at improving oxygenation titration in real time in critical care.

ACKNOWLEDGMENTS

We thank Dr. Shira Fischer, MD, PhD, for her advice and Dr. Craig Smallwood, RRT, PhD (deceased), for his assistance with the T3 dataset.