Innovative approaches in imaging photoplethysmography for remote blood oxygen monitoring

Abstract

Peripheral Capillary Oxygen Saturation (SpO₂) has received increasing attention during the COVID-19 pandemic. Clinical investigations have demonstrated that individuals afflicted with COVID-19 exhibit notably reduced levels of SpO₂ before the deterioration of their health status. To cost-effectively enable individuals to monitor their SpO₂, this paper proposes a novel neural network model named “ITSCAN” based on Temporal Shift Module. Benefiting from the widespread use of smartphones, this model can assess an individual’s SpO₂ in real time, utilizing standard facial video footage, with a temporal granularity of seconds. The model is interweaved by two distinct branches: the motion branch, responsible for extracting spatiotemporal data features and the appearance branch, focusing on the correlation between feature channels and the location information of feature map using coordinate attention mechanisms. Accordingly, the SpO₂ estimator generates the corresponding SpO₂ value. This paper summarizes for the first time 5 loss functions commonly used in the SpO₂ estimation model. Subsequently, a novel loss function has been contributed through the examination of various combinations and careful selection of hyperparameters. Comprehensive ablation experiments analyze the independent impact of each module on the overall model performance. Finally, the experimental results based on the public dataset (VIPL-HR) show that our model has obvious advantages in MAE (1.10%) and RMSE (1.19%) compared with related work, which implies more accuracy of the proposed method to contribute to public health.

Introduction

SpO₂ is an important indicator to reflect the human cardiovascular and cerebrovascular function for clinical medical monitoring¹. In addition, many lung diseases can be detected early by monitoring abnormal values of SpO₂. For example, it has been reported that some COVID-19 patients did not show respiratory symptoms, only clinically significant hypoxemia². Thus, timely detection of changes in SpO₂ is of great essential to the protection of life and health. Conventional SpO₂ detection methods can be divided into two types: arterial blood gas (ABG) analysis³ and photoplethysmography (PPG)⁴. ABG needs to collect blood from the human body first and measure the SpO₂ partial pressure through electrochemical analysis to obtain the SpO₂ saturation. PPG generally uses a finger-clip photoelectric sensor to calculate the SpO₂ saturation by measuring the reflected light intensity of two incident light sources with different wavelengths, such as the red light wavelength (660–700 nm) and the infrared (IR) light wavelength (800–950 nm)⁵. Both methods belong to the contact measurement ways, which suggests they are not suitable for patients requiring non-contact monitoring even for skin damage. To achieve user-friendly non-contact measurements, Imaging PhotoPlethysMography (IPPG) is widely used due to its advantages of capturing physiological measurements with digital cameras and ambient light⁶.

There have been some studies on the relationship between IPPG and SpO₂, but lots of them are still in their infancy. The Radio of Radios method (ROR) by regression is a commonly used method to predict SpO₂, which transitions the range of datasets from the infrared to the visible domain. For example, Kong et al. use a band-pass filter to lock red and green channels as input variables, and predict SpO₂ without introducing additional light sources⁷; Al-Naji et al. introduce complete Ensemble Empirical Mode Decomposition (EEMD) and Independent Component Analysis (ICA) to extract the features of facial video as the input of ROR to predict SpO₂⁸; Sun et al. improve the ROR algorithm and uses Multiple Linear Regression (MLR) to extract red and blue channel for predicting SpO₂⁹; Luo et al. design a special sampling device based on the facial video to obtain more comprehensive ROR input variables¹⁰; Rosa et al. utilize Eulerian Video Magnification (EVM) to preprocess facial video to amplify changes in reflected light for more sensitive input variables¹¹. Shao et al.¹² preset the ROR standard model to align individuals with the standard model. Hamoud et al.¹³ innovatively used XGBoost Regression to achieve SPO2 measurement based on ROR. Chan et al.¹⁴ applied the ROR for two combinations to access the accuracy under four lighting conditions. The main innovations of researchers using conventional methods are reflected in the preprocessing of data, while there are few algorithm innovations for predicting SpO₂. Another commonly used method for predicting SpO₂ is called the Pulse Blood Volume (PBV) method¹⁵. Unlike ROR, which needs to capture the DC and AC components of two different exact wavelengths of visible light, PBV utilizes the different relative pulsatile amplitudes in the color channels to distinguish changes in blood volume caused by SpO₂. For example, Van Gastel et al. proposed an Adaptive PBV method (APBV) for BVP signals extraction from multiple ROIs, followed by their mapping to different SpO₂ levels ranging from 65 to 100% with an interval of 5%¹⁶. Although PBV is simpler in terms of data processing, this comes at the cost of reduced accuracy. Moreover, the above two conventional methods have poor anti-interference ability and have extremely high requirements for the quality of the datasets.

With the rapid development of machine learning, especially deep neural network, a rapidly increasing number of researchers have applied them to biomedical scenarios and daily health monitoring. In particular, deep learning techniques have matured in predicting HRV metrics¹⁷, so deep learning algorithm is also used for the contactless SpO₂ measurement. Zheng¹⁸ proposed a hybrid model, which has a mixed network architecture consisting of CNN, convolutional long short-term memory (convLSTM), and video vision transformer (ViViT). Peng et al.¹⁹ proposed a contrastive learning spatiotemporal attention network, an innovative semi-supervised network for video-based SpO₂ estimation. Cheng et al.²⁰ used the spatial temporal representation to record features and CNN consisted with EfficientNet-B3 to predict SpO₂ value. Mathew et al.²¹ first propose the convolutional neural network based on a noncontact SpO₂ estimation scheme using a smartphone camera. This paper compares the influence of the sequence of processing temporal features and spatial features on the SpO₂ estimation results and finally concludes that processing the two features at the same time has the smallest Mean Absolute Error (MAE) and Root Mean Square Error (RMSE). However, without an attention mechanism, model training cannot focus on skin pixels, causing serious interference from environmental noise and low robustness. Yusuke et al. decompose the input video into direct current (DC) and alternating current (AC) components using a filter. These components are subsequently input into ResNet18 separately, and the SpO₂ value is ultimately derived through a fully connected layer²².

Nevertheless, they use a self-sampled dataset, lacking comparisons with other researchers, and the structure of directly outputting SpO₂ using a fully connected layer is open to debate. Casalino et al.²³ overcame one of the main issues of non-contact SpO₂ measurement based on pure data sets, which is the frequent change of perspective viewpoint due to head movements, which makes it more difficult to identify the face and measure SpO₂. Verkruysse et al.²⁴ used two public datasets (rPPG datasets and BIDMC datasets) to extract multimodal physiological information including SpO₂. In model architecture, Researchers have undertaken numerous efforts. Hu et al.²⁵ first introduced Residual and Coordinate Attention (RCA) into the SpO₂ estimation module as feature extraction network. This paper constructs a Color Channel Model (CCM) based on an image generator and a Network-Based Model (NBM), finally combining the two into the Multi-Model Fusion Model (MMFM), which achieves the optimal solution. Nonetheless, the researchers solely took into account the spatial features of facial videos, neglecting the spatiotemporal features of SpO₂. Empirical investigations have conclusively demonstrated that the sequence in which temporal and spatial features are extracted exerts a discernible influence on the accuracy of SpO₂ prediction²¹. It is imperative to attain concurrent extraction of both temporal and spatial features to optimize predictive performance. From the evaluation metrics, Bal²⁶ observed the Pearson correlations by comparing the Pearson coefficient between our SpO₂ estimations and the commercial oximeter readings. Stogiannopoulos et al.²⁷ used the RMSE and Pearson to estimating SpO₂ values. Verkruysse et al.²⁸ used the MAE to evaluate, which was validated in controlled conditions for an SpO₂ range of 83–100%. Pirzada²⁹ used the ROR method for measurement, but the metrics was also RMSE between the estimated SpO₂ and the real value. Avram et al.³⁰ explained the quality of the work by judging whether the RMSE result exceeds the FDA-recognized ISO 81060-2-61:2017 standard.

Inspired by the contactless Heart Rate (HR) estimation using IPPG³¹, this paper utilizes the Improved Temporal Shift Convolutional Attention Network (ITSCAN) for SpO₂ estimation for the first time. In light of the prevailing challenges encountered in contemporary SpO₂ estimation tasks, several noteworthy issues merit consideration:

1. Inadequate Feature Extraction Methodology The current models in use exhibit limitations in feature extraction as they do not concurrently account for temporal and spatial characteristics. Moreover, they lack an attention mechanism, precluding the capacity to selectively focus on skin pixels of utmost relevance. Additionally, the output generated via the fully connected layers lacks interpretability, impeding the model’s transparency and the ability to discern the basis for predictions.

2. Lack of Rigorous Loss Function Selection A second pertinent concern pertains to the absence of a systematic approach in the selection of appropriate loss functions for the SpO₂ estimation task. This omission has profound implications for the model’s performance and convergence properties. An in-depth analysis and rationale for the chosen loss functions are warranted to ensure their alignment with the specific characteristics of the problem at hand, as well as to optimize training dynamics.

3. Limited Dataset Utilization and Reproducibility Concerns A third salient issue revolves around the predominant reliance on self-built datasets by most researchers within the field. This practice raises concerns regarding the generalizability of findings and the robustness of experimental outcomes.

In response to the aforementioned challenges, this paper makes the following substantial contributions:

1. We propose a novel deep learning model architecture for SpO₂ estimation, which is consists of two distinct branches, seamlessly interconnected via a coordinate attention mechanism. The model introduces TSM to extract spatiotemporal features, employs Residual Network to capture pertinent facial skin pixels, combines skin pixel positions with feature maps through coordinate attention, and finally deploys a specialized SpO₂ Estimator in lieu of conventional fully connected layer for more robust estimation.

2. In a pioneering effort, we consolidate prevalent loss functions commonly employed in SpO₂ estimation tasks. Subsequently, we conduct comprehensive comparative experiments to discern optimal combinations of these loss functions that best align with our model. Significantly, we introduce a novel composite loss function encompassing the Negative Pearson coefficient (NegPearson) and SmoothL1 for the precise estimation of SpO₂ value.

3. To facilitate comparison with the related work, we conduct rigorous experimentation on public datasets (VIPL-HR) and compare the results with traditional methods and contemporary deep learning methods in recent literature. Judging from the results, the proposed method out perms in all aspects, especially in RMSE which shows significant advantages.

Methods

Datasets

In the current research on contactless SpO₂ estimation, most researchers use self-built datasets instead of public ones, and most of them have fewer subjects and scenes. The experiments in this paper are conducted on the public datasets of VIPL-HR for easier comparison and more convincing³². The VIPL-HR database contains 9 scenarios for 107 subjects recorded by 4 different devices, which note the HR, SpO₂, and Blood Volume Pulse (BVP) signal of each video at the same time.

In order to verify the running status of the algorithm on devices with poor performance, this paper selects SOURCE1, in which the video is recorded by a Logitech C310 web camera with 25 frames and a resolution of $960\times 720$, as the dataset. 80% of the data was used as the training set, 10% as the validation set, and 10% as the test set. The SpO₂ starts recording at the same time as the video, and a SpO₂ value is recorded every second. The histogram of the SpO₂ value distribution of the data set is shown in Fig. 1. The range of SpO₂ from VIPL-HR is 84 to 99.

Figure 1.

The histogram of label distribution of the dataset VIPL-HR, and label is SpO₂ level.

Traditional method

According to the literature³³, SpO₂ is the percentage of oxygenated hemoglobin in peripheral capillaries, which can be expressed as Eq. (1).

$${\text{SpO}}_2=\frac{C_{Hb{O}_2}}{\left(C_{Hb{O}_2}+C_{\mathit{Hb}}\right)}\times 100\%$$ 1

where $C_{\mathit{Hb}}$ is the concentration of deoxygenated hemoglobin and $C_{Hb{O}_2}$ is the concentration of oxygenated hemoglobin. The Beer–Lambert Law states the light absorption by a substance in a solution is proportional to its concentration³⁴. Set the light intensity incident on the tissue as $I_0$. After passing through human tissue, the intensity of transmitted light $I$ can be expressed as Eq. (2).

$$I=I_0{e}^{-{\epsilon }_0{C}_{0}l-({\epsilon }_{Hb{O}_2}{C}_{Hb{O}_2}+{\epsilon }_{\mathit{Hb}}{C}_{\mathit{Hb}})L}$$ 2

where ${\epsilon }_0$, $C_0$, $l$ represent the total extinction coefficient of non-pulsatile components in tissue and venous blood, the concentration of light-absorbing substances, and distance travelled by light in tissue, respectively. The ${\epsilon }_{\mathit{Hb}}$ and ${\epsilon }_{Hb{O}_2}$ represent the extinction coefficient of deoxygenated hemoglobin and the extinction coefficient of oxygenated hemoglobin. The $L$ represents the distance travelled by light in arterial blood. Then, the absorbance of human tissue $\mathit{Ab}$ can be defined as Eq. (3).

$$Ab=-log\frac{I}{I_0}={\epsilon }_0{C}_{0}l+\left({\epsilon }_{Hb{O}_2}{C}_{Hb{O}_2}+{\epsilon }_{\mathit{Hb}}{C}_{\mathit{Hb}}\right)L$$ 3

As the human heart beats, the light intensity of light penetrating arteries will reach the minimum and maximum, and the absorbance of human tissue fluctuates between maximum and minimum values. Obviously, ${\epsilon }_0{C}_{0}l$ can be cancelled by the max $A{b}_{max}$ and the min $A{b}_{min}$, which means the influence of blood tissue can be eliminated by changes in light absorption caused by blood volume fluctuations. It is shown in Eq. (4).

$$\begin{array}{cc}& \mathrm{\Delta }Ab=A{b}_{max}-A{b}_{min}=\left(-log\frac{I_{\mathit{DC}}}{I_0}\right)-\left(-log\frac{I_{\mathit{AC}}}{I_0}\right)=log\frac{I_{\mathit{AC}}}{I_{\mathit{DC}}}\hfill \\ \hfill & \propto \frac{I_{\mathit{AC}}}{I_{\mathit{DC}}}=\left({\epsilon }_{Hb{O}_2}{C}_{Hb{O}_2}+{\epsilon }_{\mathit{Hb}}{C}_{\mathit{Hb}}\right)\mathrm{\Delta }L\hfill \\ \hfill \end{array}$$ 4

where $I_{\mathit{DC}}$, $I_{\mathit{AC}}$ represent the DC component and the AC component, respectively. $\mathrm{\Delta }L$ represents the difference of the distance travelled by light. For the same subject, $\mathrm{\Delta }L$ is the same at different wavelengths. Obviously, in the scene of two different wavelengths of light, the influence can be eliminated, which is shown in Eq. (5).

$$\frac{D_{{\lambda }_1}}{D_{{\lambda }_2}}=\frac{I_{\mathit{AC}}^1/I_{\mathit{DC}}^1}{I_{\mathit{AC}}^2/I_{\mathit{DC}}^2}=\frac{{\epsilon }_{Hb{O}_2}^1{C}_{Hb{O}_2}+{\epsilon }_{\mathit{Hb}}^1{C}_{\mathit{Hb}}}{{\epsilon }_{Hb{O}_2}^2{C}_{Hb{O}_2}+{\epsilon }_{\mathit{Hb}}^2{C}_{\mathit{Hb}}}$$ 5

where ${\lambda }_1$ and ${\lambda }_2$ represent the different wavelengths, and $D_{\lambda }$ represents the radio of the AC component and the DC component. Combined with Eqs. (1) and (5), we can rewrite the calculation model as Eq. (6).

$${\text{SpO}}_2=\frac{{\epsilon }_{\mathit{Hb}}^2·\frac{D_{{\lambda }_1}}{D_{{\lambda }_2}}-{\epsilon }_{\mathit{Hb}}^1}{\left({\epsilon }_{Hb{O}_2}^1-{\epsilon }_{\mathit{Hb}}^1\right)-\left({\epsilon }_{Hb{O}_2}^2-{\epsilon }_{\mathit{Hb}}^2\right)·\frac{D_{{\lambda }_1}}{D_{{\lambda }_2}}}$$ 6

According to the optical absorption spectra of HbO₂ and Hb shown in Fig. 2, when the absorption of Hb and HbO₂ of light of wavelength ${\lambda }_2$ is equal, Eq. (6) can be written as Eq. (7).

$${\text{SpO}}_2=\frac{{\epsilon }_{\mathit{Hb}}^1}{{\epsilon }_{\mathit{Hb}}^1-{\epsilon }_{Hb{O}_2}^1}-\frac{{\epsilon }_{\mathit{Hb}}^2}{{\epsilon }_{\mathit{Hb}}^1-{\epsilon }_{Hb{O}_2}^1}·\frac{D_{{\lambda }_1}}{D_{{\lambda }_2}}$$ 7

Figure 2.

Absorption spectrum of HbO₂ and Hb¹⁰.

Therefore, the red light with a wavelength of 660 nm is usually referred to as ${\lambda }_1$. In the visible light range, light with wavelengths of 440 nm and 520 nm is usually used as ${\lambda }_2$. For each subject, this paper gets its BVP signal by video, and then calculates the average peak-to-peak and average values of the BVP signals of different color channels as AC and DC components respectively, and finally fits the SpO₂ curve.

Another traditional method PBV is not discussed in this paper because it can only roughly calculate SpO₂.

Deep learning method ITSCAN

The proposed SpO₂ estimation model based on deep learning is shown in Fig. 3, which consists of a motion branch and an appearance branch. And the identifiable image in Fig. 3 is taken from a dataset, named VIPL-HR. The appearance branch is responsible for capturing the detailed features of the face in the video by the Residual Network, and introducing it into the motion branch through the attention mechanism. The motion branch uses Temporal Shift Module (TSM)³⁵ instead of 3D Convolution Neural Network (3DCNN) to extract spatial–temporal features, which helps to reduce calculation time. Different from TSCAN’s frame-by-frame estimation of HR, ITSCAN’s motion branch introduces a SpO₂ Estimator at the end to map the features learned by the model into a separate SpO₂, realizing second-by-second SpO₂ prediction. By doing so, ITSCAN not only significantly reduces computational time by only calculating the attention mask once, but also captures most of the pixels that contain skin and reduces camera quantization error. ITSCAN is derived from MTTS-CAN, but they exhibit distinct differences in task types, application scenarios, and model architectures. MTTS-CAN. MTTS-CAN is used to measure HR and Resp by introducing self-attention masks twice in regression tasks, while ITSCAN is used to measure SpO₂ by introducing coordinate attention, residual network and SpO2 estimator in classification.

Figure 3.

Overview of the ITSCAN model.

TSM network

This section introduces a TSM Network to extract features with physiological information from face videos and uses them to calculate the SpO₂. The TSM Network is shown in Fig. 4.

Figure 4.

The TSM network module operates by using frame dislocation and 2D CNN to achieve synchronous extraction of spatiotemporal features.

2D CNN is widely used to extract image features, but IPPG needs to extract more complex spatiotemporal features. Therefore, 3D CNN, which can better take into account spatiotemporal features, is used for video processing. However, there is still a problem with 3D CNN because it needs to process temporal and spatial features at the same time, resulting in a large computational overhead. While 2D CNN can guarantee computational efficiency but cannot effectively extract temporal features. TSM is naturally proposed as a compromise between 3D CNN and 2D CNN, the schematic diagram of which is shown in Fig. 5.

Figure 5.

The schematic diagram of TSM specific operations.

In this paper, the total number of channels is divided by 3, 1/3 of the channels shift forward along the temporal axis, the other 1/3 shifts backward along the temporal axis, and the remaining 1/3 channels are unchanged. When some channel shifting exceeds the theoretical time length, the excess frames are truncated and the empty part is filled with zero. TSM realizes the exchange of spatial information between different frames so that the subsequent 2D CNN operation can extract spatial features including other time dimensions. Moreover, the operations of truncation and zero padding also ensure the invariance of the original time sequence.

The coordinate attention

Affected by the environment, large amplitude head movements and light conditions will affect the extraction of facial features, resulting in distortion of SpO₂ estimation³⁶. Meanwhile, to minimize the negative effects introduced by shifting tensor, an attention module is inserted in ITSCAN. The coordinated attention mechanism performs convolution, normalization, pooling, and activation operations on the horizontal and vertical features information of the image, realizing the integration of spatial coordinate information and constructing a coordinate-aware attention map. In the paper, the coordinate attention module is embedded into the appearance branch, and the network structure is shown in Fig. 6.

Figure 6.

The coordinate attention module structure first extracts the height and width features of the input data independently, and then learns them comprehensively.

The input can be expressed as $F=[f_1,f_2\dots ,f_n]$. Through the average pooling layer, the input $F$ is encoded separately in the horizontal and vertical directions, allowing the attention module to capture the precise location of facial features. Thus, the output of the c-th channel at width $w$ can be written as Eq. (8), and the output of the c-th channel at height $h$ can be written as Eq. (9).

$$z_c^w(w)=\frac{1}{H}\sum _{0\le i\le H}{f}_c(j,w)$$ 8

$$z_c^h(h)=\frac{1}{W}\sum _{0\le j\le W}{f}_c(h,j)$$ 9

$H$ and $W$ represent the height and width of the image respectively. The size of the horizontal feature map $Z^h$ is $[H,1]$ and the size of the vector feature map $Z^w$ is $[1,W]$. Both are merged into a new feature map, the size of which is $[H,W]$. The results are passed through the $1\times 1$ convolution layer, normalization layer and $\text{Re}LU$ activation layer respectively.

Then, the features are evenly divided into part $F_1$ and $F_2$ according to the second dimension, and then sent to the $1\times 1$ convolution layer respectively, so that their dimensions are again consistent with the dimensions of the input features of the coordinate attention module. Finally, the output is denoted by Eq. (10).

$$\text{Output}=F\times F_1\times F_2$$ 10

Residual network

The paper refers to the literature²⁵ to construct a residual network for feature extraction as shown in Fig. 7. In the residual network, the introduction of jump links greatly alleviates the vanishing gradient problem. At the same time, directly connecting the input features to the output via a bypass reduces information loss and further improves network performance. After experimental testing, when the number of residual blocks is 2, the residual network has the best effect.

Figure 7.

Specific architecture of the module Residual Network, which includes two layers of Residual Block.

SpO₂ estimator

The SpO₂ Estimator is shown in Fig. 8. It maps the high-dimensional features to SpO₂ corresponding to time. Since SpO₂ can equal interval discrete values from 1 to 100%, i.e., the sequential vector $S=[1,2,\dots ,100]$. The channels of input are compressed to 100 by a $1\times 1$ convolution layer, consisting with $S$. Subsequently, obtain the feature map with these 100 channels by a normalization layer and a Softmax activation layer.

Figure 8.

Specific architecture of the SpO₂ estimator. Map the input features to SpO₂ value through the softmax function.

The loss function

In the SpO₂ estimation task, the MAE and RMSE between the predicted result and the real result are the targets that researchers care about. According to NBR ISO 80601-2-61:2015, to validate equipment for SpO₂ estimation, the RMSE value must be less than 2% of the declared range¹¹. MAE is also the smaller the better. At the same time, if the change trend of the estimated SpO₂ is basically consistent with the actual value, it means that the estimated SpO₂ can reflect the real situation. So, The Pearson coefficient is naturally also used to evaluate the quality of prediction results. The formulas are defined from Eqs. (11) to (13).

$$\text{MAE}=\frac{1}{T}\sum _{i=1}^T\left|X_i-Y_i\right|$$ 11

$$\text{RMSE}=\sqrt{\frac{1}{T}\sum _{i=1}^T{(X_i-Y_i)}^2}$$ 12

$$\text{Pearson}=\frac{T{\sum }_{i=1}^T{X}_i{Y}_i-{\sum }_{i=1}^T{X}_i{\sum }_{i=1}^T{Y}_i}{\sqrt{\left(T{\sum }_{i=1}^T{X}_i^2-{\left({\sum }_{i=1}^T{X}_i\right)}^2\right)\left(T{\sum }_{i=1}^T{Y}_i^2-{\left({\sum }_{i=1}^T{Y}_i\right)}^2\right)}}$$ 13

where T is the length of the SpO₂ array, X is the gold standard SpO₂ series, and Y is the estimated SpO₂ series.

In predicting the SpO₂ task using deep neural networks, there are a total of 5 loss functions that are commonly used, namely MSE²², RMSE²¹, SmoothL1²⁵, NegPearson²², and cross entropy²⁵. The SmoothL1, NegPearson, and Cross Entropy are defined from Eqs. (14) to (16). The paper compares the effectiveness of new loss functions formed by linear combinations of the above 5 loss functions in predicting SpO₂. Finally, it was concluded that the combination of SmoothL1 and NegPearson has the best results. Specific experimental results are displayed in Experiments.

$$\text{SmoothL1}=\frac{1}{T}\sum _{i=1}^T\left\{\begin{array}{cc}0.5\times {(Y_i-X_i)}^2,\hfill & \left|Y_i-X_i\right|<1\\ \left|Y_i-X_i\right|-0.5,\hfill & \mathit{otherwise}\end{array}\right)$$ 14

$$\text{NegPearson}=1-\left|\text{Pearson}\right|$$ 15

$$\text{CrossEntropy}=-\sum _{i=1}^T\sum _{j=1}^N{P}_{X_{\mathit{ij}}}log{P}_{Y_{\mathit{ij}}}$$ 16

where T is the length of the SpO₂ array, X is the gold standard SpO₂, and Y is the estimated SpO₂. ${\text{P}}_{{\text{X}}_{\text{ij}}}$ represents the probability that the gold standard SpO₂ at time i is estimated to be j, and ${\text{P}}_{{\text{Y}}_{\text{ij}}}$ represents the probability that the estimated SpO₂ at time i is estimated to be j.

Results and discussion

Experiment setting

In preprocessing, the original video is segmented into groups of 25 frames because each 25 frames represent one piece of recorded SpO₂. Use the OpenCV open haarcascade_frontalface_default.xml file to locate the position of the face part in the video. Moreover, to avoid missed detection, the height and width of the recognition anchor frame are multiplied by 1.5 times. In the process of model training, the AdamW decay approach is implemented, with a learning rate of 1e−4. Each training iteration consists of 20 epochs and is conducted on a dual setup of GeForce RTX 2080 Ti GPUs.

Considering that there are two branches in the model, the original video image is processed in two different normalization ways. The image input to the motion branch is firstly subjected to difference operation and then normalized, and the image input to the appearance branch is only normalized. During normalization, this paper compresses each frame image to the same standard size. Therefore, this paper compares the ITSCAN model performance under different input image sizes. This paper summarizes the commonly used loss functions for predicting SpO₂, and linearly combines them to select the loss function with the best results. Likewise, this paper uses ablation experiments to verify the impact of each module on the prediction results. In conclusion, this paper conducts a comprehensive comparative analysis, aligning its findings with those of contemporaneous researchers. In all tables, all values are average values, and the bold values indicate minimal or maximum errors. This paper uses MAE, RMSE, and Pearson coefficients to evaluate the experimental results. Example test prediction curves are shown for reference in Fig. 9.

Figure 9.

Prediction curves under different circumstances.

Where the blue line represents the individual’s actual SpO₂ and the red line represents the predicted SpO₂ by ITSCAN.

Results and discussion

Image size

To verify the impact of different input image sizes on the experimental results, the facial areas are segmented from the original image using a rectangular anchor frame and resized to various sizes. Table 1 presents the prediction results under different input image sizes, which is the optimal loss function at this time.

Table 1.

Comparative experimentation of image sizes.

Image size	MAE (%)	RMSE (%)	Pearson
64 × 64	1.28	1.29	0.355
128 × 128	1.37	1.50	0.312
256 × 256	1.10	1.19	0.375
320 × 320	1.32	1.39	0.343
400 × 400	1.31	1.38	0.302

According to Table 1, increasing the pixel size of images can indeed obtain more texture and contextual information, which is beneficial to capture better features. In addition, it is easier to obtain discriminative features when images become large. On the one hand, the utilization of excessively large input images results in an increase in model size and complexity, which can lead to overfitting. On the other hand, this approach is susceptible to introducing superfluous features that may subsequently reduce the model’s predictive accuracy. From the experiment, the input image size is set to $256\times 256$, which is more suitable.

A comprehensive investigation into the influence of input image dimensions on model performance within the context of the PURE dataset is conducted²⁵. Specifically, the study evaluated 7 distinct image sizes, namely 32, 112, 196, 224, 304, 352, and 448 pixels. The findings demonstrated that the employment of the $224\times 224$ pixel image dimensions yielded the most favorable outcomes in terms of model performance, which is match with our experiment ($256\times 256$).

Time length

To examine the effect of input data over varying time lengths on model performance, this paper evaluates the model using input data at different time scales. Since the video frame rate of VIPL-HR is 25 fps, the length of the input data is an integer multiple of 25 frames. Table 2 presents the prediction results under different input time length, which is the optimal loss function at this time.

Table 2.

Comparative experimentation of time length.

Time length (s)	MAE (%)	RMSE (%)	Pearson
1	1.10	1.19	0.375
2	1.03	1.21	0.368
3	1.35	1.44	0.351
5	1.37	1.41	0.321

According to Table 2, as the length of input data increases, the overall model performance exhibits a downward trend. This phenomenon may be attributed to the excessive amount of input data combined with the minimal variation in SpO₂ as a label, which hinders the model’s ability to effectively learn features. However, the MAE indicator slightly improves from the 1 to 2-s time scale, indicating that the model retains a certain level of processing capability for 2-s length data. In summary, the model proposed in this paper achieves SpO₂ evaluation within seconds.

Loss function

A suitable loss function helps the model adapt to the data. This paper evaluates the performance of several commonly used loss functions in SpO₂ prediction tasks. The results are shown in Table 3. For convenience of expression, MSE is denoted as M, RMSE is denoted as R, SmoothL1 is denoted as S, CrossEntropy is denoted as C, and NegPearson is denoted as N. At this juncture, the weights associated with the components of the linear combination within the loss function are equitably set.

Table 3.

Comparative experimentation of loss functions.

Loss Function	MAE (%)	RMSE (%)	Pearson	Loss function	MAE (%)	RMSE (%)	Pearson
MSE	1.88	1.99	0.304	M + R + S	1.42	1.50	0.294
RMSE	1.64	1.74	0.313	M + R + N	1.46	1.54	0.274
SmoothL1	1.43	1.51	0.281	M + R + C	1.98	2.09	0.211
NegPearson	5.55	5.63	0.329	M + S + N	1.46	1.54	0.336
Cross Entropy	2.14	2.24	Nan	M + S + C	2.00	2.11	0.295
M + R	1.46	1.55	0.291	M + N + C	1.59	1.69	0.247
M + S	1.35	1.43	0.324	R + S + N	1.88	1.98	0.287
M + N	1.43	1.51	0.290	R + S + C	1.72	1.82	0.289
M + C	1.46	1.54	0.286	R + N + C	2.01	2.11	0.238
R + S	1.39	1.47	0.275	S + N + C	2.07	2.18	0.210
R + N	1.49	1.58	0.194	M + R + S + N	1.46	1.54	0.303
R + C	1.58	1.68	0.235	M + R + S + C	2.07	2.18	0.294
S + N	1.46	1.56	0.318	M + S + N + C	1.59	1.79	0.228
S + C	1.59	1.69	0.245	R + S + N + C	1.79	1.89	0.275
N + C	4.39	4.39	0.214	ALL	1.51	1.61	0.301

According to Table 3, bold fonts represent the best results, and the results exceeding 0.3 in the Pearson coefficient are highlighted in italics font. It is evident that selecting M and S as the loss functions leads to the attainment of minimum values for MAE and RMSE, respectively. Meanwhile, loss functions associated with green font consistently incorporate combinations of M, S, or N. Despite the fact that utilizing the sole loss function S results in a Pearson coefficient of only 0.281, substantial enhancements in both MAE and RMSE are evident. Therefore, this does not preclude the inclusion of S in the subsequent discussion within this article. Choosing the simplest possible loss function can reduce the space complexity of model calculation. Consequently, when indicators of M + S and M + N + S are not much different, it is best to use M + S as the ITSCAN model loss function. Based on this idea, this paper considers pairwise combinations of M, N, and S to explore the optimal combination weight. The results are shown in Fig. 10.

Figure 10.

The impact of hyperparameters on model results under the combination of three different loss functions.

Absolutely, in the case where the combination of the SmoothL1 and NegPearson loss functions is employed ($\alpha =0.8$), MAE attains its minimum value ($MAE=1.10\%$), while RMSE reaches its minimum value ($RMSE=1.19\%$). Additionally, the Pearson correlation coefficient achieves its maximum value under this configuration ($Pearson=0.375$).

Ablation experiment

The ITSCAN model introduces a total of four modules, namely the TSM Network, Coordinate attention network, Residual network and SpO₂ estimator. To verify the impact of each network on the overall ITSCAN and thus illustrate the necessity of adding these modules, this paper conducts ablation experiments by replacing modules one by one.

Use ordinary 2D convolutional layers instead of a Residual network to extract facial features. Only the convolution operation is performed between the TSM network output and the appearance branch output instead of the coordinate attention network. Remove the TSM network architecture and exclusively employ convolutional layers for feature extraction. Replace the SpO₂ estimation network with a fully connected layer. The ablation experiment results are shown in Table 4.

Table 4.

Comparative experimentation of model modules.

	MAE (%)	RMSE (%)	Pearson
ITSCAN without a TSM network	1.42	1.51	0.281
ITSCAN without Residual network	1.71	1.81	0.343
ITSCAN without Coordinate attention	1.95	2.05	0.243
ITSCAN without SpO2 estimator	2.70	2.87	0.319
ITSCAN	1.10	1.19	0.375

It is evident that in the absence of the SpO₂ estimator incorporation, the predicted SpO₂ values exhibit a notable discrepancy, resulting in MAE and RMSE values of 2.70% and 2.87%, respectively. This observation underscores the advantageous role of the SpO₂ estimator in enhancing the model’s adaptability to the SpO₂ prediction task. Simultaneously, the absence of coordinate attention manifests as the most unfavorable overall SpO₂ estimation outcome within the model, characterized by the lowest Pearson correlation coefficient and the deviations of MAE and RMSE are second only to the absence of the SpO₂ estimator. This phenomenon arises from the fact that coordinate attention facilitates the model’s heightened focus on skin pixels directly associated with SpO₂ estimation, effectively mitigating the influence of extraneous background noise. Consequently, this attention mechanism proves notably advantageous for the model’s acquisition of pertinent and discriminative features. The inclusion of the TSM network serves the purpose of concurrently capturing both temporal and spatial attributes within the video data. Consequently, following the exclusion of the TSM network, MAE and RMSE are able to sustain relatively favourable performance outcomes. However, the Pearson correlation, which exhibits a strong association with the temporal variable, experiences a substantial deterioration in performance. Finally, our investigation showed that changes to the residual network had minimal impact on the overall model performance. This can be attributed to its main function of extracting image features from the input of the appearance branch. Although optimizing the residual network can improve model performance, omitting it does not significantly affect the experimental results.

Comparative methods

To fully verify the validity and objectivity of the experiment results, this paper compares it with other researchers’ papers, including ROR⁸, standard ROR¹², convLSTM¹⁸, CL-SPO2NET¹⁹, CNN with Efficient-B3²⁰,deep learning with ResNet18 and MMTM²², MMFM²⁵, and deep learning with temporal convolution³¹, which are shown in Table 5. The above refers to the traditional method of SpO₂ estimation and the deep learning model that has emerged in recent years. All comparative experiments are conducted on the VIPL-HR dataset. Al-Naji et al.⁸ uses the EEMD and ICA improved ROR method. Shao Qijia et al.¹² compared the ROR extracted by different individuals with the standard ROR to achieve the generalization of the ROR. Zheng¹⁸ proposed a hybrid network which has a mixed architecture consisting of convLSTM and ViViT. CL-SPO2NET¹⁹ introduced the 3D CNN to capture spatial temporal features. Cheng et al.²⁰ used the spatial temporal representation to record features and CNN with EfficientNet-B3 to predict SpO₂ value. Akamatsu et al.²² introduced a dual-branch model incorporating both AC and DC components. This model incorporates a Multi-Modal Transfer Module (MMTM)³⁷ serving as an intermediate fusion mechanism, enabling the integration of intermediate layers from any CNN model with a Squeeze and Excitation (SE) module³⁸ for intermodal information exchange. Concurrently, the DC and AC constituents are extracted from the original video utilizing the ResNet18. The model used in²⁵ consists of three parts: RCA network, Color Channel model and MMF model. The RCA network includes a Residual Block to extract features and Coordinate attention to enhance features. MMF model and the Color Channel model are two branches respectively, and both use the output of RCA as input. Liu et al.³¹ employed an end-to-end neural network, where feature extraction is carried out through a four-layer temporal network, culminating in the compression of the network’s output to estimate SpO₂.

Table 5.

Comparison of different SpO₂ estimation method.

Model	MAE (%)	RMSE (%)	Pearson
ROR8	4.13	4.77	None
Standard ROR12	2.43	3.75	None
convLSTM and ViViT18	1.51	1.61	0.362
CL-SPO2NET19	1.83	1.73	0.401
CNN with Efficient-B320	1.42	1.71	0.372
ResNet18 and MMTM22	1.23	1.52	0.471
MMFM25	1.04	1.48	0.354
Temporal convolution31	1.71	2.29	0.409
ITSCAN	1.10	1.19	0.375

According to Table 5, conventional SpO₂ estimation methodologies exhibit significant predictive deviations and limited robustness. Furthermore, the absence of an attention mechanism within the model yields suboptimal results in SpO₂ assessment, such as³¹. Considering both the spatiotemporal attributes of the data and the specific demands of the SpO₂ assessment task²⁵, and ITSCAN, which developed the SpO₂ estimator, demonstrated superior performance in terms of MAE and RMSE metrics, respectively. Conversely²², which relied on the output of fully connected layers, exhibited enhanced capability in capturing overall SpO₂ variations, as reflected by its higher Pearson correlation coefficient.

Conclusion

This paper proposes an enhanced neural network built upon TSCAN. While preserving the TSM network, novel additions include the incorporation of the Residual network and the Coordinate attention mechanism to augment the model’s data sensitivity. Concurrently, a SpO₂ estimator is integrated at the end to better suit the task of SpO₂ estimation. Each module of the model reaches its optimal state through an ablation experiment, and the input image size is also determined through experimental comparison. This paper also refers to the five loss functions commonly used in SpO₂ estimation tasks, and determines the model loss function through experimental comparison. Meantime, this paper presents an overview of the primary models employed by researchers in the task of SpO₂ estimator over the years and conducts a comparative analysis with the most recent and exemplary findings. The outcomes demonstrate that the proposed method, occupied the top and second positions in terms of RMSE and MAE, achieving values of 1.19% and 1.10%, respectively. Notably, the RMSE indicator exhibits a 0.29% lower deviation than the second-ranked approach, while the MAE is only 0.06% larger than the first-ranked approach.

Author contributions

All authors contributed to the study conception and design. Material preparation, and analysis were performed by S.Z.. Data collection was performed by S.Z. and Y.Y. Experiments’ result was checked by S.L. Paper formatting was checked by X.J. and C.S. The first draft of the manuscript was written by S.Z. and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Data availability

The data that support the findings of this study are available from the Visual Information Processing and Learning (VIPL) research group attached to the Institute of Computing Technology of the Chinese Academy of Sciences (ICT, CAS) and the key Intelligent Information Processing Laboratory of the Chinese Academy of Sciences. Data are available at https://vipl.ict.ac.cn/resources/databases/201811/t20181129_32716.html with the permission of VIPL by downloading the VIPL-HRDatabase Release Agreement and emailing to hanhu[at]ict.ac.cn.

Competing interests

The authors declare no competing interests.