Contactless blood oxygen estimation from face videos: A multi-model fusion method based on deep learning

Abstract

Blood Oxygen (${\text{SpO}}_2$), a key indicator of respiratory function, has received increasing attention during the COVID-19 pandemic. Clinical results show that patients with COVID-19 likely have distinct lower ${\text{SpO}}_2$ before the onset of significant symptoms. Aiming at the shortcomings of current methods for monitoring ${\text{SpO}}_2$ by face videos, this paper proposes a novel multi-model fusion method based on deep learning for ${\text{SpO}}_2$ estimation. The method includes the feature extraction network named Residuals and Coordinate Attention (RCA) and the multi-model fusion ${\text{SpO}}_2$ estimation module. The RCA network uses the residual block cascade and coordinate attention mechanism to focus on the correlation between feature channels and the location information of feature space. The multi-model fusion module includes the Color Channel Model (CCM) and the Network-Based Model(NBM). To fully use the color feature information in face videos, an image generator is constructed in the CCM to calculate ${\text{SpO}}_2$ by reconstructing the red and blue channel signals. Besides, to reduce the disturbance of other physiological signals, a novel two-part loss function is designed in the NBM. Given the complementarity of the features and models that CCM and NBM focus on, a Multi-Model Fusion Model(MMFM) is constructed. The experimental results on the PURE and VIPL-HR datasets show that three models meet the clinical requirement(the mean absolute error $⩽$ 2%) and demonstrate that the multi-model fusion can fully exploit the ${\text{SpO}}_2$ features of face videos and improve the ${\text{SpO}}_2$ estimation performance. Our research achievements will facilitate applications in remote medicine and home health.

1. Introduction

${\text{SpO}}_2$ saturation reflects the function of the cardiopulmonary and respiratory system and is an essential physiological indicator for clinical medical monitoring [1]. Many lung diseases can be detected by detecting abnormal values of ${\text{SpO}}_2$, such as SARS coronavirus in 2003, MERS, the Middle East respiratory syndrome coronavirus in 2012, and COVID-19 which spread uncontrollably in the current period. The literature [2] stated that patients with pulmonary infections did not feel shortness of breath in the early stages, but their ${\text{SpO}}_2$ levels were decreasing. Therefore, monitoring ${\text{SpO}}_2$, an important physiological parameter, is essential to avoid disease deterioration. In addition, regarding blood vessel detection, anomaly and blood glucose disease can be determined from the data collected from blood glucose meter [3].

Traditional contact pulse ${\text{SpO}}_2$ meters [4] are widely used in routine and critical scenarios, but they are challenging to perform in particular scenarios, such as possible injury to burn patients and inaccurate ${\text{SpO}}_2$ monitoring in patients with hand and foot tremors. In addition, pulmonary diseases are highly contagious in general, and incomplete disinfection of healthcare workers and healthcare tools may lead to the secondary transmission of pathogens. In these scenarios, contactless measurement of physiological parameters becomes an effective method for estimating ${\text{SpO}}_2$. Several scholars have already studied the implementation of contactless monitoring of vital characteristics [5], for example, Remote Photo-plethysmography (rPPG), Image Photo-plethysmography (iPPG), contactless pulse ${\text{SpO}}_2$ measurement, and others. To our best knowledge, a few articles studied contactless ${\text{SpO}}_2$ estimation methods based on videos. Among them, most of the rPPG-based ${\text{SpO}}_2$ estimation methods use traditional signal processing methods, such as filtering [6], signal interpolation [7], and fast Fourier transform [8].

However, with the rapid growth of deep learning techniques, an increasing number of scholars have applied them to biomedical scenarios and daily health monitoring and achieved desirable results, such as detecting COVID-19 by chest X-ray images [9] and physiological indicator estimation [10]. In particular, deep learning techniques regarding heart rate estimation have matured in physiological indicator estimation. However, the application of deep learning to contactless ${\text{SpO}}_2$ estimation is still at an early stage of development, and the literature [11] provided a systematic review of them. Last year, J. Mathew et al. proposed a study combining deep learning with contactless ${\text{SpO}}_2$ estimation in hands by their self-built dataset and achieved the clinical requirement [12]. Apart from the literature [12], no other literature uses deep learning techniques for non-contact ${\text{SpO}}_2$ estimation tasks. However, considering the maturity and convenience of facial estimation and inspired by the previous studies of rPPG-based heart rate estimation literature [13] and the contactless blood oxygen estimation literature [14], a contactless ${\text{SpO}}_2$ estimation method combined with deep learning based on face videos is proposed in this paper. Experimental results show that the ${\text{SpO}}_2$ estimation values of this paper's method are comparable to those values obtained by the commercial pulse ${\text{SpO}}_2$ meter, and the ${\text{SpO}}_2$ measurement error in this paper is within an acceptable range according to the literature [15].

The main contributions of the work in this paper are as follows.

• 1)To fully extract ${\text{SpO}}_2$ features from face videos by deep learning, a network named Residuals and Coordinate Attention(RCA) for feature extraction is constructed. It is a combination of residual block cascade and coordinate attention mechanism. Thus, the RCA network not only learns the high-dimensional features of the image, but also focuses on the correlation relationship between feature channels and the location information in the feature space. In this way, the RCA network has a stronger feature representation capability.
• 2)To use the classical color channel ratio algorithm to calculate ${\text{SpO}}_2$, a ${\text{SpO}}_2$ estimation model named Color Channel Model(CCM) for face videos is constructed, which is based on an image generator. CCM firstly uses residual blocks and transposed convolution to reconstruct feature maps learned by the RCA network into RGB images. Then, CCM calculates ${\text{SpO}}_2$ using the red/blue color channel ratio method.
• 3)To facilitate the network to focus on learning ${\text{SpO}}_2$ features in many aspects, a ${\text{SpO}}_2$ estimation model named Network-Based Model(NBM) is constructed. The NBM is based on a network and a two-part loss function. Then, it is fused with the CCM model into a multi-model fusion ${\text{SpO}}_2$ estimation model named Multi-Model Fusion Model(MMFM), which constructs a loss function that takes full advantage of the features that CCM and NBM focus on as well as the complementarity between them. The CCM, RBM and MMFM perform ${\text{SpO}}_2$ estimation experiments on the public datasets (PURE [16] and VIPL-HR [17], [18]) to achieve accurate ${\text{SpO}}_2$ estimation.

2. Related work

${\text{SpO}}_2$ is the oxygen concentration in the blood and represents the ratio of oxygenated hemoglobin to total hemoglobin, with a normal range of 95 % to 100 %. The medical device commonly used to measure arterial ${\text{SpO}}_2$ is the pulse ${\text{SpO}}_2$ meter, which is based on the use of photo-plethysmography (PPG) to assess changes in blood volume in the human tissue microcirculation [19]. However, the pulse ${\text{SpO}}_2$ meter has many limitations, such as possible injury to burn patients and inaccurate ${\text{SpO}}_2$ monitoring in patients with hand and foot tremors [20]. In addition, there is a risk of virus infection when using contact oximeter to check the physiological health of patients with infectious diseases. So researchers proposed the remote photo-plethysmography (rPPG) which is a contactless method for estimating pulse waves generated by the heart through peripheral blood perfusion measurements, using a video camera to capture video of body parts, and a series of processing of the video to obtain the desired physiological information, such as heart rate [21], respiration rate [22] and heart rate variability [23].

In recent years, many scholars have started to use videos to investigate contactless measurements of ${\text{SpO}}_2$, but most of these studies have used conventional signal processing equipment and methods. The literature [14] used a CMOS camera to record PPG signals for ${\text{SpO}}_2$ estimation, with the light source alternating between orange light at 611 nm and near-infrared light at 880 nm. ${\text{SpO}}_2$ was derived from the pulsatile component(AC) and the nonpulsatile component(DC) component analysis of the tracked PPG signals. The literature [24] proposed a method using iris tissue irradiated by two light wavelengths (630 nm and 940 nm) for reflectance images to assess ${\text{SpO}}_2$ levels. The literature [25] developed a contactless skin ${\text{SpO}}_2$ imaging system that uses reflected images of superficial tissue skin to create a map of ${\text{SpO}}_2$ distribution across the measurement area to assess ${\text{SpO}}_2$, heart rate(HR) and blood flow velocity(BFV). The literature [26] used two orthogonal vectors in RGB color space to extract the PPG signal and used a denoising algorithm based on a double-tree composite wavelet transform to reduce illumination and motion artifacts.

With the continuous development of deep learning techniques, there are a proliferation of studies using them for video-based physiological metrics estimation with excellent results, such as heart rate and respiration rate. The literature [17] proposed an end-to-end convolutional attention network to detect blood volume pulses in face videos, which in turn performs frequency analysis of the detected pulse signal to track heart rate and respiration rate. The literature [27] designed a DeepPhys model based on a two-stream approach based on a skin reflection model, which used the appearance module to provide attention to guide the learning of the motion module and thus recovered a more robust rPPG signal. The literature [13] divided the input videos into segments and applied a time-domain segment subnet to extract spatial and temporal information. The literature [28] was built on deep learning to construct temporal signals and used Action Units (AUs) for signal denoising to improve HR estimation.

However, the research on deep learning algorithms for ${\text{SpO}}_2$ estimation by videos is still in its early stages. The literature [29] proposed a convolutional neural network architecture for contact ${\text{SpO}}_2$ monitoring from smartphone cameras. Although they performed better than traditional ratio methods, they still had the drawbacks associated with contact measurements. In July 2021, J. Mathew et al. proposed a study combining deep learning with contactless ${\text{SpO}}_2$ estimation in hands [12]. Experiments were conducted to collect 14 volunteers' hand videos to estimate ${\text{SpO}}_2$, containing both normal breathing and breath-holding states, with a MAE of around 2 %, which aligns with clinical indications. However, the dataset collected in this literature is not public, and no relevant experiments have been performed on public datasets.

Inspired by the above articles, this paper proposes a contactless ${\text{SpO}}_2$ estimation method based on deep learning with multi-model fusion, and conducts rich experiments on the public datasets (PURE and VIPL-HR). PURE and VIPL-HR contain face videos and physiological signal recordings of faces in rich scenes, such as stillness, brightness, darkness, head movement and physical post-exercise state. In PURE dataset, the ground truth value of ${\text{SpO}}_2$ have been captured using a finger clip pulse oximeter (pulox CMS50E). In VIPL-HR dataset, the ground truth value of ${\text{SpO}}_2$ have been captured using the CONTEC CMS60C BVP sensor. There are more than one hundred volunteers involved in the dataset construction work. Rich experiments verify that the RCA network can adequately extract ${\text{SpO}}_2$ features from face videos, and the fusion model can achieve better ${\text{SpO}}_2$ estimation. These also confirm that deep learning techniques can estimate ${\text{SpO}}_2$ effectively and be applied to health screening and remote health assessment.

3. Material and methods

The proposed multi-model fused ${\text{SpO}}_2$ estimation model based on deep learning is shown in Fig. 1 , which consists of two parts: the RCA network and the multi-model ${\text{SpO}}_2$ estimation module. RCA network mainly consists of the residual network module and the coordinate attention module, which is used to extract physiological features from the facial image sequences. The multi-model ${\text{SpO}}_2$ estimation module includes Color Channel Model, the Network-Based Model, and the Multi-Model Fused Model. ${\text{SpO}}_2$ is calculated by feeding the features extracted by the RCA network into the ${\text{SpO}}_2$ estimation module.

Fig. 1.

Structure of ${\text{SpO}}_2$ estimation network model based on deep learning.

3.1. RCA network

This section introduces an RCA network to extract feature signals with physiological information from face videos and use them to calculate the ${\text{SpO}}_2$. Fig. 1 shows the structure of this RCA network. In terms of logical structure, RCA network and the conventional deep CNN [30] are similar. Both of them mainly consist of these types of layers: input layer, convolutional layer, activation function layer, pooling layer and fully connected layer. However, the difference is that the RCA network has residual blocks and attention composition. Residual blocks can alleviate the gradient disappearance problem in the conventional deep CNN by using jump connections. In addition, residual blocks can protect the integrity of the information. Besides, by using attention, more representative feature information can be extracted and result in a higher computational accuracy.

The input image sequence is denoted as $V$ and the pixel size of images is 3 * 224 * 224. Firstly, $V$ is fed into a convolution layer with the kernel size of $\left[7,7\right]$ to downsample, namely, $\mathit{Con}{v}_{7\times 7}$. Then, batch normalization(BN) and ReLu activation function($\delta$) are performed to accelerate the convergence of the network, and finally, global average pooling(GAP) output $F^{\prime }$:

$$F^{\prime }=GAP(\delta (BN(Con{v}_{7\times 7}(V))))$$ (1)

3.1.1. Residual module

To extract the high-dimensional features from $F^{\prime }$, $F^{\prime }$ is fed into the Residual Block (RB) cascade, and the features $\overline{F}$ are obtained. The residual module in this paper is constructed based on ideas from the literature [31]. It alleviates the gradient disappearance problem by using jump connections. In addition, the residual block protects the integrity of the information by passing the input information directly to the output in a bypass, reducing information loss and attrition and improving the learning capability of the network. Besides, to improve network performance and obtain a larger perceptual field to help capture the features that need to be learned and attended to, the network depth is increased by cascading two residual blocks. The structure is shown in Fig. 2 .

Fig. 2.

Residual block cascade structure.

The formula for a single residual module is shown in Equation (2). $\delta$ is the ReLu activation function. ${\text{Conv}}_{3\times 3}$ is a convolution layer with the kernel size of $\left[3,3\right]$, and ${\text{Conv}}_{1\times 1}$ is a convolution layer with the kernel size of $\left[1,1\right]$. $\oplus$ represents the Element-wise Sum.

$$\mathit{RB}(F^{\prime })\text{=}\delta {\text{(Conv}}_{3\times 3}\text{(}\delta {\text{(Conv}}_{3\times 3}(F^{\prime })))\oplus Con{v}_{1\times 1}(F^{\prime }))$$ (2)

The output of the cascade of the two residual blocks is $\overline{F}$ as Equation (3):

$$\overline{F}=R{B}_2(R{B}_1(F^{\prime }))$$ (3)

3.1.2. Coordinate attention module

Large-amplitude head movements and changes in lighting conditions during the recovery of the rPPG signal cycle can interfere with signal estimation, resulting in excessive fluctuations in feature information changes, which can interfere with the signal periodicity [13]. However, the coordinate attention module [32] can focus on the relationship between feature channels and the location information in the feature space. Therefore, we add it after extracting the acquired high-dimensional features to enhance the feature representation.

Coordinate attention refers to embedding location information into channel attention and decomposing channel attention into two parallel 1D feature encoding processes, i.e., pooling, convolution and activation operations for feature information in the X (horizontal) and Y (vertical) directions respectively, to effectively integrate spatial coordinate information into the generated attention graph and construct a coordinate aware attention graph. In this paper, the coordinate attention idea is embedded into the RCA network, and the network structure is shown in Fig. 1. First, the high-dimensional features $\overline{F}=\left[{\overline{f}}_1,{\overline{f}}_2,...,{\overline{f}}_C\right]$ are used as input. Each channel is encoded along the horizontal and vertical directions using pooling layers of size $\left(H,1\right)$ and $\left(1,W\right)$ respectively to enable the attention module to capture the precise position information of the features. Thus, the output of the c-th channel at height h can be formulated as Equation (4). $\sum$ stands for summation.

$$z_c^h(h)=\frac{1}{W}\sum _{{\overline{f}}_c(j,w)}{\overline{f}}_c(h,i)$$ (4)

Similarly, the output of the c-th channel at width w can be written as Equation (5):

$$z_c^w(w)=\frac{1}{H}\sum _{0⩽i<H}{\overline{f}}_c(j,w)$$ (5)

${\overline{f}}_c$ represents the c-th channel of $\overline{F}$. The above transformations perform feature aggregation along two spatial directions to obtain a pair of direction-aware feature maps $[z^h,z^w]$. They also enable the attention module to capture long-term dependencies along one spatial direction and preserve precise location information along with the other, which helps the network locate the target of interest more accurately.

Then, results of the two transformations are concatenated and transform operations are performed on them using the $1\times 1$ convolutional transform function $F_1$ as shown in Equation (6).

$$f_c=\delta (F_1([z_c^h,z_c^w]))$$ (6)

Here, $\delta$ is ReLu activation function. $f_c$ is the intermediate feature mapping result of encoding the spatial information of the c-th channel in the horizontal and vertical directions. Decompose $f_c$ into two separate tensors $f_c^h\in R^{C/r\times H}$ and $f_c^w\in R^{C/r\times W}$ along the spatial dimensions. $f_c^h$ and $f_c^w$ are transformed into tensors with the same number of channels by using two $1\times 1$ convolution transforms $F_h$ and $F_w$ respectively to obtain $g_c^h$ and $g_c^w$ as shown in Equation (7).

$$\begin{array}{c}{g}_c^h=\sigma (F_h(f_c^h)),\\ g_c^w=\sigma (F_w(f_c^w)).\end{array}$$ (7)

Here, $\sigma$ is Sigmoid activation function. Subsequently, $g_c^h$ and $g_c^w$ are expanded using the function named expand_as so that both of them are of the same size as the input feature $\overline{F}$ so as to serve as the attentional weights of $\overline{F}$. The output of the coordinate attentive module is $F^{″}=\left[f_1^{″},f_2^{″},...,f_C^{″}\right]$, in which the component of the c-th channel of $F^{″}$ is:

$$f_c^{″}(i,j)=f_c(i,j)\times g_c^h(i)\times g_c^w(j)$$ (8)

3.2. Multi-Model fusion for ${\text{SpO}}_2$ estimation

Existing research results show that the color channel ratio method based on videos has achieved certain results, such as the literature [26]. In order to combine the advantages of the Color Channel Model and the Network-Based Model, the features that both models focus on and the complementarity between the models are fully utilized. In this paper, a multi-model fusion ${\text{SpO}}_2$ estimation module is constructed, as shown in Fig. 1. The Color Channel Model reconstructs RGB images from feature maps by the image generator, and then calculates the ${\text{SpO}}_2$ value using the red and blue color channel AC/DC ratio analysis. The Network-Based Model is a simple network model based on deep learning techniques, and ${\text{SpO}}_2$ is estimated directly from the feature maps output of the RCA network.

3.2.1. Color channel model

The structure of the Color Channel Model is shown in Fig. 1. The module consists of the image generator and the ${\text{SpO}}_2$ Estimator1. In order to calculate the ${\text{SpO}}_2$ using the signals from the red and blue color channels, the features $F^{″}$ learned by the RCA network are first reconstructed into the RGB feature map $\mathit{feat}$ by the image generator, and then fed into the ${\text{SpO}}_2$ Estimator1 to obtain $\mathit{Sp}{O}_{2_{\mathit{CCM}}}$. In addition, the loss functions $\mathit{Los}{s}_{\mathit{img}}$ and $\mathit{Los}{s}_{CCM\_\text{S}}$ are constructed for the feature map $\mathit{Sp}{O}_{2_{\mathit{CCM}}}$ and the estimation value $\mathit{Sp}{O}_{2_{\mathit{CCM}}}$ respectively to improve the ${\text{SpO}}_2$ estimation accuracy.

The feature $F^{″}$ is first fed into the image generator to reconstruct the RGB image. The structure of the image generator is shown in Fig. 3 . This means that $F^{″}$ is first fed into the residual block. Then, send the result to a transposed convolution kernel of size $[3,3]$ which named $\mathit{Conv}{T}_{3\times 3}$ for upsampling. Next, send it into normalization and non-linear activation to obtain $I^{\prime }$. Finally, $I^{\prime }$ is fed into a convolution kernel of size $[7,7]$ named $\mathit{Con}{v}_{7\times 7}$ to obtain the RGB feature map $\mathit{feat}$ which pixel size is 3 * 224 * 224.

Fig. 3.

Structure of Image Generator.

Then, the reconstructed $\mathit{feat}$ is fed into ${\text{SpO}}_2$ Estimator1. Here, the signals of the red and blue channels are calculated by conversion to obtain cardiovascular pulse wave signals that can replace two different wavelengths (660 nm and 940 nm) in the pulse ${\text{SpO}}_2$ meter [4], thus using the AC/DC ratio estimation, whose mathematical equation for the conversion calculation is shown in Equation (9):

$$\mathit{Sp}{O}_{2CCM}=A-B\frac{A{C}_{\mathit{RED}}/D{C}_{\mathit{RED}}}{A{C}_{\mathit{BLUE}}/D{C}_{\mathit{BLUE}}}$$ (9)

Here, $A{C}_{\mathit{RED}}$ and $A{C}_{\mathit{BLUE}}$ represent the standard deviation of the red and blue channels respectively. $D{C}_{\mathit{RED}}$ and $D{C}_{\mathit{BLUE}}$ represent the mean of the red and blue channels respectively, with fixed coefficients A = 125 and B = 26 based on the empirical evaluation [4].

In this section, loss functions are designed for $\mathit{feat}$ and $\mathit{Sp}{O}_{2_{\mathit{CCM}}}$ respectively. In order to learn more ${\text{SpO}}_2$-related information and preserve original image features as much as possible during the RCA network learning and feature map reconstruction, the L1Loss function is used to evaluate the loss between the reconstructed RGB feature map $\mathit{feat}$ and the original map $\mathit{img}$ to obtain $\mathit{Los}{s}_{\mathit{img}}$. H, W, and C represent the image height, width, and number of channels respectively.

$$\mathit{Los}{s}_{\mathit{img}}(img,feat)=\frac{1}{H\times W\times C}\sum _{i=1}^H\sum _{j=1}^W\sum _{c=1}^C\left|\mathit{im}{g}_{i,j,c}-fea{t}_{i,j,c}\right|$$ (10)

Besides, for guiding the Color Channel Model learning to obtain more accurate ${\text{SpO}}_2$ estimation values, the Smooth L1 loss function is used to evaluate the loss between $\mathit{Sp}{O}_{2_{\mathit{CCM}}}$ and the ground truth $\mathit{SpO}{2}_{\mathit{gt}}$ to obtain $\mathit{Los}{s}_{CCM\_\text{S}}$:

$$\mathit{Los}{s}_{CCM\_S}(x,y)=\frac{1}{L}\sum _{i=1}^L\left\{\begin{array}{c}0.5\ast {(y_i-x_i)}^2,if\left|y_i-x_i\right|<1\\ \left|y_i-x_i\right|-0.5,otherwise\end{array}\right)$$ (11)

Here, L represents the length of the input video frame, $x$ and $y$ represent the estimated value $\mathit{Sp}{O}_{2_{\mathit{CCM}}}$ and the ground truth $\mathit{SpO}{2}_{\mathit{gt}}$ respectively. When the absolute value of the difference between $x$ and $y$ is less than 1, L2 Loss is used. When the difference is larger, the translation of L1 Loss is used. The Smooth L1 loss function has a constant gradient when the difference between $x$ and $y$ is large solves the problem of large gradients destroying the training parameters in L2 loss. When the difference is small, the gradient decreases dynamically, solving the problem of difficult convergence in L1 loss.

By designing ${\lambda }_{\mathit{img}}$ and ${\lambda }_{CCM\_S}$ as balancing parameters to adjust the importance of $\mathit{Los}{s}_{\mathit{img}}$ and $\mathit{Los}{s}_{\mathit{CCM}\text{- S}}$, the total loss of the Color Channel Model is denoted to estimate the ${\text{SpO}}_2$ value as $\mathit{Los}{s}_{\mathit{CCM}}$:

$$\mathit{Los}{s}_{\mathit{CCM}}={\lambda }_{\mathit{img}}\times Los{s}_{\mathit{img}}+{\lambda }_{CCM\_S}\times Los{s}_{CCM\_\text{S}}$$ (12)

3.2.2. Network-based model

The structure of the Network-Based Model is shown in Fig. 1. $F^{″}$ learned by the RCA network is fed into the ${\text{SpO}}_2$ Estimator2 to calculate the ${\text{SpO}}_2$ value $\mathit{Sp}{O}_{2_{\mathit{NBM}}}$. The two-part loss functions including $\mathit{Los}{s}_{\mathit{label}}$ and $\mathit{Los}{s}_{NBM\_S}$ between $\mathit{Sp}{O}_{2_{\mathit{NBM}}}$ and the ${\text{SpO}}_2$ ground truth $\mathit{SpO}{2}_{\mathit{gt}}$ monitored by the pulse oximeter are constructed to optimize the model, so that the model can focus on more ${\text{SpO}}_2$ information and improve the accuracy of ${\text{SpO}}_2$ estimation.

The structure of ${\text{SpO}}_2$ Estimator2 is shown in Fig. 4 . Firstly, the high-dimensional features $F^{″}$ generated by the RCA network are globally averaged and pooled to replace the fully connected operation to reduce the network parameters and overfitting. Then, feed it into a convolution named $\mathit{Con}{v}_{1\times 1}$ with the kernel of size $[1,1]$ to compress the number of channels, increase the nonlinearity of the network and obtain the feature map with 100 channels. Then, activate it by the $\mathit{SoftMax}$ function to obtain $F^{\mathit{pre}}$. Since ${\text{SpO}}_2$ can take values from 1 % to 100 %, the sequential vector $S=\left[1,2,...,100\right]$ is constructed and multiplied with $F^{\mathit{pre}}$ to obtain the estimated value $\mathit{Sp}{O}_{2_{\mathit{NBM}}}$.

$$F^{\mathit{pre}}=\delta (Con{v}_{1\times 1}(GAP(F^{″})))$$ (13)

$$\mathit{Sp}{O}_{2NBM}\text{=}\sum _{i=1}^{100}{S}_i\times F_i^{\mathit{pre}}$$ (14)

Fig. 4.

Structure of ${\text{SpO}}_2$ Estimator2.

Since the face videos contain not only ${\text{SpO}}_2$-related signals, but also some other physiological signals. To motivate the network to learn more ${\text{SpO}}_2$ signals than other signals, a two-part loss function is designed for $F^{\mathit{pre}}$ and $\mathit{Sp}{O}_{2_{\mathit{NBM}}}$ in this section. First, the ${\text{SpO}}_2$ ground truth $\mathit{SpO}{2}_{\mathit{gt}}$ is processed by the label distribution learning technique [33] to obtain a matrix label $l$ of the same size as $F^{\mathit{pre}}$. The Kullback-Leibler Dispersion [34] (KLD) loss $\mathit{Los}{s}_{\mathit{label}}$ between $F^{\mathit{pre}}$ and $l$ is constructed so that the real label $l$ is used to direct $F^{\mathit{pre}}$ to focus on more ${\text{SpO}}_2$ information. The rule for calculating $\mathit{Los}{s}_{\mathit{label}}$ is shown in Equation (15).

$$\mathit{Los}{s}_{\mathit{label}}(l\Vert F^{\mathit{pre}})=\sum _{i=1}^T\sum _{j=1}^{100}[l_{\mathit{ij}}\log l_{\mathit{ij}}-l_{\mathit{ij}}\log F_{\mathit{ij}}^{\mathit{pre}}]$$ (15)

Besides, to guide the Color Channel Model learning to obtain more accurate ${\text{SpO}}_2$ estimation values, the Smooth L1 loss function is used to evaluate the loss between $\mathit{Sp}{O}_{2_{\mathit{NBM}}}$ and the ground truth $\mathit{SpO}{2}_{\mathit{gt}}$ to obtain $\mathit{Los}{s}_{NBM\_S}$. It follows the same rules as Equation (11).

$$\mathit{Los}{s}_{NBM\_S}(x,y)=\frac{1}{L}\sum _{i=1}^L\left\{\begin{array}{c}0.5\ast {(y_i-x_i)}^2,if\left|y_i-x_i\right|<1\\ \left|y_i-x_i\right|-0.5,otherwise\end{array}\right)$$ (16)

Here, $x$ and $y$ represent the estimated value $\mathit{Sp}{O}_{2_{\mathit{NBM}}}$ and the ground truth $\mathit{SpO}{2}_{\mathit{gt}}$ respectively.

By designing ${\lambda }_{\mathit{label}}$ and ${\lambda }_{NBM\_S}$ as balancing parameters to adjust the importance of $\mathit{Los}{s}_{\mathit{label}}$ and $\mathit{Los}{s}_{NBM\_S}$, the total loss of the Network-Based Model is denoted to estimate the ${\text{SpO}}_2$ value as $\mathit{Los}{s}_{\mathit{NBM}}$:

$$\mathit{Los}{s}_{\mathit{NBM}}={\lambda }_{\mathit{label}}\times Los{s}_{\mathit{label}}+{\lambda }_{NBM\_S}\times Los{s}_{\mathit{NBM}}$$ (17)

3.2.3. Multi-Model fusion model

To facilitate the network to focus on and learn ${\text{SpO}}_2$ features in many aspects and to improve the accuracy of ${\text{SpO}}_2$ estimation, the Color Channel Model is combined with the Network-Based Model to construct a Multi-Model Fusion Model (MMFM). Using the SmoothL1Loss loss function, a new loss named $\mathit{Los}{s}_{\mathit{MMFM}}$ is constructed for the estimation values $\mathit{Sp}{O}_{2_{\mathit{CCM}}}$ and $\mathit{Sp}{O}_{2_{\mathit{NBM}}}$ of the two models, allowing the two models bootstrap each other, making full use of the features of interest to both and the complementarity between them.

$$\mathit{Los}{s}_{\mathit{MMFM}}(x,y)=\frac{1}{L}\sum _{i=1}^L\left\{\begin{array}{c}0.5\ast {(y_i-x_i)}^2,if\left|y_i-x_i\right|<1\\ \left|y_i-x_i\right|-0.5,otherwise\end{array}\right)$$ (18)

Here, $x$ and $y$ represent $\mathit{Sp}{O}_{2_{\mathit{CCM}}}$ and $\mathit{Sp}{O}_{2_{\mathit{NBM}}}$ respectively. It follows the same calculation rules as Equation(11) and Equation(16). The importance of $\mathit{Los}{s}_{\mathit{NBM}}$, $\mathit{Los}{s}_{\mathit{CCM}}$ and $\mathit{Los}{s}_{\mathit{MMFM}}$ is adjusted by designing equilibrium parameters ${\lambda }_{\mathit{NBM}}$, ${\lambda }_{\mathit{CCM}}$ and ${\lambda }_{\mathit{MMFM}}$, respectively.

$$\mathit{Los}{s}_{\mathit{total}}={\lambda }_{\mathit{NBM}}\times Los{s}_{\mathit{NBM}}+{\lambda }_{\mathit{CCM}}\times Los{s}_{\mathit{CCM}}+{\lambda }_{\mathit{MMFM}}\times Los{s}_{\mathit{MMFM}}$$

4. Implementation processes

This section introduces the implementation processes of the remote multi-model fusion ${\text{SpO}}_2$ estimation model based on deep learning proposed in this paper. The model is divided into three parts: facial video preprocessing, the RCA network model training and ${\text{SpO}}_2$ estimation.

4.1. Video preprocessing

In this paper, the PURE and VIPL-HR datasets are chosen for experiments, which contain face video sequences, timestamped text and ${\text{SpO}}_2$-valued text corresponding to video frames. All video sequences need to be processed by face detection, localization (Fig. 5 ) and normalization. The specific steps are as follows.

Fig. 5.

Facial cutting process.

Step 1. Face detection. Firstly, the images are input to a multi-task convolutional neural network (MTCNN) [35] for face detection and localization.

Step 2. Key landmark localization. During the key landmark localization process, we pre-downloaded an open-source file containing the 81 landmarks [36] of the face. Subsequently, we take the file as an argument and call the method named shape_predictor of the Dlib library to achieve the localization of the key landmarks.

Step 3.The selection of regions of interest(ROI). The facial landmarks obtained in the previous stage are processed to locate the ROI region: the maximum and minimum values of the ×, y axes of 81 points are combined to form the four corner coordinates of the ROI and cropped.

Step 4. The facial image sequence normalization. All the facial images in this paper are normalized to 224 × 224 pixel RGB images.

4.2. RCA network model training

The processed facial image sequence $V\in R^{C\times L\times H\times W}$ is fed into the RCA network for feature extraction, where $C$, $L$, $H$ and $W$ represent the number of channels, length, height and width of the input video respectively.

$V$ is firstly subjected to $7\times 7$ convolutional transform and pooling, and then high-dimensional features with strong representation ability are extracted by residual block cascade and coordinate attention module. The image information is learned and ${\text{SpO}}_2$ values are estimated from them by three ${\text{SpO}}_2$ estimation models respectively. The model convergence speed is accelerated by adjusting the optimizer, learning rate, loss function, and other hyperparameters to obtain the best results.

4.3. SpO₂ estimation

The high-dimensional features are extracted from the RCA network, and the ${\text{SpO}}_2$ values are estimated by Color Channel Model, Network-Based Model, and Multi-Model Fusion Model respectively. It should be mentioned that the estimated values of both models and the corresponding errors are obtained in the fusion estimation model. The estimated value with the smallest error is taken as the result of the fusion estimation model. The effect is compared with the other two models to select the best model.

In addition, to avoid frame redundancy, the frame number is selected according to the setting of batch_size. In other words, the median of ${\text{SpO}}_2$ ground truth in T frames is used as $\mathit{SpO}{2}_{\mathit{gt}}$. The estimated values are obtained by averaging the T frames of ${\text{SpO}}_2$ values output from the three models. It also reduces the error caused by single-frame computation without considering time characteristics and noise.

5. Experimental results

5.1. Experimental datasets

Most of the datasets used in the existing ${\text{SpO}}_2$ research literature were taken by the authors themselves and are not public, and most of them had fewer subjects and scenes. Our experiments are conducted on the public datasets of PURE [16] and VIPL-HR [17], [18] for easier comparison. Both contain sufficient study subjects and rich scenarios to verify the robustness of the model. In addition, using the public dataset can facilitate method comparisons for subsequent studies. In the experiments, two datasets are divided 6:4 between the training and test datasets, and the information about them is shown in Table 1 .

Table 1.

Information of PURE and VIPL-HR.

Dataset	Num	Scenes	Acquisition equipment	Frame rate
PURE	10	•Steady •Speck •Slow translate •Fast translate •Small rotate •Medium rotate	•eco274CVGE Camera	30fps
VIPL-HR	107	•Steady •Head-move •Speck •Dark •Bright •Remote •After skipping	•RealSense F200 color camera	30fps

Fig. 6 (a) and (b) show histograms of the distribution of ${\text{SpO}}_2$ values collected from the PURE and VIPL-HR datasets respectively. The range of ${\text{SpO}}_2$ collected from the PURE dataset is 89–99 and the range from VIPL-HR is 86–99.

Fig. 6.

Histogram of the distribution of ${\text{SpO}}_2$ values.

5.2. Experimental settings

In this paper, MTCNN [35] model is used to detect and locate the face region of the original video, select the ROI and crop it, and normalize the face images to 224 × 224 pixels. We conduct experiments on the PURE dataset using the CCM model as a benchmark. The PURE dataset is small according to Table 1. Small dataset is more likely to show uncontrollable performance during the training process due to parameter changes. Therefore, experiments on the PURE dataset are more representative. Table 2 shows the experimental results for different image sizes. In all tables, the bold values indicate minimal errors.

Table 2.

Comparative experimentation of image sizes.

Dataset/Image size	PURE
	MAE (%)	RMSE (%)
3 * 32 * 32	1.02	1.48
3 * 112 * 112	0.83	1.07
3 * 196 * 196	0.72	0.97
3 * 224 * 224	0.65	0.85
3 * 304 * 304	0.76	0.99
3 * 352 * 352	0.78	1.05
3 * 448 * 448	0.88	1.22

According to Table2, increasing the pixel size of images, the images contain more texture and contextual information, which capture better features. In addition, it is easier to obtain discriminative features when images become larger. However, when the size becomes larger to a certain extent, the model complexity is too large, which tends to lead to overfitting and the experimental performance decreases instead. In addition, the computational overhead also becomes much larger accordingly. From experiments, the input image size is set to 3 * 224 * 224, which is more suitable.

Then, the image sequences are fed into RCA network model to extract features, and ${\text{SpO}}_2$ values are calculated using each of the three methods. In order to select the optimal optimizer and learning rate, rich ablation experiments are implemented. Four optimizers (RMSProp, Adagrad, Adam, AdamW) with different learning rates are used for model training and testing. Fig. 7 shows the comparative experimentation of different learning rates and optimizers.

Fig. 7.

Comparative experimentation of different learning rates and optimizers.

From Fig. 7, when the RMSProp optimizer is used and the learning rate is 0.02 or 0.05, the situation arises where the model is difficult to converge. When the Adam or AdamW optimizer are used and the learning rate is 0.05, our model is also difficult to converge. As a whole, it is optimal to set the learning rate to 0.01. During the training process, the Adam optimizer is used to continuously adjust the learning rate to achieve the best results.

In general, the optimal experimental setup is determined with rich experiments. The $\mathit{Los}{s}_{\mathit{total}}$ in Section 3.2.3 is used as the loss function. The Adam optimizer with a weight decay of 1e−4 is employed. The learning rate is set to 0.01. The ReduceLROnPlateau scheduler with a patience of 20 and factor of 0.1 is employed. The maximal epoch number and early stopping counter are set to 100 and 20, respectively. The batch_size is set to 50 frames on the PURE dataset. Extensive experiments verify that the above choice of parameters is optimal. Then, the converged model is migrated to the VIPL-HR dataset, and some hyperparameters are fine-tuned. The model network frameworks are all implemented based on PyTorch, and the graphics card used for the experiments is NVIDIA GTX1080TI.

5.3. Evaluation metrics

In order to verify the validity of the model, the mean absolute error (MAE) and root mean square error (RMSE) are used to present the experimental results, and the MAE and RMSE are calculated as shown in Equation (20), (21):

$$\mathit{MAE}\text{=}\frac{{\sum }_{i=1}^L\left|\mathit{SpO}{2}_{\mathit{pre}}(i)-SpO{2}_{\mathit{gt}}(i)\right|}{L}$$ (20)

$$\mathit{RMSE}\text{=}\sqrt{\frac{\sum _{i=1}^L{\left(\mathit{SpO}{2}_{\mathit{pre}}(i)-SpO{2}_{\mathit{gt}}(i)\right)}^2}{L}}$$ (21)

Here, L represents the length of the input video frame, $\mathit{SpO}{2}_{\mathit{pre}}$ represents the estimated value of ${\text{SpO}}_2$, and $\mathit{SpO}{2}_{\mathit{gt}}$ represents the ${\text{SpO}}_2$ ground truth.

5.4. Experiment and discussion

As none of the studies on contactless estimation of ${\text{SpO}}_2$ have published its datasets and algorithms, this paper will use medical indicators to measure the model's validity. If MAE between the estimated ${\text{SpO}}_2$ value of the model and the ${\text{SpO}}_2$ ground truth is within 2 %, ${\text{SpO}}_2$ is valid and the model is reliable [15].

5.4.1. Max-min coordinate face image cropping

In this paper, the maximum and minimum values of the × and y coordinates of 81 key points of the face are combined into the four corner point coordinates of ROI. The face is cropped by the coordinates, which preserves the most effective area of the face and reduces the effect of some areas of the face being occluded. In order to verify this conclusion, this paper does a comparison experiment between full face cropping and maximum-minimum boundary cropping (m_face), and here the Color Channel Model is used for verification. The results are shown in Table 3 . ${\text{SpO}}_2$ values are given in %. Therefore, MAE and RMSE are also in %. We mark it in all of the experimental tables accordingly.

Table 3.

Comparative experimentation of facial cutting methods.

Dataset/Crop Type	PURE		VIPL-HR
	MAE (%)	RMSE (%)	MAE (%)	RMSE (%)
all_face	0.86	1.18	1.15	1.61
m_face	0.65	0.85	1.04	1.48

The proposed image cropping method calculates the ${\text{SpO}}_2$ values for the PURE and VIPL-HR datasets respectively. Fig. 8 and Fig. 9 show the comparison of the ${\text{SpO}}_2$ estimation values and the reference values for the two datasets respectively. The horizontal label in Fig. 8 and Fig. 9 is “Batch”. It means the ${\text{SpO}}_2$ value calculated for the nth batch of the image sequences. In order to closely reflect the true human ${\text{SpO}}_2$ status, the estimated ${\text{SpO}}_2$ are retained to two decimal places. The retention of these two decimals gives rise to reasonable fluctuations in Fig. 8 and Fig. 9.

Fig. 8.

Experimental effect of PURE dataset.

Fig. 9.

Experimental effect of VIPL-HR dataset.

It is of interest to note the abrupt changes in Fig. 8 and Fig. 9 that the model failed to detect, such as the point around 80 in Fig. 8. By analysing the details of the original dataset, the reason for the problem has been found. In fact, during the construction of the dataset, there are discontinuities and interruptions when shooting different scenes of the same object. Therefore, during this interval, the ${\text{SpO}}_2$ status may change, and naturally the data within the corresponding dataset may also change abruptly. As datasets are feed as a whole continuously into the model for training and testing, the model does not immediately capture the sudden changes in ${\text{SpO}}_2$ values due to scene switching, yet it is relatively seldom the case in general. In addition, undetectable abrupt changes are within 1 % and return to pre-change ${\text{SpO}}_2$ status rapidly. In practice, in the actual scenario, the permissible error range is 2 %. It indicates that the estimation error due to the switching of detection objects and scenes is still acceptable. But whether or not it is these reasons that brought about the changes, when the magnitude exceeds 1 %, our model is still able to capture the change relatively quickly and estimate close ${\text{SpO}}_2$ values. For example, the drop in ${\text{SpO}}_2$ values at point around 120 in Fig. 8 and the rise at point around 160 in Fig. 9. It indicates that our model is effective and has application values.

Besides, since the normal range of ${\text{SpO}}_2$ values in the human body are 95 %–100 %, values below 95 % are considered to be low ${\text{SpO}}_2$ values. The proportion of low ${\text{SpO}}_2$ values is relatively small. The PURE dataset captures ${\text{SpO}}_2$ values once per frame at a frame rate of 30 fps and the percentage of the low ${\text{SpO}}_2$ value is 4.9 %. In contrast, the VIPL-HR dataset collects ${\text{SpO}}_2$ values once every second and the percentage of the low ${\text{SpO}}_2$ value is 5.8 %. However, in clinical scenarios, low ${\text{SpO}}_2$ samples need to be given more attention. Therefore, in addition to giving the overall ${\text{SpO}}_2$ estimation errors for PURE and VIPL-HR dataset, low ${\text{SpO}}_2$ values (<95) have been purposefully extracted for error calculations with the ${\text{SpO}}_2$ values calculated by the Color Channel Model. The MAE in PURE dataset is 1.28 %, and the MAE in VIPL-HR dataset is 1.57 %. From the experimental results, even with the low ${\text{SpO}}_2$ scenarios, the MAEs are still below 2 % and meeting the clinical target. It demonstrates that our model is valid for the low ${\text{SpO}}_2$ situations as well.

5.4.2. RCA network

In this section, rich ablation experiments are conducted for the RCA network model, and each module and parameter inside the model, and the effectiveness of the model and each module is verified on the PURE and VIPL-HR datasets using the Color Channel Model.

First, the setting of batch_size affects the experimental results and reasonableness. Each batch will get the corresponding ${\text{SpO}}_2$ estimation value and the average of the corresponding batch_size ${\text{SpO}}_2$ ground truth, from which the overall MAE and RMSE of the test sets of PURE and VIPL-HR are calculated. Due to the limitations of the hardware device, a too large batch_size will result in insufficient memory to allow for training. A too small bitch_size will result in too little data being trained in a single batch. It may lead to large experimental errors and unstable experimental results. At this point, it is difficult to extract more global features. Therefore, we try to choose a multiple of 10 in the appropriate region for the ablation experiments. Besides, due to the experimental setup, the minimum value of batch_size is 15. Fig. 10 shows the comparison experiments of different batch_size.

Fig. 10.

Plot of batch_size comparison experiments.

From Table 1 and Fig. 10, the PURE dataset is small and it is significantly affected by batch_size. In the process of randomly selecting a small batch_size and training, some uncontrollable factors are generated, resulting in fluctuations in the curve. Nevertheless, as the batch_size increases, PURE as a whole also produces the effect within the expectation, which is consistent with the training effect of large dataset. Finally, we choose 50 as batch_size after several experimental comparisons. Combined with the frame rate of the dataset, it can be considered reasonable to do the estimation and error calculation every less than 2 s with this setting. Subsequent experiments were set to a batch size of 50 frames.

Afterward, to verify the effectiveness of the RCA network and the internal modules for extracting ${\text{SpO}}_2$ features, the RCA network is replaced by the feature extraction modules of classical deep neural networks (ResNet-50[31] and Inception v3[37]). The FLOPs and Params are used to measure the time complexity and memory complexity of the model respectively. FLOPs refer to floating point operations which is understood as the amount of computation. It can be used to measure the complexity of an algorithm or a model. Params refers to the number of parameters which represents the amount of memory used. Then, ablation experiments for residual blocks (RB) and the coordinate attention (CA) in the RCA are conducted. The above experiments are based on the Color Channel Model. According to Table 4 , the RCA network outperforms the other networks in terms of feature extraction capability. In addition, RCA networks have a lower time complexity and space complexity. It means that RCA network is more efficient and uses less memory. Our proposed model do not require much capturing equipment and calculation environment. Besides, both RB and CA can effectively improve the accuracy of ${\text{SpO}}_2$ estimation and reduce MAE and RMSE. RB reduces information attrition and improves the learning capability. Besides, CA can focuses on feature channels and the location information, and enhances the feature representation.

Table 4.

Ablation experiments with RCA network.

Dataset/Model	FLOPs (GMac)	Params (M)	PURE		VIPL-HR
			MAE (%)	RMSE (%)	MAE (%)	RMSE (%)
ResNet-50 [31]	5.12	24.05	1.02	1.49	1.50	1.77
Inception v3 [37]	3.67	22.68	1.04	1.47	1.14	1.51
RCA w/o CA	2.048	1.077	0.74	1.05	1.11	1.50
RCA w/o RB	0.94	0.11	1.02	1.44	1.15	1.53
RCA	2.05	1.08	0.65	0.85	1.04	1.48

Next, the selection experiment of the number of cascades for the cascaded residual network is conducted. According to Table 5 , the best results were obtained when two residual blocks were used for cascading, extracting the most effective high-dimensional features. Besides, using two residual blocks does not unduly increase FLOPs and Params.

Table 5.

Cascade number selection experiment for cascade residual blocks.

Dataset/RB num	FLOPs (GMac)	Params (M)	PURE		VIPL-HR
			MAE (%)	RMSE (%)	MAE (%)	RMSE (%)
1	1.4	0.26	0.79	1.02	1.12	1.54
2	2.05	1.08	0.65	0.85	1.04	1.48
3	2.69	4.36	0.89	1.15	1.10	1.50
4	3.33	17.47	1.02	1.44	1.15	1.52

After determining the experimental setup and the structure of the RCA model, visualisations were used to show the facial regions that the RCA model learned and focused on.

First, a set of images of faces wearing glasses and hair coverings and their attention graphs are as shown in Fig. 11 . As can be seen from them, the RCA network focuses more on the cheek and forehead regions. Meanwhile, the model reduces the attention to the eyes, eyeglasses, hair, lips and chin.

Fig. 11.

Visualisation of scenes with eyeglasses.

In addition, the visual scenes of facial rotation and movement and specking are as shown in Fig. 12 . As can be seen from them, the RCA network also reduces the focus on teeth and male beards. In non-stationary scenes, the RCA network still accurately focuses on the cheek and forehead regions.

Fig. 12.

Visualisation of non-stationary scenes.

In general, ${\text{SpO}}_2$ needs to be estimated from bare skin. Not focusing on disruptions such as beards, teeth, hair, eyes and eyeglasses helps to improve the accuracy of the model in detecting ${\text{SpO}}_2$.

Besides, to explore the effectiveness of deep learning in greater depth, a ${\text{SpO}}_2$ estimation experiment with traditional color channel segmentation is designed for comparison. The red and blue channel of the original image sequence are separated and fed into the RCA network separately to extract the signal features. The corresponding standard deviation and mean values were calculated respectively, and the ${\text{SpO}}_2$ values were calculated by substituting into the ${\text{SpO}}_2$ calculation Equation (9). Table 6 shows the experimental results, which demonstrate the effectiveness of deep learning in terms of the ${\text{SpO}}_2$ estimation accuracy.

Table 6.

Red and blue color channel method selection experiment.

Dataset/Channel Method	PURE		VIPL-HR
	MAE (%)	RMSE (%)	MAE (%)	RMSE (%)
Tradition Color Model	0.92	1.17	1.18	1.60
CCM	0.65	0.85	1.04	1.48

5.4.3. Comparative study of three ${\text{SpO}}_2$ estimation models

After the experiments in the previous section, the RCA network structure is determined, and then the three ${\text{SpO}}_2$ estimation models are trained and tested separately, the results are shown in Table 7 .

Table 7.

${\text{SpO}}_2$ estimation model comparison experiment.

Dataset/Estimate Model	FLOPs (GMac)	Params (M)	PURE		VIPL-HR
			MAE (%)	RMSE (%)	MAE (%)	RMSE (%)
CCM	2.05	1.08	0.65	0.85	1.04	1.48
NBM	1.00	0.70	0.86	1.04	1.10	1.50
MMFM	3.05	1.77	0.63	0.80	1.00	1.43

As can be seen from the results, the MMFM has the lowest error in ${\text{SpO}}_2$ estimation, the conventional CCM has the middle error, and the NBM has the highest error, but all of them are within 2 % of the clinical target. The MMFM effectively combines the advantages of the NBM and the CCM, making full use of the characteristics and complementarity between the two models to achieve relatively better estimation results. The complexity of MMFM is the sum of the complexity of CCM and NBN.

Besides, to evaluate the real-time property and the latency property, extensive experiments are done. Specifically, the time required to load the pre-trained model, and the time required to load the batch_size (50) frames and feed them into the models to estimate the ${\text{SpO}}_2$ are calculated. Separate experiments are conducted on the PURE and VIPL-HR datasets. Both of them have similar loading and estimating times, so their average time are chosen as the result. In Color Channel Model, the loading time is about 1.1 s, and the estimating time is about 0.5 s. In Network-Based Model, the loading time is about 0.4 s, and the estimating time is about 0.5 s. In Multi-Model Fusion Model, the loading time is about1.6 s, and the estimating time is about 0.5 s. Overall, the CCM, NBM and MMFM all take around 0.5 s to estimate ${\text{SpO}}_2$ after the pre-trained models are loaded. The delay of 0.5 s is acceptable in terms of timeliness.

In addition, ablation experiments on the loss functions are conducted to verify the effectiveness of $\mathit{Los}{s}_{\mathit{img}}$ and $\mathit{Los}{s}_{\mathit{label}}$ for CCM, NBM and MMFM respectively. The experimental results are shown in Table 8 , which can confirm that the reduced image loss and KLD loss help to improve the model estimation accuracy and reduce its error.

Table 8.

Loss function ablation experiments.

	$\mathit{Los}{s}_{\mathit{img}}$	$\mathit{Los}{s}_{\mathit{label}}$	PURE		VIPL-HR
			MAE (%)	RMSE (%)	MAE (%)	RMSE (%)
CCM	✗	—	0.81	1.05	1.15	1.55
	✓	—	0.65	0.85	1.04	1.48
NBM	—	✗	0.89	1.13	1.19	1.56
	—	✓	0.86	1.04	1.10	1.50
MMFM	✗	✗	0.87	1.06	1.18	1.59
	✓	✗	0.77	1.01	1.08	1.46
	✗	✓	0.82	0.99	1.14	1.48
	✓	✓	0.63	0.80	1.00	1.43

5.4.4. Sub-scene verification

To analyze whether the RCA model still maintains better performance under head motion states and different lighting conditions, three ${\text{SpO}}_2$ estimation models will be individually trained and validated for several unstable scenes in the PURE and VIPL-HR datasets respectively to improve the robustness of the model. The experimental results are shown in Fig. 13 and Fig. 14 .

Fig. 13.

Comparison of experimental results of the three models in PURE by scene.

Fig. 14.

Comparison of experimental results of the three models in VIPL-HR by scene.

According to Fig. 13, the estimation errors of MMFM are lower than those of NBM and CCM in different scenes of the PURE dataset. NBM outperforms CCM only in the steady scenes and performs worse than CCM in the rest of the unstable scenes.

According to Fig. 14, each of the three estimation models has advantages and disadvantages in different scenarios in the VIPL-HR dataset, but all of them are relatively small in difference. Based on the presentation of the dataset in Table 1, it can be seen that VIPL-HR is large compared to PURE. In the smaller PURE dataset, the results of the three estimation models differ significantly and the Color Channel Model significantly outperforms the Network-Based Model. In the larger VIPL-HR dataset, the difference between the three models is tiny. Through these experiments, this paper concludes that when the experimental samples are large enough and the learning objects are large enough, the features extracted from the RCA network can gradually be detected by the network model with comparable accuracy to that of the Color Channel Model. In addition, MMFM takes into account the advantages of CCM and NBM to achieve a more accurate result.

5.4.5. Comparison with other models

We summarise articles on contactless ${\text{SpO}}_2$ estimation over the years. From these, representative articles are selected to compare the experimental results with MMFM. Since the code and dataset of them are not public, the MAEs in these articles are extracted for comparison as shown in Table 9 . The Num denotes the number of volunteers involved in the construction of these datasets. DL denotes Deep Learning. Area denotes the area of the body to be photographed.

Table 9.

Comparison of different contactless ${\text{SpO}}_2$ estimation methods.

Method	Dataset	Num	MAE (%)	DL	Area
[25] 2016	Self-built	9	1.00	✗	hands
[38] 2020	Self-built	9	0.85	✗	face
[39] 2020	Self-built	21	0.83	✗	face
[40] 2021	Self-built	25	1.7	✗	face
[12] 2021	Self-built	14	1.81	✓	hands
[41] 2022	BIDMC PPG [42]	52	1.45	✗	PPG signal
*[38] 2020	PURE	10	1.51	✗	face
	VIPL-HR	107	2.32
*[39] 2020	PURE	10	1.30	✗	face
	VIPL-HR	107	2.02
Ours-MMFM	PURE	10	0.63	✓	face
	VIPL-HR	107	1.00

The literature [38] presented a ${\text{SpO}}_2$ monitoring method without physical contact with the patient using imaging photoplethysmography (iPPG). Firstly, the authors used a camera to capture videos and extracted the forehead area as an ROI. Then, they performed Eulerian video magnification (EVM) [43] on the facial videos to amplify the changes in skin color due to the heart cycle. Lastly, the red and blue channel ratio method is used to calculate ${\text{SpO}}_2$ values.

The literature [39] presented a ${\text{SpO}}_2$ monitoring method by using a video camera. The authors used the face detector provided by the Dlib library to detect faces in the images. Then, they select the forehead and left and right cheeks as three ROIs. To track the ROIs in the videos, they used the Kanade-Lucas-Tomasi (KLT) [44] tracking method. The next step was the analysis of the RGB signals coming from the ROIs marked in each video frame of a short sequence. To avoid noise, the signals are improved through a preprocessing phase. Next, they used Power Spectral Density (PSD) through Welch’s method [45] to select the most informative ROI. Finally, they used the chosen ROI to calculate ${\text{SpO}}_2$ values by the red and blue channel ratio method.

As the source code of the literature [38], [39] is not publicly available, we reproduce the model based on details from the literature and do comparative experiments on the PURE and VIPL-HR datasets. Therefore, we mark them with * in Table 9. By reproducing the models from the literature [38], [39] and conducting comparative experiments on the PURE and VIPL-HR datasets, the RCA network and the MMFM model are proven to be more effective in extracting ${\text{SpO}}_2$-related feature from facial image sequences. Besides, it demonstrates the effectiveness of deep learning in the field of contactless ${\text{SpO}}_2$ measurement as well.

In addition, we conduct an experimental comparison with the literature [41] published in August 2022. It is an extremely new literature. It utilizes PPG signals and machine learning technology to extract ${\text{SpO}}_2$ and achieve clinical indicators. But in general, MMFM can adapt better to the ${\text{SpO}}_2$ estimation task. Furthermore, it confirms the effectiveness of deep learning in the area of contactless ${\text{SpO}}_2$ detection.

Since there is no method based on deep learning for Sp02 estimation by face videos, it is important to compare our model with a simple deep neural network trained with a regression loss. Therefore, we experiment with ${\text{SpO}}_2$ estimation task on Classical network models(ResNet-18[31] and VGG16[46]) with L1 Loss and compare these results. According to Table 10 , the MMFM has a slightly higher time complexity than ResNet-18, but the lowest spatial complexity of the three models. The MMFM has the lowest detection error and highest accuracy. Overall, the estimation model of this paper is more applicable to the ${\text{SpO}}_2$ estimation task.

Table 10.

Deep learning model comparison experiments.

Dataset/Model	FLOPs (GMac)	Params (M)	PURE		VIPL-HR
			MAE (%)	RMSE (%)	MAE (%)	RMSE (%)
VGG16 [46]	15.38	72.34	2.00	2.66	2.94	3.21
RestNet-18 [31]	1.82	11.18	1.16	1.37	1.98	2.23
Ours-MMFM	3.05	1.77	0.63	0.80	1.00	1.43

6. Conclusion

In contactless estimating of ${\text{SpO}}_2$, traditional contactless methods require elaborate and detailed signal processing techniques to meet medical specifications. How to develop deep learning techniques for contactless ${\text{SpO}}_2$ estimation for better accuracy is the focus of our research. The multi-model fusion method based on deep learning is proposed in this paper. Firstly, the maximum and minimum corner cropping of faces is proposed. It can preserve the most effective information about faces and reduce the impact of facial occlusion. Next, we combine the residual network module and the coordinate attention module as the RCA network to obtain a high-dimensional feature map with greater representation. Finally, the ${\text{SpO}}_2$ values are calculated by three ${\text{SpO}}_2$ estimation models(CCM, NBM and MMFM). Their advantages and disadvantages are compared. From the experimental results on the PURE and VIPL-HR datasets, the MAEs of three models are less than 2 % and all of them meet the clinical requirement. It confirms the effectiveness of deep learning for contactless ${\text{SpO}}_2$ estimation from face videos. Additionally, the MMFM combines the advantages of CCM and NBM effectively. It fully uses the features of interest and complementarity between the two models to achieve more accurate estimation results.

In future research, we will focus on how to construct deep learning models to obtain a more accurate ${\text{SpO}}_2$ estimation. For example, we can design our network models based on large pre-trained models for migration learning, and use the latest relevant variants based on Transformer instead of the traditional convolution. Beisides, we will focus on improving the ability to deal with small, transient changes in ${\text{SpO}}_2$ status for more valuable applications. Furthermore, how to minimize the redundancy of videos and how to achieve real-time monitoring to achieve practical value are also directions for our future research. We will continue to improve model accuracy and performance and apply it to telemedicine and home health.

CRediT authorship contribution statement

Min Hu: Conceptualization, Methodology, Software, Formal analysis, Investigation, Writing – original draft, Writing – review & editing, Visualization. Xia Wu: Conceptualization, Methodology, Software, Writing – original draft, Investigation, Data curation, Formal analysis, Writing – review & editing, Visualization. Xiaohua Wang: Conceptualization, Validation, Writing – review & editing, Supervision. Yan Xing: Conceptualization, Investigation, Validation, Resources, Supervision. Ning An: Conceptualization, Writing – review & editing, Resources, Supervision. Piao Shi: Validation, Resources, Writing – review & editing, Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

Data will be made available on request.