Robust blood pressure measurement from facial videos in diverse environments

Abstract

Blood pressure (BP) management is important worldwide, and BP monitoring is a crucial aspect of maintaining good health. Traditional BP meter measures BP independently in various situations, such as at home or work, using a cuff to maintain a stable condition. However, these devices can causes a foreign body sensation and discomfort, and are not always practical for periodic monitoring. As a result, studies have been conducted on the use of photoplethysmography (PPG) for measuring BP. However, PPG also has limitations similar to those of traditional BP meters, as it requires the placement of sensors on two regions of the body (fingers or toes). To address this issue, researchers have conducted studies on non-contact methods for measuring BP using face and hand videos. These studies have utilized two cameras to measure PTT and have focused on internal environments, resulting in low accuracy of BP measurement in external environments. We proposes a method for robust BP measurement using pulse wave velocity (PWV) and PTT calculated from facial videos. PTT is estimated by measuring the phase difference between two different regions of interest (ROIs) and PWV is calculated using PTT and the actual distance between two ROIs. In addition, our proposed method extracts the pulse wave from the ROI to measure BP. The actual distance between the ROIs and PTT are estimated using the two extracted pulse waves, and BP is then measured using PWV and PTT. To evaluate the BP measurement performance, the BP calculated from both BP meters and facial videos (in indoor, outdoor, driving car, and flying drone environments) are compared. Our results reveal that the proposed method can robustly measure BP in diverse environments.

1. Introduction

Blood pressure (BP) is a critical component of vital signs used to assess an individual's health status. Hypertension, which is a risk factor for cariovascular diseases, is most commonly observed in patients [[1], [2], [3], [4], [5], [6], [7]]. BP is the pressure generated in the blood vessels as blood is pumped out of the heart, and this pressure varies according to the heart's contraction and relaxation. Systolic BP (maximum pressure) and diastolic BP (minimum pressure) are the two types of BP. Hypertension may result in complications such as heart failure, myocardial infarction, cerebral infarction, and cerebral hemorrhage, while hypotension can lead to decreased metabolism and brain function [2,[7], [8], [9]].

According to the 2021 report on noncommunicable diseases by the World Health Organization (WHO), CVDs account for 32% of global deaths, with the majority of deaths resulting from hypertension [1,[10], [11], [12], [13], [14]]. Hypertension is generally asymptomatic and difficult to detect early [12], therefore, continuous BP monitoring is critical as it can prevent CVDs and reduce the risk of CVD-related complications in individuals who are at risk for hypertension [1]. The existing method for monitoring BP requires individuals to wear a cuff mounted on a BP meter for approximately 30s to obtain a highly accurate measurement [10,15]. However, this method restricts user movement, and incomplete contact with the cuff may result in the need for repeated BP measurements. Thus, it is not suitable for continuous monitoring. Moreover, for continuous BP monitoring, individuals must carry the BP meter with them at all times, and the cuff must be in firm contact during BP measurement [2,[6], [7], [8],10,16,17]. Traditional BP measurement methods using a cuff mounted BP meter are accurate, but require the user to remain still and can be uncomfortable due to cuff inflation. Furthermore, the need for one device per person makes it difficult to measure BP in large populations. To address these issures, researchers have investigated alternative methods using pulse waves [1,3,5,6,[9], [10], [11], [12], [13],[15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30]]and pulse transit time (PTT) [1,2,8,12,14,18,21,28,31,[32], [33], [34], [35], [36]], as well as PPG-based pulse wave velocity (PWV) [37,38] to measure BP without using a BP meter. While PPG and ECG devices enabled continuous BP monitoring, these methods still pose challenges. To overcome these challenges, non-contact methods have been developed that extract pulse waves from facial and palm images, minimizing user intervention and utilizing PTT calculated from the two pulse waves to monitor BP. These non-contact methods are suitable for use not only in internal environments but also in moving cars [29,31] and for individuals during sleep [18], making them a promising and versatile method for real-time BP monitoring in various fields. Studies on noncontact BP measurement have utilized various regression analysis methods such as linear regression (LR) [4,8,9,12,18,21,25,27,31,[32], [33], [34], [35]], multiple regression analysis (MRA) [2,3,6,11,16,20,29,30,33], and deep learning approaches such as CNN and LSTM [1,5,10,13,17,19,23,24,26,39]. These methods are used to predict the PTT between two pulse waves, or to extract features of the pulse wave (such as amplitude, width, and slope) to measure BP. While previous studies have demonstrated the feasibility of measuring blood pressure (BP) using facial and palm videos, they have mostly been conducted in controlled laboratory environments. However, accurate BP measurement in real-world outdoor settings (such as on park benches, in cars, on mountains, or using drones) remains a challenge due to unstable pulse waves caused by user movement and changes in illumination. Furthermore, these studies have relied on the use of pulse transit time (PTT) and limited user physiological information, as calculating the distance from the heart to the face and palms presents significant challenges. While the peak interval between pulse waves derived from the face and palms can be calculated, this limitation complicates the estimation of pulse wave velocity (PWV), which is strongly correlated with BP. To address this challenge, this study applies a moving average filter (MAF) to detect regions of interest (ROI) despite user body tremors and illumination changes in the external environment to ensure that the ROI could be stably detected. The ROI is then specified in the facial video captured in both internal or external environments, and the RGB color system of the ROI is converted to the YCgCo color system to calculate the Cg color signal. The Cg signal is then processed using FFT and iFFT, which limit the pulse-related frequency band (0.67–3.34 Hz)t o extract pulse waves. The actual distance between the two ROIs and PTT are estimated from the extracted pulse waves, the PWV is calculated using the PTT and actual distance. This approach enables robust BP measurement using PWV and PTT in diverse environments.

2. Related works

Non-contact remote BP measurements is increasingly gaining attention for addressing CVD-related issues and for commercial and academic purposes globally. In recent years, several studies have demonstrated the use of contactless general cameras for BP measurement monitoring, which offer superior advantages. Non-contact BP measurement enables continuous monitoring without pressure on the forearm, and the results are periodically output as bio-signals, such as magnetic waves and skin temperature, in contrast the one-time measurement (30s) used at home.

Existing studies differ in several ways, including how they specify the ROI for BP measurement, separate RGB color data from the specified ROI and preprocess it, extract pulse waves, and calculate PTT. The BP measurement process can be summarized as follows.

• (1)This method uses features of pulse waves extracted by applying ICA and Kalman filter to RGB color signals or the PTT calculated from two extracted ICA-based pulse waves [21].
• (2)This method uses features of pulse waves extracted by applying BPF to RGB color signals or the PTT calculated from two extracted BPF-based pulse waves [2,5,8,9,[12], [13], [14], [15],20,[22], [23], [24],[26], [27], [28],31,32,34,36].
• (3)This method employs features of pulse waves extracted by applying HPF to RGB color signals or the PTT calculated from two extracted HPF-based pulse waves [22,25].
• (4)This method employs features of pulse waves extracted by applying LPF to RGB color signals or the PTT calculated from two extracted LPF-based pulse waves [3,19,22,33].
• (5)This method uses features of pulse waves extracted by applying MAF to RGB color signals or PTT calculated from two extracted MAF-based pulse waves [30].
• (6)This method uses features of pulse waves extracted by applying a pair filter to RGB color signals or the PTT calculated from two extracted pair filter-based pulse waves [35].

Most existing studies extract pulse waves using filtering methods and measure BP using features of pulse waves [1,3,5,6,[9], [10], [11], [12], [13],[15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30]] or PTT [1,2,8,12,14,18,21,28,31,[32], [33], [34], [35], [36]] of facial videos. However, studies on other PPG-based BP observations have shown that PWV [37,38] has a higher correlation with BP compared with PTT. Nevertheless, PTT is often used because calculating the PWV using facial videos requires knowing the actual distance between ROIs. Morover, sudden illumination changes in RGB colors were not considered because the subject was allowed to remained stationary when measuring BP using color data from the ROI in the face video, and data were collected only in a constant indoor environment with negligible light change. Thus, all reported results achieved high accuracy due to simple data with little noise in controlled settings. However, if BP is measured in outdoor environments (outdoor benches and mountains), automobiles, and drone cameras, the camera may continue to shake when detecting the user's face, and illumination can change rapidly. Few studies have explored how these methods performed with challenging data when both illumination changes and minute subject body tremors are present. New datasets must be collected to apply and evaluate novel methods, and repetitive data collection consumes time.

Fig. 1 depicts the process of non-contact BP measurement, in which the optimal ROI is specified from facial images captured by a camera, and stable face detection is performed. Then, BP is measured using the estimated PTT and PWV, which are extracted from the Cg color signals obtained from the specified ROIs.

Fig. 1.

Flowchart of systolic & diastolic blood pressure measurement.

3. Robust blood pressure measurement method

In this study, a robust BP measurement method using facial videos in diverse environments was proposed. The preprocessing process involves applying stable facial detection and transforming the RGB color medel to the YCgCo color model in the skin ROI, where arterial blood flows from the face, in order to calculate the Cg signal. By applying FFT to the Cg signals extracted from two different ROIs, we calculate the maximum peak value (${Peak}_{\mathit{max}}$), frequency value ($f_{\max \_peak}$) for ${Peak}_{\mathit{max}}$, and frequency $phase$ using frequency-domain analysis. $PTT$ is estimated using $phase$ and $f_{\max \_peak}$, and $PWV$ is calculated using $PTT$ and the actual distance between the pixels of two different ROIs for the center point. Finally, we measure systolic blood pressure (SBP) and diastolic blood pressure (DBP) using the features.

3.1. ROI stabilization

The Haar-like feature is an object detection algorithm that uses AdaBoost Learning, which is a machine learning method widely used for face detection. This method detects an object by calculating the difference in brightness in a specific area, as shown in Fig. 2.

Fig. 2.

Expended Haar-like feature.

The Haar feature detects facial features such as the eyebrows and philtrum to locate the face. It has a high detection speed and provides accuracy results. Accurate and efficient face detection is crucial for the measurement of BP using facial videos, as it enables the detection of subtle changes in skin color associated with variations in blood flow.

As shown in Fig. 3, the $x$ and $y$ coordinates of the upper left point of the detected face area were defined, with the height and width of the detected face denoted by H and W respectively. Thanks to the accurate detection provided by the Haar algorithm, the face area could be captured even with large variations in coordinates, enabling the exclusion of any noise except for changes in skin color data.

Fig. 3.

Face detection using Haar feature.

Fig. 4 shows the original $x$ coordinates of the detected ROI, which exhibit high variability. To overcome this issue, in this study, MAF was applied by removing high-frequency components to eliminate noise other than skin color changes. Thus, the X and Y coordinates of rapidly changing faces were stabilized.

Fig. 4.

X-coordinate variability of ROI.

As shown in Fig. 5, the high-frequency components were eliminated for stable ROI detection by applying MAF to the $x$ and $y$ coordinates of n samples, and the detected face $x$ and $y$ coordinates with large changes were stably improved, as expressed in Eq. (1).

$$\overline{{roi\_x}_i}=\frac{{roi\_x}_{i-n+1}+{roi\_x}_{i-n+2}+{roi\_x}_{i-n+3}+\cdots +{roi\_x}_i}{n}\text{,}$$ (1)

where $\overline{{roi\_x}_i}$ denotes the average of $x$ and $y$ coordinates of $roi$ for $n$ samples, and ${roi\_x}_i$ denotes the $x$ and $y$ coordinates of the actual detected ROI per frame.

Fig. 5.

ROI stabilization process.

As shown in Fig. 4, Fig. 6, the calibrated $x$ coordinate variability can be utilized to measure BP with high accuracy, as it effectively removes noise other than skin color changes through the proposed ROI stabilization method.

Fig. 6.

Calibrated x-coordinate variability of ROI (n = 15).

3.2. ROI selection

In this study, we measured BP from facial videos by detecting skin color changes in the skin ROI where arterial blood flows. The PTT represents the peak delay of the pulse waves for the cardiac response.

PTT, an important feature for BP measurement, represents the delay of the pulse waves from the aortic valve to the peripheral stage. As shown in Fig. 7, Cg signals were calculated in two different skin regions (${roi}_1$ and ${roi}_2$) to determin PTT. We set roi_1 and roi_2 in close proximity to the direction of blood flow, and observed that the use of these regions of interest resulted in less noise and enabled more precise observation of changes in blood flow compared to when ROIs were placed in each left and right cheek individually. This was confirmed through analysis of the color signals. Moreover, this was found to be a crucial factor in the extraction of the pulse waveform and detection of peak points.

Fig. 7.

Comparison of roi (${roi}_1$ and ${roi}_2$).

3.3. Pulse wave extraction

To compare the performance with the proposed method, BP was estimated by defining the peak interval between ${roi}_1$ and ${roi}_2$ of the pulse waves calculated in two different regions as the PTT. Prior to calculating the PTT, the pulse wave was obtained using the Cg signal. It should be noted that the Cg color data is more suitable for pulse wave extraction than the R, G, and B color data, which are sensitive to illumination change. The Cg for the ROI is expressed as follows Eq. (2).

$${Cg\_signal}^i=\left[{Cg\_data}_1^i,{Cg\_data}_2^i,\cdots ,{Cg\_data}_n^i\right]\text{,}$$ (2)

Where ${Cg\_signal}^i$ denotes a set of ${cg\_data}_n^i$, $n$ represents the total number of frames, and the ${cg\_data}_n^i$ represents color conversion using $r,g,\text{and}bcolordata\text{,}$ expressed as follows Eq. (3).

$${Cg\_data}_n^i=128+\left(-0.25\times {\overline{r\_data}}_n^i+0.5\times {\overline{g\_data}}_n^i-0.25\times {\bar{b}\_data}_n^i\right)\text{,}$$ (3)

where ${\overline{r\_data}}_k^i$, ${\overline{g\_data}}_k^i$, and ${\overline{b\_data}}_k^i$ represent the average of the $r$, $g$, and b $\text{color}\text{data}$ of all pixels in the skin $roi$, respectively, and $k$ is the value calculated for each pixel of one frame for the $i$ th $roi$, which is expressed by Eq. (4), as follows.

$${\overline{color\_data}}_k^i=\frac{1}{p^i}\sum _{\left(x,y\right)\in R^i}{color\_data}_k\left(x,y\right)\text{,}$$ (4)

where $p^i$ is the number of all pixels in the $i$ th $roi$, ${\overline{color\_data}}^i$ is the setting $i$ th $roi$, $\left(x,y\right)$ are the x and y coordinates of a pixel, and ${color\_data}_n$ represents the $r$, g, and b color values.

As shown in Fig. 8, the frequency domain can be identified by applying the FFT to the Cg signal calculated in the $ithroi$. The pulse-related frequency band was 0.67–3.34 Hz, which is predicted to be 85.002 ($f_{\max \_peak}\times 60$) bpm when estimating a pulse using a frequency value and approximately 85 bpm when using a PPG device.

Fig. 8.

Cg signal extraction for 30s (Left) and FFT result of pulse-related frequency domain (Right).

Zero-padding was applied, excluding pulse-related frequency bands from the entire frequency amplitude calculated by applying FFT, which is expressed as Eq. (5). Pulse waves were extracted by applying iFFT to estimate the PTT using the pulse peak interval of the ${roi}_1$ and ${roi}_2$.

Fig. 9 illustrates the process of obtaining pulse waves by applying FFT-iFFT. A pulse wave was obtained by applying iFFT to the frequency values to which zero padding was applied. Zero padding is expressed using Eq. (5).

$$x_{zp}\left(n\right)=x\left(n\right),f_{\text{pulse}\_\mathit{min}}\le n\le f_{\text{pulse}\_\mathit{max}},=0,n<f_{pulse\_\mathit{min}},n>f_{pulse\_\mathit{max}}\text{,}$$ (5)

where $x$ represents the values in the frequency domain calculated by applying FFT to the Cg signal, $n$ represents the total number of samples, and $x_{zp}$ is the frequency data to which zero padding is applied. Zero-padding was applied to all areas except the pulse-related frequency band ($f_{pulse\_\mathit{min}}$ $–f_{pulse\_\mathit{max}}$ (Hz)), which is expressed by Eq. (6).

$$X\left(n\right)=\frac{1}{N}\sum _{k=0}^{N-1}{x}_{zp}\left(k\right)\times e^{j\left(\frac{2\pi }{N}\right)nk},\left(n=0,1,\cdots ,N-1\right)\text{.}$$ (6)

Fig. 9.

Extraction process of the pulse wave to measure the BP.

3.4. Robust blood pressure measurement method

In this study, the Cg signal calculated by the stable ROI Detection method was employed, PTT was calculated based on the phase difference between ${roi}_1$ and ${roi}_2$, and PWV was estimated using the actual distance between ${roi}_1$ and ${roi}_2$, and PTT to measure robust blood pressure.

3.4.1. ${Peak}_{\mathit{max}}$, $f_{\mathit{max}\_peak}$, and $phase$ difference for $f_{\mathit{max}\_peak}$ extraction

As shown in Fig. 10, to measure BP, the maximum peak of the pulse-related frequency band was detected to calculate the ${\mathit{P}\mathit{e}\mathit{a}\mathit{k}}_{\mathit{max}}$ and ${\mathit{f}}_{\mathit{max}\_\mathit{p}\mathit{e}\mathit{a}\mathit{k}}$. In addition, the phase of ${\mathit{f}}_{\mathit{max}\_\mathit{p}\mathit{e}\mathit{a}\mathit{k}}$ was extracted to calculate the phase difference between ${roi}_1$ and ${roi}_2$, which is expressed as follows Eq. (7).

$${\mathrm{\theta }}_d=\{\begin{array}{c}(2\pi -\left|{\mathrm{\theta }}_{{roi}_1}-{\mathrm{\theta }}_{{roi}_2}\right|if\left|{\mathrm{\theta }}_{{roi}_1}-{\mathrm{\theta }}_{{roi}_2}\right|>\pi \\ \left|{\mathrm{\theta }}_{{roi}_1}-{\mathrm{\theta }}_{{roi}_2}\right|otherwise\end{array}\text{,}$$ (7)

where ${\mathrm{\theta }}_d$ denotes the phase difference calculated for ${\text{roi}}_1$ and ${\text{roi}}_2$, ${\text{and}\mathrm{\theta }}_{{roi}_1}$ and ${\mathrm{\theta }}_{{roi}_2}$ are the phase values calculated for ${roi}_1$ and ${roi}_2$, respectively.

Fig. 10.

Peak detection and phase extraction of ${roi}_1$ and ${roi}_2$

3.4.2. PTT calculation using ${\mathit{f}}_{\mathit{max}\_\mathit{p}\mathit{e}\mathit{a}\mathit{k}}$ and difference of $\mathit{p}\mathit{h}\mathit{a}\mathit{s}\mathit{e}$

PTT is an essential component for calculating not only PWV but also BP. where PTT was estimated using the phase difference calculated by Eq. (8) and ${\mathit{f}}_{\mathit{m}\mathit{a}\mathit{x}\_\mathit{p}\mathit{e}\mathit{a}\mathit{k}}$ detected by ${roi}_1$. PTT can be expressed as follows.

$$PTT=\{\begin{array}{c}\frac{\left(2\pi -{\theta }_d\right)}{2\pi }\times \frac{1}{f_{\mathit{max}\_peak}}if{\theta }_d>\pi \\ \frac{{\theta }_d}{2\pi }\times \frac{1}{f_{\mathit{max}\_peak}}otherwise\end{array}\text{.}$$ (8)

3.4.3. Distance calculation between the center pixels of ${\text{roi}}_1$ and ${\text{roi}}_2$

As shown in Fig. 11, prior to calculating the distance between the center pixcels of ${\text{roi}}_1$ and ${\text{roi}}_2$ for estimating PWV, a dataset of the relationship between the distance from the camera to the user, divided into 5 cm intervals from 25 cm to 50 cm, and the width of the detected face was collected to estimate the distance between the camera and the user.

Fig. 11.

Relationship dataset collection between the distance of camera-user and the detected face width.

To establish a relationship formula, an experiment was conducted while converting the distance between the camera and user to 25–50 cm in Fig. 12(a). The results are illustrated in Fig. 12(b), and the regression curve was extracted from a dataset using Eq. (9), as follows.

$$D_{camera-user}=a_{face}\times {width}_{face}^2+b_{face}\times {width}_{face}+c_{face}\text{,}$$ (9)

where $D_{camera-user}$ denotes the distance between the camera and user, ${width}_{face}$ is the $width$ of the detected face $a_{face}$ and $b_{face}$ represent each coefficient for the regression curve, and $c_{face}$ is a constant. where $a_{face}$, $b_{face}$, and $c_{face}$ were calculated by regression analysis based on the collected dataset (Fig. 12(a)). $D_{camera-user}$ was estimated by applying the detected face width to the regression curve formula extracted from Fig. 12(a). As illustrated in Fig. 12(b), the actual $D_{onepixel}$ can be calculated using the estimated $D_{camera-user}$, expressed by Eq. (10), as follows. Based on this, the actual distance between ${\text{roi}}_1$ and ${\text{roi}}_2$ can be predicted using $D_{one-pixel}$.

$$D_{onepixcel}=a_D\times D_{camera-user}^2+b_D\times D_{camera-user}+c_D\text{,}$$ (10)

where $D_{onepixel}$ is the actual distance of one pixel according to the distance between the camera and user, $a_D$, $b_D$ are coefficients of the regression curve, and $c_D$ is a constant. where $a_D$, $b_D$, and $c_D$ were calculated by regression analysis based on the collected dataset (Fig. 12(b)). As shown in Fig. 13, the actual distance from ${\text{roi}}_1$ and ${\text{roi}}_2$ can be calculated using the distance of one pixel, as follows Eq. (11).

$$D_{centerof{roi}_{1}and{roi}_2}=\sqrt{x_{distance}^2+y_{distance}^2}\text{.}$$ (11)

Fig. 12.

Regression Analysis of Face width and Pixcel distance.

Fig. 13.

Distance between ${\text{roi}}_1$ and ${\text{roi}}_2$

3.4.4. PWV calculation

PWV is a useful indicator of arterial stiffness. It varies with age, because the arteries are elastic in young people, PWV is relatively low in young people and increases with age. PWV estimation methods are classified according to the measurement site. where the brachial-ankle PWV (baPWV), which is relatively easy to record, was adopted. The baPWV attaches a pulse wave extraction device to the ankle and upper arm, as illustrated in Fig. 14, and calculates the velocity using the pulse wave velocity difference value and distance between the two different ROIs (see Fig. 15).where PWV was estimated using PTT and the distance calculated from ${\text{roi}}_1$ and ${\text{roi}}_2$, as shown in Eq. (12). The transmission speed of the pulse wave was calculated using the calculated PTT and distance.

$$PWV=\frac{D_{centerof{roi}_{1}and{roi}_2}}{PTT}\text{.}$$ (12)

Fig. 14.

baPWV measurement.

Fig. 15.

PTT calculation using Pulse wave of ${roi}_1$ and ${roi}_2$

3.4.5. Robust blood pressure measurement using feature MRA

In this study, equations for BP measurement were extracted by applying a feature dataset collected for MRA to measure SBP and DBP using features estimated with the Cg signals calculated in ${\text{roi}}_1$ and ${\text{roi}}_2$. The SBP and DBP were calculated as follows Eq. (13).

$$Y_{SBP\left(orDBP\right)}=\alpha {+\omega }_1\times PWV{+\omega }_2\times PTT{+\omega }_3\times {Peak}_{\mathit{max}}$$

$${+\omega }_4\times f_{\max \_\text{peak}}{+\omega }_5\times W{+\omega }_6\times H\text{,}$$ (13)

where $Y_{BP}$ denotes a dependent variable, indicating SBP or DBP, ${\omega }_1–{\omega }_6$ represent the coefficient for each independent variable, $PWV$ is a pulse wave velocity calculated in ${roi}_1$ and ${roi}_2$, PTT is the pulse transit time (s), ${peak}_{\mathit{max}}$ is maximum peak of pulse-related frequency domain, $f_{\max \_peak}$ is a frequency(Hz) corresponding to ${peak}_{\mathit{max}}$, and W and H are the user body weight (kg) and height (cm), respectively.

3.5. Blood pressure measurement method using pulse wave

As illustrated in Fig. 16, PTT was calculated using the peak of ${roi}_1$ and the peak of ${roi}_2$ adjacent interval extracted from two different ROIs, which are the most commonly used cases in existing studies. Further, BP was estimated using PTT, which is expressed as follows Eq. (5).

$${ptt}_i=\left|t\left(i_{th}{peak}_{{roi}_1}\right)-t\left(j_{th}{peak}_{{roi}_2}\right)\right|,iandj=1,2,\dots ,n\text{,}$$ (14)

where ${ptt}_i$ denotes the $i$ th ${peak}_{{roi}_1}$ of ${roi}_1$ and the $j$ th ${peak}_{{roi}_2}$ of ${roi}_2$ adjacent interval, |∙| represents the absolute operator, and $t$ is a time function and time for the $i$ th or $j$ th peak. The unit of ${ptt}_i$ is s. The $i$ and $j$ may be the same, or different values can be substituted. The average of ${ptt}_i$ ($\overline{PTT}$) for the pulse wave calculated by ${roi}_1$ and ${roi}_2$ is expressed as follows.

$$\overline{PTT}=\frac{1}{n}\sum _{i=1}^n{ptt}_i\text{.}$$ (15)

where BP was measured using $\overline{PTT}$ (calculated from Eq. (15)) and the user body weight (kg) and height (cm), as follows Eq. (16).

$$Y_{SBP\left(orDBP\right)}=\alpha +{\omega }_1\times \overline{PTT}+{\omega }_2\times W+{\omega }_3\times H\text{.}$$ (16)

Fig. 16.

Multiple roi.

3.6. Weighted average blood pressure for $i$ th of ROI

Depending on the facial area considered in the image, the illumination change may be applied differently to the left and right sides. More light may be applied to the left or right side around the center of the face. Even if the color difference component is employed, removing all noise from the illumination change is challenging, therefore, the BP calculated for both cheeks may differ depending on the amount of illumination. As illustrated in Fig. 16, SBP and DBP were measured using a weighted average of ${ROI}_1$ and ${ROI}_2$, as follows Eq. (17).

$${BP\left(SBPorDBP\right)}_{wa}={\omega }_1\times {BP}_{{ROI}_1}+{\omega }_2\times {BP}_{{ROI}_2}+\cdots +{\omega }_n\times {BP}_{{ROI}_n}\text{.}$$ (17)

where a weighted average of 0.5 was applied to the BP calculated in ${ROI}_1$ (${roi}_1$-${roi}_2$) and ${ROI}_2$ (${roi}_1$-${roi}_2$) of both cheeks and the calculated BP.

4. Experimental results

4.1. Recorded dataset

To evaluate the SBP and DBP measurement performance of the proposed method, the BP was measured using a BP meter utilized in homes and medical facilities, and facial videos were collected. Measurement using the BP meter required wearing the cuff and took approximately 30s, simultaneously, a facial video of 30s was taken. Prior to the recording and BP measurement, facial videos and BP data of the subjects used in this study were obtained from approved data by an accredited Institutional Review Board (IRB). All subjects provided written informed consent for the use of their video and BP data for research purposes before conducting the experiment.

4.1.1. Internal dataset

For the BP measurement experiment using facial video in the indoor environment, a total of 400 facial videos 30-s long were generated for 20 subjects (15 males and five females) using the front camera of a smart device (Galaxy 10+). Additionally, SBP and DBP measured for 30 s using the cuff of the BP meter were collected. Facial video and BP measurements were performed simultaneously, and the video resolution of the smart device was set to 640 × 480 and 30 fps. The dataset collection process for the indoor environment is illustrated in Fig. 17.

Fig. 17.

Dataset collection in an indoor environment.

4.1.2. External dataset

For the BP measurement experiment using facial video in the external environment, a total of 120 facial videos, 30-s long, were generated for 6 subjects (5 males and 1 female) using the front camera of a smart device (Galaxy 10+), additionally, SBP and DBP measured for 30 s using the cuff of the BP meter were collected. The dataset collection process for the external environment is illustrated in Fig. 18, Fig. 19. In addition, for outdoor environments such as cars, drones, and benches, the cuff was elevated to the level of the heart using an height adjustable desk and a sitting desk to maintain a stable position, and facial video recording and blood pressure measurement were conducted.

Fig. 18.

Dataset collection in a driving car.

Fig. 19.

Dataset collection in a flying drone.

Cg signals calculated from external environments were extracted using a camera mounted on a driving car or flying drone, which rendered difficulty in measuring BP with high accuracy as the face may not be detected owing to illumination changes or body or object shaking.

The Cg signal was corrected by applying interpolation to measure the robust blood pressure in diverse environments. As shown in Fig. 20, a cubic spline correction was utilized.

Fig. 20.

Interpolation process of a Cg signal including nondetection.

The cubic spline interpolation is an algorithm that connects two broken points. Note that the curve connecting the two points is a cubic polynomial, thus, a cubic spline was applied to the undetected area, the functional values of the two curves at each point must be the same, and the derivatives of the two curves at each point must also be the same. The undetected part was connected and the BP was measured using the interpolated Cg signal, as illustrated in Fig. 20.

4.1.3. Histogram of blood pressure

Fig. 21 shows the distribution of the total 520 BP datasets collected in indoor (400) and outdoor (120) environments. The systolic blood pressure (Fig. 21(a)) ranged from a minimum of 110 to a maximum of 170, while the diastolic blood pressure (Fig. 21(b)) ranged from a minimum of 56 to a maximum of 90.

Fig. 21.

Histogram of SBP and DBP

4.2. Performance evaluation

The performance of the proposed method was evaluated by calculating the mean absolute error (MAE), root mean square error (RMSE), and mean absolute percentage error (MAPE) using the BP error ($y_i-\widehat{y_i}$), which is the difference between $y_i$ (systolic and diastolic BPs measured by BP meter) and $\widehat{y_i}$ (systolic and diastolic BPs measured by facial video) for the collected dataset. This is expressed as follows Eq. (18).

$$MAE=\frac{1}{n}\sum _{i=1}^n\left|y_i-{\hat{y}}_i\right|,MAPE(\%)=\frac{1}{n}\sum _{i=1}^n\frac{\left|y_i-{\hat{y}}_i\right|}{y_i}\text{,}$$ (18)

where |∙| is the absolute operator for calculating the MAE and MAPE, and $n$ represents the total number of BPs calculated from the th facial video. Nota that low MAE, RMSE, and MAPE values indicate that the SBP and DBP calculated in the facial videos are close to the BP measured by the BP meter.

4.2.1. Performance evaluation for stable and unstable ROI

Fig. 22 shows the results of systolic/diastolic BP comparison measured using the Cg signal calculated in a stable ROI (Fig. 22(a) systolic BP using stable ROI, Fig. 22(b) diastolic BP using stable ROI), and an unstable ROI (Fig. 22 (c) systolic BP using unstable ROI, Fig. 22 (d) diastolic BP using unstable ROI) to measure robust BP in diverse environments.

Fig. 22.

Comparison of BP calculation results by stable and original ROI.

To evaluate the performance of the proposed stable ROI detection method, a comparative experiment with BP measured in the original ROI was conducted, noise (such as illumination changes and unstable ROI detection) was removed, and systolic/binary BP was measured with a low error, as shown in Table 1.

Table 1.

Comparative results of SBP and DBP on stable and original ROI.

Methods		Analysis of difference conditions
		MAE	MAPE (%)
ROI Stabilization	Systolic BP	4.21	3.13
	Diastolic BP	3.61	4.89
Original	Systolic BP	4.70	3.50
	Diastolic BP	4.08	5.53

BP was calculated using the Cg signal calculated in the original ROI detection and ROI stabilization conditions using the collected internal dataset. The experimental results demonstrated that systolic BP before and after ROI stabilization exhibited a MAPE of 3.50 % and 3.13 %, respectively, corresponding to an improvement by 0.37 %. In addition, the diastolic BP before and after ROI stabilization exhibited a MAPE of 5.53 % and 4.89 %, respectively, corresponding to an improvement by 0.64 %.

Fig. 23 provides the mean error (ME), standard deviation error (SDE), and Bland-Altman analysis between BP measured by a cuff-based device and the FBP estimated using the Stable ROI (Fig. 23(a)) and Original ROI (Fig. 25(b)) for all subjects in indoor and outdoor environments. Fig. 23(a) indicates the approximation of the proposed method to the reference BP measured by the cuff-based device.

Fig. 23.

(a) and (b) Bland-Altman plot BP estimation. (a) from Stable ROI. (b) from Original ROI.

Fig. 25.

(a) and (b) Bland-Altman plot BP. (a) from PWV/PTT. (b) from only PTT on internal dataset.

4.2.2. Performance evaluation for internal dataset

Fig. 24 illustrates the results of a performance comparison experiment between the proposed BP measurement method (Fig. 24(a) systolic BP using PWV and PTT (Proposed), Fig. 24(b) diastolic BP using PWV and PTT (Proposed)) for the collected indoor dataset, and the BP measurement method using PTT (Fig. 24(c) systolic BP using PTT, Fig. 24(d) diastolic BP using PTT). The scatter diagram and correlation coefficient of systolic/diastolic BP calculated using facial videos and the BP meter are presented.where BP in both cheeks (${ROI}_1$, ${ROI}_2$) was calculated, as illustrated in Fig. 16, and a weighted average method was applied to reduce the noise of illumination changes. As presented in Table 2, an improvement in BP was confirmed when the weighted average was applied. A comparison of the results of the PTT-based and proposed methods revealed that the MAPE of systolic BP was 2.96 % (proposed) and 3.52 % (using PTT), whereas that of diastolic BP was 4.72 % (proposed) and 5.26 % (using PTT).

Fig. 24.

Comparison of BP measurement results using PWV and PTT.

Table 2.

Comparative results of BP on internal dataset.

Methods			Analysis of difference conditions
			MAE	MAPE (%)
Weighted Average	PWV/PTT-based (Proposed)	Systolic BP	4.01	2.96
		Diastolic BP	3.53	4.72
	PTT-based	Systolic BP	4.77	3.52
		Diastolic BP	3.81	5.26

Fig. 25 provides the ME, SDE, and Bland-Altman analysis between BP measured by a cuff-based device and the FBP estimated using the PWV/PTT based method (Fig. 25(a)) and only PTT based method (Fig. 25(b)) for all subjects in indoor environments. Fig. 25(a) indicates the approximation of the proposed PWV/PTT based method to the reference BP measured by the cuff-based device.

4.2.3. Performance evaluation for external dataset

As illustrated in Fig. 19, Fig. 20, measure BP with high accuracy in flying drones and driving vehicles is challenging owing to illumination changes and body tremors. In this study, to evaluate the performance of the proposed method in diverse environments, an experiment was conducted by applying it to a dataset of facial videos taken from outdoors, a moving car, and flying drone, and the results are presented in Table 3.

Table 3.

Comparative results of BP on external dataset.

External environment (Weighted average application)
Conditions	Metrics	PWV/PTT-based (Proposed)		PTT-based
		Systolic BP	Diastolic BP	Systolic BP	Diastolic BP
Outdoor	MAE	4.66	3.88	5.93	4.21
	MAPE (%)	3.48	5.23	4.58	6.33
Driving car	MAE	5.74	6.13	7.88	8.03
	MAPE (%)	5.17	7.12	8.72	9.26
Flying drone	MAE	5.93	6.86	8.99	9.18
	MAPE (%)	6.72	7.37	10.04	10.07

As presented in Table 3, the MAPE of systolic/diastolic BP for the proposed method and PTT-based method improved by 1.37 %/1.10 % outdoors (3.48 %/5.23 % (Proposed) and 4.85 %/6.33 % (using PTT)), 3.55 %/2.14 % in moving cars (5.17 %, 7.12 % (proposed) and 8.72 %/9.26 % (using PTT)), and 3.32 %/2.70 % in flying drones (6.72 %, 7.37 % (proposed), and 10.04 %/10.07 % (using PTT)). As indicated in the experimental results, the method proposed enables the measurement of systolic/diastolic BPs with a higher accuracy for external environments compared with for internal environments.

Fig. 26provides the ME, SDE, and Bland-Altman analysis between BP measured by a cuff-based device and the FBP estimated using the PWV/PTT based method (Fig. 26(a)) and only PTT based method (Fig. 26(b)) for all subjects in external environments. Fig. 26(a) plot indicates the approximation of the proposed PWV/PTT based method to the reference BP measured by the cuff-based device.

Fig. 26.

(a) and (b) Bland-Altman plot BP. (a) from PWV/PTT. (b) from only PTT on external dataset.

5. Discussion

PWV and PTT have been reported to be closely related to cardiovascular health, including BP (systolic/diastolic BP, and mean arterial pressure), based on numerous studies conducted in the medical/biomedical engineering fields. In previous PPG-based BP estimation studies, PWV estimated using two PPG devices was found to have higher correlation with blood pressure than PTT [37]. However, in existing non-contact skin imaging-based BP estimation methods, PTT was used to estimate BP as there were limitations in calculating the difference in distance from the heart to the two points even if different two regions of interest (ROIs), such as palm and face, were captured by the camera, as sFinallyhown in Fig. 14. In addition, in previous studies, the color signal extracted from the ROI was unstable due to shaking or frequent coordinate changes in face detection, which could cause errors when estimating PTT using the color signal.

In this study, ROIs were selected based on the area where arterial blood flows in facial images, and MAF was applied to address the aforementioned issues. While there are various methods, such as wavelet filtering, to remove high-frequency components other than MAF, considering them, our approach is likely to be further improved in the future. However, since the smart device environment is lighter than the PC environment, MAF was applied considering that the wavelet filtering method, which requires more computational resources than MAF, could reduce face detection rate during continuous calculation of facial detection and skin color data. And, PTT and PWV were estimated using the pulse signals generated from ${roi}_1$ and ${roi}_2$ and the actual distance between pixels from ${roi}_1$ and ${roi}_2$, and high performance was achieved when measuring blood pressure using the estimated feature values.

We observed the possible limitations of our method during the experiment. Our approach calculates the PTT from the pulse signal in the ROIs of the facial artery. However, changes in facial angle and long-distance facial videos decrease the accuracy of blood pressure (BP) measurement. Moreover, if we consider the possible limitations by addressing the issues, there is a high possibility of improving our approach in the future. Our research methodology focused on estimating blood pressure (BP) using PTT, PWV, and mean arterial pressure (MAP) estimated from facial videos, with consideration for minimal frame down using the camera of a smart device that can be used in various environments. Therefore, we did not include the part that addresses the jitter problem between video frames. However, we will conduct extensive experiments considering various problems that may arise from video images in our future approach, such as jitter.

6. Conclusions

In this study, we proposed a robust method for measuring systolic/diastolic BP in diverse environments using facial videos. The proposed method employed the ROI stabilization method and PWV calculation, which enabled high-performance BP measurement even in an external environment. Furthermore, the weighted average method, which considered the amount of illumination on both cheeks, was applied to further improve the BP measurement performance. The method can be used to measure BP anytime/anywhere using a smart device camera, and continuous BP monitoring is possible without physical contact with the device. In future studies, a lightweight deep learning-based model that can be mounted on a smart device will be generated and compared with the proposed method, and bio-signal analysis related to body temperature, stress, and depression disorder will be performed, in addition to BP.

CRediT authorship contribution statement

Jin-soo Park: Conceptualization, Conceptualization, Data curation, Formal analysis, Methodology, Software, Validation, Visualization. Kwang-Seok Hong: Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Supervision.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.