Evaluating the robustness of a contact-less mHealth solution for personal and remote monitoring of blood oxygen saturation

Abstract

MHealth technologies play a fundamental role in epidemiological situations such as the ongoing outbreak of COVID-19 because they allow people to self-monitor their health status (e.g. vital parameters) at any time and place, without necessarily having to physically go to a medical clinic. Among vital parameters, special care should be given to monitor blood oxygen saturation (SpO₂), whose abnormal values are a warning sign for potential COVID-19 infection. SpO₂ is commonly measured through the pulse oximeter that requires skin contact and hence could be a potential way of spreading contagious infections. To overcome this problem, we have recently developed a contact-less mHealth solution that can measure blood oxygen saturation without any contact device but simply processing short facial videos acquired by any common mobile device equipped with a camera. Facial video frames are processed in real-time to extract the remote photoplethysmographic signal useful to estimate the SpO₂ value. Such a solution promises to be an easy-to-use tool for both personal and remote monitoring of SpO₂. However, the use of mobile devices in daily situations holds some challenges in comparison to the controlled laboratory scenarios. One main issue is the frequent change of perspective viewpoint due to head movements, which makes it more difficult to identify the face and measure SpO₂. The focus of this work is to assess the robustness of our mHealth solution to head movements. To this aim, we carry out a pilot study on the benchmark PURE dataset that takes into account different head movements during the measurement. Experimental results show that the SpO₂ values obtained by our solution are not only reliable, since they are comparable with those obtained with a pulse oximeter, but are also insensitive to head motion, thus allowing a natural interaction with the mobile acquisition device.

Introduction

The current COVID-19 (Coronavirus Disease 2019) pandemic has changed our daily lives, by imposing some strict behavioural uses, such as wearing mask, distancing, and in most severe conditions lockdown. At the time of writing this paper, many countries worldwide are still trying to limit the spread of the virus by requiring people to remain in self-isolation at home and limit human contacts as much as possible. On the other side, hospital facilities have been taken by storm by the huge inflow of Covid patients, also causing slowdowns in treatment of other diseases. Despite the arrival of the first vaccines against Corona virus, these precautionary measures will have to be observed for a long time to come.

In this context, mHealth, which is a new frontier of telehealth encompassing the use of mobile and wireless technologies to support digital health services, assumes crucial importance (Lupton 2013; Swan 2012; Ardito et al. 2020; Tripathi et al. 2020). Thanks to mHealth solutions, that allow people to self-monitor their vital parameters in any place and at any time through the use of mobile devices, patients can have their vital signs continuously monitored without having to physically go to a medical clinic, thus reducing the risk of virus transmission (Kim and Choi 2020). Moreover, the monitoring of vital signs undertaken by mobile devices allow immediate medical intervention when there is a deterioration in the health status, and reduces unnecessary hospital admissions, time and costs spent on transport, and health-related costs. Actually, the potential for using mHealth and in general telehealth solutions during emergencies has been emphasized in Lurie and Carr (2018) and even WHO (World Health Organization) has given specific highlight of telehealth as a powerful tool to curb the spread of the Corona virus and for early detection of the disease at the very onset. In particular, monitoring blood oxygen saturation (SpO₂) is of fundamental importance for early detection of COVID-19 since abnormal values of this vital parameter are a warning sign for potential infection even before patients start having short breath. Indeed, it has been observed1. that the SpO₂ values may fall below $90\%$ much more before the first symptoms of Covid pneumonia appear. SpO₂ monitoring can be easily done at home through the pulse oximeter that however requires skin contact and hence could be a potential way of spreading contagious infections.

To overcome this problem, in Casalino et al. (2020) we have recently developed a mHealth solution for contact-less estimation of blood oxygen saturation. Its basic idea is based on extracting the photoplethysmographic signal from facial video frames, hence it does not require any skin contact for measurement of the vital sign. The first experimental evidences reported in Casalino et al. (2020) have shown that our mHealth solution works well in controlled environments and provides several advantages. It is cheap, since it just requires a mobile device equipped with a camera (smartphone, laptop, tablet, ...), and does not need specific medical devices. It is contact-less, thus avoiding potential infections. It is easy-to-use, since the patient experiences the self-monitoring of vital parameters through a simple and natural action that can be part of his daily routine. Moreover, it can be easily integrated within a patient-centered telehealth platform for both personal and remote health monitoring. Finally, it is fast, since it provides reliable SpO₂ measurements in a couple of seconds. Although the proposed solution has been calibrated to perform the measurement of blood oxygen saturation aligned with a common pulse oximeter, there is still a lack of validation studies concerning its applications in real world scenarios. Indeed, measuring the SpO₂ using mobile devices in real-world daily situations holds some challenges in comparison to the controlled laboratory scenarios where the first experiments of our solution were conducted. One main issue is the frequent change of perspective viewpoint due to head movements, which leads to challenges in detecting the human’s face and computing an estimate of SpO₂. Hence, to better assess the validity of our solution for in-the-wild scenarios, it is fundamental to assess its robustness to head movements. This is the goal of the present work. To understand to what extent our mHealth solution is robust to head movements, in this paper we carry out a pilot study on the PURE benchmark dataset that takes into account different head movements during the SpO₂ measurement. Experimental results have shown that measurements obtained by our contact-less solution are reliable since they are comparable with those obtained using a commercial pulse oximeter, and are insensitive to head movements.

The paper is organized as follows. In Sect. 2 we overview some related works.

Section 3 describes the problem of SpO₂ measurement via photopletismography. Section 4 summarises the proposed solution. In Sect. 5 we discuss experimental results on a benchmark dataset to evaluate how head motion affects the SpO₂ measurements. Conclusions and future works are outlined in Sect. 6.

Related works

Several mHealth solutions have been recently proposed that allow patients to self-monitor their health status and stay remotely in touch with physicians, thus avoiding them to go physically to clinical centers. Most of these solutions rely on the well established technology known as remote photopletismography (rPPG) that has been largely used to develop contact-less solutions for monitoring vital parameters including SpO₂. In the pioneering work by Verkruysse et al. (2008), a digital camera in ambient light is used to estimate heart rate and breath rate values based on rPPG. Poh et al. (2010) proposed the use of the blind source separation of colour channels into independent components to process facial videos for heart rate estimation. Hassan et al. (2017) and Rouast et al. (2018) present an overview of rPPG techniques based on images of faces for heart rate estimation. Since vital parameters must be correctly estimated by means of trustable devices, several works studied the limits of these approaches, due to their sensitivity to external factors, such as light in Yang et al. (2021) and Song et al. (2021), variations in environmental conditions and subject demographics in Dasari et al. (2021), presence of face masks in Speth et al. (2021), and motion in Liu et al. (2021). However, none of these works studied the influence of those factors on the measurement of blood oxygen saturation. Only Allado et al. (2021) have recently designed a clinical trial to evaluate the effectiveness of rPPG-based measurements of blood oxygen saturation (as well as heart rate and respiratory rate) compared to those obtained with contact devices.

First attempts to use PPG-based measurements employed specialized and expensive devices that are difficult to retrieve and use, thus largely excluding a portion of the potential target users. More recently, low cost devices or common devices such as smartphones have been employed for large-scale adoption in daily life. Many free or low-cost software applications specifically designed for self-measuring of vital parameters through smartphones have been recently developed. For example Tayfur and Afacan (2019) use built-in sensor and light source of a Samsung Galaxy S8 smartphone to measure heart rate and SpO₂. Chan et al. (2019) evaluates the reliability of a smartphone oximeter in measuring heart rate and SpO₂ during rehabilitation therapies. Tomlinson et al. (2018) developed a camera-based application and probe-based application using Apple iPhone 5 and iPhone 6 to measure heart rate and SpO₂ of healthy children and the triage values of pediatric patients. In Sahoo et al. (2017), a mobile app is used to track patients’ heart activities and to send a warning if anomalous values are detected. In Storz et al. (2021) a mobile app has been designed to support individual monitoring of vital parameters of outpatients.

Further examples on the use of mobile applications for chronic disease monitoring have been systematically reviewed in Moses et al. (2021) and El-Rashidy et al. (2021). Mhealth solutions are widely used for monitoring patients affected by different pathological conditions, such as patients undergoing haemodialysis treatment in Villarroel et al. (2020), patients under home oxygen therapy in Naranjo-Rojas et al. (2021), patients affected by bipolar disorder in Casalino et al. (2021). Currently, mobile solutions have been shown to be also very useful during COVID-19 pandemic. A wide overview, discussing the use of mobile apps for Covid-19 is provided by Kondylakis et al. (2020). As an example El-Rashidy et al. (2020) propose a mobile GPS-based app is used for monitoring patients which are infected or suspected of COVID-19.

Besides smartphones, other cheap and ergonomic mobile devices have been proposed for personal and remote measurement of vital parameters. An example is the smart bracelet developed within the Telemachus (Telemedicine Monitoring and Collaborative Hub-and-Spoke System) project2. that allows to remotely monitor Covid patients in all phases of the disease by detecting the vital parameters of the wearer and communicating via geolocalization or via Bluetooth. Another device for data collection and remote monitoring of COVID-19 patients in home isolation is described in the cosinuss${}^{\circ }$ project3. It consists of a device in the wearable in-ear using photoplethysmography to monitor vital parameters, a data gateway and a cloud database server (Baldinger et al. 2021). In Sabbadini et al. (2021) heart rate and breathing rate are measured through the use of a face mask using the photoplethysmography technique to detect the presence of environmental and physical stress factors in workers.

Despite the current proliferation of personal mobile/wearable devices for vital parameter self-monitoring, the state-of-art literature shows that the most suitable sensor for contact-less measurement of SpO₂ is the video camera that satisfies both easy-of-use and contact-less requirements. The present work represents an advancement of the state of the art by introducing a general camera-based mhealth solution for the reliable measurement of blood oxygen saturation. Given the absence of studies on the influence of external factors on rPPG measurement of blood oxygen saturation, in this study we investigate the influence of head movements to better assess the robustness of our rPPG-based solution.

Problem definition

The problem we face in this work is to estimate the value of blood oxygen saturation through a contact-less solution exploiting remote photopletismography. According to Challoner (1979), the photoplethysmography (PPG) signal represents the variation of lights due to cardiovascular activities in some regions of the face tissue. These variations can be used to estimate the hearth rate and the blood oxygen saturation (Allen 2007). Specifically, according to Kong et al. (2013), an estimation of SpO₂ can be obtained by considering the cardiovascular pulse wave signal at two different wavelengths, namely the red and blue bands, being the blue band representative of the infrared wavelength used in the traditional pulse oximeter.

The underlying principle is that the impulse of cardio-vascular wave that flows through the body periodically, causes stretch in the vessel walls, with consequent fluctuations in blood volume. These fluctuations modulate the absorbency of light passing through a given volume of tissue, so it is possible to evaluate the variation of light during a normal cardiac cycle. There are two methods to implement PPG measurement of SpO₂. The first method is contact PPG, in which the user places his/her finger on the sensor and the flash light provides the light source required for the blood cells to become visible (Kim et al. 1986). The second method is contact-less or remote PPG, in which the camera is held in front of the user’s face and the measurement is undertaken without the need for skin contact. For the motivations described in section I, we chose to implement a contact-less PPG estimation of SpO₂. The following sections outlines the methodological pipeline underlying out contact-less solution.

The proposed mHealth solution

Our approach for contact-less measurement of blood oxygen saturation is based on processing a short video frame sequence that capture the face of the user in order to extract the photopletismographic signal and derive an estimate of SpO₂. The detailed description of the methods underlying our solution are described in Casalino et al. (2020). Here we briefly describe the main phases of the methodological pipeline (Fig. 1).

Fig. 1.

Methodological pipeline for contact-less SpO₂ measurement

In the first phase, a short video capturing the user’s face is acquired through the video camera and a sequence of video frames 2 seconds long is processed. Resolution of the video frames depend on the characteristics of the video camera integrated in the mobile device. Each frame is processed for face detection and three regions of interest (ROIs) are identified, specifically the two cheeks and the forehead, which represent the areas where the plethysmographic signal is most evident (Challoner 1979). We used three different regions in order to make the acquisition more robust to occlusion, such as glasses, beard and so on, as it has been proven in our previous work by Casalino et al. (2020). To cope with head movements, a fourth ROI is located in the middle of the face and used for face tracking. This further ROI is connected with the other three ROIs (cheeks and forehead), so that together they can indicate the movement of the face. This enables face tracking that makes the system less sensitive to movements and thus avoids the user having to stand still in front of the camera.

The second phase is devoted to extract the photopletismographic signal from the identified ROIs. Signal processing techniques are applied to the three different ROIs for noise reduction, and the most informative ROI is selected for SpO₂ measurement, as suggested in Solomon (1991). Particularly, for each of the three ROI, the RGB signals are extracted. Then for each of the three channels (red, blue and green) and for each frame in the video, the average value over all the pixels is computed. Thus a three dimensional signal is derived for each frame. High-frequency noise in signals are filtered (Speake and Mersereau (1981)) and the the chrominance method (De Haan and Jeanne (2013)) is used to limit the influence of motion and illumination variations on the signal. Finally linear interpolation is applied to uniform the signal.

After signal cleaning, red and blue components of the signal are used to compute SpO₂ value according to the following formula that was suggested by Kong et al. (2013):

$$\begin{array}{c}\hfill Sp{O}_2=A-B\frac{A{C}_{\mathit{RED}}/D{C}_{\mathit{RED}}}{A{C}_{\mathit{BLUE}}/D{C}_{\mathit{BLUE}}}\end{array}$$ 1

where the $A{C}_{\mathit{RED}}$ and $A{C}_{\mathit{BLUE}}$, $D{C}_{\mathit{RED}}$ and $D{C}_{\mathit{BLUE}}$, are respectively the standard deviations and the mean of the red and blue signals computed from the sequence frames. A and B are calibration parameters. We used $A=125$ and $B=26$ as in Kong et al. (2013).

Using the above described process, an estimation of blood oxygen saturation is obtained in real-time. Indeed, the whole process requires only a couple of seconds to obtain the SpO₂ value. The methodology was implemented as a web-based application, hence after acquiring the short video, all the processing steps run on a server, and web-services are used for communication. Being web-based, our solution can be employed on different mobile devices (smartphone, tablet, laptop, mobile robots,....), as long as they are equipped with a camera and are able to connect to internet.

The described pipeline was implemented as a prototype in Casalino et al. (2020). Preliminary results under controlled conditions of in-vitro experiments have shown that the methodology is reliable and comparable to a classical pulse oximeter under controlled conditions, i.e. when the users were required to stay still in front of the camera. However, to definitively assert the robustness of the method, we need to test it in more real scenarios. Actually, in real situations of use of our method on mobile devices, it is essential to determine if the measurement is susceptible to changes in case of head movements. The aim of this work is indeed to evaluate how robust the developed method is to the movements that the user may naturally make during the interaction with the mobile device. To this aim, we have conducted a pilot study to test the method on a dataset including facial videos of users characterized by different head movements. The following section shows the results of this study.

Experiments

The goal of this work is to examine the influence of artifacts introduced by head motion on the SpO₂ measurements made by our contact-less solution originally proposed in Casalino et al. (2020). To this aim we used the PURE dataset (Stricker et al. 2014). It is composed of videos of 10 subjects (8 males, 2 females) standing in front of a camera at a distance of about 1.1 meters and performing 6 different head motions, for a total of 60 videos. The videos were recorded with an RGB digital camera. Each video has a length of about 1 minute and was captured at a frame rate of 30 Hz with a cropped of resolution of 640 × 480 pixels. The dataset provides Ground Truth values of heart rate and SpO₂ that were measured with a pulse oximeter at the same time as the videos were recorded. The adopted pulse oximeter is a CMS50E with a sampling rate of 60 Hz. Natural lighting condition were used with frontal daylight on the subjects, and changing illumination conditions due to clouds. In all controlled setups the user was sitting in front of the camera. The following six different configurations are considered:

• Steady (S). This setup is used as baseline for measurements. The subject looked at the camera while avoiding head motion.

• Talking (T). The subject talked in front of the camera while avoiding head motion.

• Slow Translation (ST). The subject performed slow head movements parallel to the camera plane.

• Fast Translation (FT). The subject performed head movements parallel to the camera plane, twice faster than in the ST setup.

• Small Rotation (SR). The subject performed a predefined sequence of head rotation (angles about 20${}^{\circ }$).

• Medium Rotation (MR). The subject performed a predefined sequence of head rotation (angles about 35${}^{\circ }$).

Figure 2 shows frames of a single subject in the six different poses.4 We processed each video using our method by extracting a number of measurements that is proportional to the length of the video. We obtained an average number of 20 measures for each video. Due to the higher sampling rate of the pulse oximeter, there are more ground truth values than the values obtained by our method. To enable comparison between our measurements and the pulse oximeter measurements, for each video we calculated the mean value of the SpO₂ values computed by our system and the mean value of the corresponding SpO₂ ground truth values. We quantitatively evaluated the accordance between the two measurements by using Bland-Altman (B&A) analysis, Pearson Correlation, Mean Absolute Error (MAE), and Root Mean Square Error (RMSE). Figure 5 shows Bland-Altman graphs that visualise the differences between the two measurements and define the Level of Agreement (LoA). For reliable measurements the $95\%$ of the differences should be within the LoA, that are the dotted lines in the graph (lower and upper limits LoA). The central line represents the mean bias. Values lower than zero indicates that the proposed approach under-estimate the ground truth. On the contrary, values higher than zero suggest that the proposed approach returns values that are higher than the real ones. For a good agreement mean bias should be zero or close to it. We highlight that due to measurements errors related to the pulse oximeter, SpO₂ is considered valid if the mean bias is within $\pm 2\%$ (Chan et al. 2019). In detail, Fig. 5 shows the B&A graph with all mean values of the measures without considering the movement of the subjects. We can observe that there is agreement between the two measurements since all the points except two are within the LoA. In addition, we computed the B&A graph based on subject motion (Fig. 3). It can be seen that the movements slightly affect the agreement but there are no significant differences among the graphs of different movements. All the points are between the LoA, except for one measurement in settings S, ST and MR, that are still very close to the LoA bands. Figure 6 shows the average SpO₂ values obtained from ground truth and our system for each subject without accounting for motion. Overall, it can be seen that the values in the plot are in agreement with each other and there are no significant differences between the ground truth values and the values in our system. This is also supported by the total MAE value (1.332) in Table 1. In addition, Fig. 4 shows the average values of ground truth and our system, for each subject, in relation to motion. Again, there are no significant differences between the values. Table 1 shows that comparable results of MAE and RMSE are obtained for the six movements. Indeed if we consider the steady case as a baseline, low differences are observed in the other cases. Even with abrupt changes, as fast translation and medium rotation, we obtain low error values. Pearson analysis returns comparable results for the setups S, T, FT and MR. Surprisingly the worst results are obtained with ST and SR setups. We observed that these results depend on significant errors in rPPG acquisitions that were due to bad light conditions. In particular, three out ten subject videos had low lights in the Slow Translations setting. On the contrary, in the Fast Translation setting the scene was well lite, and consequently lower errors were observed. Thus we conclude that low illumination may affect the rPPG-based measurement. However, it is worth noting that the proposed methodology acquires the rPPG signal from a sequence of frames that together compose the whole movement. Thus, during real-time acquisition in complex setups such as ST or FT, the user will still be stationary in some frames, as in Steady setup. In fact, in Fig. 2, frames corresponding to ST and FT setups show the user’s head in front of the camera. Moreover, even though the setups were controlled, the users might perform slightly different movements. Leveraging these results, we can conclude that SpO₂ measurements obtained by the proposed contact-less solution approximate quite well measurements given by the pulse oximeter, and the movements do not affect significantly the estimate of SpO₂. This is also obtained thanks to the tracking module that allows the system to cope with head movements such as slow rotations. Indeed in case of head movements, the ROIs are still tracked by the system (as shown in Fig. 2), thus the rPPG signal is still correctly extracted from the ROIs.

Fig. 2.

Examples of video frames from the PURE Dataset with different setup: steady (a), talking (b), slow translation (c), fast translation (d), small rotation (e) and medium rotation (f)

Fig. 5.

Bland-Altman plot

Fig. 3.

Bland-Altman plots of measurements obtained in steady (a), talking (b) slow translation (c), fast translation (d), small rotation (e) and medium rotation (f) setup

Fig. 6.

Comparing the SpO₂ values measured by our system with the ground truth

Table 1.

Comparison between pulse oximeter and the proposed system

	Pearson	MAE	RMSE
Total	0.535	1.332	1.637
Steady	0.700	1.439	1.879
Talking	0.692	1.015	1.188
Slow translation	0.407	1.262	1.701
Fast translation	0.579	1.697	1.893
Small rotation	0.341	1.747	1.881
Medium rotation	0.691	0.834	1.063

Fig. 4.

Bland-Altman plots of measurements obtained in steady (a), talking (b) slow translation (c), fast translation (d), small rotation (e) and medium rotation (f) setup

Conclusion

In this paper a mHealth solution for personal and remote measurement of blood oxygen saturation has been presented. The key feature of our contact-less solution is the use of a video camera to acquire video frames of the face from which the plethysmographic signal is extracted and finally an estimate of the SpO₂ value is derived. The proposed solution allows a continuous personal control of the blood oxygen saturation in any place, avoiding the use of dedicated measuring devices (e.g. saturometers, pulse oximeters), but simply using a mobile device equipped with a camera. Since it is designed to be as portable as possible, our solution can run on different mobile devices of daily use, thanks to a web-based system, thus making the measurement easy, non-invasive and pervasive. The study in this work was mainly aimed to evaluate the robustness of the proposed approach to different motion setups. Experimental results showed no significant differences among the SpO₂ measurements in presence of different slight head movements. We believe the robustness of the method is achieved also thanks to the tracking mechanism that has been implemented as a main step of the methodological pipeline. On the overall, we can conclude that the proposed mHealth solution is robust enough to be used in real scenarios where the user should be free to move and not be forced to be still in front of the camera. This enhances the user experience, as well as the acceptability, leading the user needs to trust the technology and feel safe while using it. However, experimental results have shown a certain influence of the illumination conditions on the acquisition. Future works will be addressed to better study light conditions that could affect the real-time measurement in indoor environments. Moreover, the dataset considered in this work mainly encompasses videos of healthy people. We plan to test the robustness of the method on data coming from sick patients, who have critical values of SpO₂. Unfortunately such cases are difficult to collect at the moment, due to the pandemic situation, but we plan to collect more data in the future. With the growing use of mobile technology in the health field, we foresee that patients will increasingly use mobile devices to monitor daily their vital parameters or for personal triage assessment before going to the hospital. For this reason, we are currently extending our solution so as to enable contact-less measurement of other vital parameters besides SpO₂, such as temperature, heart rate, and breathing rate. Also, we are working to integrate it into a telehealth platform so as the acquired data is sent in real time to physicians who can review it from mobile through an app and from desktop through a web dashboard.