Cost-Effective Solution of Remote Photoplethysmography Capable of Real-Time, Multi-Subject Monitoring with Social Distancing

Abstract

Recent advances in remote-photoplethysmography (rPPG) have enabled the measurement of heart rate (HR), oxygen saturation (SpO₂), and blood pressure (BP) in a fully contactless manner. These techniques are increasingly applied clinically given a desire to minimize exposure to individuals with infectious symptoms. However, accurate rPPG estimation often leads to heavy loading in computation that either limits its real-time capacity or results in a costly setup. Additionally, acquiring rPPG while maintaining protective distance would require high resolution cameras to ensure adequate pixels coverage for the region of interest, increasing computational burden. Here, we propose a cost-effective platform capable of the real-time, continuous, multi-subject monitoring while maintaining social distancing. The platform is composed of a centralized computing unit and multiple low-cost wireless cameras. We demonstrate that the central computing unit is able to simultaneously handle continuous rPPG monitoring of five subjects with social distancing without compromising the frame rate and rPPG accuracy.

I. Introduction

The COVID-19 pandemic has placed a greater need on methods for contactless monitoring of patients in hospital settings with social distancing [1]. A key component of monitoring is assessing vital signs in a frequency that is dicatated by the disease process. This process is facilitated in-person by healthcare workers, but may place them at risk of exposure to disease [2]. Additionally, recent staffing shortages may result in a limited workforce that may limit the ability for in-person assessments. The application of a contactless system that unobtrusively measures vital signs is therefore an important advance in clinical medicine. By minimizing in-person contact with individuals, healthcare workers can prevent disease transmission and conserve personal protective equipment [3], [4]. Recent developments in computer vision have enabled measuring skin temperature, respiratory rate, heart rate, oxygen saturation, and blood pressure via infrared, RGB, and monochrome cameras [5]–[9]. Despite these advances, contactless vital sign monitoring is frequently infeasible at scale because these tasks rely on expensive cameras and powerful computing units for real-time image processing. Moreover, they often require the measured subject to be close to the camera (less than 1 m) to maximize the number of pixels covering the region of interest(ROI) [9]–[11]. This can relax camera resolution requirements, and consequently increase the possible video streaming frame-rate to improve algorithm performance for many of the aforementioned tasks [12]. These criteria limit either the rPPG’s real-time capacity or its deployment scalability for simultaneous multi-subject monitoring in a hospital setting to mitigate the spreading virus.

Here, we are proposing a cost-effective solution for real-time rPPG monitoring of multiple subjects with social distancing. This system is composed of several low-cost, off the shelf, wireless machine vision cameras sharing one powerful centralized computing unit. To overcome the reduced number of ROI pixels or video streaming frame rate when increasing the measuring distance shown in Fig. 1, we developed a closed-loop ROI mean algorithm that allows a camera to only capture the ROI and transmit the rPPG information to the computing unit which fundamentally removes the resolution-framerate tradeoff in the data transmission of high-resolution images.

Fig. 1.

Distance effect on the number of pixels covering the ROI as well as the mean absolute error (MAE) of rPPG by means of a camera with fixed resolution.

The proposed closed-loop ROI mean algorithm enables the use of a cost-effective WIFI-camera with a data transmission speed of 12 Mbs to achieve a similar rPPG performance as an expensive camera using USB 3.0 with a data transmission speed of 600 Mbs. Through multithreading of the central computing unit, simultaneous monitoring of multiple subjects can be easily implemented by increasing the number of low-cost wireless cameras.

II. Materials and Methods

We utilize the Nvidia Jetson AGX Xavier (Jetson) to implement the shared centralized computing unit, because it is a standalone device capable of simultaneous communication with multiple WIFI-cameras, and it provides computer vision acceleration hardware. This device is in charge of collecting rPPG information from multiple cameras, detecting and tracking the ROIs in a frame, sending the ROI location back to each individual camera, and calculating the vital signs of each individual subject. The OpenMV H7 Plus (OMV-Cam) is chosen as the low-cost, open-sourced, wireless machine vision camera. It has a maximum resolution of 2952 × 1944 (5 MP) with a color depth of RGB565. The device has an ARM Cortex M7 processor which allows it to perform simple computations.

A. Closed-Loop ROI Mean

The core of closed-loop ROI mean is using the Jetson to perform ROI detection and tracking to inform each OMV-Cam where the ROI is, then the OMV-Cam can only capture the pixels covering the ROI, compute the mean RGB values of the ROI pixels, and transmit the computed mean values back to the Jetson (Fig. 2). The location of the ROI would be updated every second to handle any body movement by taking a full-sized JEPG image with 80% quality factor. In the setup, we used face detection and facial landmark tracking to estimate the ROI (the forehead in this case) of subjects wearing a mask. The ROI size is 440 × 270 (118 k pixels) when the distance between the measured subject and the camera is 2 m and the camera is equipped with a 10-degree field of view lens.

Fig. 2.

Closed-loop ROI mean architecture of the multi-wireless cameras sharing one computing unit.

B. rPPG recordings

A FLIR Blackfly USB3 RGB camera with 12.3 MP resolution is served as the reference for evaluating the low-cost OpenMV camera with the same resolution. The resolution of the OMV-Cam and the FLIR Blackfly was set to 2000 × 1500 (1500p) and 1280 × 720 (720p), respectively. We recorded the rPPG from eleven subjects with social distancing in which the camera is placed 2 meters away from the measured subjects in an uncontrolled light environment. The age range of the subjects is between 20 and 30 years old with diversified skin tones. Two outliers, subjects 3 and 7, were excluded.

III. Results and Discussion

To develop a cost-effect solution for rPPG, we’ve investigated various criteria in the hardware required in terms of camera color depth, camera resolution, and rPPG streaming frame rate. We aim to find a proper balance between the cost and detection accuracy. To reduce the overall cost, we empower the rPPG performance in low-cost cameras by the proposed closed-loop ROI mean algorithm as well as enable the shared computation of multi-wireless machine vision cameras.

A. Color Depth Effect

State-of-the-art rPPG signals have been captured by a camera with RGB888 quantization (24 bits color depth). To verify if the OMV-Cam with RGB565 is qualified for rPPG measurement, we constructed a rPPG dataset with various color depth by lowering the color bit resolutions from the UBFC dataset. Table I shows the mean absolute error (MAE) of heart rate estimation with varying the color depth. Reducing the color depth from 24 bits to 16 bits and 80 bits only results in less than 10% and 20% increase of MAE, respectively. However, the MAE dramatically increases from 4.7 bpm to 8.8 bpm when the color depth is reduced from 24 bits to 10 bits. The results support that the OMVCam with RGB565 as a cost-effective camera is capable of rPPG analyses.

TABLE I.

Color depth effect on the rPPG accuracy.

Pixel resolution	RGB888	RGB565	RGB454	RGB343
# Bits	24	16	13	10
MAE [bpm]	4.7	5.1	5.8	8.8

B. Spatial and Temporal Resolution Effect

It is known that the higher spatial and temporal resolution the better for rPPG analyses. However, rPPG captured by a high-resolution camera increases not only the cost but also the computation loading, thus resulting in lowering the video streaming frame rate. Fig. 3 shows the MAE of heart rate estimation with varying the frame rate. Reducing the frame rate from 45 fps to 22 fps and 11 fps would increase the MAE from 4.5 bpm to 5.2 bpm and 6.3 bpm respectively. Fig. 4 also shows that reducing the camera resolution from 1500 p to 640 p would cause the total pixels numbers covering the ROI to reduce from 118 k to 8 k, thus resulting in the MAE from 4.7 bpm to 13.8 bpm. The increased MAE at lower camera resolution is caused by the pixel binning effect.

Fig. 3.

rPPG evaluation with varying the the temporal resolution.

Fig. 4.

The MAE of estimated heart rate versus camera resolution and number of pixels covering the ROI.

C. Closed-Loop ROI Mean

To fundamentally address the bottleneck in data transmission, we developed the closed-loop ROI mean technique that allows the camera to only capture the ROI pixels and pre-process the captured ROI pixels prior to transmitting the image to the central computing unit. In this way, only one-pixel size data that contains the mean RGB value of the ROI need to be transmitted. We evaluate the closed-loop ROI mean method performance on the OMV-Cam and compare it to the FLIR camera capable of USB 3.0 data transmission as shown in Fig. 5. The closed-loop ROI mean powered wireless camera shows competitive rPPG performance to the FLIR camera. It is worth mentioning that the average frame rate of the OMVCam via WIFI (12 Mbs) powered by the closed-loop ROI mean algorithm and the FLIR Blackfly via USB 3.0 cable (600 Mbs) was 25 and 45 fps, respectively.

Fig. 5.

Error analyses of the dataset recorded by means of the FLIR Blackfly USB3 RGB camera as well as the OpenMv wireless machine vision camera.

We also demonstrate that five closed-loop ROI mean powered OMV-Cams can be operated simultaneously via a single Jetson via multithreading without compromising the rPPG performance as shown in Fig. 6. All the devices can maintain the frame rate above 20 fps.

Fig. 6.

Multi-subject simultaneous rPPG monitoring.

IV. Conclusion

We developed a cost-effective solution of rPPG to enable simultaneous monitoring of multi-subject in real-time with social distancing via low-cost, open-sourced wireless cameras. We also investigated the effect of color depth as well as the spatial and temporal of a camera on the rPPG accuracy. To equip the low-cost wireless cameras with comparable rPPG performance to a costly FLIR Blackfly USB 3.0 camera, we develop a closed-loop ROI mean algorithm that allows a camera to only capture the region of interest and transmit pre-processed rPPG information to reduce the burden on the wireless data transmission as well as the imaging processing on the central computing unit. We demonstrate that the proposed cost-effective solution can enable at least five wireless cameras to simultaneously operate and be controlled by a single computing unit without compromising each individual performance. Taken together, this suggests that off the shelf camera systems as described could be scaled and leveraged to facilitate vital sign measurement in hospital systems.