Real-time social distance alerting and contact tracing using image processing

Abstract

Social distancing has been enforced by many government and health organizations as an imperative measure to stop the spread of the ongoing COVID-19 crisis. However, current surveillance systems do not have the ability to detect breaches in social distancing nor real-time contact tracing. This chapter aims to provide a novel solution to this problem with the help of image processing techniques. The proposed system will generate structured data involving social distancing breaches and face mask detection based on surveillance footages. In the case of a person testing positive for the virus, it will also project the susceptible victims who have been in contact with the person through contact tracing, An installation of this kind will decrease the spread substantially and enable real-time contact tracing.

1. Introduction

One of the ways to control the spread of infection without the use of pharmaceutical methods is social distancing: which suggests that two individuals must not be in close vicinity of each other, maintaining at least 2 m (6 feet) distance between the individuals to curb the transfer of pathogens between them [1]. This method has shown to decrease the death rates because of the inability of the disease to disseminate. It has been previously practiced during previous outbreaks and pandemics such as the Spanish Flu during 1918 [2] and Ebola outbreak [3,4]. Both have proven to curb the spread of the infectious influenza by imposing strict lockdown measures such as closing schools, banning public gatherings, and the compulsory use of masks. It is especially effective for highly infectious diseases which are generally airborne and spread through cough or sneeze droplets, or diseases which spread through direct or indirect physical touch. COVID-19 has been proven to spread predominantly among people who are in close contact with each other [5], and countries have enforced social distancing measures such as the following to reduce the impact of the same:

• •Nationwide lockdowns were imposed in more than 190 countries [6] around the world where geographic quarantine enforcement was implemented, places of public gathering were shut down, and nonessential services were limited from operating.
• •Remote education [7,8] was promoted through video conferencing services such as Zoom, Google Meet, etc., and in-person interaction in schools, colleges, universities, and open areas was reduced by eliminating physical classes. Even graduation ceremonies of many schools were either postponed or happened virtually.
• •Companies across the world which have support for remote communication and work infrastructure have enforced employees to work from home [9].
• •Sporting events have been postponed; the London marathon which was supposed to be held on April 26, 2020 has been put away till October 4, 2020 [10]. The opening ceremony of the Tokyo Olympics which was to be held on July 24, 2020 has been postponed by a year [11]. The International Tennis Federation has also confirmed that 900 tournaments varied across all its circuits have been postponed [12].
• •People immigration was constricted in various modes of intracountry as well as intercountry transportation [13].
• •Proper hygiene and health protocols were installed in hospitals as well as public places. Markers and graphic cues have been placed at select places to highlight minimum separation [14,15].

As lockdowns across countries get eased, it is imperative to house quality surveillance systems which ensure whether minimum separation is being followed by individuals at places of interest. In India, artificial intelligence–equipped drones are being used to keep an eye on people who breach evening curfews [16]. Singapore has begun trials for robot dogs, developed by Boston dynamics, to patrol major parks and monitor the densities of public crowding [17]. Surveillance cameras in France [18] and thermal imaging technology in some parts of Britain [19] have also started to monitor social distancing compliance. These surveillance systems use machine-learning techniques to quickly identify and alert breaches.

This chapter first introduces the concept of flattening the curve which explains why social distancing works, then explores the currently available contact tracing methods, and proposes a surveillance system to identify susceptible members. The concept of identifying susceptible people in image frames essentially boils down to a classification problem where the object to be detected is a person. This task involves two steps, first is to identify the features and second is the choice of learning algorithm. The outcome of this is largely based on the efficacy of the feature selection technique. Deep learning techniques automate the feature selection process, thereby eliminating the errors that could rise out of hand-engineered pipelines. Deep learning methods which revolve around object detection largely use either of recurrent convolutional neureal networks (R-CNN) [[20], [21], [22]], single shot detection (SSD) [23], or you only look once (YOLO) [24]. R–CNN methods proposed by Girshick et al. [[20], [21], [22]] incorporates a two-stage approach where first the bounding boxes along with the region of interests are identified and then the regions are passed onto the classification phase. To reduce the latency, in YOLO and SSD, the two steps are combined into one. Thus, SSD and YOLO seem to be a good fit for real-time surveillance systems which require lower latencies and is considered for the implementation of the proposed system.

2. Flattening the curve

The idea of dampening the virus spread rate so that minimal number of people tends to seek treatment is referred to as “flattening the curve” [25]. Social distancing satisfies this objective to reduce the R₀, which is the rate at which the infection multiplies to other people to a value lesser than one or in other words flatten the curve [26]. Social distancing can also help in increasing the doubling time $T_d$ which is the time frame in which the growth of the infected population doubles and is given by $T_d=t\frac{ln\left(2\right)}{ln\left(1+r/100\right)}$ where r is the growth rate [25]. In this chapter, we consider the susceptible, exposed, infected, and recovered (SEIR) model to demonstrate this.

2.1. Susceptible, exposed, infected, and recovered model

The SEIR model [27] which stands for susceptible, exposed, infected, and recovered, respectively, buckets the entire population into the following categories:

• •Susceptible: This bucket consists of people who have not yet contracted the virus neither have immunity to it.
• •Exposed: This bucket consists of people who have been exposed to the virus but are yet to show symptoms as the virus is in the incubation stage.
• •Infected: This bucket consists of people who are either mildly, severely, or critically ill because of the virus. At this stage, not every individual requires hospitalization or ICU attention unless belonging to the latter half of the symptoms. The symptoms may worsen from one category to another. This stage also shows the mortality rate of the virus.
• •Recovered: This bucket consists of people who have completely recovered from the said disease. It is assumed people belonging to this category do not relapse and fall into infected category again (Fig. 15.1 )

Figure 15.1.

Susceptible, exposed, infected, and recovered flow.

There are four constants in addition to SEIR which define the model: $\alpha$, $\beta$, $\sigma$, and $\gamma$. $\alpha$ is the inverse of the virus incubation period, for COVID-19 this is approximated to 5.2 days [28]. The probability that an infected person spreads the disease to a susceptible person is defined as $\beta$. $\sigma$ represents the incubation rate and $\gamma$ represents the recovery rate. $\gamma$ is defined as $1/D$ where $D$ is the average duration of the infection. These constants are related to the SEIR model through first order and second order differential equations with time as an independent variable [27] (Table 15.1 ).

Table 15.1.

Differential equations for SEIR.

Variable	Supporting differential equation for susceptible, exposed, infected, and recovered
S (suspected)	$\frac{dS}{dt}=-\beta ·S·I$
E (exposed)	$\frac{dE}{dt}=\beta ·S·I-\alpha ·E$
I (infected)	$\frac{dI}{dt}=\alpha ·E-\gamma ·I$
R (recovered)	$\frac{dR}{dt}=\gamma ·I$

The entire population is the sum of the constants S, E, I, and R: $N=S+E+I+R$. $R0$ which is the basic reproductive number of the virus is given by $R0=\beta /\gamma$ [27] (Fig. 15.2 ).

Figure 15.2.

Susceptible, exposed, infected, and recovered plot for α = 0.1995, β = 1.74987, γ = 0.4991 and ρ = 0.7.

To the above set of differential equations, we can include the element of social distancing through $\rho$ which is the population interaction factor, the greater the $\rho$, the more is the interaction between the population. Therefore, suspected calculation now becomes $\frac{dS}{dt}=-\rho ·\beta ·S·I$ and exposed becomes $\frac{dE}{dt}=\rho ·\beta ·S·I-\alpha ·E$ [30] (Table 15.2 ).

Table 15.2.

Top $R0$ values for Indian states for COVID-19 (as of April 24, 2020) [29].

State	Confirmed cases	Reproductive number $\mathit{R}0$
Maharashtra	6817	2.2
Gujarat	2815	3.33
Delhi	2514	1.27
Rajasthan	2061	1.97
Madhya Pradesh	1846	2.25
Tamil Nadu	1821	0.93
Uttar Pradesh	1778	2.2
Andhra Pradesh	1016	1.55
Telangana	983	1.61
West Bengal	571	2.31

To understand the effectiveness of social distancing in curbing the spread of infection, we plot the graph showing the infection levels at various interaction factors among the population. We can see as the rate of interaction among the population increases, we get a higher curve and as the rate of interaction reduces, we get a flatter curve. With the use of the same model, let us understand how early cessation or relaxation of a lockdown or social distancing measures can lead to a second curve which is often higher than the first curve. Initially interaction among the population is at 0.413 and then because of relaxation it has led to an interaction of 0.815 among the population (Figure 15.3, Figure 15.4 ).

Figure 15.3.

Effectiveness of social distancing for various interaction factors.

Figure 15.4.

Demonstration of double peak for early cessation.

With the help of a longer lockdown, we can reduce the population that can be susceptible to the virus over the course of time. This is known as “herd immunity” where the population is given immunity to a virus through indirect exposure. This ensures that once the social distancing measures are relaxed, the curve does not become high again [30].

3. Contact tracing

Contact tracing is a process used to track down the people who might have come in contact with an infectious virus such as COVID-19, these people are then kept in close observation to ensure that there is no further transmission of the infection. Contact tracing needs a good amount of technology and efficiency to be successful on a large scale. This process can be further broken as below [31]:

• •When a contact is confirmed to have been infected or tested positive for an infection, their contacts must be identified, their contacts could be from various points such as work, family, recreational spots, etc.
• •All the contacts who have been identified should be reached out to and informed about the infection and asked to be quarantined/isolated depending on nature of infection
• •Identified contacts must be reached out to often to understand if they are developing any symptoms associated with the infection.

The methods that are available for contact tracing are either by using mobile applications or by manual methods. Manual tracking process involves contact tracing through manual efforts by identifying all the close contacts of the infected person and informing them. This is an arduous and error prone process and is largely discouraged. Governments and private organizations have come up with mobile applications which once installed use Bluetooth low energy and help in tracking an infected person or inform a person if they have encountered an infected person; The Australian government launched Covidsafe for COVID-19 using Bluetooth to track individuals who have been in close contact with each other [32]. In India, the contact tracing app Aarogya Setu which is based on the same principle has been downloaded over 100 million times, and the government has made it mandatory for government and private sector employees to download it [33].

4. Proposed system for identification of susceptible members

The proposed system can be broadly described as an engine which extracts relevant information from a footage which is fed as an input into the system (Fig. 15.5 ).

Figure 15.5.

Proposed system architecture.

The input is a surveillance footage which is passed into the analysis engine. The analysis engine handles the following tasks:

• •The engine first identifies unique people in the frame using image processing.
• •The engine then assigns a score based on the social distancing breach and face mask detection.
• •The output of the engine is a graph which can be used for real-time contact tracing and identify susceptible members.

4.1. Identification of unique people in the frame

In this chapter, we consider YOLO [23] and SSD [24] cross trained with MobileNet [34] as our prediction networks for identifying people within a given frame.

4.1.1. SSD—single shot detection

SSD [24] is a convolutional neural network which is feed forward in nature. The architecture of an SSD model can be broken up into two main parts: a backbone network and an SSD head. Generally pretrained image classification models such as VGG-16 or ResNet34 can be exploited as these backbone networks because of their accuracy in classifying images and the reduction in training time. This network gives an image with the relevant features which now must be extracted for semantic meaning. The SSD heads are the layers which are responsible for having the input of an image of arbitrary size else we are restricted by the dimensions of the base network. These are also CONV layers which gradually reduces the size of the image volume. Every CONV layer which is present is attached to the prediction layer and this is what allows for the variations in the size of the input section which is being considered and thus reducing the need for resampling the feature maps. An ameliorated version of the multibox algorithm is used for the bounding box suggestions. For every class present in the dataset, we have ground-truth bounding boxes. Bounding boxes which have already been calculated based on the sizes and locations of the ground-truth ones are called priors because of the use of a prior probability distribution [24] (Fig. 15.6 ).

Figure 15.6.

Backbone network and single shot detection heads.

4.1.2. MobileNet single shot detection

MobileNet architecture was created to ensure that the network gives faster results with a smaller space complexity [34]. Standard convolution is done on the dimensions of the input, output channels and the feature map vector. Let us assume $S_f$ as the sizes of the input feature maps, $N_i$ as the number of input channels, $N_o$ as the number of output channels, and $S_k$ as the kernel size. The complexity of evaluating with this filter is $S_f^2\ast N_i\ast N_o\ast S_k^2$. A depth wise convolution filter has the complexity as $S_f^2\ast N_o\ast S_k^2$ since only a single convolution is mapped on each on every input. Then there are pointwise convolutions where the kernel size is 1 with the complexity being $S_f^2\ast N_i\ast N_o$. MobileNet architecture combines both depth wise and point wise convolution into depth wise separable convolution which has a reduction of complexity by $\frac{1}{N_i}+\frac{1}{S_k^2}$. Apart from the first layer of the architecture which is fully convolutional, everything else is on depth wise separable convolutions post which batch normalization and rectified linear unit (ReLU) is applied. It also accommodates a width multiplier which helps in reducing the number of channels and a resolution multiplier which reduces the input [34]. In this chapter, we considered a caffe implementation where SSD is used as the base model in MobileNet which is around 45 MB in size. This enables the model to reside on portable devices with stringent memory requirements (Figure 15.7, Figure 15.8 ).

Figure 15.7.

Types of convolutions.

Figure 15.8.

Application of Batch normalization and rectified linear unit.

4.1.3. YOLO—you only look once

YOLO is an extremely popular detection network for the identification of objects. There are three variants to this network: YOLO [23], YOLOv2 [35], and YOLOv3 [36]. The first version YOLO consisted of 24 CONV layers accompanied by two fully connected layers at the end. ImageNet classification dataset which houses 1000 classes and with a resolution of 224 × 224 is used to pretrain the first 20 CONV layers. The last four layers is coupled with two fully connected layers for detecting objects. The resolution is also increased to 448 × 448 to increase the granularity of object detection. But the first version of YOLO had a major problem in object detection if objects were closer to each other. This was improvised in YOLOv2 which used darknet-19 where batch normalization and anchor boxes were introduced. The ameliorated version was also able to take a higher resolution as input and was able to detect in-depth features. The network was also trained using multiple dimensions of the same class to ensure that varying sizes of the object can be detected. The YOLOv3 is improvised from darknet-53 architecture by stacking up additional 53 layers. This was improvised from the previous version by incorporating residual blocks and skip connections. This version also handles upsampling. Since there is an addition 53 layers, this caused the architecture to be slower however ensured better accuracy than its previous versions (Fig. 15.9 ).

Figure 15.9.

You only look once V1 architecture.

The output of the YOLO network is a prediction vector consisting of five normalized components (center_x, center_y, box_w, box_h, prediction_confidence) where center_x, center_y is the center point coordinates, box_w, box_h is the bounding box dimensions, and confidence is the product of the probability of an object present and intersection over union between predicted box and ground truth.

In this chapter, we use YOLOv3 over its predecessors considering its high accuracy. The model consumed a space of around 240 MB which is suitable systems without memory constraints (Fig. 15.10 ).

Figure 15.10.

Prediction vector normalization.

4.1.4. Implementation

The image processing module that we implemented in a frame can be broken up into three components:

• •Identification of unique members
• •Social distance breach identification
• •Identifying whether an individual is wearing a mask or not

The video is processed to first identify people in a frame using MobileNet-SSD and YOLOv3 which is pretrained for recognizing person as an object. The centroid of the individual identified is continuously tracked. To make sure that we assign an individual an ID to keep track of, we compare the difference in centroids of subsequent frames. In the unique person tracking implementation screenshot, we have identified two individuals as Person 0 and Person 3 (Person 1 and Person 2 existed at a previous point in time and have escaped the current frame). We define a threshold and if the centroid difference is below this threshold, the ID is retained. We then calibrate a specific distance as reference in the frame and examine whether there is a social distancing breach or not. People who have caused social distancing breach is continuously tracked. Thereafter we subject the frame to see whether the person is wearing a face mask or not using MobileNet-SSD and YOLOv3. The training set for detecting face mask was generated by web scraping 1400 faces which contain mask, and which do not contain masks. The face mask module is integrated with the person identification module to produce a unified result. We define the prediction accuracy $a$ of the system as $\frac{c}{t}$ where c is the total number of human faces correctly classified and t is the total number of human faces in the video. Once these steps are completed, we forward this information to a graph tracing algorithm which constructs a structured data and can be later used for querying. The dataset used to analyze this measure was generated by scraping 250 pedestrian videos from publicly available sources of 60 s duration each which contained a good mix of individuals with face masks and without face mask (Figure 15.11, Figure 15.12, Figure 15.13 ) (Table 15.3 ).

Figure 15.11.

Unique person tracking implementation.

Figure 15.12.

Face mask module implementation.

Figure 15.13.

Integration of person identification with face mask detection.

Table 15.3.

Accuracy of integrated person tracking and face mask detection system.

Model	Memory consumed by model	Accuracy on dataset: $a$
MobileNet-SSD	45 MB	94.33%
YOLOv3	240 MB	98.29%

4.2. Susceptibility score as a concept for contact tracing

Each person who is identified in a frame is given a susceptibility score which is a measure of their ability to contract the disease. Thereafter we establish a set of heuristic rules upon which increments happens to this score (Table 15.4 ):

Table 15.4.

Susceptibility score calculation.

Frame analysis	Addition to susceptibility score
Identification of a unique individual	+1
Individual is not wearing a mask	+1
For every unique social distance breach	+1 for both the individuals involved
Individual was initially wearing a mask but has removed it (and for every mask activity that follows this action)	+1
Individual was initially not wearing a mask but is now wearing a mask (and for every mask activity that follows this action)	+0

The susceptibility scores are maintained in a map where the key is the person identifier and the value being the susceptibility score. The people who breach social distancing are the people who have been in close contact with each other beyond a distance threshold. This information is maintained in the form of an adjacency list with the timestamp.

Let us understand this with the help of an example. Consider that we have identified four individuals in a frame: (A, B, C, D). This translates to Fig. 15.14 , where the susceptibility score is mentioned inside the parenthesis within the person node and the adjacency list tracks the people who have been in contact with each other (breach in social distancing). They are initially assigned a value of 1.

Figure 15.14.

Susceptibility score initialization at time t0.

Post that, the check for whether a person is wearing a mask or not happens. Let us consider that B is not wearing a mask and that A, C, and D are wearing masks at t0. The susceptibility score of B is incremented by a value of 1, but there is no increment for rest of the candidates (Fig. 15.15 ).

Figure 15.15.

No face mask increment of susceptibility score at time t0.

Now in the next frame, let us consider that A and B commit a social distance breach at t1. A has an increment of 1 to the susceptibility score and B has an increment of 1 to the susceptibility score. Notice that B has now been added to the adjacency list of A and A has been added to the adjacency list of B. The time stamp of the breach has also been added in Fig. 15.16 . The time stamp will not be updated until A and B continue to be in social branch.

Figure 15.16.

Social distance breach increment of susceptibility score at time t1.

Let us consider that even C and D commit a social distance breach to understand the change in values. So, we can see that D has been added to the adjacency list of C, and C has been added to the adjacency list of D. The values of C and D have been incremented as well in Fig. 15.17 since there was a social distance breach.

Figure 15.17.

Social distance breach increment to susceptibility score at time t2.

Now let us consider the case that A and B got separated and reunited at a time t3 in Fig. 15.18 . The susceptibility scores of A and B will not be updated since it is not a unique meet. However, the time stamp t3 will be updated in the adjacency list.

Figure 15.18.

Timestamp addition for new meet at time t3.

Let us now add two more candidates E and F at a time t4 in Fig. 15.19 . E is not wearing a mask and F is wearing a mask. By similar logic, E now has a score of 3 because E is not wearing a mask, and has two social distancing breaches (E, A) and (E, D). The adjacency list has been updated likewise.

Figure 15.19.

Addition of two new candidates at time t4.

Let us now consider the case of F removing the mask and E putting a mask on at a time t5 in Fig. 15.20 . The susceptibility score of F's has been updated to 2 since F has removed the mask and is now increasing the susceptibility. However, the score of E is not updated because he has already been exposed.

Figure 15.20.

Susceptibility score update based on face mask status at time t5.

Now let us discuss how contract tracing will be applied in Fig. 15.21 when we get to know F and A have been infected. Since the adjacency list has already been populated, we can exploit it to find out the most susceptible members. F has not been in contact with anyone. A has been in contact with B at time t1, and time t3. Therefore, B is susceptible. Now let since B has not been in contact with any other member, we shall stop probing further than B. Moving on to the next member, A has been in contact with E at time t4 and hence it is also susceptible. We now look at the contacts of E with time t4 and above. Since D fits the description, it is also susceptible. Therefore, the susceptible members are B, E, and D. The general rule of thumb here is we look at the events in the list and find out the susceptible members who have been in contact greater than or equal to the current time being probed. For real-time analysis, without contact tracing, we can the exploit the map which holds the susceptible values to understand who all are susceptible.

Figure 15.21.

Contact tracing for infection detected at time t6.

5. Conclusion

Social distancing is a pivotal measure to reduce the current infection spread and avoid overcrowding in hospitals. Thus, organizations need to install surveillance applications which can in real-time highlight the relevant social distancing breaches and produce the list of susceptible members. The application proposed in this chapter achieved a good amount of accuracy on surveillance footages with low-cost development. Future work can be focused on real-time face identification, where we already have the knowledge of the faces appearing in the video and the idea of blockchain based networks can be probed where time transactions pertaining to social distancing breaches can be stored.