iWash: A smartwatch handwashing quality assessment and reminder system with real-time feedback in the context of infectious disease

Abstract

Washing hands properly and frequently is the simplest and most cost-effective interventions to prevent the spread of infectious diseases. People are often ignorant about proper handwashing in different situations and do not know if they wash hands properly. Smartwatches are found to be effective for assessing the quality of handwashing. However, the existing smartwatch based systems are not comprehensive enough in terms of achieving accuracy as well as reminding people to handwash and providing feedback to the user about the quality of handwashing. On-device processing is often required to provide real-time feedback to the user, and so it is important to develop a system that runs efficiently on low-resource devices like smartwatches. However, none of the existing systems for handwashing quality assessment are optimized for on-device processing. We present iWash, a comprehensive system for quality assessment and context-aware reminders for handwashing with real-time feedback using smartwatches. iWash is a hybrid deep neural network based system that is optimized for on-device processing to ensure high accuracy with minimal processing time and battery usage. Additionally, it is a context-aware system that detects when the user is entering home using a Bluetooth beacon and provides reminders to wash hands. iWash also offers touch-free interaction between the user and the smartwatch that minimizes the risk of germ transmission. We collected a real-life dataset and conducted extensive evaluations to demonstrate the performance of iWash. Compared to existing handwashing quality assessment systems, we achieve around 12% higher accuracy for quality assessment, as well as we reduce the processing time and battery usage by around 37% and 10%, respectively.

1. Introduction

Maintaining hand hygiene is crucial for preventing the spread of infectious diseases, and it can help reduce mortality, morbidity and healthcare costs (Allegranzi & Pittet, 2009). The rise of the COVID-19 pandemic has shown how critical it is to wash hands frequently and properly to reduce the transmission of lethal viruses and germs. The pandemic has put the entire human race in an unprecedented crisis and cost many lives (Meter, 2020). Since no vaccine has been invented yet, experts have identified frequent handwashing and social distancing as the two most effective ways to reduce the spread of the virus. Since the beginning of this pandemic, scientists have discovered even more significant facts about handwashing. A recent study shows that a country's handwashing culture is a very good predictor of the magnitude of its spread of COVID-19 (Pogrebna & Kharlamov, 2020). However, adherence to proper handwashing is not as simple as it looks. According to guidelines by the World Health Organization (WHO) (WHO), proper handwashing consists of several steps, which ensures that every area of the hands is properly disinfected. However, the average persons as well as healthcare workers are often ignorant about proper handwashing in different situations (Tremblay et al., 2019). Moreover, since it is difficult to remember all the steps, people often miss one or several steps, and consequently compromise their protection against viruses and germs. A recent study shows that 95% of people worldwide do not wash hands correctly (Atlantic, 2020), while frequent and proper handwashing could save a million deaths a year (CDC, 2020). As such, a hand hygiene assessment system that reminds users to wash hands when necessary, assesses the quality of handwashing, and provides feedback in real-time is of utmost importance, particularly in the current context of COVID-19 pandemic.

Over the years, a number of hand hygiene assessment systems have been developed to assess the quality of handwashing events in terms of following the guidelines as well as to remind users. Most of these systems are based on monitoring and assessing by direct observations (Arias et al., 2016) (Tschudin-Sutter et al., 2015) (Szilágyi et al., 2013), or by a video camera placed inside the sink (Llorca et al., 2011) (Xia et al., 2015) (Hoey et al., 2010). There are several reasons why these approaches do not work for assessing handwashing quality in the context of infectious diseases. Direct human observation is expensive and would be highly impractical in the context of social distancing. On the other hand, capturing user movements via video camera have the potential to leak sensitive information. Moreover, these approaches are not ubiquitous, have installation costs, and do not work when the user is outside his home (e.g, travelling). To overcome these limitations, several smartwatch based automated systems have been developed for assessing the handwashing quality (Li et al., 2018) (Galluzzi et al., 2015) (Wang et al., 2020) as well as reminding the users (Mondol and Stankovic, 2015). These systems utilize the sensing capabilities of the smartwatches, are ubiquitous and very much suitable for the daily life settings given the growing popularity of the consumer-grade smartwatches.

The existing smartwatch based handwashing assessment systems have several limitations in the context of infectious disease. First, these systems focus on accuracy only, and do not provide any feedback to the user whether he/she washed hands properly or for enough duration. In different situations, such feedback could help taking immediate measures which, in turn, would be crucial to stop the further spread of the viruses and germs. For example, if a person entering home from outside does not wash hands properly, he/she could pass the germs to family and friends and infect them too. So, a real-time feedback provided to the user about handwashing could help stop further transmission of the viruses or germs. The existing systems offload the smartwatch sensor data to a cloud server and fetch the assessment results. These systems can be straightforwardly extended to provide feedback in real-time, however such systems are susceptible to network connectivity and bandwidth problems (Medium, 2018), particularly for the smartwatches that are ubiquitous. These problems can be addressed through on-device processing that do not require network connectivity to process data and provide feedback in real-time such as our solution.

People often forget to wash hands when necessary. For example, it is very important to wash hands after returning home, particularly in the context of infectious disease like COVID-19 (CDC). However, existing systems lack the ability to provide reminders in such situations. We need a comprehensive system that guarantees high accuracy of handwashing quality assessment, runs efficiently on a smartwatch to facilitate real-time feedback, and reminds the user to wash hands in different contexts. In addition to that, the system has to interact with the user in a touch-free manner in order to avoid further contamination. However, there exist several challenges towards developing such a system.

First, a proper handwashing involves a set of steps (WHO) where the same step can be performed differently by different people and even differently by the same person at different times. Consequently, there are confounding gestures in different steps. The diversity in the gestures for same step as well as confounding gestures in different steps make it difficult to identify which steps a user performs or misses in a handwashing event. Though this problem can be alleviated by using sensor data from both hands, wearing smartwatches on both wrists is neither convenient nor practical in free-living context. It is challenging to detect the steps of proper handwashing, particularly using a single wrist device.

Second, resources like processing capacity and battery life available in smartwatches are low. High accuracy solutions like neural network based models usually require a significant amount of resources. It is challenging to run such models on the smartwatches with low latency in order to provide real-time feedback.

Third, different smartwatches have different amounts of resources as well as different numbers of other applications that need to share the resources. In contrast to the solutions that require a fixed amount of resources, a solution that can trade-off between accuracy and resource requirements is more desirable because such an adaptive solution can provide room for other applications when needed as well as they can be used on devices with a wide range of resources.

Fourth, a touch-free interaction between the watch and the user is needed to avoid further contamination, particularly in the context of infectious diseases. The tiny display of a smartwatch is also not suitable in providing detailed assessment results to the users.

In this paper, we address each of the aforementioned challenges in a smartwatch based handwashing quality assessment system. We present iWash, a novel handwashing quality assessment system on a smartwatch that assesses whether the user washed hands properly in terms of WHO guidelines with high accuracy. It then provides the user with real-time feedback about the quality of the handwashing using voice, and also reminds the user to wash hands frequently and specifically in the situation when the user enters the home. The main contributions of this paper are:

• •Handwashing quality assessment system on a smartwatch with high accuracy: We present iWash, a comprehensive handwashing quality assessment and a real-time reminder system on a smartwatch. We employ a hybrid deep neural network (combination of Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN)) based method to detect the handwashing quality in terms of the standard guidelines and achieve around 12% accuracy improvement over the state-of-the-art systems.
• •Efficient smartwatch based system for providing immediate feedback: For the first time in smartwatch-based handwashing quality assessment solutions, we employ model compression technique to optimize the system in terms of the smartwatch constraints to run the system faster and with lower battery usage in addition to guaranteeing high accuracy. We further show that our system is able to trade-off between accuracy and the resource requirements, and thus offers an adaptive solution as per the device specification.
• •A context-aware reminder system: In the context of the infectious diseases, iWash provides routine reminders for washing hands and also for the situation when the user enters his home.
• •Touch-free interaction: iWash interacts with the users in a touch-free way, a critical requirement for a handwashing system to avoid further contamination from touching the devices. It intelligently uses voice interaction to overcome the small display challenge of the smartwatch and provides useful feedback to the user.
• •Real-life data collection: As there is no widely used publicly available dataset for handwashing quality assessment, we collected handwashing data from 14 participants in real-life setting and conducted extensive evaluations to demonstrate the performance superiority of iWash compared to the state-of-the-art systems.

The rest of the paper is organized as follows. Section 2 provides the background knowledge to better understand the problem. Section 3 provides the basic overview of iWash. Section 4 describes the details of each system component of iWash. Section 5 experimentally evaluates the claims made in this paper. Section 6 summarizes the relevant research works. Section 7, 8 conclude the paper with some pointers for possible future works.

2. Background

2.1. Handwashing guidelines by WHO

To maintain the hand hygiene properly, WHO suggests that one should follow either handrub using an alcohol-based formulation or handwash with soap and water (WHO). The alcohol-based handrub systems are the preferred means for routine decontamination of hands for all clinical environments, while the handwash with soap and water is recommended when hands are visibly soiled (WHO). Both procedure consists of several steps which ensures that every portion of the hand is properly disinfected. The details of these steps for both approaches are shown in Fig. 1 . While the soap or the alcohol applying step on the hands are different, we can see that the key steps for cleaning different portions of the hands (step 2 through until step 7) are the same. So, to cover both of the approaches, the existing handwashing quality assessments systems have focused on whether the steps 2 through 7 were properly performed by the user (Li et al., 2018) (Wang et al., 2020). Moreover, by carefully observing these steps, we can see that the steps 3, 5, 6, and 7 involves an action to be repeated by opposite hands. For example, the step 3 suggests that one should put the right palm over left dorsum with interlaced fingers and vice versa. It means the user should put the right palm over left dorsum with interlaced fingers and also put the left palm over right dorsum with interlaced fingers. So, for the convenience of evaluation, we break such steps into two steps and assign number for all the individual steps. The steps from the WHO handrubbing and handwashing guidelines and the corresponding step numbers in iWash are presented in Table 1 .

Fig. 1.

WHO guidelines for maintaining hand hygiene, Source: WHO. How to Handrub?/How to Handwash? (WHO).

Table 1.

Steps for performing alcohol-based handrub and handwash with soap and water.

WHO step no.	Alcohol-based handrub	Handwash with soap and water	iWash step no.
1	Apply sanitizer	Apply soap	N/A
2	Rub Palm	Rub Palm	1
3a	Rub Right Dorsum	Rub Right Dorsum	2
3b	Rub Left Dorsum	Rub Left Dorsum	3
4	Interlock fingers	Interlock fingers	4
5a	Twist Right Knuckles	Twist Right Knuckles	5
5b	Twist Left Knuckles	Twist Left Knuckles	6
6a	Rub Right Thumb	Rub Right Thumb	7
6b	Rub Left Thumb	Rub Left Thumb	8
7a	Scrub Right Fingertip	Scrub Right Fingertip	9
7b	Scrub Left Fingertip	Scrub Left Fingertip	10
8	N/A	Rinse hands	N/A
9	N/A	Dry hands with towel	N/A
10	N/A	Turn off faucet	N/A

2.2. What is a quality assessment system for handwashing

In most of the previous works, the adherence to the WHO handrub and handwash guidelines were measured (Li et al., 2018) (Galluzzi et al., 2015) (Wang et al., 2020) (Mondol and Stankovic, 2015) to assess the quality of hand hygiene. In other words, a handwashing or handrubbing event is assessed as a proper one if the user followed the steps in the guidelines properly. Additionally, in the context of infectious diseases, CDC(Centers for Disease Control and Prevention) recommends performing handwashing or handrubbing for a period of a minimum of 20 s (CDC). So, in this paper, by handwashing quality, we focus on the steps and the duration of the handwashing or handrubbing event performed by the user with respect to these guidelines. Moreover, for the sake of uniformity, we use the term handwashing throughout the paper, which covers both handwash and handrub approaches.

In a smartwatch based handwashing quality assessment system, the readings of the sensing modalities embedded in the smartwatch are used to identify the individual handwashing steps. In particular, when the user performs the handwashing steps, it involves movements of the wrists, which is captured by the IMU (Inetial Motion Unit) sensors of the smartwatch. These sensors readings are used to identify the corresponding handwashing steps.

3. System overview of iWash

In this section we discuss the system overview of iWash. Fig. 2 depicts the overall system. The user wears a smartwatch where the iWash application is previously installed. iWash includes a voice interactive reminder system that is used to remind the user through the smartwatch. iWash identifies the situation when the user enters home from outside and immediately provides a reminder to the user to wash hands. Also, the user is periodically reminded to wash hands in order to maintain the hand hygiene throughout the day. When the user starts to wash hands, iWash collects the accelerometer, gyroscope, and magnetometer sensor readings of the smartwatch throughout the handwashing event. The start and the end of the event are marked with the help of the voice interaction with the user. Following the handwashing event, several statistical features are extracted from the data. The features are fed to a model to identify the handwashing steps following the guidelines by WHO. To build the model, we first develop a model which identifies the handwashing steps with high accuracy, however the model requires heavy computation. Then, we employ a model compression technique to create an optimized version of the model for running on the low-resource smartwatch. While optimizing the model, we focus on minimizing the prediction latency to meet the requirements for providing immediate feedback a well as preserving the high accuracy to identify the handwashing steps. iWash then identifies the steps missed by the user for a proper handwashing, and immediately reports it to the user through a voice feedback. iWash also reports to the user if he/she did not wash hands for a minimum duration of 20 s. In this way, iWash assesses the quality of the handwashing performed by the user and provides immediate feedback to the user.

Fig. 2.

System overview of iWash.

4. System design of iWash

Based on discussion of the system overview in the previous section, the system design of iWash has the following five major components.

• •Features extraction from a handwashing event
• •A hybrid deep neural network architecture for high accuracy model development
• •A reinforcement learning based model optimization for a smartwatch
• •Identification of missing steps and feedback generation
• •Voice interaction and reminder module

We provide detailed description for each of the components as below:

4.1. Features extraction from handwashing event

During a handwashing event, data are collected from the accelerometer, gyroscope and magnetometer sensors of the smartwatch. The sensor data are collected at 50Hz sampling rate. Each of the sensors provides data signals along the x, y, and z axes. Some pre-processing is needed to remove the noisy artifacts from the signals. Specifically, we pass the raw signals through a FIR (Finite Impulse Response) filter to remove the high-frequency vibration noises similar to (Samyoun et al., 2020). We window all the pre-processed data with a window size of 0.06 s s, and with 70% overlap between subsequent windows. These values are picked according to the state-of-the-art paper (Li et al., 2018) which led to the best results for handwashing quality assessment in their work. Once the windows is generated, we extract several statistical features from each window, such as the mean, standard deviation, kurtosis and skew. Moreover, previous works have shown that the empirical cumulative distribution function representation (ECDF) of the sensor data is effective in preserving the statistical characteristics of the data (Hammerla et al., 2013). Consequently, in addition to the mean, standard deviation, kurtosis and skew, we also compute the ECDF feature from the sensor data along the three axes.

4.2. A hybrid deep neural network architecture for iWash

To develop a model that identifies handwashing steps with high accuracy, we develop a hybrid deep neural network as shown in Fig. 3 . The model is a parallel combination of CNN and RNN. CNN and RNN allow capturing the spatial and temporal correlation present in input sensor data, respectively. As a result, the combination of these two networks facilitate the identification of handwashing steps with high accuracy.

Fig. 3.

Hybrid deep neural network architecture for iWash.

After the sensor data are windowed from the previous step, the windowed data are provided to the hybrid deep neural network. The network identifies the handwashing step for each window. The inputs to the hybrid network are the statistical features for a window and the activation vector for the previous window data. Activation vector for a window is the concatenation of the CNN and RNN outputs for the window data. By considering the activation vector for the previous window data in addition to the statistical features of the current window data as input to the hybrid network, we allow the flow of information from the previous windows. A complete handwashing event consists of a sequence of handwashing steps, so identification of a handwashing step is facilitated if the previous handwashing steps are identified and kept in consideration. In essence, such flow of information from the previous windows facilitates the accurate identification of the subsequent steps.

CNN Architecture: The CNN consists of three convolutional layers. The filter size of the three convolutional layers are 3, 5, and 7, respectively. The number of filters in the three convolutional layers are 128, 256, and 512, respectively. The values of the filter size and number of filters of a convolutional layer are chosen based on evaluation on a validation set. The convolutional layers capture the correlation among data from multiple sensors across multiple axes (i.e., x, y, and z). The output of each convolutional layer passes through the Relu activation function (Agarap, 1803). Relu activation introduces non-linearity in the network which facilitates higher accuracy. The squeeze and excite block (Karim et al., 2019) after the convolutional layers help to capture contextual information present in input features. Finally, the average pooling layer reduces the number of parameters and contributes in increasing accuracy by introducing transitional invariance in the output of a convolutional layer (Goodfellow et al., 2016).

RNN Architecture: We use a LSTM (Long Term Short Memory) network as the representative of RNN due to its higher performance over the other RNN variants (Rathor, 2018). The LSTM network has 50 cells, which is chosen based on evaluation on a validation set. Before the LSTM network, the input feature are passed through a self attention layer (Karim et al., 2019). The self attention layer facilitates the architecture to learn which input features need to be provided higher attention (or weights). Overall, the RNN architecture captures the temporal relationship present among input features.

Concatenation of CNN and RNN Outputs: The outputs of the CNN and RNN are concatenated. As mentioned before, the concatenated vector (or activation vector) of the CNN and RNN outputs are incorporated with the input features of the next window. The concatenated vector is also passed through a fully connected network that outputs the handwashing step for the current window. The fully connected network has 3 layers, each having a hidden state vector size of 250.

4.3. A reinforcement learning based model optimization for smartwatch

Smartwatches have limited resources in terms of computation and energy. To run iWash efficiently on the smartwatch, we need to optimize the high accuracy model described in the previous section so that it runs with minimal prediction latency and battery resource usage. We also need to preserve the accuracy of the high accuracy model during this optimization. To overcome this challenge, we employ a model compression technique that compresses the high accuracy model.

Specifically, we adopt AMC (He et al., 2018) as this is a state-of-the-art work on model compression focusing on the optimization of both accuracy and prediction latency. As battery usage depends on the computation duration, optimizing for prediction latency also ensures optimization for battery usage.

In a deep learning model, the number of FLoating point OPerations (FLOPs) represents how many computations (multiplications and additions) are required for the model. Thus, the greater the number of FLOPs, the higher the prediction latency is. We need to set the maximum number of FLOPs allowed in the optimized model. Specifically, we consider a hyper-parameter $\alpha \in \left(\mathrm{0,1}\right)$ that represents the fraction of FLOPs of the high accuracy model allowed in the optimized model. Thus, if the number of FLOPs of the high accuracy model is N and we set $\alpha =0.5$, then the maximum number of FLOPs allowed in the optimized model is $0.5\text{∗}N$.

We leverage reinforcement learning to efficiently and automatically explore the large design space of deep neural network models with the goal of finding a model that is optimal in terms of both accuracy and prediction latency. Fig. 4 provides an overview of the reinforcement learning based model optimization procedure. At each iteration of the reinforcement learning, we compress the high accuracy model in a layer-by-layer manner by following an actor-critic algorithm. For each layer, we generate an embedding of the layer. The embedding depends on the structure of the layer, number of FLOPs of the layer, total number of FLOPs reduced in previous layers, and the number of remaining FLOPs in the subsequent layers. Based on the embedding, the actor network generates a sparsity ratio (a real value between 0 and 1) for the layer. The layer is compressed with the sparsity ratio. The higher the sparsity ratio, the more the layer is compressed. If the sparsity ratio is s, the layer will be compressed by $s\ast 100\text{\%}$. Then, the next layer is processed in the similar manner. After all the layer are compressed, the critic network evaluates the resultant model in terms of accuracy and number of FLOPs. During the evaluation, it is also checked whether the number of FLOPs is less than the maximum allowed. We run 100 such iterations. Finally, the reinforcement learning agent learns the optimal sparsity ratio for each layer of the high accuracy model. Each layer is compressed based on the corresponding sparsity ratio to generate the final optimized model.

Fig. 4.

Overview of model optimization in iWash.

Specifically, at each iteration of the reinforcement learning, we try to minimize the following loss function:

$$\mathcal{L}=L\left(\mathbf{y},O_C\left(\mathbf{x}|\mathbf{s}\right)\right)\text{∗log}\left(\#FLOPs\right)$$ (1)

where,

$$\mathbf{s}=O_A\left(\mathbf{E}\right)$$ (2)

In Equation (1), $\mathbf{y}$ and $\mathbf{x}$ represent the ground truth and input features, respectively. $\mathbf{s}$ denotes the sparsity ratio generated by actor network ($O_A$, Equation (2)) given embedding of the model layers (E, Equation (2)). L denotes the mean squared loss between ground truth ($\mathbf{y}$) and the output generated by critic network ($O_C$). $\#FLOP$ represents the number of FLOPs in the model after compressing all the layers according to the corresponding sparsity ratios. Each of the actor and critic networks has two hidden layers, each layer has hidden state vector size of 300.

Overall, in Equation (1), $\mathcal{L}$ is the multiplication of two terms. The first term finds the loss between ground truth $\mathbf{y}$ and the output generated by critic network $O_C$ where the critic network considers the action (sparsity ratio for each layer, $\mathbf{s}$) generated by the actor network. The second term is the logarithm of the number of FLOPs in the model after compressing each layer according to the corresponding sparsity ratio. Thus if we minimize Equation (1), we try to maximize accuracy and minimize number of FLOPs. We use Adam optimizer (Kingma and Ba, 2014) to minimize Equation (1). By the minimization of Equation (1), we find the optimal parameter values for the actor and critic networks. When both the actor and critic networks have optimal parameter values, the action (sparsity ratio for each layer) generated by the actor network is also optimal. The optimal sparsity ratios ensure the optimal compressed model.

As mentioned before, choosing the value of α is a design choice. With the increase of the value of this hyper-parameter, the prediction latency increases but the accuracy also increases as the model is allowed to be more complex. Due to such trade-off between accuracy and number of FLOPs which is controllable by tuning the value of α, iWash has the capability of being device adaptive. For low-resource devices, we can choose a lower value of α resulting in reduced number of FLOPs. This ensures that the model runs efficiently on the device with a little sacrifice of accuracy. On the other hand, for the devices having high available resources, setting a higher value of α involves higher number of FLOPs and results in high accuracy. The details of this trade-off and device adaptivity are discussed in Section 5.5.4 and Fig. 12 illustrates the trade-off. For our scenario, we consider $\alpha =$ 0.5 as it provides a satisfactory trade-off between accuracy and the number of FLOPs.

Fig. 12.

Trade-off between accuracy and number of FLOPs in iWash.

4.4. Identification of missing steps and feedback generation

As per the claim of this paper, iWash assesses the quality of handwashing in terms of the steps in the WHO guidelines and the duration of the handwashing. Using the model described above, iWash identifies the steps from a handwashing event performed by a user on the smartwatch. It then compares these steps to those in the WHO guidelines, and thus finds which step(s) has been missed by the user, if any. It then generates a feedback message for each of the missing step which contains a brief description about that step. Moreover, iWash calculates the duration of the handwashing event from the start and the end of the handwashing found from the voice interaction with the user. If the duration is less than 20 s, a feedback message is generated mentioning that the user did not wash hands for enough duration. However, if there are no missing steps and the duration is at least 20 s, a feedback message is generated mentioning that the user washed hands perfectly. Table 2 shows the list of the feedback messages provided by iWash to the user. If there are multiple feedback messages, the messages are combined into a single message at the end. The feedback is then provided to the user using voice at the end of the interaction with the user.

Table 2.

Feedback provided to user for the corresponding quality assessed by iWash.

Assessed quality	Feedback provided to user
Missed step 1	Didn't rub both hands palm to palm
Missed step 2	Didn't rub right palm over left dorsum properly
Missed step 3	Didn't rub left palm over right dorsum properly
Missed step 4	Didn't put palm to palm with fingers interlaced properly
Missed step 5	Didn't clean right fingertips interlocked in left palm properly
Missed step 6	Didn't clean left fingertips interlocked in right palm properly
Missed step 7	Didn't rub left thumb clasped in right palm properly
Missed step 8	Didn't rub right thumb clasped in left palm properly
Missed step 9	Didn't rotationally rub right fingers on left palm properly
Missed step 10	Didn't rotationally rub left fingers on right palm properly
Duration less than 20 s	Didn't wash hands for enough duration
No missed steps and duration at least 20 s	Great job! You washed your hands perfectly.

4.5. Reminder and interaction module

This module of iWash is used to remind the user for washing hands in different situations and interacting with the user throughout the handwashing event. The flow of a reminder in iWash is demonstrated in Fig. 5 . When the user wearing the watch approaches the door from the outside, he/she receives a reminder on the watch. The user is also reminded to wash hands if iWash haven't recorded any handwashing event during the last 1 h. If the user confirms starting handwashing during the interaction, iWash collects the data during the handwashing event and assesses the quality of the handwashing, which is immediately provided the to user in the form of a voice feedback. Moreover, during the interaction, if the user wants to wash hands later at a specific time, iWash reminds the user accordingly at the rescheduled time. The detailed functionalities of the reminder and interaction module are discussed below.

Fig. 5.

Flowchart of the reminder and interaction module in iWash.

4.5.1. Providing reminders when the user is entering home

iWash includes a low-energy, proximity Bluetooth beacon (Estimote) set at the door of a room. The proximity beacon is a small wireless sensor that can be attached to any object. It broadcasts tiny radio signals which can be sensed by a smartwatch. Each sensed beacon signal on the smartwatch is associated with a Received Signal Strength (RSSI) value, which indicates how distant the beacon is from the smartwatch. The beacon has a unique advertisement identifier, which is programmed into the smartwatch and is used to identify the specific beacon. The iWash application installed on the smartwatch runs in background which continuously receives signals from any surrounding beacon. Therefore, when a user wearing the smartwatch approaches the door, iWash receives strong signals from the door beacon continuously. We use a simple scheme for detecting the home entrance. If the average RSSI value received from a beacon within the last 15 s is greater than a threshold value (−60 dB), and if the beacon identifier matches with that of the door beacon, iWash identifies the situation as the user is entering the home. Here the parameters, such as, −60 dB and 15 s are picked as a design choice, which led to satisfying results for the detection of the home entrance situation compared to other situations, such as the home exit. Moreover, on detecting the home entry of the user, iWash invokes the reminder system to remind him/her for washing hands.

4.5.2. Providing routine reminders

In the context of the infectious disease, it is also important to wash hands frequently throughout the day (CDC), whether the user is inside or outside the home. Moreover, smartwatches are very effective for notifying the user in all situation. iWash provides daily handwashing reminders to the user between 9 AM to 9 PM. It picks the reminder times intelligently. Whenever the user performs a handwashing event following a voice interaction, iWash records the time of the event as the last handwashing time. One hour from that time, it reminds the user with a voice message that it would be a good idea to wash hands now, since he/she did not wash hands during the last hour. This design choice ensures that the user is reminded in a routine way, but he/she is not bothered with too many reminders.

4.5.3. Voice interaction with the user

The small display size of the smartwatch display poses a serious challenge to interact with the user and to provide the detailed feedback about handwashing. In order to overcome this challenge, iWash uses voice input and output instead of any text for interaction. The voice interaction ensures that the user is interacted with in a touchless manner, which is critical to avoid further contamination of any virus or germ. While providing the reminder, iWash provides a local notification with a haptic vibration to alert the user. It then initiates the voice interaction with the user and asks him/her to wash hands. It uses the speaker of the smartwatch and leverages a text-to-speech framework for this purpose. When the user responds using any voice utterance, iWash takes the input via the microphone of the smartwatch, and converts the audio to text using a speech-to-text framework. Then it parses the text using a keyword based mechanism similar to (Mondol et al., 2016) (Samyoun et al., 2019) to understand the intent of the user. When the user confirms starting the handwashing, iWash starts collecting data from the accelerometer, gyroscope, and magnetometer sensors of the smartwatch. When the user confirms being done with handwashing, it stops collecting data and begins the quality assessment with the collected data. An example of the interaction between iWash and the user is demonstrated in Fig. 6 .

Fig. 6.

Flow of interaction of iWash with the user.

4.5.4. Postponing a reminder

The user may not be in a situation or may not be willing to wash hands immediately after being reminded. In such cases, iWash allows the user to reschedule a reminder at a later time using voice. The user can do so by simply asking iWash to remind him after a specific number of minutes. iWash finds out the specific time the user want to be reminded from the utterance text and sets the reminder accordingly. This design choice ensures that the user is reminded at a time of his/her own preference for washing hands.

5. Evaluation

We performed a thorough evaluation to demonstrate the performance superiority achieved by iWash compared to the state-of-the-art smartwatch-based handwashing quality assessment systems. The key questions that drive our evaluations are:

• 1.How accurately can iWash assess whether the user washed hands properly compared to the state-of-the-art systems?
• 2.In addition to high accuracy, can iWash minimize the prediction latency of the feedback provided to the user as per the claim of the paper?
• 3.How much battery of the smartwatch is consumed by iWash compared to the state-of-the-art systems?

5.1. Real-life dataset collection

We collected real-life handwashing data from 14 participants. The age range of the participants was from 25 to 35 years. Among the 14 participants, 10 were male and 4 were female. We developed a data collection app for the Apple watch. The app collects readings from the accelerometer, gyroscope, and magnetometer sensors of the smartwatch with 50Hz sampling rate. An instructional video on the WHO handwashing guidelines was shown to the participants so that they could learn the process. After 3 practice sessions, we collected data of 19 handwashing sessions from each participant. During each session, the participant wore the smartwatch on the right hand with the data collection app running on it and then performed a handwashing event. The participants were asked to wear the watch in their own ways as well as to perform the handwashing steps in their own ways. All the sessions were video-taped for collecting the ground truth of the handwashing steps. The collected data were used for the training of iWash.

5.2. Training of iWash

We trained the high accuracy model (Section 4.2) with the collected real-life dataset and then optimized the model for minimizing prediction latency while preserving accuracy (Section 4.3). The training was performed in a server that has a Intel processor with 10 cores, 256 GB RAM, and 4 Nvidia RTX 2080 Ti GPUs. We used Keras (Cholletet al., 2015) to implement and optimize the model.

5.3. Implementation of iWash

The implementation of iWash requires a smartwatch and a proximity Bluetooth beacon. In this work, we implemented iWash in real-life setting, where we used an Apple watch series 4 (Apple) and an Estimote BLE beacon (Estimote). The iWash smartwatch application was developed on the watchOS platform and the trained optimized model (Section 5.2) was integrated with it. The iWash application was installed on the Apple watch where it was running on background.

Estimote beacons are one of the most popular, low-energy, proximity bluetooth beacons. An estimote beacon was fitted at the door of the home and the user was wearing the Apple watch. As described in Section 4.5, using the beacon and the smartwatch application, iWash reminds the user for handwashing and engages the user in a voice interaction (shown in Fig. 7 ).

Fig. 7.

iWash implementation, with the beacon fitted at the door, user receiving reminder on the smartwatch and responding via voice interaction.

We deployed iWash with the 14 participants for testing. During testing, each participant did 1 session wearing the smartwatch with the iWash application installed. Following the handwashing, iWash provided a voice feedback to the user stating if he/she washed hands properly or not. If not, the feedback included if the user missed any steps and if the user did not wash for enough duration for a proper handwashing.

5.4. Experimental setup of iWash

5.4.1. Compared systems

We compared the performance of iWash against several state-of-the-art handwashing quality assessment systems on real-life handwashing data. These baselines were chosen considering that these systems identified the handwashing steps according to the WHO guidelines. All systems were trained on the collected real-life data (Section 5.1). We evaluated the systems on the test data mentioned in Section 5.3. A short description of the baselines are provided below:

• •WristWash: WristWash (Li et al., 2018) is an handwashing quality assessment system that provides a Hidden Markov Model-based analysis for assessing the steps according to the WHO guidelines.
• •H2DTR-NN: This work (Galluzzi et al., 2015) measures the duration and the quality of hand hygiene according to WHO guidelines using different techniques. The best classification results were obtained using a neural network based solution and a k-nearest neighbor based solution. H2DTR-NN represents the neural network based solution presented in the paper.
• •H2DTR-kNN: H2DTR-kNN represents the k-nearest neighbor based solution presented in above paper (Galluzzi et al., 2015).

As per the implementation details described in these papers, both systems require a server where the model for assessing the handwashing quality is loaded and run. When the user performs handwashing, the smartwatch sends the handwashing data to the server. The model running at the server then assesses the handwashing quality, and sends the result back to the smartwatch. In our implementation, the network bandwidth between the smartwatch and the server was 1 Gbps.

5.4.2. Performance metrics

• •
  Accuracy: To evaluate the performances of the systems in identifying the handwashing steps, we calculate accuracy, a commonly used performance metric. Accuracy of a handwashing step is defined as:

  $$\text{Accuracy of a step}=\frac{\text{Number of times the step was identified correctly}}{\text{Total number of times the step appeared in the handwashing data}}$$ (3)
• •Prediction latency: It measures the average time required by the system to provide the quality result of a handwashing event to the user.
• •Battery usage: It measures the amount of the battery consumed by the handwashing quality assessment system on the smartwatch.

5.5. Experimental results

5.5.1. Accuracy

We conducted thorough experiments to evaluate how accurately iWash assesses the quality of handwashing performed by an user compared to the state-of-the-art systems. Moreover, as previous systems (Li et al., 2018) have evaluated the accuracy performance for both user-independent and user-dependent models, we also conducted experiments in both settings to show the performance superiority achieved by iWash.

User-independent Model: For the user-independent model, the leave-one-participant-out (LOPO) validation was carried out where the data from 13 participants were used for training, and data from the remaining participant was used for testing. The process was repeated for every participant and the average accuracy score for each of the handwashing steps was calculated.

Fig. 8 presents the results for the handwashing steps detection accuracy of the systems for user-independent model. On average, iWash provided around 27.7%, 12.3%, and 14.6% better accuracy than WristWash, H2DTR-NN, and H2DTR-kNN, respectively. This performance improvement occurred because unlike the compared systems, iWash employs a hybrid deep neural network that effectively captures the correlation present among multiple sensors and the temporal correlation present in sensor data.

Fig. 8.

Handwashing steps detection accuracy of the systems for user-independent model.

User-dependent Model: For the user-dependent model, each of the compared system was trained and evaluated using the data from each participant individually. The process was conducted for every participant and the average accuracy score was calculated for each of handwashing steps. Fig. 9 presents the results for the handwashing steps detection accuracy of the systems for user-dependent model. On average, iWash provided around 15.6%, 4.2%, and 5.7% higher accuracy than WristWash, H2DTR-NN, and H2DTR-kNN, respectively. The superior performance of iWash occurred for the same reason as described in the user-independent model. Also, each of the systems provided at least 4% accuracy improvement in the user-dependent model than its corresponding user-independent model. This happened because the user-dependent model that has been trained on only a single individual's handwashing data, can better capture his/her unique features of hand movement during handwashing.

Fig. 9.

Handwashing steps detection accuracy of the systems for user-dependent model.

5.5.2. Prediction latency

The time required by the system to provide the handwashing quality result to user was measured for iWash and for the compared systems.

Fig. 10 shows the performances of iWash and the compared systems in terms of prediction latency. iWash provided around 46.2%, 50.1%, and 36.7% improvement in prediction latency compared to WristWash, H2DTR-NN, and H2DTR-kNN, respectively. The reason behind this performance improvement is that contrary to the compared systems, iWash is optimized for both accuracy and prediction latency.

Fig. 10.

Prediction latency comparison.

5.5.3. Battery usage

To show the effectiveness of iWash for use in daily life, we evaluated the daily battery usage of the iWash compared to the state-of-the-art systems, which is demonstrated in Fig. 11 . Each of compared systems were installed at a time on the smartwatch for evaluation. With each system installed, the battery level of the smartwatch was recorded at various times of the day from 8 a.m. until 7 a.m. next morning. Between 7 a.m. and 8AM the smartwatch was charged every day. The experiment was performed for 7 days on an Apple watch series 4 and we report here the average battery usage. Other than the installed systems, only the default Apple watch applications (Inc, 2020) were running on the watch. A user performed 10 handwashing events on daily basis at various times of the day. The intuition behind this is that, according to previous study, on average a person washes hands 10 times a day (Institute, 2109). The result shows that without having any of the systems installed, the battery level of the smartwatch fell to 26% by the end of the day, whereas using iWash, the battery level of the smartwatch fell to 22% by the end of the day. Thus, it is evident that iWash does not consume any significant battery for the daily life use, while providing a handwashing assessment system to the user on the smartwatch with high accuracy and low prediction latency. On the other hand, using the other compared systems, the battery level dropped to a level of 8%–10% making those systems less effective for daily basis usage. Now-a-days, most of the people use multiple apps on smartwatch throughout a day (Jeong et al., 2017), this is where the additional battery level for iWash will be useful. The reason for the performance superiority of iWash is, unlike the other systems, iWash is optimized so that it effectively runs on the smartwatch with minimal battery usage.

Fig. 11.

Battery usage comparison.

5.5.4. Device adaptivity of iWash

The number of FLOPs works as a delegate for prediction latency or consumed energy (Li et al., 2020). The greater the number of FLOPs, the higher the prediction latency or consumed energy is. As explained in Section 4.3, the hyper-parameter α introduces a trade-off between accuracy and the number of FLOPs in the deep learning model of iWash. Fig. 12 illustrates the trade-off. Here, accuracy refers to the average accuracy of the handwashing steps in the user-independent model. We observe that with the increase of the value of α, the accuracy increased, but the number of FLOPs also increased which in turn increased the prediction latency or battery usage of iWash. This occurred because with the increase of α, the model was allowed to be more complex facilitating higher accuracy, albeit requiring higher computation. By controlling the value of α, we can fulfill the resource constraints of a low-resource device like a smartwatch. In our current implementation, we consider the value of α as 0.5. If a smartwatch has less resources or we need to make further room for other applications running on the device, we can consider the value of α less than 0.5. From Fig. 12, we observe that, when the value of α decreased from 0.5 to 0.3, number of FLOPs decreased by around 41%, but accuracy decreased by only 1.2%. Thus, by sacrificing accuracy a little, we can further decrease the battery usage of iWash. Similarly, for future smartwatches that are expected to have higher battery resources, we can consider a higher value of α that allows achieving higher accuracy. The baseline systems do not have such device adaptivity and, hence, we do not report them in this experiment.

5.6. Overall results

Overall, the results show that iWash is a comprehensive system that overcomes several limitations of the state-of-the-art handwashing assessment systems in the context of the infectious diseases. Section 5.5.1 shows that iWash is able to detect the handwashing steps according to the WHO guidelines with a significantly higher accuracy than the other systems for different settings. Also, the results in Section 5.5.2, 5.5.3 demonstrates that iWash is designed and implemented in a way optimized for running on a smartwatch, as it provides faster processing time and consumes less battery than the other systems to facilitate real-time feedback provided to the user. Section 5.5.4 shows that iWash can be tuned to adapt as per the resource availability of the devices unlike the state-of-the-art systems which offers no flexibility.

6. Related work

6.1. Camera-based handwashing quality assessment systems

Many past works on handwashing quality assessment are based on a camera or vision based systems. These approaches usually require a video camera or a depth sensor camera to be placed above the sink (Llorca et al., 2011) (Xia et al., 2015) (Hoey et al., 2010) (Ltd, 2020). The camera records the hand movements performed by the user during the hand hygiene activity. The images or videos from the camera are sent to a cloud server, where the data is processed using vision-based approaches to assess the hand hygiene activities. For example, Llorca et al. (Llorca et al., 2011) used such an approach using a RGB camera to access the hand hygiene activities of the healthcare workers. Xia et al. (Xia et al., 2015) used similar approaches to recognize hand hygiene poses using RGB-D Videos. However, there are several reasons such approaches are not suitable for daily life settings. First, placing a camera in a bathroom or kitchen is potentially privacy invasive. Second, these approaches are not ubiquitous. For example, a camera-based system will not work when the user is outside the home or when the user follows the hand rubbing approach which does not require a sink. Third, such approaches require substantial installation and maintenance costs (Boyce, 2017), as well as involves heavy computation for the analysis of images and videos.

6.2. Wearable-based handwashing quality assessment and reminder systems

Several wearables-based systems (Li et al., 2018) (Galluzzi et al., 2015) (Wang et al., 2020) (Mondol and Stankovic, 2015) have been employed for the assessment of handwashing quality. Such approaches overcome all the limitations of the camera-based systems. It does not require installing any camera on the bathroom or kitchen, and works well even when the user in outside the home. This approach usually collects the wrist movements data during a handwashing event using the wrist-worn sensors and then uses different techniques for detecting the handwashing steps from the event according to the WHO guidelines. Additionally, some works integrate handwashing reminders (Mondol and Stankovic, 2015) on the smartwatch too. WristWash (Li et al., 2018) is a wrist-worn platform that provides offline analysis for assessing the steps according the WHO guidelines using Hidden Markov Model-based method. Galluzzi et al. (Galluzzi et al., 2015) used different machine learning based techniques for recognizing the handwashing duration and steps according to the WHO guidelines using the wrist-worn sensors. However, there are several reasons why all these systems are not comprehensive enough in the context of infectious diseases. First, most of these works are limited to offline analysis only and none of these provide any feedback to the user whether he/she washed hands properly. Second, none of these systems focused on optimizing the systems for running efficiently on smartwatches in addition to guaranteeing accuracy. Third, none of these systems provide any reminders depending on the user context or offer any interaction in touch-free way. Compared to these systems, we achieve higher accuracy for quality assessment, as well as minimize the smartwatch processing time and battery usage to facilitate the immediate feedback. Moreover, in the context of infectious diseases, our system provide the feedback to the users in real-time if he/she did not wash hands properly, reminds the users when entering home and offers touch-free interaction with the user.

6.3. Other approaches

In the past, adherence to the WHO guidelines was also measured from the direct observation by trained auditors (Arias et al., 2016) (Tschudin-Sutter et al., 2015) (Szilágyi et al., 2013). However, such approaches reported very poor compliance to the WHO guidelines (Tschudin-Sutter et al., 2015) (Szilágyi et al., 2013). Moreover, such approaches are not automated and are impractical in the context continuous monitoring and social distancing. Previous works have also used automated dispenser based systems to count the handwashing events electronically (Kinsella et al., 2007) (Marra et al., 2010). However, such approaches are not ubiquitous, and can not detect if the user performed the handwashing properly.

6.4. Handwashing applications since the COVID-19 outbreak

Since the COVID-19 outbreak, measures for spreading handwashing awareness and to improve handwashing practice have become widespread. In line with that, now-a-days commercially available smartwatches and wearables companies are trying to add handwashing features utilizing the embedded sensors. Samsung has developed a handwashing app to the Galaxy smartwatches (Samsung, 2020) that will provide reminders to wash hands regularly and track the duration of the handwashing. Google has also added similar features to their smartwatch operating system (Google). Apple recently announced that it will add handwashing features in the next watchOS version, such as detecting handwashing and it's duration automatically (Apple, 2020). However, so far none of these companies have found an established solution to assess the quality of handwashing with respect to the standard guidelines or to provide any feedback to the user whether he/she washed hands properly. Moreover, most of these companies are still working on these features, none have brought any comprehensive system so far. Nevertheless, the current trend of the industry shows the promise of smartwatch based handwashing applications for the future.

7. Discussion and future work

• •Potential use-cases of iWash: The performance superiority achieved by iWash will also be very significant in clinical environments, that is for healthcare workers where hand hygiene measures needs to be strictly followed. A study (Stimpfel et al., 2019) shows that the daily average work shift duration of an healthcare worker is 12 h. The battery usage evaluation of iWash shows that the smartwatch battery level with the iWash installed and used on a regular basis can last for the whole day. Therefore, iWash can be effectively worn by a healthcare worker, while it can provide them feedback about their handwashing quality with a high accuracy, which will help reduce the healthcare-associated infections. Moreover, previous studies (Jeong et al., 2017) have shown that the average daily wear times of a smartwatch are typically 8–10 hours depending on the weekend/weekdays. Since iWash does not consume any significant amount of battery life (Section 5.5.3) compared to not having a handwashing quality assessment system, it will be very effective for improving the hand hygiene practice in daily life.
• •Privacy: The whole system is deployed in smartwatch and requires no connection with any third-party entity (e.g., cloud provider). Thus, handwashing data do not leave the smartwatch and privacy of user data is ensured.
• •Effect of handedness and the order of the handwashing steps: Since washing hands involves movement of both hands, using data from both wrists would provide better performance in assessing the quality of the handwashing events. However, it is not practical to wear smartwatches on both hands, particularly in free living context. Our system requires the user to wear the smartwatch only on one wrist. In this work, we evaluated our system on the data collected from the right hand. Considering the fact that our method is agnostic to the placement of the watch, it would also work for the left hand. In future, we will perform evaluations for the left hand too. Moreover, iWash was trained to detect the individual handwashing steps and during evaluation, the users performed the handwashing steps in arbitrary order. So the order of the handwashing steps does not effect the performance of iWash.
• •Larger user study in the context of infectious diseases: The focus of this work is to develop a comprehensive system for handwashing quality assessment on a smartwatch, that is more accurate, runs faster and consumes less battery compared to the state-of-the-art, as well as integrates context-aware reminder and touch-free interaction with the user. Applying the system on a larger group of people will be the next step to demonstrate its capability in reducing the spread of infectious diseases - which is beyond the scope of this paper.
• •Effect of the handwashing related apps on user behavior: Previous research shows that the smartphone applications helps to reduce health care–associated infections (HAIs) in clinical context by providing easy access to the step-by-step and monitoring support (Schnall & Iribarren, 2015). Since the COVID-19 pandemic, the smartwatch and wearable companies have started focusing on the handwashing related apps to improve the handwashing practice among users, which has been discussed in Section 6.1.
• •Voice-based cognitive assistant: Modern consumer-grade smartwatches have a built-in speaker and microphone. Using these, previous work (Samyoun et al., 2019) used the idea of voice-interactive assistants for stroke patients that can interact with the user about any query regarding the medication and exercise. Similar to that, iWash can be extended to a voice-based cognitive assistant for any pandemic situation. Besides offering handwashing reminders and voice interaction, in future iWash can integrate up-to-date public information about the pandemic downloaded to the smartwatch daily, such as, the current status of the pandemic, current restrictions, etc. Users can then, at any time, ask for information from smartwatch on these topics and receive a verbal response.

8. Conclusion

Proper handwashing is widely acknowledged to be one of the most important activities for reducing the spread of infection. In this paper, we designed and implemented iWash. To the best of our knowledge, iWash is the most comprehensive system for smartwatch handwashing quality assessment, that achieves significantly higher accuracy compared to the state-of-the-art handwashing quality assessment systems. It runs on a smartwatch with minimal processing time and battery usage facilitating real-time feedback to the user. It also provides routine and context-aware reminders, and offers touch-free interaction. It is also adaptive according to the device requirements without sacrificing accuracy significantly. We collected real-life dataset and extensive evaluations on the dataset demonstrate that iWash achieves around 12% higher accuracy, 37% lower processing time, and 10% lower battery usage than the state-of-the-art handwashing quality assessment systems.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.