Accuracy of a radiofrequency identification (RFID) badge system to monitor hand hygiene behavior during routine clinical activities

Abstract

Background

Hand hygiene (HH) is a critical part of infection prevention in healthcare settings. Hospitals around the world continuously struggle to improve healthcare personnel (HCP) HH compliance. The current gold standard for monitoring compliance is direct observation; however this method is time consuming and costly. One emerging area of interest involves automated systems for monitoring HH behavior such as radiofrequency identification (RFID) tracking systems.

Methods

To assess the accuracy of a commercially available RFID system in detecting HCP HH behavior, we compared direct observation to data collected by the RFID system in a simulated validation setting and to a real-life clinical setting over two hospitals.

Results

A total of 1554 HH events were observed. Accuracy for identifying HH events was high in the simulated validation setting (88.5%) but relatively low in the real-life clinical setting (52.4%). This difference was significant (p<0.01). Accuracy for detecting HCP movement into and out of patient rooms was also high in the simulated setting but not in the real-life clinical setting (100% on entry and exit in simulated setting vs. 54.3% entry and 49.5% exit in real-life clinical setting, p<.01).

Conclusions

In this validation study of an RFID system, almost half of the HH events were missed. More research is necessary to further develop these systems and improve accuracy prior to widespread adoption.

Background

Hand hygiene (HH) is a critical part of infection prevention in healthcare settings.^{1, 2} Hospitals around the world continuously struggle to improve healthcare personnel (HCP) HH compliance. The current gold standard for monitoring compliance is direct observation; however this method is time consuming, costly, and only captures a small percentage of HH opportunities.^{2, 3} Another concern is the Hawthorne effect, which inflates compliance rates when an observer is present.^{1, 4} One emerging area of interest involves automated systems for monitoring hand hygiene behavior.³ There are many types of automated systems, including counting systems, ultrasound, infrared, WiFi, and radiofrequency identification (RFID), among others.^{3, 5, 6} These systems may incorporate features such as direct performance feedback to HCP and have recently been reported to significantly improve compliance.^{3, 7, 8} However, in order to be effective, electronic compliance monitoring systems must be reliable, minimally intrusive, and accepted by HCP. A study using structured HCP focus groups showed that accuracy is the most common concern HCP have about electronic badge monitoring systems.⁹ Prior to widespread adoption of costly technology, it is important to understand the capacity of these systems to accurately monitor hand hygiene compliance in real-life clinical settings. No previous studies have systematically assessed the accuracy of RFID badge systems across multiple hospitals or compared performance between simulated and real-life clinical settings.³

The aim of this study was to assess the accuracy of a proprietary RFID badge system for detecting HCP activity including room entry, room exit and hand hygiene compliance in a real-life clinical practice at two large academic medical centers.

Methods

We reviewed several hand hygiene monitoring systems (see Table 1 for detail). We wanted a system that was easy to install using existing alcohol gel and soap dispensers, would not require a change in HCP behavior, and could give reports on individual HCP. We did not want a system that would require a change in HCP behavior or that reminded HCW to wash their hands. We selected the nGage™ system (Proventix Systems, Inc.) because it met all of our criteria. After obtaining institutional review board (IRB) approval, the RFID system for monitoring hand hygiene was installed in intensive care units of two academic medical centers (550 and 757 beds). The RFID system worked by having HCP wear badges designated with unique identifiers such that each room entry, exit, and HH event could be attributed to a single individual by “readers” installed adjacent to soap and alcohol-based hand rub dispensers, both inside and outside of patient rooms. The readers displayed a personalized message (i.e. “Thank you for washing, Nurse X”) whenever a HCP used the dispenser. The readers also displayed non-personalized messages including weather updates, hand hygiene tips, current events or fun facts when the dispenser was triggered. Each reader transmitted data wirelessly to an access point and data were stored on an off-site secure server. Study staff accessed data through a web-based interface.

Table 1. Comparison of Electronic Hand Hygiene Monitoring Systems commercially available to monitor individual hand hygiene compliance and key criteria for selecting the system used (not all systems were available at the time of the study).

Monitoring system	HCP tracking method	Technology*	Requires change in HCP behavior	Badge/wristband reminds HCP to wash	Use with existing dispensers
nGage™	Badge	R/W			✓
HyGreen	Badge	W/IR	✓	✓	✓
BIOVIGIL	Badge	IR/P	✓	✓	✓
Versus SafeHaven™	Badge	R/IR
UltraClenz Patient Safeguard System™	Badge	P		✓
Hyginex	Wristband	W		✓
MedSense	Badge	W		✓
HandGiene HHMS™	Badge or wristband	R/W
IntelligentM	Wristband	R	Unknown	✓	✓

^*R=RFID, W=WiFi, IR=Infrared, P=Proprietary

Phase 1 of the study, the simulated setting, involved validation of reader accuracy with research staff following a planned path through the units entering each room and using each room and hallway hand hygiene dispenser while wearing a badge. Several common scenarios were followed, including differences in badge placement (e.g., lapel pocket, waistband) and different positions (e.g., facing dispenser, standing sideways) for using the dispenser. Data from the electronic readers were compared to actual behavior (i.e., the planned path) to determine accuracy.

In Phase 2, the real-life clinical setting, HCP who consented to participation were assigned an RFID badge and given standard instructions on how to wear the badge. HCP were not instructed to modify their behavior in any way. Each HCP wore the badge during their entire 12-hour work shift. Their schedules were obtained so that research staff could overtly observe their actual behavior during a normal work shift. These direct observations (gold standard) were compared to the data recorded by the RFID system using chi-square or Fisher's exact test.

Results

There were no electronic conflicts or interference between RFID and medical devices or other technology reported at either site. In the simulated setting, research staff completed 1197 hand washes. Of these, 1059 (88.5%) were accurately attributed to the correct RFID badge. Accuracy was highest when participants stood directly in front of the dispenser with the badge on their upper lapel (370/381, 97.1%). Accuracy was lower when participants stood perpendicular to the dispenser; 97 of 113 (85.8%) washes were properly attributed when participants stood perpendicular to the dispenser and wore the badge on the side closer to the dispenser and 92/122 (75.4%) were properly attributed when participants stood perpendicular to the dispenser but wore their badge on the side farther from the dispenser. Additionally, 170 room entries and exits were observed and 170 (100%) were properly detected by the RFID system. A summary of Phase 1 results can be found in Figure 1. In Phase 2, 31 HCP were enrolled in the study. The study period lasted six months. After excluding a small number of observations due to improper badge positioning, 357 observed HH events were included in the analysis. Of these, 187 (52.4%) were properly attributed by the RFID system to the HCP who performed the HH behavior. On room entry, 94 entries were observed and 51 (54.3%) were accurately recorded by the RFID system. On room exit, 93 room exits were observed and 46 (49.5%) were accurately recorded by the RFID system. The difference in accuracy in HH detection between study conditions and actual conditions was statistically significant (X²=225.38, p<.01). The difference in detection of room entry and room exit was also statistically significant (p<.01). These results are summarized in Figure 2. An image illustrating interaction between healthcare personnel position and badge functioning is displayed in Figure 3. Various HCP positions that were associated with improper data are presented in the figure.

Figure 1.

Radiofrequency Identification hand hygiene monitoring system accuracy in controlled situation with volunteers of average height (between 5′4” and 5′10”) wearing badges in different positions as well as short (5′3”) and tall (6′3”) volunteers.

Figure 2. Radiofrequency identification detection hand hygiene system accuracy in simulated validation phase vs. real-life clinical practice.

Figure 3.

Example of radiofrequency identification (RFID) badge detection system used in a hospital unit with fields detecting HCP in a patient room (blue) and when using a hand hygiene dispenser (yellow). Multiple sample HCPs are depicted with a badge in place on their left breast pocket. The badge is green when HCP are accurately positioned to be recognized by the system for room entry or hand hygiene. Reasons for misreading include badge being blocked by HCP bodies (1) and being outside the field of detection (2).

Discussion/Conclusions

In this multicenter study we found that hand hygiene compliance monitoring with RFID performed well in a simulated setting that maximized the accuracy of the technology. However, during real-life clinical activities, the same system correctly identified approximately half of hand hygiene events. Reasons for such poor performance in the real-life clinical setting included improper positioning of HCP in front of the hand hygiene dispensers and quick “fly by” hand washing.

The poor performance of a well-developed RFID hand hygiene monitoring system that performed well in a simulated setting is disappointing but not unexpected. The technology necessary to monitor human behavior is complex. For example, RFID signals are absorbed by water, rendering the tag unable to transmit a signal to the sensor if any portion of the body is between the badge and the reader. In Phase 1, we showed high accuracy when the badged test subject was instructed to stand directly in front of the dispenser when performing HH and to enter each room within range of the reader; however this was not requested in Phase 2 where HCP performed normal activities. RFID signals are also capable of transmitting through walls³ and only a single in-room sensor was used, so HCP were occasionally recorded as entering or exiting a room adjacent to the room they actually entered or exited.

The current gold standard, direct observation, suffers from limitations in that it only detects an extremely small fraction of HH opportunities each day and most hospitals cannot maintain the secrecy of the observers. Furthermore, because of lack of individual data, this approach is unable to identify individuals with consistently poor or outstanding performance. Finally, direct observations have been shown to contribute to a Hawthorne effect which inaccurately inflates compliance rates.⁴ The potential of an automated system to overcome these challenges is appealing, although not yet realized. The current system was able to correctly assess approximately half of all HH opportunities.

A recent abstract highlights the importance of HCP confidence that the system is actually monitoring their compliance accurately. The study showed that poor accuracy can lead to a counterproductive reduction in hand hygiene compliance. In this study, an automated system with 60% accuracy for detecting HCP HH was introduced. After introduction of the system, compliance decreased by 36%.¹⁰ For an RFID system to be effective, it must be extremely sophisticated to account for the many activities that occur in everyday clinical settings. Additionally, the system should not require a change in HCP behavior, i.e. HCP being responsible for ensuring that the system is correctly reading their badge. These continue to be unrealized goals for all systems to date.

Limitations of this study include: 1) Only 31 HCP were enrolled, limiting the extent of variation in body habitus and behaviors of HCP; and 2) we did not monitor overall unit level hand hygiene compliance before and after installing the RFID system. This study had strengths including that it was conducted at multiple sites with a commercially available system with a formal protocol for validation both in an idealized setting and routine clinical activities.

In conclusion, in the first multisite validation of a commercially available RFID hand hygiene monitoring system, we found the system was very accurate in an idealized setting but correctly attributed one half of HCP hand hygiene events during routine clinical activities. Validation of automated systems for hand hygiene compliance must be tested in actual clinical practice. Otherwise, estimates of accuracy will overestimate accuracy in real-world clinical settings. Infection preventionists and hospital epidemiologists must understand the limitations of using technology as a monitoring tool. Continued study is necessary to determine if such systems can be used to monitor hand hygiene compliance and whether they can improve overall hand hygiene compliance.