Multidimensional evaluation of a radio frequency identification wi-fi location tracking system in an acute-care hospital setting

Abstract

Real-time locating systems (RTLS) have the potential to enhance healthcare systems through the live tracking of assets, patients and staff. This study evaluated a commercially available RTLS system deployed in a clinical setting, with three objectives: (1) assessment of the location accuracy of the technology in a clinical setting; (2) assessment of the value of asset tracking to staff; and (3) assessment of threshold monitoring applications developed for patient tracking and inventory control. Simulated daily activities were monitored by RTLS and compared with direct research team observations. Staff surveys and interviews concerning the system's effectiveness and accuracy were also conducted and analyzed. The study showed only modest location accuracy, and mixed reactions in staff interviews. These findings reveal that the technology needs to be refined further for better specific location accuracy before full-scale implementation can be recommended.

The development of radio frequency identification (RFID) devices has led to prolific uptake in many sectors, including healthcare, in which it is estimated that overall spending for RFID tags in healthcare could surpass US$2 billion in 2018.¹ Claims of the potential benefits of RFID are many, with suggestions that RFID technology will improve supply chain efficiency and patient outcomes, increase the speed for critical treatments, provide better tracking of patient drug treatment, and lower labor costs.² As a result of these claims, many hospitals around the world have invested in, and implemented, RFID tags.³

Despite this early uptake, RFID lacks the ability to give the live location and real-time tracking of an asset or person. In theory, real-time locating systems (RTLS), which combine RFID and wi-fi, will allow for real-time data gathering and live asset tracking. RTLS can calculate distances, range, time or direction in order to determine the location of objects, and is capable of providing stored data that can be used for analysis and reporting.⁴ This information can be applied to patient safety in the form of patient tracking and identification, and can provide valuable information required to meet patient safety goals such as those proposed by the Joint Commission in 2011.⁵ While the features of RTLS provide the potential to conduct numerous research studies and a plethora of potential patient safety benefits, there is a need for formal evaluation of this technology in the clinical setting.

Although there is much internet-based information concerning the accuracy and uses of RTLS in hospitals, there is a dearth of published articles. Many of the articles published to date discuss the use of active RFID tags within a custom algorithm developed internally for live tracking and location.^6–10 Of the published RTLS studies, many were either conducted by industry or are industry-sponsored evaluations of a technology.¹¹ Among non-industry-funded research, we identified only one study that tested commercial RTLS and it contained little information on location accuracy testing, and only subjective assessments of system benefits, with limited sample size.¹² The body of literature in this area is thus rather incomplete. More multifaceted and multidisciplinary evaluations of commercial RTLS need to be conducted before widespread hospital investment can be recommended.

Throughout this study, AeroScout RTLS technology was used. In 2009, AeroScout arose as a leading vendor of RTLS technology for healthcare, holding 45% of the market.¹³ Its technology was implemented in over 500 organizations,¹⁴ and AeroScout added 30 new hospitals to its list in the first quarter in 2009.¹⁴ It was reasonable to assume that AeroScout would provide a representation of other vendors such as RadarFind, Ekahau, Radianse and Awarepoint, to name a few, who provide similar products that run on wi-fi-based RFID tags.¹⁵ IBM supplied and supported RFID and RTLS technology for a formal evaluation in a clinical setting. The objectives of this evaluation were threefold: (1) to assess the location accuracy of the technology in a clinical setting; (2) to assess the value of asset tracking to staff; and (3) to assess threshold monitoring applications for patient wandering and inventory control.

Methods

RTLS technology

The RTLS system assessed uses battery-powered RFID tags operating at a 2.45 GHz frequency. The wireless network receives the signal from the active RFID tag at predetermined time intervals (beacon rates), which allows the system to convey the position of the asset to AeroScout's online computer system, MobileView, accessible through secure login on hospital computers. All tags that are active on a given unit are visible in MobileView, and each tag is given a corresponding label and icon that reflects its purpose. AeroScout's RTLS also allows users to perform threshold monitoring. Threshold boundaries are defined in the AeroScout Mobileview software using hardware exciters located at the physical location of the threshold boundary (eg, doorways). RTLS tags entering and exiting the threshold boundaries are detected by the exciters. Events are defined for each threshold boundary in the Mobileview application, and each event generates a specific alert in the form of an email, a short message service alert, hyper-text transfer protocol postings, or pop-up notifications on computers and other network devices. These alerts are used to notify clinicians or staff that a threshold event has taken place. For this study, AeroScout supplied models T2 and T3 and the EX2000 and EX3200 exciters.¹⁶

Evaluation methods

This evaluative study was conducted on the Ward of the 21st Century (W21C—see http://www.w21c.org), a medical teaching unit affiliated with the University of Calgary. The RTLS system was installed on the W21C medical ward over the existing wi-fi network. A site survey determined wi-fi coverage and signal strength to be effective for the space layout. Staff training and a step-by-step diluted mesh calibration guide ensured the system was functioning consistently throughout the project.

During the pilot testing and validation stage, simultaneous tracking of data relating to time, position and interactions of medical personnel were recorded by the RTLS system and by a human observer recording position benchmarks on a map of the hospital floor plan. Two sets of tests were conducted: static and dynamic. Static tests involved a researcher walking a random path and stopping for full minute periods to record their position. Dynamic tests, meanwhile, involved having a researcher walk naturally in a predetermined path around the unit, and making arbitrary/unplanned stops periodically. Any deviations from the path and stops were recorded. Some dynamic tests involved giving a nurse or a doctor one tag to be worn throughout their shift. All person-observed data were entered manually into a specialized computer program developed internally called the clinical path tracking and visualization system, which generated a digital picture of the path undertaken by the researcher.¹⁷ The digital picture was compared visually with the picture generated by AeroScout's Mobileview tracking system.

In judging accuracy, we will only refer to qualitative observations. However, during the data collection stages of this study, an operational definition of ±2 m was designated as being accurate, and thus all results were derived from this definition.

Asset tracking

Researchers approached staff and provided training on the use of the technology, followed by a 1-week period to allow for familiarization. A follow-up survey was administered to assess the utility of the technology as a clinical tool for asset tracking. The survey asked nurses to indicate how many times they used the technology, if it helped them to locate assets and how far away the asset was from where the system said it would be, why they used the system, suggestions for improvement and if they could see a future use for it on the unit. In order to assess if RTLS was useful to the clinical area, participants had to record at least one instance (in an 8-h shift) when the system helped them to find an asset. We considered a use anywhere from two to five instances to be ‘moderately useful’ and more than five instances to be ‘very useful’. Forty surveys were handed out over a 4-month period.

Threshold monitoring

Two different threshold monitoring situations were simulated: patient wandering and inventory management. The goals were to determine how effective RTLS was in detecting patient movement and real-time inventory monitoring. To simulate monitoring for wandering patients, the threshold boundary was established at the entrance/exit point of one hallway via exciters mounted in the ceiling with AeroScout's System Manager software defining the detection radius. Two exciters were used to reduce false entrances and exits through the hallway threshold. Researchers wore RFID tags and simulated patient wandering by moving through the designated threshold boundary. Entrance events were triggered when a patient moved underneath the first hallway exciter and passed the second exciter within the 30-s time frame. All detected events triggered an alert in the form of a custom pop-up on researchers' computers. The effectiveness, accuracy and timeliness of the pop-up messages were measured. Ten entrance and exit events in multiple directions were recorded. For inventory management testing, an exciter was placed in a cabinet with tags representing valuable inventory. Tags were then removed from the cabinet to simulate depletion of inventory for a total of 26 removal events. RTLS was observed to see if the removal of inventory was accurately recorded, and to determine how effective the system was in alerting the need for re-stocking.

Results

Validation of the technology

IBM conducted a survey of the wireless network on the test unit, and used the information from this survey to suggest specific locations to set up additional wireless access points and chokepoint detectors to our existing wireless network. Figure 1 shows the resulting set up of the access points on the test unit. The technology was calibrated once in the beginning, and then recalibrated three times in the span of 3 months to ensure maximum accuracy or otherwise in efforts to improve accuracy.

Figure 1.

The set-up of the wireless access points and chokepoint detectors for superior performance of the real-time locating system (RTLS), as directed by IBM.

Eight static tests were performed with 239 min of data and 20 dynamic tests were conducted with 4796 min of data. Figure 2 shows a typical test trial in which we evaluated the validity of location measurements made by the system. Figure 2 (panel A) shows the path the researcher walked, as recorded by a human observer. Figure 2 (panel B), shows AeroScout's MobileView RTLS-based rendering for the same path. Comparison of panels A and B indicates that AeroScout's system did not accurately report locations in the hallways, often indicating that the researcher was situated in patient rooms. A visual comparison of panel A with B shows that the average location inaccuracy is 2 m or greater (as the width of the hallway from wall to wall is 2.5 m).

Figure 2.

A typical example of the real-time locating system (RTLS). (A) Clinical path tracking and visualization system for medical assets. (B) AeroScout's MobileView generation of the same path. Two points were selected on panel A, representing a 1-min time frame. During the 1-min time interval, AeroScout recorded seven points at each of the two selected locations. The AeroScout generated points that correspond to the researcher's actual location (circled in panel A) are circled in panel B.

Figure 3 shows the suboptimal results of the dynamic location testing for a nurse wearing a tag throughout a full shift. The combination of location inaccuracy and beacon rate challenges (frequent rates produce too many data points; infrequent rates miss relevant movements) indicated limited utility for dynamic time–motion uses.

Figure 3.

Result from a dynamic test involving one nurse wearing one tag for 6 h.

Asset tracking

Figure 4 summarizes the survey results, demonstrating that the system only sometimes helped users find the tagged asset, but in most instances the asset was found quite distant from the depicted location. Table 1 extends these results by presenting the nurses' suggestions to improve the system.

Figure 4.

Results from the surveys handed out to nurses concerning asset tracking (n=13).

Table 1.

Summary of suggestions given by nurses about the current RTLS system to improve its utility (n=13)

Suggestions	No of nurses who made the suggestion
Need for increased accuracy	4
Increased visual stimulation and simplification of system, including clarification of icons and map	4
System to indicate the specific location of the asset (eg, the room no)	4
Creation of ‘cheat sheets’ on each computer on how to use the system	3
Need for hospital-wide implementation in order to track asset once it is off the unit	1
Total suggestions*	16

^*Total suggestions is higher than total participants who replied to the study as some nurses made more than one suggestion, or their suggestion would fall under two categories.

RTLS, real-time locating system.

Comments on the surveys revealed negative views on the system's accuracy (eg, one nurse stated ‘the system does not work at all’, another that the system would give a location of the asset but the asset could not be found in that area) and questioned its use when their role required little need for equipment, when low technology options were easier and available to them (eg, asking colleagues), or when they already knew where all the equipment was stored. Despite these comments, participants nevertheless could see a potential future use for the technology for minimizing hospital cost and decreasing search time of assets, especially in assets that are commonly shared.

Threshold monitoring

For patient wandering the AeroScout RTLS provided an accuracy rate of 100% in detecting the total of 20 threshold enter and exit events, generating no false positives. Different movement paths did not have an effect on the detection accuracy of the RTLS. The RTLS also successfully recorded 100% of the absence or ‘re-stock’ events that were simulated, again with no false positives recorded. In both scenarios, pop-up alerts generated by MobileView occurred immediately (within 1 s) on the computers when a threshold event was triggered.

Discussion

While it was evident that the RTLS system is capable of tracking assets, for applications that require only modest accuracy for a precise location, the system has limited utility based on our findings. Moreover, the frequent calibrations required to maximize accuracy are not feasible on a large scale due to cost and time. Surveys of nurses responding to asset tracking revealed that they were generally discouraged from using the system due to its inaccuracy and its inability ultimately to reduce search time given the lower technology, yet entirely viable, options available (eg, asking colleagues). In contrast, the threshold monitoring simulation found the technology to be effective and precise in detecting threshold events, an indicator of potential value to certain patient-monitoring functions.

The limited body of literature describing RTLS evaluations identifies the potential benefits achieved by the technology, stressing its capability to track and identify difficult patients,⁶ improve equipment workflow while reducing hospital cost,¹¹ and assist providers and save time with asset tracking.¹² However, this earlier research has not analyzed the location accuracy abilities of commercial systems, and given that accuracy is a prerequisite for at least some of these benefits, it was imperative that a study such as ours be conducted. Throughout our study, we refer only qualitatively to accuracy measurements, instead of the original ±2 m, due to subsequent conversations with the companies indicating that our benchmark accuracy of ±2 m is probably not achievable with the current generation of RTLS technology. This information underlines an important shortcoming in the technology. Our study showed that the system evaluated functions well for locating the general identity of assets and people (eg, which wing of the unit they are in), but failed in pinpointing precise locations. The modest location capabilities of the system will limit its utility for functions such as refined location of assets or time–motion analysis of healthcare providers. Based on this finding, it would appear that at least some of the marketing claims surrounding the technology in domains of ‘clinical flow’ and ‘quality of care’ may be unsubstantiated based on the current generation of technology and current evidence of impact. Other potential uses in the realm of patient safety, such as eliminating patient identification error, may not be feasible at this time due to the system's inability to position a person accurately to one room. In addition, many hospital floor plans include more than one patient per room and the emergency departments occasionally place patients in chairs, decreasing the distance between patients and further increasing the importance of optimal accuracy. In these instances, even if the system was able to pinpoint accuracy to one room, this would not be sufficient. Future generations of RTLS technology will need to achieve better location accuracy to yield the potential clinical benefits that are proposed to arise from hospital investment in such systems.

Study limitations

The technology was set up on only one unit within the hospital, thereby limiting our ability to track tags once they left the unit. Furthermore, we only assessed one RTLS technology produced and marketed by AeroScout. Such RTLS technologies are continually evolving and changing within the healthcare field. Since the time this system was tested, upgrades have been made to the technology both through AeroScout and through other technology development companies, and it is thus possible that more contemporary systems would perform better.

Lessons learned

AeroScout's RTLS was implemented on an inpatient medical unit in a Canadian tertiary care centre, and the overall performance and clinical utility in evaluative testing were found to be rather modest, primarily because of suboptimal specific location accuracy. Our global conclusions for the system tested are that there should not be a major health system investment for widespread implementation of this technology unless the proposed use is in a realm where high system performance can be counted on. Future generations of RTLS may have continuing promise as an information technology tool in healthcare settings, but we strongly recommend that similar multidimensional assessments of technology be undertaken before major investment in these technologies is made.