Electronic hand hygiene monitoring: accuracy, impact on the Hawthorne effect and efficiency

Abstract

Objectives:

To evaluate a type five electronic monitoring system (EMS) for hand hygiene (HH) adherence with respect to accuracy and ability to avoid the Hawthorne effect.

Design:

HH events were observed manually and electronically. The agreement between the two observation methods was evaluated. Continuous electronic measurement was made throughout the study.

Setting:

An acute 31-bed medical ward in a National Health Service trust in London, United Kingdom.

Participants:

Staff working or attached to the ward.

Intervention:

A newly developed type five EMS that can measure disinfectant dispenser usage as well as continuous movements of health workers throughout the ward with arm-length precision and analyse HH adherence was installed at the ward.

Results:

A total of 294 HH events were observed in five sessions by an observer previously unknown to the ward. There was concordance between HH adherence assessed by manual observer and the EMS on 84% (79.1%–89.9%) of the occasions. During the five observation sessions, the observed HH adherence increased from 24% to 76% while the EMS measurements immediately before the arrival of the observer remained constant for all sessions.

Conclusion:

The 84% agreement between the EMS and the manual observation suggest a high level of precision for the evaluated system. The Hawthorne effect (higher rate of HH performance) was clearly seen in the increase by a factor of three in the manually observed adherence from session to session as the health workers became more aware of them being observed. The EMS was able to avoid the Hawthorne effect when the observer was not present.

Introduction

Healthcare-associated infections are the most common adverse events affecting inpatients (World Health Organization, 2011). Hand hygiene (HH) is the most important preventative measure (World Health Organization, 2009). Numerous strategies are used to promote HH but adherence to HH protocols is suboptimal and difficult to sustain (Gould et al., 2017a). Routine HH audits are undertaken in many countries as part of quality assurance, usually by directly observing practice. This approach is subject to bias (Jeanes et al., 2019). Audit periods are usually brief (15–20 min), giving an overview of practice relating to the behaviour of a few staff at a single point in time and space. This results in sampling and selection bias (Jeanes et al., 2019). Auditors may fail to document HH opportunities and events through poor vantage or because clinical activities take place behind bedside curtains (Gould et al., 2017b). Health workers are likely to perform HH more often if they know they are watched, resulting in a transient Hawthorne (increased productivity) effect (McCambridge et al., 2014). It has also been reported that health workers avoid performing complex procedures while being observed, thus affecting patient care (Gould et al., 2017b).

Electronic monitoring systems (EMSs) are an alternative to manually documenting HH (Ward et al., 2014). They record HH events constantly in real time, apply consistent algorithms to monitor adherence and, in theory, should overcome the Hawthorne effect and other sources of bias (Srigley et al., 2014). Increase in adherence to HH protocols demonstrated through the use of EMSs has been associated with a decline in methycillin-resistant Staphylococcus aureus infection (Kelly et al., 2016). Numerous EMSs have been developed, and a rapid recent technological development makes it relevant to speak about several types of EMSs (Box 1 and Figure 1).

Box 1. Defining the various types of electronic monitoring systems (EMS).

Five types of EMS

Generally, an EMS has two functions: (1) measure adherence; and (2) present data to healthcare workers. The focus of the below structure is the measurement function, since this will lay the foundation of the accuracy with which the system can operate. The definition of Adherence is Usage of disinfectant divided by Opportunities when disinfectant should have been used. All EMSs measure Usage of disinfectant by means of connected dispensers or other means to detect dispenser usage. Therefore, the EMSs differ mainly in the way they measure Opportunities for HH, summarized in Figure 1.

The first-type systems measure consumption as a proxy for adherence. Opportunities are not measured, but a predefined optimal consumption rate, modified by ward-specific parameters, can be used in place of measured opportunities to arrive at a theoretical adherence rate.

The second-type systems introduced a passage-counter measuring the number of people passing certain strategic points in the ward as an indicator of the Opportunities; the more people passing the more opportunities for HH.

Neither the first nor the second-type systems can distinguish between health workers, patients or visitors. The adherence is, therefore, not indicative only of the health workers but rather of all people visiting the ward.

The third-type systems introduce badges or other means to identify individuals, especially health workers, as well as ways to detect movement in and out of rooms to detect wash-in/wash-out opportunities. The systems cannot detect movement between beds in the same room neither repeated need for HH within the same room.

The fourth-type systems add granularity to detect movement also between beds by placing beacons on or near beds to generate bed-centric zones, typically round or oval in shape since they originate from one beacon. The fourth-type systems can detect presence within these zones but not movement within the zones, neither detect movements outside of the various zones.

The fifth-type systems utilize high-resolution indoor positioning technology that enables continuous detection of movement with arm-length precision throughout the applicable areas of the ward. This allows the system flexibility to adapt to more complex work patterns within patient rooms and throughout the ward as, for example, in the dirty utilities room and other such areas. It also allows for flexibility to define zones in any shape.

Figure 1.

Showing typical examples of the EMSs from type 1 (top left) to type 5 (bottom), with the dotted red lines indicating sensors (type 2) or zones (types 3–5) for HH. Note the dotted line in the type 5 image, indicating detailed movement of the HCW. In types 3 and 4, the HCW appears/disappears in the various zones and no detailed movement measurement is possible. EMS, electronic monitoring system; HCW, healthcare worker; HH, hand hygiene.

Although previous types of EMSs have been evaluated, approaches vary and, perhaps more significantly, the various types differ in function and sophistication from each other, making it difficult to compare functionality and results (Ward et al 2014, Srigley et al 2015). The objective of the present prospective cohort study was to undertake a comprehensive evaluation of the accuracy of a previously unexamined fifth type EMS, its ability to overcome the Hawthorne effect (McCambridge, Witton, and Elbourne 2014) and efficiency compared to manual audit.

Methods

Study site

Data were collected in a busy acute 31-bed medical ward in a National Health Service trust in London, United Kingdom serving a local population of approximately 600,000 people and offering a wide range of secondary and tertiary care services. Patient throughput was high. The ward was equipped with three single rooms and seven bays each with four beds. There are 39 ward-based staff and 15 other health workers attached to the ward (doctors, pharmacist, housekeepers). HH is audited manually by observation every month. All permanent ward-based and ward-attached health workers were eligible to participate in the study.

The fifth type of electronic hand hygiene monitoring system

The Tork Vision Hand Hygiene System (Essity AB, Sweden) is a fifth-type EMS. It incorporates antennae mounted in the ward ceiling, tags worn by health workers and Internet-connected alcohol hand-rub dispensers placed at every bed space and in the ward corridor. The antennae and tags document health workers’ movements with high (< 0.5m) precision in real time, their usage of disinfectants and measure adherence to the World Health Organization’s Moments 1, 4 and 5 to match the moments the EMS can measure. Tags were worn by all groups of staff who were working on the ward and those who came to the ward on a regular basis (e.g. physiotherapists, pharmacists).

A HH opportunity is defined when a health worker wearing a tag is present in the patient zone for at least 10 s. Visits < 10 s in which the health worker is considered by the EMS as less likely to touch the patient or surfaces in the zone are filtered out. In this type of EMS, we were evaluating the patient zone as a rectangular space around the bed with an arm-length distance to the bed. The advantage of this arrangement is that outside this space the health worker cannot reach the patient or the bed.

The EMS records usage of alcohol hand-rub when a health worker is within an arm-length distance from a dispenser that is activated. An adherent HH event is recorded when the health worker has cleansed hands with alcohol hand-rub on entering the patient zone, moves between or leaves the patient zone (Figure 2). Data are analysed and stored in a cloud-based information technology infrastructure. It is possible to document adherence for individual health workers and to aggregate data for clinical teams, wards or the entire organisation. Individual health workers are identified by their tag but aggregated data are always anonymised. The EMS relays individual data to the health worker via email and/or SMS/text messages and aggregated data for the team as a whole via a screen placed in the ward.

Figure 2.

The Tork Vision Hand Hygiene system, Essity Hygiene and Health AB, measures adherence to hand hygiene before, between and after movements of health worker in relation to patient beds.

Study design

The present prospective cohort study explores the accuracy of the fifth-type EMS, its ability to overcome the Hawthorne effect and efficiency compared to manual audit. Data collection with the EMS took place from March 2018 to June 2019. Manual observation was taken as the ‘gold standard’ measure of HH adherence in accordance with international guidelines (World Health Organization, 2009). It was conducted for five sessions (mornings, afternoons and evenings during weekdays, over a weekend and at night) by a trained auditor previously not well-known to the staff at times of usual ward activity between December 2018 and January 2019. No other HH intervention was in progress except routine institutional audits. Manual observation took place in one or two four-bedded bays, but the vantage of the auditor changed during each audit period. Manual observations were collected and compared to the simultaneously collected EMS data (Figure 3).

Figure 3.

The protocol used during manual observation showing time, bed number, information on professional group and what was touched for each manually observed event. The columns ‘HH performed’ indicate if HH was performed correctly (‘yes’) or incorrectly (‘no’) during the event before going to the first bed, between beds or after leaving the last bed according to the manual observation (Figure 2). The colours were added later when comparing each manually observed event with the corresponding EMS observation. The meaning of the colours is the same as is used in Table 1. EMS, electronic monitoring system; HH, hand hygiene.

Ethical clearance and participation

Approval for the study was given by the ethics committee of the university hosting the work. Participation in the evaluation was voluntary. Those who did not want to take part could have declined to wear tags. There were no refusals, however.

Results

Five periods of manual observation took place over 21 h, resulting in the documentation of 294 HH events (Table 1). There was concordance between HH adherence assessed by the manual observer and the EMS on 84% of the occasions (n = 128/153, 95% confidence interval [CI] = 79.1–89.9).

Table 1.

The distribution of the 294 observations made during the five observational sessions and how many were adherent and non-adherent by the respective observational method.

Electronic system	Observer adherent	Observer not adherent

Green indicates that the EMS and manual observer agrees, Red that they disagree and Yellow that the event was observed manually but not by the EMS.

A total of 153 events were observed manually and by the EMS (52.1%). Of the 72 EMS-assessed adherent events, the auditor agreed in 70 cases, resulting in a true-positive rate of 97% (one-sided 95% CI = 95.8–100). Of the 81 EMS-assessed non-adherent events, the observer agreed in 58 cases, resulting in a true-negative rate of 72% (95% CI = 62.2–81.8).

Mean adherence per audit session for the 153 individual events is shown in Figure 4. The EMS and manually audited values are plotted on the y-axis and x-axis, respectively. If the two methods of assessment had yielded identical outcomes, a diagonal line (y=x) would have resulted. In view of the differences between the two assessment methods, an adjustment factor was introduced to translate EMS-assessed data into the manually observed domain. The adjustment factor was estimated using direct comparison to 1.29 (95% CI using Monte Carlo simulation = 1.05–1.61). The result indicated that an EMS-assessed value of 10% adherence translated into an auditor-measured value of 12.9%.

Figure 4.

Showing the average adherence rate per observational session for the observer (x-axis) versus the EMS (y-axis). The blue line represents the original values being below the dotted reference y=x line. The orange line includes the amplification factor 1.29, calibrating the two systems.

A total of 141 HH events documented manually (47.9%) were not captured by the EMS. Inspection of the data indicated that either the health worker had occupied the patient zone for < 10 s or was not wearing a tag, the latter reflecting the relative high proportion of transient staff (students, bank and agency staff, junior doctors undertaking six-month contracts) on the ward.

Hawthorne effect

Auditor-assessed HH adherence was 24% during the first period of manual observation, increasing to 44%, 57%, 68% and 76% for each successive audit session. Increment followed a logistic regression curve as a function of the audit session number and is statistically significant (odds ratio [OR] = 1.75, P < 0.0005) (Figure 5). Comparison between the auditor-assessed data and the EMS-assessed for times when the observer were present at the ward is shown in Figure 4.

Figure 5.

Showing the average observer and EMS measured adherence. The EMS rates are measured 3 h before the arrival of the observer for each of the observation sessions. It also shows the logistic regression curve.

As a baseline for each observational session, the EMS recorded data for the 3 h before each session was used. This baseline adherence was constant with a mean adherence of 10.3% for all sessions (Figure 5) (n = 1232, range per individual observation session = 9–11%, 95% CI = 10.3% ± 1.7). Using the correction factor, adjusted baseline adherence assessed by the EMS was 13.3% (n = 1232, range per individual observation session = 12–14%, 95% CI = 13.3% ± 1.9).

Efficiency

Five sessions of manual observation (21 h) resulted in the documentation of 294 auditor-assessed HH opportunities for 4–8 patients per session, giving a mean of 14 HH events/h. This included all manually observed HH events throughout the ward including nurses not wearing a tag and hence not documented by the EMS. Throughout the sessions of manual observation, the EMS documented 2623 HH opportunities over 36 h for all 31 patients on the ward, giving a mean observation of 73 HH events/h for the subset of nurses wearing a tag.

Discussion

The present study appears to be the first evaluation of what we have described as a fifth-type EMS. Agreement between HH adherence data collected by the two audit methods was high (84%). The EMS can therefore be considered to demonstrate a high level of precision compared to manual observation which traditionally has been accepted as the gold standard HH audit method (World Health Organization, 2009). Concordance between EMSs and manual audit has varied in previous studies. In some evaluations level of concordance had been considered acceptable (Sharma et al., 2012) but not in others (Hagel et al., 2015; Morgan et al., 2012). A number of limitations to the EMS in terms of accuracy emerged, however. Underestimation of adherence could occur if the health workers occupy the patient zone for > 10 s without initiating patient contact. This situation would arise relatively seldom as most visits within the patient zone for > 10 s would involve touching either the patient or his/her surroundings. Likewise, all procedures < 10 s are omitted, also affecting the data. However, the shorter the visit the less likely it is to involve touching. The 10 s is adjustable in the system and was arrived at as a suitable threshold in discussion with the ward and authors of the present study. Finally, ambiguous signals (‘noise’) in the electronic data can occur if health workers stand at the periphery of the zone to hold a conversation or monitor the patient’s condition without initiating physical contact.

It is also possible that manual audit might impact on adherence compared to the EMS in view of the well-known limitations associated with direct observation of HH. Manual audit is a complex activity requiring considerable expertise, constant attention and often rapid decision-making. Errors can result through poor vantage or the fast pace of work in the clinical environment (Gould et al., 2007), especially when, as in this study, more than one health worker is observed simultaneously (FitzGerald et al., 2013). In one study where adherence was lower when documented manually than by EMS, the difference was attributed to poor vantage of the auditors and inability to record HH accurately when health workers were busy (Hagel et al., 2015).

The Hawthorne effect has been identified as a major bias in studies designed to establish the effectiveness of interventions to promote HH (Fuller et al., 2012; Martín-Madrazo et al., 2009; Stewardson et al., 2016). Previous studies have suggested that EMSs can overcome this source of bias (Ellingson et al., 2011; Srigley et al., 2014).

The present study supports these findings and measures the Hawthorne effect to be increased by a factor of three, from the initial 24% adherence when the observer was unknown to the health workers to 76% during the final observation session when the observer had become more familiar to the staff. The statistically significant increase of adherence as a function of the observational session number further supports this finding. At the same time, the EMS-measured values during the 3 h preceding the arrival of the auditor remained constant at approximately 9%–11% throughout the five observational sessions.

These findings indicate that adherence values derived from manual observation may be subject to a Hawthorne effect that overestimates adherence. The EMS data mirrored this increase when the observer was present (Figure 4) but did not show a similar effect when the observer was not present on the ward (Figure 5). From this, we conclude that the EMS prevented the Hawthorne effect and its use enabled us to quantify it.

The initial level of adherence (24%) documented by direct observation was close to figures quoted by other authors during observational studies where observation has been covert (Kingston et al., 2016). They were also close to the values registered by the EMS before the arrival of the auditor. The final levels of adherence (76%) matched rates often reported after routine institutional audits where the auditor is familiar to staff (Scherer et al., 2019). The same high levels are also reported in studies after intense efforts have been made to increase adherence by making health workers aware that audit is in progress (Kohli et al., 2009). The very high levels of adherence frequently documented during routine, overt manual HH audit are very likely to be inflated by the Hawthorne effect (Scherer et al., 2019). The levels of adherence registered for the ward by the EMS are more likely to reflect ‘real-world’ adherence levels than to represent unusually low levels compared to those reported from other organisations.

The EMS obtained five times more data than manual observation over the same period. Since approximately half of the healthcare workers were tagged permanent staff, the efficiency of the EMS is estimated to be tenfold compared to manual monitoring. Less effort is required than with manual observation which is laborious, expensive in terms of manpower and subject to bias, delivering only a ‘snapshot’ of HH performance over a short time frame at only one ward location (Gould et al., 2007; Srigley et al., 2014). Electronic monitoring overcomes these problems because data are generated continuously and objectively throughout the ward and for all staff, using the same evaluation criteria for everybody at all times. It is therefore possible to identify trends in HH behaviour over time that would not be meaningful with the small and potentially biased datasets generated by manual audit.

The 84% agreement between the EMS and manual observation suggest a high level of precision for the evaluated system. The Hawthorne effect was clearly seen in the increase by a factor three in the manually observed adherence from session to session as the health workers became more aware of being observed. The EMS was able to avoid the Hawthorne effect when the observer was not present.

Study limitations

Data collection took place on a single ward in one organisation and transferability of the findings is unknown. The ward was typical for the NHS in terms of acuity, patient throughput, staffing and workload, however. Conclusions about the precision of the EMS and Hawthorne effect were based on five episodes of direct observation only. Manual audit ceased because a large number of HH opportunities and events had been documented by both methods. Finally the EMS we evaluated documents only Moments 1, 4 and 5. Difficulties monitoring all five moments have been noted by previous authors using manual methods because of the rapid pace of work in the clinical environment (FitzGerald et al., 2013) and because bedside curtains obscure clinical events (Chung et al., 2016). Boyce (2017) has recently argued that under these circumstances it is acceptable to monitor Moments 1, 4 and 5 and that when using an EMS, the amount of time saved by collecting data manually can be used to good advantage identifying barriers to HH, finding solutions and coaching health workers.