A standardised approach to validate both the accuracy and precision of electronic hand hygiene monitoring systems is needed

Dear Editor,

We read with interest the study by Gould et al. (2020) evaluating the precision of a specific electronic hand hygiene monitoring system (EMS): the Tork Vision Hand Hygiene System (TVHHS).

EMS are an innovative solution to the perennial problem of accurately measuring compliance with hand hygiene. However, without robust validation data in the clinical environment, infection prevention and control (IPC) teams will be reluctant to procure such systems. A 2014 systematic review of EMS (Ward et al., 2014) found that of 42 articles analysed, only 20% provided a methodology for demonstrating the accuracy of EMS. Furthermore, for those that did, there was a lack of a consistent and robust methodological approach.

Limper et al. (2016) recommended the use of simple epidemiological statistics to validate the accuracy of EMS technology, as per validating any given clinical intervention or laboratory test. They specifically suggest that sensitivity and positive predictive value (PPV) are key metrics which should be measured. Using such a standardized approach to evaluate the accuracy of EMS will allow for valid comparisons to be made against direct observation (DO), as well as between different EMS manufacturers.

A measurement system, such as an EMS, is considered valid if it is both accurate and precise. Accuracy of a test or intervention relates to how close a test measurement (e.g. EMS data) is to the true test result (i.e. DO). Precision, however, refers to how close replicate values are to each other.

Gould et al. (2020) found that in 84% of HH observations, there was agreement (i.e. concordance) between EMS and DO. They concluded that this was an acceptable level of precision for the EMS. However, they have not commented upon the accuracy of the TVHHS. To investigate this further, we searched for any accuracy data previously published by the manufacturer. Unfortunately, none was available from their website. We therefore decided to use the epidemiological statistics recommended by Limper et al. (2016) to evaluate the accuracy of the TVHHS. The study data on HH events provided by Gould et al. gave absolute numbers of HH events deemed as adherent or not to protocol by both the EMS and DO. This information was then used to determine the sensitivity (70/93, 75%), specificity (58/60, 97%), PPV (70/72, 97%) and negative predictive value (NPV) (58/81, 72%) for the TVHHS. While the PPV is high, sensitivity is low. Put another way, the EMS was not accurately able to detect cases where staff were adherent with HH protocol. The TVHHS also had a tendency to overly detect poor adherence, generating a lot of negative values, as demonstrated by the NPV. This will potentially antagonise staff and could create distrust of the system.

In another approach to validate an EMS, Limper et al. (2017) assessed the accuracy of an EMS manufactured by GOJO Industries. The accuracy of the EMS was established during the behavioural validation phase in a working ward; sensitivity was 92.75% and PPV 84.4%. Different approaches to validate EMS suggest there needs to be a consensus as to how this is measured and what level of accuracy is acceptable to healthcare workers, IPC teams and procurement managers.

In the absence of the Hawthorne effect, the HH compliance rate was 10.3% when measured using TVHHS (Gould et al., 2020). This is clearly concerning. Previously, it has been documented that compliance with HH is in the region of 45% when measured covertly (Pan et al., 2013). The low sensitivity of the TVHHS makes this 10.3% figure questionable. Despite this, it is likely that the true rate of compliance with HH could be lower than previously anticipated. Gould et al’s findings provide a compelling argument for the use of EMS to give a more representative and realistic picture of HH compliance across healthcare organisations.

When discussing EMS accuracy, the fallibility of measuring against DO as the best available standard cannot be ignored. Although DO remains the World Health Organization’s gold standard, it is widely recognised that DO itself is an inaccurate measurement of HH compliance. As discussed by Gould et al. (2020), DO is subject to bias such as the Hawthorne effect, sampling and selection bias. It is also well recognised that DO captures only a small fraction of total HH events.

The proposed EMS classification system by Gould et al. (2020) is an interesting and helpful addition. However, there are some general issues with EMS that need to be commented upon. Type 3–5 EMS all track staff movement within the healthcare environment. These EMS require a change in staff workflow through the need to wear an additional badge (or other such tracking equipment). Such change in practice can be a barrier to adoption. In the study by Gould et al., 141 (47.9%) of HH events were not captured by EMS, which the authors largely attributed to the presence of transient staff, such as students and agency workers. However, an additional explanation is that the staff study cohort may not have been wearing their badge simply due to the extra effort that it required of them. Overall, the study by Gould et al. demonstrates how challenging it is to undertake rigorous EMS validation in a busy clinical environment.

If the problem of poor compliance with HH is to be solved, an accurate method of HH monitoring is surely the first step forward. The IPC community must agree which statistics should be used when evaluating the accuracy of a given EMS. This will allow IPC professionals to make informed decisions regarding their chosen method when clinical validation is undertaken. Limper et al. (2016) made a sensible proposal to use fundamental epidemiological statistics such as sensitivity and PPV. Drawing upon these statistics from Gould et al.’s results allows us to conclude that the TVHHS EMS does not appear to achieve a high sensitivity, and so its accuracy remains a concern.