The precision of human-generated-hand-hygiene observations: a comparison of human observation with an automated monitoring system

Abstract

We compared nearly 1400 hand-hygiene-related events observed by an automated system and by human observations. The records differed for 38% of these events. Two likely explanations for the inconsistencies were the distance between the observer and the event and the busyness of the clinic.

Hand-hygiene adherence is measured almost exclusively via direct human observation. While considered the ”gold standard” [1], direct observation is susceptible to observer biases [1, 2], and the reliability is affected by sporadic or inconsistent sampling [1]. Several emerging technologies [3-8] to measure hand-hygiene adherence offer alternatives to human observation. Until these become widely adopted, hand-hygiene adherence will likely continue to be measured by human observers. Thus, a better understanding of the limitations of human observations is needed. This paper demonstrates how the accuracy of human observers is affected by activity in the observed area and by the distance between the observer and hand-hygiene events.

METHODS

The experiment was conducted on three consecutive Tuesdays in an outpatient clinic located in a single hallway. Hand-hygiene dispensers mounted outside the examination room doors were replaced with a dispensing mechanism modified to broadcast a radio transmission each time the dispenser was used. Each doorway was outfitted with an instrument that recorded the time when an infrared beam was interrupted as an indicator that someone had crossed the threshold. This instrument also received and recorded radio broadcasts from the dispensers. The dispensing mechanism and infrared beam instruments were each custom built, equipped with a microprocessor, materials for wireless communication and flash memory.

A human observer sat unobtrusively at the end of the clinic hallway. One observer worked each morning, the other each afternoon. The observer recorded every time a person entered or exited each room, defined as a threshold event. She also recorded whether or not the person used a hand-sanitizer dispenser when crossing the threshold. Observations were coded as: a) no wash in; b) no wash out; d) wash in; d) wash out; e) wash in the hallway without entering or exiting a room. On the first day, each of the two observers collected this information for every event over a shift of approximately four hours. On the second Tuesday, each observer recorded the same information, but this time for only one hour during the four-hour span. On the third observation day the clinic was only active during the morning and only three rooms were being used. On this day the observer was positioned close to the active rooms and recorded the same information for those three rooms.

We compared the human observations from the first and second days to the machine records from the same time period. The records from the infrared receivers were synchronized with the data from the human observers. A timeline of the automated record was then produced and compared to a timeline of the human observations for each threshold-crossing- and hand-sanitizing event. Inconsistencies between the records were coded as: 1) missed hand-hygiene event, 2) missed threshold event, 3) false hand-hygiene event, 4) false threshold event, and 5) incorrect room. A missed hand-hygiene event or missed threshold event occurred when the written record did not include an event perceived by the electronic system. A false hand-hygiene event or false threshold event occurred when a hand-hygiene event or threshold crossing appeared in the written record but not in the electronic record. An incorrect room event occurred when the room number in the written record corresponded most closely to a threshold event with a different room number in the electronic record.

Each inconsistency was then coded with four factors that the team believed might affect the reliability of human observations. These four factors were: observer ID, number of minutes observing (to control for observer fatigue), distance between the observer and the room at which the inconsistency occurred (to control for range of view), and clinic activity (to control for degree of hallway traffic). Next, we used logistic regression to determine which factors were associated with the occurrence of an inconsistency. A similar analysis was applied to hand-hygiene-adherence data from the third day of the experiment (excluding ID of observer). Finally, three types of errors (incorrect room, hand-hygiene events, and threshold events) were again modeled using logistic regression.

All statistical analyses were performed using R, version 2.12.2 (R Foundation).

RESULTS

Nearly 1400 unique threshold events were recorded by both the human observers and the equipment. Approximately 62% of these events were consistent between the two systems. Table 1 provides the tally and proportion of each inconsistency. Clinic activity and distance were significant predictors of inconsistencies, while the user and minutes observing were not. Table 2 describes the resulting model fit.

Table 1.

Counts and percentages of human errors.

Error Type	Day 1 Count	Percent	Day 2 Count	Percent	Day 3 Count	Percent	Total Count	Percent
False Threshold Event	23	2.39	13	4.51	10	6.80	46	3.29
False Hand Hygiene Event	5	0.52	2	0.69	0	0.00	7	0.50
Missed Threshold Event	284	29.52	41	14.24	21	14.29	346	24.77
Missed Hand Hygiene Event	39	4.05	1	0.35	4	2.72	44	3.15
Incorrect Room	66	6.86	14	4.86	2	1.36	82	5.87
No Error	545	56.65	217	75.35	110	74.83	872	62.42
Total	962	100.0	288	100.00	147	100.00	1397	100.00

Table 2.

Coefficient estimates, standard errors, and p-values from the logistic regression model.

Coefficient	Estimate	Standard Error	Odds Ratio	95% CI for Odds Ratio	Z Value	P Value
Intercept	-1.197	0.133	3.310	2.551, 4.296	-8.964	<0.001
Clinic Activity*	0.021	0.009	1.021	1.003, 1.039	2.455	0.014
Distance	1.309	0.132	3.702	2.858, 4.796	9.880	<0.001

^*The clinic activity level is the number of events occurring within the current and three preceding minutes.

Holding distance constant, the odds of an error are 1.11 (1.02 - 1.21 95% CI) times greater for every additional five events observed within a 3-minute-time span. Likewise, holding clinic activity constant, we find that the odds of an error in the far rooms are 3.70 (2.86 - 4.80 95% CI) times the odds of an error in the closer rooms.

When specifically modeling threshold-event errors (errors related to room entry or exit), we found a significant association with distance (2.68 OR; 2.06 - 3.50 95% CI). Distance was also found to be associated with hand-hygiene-event errors (2.02 OR; 1.12 - 3.62 95% CI). The wrong-room-assignment errors were found to be significantly associated with both the user (3.58 OR; 2.00 – 6.38 95% CI) and distance (3.20 OR; 1.99 – 5.14, 95% CI).

DISCUSSION

The frequency of inconsistencies (38%) between the human observer and the automated system was surprising. After eliminating the possibility of systematic equipment errors, we found that the most likely explanation for the inconsistencies was the distance between the observer and the observed event and the clinic activity level.

Existing guidance for human observation of hand-hygiene compliance often recommends increasing the number of observations collected to improve data accuracy [10]. However, this assumes that observations are all equal with respect to their accuracy. Our results show that errors are dependent upon the circumstances of the observations. The lower accuracy noted with high clinic traffic likely correlates with missed events because of near-simultaneous threshold crossings in multiple rooms.

Our study has several limitations. First, it was performed in a single medical center. Second, our statistical analysis treated all of the inconsistencies as equally important. Thus one type of error may have excessive influence on our results. For example, if we removed from our model missed machine threshold events, the busyness covariate is no longer significant (p = .14). However, distance was still strongly associated with all of the remaining types of errors. Furthermore, the data did not allow for subject-level effects such as the individual nurses and doctors triggering the events. Finally, our results assume that the hardware was the “gold standard” and all errors were the fault of the human observers, which may be inaccurate. For example, if two people entered a room in rapid succession, an observer may have counted two entries with the equipment only counting one.

Despite our limitations, we demonstrate that the accuracy of an observer may depend upon when and where observers are asked to audit. By increasing the number of observations through more-frequent events or over a longer distance, accuracy may suffer. The degree to which our results apply to other settings warrants further investigation. When planning observations, it may be beneficial to limit the number of observations during a specific time period and limit the distance between observers and the healthcare workers under observation.