Reactivity in Rapidly Collected Hygiene and Toilet Spot Check Measurements: A Cautionary Note for Longitudinal Studies

Abstract

Discreet collection of spot check observations to measure household hygiene conditions is a common measurement technique in epidemiologic studies of hygiene in low-income countries. The objective of this study was to determine whether the collection of spot check observations in longitudinal studies could itself induce reactivity (i.e., change participant behavior). We analyzed data from a 12-month prospective cohort study in rural Tamil Nadu, India that was conducted in the absence of any hygiene or toilet promotion activities. Our data included hygiene and toilet spot checks from 10,427 household visits. We found substantial evidence of participant reactivity to spot check observations of hygiene practices that were easy to modify on short notice. For example, soap observed at the household's primary handwashing location increased from 49% at enrollment to 81% by the fourth visit and remained at or above 77% for the remainder of the study.

The measurement of hygiene and toilet use behavior is notoriously difficult because of potential courtesy bias—participants overreporting behaviors that they perceive as desirable—and the variability in handwashing practices within a household.¹ Investigators frequently conduct intensive measurement of household behavior using 3- to 5-hour-long structured observations that collect rich information about household hygiene and defecation practices. The presence of a structured observer in the home, however, can increase the frequency of handwashing or toilet use, leading to upwardly biased measurements.^2,3 A potentially lower impact measurement approach that is also widely used in epidemiologic studies is collection of rapid, discreet spot check measurements of hygienic conditions during unannounced household visits. The extent to which such rapid spot checks could also lead to reactivity among participants is unknown. Participant reactivity that results from study measurements can constitute an important source of bias and thus, is an important consideration for longitudinal studies.⁴ The objective of this study was to examine longitudinal patterns of hygiene and toilet spot check measurements in rural Tamil Nadu, India to determine whether the measurements exhibited patterns suggestive of participant reactivity. We hypothesized that the indicators would be relatively stable in the study population over the study period, because there was no hygiene promotion intervention during that time.

We enrolled 900 households from 25 intervention and control villages in rural Tamil Nadu, India for a 12-month prospective cohort study to measure the long-term effects of a previously delivered sanitation, water supply, and hygiene promotion intervention; we have previously reported study details.^5,6 In brief, the field study took place between 5 and 47 months after the conclusion of intervention activities. Intervention activities included hygiene promotion through village-wide education campaigns and school health clubs, promotion of toilet construction, water supply distribution network upgrades, and microcredit loans to help fund private toilet and water tap construction (details in ref. 5). Data collection spanned from January of 2008 to April of 2009. All households provided informed consent at enrollment, and during that process, field staff explained that the objective of the study was to measure whether household environmental conditions were associated with child health. Field staff visited households monthly for 1 year to collect information about child illness. After asking permission, field staff also collected discreet spot check observations; each monthly visit lasted approximately 10 minutes. Spot check observations included observations of the household toilet if present (water present, soap present, and feces present on the slab), observations of the household compound environment (presence of human or animal feces and fecal odors), observation of the primary location—determined by participant report—used to wash hands (soap present and water present), and hand inspections of study children < 5 years old (palms and finger pads free of soil or mud and fingernails free of soil or mud). Field staff asked participants to identify their primary handwashing location at each visit, because it was possible for the location to change places within the compound over the course of the study. To summarize the longitudinal patterns of spot check observations, we calculated the study population mean for each spot check measurement by survey round and calculated non-parametric percentile bootstrapped confidence intervals for the means by resampling study households with replacement.

Field staff collected handwashing location spot check observations in 10,427 household visits and toilet spot check observations in 3,693 household visits. Toilet inspections only took place in 374 households with private toilets. Because of a communication error with the survey team, toilet spot checks were not included in survey rounds 2 and 3 but were added back in round 4. We have not summarized the longitudinal pattern of feces observed on the toilet slab, because it was < 2.6% in all rounds. Most spot check measurements were stable over the study period and showed no clear relationship with time in the study (Figure 1 ). For example, over the 12 survey rounds, soap was observed in 46–56% of toilet inspections (Figure 1A), and human or animal feces were observed in the living area outside the house in 22–35% of visits (Figure 1C). In contrast, soap observed at the household's primary handwashing location increased from 49% at enrollment to 81% by the fourth visit and remained at or above 77% for the remainder of the study (Figure 2A ). Children < 5 years old with hands free of visible dirt increased from 79% at enrollment to 95% by the fourth visit and remained at or above 91% for the remainder of the study (Figure 2B).

Figure 1.

Six toilet and hygiene rapid spot check indicators that showed little reactivity over 12 monthly longitudinal survey rounds between 2008 and 2009 in rural Tamil Nadu, India. (A) Soap present at the toilet. (B) Water present at the toilet. (C) Human or animal feces observed in the living area of the house. (D) Staff could smell feces during the interview. (E) Water present at the primary handwashing location. (F) Child's fingernails free of soil/mud.

Figure 2.

Two hygiene rapid spot check indicators that showed evidence of reactivity over 12 monthly longitudinal survey rounds between 2008 and 2009 in rural Tamil Nadu, India. (A) Soap present at the primary handwashing location. (B) Child's palms and finger pads free of soil/mud.

This analysis suggests that, even in the absence of intervention or promotion activities, household survey visits can result in changes in the presence of soap at a handwashing location and apparent hand cleanliness among children < 5 years old. Seasonal variation cannot explain the longitudinal changes to new equilibriums observed in Figure 2, because the data collection spanned more than 12 months. Toilet observations and other hygiene indicators that the field staff collected did not show signs of participant reactivity in this population (Figure 1).

An important question raised by our findings is whether the measured changes in soap present at the handwashing location and child hand cleanliness reflect actual changes in hygiene behavior (a “Hawthorne effect”⁷) or simply transient changes in household conditions in anticipation of field staff visits. The field team arrived in study villages on different days each month for unannounced visits, and they divided into multiple geographic areas over each village to reduce the chance that participants would anticipate their arrival. We do note, however, that both of the observations that showed reactivity were conditions that could be relatively easy to modify on short notice. For example, cleaning a child's palms and finger pads is much easier than cleaning the soil or mud from under the fingernails, and whereas we saw a large increase in the percentage of children with clean palms or finger pads over the study period, we did not observe a similar increase in the percentage of children with clean fingernails. We did not record the time of day that the field team visited each household, and therefore, we were unable to stratify the analysis by the order in which households were visited on each day to formally test the hypothesis that households visited later were more reactive than those visited earlier.

We were able to conduct an additional analysis to examine whether the observed changes in hygiene behavior because of study participation led to reduced diarrhea or acute respiratory illness prevalence among households that exhibited reactivity. If some households were consistently washing their hands more with soap, we would expect to see lower prevalence of child diarrhea and respiratory illness compared with households that did not improve their hygiene conditions.^8,9 We excluded 149 households that always had soap present at the handwashing location and 6 households that never had soap present. For the remaining 745 households, we defined new adopter households as those that had soap at their handwashing location for greater than six of the rounds that followed round 4 (during the new equilibrium) (Figure 2A) and inconsistent users as those that did not meet this definition. New adopter households were slightly more likely to be located in intervention (68%) than control (62%) villages (P value = 0.11 for comparison). We considered the following additional possible confounding covariates in the analysis: quintiles of asset-based wealth index,⁵ caregiver education, kitchen located inside the home, and scheduled caste status. Of these covariates, all were positively associated with being a new adopter versus inconsistent user, except for scheduled caste status, which was negatively associated with being a new adopter (P value < 0.05 for all comparisons). Diarrhea prevalence among children < 5 years old was not statistically different between new adopters (1.8%) and inconsistent users (1.7%; P value for difference = 0.75), which suggests that the increases in observed soap at the handwashing location did not lead to reductions in child diarrhea risk. However, new adopters had lower acute respiratory illness prevalence (8.4% of new adopters versus 10.1% of inconsistent users; P value for difference = 0.05), even after adjusting for wealth index,⁵ caregiver education, scheduled caste status, kitchen location (inside versus outside house), and intervention village membership in a multivariable regression (adjusted difference = −1.9%; P = 0.04). This additional observational analysis lends mixed evidence to suggest that changes observed by spot check observations may reflect true “Hawthorne effects” through habitual changes in behavior.

Future studies that use spot check hygiene measurements should plan for the possibility of participant reactivity in their design and measurement approach. Participant reactivity that results from the measurement of study outcomes can constitute an important source of bias for epidemiologic studies.⁴ In this study, field staff opportunistically collected spot check measurements alongside the primary outcome (diarrhea measurements) at every household visit; the extremely narrow confidence intervals on most of the hygiene and toilet indicators (Figures 1 and 2) suggest that we could have collected this information from a subsample. If spot check measurements were collected from a rotating subsample of the population, then it would decrease the frequency of measurement in any single household and potentially reduce the chance for reactivity. The two hygiene spot check observations that exhibited reactivity to monthly visits in this study were those that were relatively easy to modify and observations that study participants could readily perceive. Field staff asked about the primary location used to wash hands, which provided a cue to participants that they were interested in handwashing. Field staff also asked to observe the hands of children, which also enabled participants to discern an interest in hand cleanliness. Spot check measurements of conditions that are more difficult to quickly modify would be less likely to exhibit this bias in longitudinal studies. Additionally, spot check measurements that can be collected discreetly without the awareness of participants would be less susceptible to reactivity.