Measuring User Compliance and Cost Effectiveness of Safe Drinking Water Programs: A Cluster-Randomized Study of Household Ultraviolet Disinfection in Rural Mexico

Abstract.

Low adoption and compliance levels for household water treatment and safe storage (HWTS) technologies have made it challenging for these systems to achieve measurable health benefits in the developing world. User compliance remains an inconsistently defined and poorly understood feature of HWTS programs. In this article, we develop a comprehensive approach to understanding HWTS compliance. First, our Safe Drinking Water Compliance Framework disaggregates and measures the components of compliance from initial adoption of the HWTS to exclusive consumption of treated water. We apply this framework to an ultraviolet (UV)–based safe water system in a cluster-randomized controlled trial in rural Mexico. Second, we evaluate a no-frills (or “Basic”) variant of the program as well as an improved (or “Enhanced”) variant, to test if subtle changes in the user interface of HWTS programs could improve compliance. Finally, we perform a full-cost analysis of both variants to assess their cost effectiveness (CE) in achieving compliance. We define “compliance” strictly as the habit of consuming safe water. We find that compliance was significantly higher in the groups where the UV program variants were rolled out than in the control groups. The Enhanced variant performed better immediately postintervention than the Basic, but compliance (and thus CE) degraded with time such that no effective difference remained between the two versions of the program.

INTRODUCTION

Many households in low- and middle-income countries rely on contaminated drinking water,^1,2 which contributes to widespread gastrointestinal and other illness.^3,4 Household water treatment and safe storage (HWTS) systems can enable users to treat their water at the point of use, making it safe to drink. To meet the diverse needs of underserved communities, the HWTS approach offers a broad set of technological options to lower the economic barriers to access, and decentralize operation and management responsibilities.^5–8

HWTS programs (indeed, any consumer-oriented technology-based programs) need their users to adhere to (or “comply” with) the requirements for correct and consistent use of that particular technology or practice, including maintenance and purchase of consumables or replacement parts. Studies have shown that HWTS can improve health outcomes,^6,9,10 but that low compliance is the norm outside pilot projects and epidemiological trials.^11–14 Quantitative microbiological risk assessment models predict that the health benefits from water quality interventions drop with even occasional consumption of untreated drinking water.^15–17 User compliance is essential for HWTS to achieve its intended health effects, but it remains a major challenge even after decades of HWTS promotion. It must be better understood if safe water programs are to be effective.

Compliance remains unevenly studied and inconsistently defined in the HWTS literature. Many previous studies have estimated compliance based on water quality tests, residual disinfectant levels, reported use, or occasional observation.^6,13,18 More rigorous metrics for assessing compliance are necessary for assessing the household-level drivers of HWTS effectiveness. HWTS interventions usually require behavior changes that are inconvenient for household members, and research has documented the many difficulties of changing practices, preferences, and perceptions.^19–22 Outside of the HWTS sector, research in behavioral economics has shown that small adjustments toward greater user convenience—or the elimination of “small hassles”²³—can lead to significant improvements in program uptake. In this article, we present a comprehensive approach to defining and measuring HWTS compliance, and test if subtle changes in the convenience of an HWTS program can improve compliance.

We present the Safe Drinking Water Compliance Framework, disaggregating compliance into five components—Adoption, Access, Knowledge, Habit, and Exclusive Use. This framework can be applied to any household-based drinking water technology. We apply the framework to an HWTS system that uses ultraviolet (UV) light to disinfect water, and that was delivered to rural communities in Mexico between 2009 and 2011 as part of a safe water program. We extend the framework to the most prevalent safe water practice in these communities before our study: purchased bottled water in 20-L plastic, narrow-necked containers (garrafones). We use the framework to assess the levels of compliance within the UV-based HWTS intervention and to evaluate the strengths and weaknesses of the accompanying outreach program. We evaluate two variants of the program (Basic and Enhanced) to test if modest improvements in the level of convenience could overcome small hassles in this context and lead to significant improvements in compliance. We also carry out a full-cost analysis of each variant to assess whether the costs of the added conveniences are worth the benefits of any additional (correct) usage of the system.

This study was conducted as part of a larger research project measuring the program’s impact on drinking water quality and health,²⁴ as well as the UV technology’s field efficacy and risks associated with posttreatment contamination.²⁵

METHODS

Study area.

Our study took place in 24 rural communities in Baja California Sur, Mexico. The region is hot and dry, with less than 200 mm annual average precipitation. Study communities ranged in size from 8 to 31 households, with each community at least half an hour from its nearest neighbor and located 1 to 4 hours away from larger towns, mostly via unpaved roads. Primary economic activities included livestock ranching, small-scale farming, and fishing. Only 14% of households had electrical grid connections; 80% used small solar panels for lighting and communication. None of the communities had functioning piped water systems. At the beginning of the study, most people relied on untreated water from local springs and shallow wells for drinking water, although some purchased garrafon-bottled water from urban vendors. Those collecting local water stored it in plastic buckets, 200-L barrels, and traditional containers made out of rock or clay (tinajas); water was mostly extracted by dipping a cup into open containers.

Mesita Azul program: Basic and Enhanced.

Fundación Cántaro Azul, a nonprofit organization based in Mexico, collaborated with our research team to develop an HWTS program consisting of a UV disinfection system (Mesita Azul, meaning “little blue table” in Spanish) and a series of outreach activities to support adoption and use of the system. The household system includes a bucket, where source water is poured in, a UV lamp (15 W; 254 nm) to inactivate bacteria, viruses, and protozoa, and a plastic narrow-necked storage container (a 20-L garrafon). The germicidal dose delivered by the system exceeds common HWTS and UV disinfection standards^26,27 to ensure proper disinfection throughout the lifetime of the system. Cántaro Azul refined the Mesita Azul program for these rural households through an iterative process of design, field tests, and user feedback before rolling it out to the study communities.

Cántaro Azul designed the Mesita Azul as an esthetically appealing and user-friendly system for treating water (Figure 1). To operate the system, users have to follow six steps:

• 1.turn on the switch and look through a plastic window to check that the UV lamp is working;
• 2.pour water through a straining cloth into the bucket;
• 3.open the bucket’s valve;
• 4.wait for water to flow by gravity through the disinfection chamber and into the garrafon;
• 5.close the bucket’s valve and open the drain; and
• 6.turn off the switch.

Figure 1.

The Mesita Azul system includes a bucket for source water, the ultraviolet disinfection chamber, a safe storage container (garrafon) for treated water, and instructions for correct use. This figure appears in color at www.ajtmh.org.

The system disinfects 20 L of water in 5 minutes. The garrafon includes a cloth wrap, which, when wetted, can keep stored water cool through evaporation.

The research team developed two variants of the Mesita Azul program: Basic and Enhanced. The Basic program included only those features that were deemed essential to promote the system’s adoption, use, and maintenance. It consisted of four sequential activities: community assessment of water quality, a community presentation on safe water and UV disinfection, installation of UV systems in households that adopted the program, and follow-up visits to homes after installation. During the community assessment, program staff established relationships with key stakeholders, assessed the feasibility of the project (e.g., presence of microbial rather than chemical contamination in water sources, presence of electricity, etc.), and recorded demographic data. During the community presentation, they discussed water and health, water quality in the region, preexisting, available water treatment options, the Mesita Azul technology, and the benefits and requirements of the program. They then launched the enrollment process and trained selected community members to maintain and repair the systems.

Households could enroll in the program if they committed to drinking only disinfected water, made a $20 (=MXN$250) one-time payment or a $24 (=MXN$300) payment over 6 months, and assumed responsibility for maintaining the UV system. We considered these households to have “adopted” the UV system. Most households adopted the Mesita Azul program in each enrolled community. In the third visit to the community, program staff installed the UV system with its safe storage container in each adopting household and trained one household member on how to operate the system. Four to six months after the installation, they carried out a follow-up visit to retrain household members if needed, address any technical problems, and collect any outstanding payments. No system was removed for failure to complete payments. The Enhanced variant of the Mesita Azul program included additional conveniences and guarantees meant to reduce “small hassles”²⁵ to adoption and compliance (Table 1).

Table 1.

Compliance barriers in household water treatment and safe storage systems and the strategies used in two program variants to minimize them

Barriers	Basic program	Enhanced program
Poor information	Regional water quality results	Individual water quality results for each household
Limited economic resources	Six-month system guarantee	Six-month system and money-back guarantee
Low operator capacity	Train one person per household	Train two people per household
Low user convenience	One safe storage container	Two safe storage containers
No safe water habit	Follow-up every 6 months	Follow-up within 1 month and then every 6 months

Safe Drinking Water Compliance Framework.

To assess the potential impact of the Mesita Azul programs, we needed to define and measure household compliance. Compliance is a multipart phenomenon; definitions of compliance range from simple adoption (with no measures of use) to correct and consistent use.⁶ We developed a comprehensive framework to define and operationalize all the components necessary to achieve the health benefits of household-based safe water programs. Unpacking “compliance” into its specific tasks for a specific program can show which components are most challenging for which household types and where strengthening the program will be most effective.

Our Safe Drinking Water Compliance Framework consists of five components: Adoption, Knowledge, Access, Habit, and Exclusive Use. All these components are directly related to procuring and consuming safe water. Adoption is defined as the initial acceptance (and, implicitly therefore, affordability) of a safe water practice or technology; Knowledge is the information necessary to carry out the safe water practice; Access is the possibility of carrying out the practice within the means and resources available in everyday life; Habit is when the safe water practice has become established and dominant; and Exclusive Use is the practice of drinking only safe water within the household. We further disaggregated each component into procurement and consumption. Procuring and consuming safe water are interlinked but they are distinct processes; operating a disinfection system and drinking disinfected water are quite different, for instance, and are carried out at different times and places, by different household members with their own motivations and barriers. The operational definitions of procurement and consumption can be adapted to the characteristics of specific safe water programs.

We adapted the Compliance Framework to the two most prevalent safe water practices observed during the trial-purchased garrafon-bottled water during baseline (Supplemental Table 1, Supplemental Information) and the Mesita Azul programs post-intervention (Table 2). The tables document the operational definitions for each component and the measures we used during household visits to verify compliance or noncompliance. We framed our interview questions (through presurvey piloting) to reduce the likelihood of recall and social desirability bias from self-reported information on the consumption of safe water. Specifically, we asked our interviewees to show us from where they drank their last glass of water; we found this to be the least “leading” way to see if drinking from safe sources was becoming normalized. We did not measure Adoption (consumption) because we expected it to be similar to Adoption (procurement), but it could be relevant where disaggregated measurements across gender or age are needed. We also did not measure Exclusive Use (procurement). Our operational definitions and measures are not an exhaustive list. They are tailored to the specific safe water practices, observable measures, user reports, and the time frame of our study.

Table 2.

Safe Drinking Water Compliance Framework adapted to the Mesita Azul program

Component	Operational definition	Measurement
Adoption
Procurement	Household acquired an ultraviolet (UV) system	Interviewer verified presence of system during visit
Consumption	Not documented	Not applicable
Knowledge
Procurement	Operator knew and carried out UV system operation steps adequately	Interviewer asked operator to treat water and evaluated him/her
Consumption	Interviewee recognized UV-treated water as better quality than untreated water	Interviewee compared water quality before and after intervention
Access
Procurement	UV system worked and was readily usable in its location and condition	Interviewer verified functionality, placement, and condition of system
Consumption	UV-treated and safely stored water was present	Interviewee reported treatment and interviewer verified presence of water
Habit
Procurement	A household member operated UV system at least once every 5 days	Interviewee or system operator reported frequency of operation
Consumption	Interviewee’s last glass of water came from, and her/his most common drinking source was, UV-treated and safely stored water	Interviewee reported where his/her last glass of water came from and interviewer recorded it. Interviewer also recorded most common source of drinking water
Exclusive Use
Procurement	Not documented	Not applicable
Consumption	Interviewee consumed only UV treated and safely stored water while in her/his household during the past 7 days	Interviewer walked interviewee through all water access points in the household and asked if she/he had consumed water from it in the past 7 days

Study design.

We conducted a cluster-randomized controlled stepped-wedge trial to evaluate the impact of the Mesita Azul program and to compare the effectiveness of the Enhanced and Basic program variants on compliance with safe water practices. We recruited 444 households in the 24 study communities. All communities started in the control arm and received the intervention in a randomized sequence. Using standard methods in STATA 10 (StataCorp LLC, College Station, TX), all 24 communities were listed in random order. The first four communities received the intervention in Step 1; sequential groups of four communities from the randomized list were enrolled in the Mesita Azul program at each sequential step. Among the communities that crossed over from control to intervention in each step, a second randomization process, determined by a coin flip, assigned two communities to the Enhanced and two to the Basic program variant. The nature of the interventions prevented blinding of households and enumerators. By the end of the trial, each community was enrolled in one of the two program variants. The study lasted 18 months and consisted of a baseline survey and follow-up surveys during each of the six sequential steps shown in Figure 2. Each step lasted for 2 months.

Figure 2.

Study design: cluster-randomized controlled stepped-wedge trial; secondary randomization assigned intervention communities to Basic and Enhanced programs.

The stepped-wedge design created intervention and control periods that allowed us to compare household water quality across communities with and without the safe water program.^24,25 We used data from both control and intervention time periods to measure the use of purchased garrafon-bottled water and the impact of the combined programs on compliance measures. We designed the second randomization process to test if the Basic and Enhanced program variants resulted in different compliance levels and to identify which one was more cost-effective in forming the habit of consuming safe water. We compared the two variants using data from the intervention periods only.

The four communities that enrolled in the program during the last step (Step 6) received an additional visit from the evaluation team to measure study outcomes. This allowed us to compare the Basic and Enhanced variants using compliance data from the first two visits after the intervention (months 0–4) for all study households. We also measured the evolution of compliance through time in a subset of the first eight clusters that crossed from control to intervention, with data from the first and the fifth observation visits after the intervention (months 0–2 and 8–10, respectively).

All participating households provided informed consent as per study protocols approved by the Office for the Protection of Human Subjects at the University of California, Berkeley (Protocol no. CPHS 2009-1-47). We registered this study at ClinicalTrials.gov (NCT01637389).

Data collection and analysis.

We defined safe water as water that was treated and stored in a narrow-necked container. Throughout the study, we observed households using commercial garrafon-bottled water, but despite a decade of promotion by health workers, we identified chlorinated or boiled water in fewer than 1% of our household visits (10 of 2,601 observations). Given the near-absence of these strategies, we included only garrafon-bottled water and UV-disinfected water in our compliance analysis.

We measured all compliance levels using the definitions in Table 2 and Supplemental Table 1, which pertain to the Mesita Azul and purchased garrafones, respectively. We used Supplemental Table 1 definitions for households in the control group; both Supplemental Table 1 and Table 2 definitions for households in the intervention group; and only Table 2 for comparisons between the Basic and Enhanced program variants. As explained earlier, our primary measure of “compliance,” specified a priori to maximize internal validity of the research, was the “Habit” of consuming safe water. Specifically, for a household to be characterized as observing the “Habit of consuming safe water,” the interviewee had to show that the last glass of water consumed was from either UV-treated water stored in a garrafon or from a purchased garrafon, and had to report that this source was also the most common source of drinking water.

We carried out secondary comparisons (i.e., of compliance measures other than “Habit”) to provide more information about the structure of compliance and to compare results within similar groups of households. Following Feise,²⁸ rather than correcting statistical significance levels for our secondary outcome measures, we suggest that the results of our secondary outcome comparisons should be treated with lower confidence because of the increased probability of Type I errors. We did not carry out compliance power calculations a priori, in part, because there were no existing data for our specific compliance outcome, and in part, because a study powered for health outcomes would very likely be powered for compliance.

We divided our analysis and results into summative evaluation (i.e., primary measure of compliance at the end of the 18-month intervention period) and formative research components (i.e., primary and secondary measures during the 18-month intervention period). We used the summative approach to rigorously measure compliance with the habit of consuming safe water, and the formative approach to identify drivers of and barriers to compliance. We used an intention-to-treat analysis for all measurements of compliance in the summative evaluations and in most of the formative research components. Because our study households were clustered into communities, they could not be considered independent units; we, therefore, calculated robust standard errors at the community level when estimating risk differences between comparison groups.²⁹ We also included fixed-effects for time-step in all analyses, to adjust for the time each community contributed to specific comparison groups during the study, so that the effect of time was not conflated with the effect of the UV programs.^24,30

A priori, we assumed that 15% of households would be practicing the habit of consuming safe water at baseline (mostly from garrafon-bottled water) and that the Basic program would increase this habit to 25%. Using the sample sizes set by the epidemiologic study (24 clusters, average size 20 households per cluster) and assuming an intracluster correlation of 0.1, 80% power, and an alpha of 0.05, we estimated that this study would be powered to detect 1) a 6% increase in safe water consumption in the combined intervention, compared with the control, using the stepped-wedge design; and 2) a 21% increase in safe water consumption in the Enhanced program, compared with the Basic program, using a parallel arm design with 12 clusters per arm and assuming one observation per treatment arm.

We managed our databases and carried out our statistical analyses using Stata 12 (StataCorp LLC).

Cost effectiveness (CE) of user compliance.

We assessed whether small program changes intended to increase user convenience could be a cost-effective strategy to increase compliance. We calculated the full costs of implementing the Enhanced and Basic variants per household for each specific compliance outcome (in our Compliance Framework). We aggregated the costs of inputs and implementation for each of the two program variants (V). We calculated average costs per household targeted by a program (CHT), including water quality testing, community meetings, food and transport, field workers’ salaries, etc. We then calculated additional costs per household that adopted the program (CHA), such as production of the UV system, household visits to train a member to operate the UV system, follow-up visits to collect payment, etc. We estimated total program costs (C) within a given period as a function of the number of households targeted (N_T) and the number of households that adopted (N_A). By definition, N_A ≤ N_T.

$$\mathit{C}\left(\mathit{V}\text{,\hspace{0.17em}}{\mathit{N}}_{\text{T}}\text{,\hspace{0.17em}}{\mathit{N}}_{\text{A}}\right)=\left({\text{CHT}}_{\text{v}}\times N_{\text{T}}\right)+\left({\text{CHA}}_{\text{v}}\times N_{\text{A}}\right)$$

We calculated the CE of the program by dividing the program costs (C) by the effects (E_i), where the effects are the number of households that complied with the program variant for each particular measure of compliance (i). CE is thus the total cost of a program variant per compliant household per measure of compliance.

$$\text{CE}\left(C,E_i\right)=C\left(V,N_{\text{T}},N_{\text{A}}\right)/E_i$$

We computed the CE of achieving the Habit of consuming safe water (our primary outcome) in the Enhanced and Basic program communities during months 0–4 following the intervention. As an exploratory exercise, we also computed the CE of secondary outcomes for this time period and the evolution in time (months 0–2 versus months 8–10) of the CE of the safe water habit in both variants.

RESULTS

User compliance in intervention (both program variants) and control periods.

As reported previously,²⁴ random assignment of intervention (I) and control (C) periods produced equivalent groups across a wide range of observable baseline characteristics (Supplemental Table 2). Measured baseline covariates, weighted by the time that communities contributed to intervention and control periods, included age, gender, education, drinking water quality, access to consuming safe water, and hygiene conditions.

All compliance metrics were expressed as the percentage of households that met a given criterion with respect to the total number of households that were targeted at the start of our study. Because the compliance framework outcomes were determined based on information collected during each household visit, compliance in one step does not imply compliance in previous or future steps. We, therefore, used a pooled analysis, adjusted with fixed-effects for time-step, to characterize compliance throughout the length of the study for the entire population. This is in line with other analyses of water quality and gastrointestinal illness in the household water-treatment sector.

For the primary outcome of the Compliance Framework, we observed a risk difference of 30.9% (95% confidence interval [CI]: 27.4%, 34.4%) in the Habit of consuming safe water between intervention (both program variants combined) (49.7%) and control (18.8%) periods. The Habit of consuming treated and safely stored water during control periods relied solely on households with garrafon-bottled water; during intervention periods it was made up of UV disinfection households (39.0%) and garrafon-bottled water households (10.7%). After controlling for clustering at the community level and step effects,²⁴ we obtained an adjusted risk difference of 35.5% (95% CI: 23.4%, 47.6%). We report community-level intracluster correlations for our primary outcomes in the Supplemental Materials (Supplemental Table 4).

We present the secondary outcomes of the Compliance Framework, for the entire time period and combining both program variants (Figure 3). We observed three trends. First, compliance levels were more than double during intervention periods, compared with control periods across all compliance components; all differences were statistically significant at the 95% level. Second, the intervention resulted in higher compliance levels in procurement than in consumption within the Knowledge, Access, and Habit components. Finally, compliance levels decreased in both groups as the definition of compliance became more stringent, moving from Adoption to the Exclusive Use of safe water. The drop in compliance levels along the procurement components was proportionately less pronounced in intervention than in control periods. This indicates that the reliability of procuring safe water was higher among households that had adopted the Mesita Azul than among those that purchased garrafones. The drop along the consumption components between Access and Exclusive Use of safe water was more pronounced in intervention than in control periods, suggesting that people in households with purchased garrafones were less likely to drink water from untreated sources than people in households with access to Mesita Azul-treated water.

Figure 3.

Compliance levels for each component of Safe Drinking Water Compliance Framework for consumers of purchased garrafones during control periods (N = 1,255 household observations) and consumers of ultraviolet-treated or purchased garrafones during intervention periods (N = 1,346 household observations). Basic and Enhanced program data are combined over the entire 18-month time period, leading to multiple observations per household. Estimates of standard errors account for clustering. The primary outcome results (Habit of consuming safe water) are shaded for emphasis.

User compliance in Enhanced and Basic programs.

Random assignment of Enhanced (E) and Basic (B) programs did not produce fully equivalent groups, possibly due to the small number of randomized units (24 communities split in two groups). By chance, the Enhanced program communities were further from urban centers, with lower levels of elementary schooling and baseline garrafon-bottled water consumption (Supplemental Table 3).

For the primary outcome of the Compliance Framework, we observed a risk difference of 17.4% (95% CI: 10.3%, 24.5%) between the Habit of consuming UV-safe water in the Enhanced (50.5%) and Basic (33.1%) programs. After accounting for clustering effects, the time of observation, and the level of garrafon-bottled water use during baseline, we obtained an adjusted risk of 15.0% (95% CI: 4.7%, 25.3%). We used the Safe Drinking Water Compliance Framework (Table 2) to measure a series of secondary outcomes; we present these results in Figure 4. Compliance levels for UV Adoption and all procurement measures were approximately 10% higher in Enhanced program communities; Access (consumption) was 18% higher and Exclusive Use (consumption) was 8% higher. Differences were statistically significant.

Figure 4.

Compliance levels for each component of the Safe Drinking Water Compliance Framework for consumers of safe water, 0–4 months post-intervention, disaggregated by program variant. The study ended 4 months after communities in the last step were enrolled. Basic program: N = 352 household observations (219 households visited up to two times); Enhanced program: N = 363 household observations (225 households visited up to two times). Estimates of standard errors account for clustering.

In a separate analysis of compliance contingent on adoption of the UV system specifically (i.e., excluding post-intervention households with garrafon-bottled water; data not shown), we found that the Enhanced program resulted in slightly higher compliance levels for all the procurement measures and significantly higher levels for Access and Habit (consumption). Differences across consumption for Knowledge, Access, and Habit were statistically significant.

Change in user compliance through time.

In our analysis of the change in user compliance over time, we found that the Habit of consuming UV water remained constant in communities that received the Basic variant (Figure 5). It fell sharply in Enhanced communities from 64.7% (first 2 months following the intervention) to 39.7% (8–10 months after the intervention) (Figure 6). At the end of 8–10 months, the habit of drinking UV water was equivalent across Enhanced and Basic program communities. Compliance levels of most procurement measures increased slightly with time for both program variants; consumption measures remained constant for Basic program communities and (mostly) decreased for Enhanced program communities. Of these differences, the reduction in the consumption Habit across time within the Enhanced variant was statistically significant. Any early advantage of the additional user conveniences within the Enhanced over the Basic program disappeared within 10 months.

Figure 5.

Compliance levels at 0–2 months (N = 61 households, one visit each) and 8–10 months (N = 65 households, one visit each) for the Basic variant of the ultraviolet program. Subset of eight (of 24) clusters analyzed. The primary outcome of interest is shaded.

Figure 6.

Compliance levels at 0–2 months (N = 68 households, one visit each) and 8–10 months (N = 78 households, one visit each) for the Enhanced variant of the ultraviolet program. Eight (of 24) clusters analyzed. The primary outcome of interest is shaded.

CE of user compliance.

Unlike many HWTS cost studies, in which cost accounting varies from partial to very partial,¹⁹ we calculated the full costs of the Mesita Azul program from the initial community assessments all the way to the follow-up visits post-intervention. Consistent with our intention-to-treat analysis, we calculated costs per targeted household, whether or not the household adopted. Costs per targeted household, CHT_v, such as community assessments and water quality testing, were $18 and $14 (in 2010 US $) for the Enhanced and Basic variants, respectively. Additional costs per adopting household, CHA_v, were $109 and $94 for the Enhanced and Basic variants, respectively. Under a perfect compliance scenario, in the first four postintervention months, the difference between the CE (per adopting household) of the Enhanced ($127) and Basic ($108) programs would have been $19. However, after accounting for actual compliance, we found a per-household CE difference of −$33 in achieving the Habit of drinking UV-safe water between the Enhanced ($208) and Basic ($241) variants.

Using the Compliance Framework (Table 2), we calculated the CE of all outcomes at the 4-month point of the Enhanced (E) and Basic (B) variants; the relevant numbers are shown in Table 3. Based on the evolution of compliance data (Figures 5 and 6), we also calculated the CE for the Habit of consuming UV-safe water during the first 2 months (E = $172; B = $195) and for the 8- to 10-month period (E = $292; B = $217), following the intervention. The relative CE of the Enhanced program over the no-frills Basic program with respect to our primary outcome of interest dissipated entirely over a 10-month observation period.

Table 3.

Cost effectiveness (CE) of ultraviolet (UV) program: costs in 2010 USD per complying household

Compliance measure	Enhanced UV program	Basic UV program
Adoption	132	114
Knowledge: procurement	148	140
Knowledge: consumption	208	158
Access: procurement	141	124
Access: consumption	174	188
Habit: procurement	174	160
Habit: consumption	208	241
Exclusive Use	308	300

CE assessed by measuring compliance 4 months after program variants were rolled out in communities. Enhanced Program: N = 363; Basic program: N = 352. The primary outcome of interest is bolded.

DISCUSSION

Compliance framework.

The evaluation of development-related interventions is often polarized along summative and formative approaches, resulting in a limited understanding of the mechanisms by which interventions work.^31,32 Combining summative and formative approaches is particularly relevant for the HWTS sector because the impacts of an intervention rely on many behavior changes in the target population. Correct and consistent use of HWTS systems (user “compliance”) remains one of the least understood and hardest to verify features of safe water programs,³³ and thus one of their least well-implemented steps.

We suggest that conceptualizing compliance as a multipart phenomenon is both intellectually and practically useful for future HWTS research. With the Safe Drinking Water Compliance Framework, we selected a primary outcome measurement to rigorously evaluate the extent to which a program effected behavior change, and also a series of secondary compliance outcomes that provided useful information about its strengths and weaknesses all along the compliance pathway. Deconstructing household compliance into its components (Adoption, Knowledge, Access, Habit, and Exclusive Use) makes it possible to identify specific program trends and potential modifications to increase compliance. By disaggregating the components further into procurement and consumption, the differences in the role of the user by age (e.g., older participants may be more resistant to new practices) or gender (e.g., women in a household may be responsible for treating and storing water safely)³⁴ can also be revealed. There are certainly other factors that can influence compliance, including economic status, social interactions, norms, and perceptions^20–22—but these tend to operate, and so can only be addressed, beyond the household level.

User compliance in intervention and control periods.

The Mesita Azul program (both variants combined) had a significant impact on increasing the Habit of consuming safe water, our primary outcome. In a review of 26 point-of-use water treatment studies that focused on behavior change, seven showed high sustained use > 50%, 13 showed a range of 1–34%, and five found initial use levels > 50% followed by a notable decline.¹³ Three papers used the physical presence of intervention materials (equivalent to Access [procurement] in our Framework), 14 used some form of water quality testing (Access [consumption]), seven used self-reporting and one used structured observation (ibid.) to assess compliance. By comparison, and with our relatively strict outcome of the Habit of safe water consumption, the 50% compliance documented in our study is high. Increasing this habit from approximately 20–50% (Figure 3) of the targeted population was a considerable achievement in communities without working piped systems, a decade of failed promotion of boiling and in-home chlorination, and garrafon-bottled water in only one-fifth of households during baseline. The observed substitution effect, in which almost half of households with a garrafon-bottled water habit switched to UV-safe water as a habit, was evidence of the program’s strengths. However, the intervention did not expand consistent and sustained safe water consumption to most of the targeted households.

Our Compliance Framework showed that postintervention compliance was significantly higher than in the control periods. The Mesita Azul program was consistently superior to garrafon-bottled water across all procurement—but not all consumption—measures. There could be several reasons for this discrepancy. The presence of a UV system in the house reduces the need to purchase garrafon-bottled water. But the Mesita and garrafon-bottled water use similar containers, and thus, once water has been procured, the barriers to regular consumption are similar. In addition, most activities during outreach and installation of the system were focused on procurement.

The downward trend in the compliance level as the components shifted from Adoption to the Exclusive Use of safe water is consistent with previous HWTS trials.^11,35–37 For the Mesita Azul program (which drove compliance during intervention periods), the drop in compliance was gradual and apparently multicausal. Our surveys showed that insufficient knowledge on how to procure safe water contributed to a slight drop in procurement (see Figure 5). Limited knowledge of the negative effects of drinking untreated water reduced consumption but did not fully account for the significant drop between Access (consumption) and the Exclusive Use of safe water. Knowledge of how a system works is never enough for sustained use as it is just one of many factors, such as social pressures or existing habits, that influence HWTS compliance.^20,38–40

In our study, the Mesita Azul program experienced proportionally less of a decline than garrafon-bottled water between adoption and the habit of procuring safe water, but a greater decline between levels of access to safe water and its exclusive consumption. This showed an established water management practice (garrafon-bottled water) resulted in a narrower gap between Access and Exclusive Use than a recently introduced alternative with a lower barrier to procurement.

User compliance in Enhanced and Basic program variants.

Our results showed that the small modifications that were meant to reduce user barriers (i.e., reduce small hassles) led to early improvements in compliance outcomes, supporting the arguments of behavioral economists that small changes can be both feasible and effective.²³ Because the randomization of the Enhanced and Basic programs did not lead to fully equivalent groups, we adjusted our analysis for baseline imbalances in garrafon-bottled use, and still found the primary outcome to be significantly higher in Enhanced program communities. Over time, however, both program variants resulted in approximately the same Habit of consuming safe water (from UV-disinfected or garrafon-bottled water). This result could have been driven by differences in group characteristics or preintervention safe water practices across both groups. Lower use of garrafon-bottled water in Enhanced communities at baseline could be an indication of higher barriers for any safe water practice. In this case, the Enhanced variant faced more challenges and, with equivalent groups, the compliance differences may have been more pronounced. Alternatively, it could be that higher barriers to securing safe water in Enhanced communities were specific to garrafon-bottled water (e.g., distance to a vendor). This could mean that the observed difference across the two groups was the result only of differences in baseline drinking water practices.

The drop in compliance observed over time for the Enhanced program, but not for the Basic (see Figures 5 and 6), suggests that the barriers specific to procuring garrafon-bottled water were not the only reason for lower baseline safe water consumption in Enhanced communities. From (qualitative) informational interviews with program staff, it seemed that the main drivers of additional early compliance in the Enhanced variant were the money-back guarantee and the (added) 1-month follow-up visit. The effect of additional household-level information on water quality was unclear, and indeed the literature on the effect of information on safe water demand and use has been mixed.^41–43 The money-back guarantee allowed households with lower priority for safe water to gain access to it without having to assume risks or make binding payment commitments. Some of these households, however, did not have the motivation or enabling conditions to face ongoing operation of the UV system and habitual consumption of UV-treated and safely stored water. The follow-up visit within 1 month of the Enhanced intervention might have postponed this effect by providing additional short-term motivation to the users or by clarifying questions that were limiting compliance.

Based on our results and analysis, we conclude that there was a high likelihood that the Enhanced program led to higher compliance levels than the Basic program while the additional program components were active. Over time, the benefits of the Enhanced program disappeared. Future experiments testing the “small hassles” hypothesis should be designed to better understand whether small conveniences lead to big improvements in uptake mainly for one-time programs (such as signing up for a bank account or adopting a safe water system) or also for programs that the user needs to sustain (such as using the account regularly or drinking safe water daily). Future work with any specific HWTS program may also benefit from pilot-testing strategies outside the current HWTS paradigm, such as expanding a narrow focus on drinking water to making all domestic water safe to drink (as suggested by our observations of multiple water access points in many households).

CONCLUSIONS

This article conceptualizes “compliance” with HWTS technologies as a multipart phenomenon. We developed the Safe Drinking Water Compliance Framework in which user compliance with HWTS programs is disaggregated into Adoption, Access, Knowledge, Habit, and Exclusive Use. Deconstructing compliance into its components allows researchers and practitioners to identify, for any given safe water approach, the practices at which use falters, and which practices can feasibly be strengthened to encourage sustained use. We applied the framework to a UV-based HWTS program, the Mesita Azul, in rural Mexico; we found that the UV program significantly improved compliance, where compliance was defined as the habit of consuming safe water. Furthermore, half the commercial garrafon-bottled water users preintervention switched to UV disinfection postintervention.

Behavioral research on poverty and health has argued that small programmatic changes to reduce hassles at the user end can lead to big improvements in uptake. The Mesita Azul program was tested in two variants: Basic and Enhanced (with additional user conveniences). We analyzed the compliance rates of each variant at each Framework component, and calculated the full costs of the program—from community assessment to post-installation follow-up visits—per household that we targeted, and per household that eventually adopted. The Enhanced variant led to higher compliance while enhancements lasted; in particular, it was more cost-effective for Habit formation. But compliance and CE in the Enhanced variant degraded with time for our primary outcome, the Habit of consuming safe water. It may be that small conveniences are more effective for one-time uptake programs, such as acquiring a bank account or getting a medical test, than for programs that need daily and sustained behavior changes.

Note: Supplemental tables appear at www.ajtmh.org.