Table 5.

Possible study designs for future WSP impact assessments, with advantages and challenges.

	Uncontrolled Study Designs		Controlled Study Designs
	Context: Site- or Country-Level Monitoring and Evaluation		Context: Research and Rigorous Impact Assessments
	Before-after Comparison	Interrupted Time Series	Matched Controls	Randomized Controlled Trial
Control group	No control group; for each site, relevant indicators are compared before and after WSP implementation	No control group; for each site, historical time series of relevant indicators are investigated to detect potential changes in slope coinciding with WSP implementation	Before WSP implementation, sites are manually assigned to a “control” or “intervention” group by matching a number of selected parameters between the two groups (e.g., system size, age, revenue, geographic setting)	Before WSP implementation, sites are randomly assigned to a “control” or “intervention” group. The randomization ensures that all possible confounding factors are equally distributed amongst the two groups.
WSP implementation	To all sites	To all sites	Only to “intervention” group	Only to “intervention” group
Data needed	Baseline and follow-up data	Historical data (pre- and post-WSP) on all relevant indicators (i.e., time series, not just baseline and follow-up data)	Inventory of all eligible study sites with data on parameters for matching Baseline and follow-up data	Inventory of all eligible study sites, ideally with data on some key parameters to confirm comparability between intervention and control groups Baseline and follow-up data
Advantages	Simplest study design (does not require a control group and only two data points per indicator: before and after) Results can be valuable for national advocacy and to encourage better monitoring/data collection practices Two rounds of data collection	Does not require a control group Provides more confidence than a simple before-after comparison that the changes observed may be associated with WSP implementation	A rigorous study design to examine associations between WSP implementation and outcomes/impacts, as long as all key parameters potentially affecting a water system’s performance (i.e., confounding factors) are used for matching Two rounds of data collection	The only study design able to establish causality, i.e., the differences between the control and intervention groups can be attributed to WSP implementation because confounding factors are equally distributed amongst the two groups Two rounds of data collection
Challenges and limitation	Causality cannot be established from a simple before-after comparison, i.e., the changes observed cannot be attributed to WSP implementation	Limitations in establishing causality (i.e., the change in slope observed cannot be rigorously attributed to WSP implementation) Multiple (>2) rounds of data collection Difficult to obtain time series of all relevant indicators, especially in low-capacity sites that do not keep rigorous records. Where available data are limited, data collection could be limited to those indicators that are most likely to show changes (as identified by prior rigorous impact assessments conducted at other sites)	Difficult to obtain data on matching parameters, especially for small water systems Risk that confounding factors may be unevenly distributed between the two groups (especially if an insufficient number of parameters are selected for matching), limiting ability to establish causality	Randomizing WSP implementation may cause ethical concerns or political frictions. To mitigate these, WSPs could be implemented in the control group at the end of data collection (i.e., staggered implementation).