Economic Evaluation of Public Health Interventions: An Application to Interventions for the Prevention of Violence Against Women and Girls Implemented by the “What Works to Prevent Violence Against Women and Girls?” Global Program

Abstract

Violence against women and girls (VAWG) has important social, economic, and public health impacts. Governments and international donors are increasing their investment in VAWG prevention programs, yet clear guidelines to assess the “value for money” of these interventions are lacking. Improved costing and economic evaluation of VAWG prevention can support programming through supporting priority setting, justifying investment, and planning the financing of VAWG prevention services. This article sets out a standardized methodology for the economic evaluation of complex, that is, multicomponent and/or multiplatform, programs designed to prevent VAWG in low- and middle-income countries (LMICs). It outlines an approach that can be used alongside the most recent guidance for the economic evaluation of public health interventions in LMICs. It defines standardized methods of data collection and analysis, outcomes, and unit costs (i.e., average costs per person reached, output or service delivered), and provides guidance to investigate the uncertainty in cost-effectiveness estimates and report results. The costing approach has been developed and piloted as part of the “What Works to Prevent Violence Against Women and Girls?” (What Works?) program in five countries. This article and its supplementary material can be used by both economists and non-economists to contribute to the generation of new cost-effectiveness data on VAWG prevention, and ultimately improve the allocative efficiency and financing across VAWG programs.

Introduction

Globally, 35% of women have experienced intimate partner violence (IPV) (Devries et al., 2013). Violence against women and girls (VAWG) has important social, economic, and public health impacts, in addition to being a human rights concern. Exposure to IPV is associated with an elevated risk of a number of adverse health outcomes, such as HIV, induced abortion, and preterm birth (World Health Organization [WHO], 2013), as well as a number of adverse mental health outcomes (Ferrari et al., 2016; WHO, 2013). Given the scale of the problem, governments and international donors are allocating increasing resources to address VAWG, particularly IPV. The United Kingdom and the World Bank alone have invested more than US$400 million in programs for, and research on, the prevention of VAWG and support to survivors in low- and middle-income countries (LMICs; Global Gender-Based Violence Task Force, 2017), and the WHO has issued guidelines on violence prevention programming for policy makers and planners (WHO, 2010).

The scale of the investment and policy focus on VAWG warrants an increased interest in assessing the value for money of VAWG interventions. Evidence on the economic value of violence prevention and development programming is, however, scarce. In addition, assessments of VAWG are methodologically challenging for three reasons. First, it is difficult to account for and value the diverse outcomes of VAWG prevention in areas such as health, education, work, social networks, and access to financial resources. Second, cost estimates are currently only available from small research projects, which may not be representative of costs of implementing at scale. Third, VAWG prevention delivery processes vary by context, and culturally specific gender norms, limiting the generalizability of costs and economic evaluation.

We outline a standardized methodology for the economic evaluation of complex VAWG prevention interventions. We developed this methodology by adapting current general guidance on economic evaluation and costing in global health (Neumann, Sanders, Russell, Siegel, & Ganiats, 2016; Vassall et al., 2017 [https://ghcosting.org/pages/standards/reference_case]; Wilkinson et al., 2016) to the characteristics of VAWG prevention programming. Analysts intending to apply our guidelines should be familiar with the general guidance too. Our aim is to provide further specification around their specific application to VAWG prevention.

“What Works to Prevent Violence Against Women and Girls?” (What Works?) is a multicountry, multiyear research and innovation program that tested the effectiveness and cost-effectiveness of VAWG prevention programs in LMICs. It is funded by the United Kingdom’s Department for International Development (DfID) and it set up an international network of implementers, VAWG prevention professionals, and researchers who co-created and evaluated these interventions. The economic evaluation component was tasked with determining the value for money of six interventions implemented alongside randomized controlled trials. The interventions were implemented in six different countries, five in sub-Saharan Africa, and one in South Asia, and across different platforms: schools, villages/communities, and adult small-group learning settings. We developed a standardized economic evaluation methodology to improve transparency in value for money estimates of VAWG interventions, as well as their comparability, replicability, and generalizability (Evans & Popova, 2016), for What Works? and beyond.

Our structure and approach to economic evaluation methods are in line with the Reference Case approach outlined in the International Decision Support Initiative (iDSI) guidelines (2014), which remain largely within the extra-welfarist framework (Brouwer, Culyer, van Exel, & Rutten, 2008) that is prevalent in economic evaluation in the health sector (Neumann et al., 2016; Vassall et al., 2017; Wilkinson et al., 2016). We do not state here whether a specific method of economic evaluation (cost-effectiveness, cost-utility, cost-consequence, or benefit–cost analysis) should be adopted, because this decision will also depend on the specific problem analysts are faced with, as well as on data availability. For methods related to costing, we also referred to the Global Health Costing Consortium (GHCC) Reference Case (Vassall et al., 2017). Both sets of guidelines are designed for global health and are consistent with the Second U.S. Panel on Cost-Effectiveness in Health and Medicine (Neumann et al., 2016) and the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) guidelines (Husereau et al., 2013).

We also referred to the U.K. DfID Value for Money (VfM) framework for the assessment of the economy, efficiency, effectiveness, and equity of its programs (DfID, 2011). Specifically, we state how our recommendations relate to DfID’s framework and its implementation. The article begins by explaining and discussing the challenges in identifying, measuring, and valuing resources and outcomes. We then list the issues to consider in analysis. We summarize our recommendations for each of the iDSI and GHCC principles in Tables A1 and A2 in the appendix. Copies of the manual and tools to accompany the guidance outlined in this article are available on the What Works? website as a project resource. ¹

The Decision Problem

The basis for any economic evaluation of an intervention is the “decision problem” (Vassall et al., 2017 [https://ghcosting.org/pages/standards/reference_case]). The decision problem outlines the investment choice and can be defined by target Population, Intervention, Comparator, and Outcome (PICO) to be achieved in a PICO statement (see Figure 1 for two examples; Richardson, Wilson, Nishikawa, & Hayward, 1995).

Figure 1.

PICO statements from two What Works? interventions.

Note. PICO = Population, Intervention, Comparator, and Outcomes; What Works? = “What Works to Prevent Violence Against Women and Girls?”

Population

The typical target populations of VAWG prevention are women and girls. However, due to the complexity of VAWG, costs and outcomes fall on other populations (externalities) and need to be considered for decision makers interested in broader social welfare. For example, male perpetrators of violence may also benefit from decreased use of violence. Children may be adversely affected by mothers’ exposure to violence (Hardt et al., 2015; Smith-Marek et al., 2015) and thus gain from a reduction in violence. Women’s improved access to economic resources may also improve child health and educational outcomes (Duflo, 2003). More generally, changing social norms and gender attitudes through targeting entire communities may result in spillover benefits to those directly and indirectly affected by violence (Stern, Heise, & McLean, 2017).

Intervention

Analysts should describe the intervention, including all activities required for its development and implementation. Broadly, VAWG interventions can be classified as workshop-based interventions (for individuals, families, or couples), economic empowerment interventions (with or without gender-empowerment components), school interventions delivered to classes, psychotherapeutic interventions including self-defense, and gender attitudes and social norms change interventions delivered to communities (often multiple components, for example, including workshops, community action teams, street theater, murals, and community radio). Workshops, gender empowerment, and programs delivered to classes are often administered over a limited number of small-group sessions (Gibbs et al., 2017; Iyengar & Ferrari, 2011). Trained facilitators deliver a curriculum, which includes sessions on gender, power, and violence; teaching a mixture of vocational and communication skills; supporting help-seeking or providing support; and skills of critical reflection. The assumption is that increased information on harmful norms and behaviors and an expanded skillset for challenging these will reduce exposure to, and perpetration of, VAWG. Community-based interventions aimed at reducing VAWG have also shown encouraging impact (Abramsky et al., 2014). They typically entail the identification and training of local volunteers. Some community-based schemes first raise VAWG awareness and then build positive concepts and practices of power in the pool of potential recruits. In turn, the most motivated recipients become community mobilizers and administer the intervention to the rest of the community (Pronyk et al., 2006; Stern & Niyibizi, 2018). Some VAWG prevention interventions are implemented using existing services (such as community financing schemes or schools).

Strengthening local services/platforms at times precedes the intervention (Pronyk et al., 2002; Stern & Niyibizi, 2018) and may incur costs. Analysts should clearly describe the platform, in terms of the existing services delivered, the scale of operation, and where services are delivered jointly. Understanding the platform is required to correctly identify the incremental costs of the investment and generalize results across settings. Specifically, the platform will influence costs and/or outcomes, through synergistic benefits, or economies of scope. Economies of scope occur when costs are reduced through improved efficiency as a result of joint production; for example, where the cost of a combined life skills and microfinance intervention is less than the cost of providing these two interventions separately.

It is also critical to clearly describe any activities required to start the VAWG intervention. Typical start-up activities for complex interventions include curriculum development, curriculum adaptation, frontline workers’ training, and obtaining permission to operate locally. Where feasible, total costs of development should be estimated and included in cost-effectiveness estimates. However, if the interventions were developed over a long period of time through iterative processes, more than five years prior to the study, costs may not be properly documented. This makes it particularly challenging to identify all relevant resources, increasing the likelihood of inaccurate costing and reducing comparability if costs can be determined for some interventions but not for others. In such cases, we recommend reporting total development costs separately, where these can be determined, and refraining from including them in the cost-effectiveness estimates. Where a program is scaling up, a clear description of any activities to scale up from the research setting stage is necessary (Evans & Popova, 2016).

Comparator

In assessing cost-effectiveness, analysts must establish incremental costs and benefits compared with the status quo (i.e., usual care or “do nothing”). In some cases, different investment options may also be compared, as they will be part of the “choice set” of interventions the decision maker is deliberating. For interventions introduced as add-ons to existing programs (Gupta et al., 2013; Iyengar & Ferrari, 2011; Pronyk et al., 2006), the existing program is considered the usual care (control) scenario. Particularly where this may be required to enable scale-up, a do-nothing comparator and the cost of strengthening the status quo should be included.

Relevant comparators that may be accessed by the target population to improve their outcomes should be included, even if not directly connected with the specific Randomised Controlled rial (RCT) or evaluation. Particular difficulties in assessing the status quo do not justify its exclusion from analysis (Sculpher, Claxton, Drummond, & McCabe, 2006). For complex, multicomponent programs already administered at scale and not evaluated with RCTs, analysts may exploit gradual rollout or other econometric or modeling techniques to characterize the comparators or create synthetic controls, rather than exclude the status quo (Jones, 2015).

Outcomes

The primary outcomes typically reported in the evaluation of VAWG prevention are the number of cases of IPV and/or disability-adjusted life years (DALYs) averted (Jan et al., 2011; Michaels-Igbokwe et al., 2016). However, these may only be a partial measure of impact: DALYs capture a subset of IPV health sequelae, and many interventions (particularly those delivered on joint platforms) are likely to affect outcomes beyond VAWG. In addition, outcomes may interact with each other. Therefore, these outcomes may not capture the full direct impact of interventions and their broader societal impact (Neumann et al., 2016).

To address this complexity pragmatically, we recommend analysts first list all relevant outcomes. This “impact inventory” should include all outcomes posited in the intervention’s theory of change (Neumann et al., 2016, p. 101), including impacts that will materialize at a later stage, beyond the immediate effects of the intervention, which may require modeling. Some of these impacts will be captured in the secondary endpoints in trials and evaluations, and will generally be included in the intervention’s theory of change. Testing of multiple outcomes in economic evaluation is handled within a Bayesian framework (Briggs, 1999), which naturally accounts for simultaneous tests (Scott & Berger, 2006). Where there are sectoral decision makers, outcomes should then be assigned to the relevant sector of the economy more likely to benefit from them (Jones, 2015; Remme, Martinez-Alvarez, & Vassall, 2017) or a cost–benefit analysis can be conducted. A list of suggested outcomes, common to the prevention of VAWG, is shown in Table 1.

Table 1.

List of Recommended Outcomes for the Economic Evaluation of Violence Prevention.

Domain	Suggested Outcomes
Partner violence	Self-reported past year physical and/or sexual partner violence, experienced or perpetrated Self-reported past year emotional partner violence, experienced or perpetrated Self-reported past year economic partner violence, experienced or perpetrated Self-reported past year controlling behavior, experienced or perpetrated
Other violence	Violence from/to non-partner (both men and women) Child exposure to intimate partner violence Child abuse
Health	Anxiety/depression Post-traumatic stress Suicidality Hazardous alcohol and/or drug use Physical injuries Preterm and/or low birthweight offspring Miscarriages or abortions	Femicide HIV/STIs Sexual risk taking Nutrition DALYs QALYs Women’s participation in health-related household decisions
Gender norms and roles	Equitable gender attitudes Women’s participation in non-health-related household decisions
Economic and socio-psychological	Individuals’ employment outside the household Individuals’ income Individuals’ ability to retain control over their own income Individuals’ consumption Savings behavior and time preference Poverty Criminality due to lack of money	School performance Absence from school School dropout (including pregnancy-related) Well-being (life satisfaction and flourishing) Locus of control Self-esteem/Self-efficacy Agency
Institutions	Proportion of victims receiving formal or informal support Improved institutional response to cases of VAWG reported Equal opportunities in the workforce

Note. STIs = sexually transmitted infections; DALYs = disability-adjusted life years; QALYs = quality-adjusted life years; VAWG = violence against women and girls.

Measuring exposure to or perpetration of VAWG

At a minimum, outcomes should be reported in natural units, including cases of physical and/or sexual VAWG (experienced or perpetrated) averted. This allows for some comparability with preexisting studies. In addition, analysts should specify the type of VAWG the intervention is tackling: For example, if the intervention tackles IPV, they will specify “cases of IPV exposure (and/or perpetration) averted.” The time period of the outcome (see below) should also be included in the outcome measure, so that, “IPV cases averted in the past year,” is reported as, “IPV-free person years.” In general, past year is an appropriate time frame, because prevention interventions may not prevent further exposure to violence and measuring lifetime exposure would mean that prevention efforts would only be counted among women and girls who have never experienced violence. Furthermore, the use of “past year” aims to strike a balance between recall bias and rarity of the event (Garcia-Moreno, Jansen, Ellsberg, Heise, & Watts, 2006). Finally, a measure of frequency of exposure to violence over the past year permits the estimation of reductions in exposure, even in women who were not free of violence in the previous year, and some studies report costs per past year incident averted (Greco et al., 2018). However, this may be problematic in settings with high rates of violence, for which shorter time periods of reference may be preferred, though this will reduce comparability with studies that measure IPV exposure over the past year.

Health outcomes

Outcomes should be converted to general comparable metrics where feasible, for potential payers in different sectors. For example, if the health sector is likely to be a payer, then estimates of DALYs averted should be derived. Key health sequelae that could be considered will vary according to the age group and social context of the intervention beneficiaries, but mental health measures such as symptoms of post-traumatic stress disorder, anxiety, depression, suicidality, and substance abuse should be included (Devries et al., 2013), as well as injuries. In some populations, sexual and reproductive health outcomes such as preterm births and miscarriages, unwanted pregnancies averted, number of sexual partners, engagement in transactional sex or sex work, condom use, sexually transmitted infections, HIV, or HSV2 (herpes simplex virus type 2, which causes genital herpes) acquisition would be relevant. DALYs can be computed directly on health outcomes using the latest methodological recommendations (Lim et al., 2012; Murray et al., 2012; Salomon et al., 2015), or indirectly with DALY weights attributed to IPV exposure (Lim et al., 2012; Murray et al., 2012; Watts & Cairncross, 2012).

Non-health outcomes

Furthermore, non-health outcomes such as educational attainment among teenagers, participation in household decision-making, controlling behaviors, and social norms on gender relations and the acceptability of violence are associated with VAWG (Fulu & Miedema, 2015; Heise, 1998). They are typically included in theories of change around VAWG, and should therefore be included in the impact inventory, and reported in a cost-consequence analysis at a minimum (Neumann et al., 2016). The set of non-health outcomes will vary across interventions and contexts. For example, an intervention to prevent sexual assault among young teenagers, such as the IMpower intervention in Kenya, could measure exposure to sexual assault as its main outcome, as well as school attendance or drop-out rates, and school grades or children’s cognitive ability as secondary outcomes. In contrast, an intervention designed to prevent IPV in couples that are members of microfinance associations, such as the Indashyikirwa couples’ intervention in Rwanda, will have IPV cases averted as a main outcome and may have participation in household decision-making, social attitudes toward IPV and gender relations, and socially harmful behaviors, such as harmful alcohol and drug consumption, as secondary outcomes.

Outcomes should be converted to general measures where feasible, for potential payers in different sectors. For example, a general measure of education, such as attainment on standard tests or of high school diploma, for the education sector, or net income for the labor sector, or DALYs/QALYs for the health sector.

Recommendations on outcomes reporting

Listing outcomes may help sectoral payers (i.e., policy makers who operate in specific sectors and may commission, and pay for, the intervention to be scaled up) but does not account for the relative value of those outcomes or interaction between outcomes. It also may not fully inform payers interested in overall economic welfare. Analysts may therefore wish to consider composite outcomes, including benefit–cost ratios (equity or non-equity adjusted) and/or using measures of economic well-being. There are several approaches for this, but there has only been limited application of these measures to VAWG prevention to date. One “first step” option is that outcomes could be used to construct “empowerment” indices (Greco, Lorgelly, & Yamabhai, 2016). Empowerment indices require extensive preparation to ensure that domains and weights used in computing the index are locally determined and relevant, while also capturing underlying dimensions of subjective/psychological well-being (Ferrari, 2016).

Another simple approach, which is currently being applied to VAWG evaluation in What Works?, is to collect measures of global well-being to proxy participants’ utility. Measures of flourishing (Diener, 2009) capture a psychological construct of empowerment likely to be affected by complex interventions (Ferrari,2016). Measures of life and health satisfaction are also widely used. These measures partly overlap with health-related quality-of-life measures (Brazier, Ratcliffe, Saloman, & Tsuchiya, 2017; Mukuria & Brazier, 2013). One option being investigated by several research groups is to use these measures to inform a benefit–cost analysis by determining the necessary monetary compensation to keep an individual at the level of well-being corresponding to their permanent income, given their health condition (Jones, 2015; Neumann et al., 2016; Powdthavee & Van Den Berg, 2011; Sunstein, 2016). Such estimates could also be complemented by modeling of costs and benefits to provide more comprehensive estimates of the benefits and costs of these interventions to society.

Perspective

In some circumstances, VAWG prevention interventions require considerable community resourcing. Therefore, a provider perspective, including the costs to the grassroots nongovernmental organization (NGO), the local public payer if public services are used by study participants, and any donor or volunteer contributions, may be appropriate in most cases. Where clients or target populations incur costs to access services, we also recommend a societal perspective (Sanders et al., 2016; Vassall et al., 2017). A societal perspective includes clients’ and households’ costs, including lost productivity from time spent off work due to illness. In some cases, a societal perspective will also include sectors which incur external costs such as police and justice system costs.

Time Horizon

In line with iDSI Principle 6, the time horizon for the evaluation should capture all costs and effects. A lifetime horizon is recommended. Only where shorter time frame is demonstrably adequate, can there be an exception. Two possible reasons would justify a shorter time frame. First, program benefits are only experienced during the intervention. However, VAWG prevention aims for sustainable changes in behavior. Therefore, analysts must use available evidence to inform decisions about the likely persistence of this longer term effect. Consideration should also be given to the expected duration of health effects to determine total DALYs. We recommend a minimum scenario with no effect duration beyond the intervention, and a scenario where duration of effect goes beyond the intervention period, supported by an adequate rationale. In such cases, decision makers should be provided with at least a summary of the evidence on longer term impact of VAWG prevention (Lim et al., 2012; WHO, 2013), even if not directly estimated in their setting (Jones, 2015).

Resource Use and Costing

VAWG interventions are varied and evolving, and policy makers will need to adapt estimates to different settings or as interventions are implemented. Detailed reporting of inputs, allocation criteria, and prices will help users of costs to adjust costs with changing prices and service delivery models and identify scope for improving efficiency with scale-up.

Type of Unit Cost

An economic cost ² is required for the economic evaluation, but financial costs, that is, costs that are effectively incurred by an organization, such as expenses or salaries, can provide useful information for decision makers assessing budget impact. In many cases, the decision maker is trying to establish whether to implement VAWG prevention (rather than invest in research) so analysts should estimate “real-world” costs that reflect the reality of implementation. Such costs can include incremental costs of removing constraints on implementation, for example, constraints to recruitment rates imposed by staff numbers (Jan et al., 2011) and staff turnover (Ferrari, Feder, Bailey, Peters, & Hollinghurst, in press).

Scope of Inputs

Many VAWG interventions add to existing services, so analysts will need to determine which costs are incremental. It may be necessary to consider economies of scope: Analysts should empirically estimate the cost of the platform (e.g., microfinance) service provision in both the intervention and the comparator, and the incremental cost obtained from their difference. If this is not feasible, incremental intervention costs should be measured using a transparent allocation procedure for joint costs; determined by the measurement of actual resource use (e.g., human resources should be allocated according to time spent).

A challenge for analysts involved in costing VAWG interventions is likely to be how to cost interventions that employ spare capacity in existing platforms. Even though this use of spare capacity may have no immediate opportunity cost, over time there may be one. For example, in a school-based intervention, VAWG programming may use teacher time. The time spent on the intervention by teachers with spare capacity may have neither immediate nor short-term economic costs. However, in the longer run, there may be an opportunity cost, in that the teacher’s time could be used to support students in other ways. Moreover, as a program is scaled up, spare capacity may not be present (Meyer-Rath et al., 2012). This information is likely to be of use to decision makers (Jones, 2015). We recommend including the cost of such spare capacity in a sensitivity analysis and discussing the extent to which excess capacity was drawn upon in the intervention. For example, in their economic evaluation of the Good School Toolkit in Uganda, Greco et al. (2018) provide evidence that teaching the Toolkit does not displace teachers’ time, and report unit costs that may be used for further analysis.

Methods for Estimating Resource Use

As VAWG prevention is in its early stages and interventions are not standardized, we recommend a micro-costing methodology (i.e., measuring the inputs used to produce a given output) to provide the level of detail requested by GHCC Principle 7 (Table A2). We recommend micro-costing even where some standardized or reference costs (average costs of services computed according to standard methodologies such as in the United Kingdom; Curtis & Burns, 2017) may be available. We recommend distinguishing between bottom-up (where all resources are observed and added up) and top-down estimates (where aggregate costs are used and allocated to different activities and interventions). To aid the costing of VAWG prevention and women’s empowerment interventions, we have developed an Excel™ costing tool and data collection instruments for public use (see https://bit.ly/2vKaGZZ). The tool must be adapted for each intervention and setting but can serve as a starting point to identify all inputs, allocation methods, and methods for calculating unit costs.

Units for the Calculation of Unit Costs

We recommend computing costs for the units listed in Table 2, developed using the GHCC framework. Providing estimates of costs of these standardized units allows for comparability across different interventions and sites.

Table 2.

Standardized Unit Costs for VAWG Interventions.

Mapping Standardized Unit Costs—VAWG Interventions
Unit costs	Quantities	Workshop-Based	Economic Empowerment	School-Based	Community-Based
Above intervention level (general project and overhead costs)
Cost of trainers’ or counselors’ recruitment	Q(trainers/counselors)	√	√	√	√
Cost of training per site a (includes training of local interventions, such as police, or health clinics)	Q(sites)	√	√	√	√
Cost of manual/curriculum development or adaptation	1	√	√	√	√
Cost of management to support the above and for general organizational needs (e.g., general fundraising for the organization)	Q(sites)	√	√	√	√
Direct intervention
Cost of managing direct intervention support per site	Q(sites)	√	√	√	√
Cost per person session (any from one-to-one, to small group, to community gatherings and activities)	Q(number of person sessions)	√	√	√	√
Cost per screening/initial assessment	Q(screenings/initial assessments)	√
Cost of recruitment	Q(persons recruited)	√	√	√
Cost per treatment completed	Q(completed treatments)	√	√	√	√
Cost per minimum cycle (i.e., minimum number of sessions)	Q(minimum cycles completed)	√	√	√	√
Cost of transport per treatment completed	Q(number of personnel trips for sessions delivery)	√	√	√	√
Cost of community entry	Q(community recruited)	√	√	√	√
Ancillary interventions
Cost of supervision per person/community treated b	Q(persons/communities treated)	√	√	√	√
Cost of supervision per counselor/facilitator trained	Q(counselors/facilitators)	√	√	√	√
Cost of managing intervention support costs per site	Q(sites)	√	√	√	√

Note. VAWG = violence against women and girls.

^aSites are different from communities. There could be more than one site per community.

^bThe cost refers to the total supervision sessions required to support facilitators/counselors to enable them to complete one treatment cycle with one person (a decision will need to be made if this is done for minimum cycles or only for completed treatment).

This framework identifies above-site, direct, and ancillary unit costs of service delivery, and aggregates them as illustrated in Figure 2. Direct service units should be further defined in each setting to capture the outputs that characterize most interventions: teaching delivered to classes, counseling sessions, radio episodes, or community-based activities for the delivery phase.

Figure 2.

Intervention unit costs (UCs) and their components.

Sampling

VAWG interventions are delivered across a range of settings. In some cases, it may be useful to generate nationally representative cost estimates. In others, a focus on specific populations, settings, or interventions may be more appropriate (Evans & Popova, 2016). Sampling design should reflect these different purposes (GHCC Principle 8; Table A2). Costing may require a two-stage sampling of communities and within them specific sites or groups (such as the sampling of community health workers [CHWs]). The sampling frame should contain information on the key cost drivers of VAWG service delivery, such as amount of services delivered per site. If the program is large enough (n > 6 sites per stratum and N > 5 strata), analysts could draw a random sample of delivery sites stratified by the relevant cost driver(s); otherwise, they should sample sites purposively.

Shadow Prices and Valuation

GHCC Principle 14 states that all shadow prices, that is, “the true social opportunity cost[s]” of inputs (McIntosh, 2010, p. 40), be labeled as such and reported alongside the source. The successful piloting and sustainability of scale-up for some VAWG prevention interventions will depend on volunteer labor (Kasteng, Settumba, Källander, Vassall, & inSCALE Study Group, 2016), whose time should be valued appropriately (Jan, Pronyk, & Kim, 2008; Kasteng et al., 2016; Michaels-Igbokwe et al., 2016). Once all tasks in a volunteer’s working day have been identified (e.g., life skills session; administrative report writing, supervision of volunteers), we recommend using replacement price, pricing volunteers’ time at the cost of their closest substitute (e.g., CHW, administrator, CHW supervisor). However, in some cases, replacement prices do not fully represent the cost of foregoing the volunteer’s time on the next best alternative. If programs are scaled up, the increasing demand on community time will mean that opportunity costs may rise, as more productive members of the community are involved.

Uncertainty

We address methodological uncertainty by providing these standardized guidelines (Briggs et al., 2012). We also recommend exploring structural uncertainty, as a minimum. This derives from model assumptions, such as discount factor and conversion rates (Dhaliwal, Duflo, Glennerster, & Tulloch, 2013; Wordsworth & Ludbrook, 2005). We also recommend that, where feasible, uncertainty around economies of scale and scope be examined, particularly when using research setting costs to inform investments in VAWG scale-up (Evans & Popova, 2016). Where empirical analysis is unfeasible, these investigations will include scenario analysis based on the estimation of a cost function to obtain the marginal cost for a given scale or scope of an intervention (Neumann et al., 2016). We recommend presenting a range of scenario analyses that consider both different levels of spare capacity and changes in costs due to different levels of fixed costs across locations as a program expands. If a program is anticipated to have high levels of coverage, then diseconomies of scale, that is, increasing unit costs compared with a situation with lower coverage, should also be explored (Briggs et al., 2012). Table A3 in the appendix reports the sensitivity analyses we recommend.

Conclusion

Donors are increasing resources targeting the prevention of VAWG in LMICs, both by implementing prevention-focused programs and adding VAWG prevention to other interventions. In the face of this increased investment, tools for evidence-based resource allocation should be upgraded.

This article lays out a standardized methodology for the economic evaluation of VAWG prevention interventions, which is also more widely applicable to complex empowerment and behavioral interventions. Such interventions are complex, with multiple components and administered across many service platforms and settings. This poses challenges to the estimation of comparable costs and means that guidance may change over time. We outline criteria for the identification of unit costs that capture core activities likely to be common across interventions. We recommend unit costs for reporting (see Table 2), as well as the measurement and valuation of resource use to improve the comparability of intervention costs across settings and provide tools to support data collection.

In addition, our standard methodology accounts for the diversity of prevention programming recipients. We identify outcomes relevant to women, men, children, and adolescents. The acknowledgment of these distinct groups of recipients, and the use of subgroup-specific outcomes, alongside common outcomes, will yield insights into any heterogeneity in programming cost-effectiveness. This will contribute valuable information for decision-making around the financing of VAWG prevention services that take intersectionality into account.

We do not provide guidance on how to incorporate the availability of health or social-support systems that provide the context for the implementation. Collecting these data would provide information on the impact the intervention may have on downstream services (e.g., a VAWG referral system beyond the intervention being studied, or reduced use of health services due to reduced exposure to VAWG) and help analysts quantify the wider costs or cost savings that the intervention is likely to determine for other services. We think this would be a fruitful area of investigation for the future and encourage efforts in this direction. Many VAWG prevention interventions rely on lay frontline workers. This is especially the case for community-based mobilization interventions. In such cases, little is known about what incentives these workers may require keeping delivering the intervention beyond the lifetime of the initial phase, thereby guaranteeing the sustainability of the intervention. Understanding their motivation for this type of work, and the incentive structure they respond to, is key to guaranteeing the sustainability of community-based interventions beyond VAWG prevention and beyond the health sector. It needs to be better understood to estimate the costs of these interventions. Finally, there remain challenges in identifying standard outcomes that may be compared across interventions, within the health sector and beyond, including as lagged secondary effects, and understanding the duration of the effect.

Our recommendations include the estimation of standardized measures of VAWG and of a synthetic measure of health impact in the DALY. These measures are not comprehensive and other measures can also be used, but as standard minimum data set, they can increase comparability with other VAWG and public health interventions and yield transparent modeling results. We recommend the identification of an “impact inventory,” whose components may be summarized in a multidimensional index, as a measure of impact and of synthetic measures of well-being to capture intervention impact on individuals’ utility. Future research should explore the links between health and non-health dimensions (economic and social outcomes) and violence, to produce estimates for use in the modeling of prevention cost-effectiveness and benefit. Further work is also needed to map and measure all relevant VAWG health sequelae in longitudinal data sets to capture the full long-term health impact of VAWG and, consequently, the health benefits of prevention.

Analysts and practitioners may encounter some obstacles in the implementation of this methodology. First, it is possible that the data are not available at the level of detail required for a micro-costing or impact measurement in line with our recommendations. In such cases, for costing, analysts should adopt a top-down gross costing approach and include the best available outcomes for impact measuring, acknowledging in both instances that they are deviating from the preferred recommendation.

Improved costing and impact measurement to account for the data needs of economic evaluations will be a critical part of the effort to scale-up and mainstream VAWG prevention. We hope that these methods provide a sound and comparable starting point to this effort.

Author Biographies

Giulia Ferrari, PhD, is a research fellow in economics at the London School of Hygiene & Tropical Medicine. She works on outcome measurement in economic evaluation, including well-being measures, and uses applied econometrics to investigate the role of economic and psychological factors in mediating the impact of prevention programming on perpetration of and exposure to violence against women and girls (VAWG).

Sergio Torres-Rueda, MSc, is a research fellow at the London School of Hygiene & Tropical Medicine. He specializes in the economic evaluation of health-related interventions, particularly in the areas of HIV and VAWG. He focuses on cost-effectiveness analyses alongside randomized controlled trials in low- and middle-income countries (LMICs).

Christine Michaels-Igbokwe, PhD, is a postdoctoral fellow at the University of Calgary. Her research interests include economic evaluation and stated preferences for health care services and policy interventions. Her work includes VAWG prevention and, more recently, on elicitation of individual preferences and economic evaluations of interventions for child and adolescent health in Canada.

Charlotte Watts, PhD FMedSci, is chief scientific adviser at the Department for International Development seconded from the London School of Hygiene & Tropical Medicine, where she is professor of social and mathematical epidemiology. She has advised the World Health Organization (WHO); led a systematic review of the global health burden of violence, and trials showing that violence is preventable; and published more than 200 academic articles.

Rachel Jewkes, MD, is the executive scientist for research strategy in the Office of the South African Medical Research Council (SAMRC) President, and an Honorary Professor at the University of the Witwatersrand School of Public Health, South Africa. She has led research on the intersection of gender inequity, gender-based violence, and health, publishing 300+ academic articles.

Anna Vassall, PhD, is professor of health economics at the London School of Hygiene and Tropical Medicine. She specializes in the economics of HIV, sexual and reproductive health, VAWG prevention, and tuberculosis in LMICs. She is leading the first large multicountry study on the cost-effectiveness of VAWG prevention programming.

Appendix

Violence Against Women and Girls (VAWG) Guidelines in Context

Table A1.

iDSI Reference Case and VAWG guidelines.

iDSI Statements of Principle (Adapted with permission from: https://www.idsihealth.org/resources/summary-table/)	VAWG Guidelines
1. Clear and transparent communication to enable the decision maker(s) to interpret methods and results (also in GHCC 1, 2, and 17).	Adherence to these guidelines is recommended, and reasons for deviation should be provided.
2. Comparator(s) to accurately reflect the decision problem (see also GHCC 1).	These guidelines describe cost data collection and analysis methods for the purpose of economic evaluation of VAWG prevention interventions, and have wider applicability to behavioral change and development interventions. A PICO statement is used to describe the intervention. This identifies all key elements of the decision problem and excludes any non-pertinent elements (also see “comparator” section)
3. Consider all available evidence relevant to the decision problem.	Include all evidence from the literature that may be relevant to the decision problem. Where this information is not available, direct communication with policy makers may provide the required information.
4. Health outcome measure is (a) appropriate to the decision problem, capturing positive and negative effects on length and quality of life, and (b) generalizable across disease states (see also GHCC 4).	Intervention outcomes should encompass (a) an effect measure in natural units—For example, years free of VAWG; (b) a health outcome measure, namely, DALYs; (c) psychological and socioeconomic measures capturing outcomes likely to be affected by the intervention; and (d) measures of well-being, to capture overall well-being impact and as a proxy for the utility individuals derived from the intervention.
5. All differences in the expected resource use and costs of delivery between intervention and comparator should be incorporated into the evaluation.	Studies should implement a bottom-up approach based on micro-costing for direct costs. Deviation from this recommendation is acceptable for management costs, which could be computed using a top-down approach. Research costs are recorded and reported but excluded from cost-effectiveness estimates. Data are to be collected with a view to informing scalability analyses.
6. Time horizon should be of sufficient length to capture all costs and effects relevant to the decision problem. An appropriate discount rate should be used to discount cost and effects to present values (see also GHCC 5).	The time horizon covers intervention adaptation and setup (i.e., training of local staff and community entry), in addition to implementation. It also includes intervention development where interventions are new and have never been tested for cost-effectiveness. Costs should be reported separately for each of these phases. Total development costs are to be reported separately only and not included in the cost-effectiveness estimates. A 3% discount rate is to be used to capture time preferences effectiveness.
7 . Non-health effects and costs that do not accrue to the health budget should be identified where relevant to the decision problem. All costs and effects should be disaggregated, either by sector of the economy or by whom they are incurred (see also GHCC 4).	Most of the costs incurred to develop and deliver VAWG prevention interventions fall under non-health budgets. We recommend quantifying and analyzing all these costs. The micro-costing approach allows for disaggregation of costs by contributing sector, where relevant.
8. The cost and effects of the intervention on sub-populations should be explored and implications characterized (see also GHCC 5, which also includes variation by site/organization).	We recommend analysts collect cost data with sufficient variation in geography, size, and platform type to analyze how these key cost drivers may affect overall intervention cost. The cost-effectiveness analysis should report on pre-specified sub-populations for trials (i.e., sub-populations listed in the trial protocol) and on sub-populations likely to differ substantially from the average in terms of costs and/or cost-effectiveness (also see iDSI Principle 8).
9. Uncertainty should be appropriately characterized (see also GHCC 16).	Model and parameter uncertainty is to be addressed with uni- and multivariate sensitivity analysis. Monte Carlo simulations are to be used to explore precision of the estimates. Intervention sustainability is to be investigated based on changes in VAWG impact post-exposure to the intervention, where the data are available.
10. The impact of implementing the intervention on health budget and on other constraints should be clearly and separately identified.	Where interventions are successful and likely to be funded, budget impact analysis will be conducted for the national or local government. Where clients or target populations incur costs to access services, we also recommend a societal perspective. A societal perspective should also include productivity losses due to VAWG and may include costs to non-health sectors such as criminal justice or social services.
11. An economic evaluation should explore the equity implications of implementing the intervention.	Analysts should explore potential inequalities in access and ability to benefit connected to baseline health or socioeconomic characteristics, or levels of VAWG exposure via, for example, subgroup analyses.

Note. This table contains a summary of our guidance for conducting economic evaluations of VAWG prevention and women’s empowerment interventions in low- and middle-income countries. The reader is encouraged to refer to the main body of the article for in-depth explanations. Our accompanying costing guidelines (at https://bit.ly/2vKaGZZ), the iDSI Reference Case and the GHCC Reference Case websites offer further insights into the concepts referred in the table. iDSI = International Decision Support Initiative; VAWG = violence against women and girls; GHCC = Global Health Costing Consortium; PICO = Population Intervention Comparator Outcome; DALYs = disability-adjusted life years.

Table A2.

Additional GHCC Reference Case Recommendations (Costs) and VAWG Guidelines.

GHCC Recommendations (Adapted with permission from: https://ghcosting.org/pages/standards/reference_case)	VAWG Guidelines
2. The perspective of the cost estimation should be defined.	We recommend transparency in perspective. The analyst should declare if the economic evaluation takes the perspective of the NGO provider or the health sector.
3. The type of unit cost estimated should be defined in terms of economic versus financial, real-world versus normative best practice, and full versus incremental cost, and whether the cost is net of future cost savings. The type of cost should be justified relevant to purpose.	We recommend the estimation of real-world full economic costs of developing and delivering the intervention and, where relevant, its “usual care” comparator. We do not recommend computing future cost savings, due to the uncertainty surrounding estimates of costs to society of VAWG.
4. The “units” in the unit costs for strategies, services, and interventions should be defined, relevant for the costing purpose, and generalizable.	We identify units relevant to a range of programs and settings (see Table 3). We recommend unit costs relevant to the type of program and setting be estimated and reported.
6. The scope of the inputs to include in the cost estimation should be defined and justified relevant to purpose. Where inputs are excluded for pragmatic reasons, these should be reported.	To identify all relevant inputs, and exclude all inputs that are not relevant, we recommend analysts compile a detailed description of inputs for the intervention development, adaptation, and implementation phases. For a template, see the intervention description workbook attached to our guidelines, downloadable from the project resources pages of the What Works? Program: https://bit.ly/2vKaGZZ
7. The methods for estimating the quantities of inputs should be described, including methods, data sources, and criteria for allocating shared costs, and the exclusion of research costs.	Studies should implement a bottom-up approach based on micro-costing for direct costs. For overhead and support costs, a top-down approach based on micro-costing is deemed acceptable, if micro-costs are not available (for definitions, see our guidelines at https://bit.ly/2vKaGZZ).
8. The sampling frame, method, and size should be determined by the precision demanded by the costing purpose and designed to minimize bias.	VAWG prevention efforts often rely on high numbers of staff and volunteers. We therefore recommend sampling a subset of sites to capture variation in service delivery. This ensures the sample is representative of the different settings where the intervention is delivered and that the data can be used in modeling scale-up analyses.
9. The selection of the data source and methods for estimating “units” for unit costs should be described, with potential biases reported in the study limitations.	We recommend identifying relevant units for unit costs based on a review of the literature in the intervention’s specific field, in the first instance. However, where publications are scarce, we also recommend interviewing implementation managers and developers to identify useful units not included in the literature.
10. Consideration should be given to the timing of data collection to minimize recall bias and, where relevant, the impact of seasonality and other differences over time.	We advise prospective data collection. However, where this is not feasible due to project timetable constraints, we advise that retrospective data collection takes place as soon as possible after end of delivery.
11. The sources for price data should reflect the price relevant to purpose and be described for each input in a way that allows for adjustment across settings.	We recommend analysts report the sources for each price used.
12. Capital costs should be appropriately amortized or depreciated to reflect the expected life of capital inputs	Capital costs are first expressed in real prices at the base year and then annuitized based on their expected life using standard tables (see, for example, Drummond et al., 2014).
13. Where relevant, appropriate discount, inflation, and currency conversion rates should be used and clearly stated.	Base case: We recommend the use of a standard 3% discount rate, the use of the GDP deflator as an inflation index, and the use of market exchange rates to convert currencies. Sensitivity analyses: Local discount rates, market exchange rates adjusted for PPA rates to take differences in price levels into account; “VAWG”-specific PPP and substitution of local prices with prices from other localities, where comparisons are being made.
14. The use and source of shadow prices, for goods where no market price exists, and for the opportunity cost of time should be reported.	Base case: We recommend the use of replacement costs as a pragmatic option in the absence of market prices. Data sources should be clearly identified and documented in the tools. Sensitivity analysis: The opportunity cost of using specific resources is computed, and sources reported.
15. Variation in the cost of the intervention by site size/organization, sub-populations, or by other drivers of heterogeneity should be explored and reported.	We recommend analysts collect cost data with sufficient variation in geography, size, and platform type to analyze how these key cost drivers may affect overall intervention cost. The cost-effectiveness analysis should report on pre-specified sub-populations (for trials), and on sub-populations likely to differ substantially from the average in terms of costs and/or cost-effectiveness (also see iDSI Principle 8).

Note. This table contains a summary of our guidance for conducting economic evaluations of VAWG prevention and women’s empowerment interventions in low- and middle-income countries. The reader is encouraged to refer to the main body of the article for in-depth explanations. Our accompanying costing guidelines (at https://bit.ly/2vKaGZZ), the iDSI Reference Case and the GHCC Reference Case websites offer further insights into the concepts referred in table. GHCC = Global Health Costing Consortium; VAWG = violence against women and girls; NGO = nongovernmental organization; GDP = gross domestic product; PPA = purchasing power adjusted; PPP = purchasing power parity; iDSI = International Decision Support Initiative.

Table A3.

Sensitivity Analyses.

Source of Uncertainty	Alternative Values/Scenarios	Recommended Sensitivity Analysis
Methodological
Discount factor	Local/national level discount factor; differential discount factors for costs and outcomes	Deterministic uni- and multivariate
PPP	World Bank PPPs, and VAWG prevention-specific PPP, where interventions are sufficiently similar	Deterministic uni- and multivariate
Time horizon	Lifetime	Probabilistic
Perspective	Societal	Probabilistic
Valuing the time of free human resources	Types of time forgone	If the volunteer is unemployed, analysts will use the gross wage in the volunteer’s potential occupational group; the individual’s gross wage, if he or she works part-time and volunteers the rest of the work time; the individual’s net wage if he or she is employed but volunteers his or her free time (Posnett & Jan, 1996).
Parameter
VAWG exposure	Vary based on evidence; expert opinion and best- or worst-case scenario	Deterministic uni- and multivariate; threshold analysis and probabilistic sensitivity analysis
DALY impact
Cost of treatment
Precision of parameter estimates (stochastic uncertainty)		Monte Carlo simulations
Subgroups with heterogeneous cost-effectiveness
Age		Multivariable sensitivity analysis to test robustness. Could include analysis where robustness of results is tested across the entire distribution (e.g., quintile regressions).
Gender
Baseline exposure to VAWG
VAWG risk factors
Alcohol consumption  Mental health issues  Small children  Pregnancy  Age at sexual debut  Age at marriage  Employment status
Urbanicity
Site accessibility

Note. PPP = purchasing power parity; VAWG = violence against women and girls; DALY = disability-adjusted life year.