Evaluation of HIV prevention programmes: the case of Avahan

Current calls to action for HIV prevention programming stress the need for more strategic, combination prevention strategies along with evaluation components to measure effectively the impacts of such efforts. The Avahan project, implemented since 2003, is one example of a combination prevention programme for a concentrated epidemic that offers a clear example of what can be achieved when a programme is based on significant ‘know your epidemic’ research before implementation, and when data are used to optimise a delivery programme to maximise coverage of services. The rapid scale-up, the documentation of quality and coverage indicators, and perhaps most importantly, the application of modern, data-driven management tools to the delivery of a large-scale public health programme, are Avahan's most exemplary qualities.

Avahan is surely the world's largest HIV prevention programme in a country with a concentrated epidemic and perhaps the largest anywhere. It focuses on reaching the populations key to the spread of HIV in India: sex workers and their clients, men who have sex with men (MSM) and drug users. The team's experience with management information systems allowed them to monitor closely the implementation of services and in close to real-time adjust their practices to improve the quality and coverage of their services.

In addition to the ‘know your epidemic’ mapping of populations at risk, Avahan used market research to inform the design of their service delivery, ensuring that the facilities were conveniently located, that they were responsive to the specific needs of the population being served, etc. Had they been selling shoes or software, they would have been able to measure their success from the day they opened for business based on revenues minus expenses. Unfortunately, however, the connection between sales of HIV prevention services (coverage and uptake) and revenue (HIV infections averted) is far less clear. As a result, it is not obvious how much of the apparent decline in HIV prevalence in the states where the programme operated can be attributed to the Avahan programme. Although it seems reasonable to assume that the massive scale-up in delivery of prevention services played an important role in contributing to the decline in prevalence, the apparent timing of the decline has raised questions about whether the decline was occurring independent of the programme efforts.¹ The likely effectiveness of behavioural HIV prevention interventions when implemented in various contexts is difficult to predict. A programme manager cannot assume that a package of ‘best practice’ interventions derived from other settings will result in reduced HIV incidence in his/her context. This is further complicated by the fact that there are often scarce efficacy/effectiveness data on some interventions in any setting. Given these uncertainties, it is essential that large programmes such as Avahan collect data that permit rigorous assessment of the impact on HIV incidence. If such data can be collected as the programme is implemented from different sites (preferably randomly determined), which receive different variations of the programme (varied by timing, intensity, or composition/character of the components of the programme), then it will be possible to measure impact, and (if successful) to make the case convincingly both for further financial support and for adapting/expanding the programme to other settings. It is also possible for the evaluation design to provide feedback long before implementation is complete, so that programmes can make changes to improve effectiveness on an ongoing basis.

The papers included in this special issue document the broad range of monitoring and evaluation activities that have accompanied the implementation of the Avahan programme, highlight some of the evaluation lessons learnt and provide an overview of the approaches that the different papers have taken to the evaluation of Avahan.

Although Avahan provides many examples of quality evaluation, had they built a prospective, more robust evaluation of impact into its implementation, they could have more convincingly demonstrated effectiveness. The impact of Avahan has been assessed through repeat cross-sectional surveys combined with modelling to confirm an effect over time; promising and highly suggestive, but not definitive. With regard to process indicators such as coverage, quality and tailoring of services, Avahan is unsurpassed; they were able to fine-tune the programme with regard to such outcomes on an ongoing basis. However, with regard to impact, might Avahan represent a missed opportunity? For example, it is unclear why they could not have implemented the programme in a randomised phased fashion (eg, stepped wedge design). They could also have included a cohort to measure incidence directly, more comprehensive baseline data, or control communities. Had Avahan used an adaptive design that permitted early peeks at outcomes, they would have been able to tailor the programme based on trends in impact (in addition to trends in coverage and quality) further optimising the likelihood of achieving the greatest impact.

Coverage and uptake: If a programme cannot achieve sufficient coverage or the target population is unwilling to participate, or if there is little uptake, then successful outcomes are unlikely. Evaluating these components may reveal that coverage was poor as a result of inefficient implementation or that poor uptake was linked to culturally inappropriate content etc. Verma et al² described methods for assessing the scale-up and coverage of large-scale HIV prevention services provided to female sex workers (FSW) and high-risk MSM during the first 5 years of Avahan. Results revealed impressively high levels of coverage as verified by survey data after 60 months: 75% of the denominator intended to be covered (69 districts) was met monthly; 85% FSW and 64% high-risk MSM ever contacted had used sexually transmitted infection services at least once.

By using data on condom sales and distribution data from programmes and the private sector, as well as sexual behaviour data obtained from general population surveys, Bradley et al³ assessed the proportion of FSW who used condoms with clients and other partners. Here they combined empirical data with modelling techniques that permitted comparisons of scenarios that discounted the number of condoms that might have been purchased for non-commercial sex. These data should be helpful in validating actual condom use when triangulated with other available data, such as self-reported condom use.

Finally, Blankenship et al⁴ assessed programme uptake by examining predictors of participation among FSW by using two rounds of serial cross-sectional surveys. Awareness of and participation in the programme increased among women who were publicly willing to admit that they were sex workers.

Quality and fidelity of interventions: Mogasale et al⁵ assessed the overall quality of coverage (towards all targeted risk groups) on a routine quarterly basis using an 80-question survey to assess the coverage, quality of services, use of referral networks, community involvement and technical support. This routine quantitative score revealed a significant improvement in programme delivery over time. Two other papers examined the association between coverage and behaviour change.

Ramakrishnan et al⁶ measured coverage by assessing whether FSW received the three core services of the programme coverage, and then examined whether this coverage was associated with knowledge and use of condoms. Lipovsek et al,⁷ who examined coverage levels from the behaviour change communication component of the programme in association with consistent condom use among male clients of FSW, determined that exposure to the programme had positive effects on these outcomes.

Effectiveness: Alery et al⁸ examined the association between prevalent infections among sex workers and their clients and antenatal care HIV (ANC) prevalence as a way to examine the contribution of commercial sex to infection in the general population. This cross-sectional assessment was conducted towards the end of the programme. Results confirmed that infection rates among FSW were positively associated with the ANC infection rate, which validated the targeting of the Avahan programme and indicated the need for continued focus in this group.

Pickles et al⁹ confirmed the findings of Alery et al⁸ using a deterministic compartmental HIV/STI transmission model parameterised using data from Belgaum and Mysore districts in south India. Multiple model fits to STI/HIV data from FSW examined whether all prevalent HIV/herpes simplex virus type 2 infections in each district could be accounted for solely by HIV/herpes simplex virus type 2 transmission between FSW/clients, and to their non-commercial partners. Although uncertainty about client population size reduced model accuracy, they nevertheless concluded that for men, most HIV infections were caused by sex with FSW, whereas for women most infections are bridging infections from their male partners who are also clients of FSW.

Finally, Ramesh et al¹⁰ came closest to examining the impact of Avahan on FSW using dynamic transmission models. They used prevalence from two cross-sectional surveys of randomly sampled FSW in five Karnataka districts to assess changes over time in STI/HIV prevalence, condom use and HIV prevention programme exposure. The survey was conducted in 2004 and 2005 and then again 27–37 months later. They also assessed exposure to the programme by asking if the FSW had been contacted by a peer educator. The results indicated a significant decrease in HIV, syphilis, gonorrhoea and chlamydia, and increased condom use with regular clients.

Costs and cost-effectiveness: Chandrashekar et al¹¹ presented excellent monitoring of costing data and noted that during the rapid scale-up of the programme, a significant reduction in average costs of the programme was observed between start-up and the first 2 years of implementation. Furthermore, large-scale interventions should quantify scale effects when planning for future resource requirements.