Evaluating Delivery Systems: Complex Evaluations and Plausibility Inference

Abstract

Delivery system evaluation is poorly defined and therefore a barrier to achieving increased coverage of interventions. We use a pre- and post-implementation cross-sectional observational study with assessment of the intermediate processes to evaluate a new delivery system for insecticide-treated nets (ITNs) in two regions of Ghana. In Volta Region, ownership of at least one net rose from 38.3% to 45.4% (P = 0.06), and 6.5% of respondents used a voucher in the purchase. In Eastern Region, ownership of a net rose from 13.7% to 26.0% (P < 0.001) and 0.5% of households used a voucher to purchase a net. Just 40.7% and 21.1% of eligible antenatal clinic (ANC) attendees were offered a voucher in Volta and Eastern Regions, respectively, and 36.0% and 30.7% used their voucher in the purchase of an ITN. Without attributing nets to the specific delivery system, in Eastern Region the success of the new system would be overestimated.

Introduction

There are many efficacious interventions, which have the potential to reduce morbidity and mortality and achieve the Millennium Development Goals (MDGs) if they are delivered effectively.^1,2 However, coverage of many of these interventions remains inadequate, limiting the achievement of improvements in population health. Improving coverage requires focusing attention on the systems through which interventions are delivered, and adapting the methods traditionally used to assess the clinical effectiveness of interventions to answer questions about the effectiveness of delivery systems at increasing coverage with essential health interventions.² Advances within this field of research are hampered by a plethora of sometimes distinct, but often overlapping terminologies and concepts. Clarity in the concepts and methods of evaluating effectiveness of interventions and their delivery systems would strengthen the evidence base for effective public health policies and programs.

Methods for assessing the effectiveness of interventions have been addressed in the literature on program evaluation^3–6 and complex evaluations;⁷ both literatures highlight the importance of understanding and assessing the intermediate steps (causal chain) between implementation and outcomes. The assessment of this causal chain has been defined by the UK Medical Research Council as process evaluation.⁷ The assessment of outcomes is seen as the priority for assessing the effectiveness of an intervention in both literatures, but the study designs used vary. The program evaluation literature has suggested that the “gold standard” randomized controlled trial (RCT) is not only operationally difficult and resource heavy, but may also be inappropriate.⁵ The use of observational studies with different levels of inference (probability, plausibility, and adequacy) that the outcome was a result of the intervention, has been proposed.³ The complex evaluation literature recommends that experimental designs (RCT, stepped wedge designs) should be used where possible, but recognizes that they are not always appropriate.⁷

Methods for assessing the effectiveness of delivery systems for public health interventions have not been defined. We propose that the approaches to evaluating delivery system effectiveness can be simpler than those for evaluating intervention effectiveness. Where the objective is to assess the effectiveness of a system that delivers, interventions of known effectiveness, and particularly its ability to achieve scaled up delivery of these interventions, the appropriate outcome measure is coverage of the intervention.

We use a case study of an evaluation of an insecticide-treated net (ITN) voucher scheme in two regions of Ghana, to show how pre- and post-implementation cross-sectional surveys with attribution of the source of intervention (ITN) can provide a plausibility level of inference that coverage outcomes were caused by the new delivery system. In this context the clinical effectiveness of the intervention (ITNs) has been proven through clinical trials and effectiveness trials in a range of settings, but the effectiveness of the delivery system (the voucher scheme) is yet to be demonstrated.

Case Study

Background.

Before introduction of the ITN voucher scheme in 2004, the proportion of households owning at least one net was 46.1% in Volta Region and 10.3% in Eastern Region.⁸ Mosquito nets had been delivered through diverse systems in both regions, with targeting of subsidies to vulnerable groups through the public sector, and promotion of widespread availability and distribution of ITNs through the private sector. Public sector delivery of nets was through the sale of nets to pregnant women and children less than 5 years of age at a price of US$2.2 in health facilities. These nets were distributed to districts from the National Malaria Control Program (NMCP). Some District Health Management Teams (DHMTs) also procured their own nets from the commercial market and sold them through health facilities using the pricing and targeting strategies recommended by the NMCP. The nets were of a variety of types: factory pre-treated, bundled with insecticide, and untreated nets with separate insecticide treatment tablets, depending on donations and availability.

As of 2003, several private sector partners were involved in the promotion and distribution of mosquito nets including: AgriMat, Vestergaard Frandsen, Transcol, and NetMark (Accra, Ghana). AgriMat marketed Dawa net, which is a factory pre-treated ITN, K-O net, which is an untreated net bundled with insecticide, and K-O Tab, which is a deltamethrin tablet for the (re)treatment of nets. By 2003 AgriMat were supplying nets to all regions of the country through their agricultural distribution networks, with the exception of the northern regions where they were not able to compete with the highly subsidized United Nations Children's Fund (UNICEF) nets. Delivery points for AgriMat products were within the formal retail sector including pharmacies, chemical sellers, and general shops. Vestergaard Frandsen marketed PermaNet, a long lasting insecticidal net (LLIN), through outlets including Total and Mobil petrol stations, pharmacies, and supermarkets. Their products were distributed by Transcol. NetMark, established by United States Agency for International Development (USAID) with the aim of increasing coverage with ITNs through partnerships with private sector companies, launched their activities in Ghana in November 2002, and worked with all the previously listed importers, wholesalers, distributors, and retailers. Formal private sector delivery points included general shops, clothes/fabric shops, wholesalers, pharmacies, and chemical sellers (private sector outlets, which are static and the goods remain at the point of sale overnight).

Within the informal retail sector the majority of mosquito nets were untreated and made from a variety of materials and fabrics,⁹ ITNs were rarely found in the informal retail sector. Informal private sector outlets included markets, local kiosks, table top vendors, and street hawkers (private sector outlets that are non-static where the goods are stored elsewhere overnight).¹⁰

Intervention evaluation.

In 2004, the Ghana National Malaria Control Program (NMCP), supported by the Department for Internal Development (DFID) and USAID, introduced an ITN voucher scheme with the aim of increasing coverage of ITNs. The scheme started in Volta Region, and within a few months it was scaled up to the adjacent region (Eastern Region). In the voucher scheme pregnant women attending antenatal clinic (ANC) were eligible for a voucher, which entitled them to a US$2.2 discount on an ITN. This discount voucher was to be used together with a “top-up” cash payment, to purchase an ITN in the retail sector. Voucher scheme ITNs were available only in outlets that had agreed to take part in the voucher scheme. The intervention therefore consisted of delivery of a subsidy in the public sector and delivery of the ITN in the formal private sector. Informal private sector providers were not invited to take part in the voucher scheme.

This intervention was implemented within the context of the pre-existing delivery systems for mosquito nets, as described previously (Figure 1). At the time of implementation of the ITN voucher scheme pilot in the two regions, the national strategy for delivery of ITNs to pregnant women was direct delivery for US$2.2 on visiting ANC. To avoid confusion, the decision was taken by the NMCP in agreement with the Regional Health Management Teams (RHMTs) that they would not supply ITNs to health facilities in Volta and Eastern Regions during the period of the voucher scheme pilot. However, during the period of the pilot many of the health facilities did actually receive ITNs from both public sector and private sector suppliers. The result of this was that during implementation of the voucher scheme some midwives exchanged vouchers for ITNs within the health facility. This was not an intended strategy for the voucher scheme and therefore ITNs delivered in this way are not attributed to the voucher scheme.

Figure 1.

Categorization of delivery systems of nets (internal comparators).

Pre- and post-implementation household surveys were undertaken in March 2004 (pre) and April 2005 (post) in Volta Region and in July 2004 (pre) and July 2006 (post) in Eastern Region. A stratified multistage cluster sampling method was used to select households across each of the two regions for both surveys. Two districts in each of the northern, central, and southern zones of each region were selected using probability proportional to population size (PPS). Thirty clusters (villages) within each of these districts were selected using PPS. Seven households were randomly selected per cluster using a modified Expanded Program of Immunization (EPI) sampling technique. Households were sampled if they had either a currently pregnant woman or a mother of a child less than 1 year of age (< 1) in the household. The same districts in each region were sampled for the post-implementation surveys, and the same sampling procedures were used to select the 30 clusters within each of the districts and seven households within each cluster. Thus, the selected clusters and households in the pre- and post-implementation surveys are independent samples. The total number of households by region and district for each survey are shown in Table 1.

Table 1.

Distribution of sample households

Volta Region			Eastern Region
District	Pre-implementation	Post-implemenation	District	Pre-implemenation	Post-implemenation
Krachi	197	204	Kwahu South	212	245
Jasikan	197	211	Manya Krobo	211	166
Kpando	168	168	East Akim	209	251
Hohoe	247	251	Suhum Kraboa Coaltar	211	168
Ketu	308	309	Akwapim South	211	149
South Tongu	115	111	Birim South	211	247
Total	1,232	1,254		1,265	1,226

Control groups and attribution.

To assess the effectiveness of the voucher scheme the other delivery systems in the two regions (see above) were defined as internal comparison groups. No external control arm was used because 1) the scheme was implemented in all districts of each of the two regions and 2) differences in contextual factors between the study regions and the other bordering regions would have precluded the usefulness of such an external control.

Attribution of nets in households to the delivery system through which they reached the household was achieved through the use of questions on the source of the net.

Respondents were asked two questions to determine the source of each of the mosquito nets owned in the household. 1) Where did you get this mosquito net? 2) Did you use a voucher to pay for this net?

Process evaluation.

Four intermediate steps in the delivery and use of a voucher subsidy were defined, which describe the causal pathway through which a mosquito net reaches a household. Four questions were included in the post-implementation survey in each region to assess these intermediate steps. 1) Whether the pregnant woman or mother of a child less than 1 year of age had attended ANC during her current/last pregnancy; 2) whether she was offered a voucher for a mosquito net during this visit; 3) whether she accepted the voucher; and 4) whether she had used the voucher to purchase a net.

Analysis.

On the basis of the reported sources, nets were categorized to public, formal private, or informal private sector nets. Formal private sector nets were further classified to unsubsidized net or voucher-subsidy net. The proportion of households having any net and the voucher-subsidy net was compared between pre- and post-implementation time points in the regions separately. Analyses were conducted using STATA 10 software (STATA, College Station, TX) adjusting for the cluster design of the surveys. Pearson's design-based F test was used to test the significance of the differences in proportions between the two cross-sectional surveys in each district.

Results

Across the four surveys in the two regions 2,019 nets were reported in the sampled households. The source of 1,891 (93.7%) of these nets was reported as known by the respondent. The remaining 128 were from sources that the respondent was not able to identify (including gifts).

Effectiveness of the voucher scheme.

In Volta Region, ownership of mosquito nets rose from 38.3% to 45.4% (P = 0.06) pre and post implementation of the voucher scheme Table 2. Formal private sector nets purchased with a voucher subsidy reached 6.5% of households. In Eastern Region, the proportion of households owning at least one net rose during one year's implementation of the voucher scheme from 15.0% to 26.0% (P < 0.001). However, formal private sector nets purchased with a voucher subsidy reached only 0.5% of households.

Table 2.

Delivery system outcomes pre and post implementation of the insecticide-treated nets (ITNs) voucher scheme

Source of net	Volta Region		Eastern Region
	Pre-implementation n (%)	Post-implementation n (%)	Pre-implementation n (%)	Post-implementation n (%)
Public sector
Directly subsidized	37 (3.0)	65 (5.2)	54 (4.3)	173 (13.7)
Voucher subsidized	0	43 (3.4)	0	49 (3.8)
Total	37 (3.0)	105 (8.4)***	54 (4.3)	222 (17.6)***
Formal private sector
Unsubsidized	18 (1.5)	50 (4.0)	27 (2.1)	11 (0.9)
Voucher subsidized	0	82 (6.5)	0	6 (0.5)
Total	18 (1.5)	130 (10.4)***	27 (2.1)	17 (1.4)
Informal private sector
Unsubsidized	392 (31.8)	341 (27.2)	93 (7.4)	55 (4.4)*
Total with at least 1 net	472 (38.3)	571 (45.4)	190 (15.0)	328 (26.0)***
Survey population	1,232	1,254	1,265	1,260

^*P ≤ 0.05;

^**P ≤ 0.005;

^***P ≤ 0.001.

An assessment of the change in proportion of nets reaching the household by alternative delivery systems pre and post implementation of the voucher scheme provides greater insight into the impact of the voucher scheme in each of the regions. In Volta Region, before the implementation of the voucher scheme, 3% of households owned at least one net that they got from the public sector, 1.5% from the formal private sector, and 31.8% from the informal private sector. Post implementation of the voucher scheme households with at least one public sector net increased to 8.4% (P < 0.001), households with a formal private sector net to 10.4% (P < 0.001), and households with an informal private sector net decreased to 27.2%. Approximately 60% of those households who got a net through the formal private sector used a voucher in the purchase. Surprisingly, 3.4% of households reported using a voucher in the process of acquiring a net through the public sector. The voucher-subsidy net was used by 9.9% of households (3.4% public sector and 6.5% formal private sector).

In Eastern Region, before implementation of the voucher scheme, 4.3% of households owned at least one net that they got from the public sector, 2.1% from the formal private sector, and 7.4% from the informal private sector. Post implementation of the voucher scheme, 17.6% (P < 0.001) of households owned a mosquito net from the public sector, 1.4% from the formal private sector, and 4.4% (P = 0.05) from the informal private sector. The increase in households owning at least one net during the voucher implementation period was through the public sector delivery system. Fifteen percent of households purchased a net directly through a public sector outlet. As in Volta Region, 4.3% of households reported purchasing a net through a public sector outlet and using a voucher subsidy in this purchase. This means that the voucher was used to purchase a net in the clinic, rather than at a retail outlet.

Delivery processes.

To benefit from the ITN voucher scheme, women must attend ANC. Attendance at ANC at least once was high, particularly in Volta Region, 92.2% and 84.4% of currently or recently pregnant women attended ANC in Volta and Eastern Regions, respectively (Figure 2). However, only 12.5% of respondents in Volta Region and 4.7% of respondents in Eastern Region said that they had used a voucher to buy a net. Examination of the four intermediate steps in the delivery process identified, shows clearly that there were two processes where the delivery of nets by the voucher subsidy broke down. These delivery disorders arose in the process of offering a voucher to eligible ANC attendees (only 40.7% and 21.1% in Volta and Eastern Regions, respectively), and in the process of using a voucher in exchange for a mosquito net (36.0% and 30.7% in Volta and Eastern Regions, respectively).

Figure 2.

Insecticide-treated nets (ITNs) voucher scheme delivery processes in Volta and Eastern Regions.

Discussion

The aggregate increase in the proportion of households owning at least one mosquito net was of borderline significance in Volta Region and significant in Eastern Region. These changes are as could be expected with implementation of a complex system such as that of a voucher scheme, in comparison with a more rapid increase in coverage that could be expected from less complex delivery systems such as campaigns. The simple pre- and post-implementation evaluation of delivery outcomes suggested that, particularly in Eastern Region, an adequate increase in the proportion of households owning at least one mosquito net occurred within the timeframe of implementing the voucher scheme.

Using information linking the coverage achieved to the specific delivery systems presents a different picture of the effectiveness of the new delivery system. Using the reported delivery point to attribute the nets in households to the specific system through which they were delivered and reached the household, suggested that only 6.5% and 0.5% of households with pregnant women or a child less than 5 years of age in Volta and Eastern Regions, respectively, had a mosquito net that was delivered in a formal private sector outlet by the voucher scheme. In Eastern Region, the majority of the increase in the coverage of ITN during the first year of implementation of the voucher scheme was caused by direct delivery of mosquito nets through the public sector.

Assessment of the delivery processes provided additional evidence that it was not plausible that the increase in ITN coverage in Eastern Region was caused by the voucher scheme because only 4.4% of respondents in sampled households reached the endpoint of the delivery system by using a voucher to purchase a mosquito net. This delivery process evaluation also identified the points at which there were “disorders” in the delivery system impeding its effective operation.

Assessing the household ownership of nets pre and post implementation of the voucher scheme provided a measure of the aggregate change in coverage that occurred during this time. However, where there is more than one delivery system operating for a particular intervention, distinguishing the contributions of each individual system to aggregate coverage becomes important. In this case, it would have been inappropriate to infer that the total change in coverage was caused by the voucher scheme. In fact if this interpretation had been made, the effectiveness of the voucher scheme would have been over estimated. Attributing the nets in the household to the system through which they were delivered provided a strong inference on the proportion of the measured change that was caused by each specific delivery system, including that of the new voucher delivery system.

Although plausible inferences can be reached using pre-post implementation studies and this study design is appropriate for the evaluation of delivery systems, one should consider the limitations and advantages with respect to other methodologies. The cross-sectional plausibility study used here is based on the comparison of outcomes in an intervention group with a non-randomized control group. Such observational studies with non-randomized controls have generally weaker internal validity than studies using randomized controls. However, robust sampling techniques would improve this validity by reducing selection bias and random errors. Our pre-post evaluation of coverage did not assess or adjust for any contextual factors, and we recognize this limitation. Nonetheless, the use of alternative delivery systems as internal controls has the advantage that each of the systems will be influenced by the same temporal trends in external contextual factors.

The discordance between the results in the two regions suggests that the voucher scheme was highly dependent on contextual factors. There are differences between the two regions in terms of the socio-economic status of the populations, levels of education, levels of economic activity, and levels of use of mosquito nets. The process evaluation suggests that the most important contextual factors are those that acted at two steps in the causal pathway of implementation of the voucher scheme, namely offer of a voucher to an eligible woman, and use of the voucher by the pregnant woman to purchase an ITN. Qualitative analysis of the factors influencing the success of the individual processes of the voucher scheme is reported elsewhere (Webster J and others, in preparation).

The finding that vouchers were being used for ITNs delivered within the clinic was an unexpected outcome of implementation of the voucher scheme. However, it corresponded to anecdotal evidence that ITNs were being sold in health facilities that had been supplied with vouchers. Because of stock outs of ITNs in the formal private sector outlets in the regions, the NMCP supplied ITNs to the districts for delivery from health facilities to pregnant women. This resulted in ANC staff having both vouchers and ITNs. The use of a voucher for an ITN delivered within a health facility was one of the outcomes of this complex situation.

The delivery of NMCP ITNs to the health facilities was a major contextual change, which resulted in the presence of two directly competing delivery systems for ITNs. The increase in the proportion of households owning a public sector directly subsidized ITN in Eastern Region (4.3–13.7%) compared with Volta Region (3.0–5.2%) over the period of implementation of the voucher scheme, suggests that the impact of this competition may have been greater in the Eastern Region. This difference is likely the result of timing, as the NMCP ITNs were distributed 9 months into the period of evaluation in Volta Region and within 2 months of the voucher scheme beginning in Eastern Region.

It could be argued that a cluster RCT of the voucher scheme with controlled implementation would have provided a better measure of the efficacy of this new delivery system. However, RCTs can have limited external validity even to the whole population of the area in which the trial was conducted in a subsample of the population. Well-conducted cross-sectional observational studies will have good generalizability to the population from which they were sampled. Therefore, if a survey is undertaken at the national level, the findings are then generalizable at the national level. Cross-sectional observational studies with a plausibility inference have the advantage that they are less complex and expensive than studies using randomized controls and are therefore more applicable at scale.

This move to a strong plausibility inference in our case study was achieved through the simple addition of two questions on the cross-sectional household surveys for attribution, and four questions on delivery processes. This study design allows relatively simple assessment of the effectiveness of a delivery system on an operational scale. Minor adaptations to the method would allow an assessment of the impact of competition between two alternative delivery systems for ITNs.

This example was of the delivery of mosquito nets, but the same basic steps would apply to delivery of other public health products such as drugs. The conceptual framework could be further developed for application to the delivery of information through communication interventions. Additionally, our model can enable comparison of delivery system strategies. For example, the effect of applying different pricing strategies, different products, or different communication messages, to an intervention delivered through a specific delivery channel could be evaluated using this approach.

A limitation of this method is that it applies to the evaluation of delivery systems. The novel methodological approach is attribution of coverage and the use of internal controls to enable a plausible inference that the outcomes were because of the intervention. This method is based on the relationship between the delivery system and coverage outcomes, which is the endpoint of a delivery system evaluation. Where an intervention is under evaluation the outcome that must be assessed is that of health impact. Although the effectiveness of a delivery system contributes substantially to the effectiveness, or health outcome of an intervention, other factors such as consistency of use or adherence to dosing schedules may also contribute. The presence of these factors and others that confound the relationship between coverage and health outcome, invalidate the use of this method for evaluation of interventions. It would not be possible to infer that the health outcomes were caused by the intervention over and above the impact of confounding factors. We would suggest, however, that a substantial number of external influences on intervention effectiveness are in fact delivery system factors.

Our cross-sectional observational study design provides a relatively simple method for assessing the effectiveness, and contributing to assessment of the cost effectiveness of delivery systems for public health interventions. Within the context of the recognition that more than one delivery system is needed to reach all target groups,¹¹ this method is able to assess the relative contributions of a number of delivery systems to a single outcome.

In one region (Volta Region), we were able to show strong plausibility that the increase in household ownership of nets was a result of the voucher scheme; and in the other region (Eastern Region) it was implausible that the increase in ownership of nets was because of the voucher scheme. However, we recognize that the differences in contextual factors between the two regions should be taken into account in interpreting this plausible inference.

Conclusion

Our case study highlights that a cross-sectional observational study design conducted pre and post implementation of a new delivery system for ITNs without an external control can provide a strong plausibility inference on the impact of the delivery system. This is achieved by attributing coverage outcomes to the delivery system through which the intervention reached the household. In the absence of this attribution, the impact of the new delivery system would have been overestimated.