An assessment of Lot Quality Assurance Sampling to evaluate malaria outcome indicators: extending malaria indicator surveys

Abstract

Background Large investments and increased global prioritization of malaria prevention and treatment have resulted in greater emphasis on programme monitoring and evaluation (M&E) in many countries. Many countries currently use large multistage cluster sample surveys to monitor malaria outcome indicators on a regional and national level. However, these surveys often mask local-level variability important to programme management. Lot Quality Assurance Sampling (LQAS) has played a valuable role for local-level programme M&E. If incorporated into these larger surveys, it would provide a comprehensive M&E plan at little, if any, extra cost.

Methods The Mozambique Ministry of Health conducted a Malaria Indicator Survey (MIS) in June and July 2007. We applied LQAS classification rules to the 345 sampled enumeration areas to demonstrate identifying high- and low-performing areas with respect to two malaria program indicators—‘household possession of any bednet’ and ‘household possession of any insecticide-treated bednet (ITN)’.

Results As shown by the MIS, no province in Mozambique achieved the 70% coverage target for household possession of bednets or ITNs. By applying LQAS classification rules to the data, we identify 266 of the 345 enumeration areas as having bednet coverage severely below the 70% target. An additional 73 were identified with low ITN coverage.

Conclusions This article demonstrates the feasibility of integrating LQAS into multistage cluster sampling surveys and using these results to support a comprehensive national, regional and local programme M&E system. Furthermore, in the recommendations we outlined how to integrate the Large Country-LQAS design into macro-surveys while still obtaining results available through current sampling practices.

Introduction

The burden of malaria has led to a massive international effort to increase prevention and treatment measures. The past 10 years have brought important advances in malaria research, as well as increases in funding by bilateral and international organizations to support malaria control.¹ This support has aided endemic countries to increase coverage with malaria interventions, including insecticide-treated bednet (ITN) distribution, indoor residual spraying, intermittent preventive treatment of pregnant women and treatment with effective antimalarial drugs.² The scale-up has led to greater emphasis on improving national monitoring and evaluation (M&E) systems for malaria programmes.³ Multistage cluster sample surveys such as the Demographic and Health Survey (DHS) and Malaria Indicator Survey (MIS) provide current estimates for malaria outcome and impact indicators at the national and regional level.^4,5 However, the complexity, cost and time needed for the execution of these surveys preclude their frequent use to monitor malaria control programmes.⁶

Frequent assessments of programme outcomes at a decentralized or sub-national level would permit programme managers to use results-based information when bringing programmes to scale, as well as to satisfy donor reporting requirements. Failure to monitor outcomes on a regular basis at a decentralized level can hide programme inadequacies and inequities, delay necessary action to improve effectiveness and lead to missed opportunities to improve programmes.⁶ As countries scale-up coverage of key malaria interventions, sub-national monitoring is essential for resource allocation and priority setting. Managers need local-level information to effectively steer and guide their programmes so they are responsive to local conditions.

Complementary methods are available to regularly monitor outcomes of malaria control programmes. Lot Quality Assurance Sampling (LQAS) is one such method that has been used to classify geographical and health programme areas based on whether a specified coverage target has been reached.^7,8 A major advantage of LQAS is that it requires smaller sample sizes to classify areas than methods used to perform other estimation analyses.⁹ Another benefit is that LQAS provides otherwise unavailable important information at the local level, where programme managers can take corrective action.¹⁰ Although LQAS has been used previously to assess the efficacy of antimalarial drugs¹¹ and to estimate malaria prevalence,¹² only recently has it been used to assess malaria outcome indicators at the local level.^13–16

The goal of this article is to demonstrate an application of LQAS integrated multistage cluster sampling to achieve information both at national and regional levels. The former also provides a measure of local performance. Specifically, this article uses existing MIS data from Mozambique (2007) to demonstrate methods to determine whether adequate bednet coverage levels have been reached within each enumeration area (EA) of the Mozambique MIS sample.^2,17 We show that solely reporting national and regional indicators hides information about local variation that would be useful for managers who implement programmes in Mozambique. We conclude with a discussion of future designs for national surveys and how they can be designed concomitantly with Large Country (LC) LQAS, a method for integrating LQAS and multistage sampling, for the purpose of programme M&E.

Methods

Data source

In June and July 2007, the Mozambican National Malaria Control Programme (NMCP), in collaboration with other national and international organizations, collected data as part of the MIS. This multistage cluster sample randomly selected households from 21 strata—urban and rural in each of the provinces plus Maputo City—from the ‘mother’ sample created by a three-stage cluster sample defined by the 1997 National Census. The National Institute of Statistics in Mozambique recommended that all nationally representative surveys use this mother sample as an initial sampling frame. The primary sampling units are sets of three to six EAs. The secondary sampling unit is a single EA within each set of sampled EAs. The tertiary sampling unit is households—15 in rural EAs and 20 in urban EAs. An EA is a group of households, 80–100 in rural areas and 120–150 in urban areas. More details of the sampling frame and sampling procedure are described elsewhere.¹⁷

Of the 346 sampled EAs, the final sample included 345 EAs (one EA in Cabo Delgado was deemed unreachable), with 5990 households. Of these, 5857 households were visited, with results available for 5745 households in the final dataset. The Mozambique MIS collected data on a variety of malaria-related indicators. For the purpose of this article, we focus our discussion on two indicators—‘household possession of any bednet’ and ‘household possession of any ITN’.

LQAS method

LQAS is a classification methodology, which, in its elemental form, is designed to identify areas of ‘high’ or ‘low’ performance. With LQAS, information on a sample is collected in an area. For each indicator, the number of successes, X, is counted and compared with a predetermined cutoff, d. If there are fewer than d successes, then the area is classified as low performance; if there are d or more successes, then the area is classified as high performance. The determination of the cutoff, d, is a function of the sample size, targets for programme coverage and types of acceptable misclassification potential at different levels of coverage.^7,18–20

For this article, the target coverage for both indicators was 70%, in accordance with the Year-1 target of the Mozambique President’s Malaria Initiative for ITN possession.²¹ Any area with ≤40% coverage was deemed severely low. The observed sample sizes at the EA level in this particular Mozambique survey vary, resulting in different cutoffs for different areas. We sought decision rules to reduce two forms of misclassification error—the probability of classifying an area with high coverage (≥70%) as low, and the probability of classifying an area with low coverage (≤40%) as high with the combined total error being <20%. In other words, for each observed sample size we aimed to find a cutoff, d, which satisfies the following three conditions:

For example, the urban areas with a target sample of 20 have two decision rules that satisfy the above criteria. For a decision rule of 11, P(X < 11 | 20, p_U = 70%) = 0.048 and P(X≥11 | 20, p_L = 40%) = 0.128. For a decision rule of 12, P(X < 12 | 20, p_U = 70%) = 0.113 and P(X≥12 | 20, p_L = 40%) = 0.057. In this case, we chose a decision rule of 12, since it results in the lowest overall error rate.

Appendix 1 provides more detail on the calculation of LQAS decision rules, including the decision rules (Table A1) for the observed sample sizes used in this article. Some of the collected sample sizes in the 2007 Mozambique MIS are too small to satisfy the above criteria. In such cases, we select a decision rule that minimizes the overall error. For example, in rural areas with a target sample of 15, we used 9 as the decision rule since this minimizes the overall classification area. In the ‘Discussion’ section, we present recommendations to ensure adequate sample sizes for LQAS classifications in future surveys.

Results

Table 1 displays the provincial and national results for the two indicators, household possession of any bednet and household possession of any ITN, as reported in the Mozambique MIS report. As seen in the second column of the table, no province in Mozambique achieved the 70% target for bednet coverage. Sofala province and Maputo Cidade reported the highest coverage at 50%. Maputo and Tete provinces had the lowest bednet coverage of ∼30%. Overall, the results for ITN coverage were much lower across all provinces, from 6 to 37%. This 48–80% decline from bednet coverage to ITN coverage indicates a gap in net retreatment or in the population obtaining already treated nets. Manica provides one exception to this trend, with only an 18% difference in coverage between the two indicators.

Table 1.

Provincial and national coverage estimates for MIS cluster-samples and LQAS result summaries for household possession of any bednet or ITN for a 70% coverage target: Mozambique, 2007

	Any Bednet in HH		Any ITN in HH
Province	Coverage proportion	EAs classified as high coverage (total EAs)	Coverage proportion	EAs classified as high coverage (total EAs)
Niassa	0.422	10 (34)	0.177	0 (34)
Cabo Delgado	0.378	8 (33)	0.196	2 (33)
Nampula	0.329	3 (36)	0.087	0 (36)
Zambezia	0.365	5 (36)	0.178	0 (36)
Tete	0.317	7 (34)	0.119	0 (34)
Manica	0.448	10 (28)	0.369	3 (28)
Sofala	0.504	16 (34)	0.217	1 (34)
Inhambane	0.323	7 (34)	0.112	0 (34)
Gaza	0.373	6 (24)	0.133	0 (24)
Maputo province	0.297	2 (32)	0.057	0 (32)
Maputo Cidade	0.486	5 (20)	0.102	0 (20)
National	0.375	79 (345)	0.158	6 (345)

HH, household.

To better understand the local variability in performance, we applied LQAS decision rules to the data from each EA. For example, bednet coverage in Manica province is estimated at 45%; however, we do not know whether any sub-provincial areas have more severe inadequacy in coverage compared with others. For illustration, we applied LQAS classification rules to each EA in Manica province (Table 2, with decision rules taken from the Appendix 1). Based on these classifications, 10 EAs were classified as reaching the coverage target. The remaining 18 EAs are classified as severely below the coverage target. Similarly, although the 37% average coverage of ITNs in Manica falls short of the target, there is important local-level variability. With LQAS, we identified 3 of the 28 areas as having reached the coverage target.

Table 2.

LQAS results for household possession of nets in Manica province, Mozambique, 2007^a

EA		Decision	Any bednet in HH		Any ITN in HH
	ID		Total	rule	Yes	LQAS	Yes	LQAS
175	18	10	8	Low	8	Low
176	20	12	14	High	11	Low
177	20	12	9	Low	8	Low
178	20	12	12	High	10	Low
179	19	11	11	High	8	Low
180	20	12	10	Low	8	Low
181	17	10	15	High	11	High
182	20	12	14	High	9	Low
183	20	12	5	Low	4	Low
184	20	12	12	High	10	Low
185	21	12	14	High	11	Low
186	20	12	10	Low	8	Low
187	20	12	13	High	7	Low
188	16	9	7	Low	6	Low
189	14	8	8	High	8	High
190	15	9	7	Low	6	Low
191	15	9	7	Low	4	Low
192	15	9	7	Low	7	Low
193	15	9	2	Low	2	Low
194	15	9	3	Low	3	Low
195	15	9	8	Low	5	Low
196	15	9	1	Low	0	Low
197	15	9	8	Low	8	Low
198	15	9	7	Low	7	Low
199	15	9	1	Low	0	Low
200	15	9	6	Low	5	Low
201	15	9	5	Low	4	Low
202	15	9	11	High	9	High
Total	480		235	10 high	187	3 high

^aDecision rules set for minimizing α + β, 70% coverage target.

HH, household.

Table 1 summarizes the results of the same classifications applied to the other provinces and Maputo city. Although the highest provincial coverage estimate is just above 50% for the indicator ‘household possession of any bednet’, all of the provinces contain several EAs classified with adequate coverage. In Sofala province, nearly half of the EAs (16 of 34) are classified as high, versus <10% in Maputo province (2 out of 32). For ITN coverage the pattern is different. Manica province has the highest percent (11%, or 3 out of 28) of areas classified as high for ITN coverage. Every other province falls <10%, and eight provinces do not display any areas with high coverage.

Discussion

To illustrate the effective use of LQAS to assess performance on malaria outcome indicators based on data collected as part of the MIS, we applied this method to an existing dataset to show that one can assess sub-national performance. Analysing MIS data from Mozambique (2007) with the LQAS method, we determine which EAs of the MIS sample are performing adequately based on 70% bednet and ITN coverage targets. We find variation in the performance of EAs that is masked by a single point estimate for the province. Because of the limited resources available to the Ministry of Health for malaria prevention, this information on local-level performance could be used to target areas with severely inadequate coverage for intensive interventions. For example, based on the LQAS classifications, we identified the 266 of the 345 lowest performing EAs. This result suggests that substantial behaviour change and communication interventions may be required to promote the use of bednets, or more intensive distribution campaigns may be needed. Classifying an area on multiple indicators refines the analysis and suggests types of interventions needed. Of the 79 EAs classified as high with respect to bednet ownership, 73 of them were classified as low with respect to ITN ownership. These areas that exhibit high bednet ownership but low ITN ownership could be targeted for specialized interventions, such as those focusing on ITN impregnation and retreatment, or distribution of long-lasting insecticide-impregnated nets (LLIN). The LQAS classification did not isolate many areas in the high category for the ITN coverage indicator because the performance in all provinces for this indicator was quite low. However, as malaria programmes strengthen and go to scale, more areas will achieve this standard, increasing the utility of LQAS to identify and focus programmes on priority actions.

Beyond this specific application, this article illustrates the feasibility of incorporating LQAS into multistage cluster sample surveys by emulating an LQAS analysis of local-level data collected for a Mozambican MIS. Previous implementation of LQAS for local programme monitoring has demonstrated that this tool can be used frequently, rapidly and cost-effectively to provide information for allocating resources.^7,9,11,22 Since LQAS is used primarily to classify areas as ‘high’ or ‘low’ (rather than estimating the exact proportion of the population covered for each local area), the method does not require large samples to maintain excellent accuracy and its results lead directly to public health action. Additionally, the simplicity of the LQAS tool empowers local programme managers to implement the classification schemes with little additional training. These favourable attributes encourage the use of LQAS for local malaria programme M&E. Despite these benefits, to our knowledge, most examples of LQAS applications only aggregate LQAS data if information is collected in ‘all’ lots or areas. Such aggregation is also feasible when a sample of the lots is selected, so long as they are sampled in a probabilistic way.²³ Recent M&E activities have created precedent for taking samples of lots rather than requiring that all be sampled.^23–25

Although the regional and national point estimates reported in the Mozambique MIS are extremely valuable for country level and international planning and monitoring, the report does not provide means to assess the variability among the EAs by reporting design effects or confidence intervals.¹⁷ Failure to report these measures of variability is a common lapse in final reports from many large surveys. Application of LQAS classifications to the data collected in multistage surveys provides a means for understanding the level of local variation around regional estimates. Furthermore, these classifications not only link directly to action, so that managers realize when substantial variability exists and can strategize effective and localized responses, but they can also be combined, as shown, to provide the same national- and regional-level information provided by the multistage surveys.

Since this analysis was conducted using a pre-existing dataset, certain limitations are inherent in our study. For example, EAs included in the MIS sample were not aligned with specific health districts in Mozambique, and, therefore, the local-level results may not directly relate to operations because the results do not correspond to a specific local manager’s supervision area. Although the EA may represent a village within a supervision area, and may be akin to a sentinel site, the results do not reflect the performance of the entire supervision area as a whole. However, this is a solvable problem and suggests how future MIS and other macro surveys can be designed to more effectively inform about district or sub-district variation. Furthermore, the target sample size in the rural areas (15 households per EA) was too small to select decision rules with a total classification error < 0.20.

In the future, all these limitations can be addressed in the design, such as described by the Large Country Lot Quality Assurance Sampling (LCLQAS) protocol.²³ Specifically, implementation of multistage cluster sampling and LQAS should consider the following in the design: first, the clusters (or EA) should be aligned with programmatically relevant supervision areas. This ensures that recommended responses based on the LQAS classifications link directly to a programme area. Secondly, the within-cluster (or EA) sample sizes should be fixed across clusters, which allows for one set of decision rules to ease training of programme managers in applying LQAS to their supervision areas. Thirdly, these fixed sample sizes must be large enough to meet the constraints for determining the decision rules—for example, we were unable to meet our target of overall classification error <20% for sample sizes less than 17. A sample size of 19 at the EA level is sufficient to maintain the error level <20% when looking at a 30% difference in programme performance (p_U – p_L = 0.30); a smaller difference or smaller error rate requires a larger sample size.

Clearly, the design of the Mozambique MIS did not sample all EAs, and, as a result, we only have information on 345 EAs. Therefore, not all areas can be classified and prioritized for intervention according to their classification. Methods of intervention in these areas without data are too numerous to cover here. Ultimately, programmes should aim to implement these surveys on an ongoing, rolling basis, so that each area has an opportunity for a localized assessment, strengthening programme response.

Conclusion

Frequent and routine M&E of malaria programmes empowers managers and increases the programme’s responsiveness and effectiveness. One long-term solution is to use the LQAS methodology as the cornerstone of an ongoing national malaria programme M&E system, as is currently proposed for Nigeria.^1,26 By collecting LQAS samples routinely in all or a random selection of health programme areas, we can classify local areas to prioritize an appropriate public health response. This solution requires an initial investment in designing macro surveys so that their information contributes to the needs of local managers and training local programme managers in data collection and LQAS analysis, but results in a sustainable M&E system that encourages data-driven decisions at decentralized levels. Furthermore, aggregation of the LQAS data only requires random sampling at all levels and knowledge of the individual probability of being sampled so that the appropriate weights can be applied when calculating the point estimate and variance estimate.²⁷ Since individuals are sampled randomly within clusters when using LQAS, then standard cluster sampling principles are applicable when reporting regional and national indicators so long as the clusters are randomly chosen. Although many countries and programmes do not have the resources to implement large-scale MIS-type surveys on a regular basis, reliable estimates with reasonable levels of accuracy may be feasible in the interim by integrating LQAS into routine activities in fewer clusters.²⁸ This allows for continuous monitoring of malaria programmes, both locally and regionally, and for priorities and interventions to be adjusted in a systematic way.

In addition to providing aggregate measures at the regional and national levels, LQAS also empowers malaria-control programme managers interested in tracking coverage at a local level to improve their service-delivery strategies and tactics. As more countries aim to control malaria by scaling-up coverage, it becomes necessary to offer alternatives to the conventional national-level surveys that provide a single set of indicators for a province/country and neither present differences across local levels nor frequent or timely measures. LQAS, either integrated with national surveys or as a backbone of a malaria M&E system, is a suitable method to determine the impact and needs of programmes in smaller geographic areas.

Funding

The Mozambique MIS was funded by the President’s Malaria Initiative. National Institutes of Health Grants T32 AI007358 and R01 EB006195 to M.P. and B.H.; and ExxonMobil to the Malaria Implementation Resource Team of the World Bank, which supported J.V.

Conflict of interest: None declared.

KEY MESSAGES.

• Incorporating LQAS into multistage cluster sample surveys provides ‘additional’ information that supports local programme management as well as the usual macro-level results reported for malaria indicators.

• The following considerations facilitate the integration of LQAS and multistage cluster sample surveys:

  • Fixed sample size at the cluster level, to allow uniform training of programme managers in applying LQAS decision rules to interpret survey data.

  • Large enough sample sizes at the cluster level to control for LQAS classification errors.

  • Delineation of cluster boundaries to match programme areas, to ensure LQAS classification actions link directly to a specific programme implementation area.

Appendix 1

LQAS technical details

The simplest form of LQAS classifies an area into two categories, for the purposes of this article, ‘high’ and ‘low’. The sample of size n is collected in an area, called a ‘lot’. The number of successes, X, is counted and compared with a decision rule, d. If fewer than d successes in an area are observed, then the area is classified as ‘low’. If d or more successes are observed, then the area is classified as ‘high’.

For a fixed n, the decision rule is determined by the following.

1. The coverage level, p_U, where an area should be classified as high.

2. The coverage level, p_L, where an area should be classified as low.

3. The amount of error, α, allowed for misclassifying an area with p_U coverage as low. (In some applications, this error is also called the provider risk—the probability that a high-performing area will be identified as low performing, and as a result receive additional resources.)

4. The amount of error, β, allowed for misclassifying an area with p_L coverage as high. (In some applications, this error is also called the consumer risk—the probability that a low-performing area will be identified as high performing, and as a result not receive additional resources.)

These values are set by the programme managers, based on subject knowledge and programme goals. For example, programmes with greater resources can increase p_L to increase the definition of a low area, or decrease the acceptable level of β, to decrease consumer risk. Any of these parameters can be changed to meet programme monitoring needs, but the decision rule will change as a result. Furthermore, the sample size may not be sufficient to meet all constraints, and, in such cases, should be increased if needed.

For the Mozambique example, discussed in this article, we set the upper coverage level to 70%, p_U = 0.70, based on national programme targets. The lower coverage level, p_L, was set to 40% to identify areas with severely inadequate coverage. Ideally, we wanted to control overall error, α + β, to <20%. However, since we were not able to control sample size in this exercise, we chose decision rules that minimize the overall error rates (Table A1).

The middle columns in the table below present the decision rules (with associated errors) used for the article for the observed sample sizes. For the purpose of transparency and clarity, we have also summarized the associated error if the next smallest or next largest decision rule had been used.

Table A1.

LQAS decision rules and alpha beta errors for samples sizes ranging from 8 to 22

				Decision rules with lowest combined error
Sample size	DR	α	β	Decision rule	Alpha (α) P(X < d|n, pU = 70%)	Beta (β) P(X ≥ d|n, pL = 40%)	DR	α	β
8	4	0.058	0.406	5	0.194	0.174	6	0.448	0.050
9	4	0.025	0.517	5	0.099	0.267	6	0.270	0.099
10	5	0.047	0.367	6	0.150	0.166	7	0.350	0.055
11	6	0.078	0.247	7	0.210	0.099	8	0.430	0.029
12	6	0.039	0.335	7	0.118	0.158	8	0.276	0.057
13	7	0.062	0.229	8	0.165	0.098	9	0.346	0.032
14	7	0.031	0.303	8	0.093	0.150	9	0.229	0.058
15	8	0.050	0.213	9	0.131	0.095	10	0.278	0.034
16	8	0.026	0.284	9	0.074	0.142	10	0.175	0.058
17	9	0.040	0.199	10	0.105	0.092	11	0.225	0.035
18	9	0.021	0.263	10	0.060	0.134	11	0.141	0.058
19	10	0.033	0.186	11	0.084	0.088	12	0.182	0.035
20	11	0.048	0.128	12	0.113	0.057	13	0.228	0.021
21	11	0.026	0.174	12	0.068	0.085	13	0.148	0.035
22	12	0.039	0.121	13	0.092	0.055	14	0.186	0.021

DR, Decision Rule.