Abstract
A malaria cohort study was conducted among young children in Machinga District, Malawi, following distribution of insecticide-treated bednets (ITNs) in May 2012. To assess ITN use, two independently sampled subsets of children (211 during survey 1 [December 2012–January 2013] and 325 during survey 2 [September–October 2013]) were randomly selected to compare the proportions of positive and negative agreement between caregiver verbal reports at monthly interviews with visual observation of the ITN at home visits. Caregiver-reported ITN use was consistently high during both surveys (98.1% and 96.0%, respectively; P = 0.17). Home visit-based ITN use fell significantly (P < 0.001) from survey 1 (98.6%) to survey 2 (88.6%). The proportions of positive agreement between caregiver report and home visit in the first and second surveys were 98.8% (95% confidence interval [CI] 97.6–99.8%) and 93.3% (95% CI 91.2–95.3%), respectively. The proportions of negative agreement in the first and second surveys were 28.6% (95% CI 0–75.0%) and 20.0% (95% CI 0.1–35.0%), respectively. ITN use by children was high in Machinga District, and caregiver reports and home visits with visual confirmation of the net demonstrated a high level of agreement for use of ITNs, but a low level of agreement when ITNs were not used.
Insecticide-treated bednets (ITNs) can dramatically reduce malaria transmission1 and are a cornerstone of malaria control programs throughout sub-Saharan Africa.2 In community-based surveys, however, use of ITNs often falls below the World Health Organization's universal coverage target.2 Evaluations of ITN use frequently rely on self- or caregiver report because direct observation at night is challenging due to logistical considerations, security issues, and community acceptance. The few studies that have validated reported ITN use through nighttime checks have found that survey-reported use generally overestimates actual ITN use by 6–32% points.3–5 Over-reporting of ITN use, usually attributed to recall and/or social desirability bias,6,7 can result in significant underestimates of ITN effectiveness in reducing malaria risk.8,9 As part of a prospective study examining the effectiveness of ITNs to prevent malaria in a fixed cohort of young children (aged 6–59 months) in six rural villages in Machinga District, Malawi, we compared caregiver verbal reports of ITN use at monthly interviews at community centers with visual confirmation of ITNs at home visits.
The cohort study was conducted from April 2012 to December 2013 and included baseline distribution of new PermaNet® 2.0 long-lasting ITNs (Vestergaard, Lausanne, Switzerland) in May 2012 to all 2,178 registered households. The distribution provided one ITN per every two household residents (plus an additional ITN for odd numbers). Caregivers of enrolled children were interviewed once per month at a community center away from their homes using a standardized questionnaire. Caregivers' responses regarding whether their child slept under an ITN on the night prior to the monthly interview served as the caregiver verbal report of ITN use.
The study protocol originally included nighttime checks on samples of enrolled children to validate caregiver-reported ITN use. After consultation with the study community, however, nighttime checks were deemed unacceptable and were replaced with unannounced daytime home visits. A study staff person, blinded to the caregivers' reports of ITN use, attempted to visit the homes of selected children within three days after the monthly interview. Using a standardized questionnaire, the staff person requested visual confirmation of the ITN the child used after the caregiver was asked whether the child slept under an ITN the night before the home visit; a child was considered to have used an ITN at the home visit if the caregiver responded positively and the child's unpackaged ITN was visually observed by the staff person. The caregivers present at the monthly interview and at the home visit were not necessarily the same individual.
Children who attended their scheduled monthly interviews were selected by simple random sampling for follow-up home visits during the wet season (survey 1: December 2012–January 2013; N = 211 [18% of enrolled children]) and the dry season (survey 2: September–October 2013; N = 325 [26% of enrolled children]). Sample sizes were calculated to provide 80% power to detect a difference of 5% (assuming 98% caregiver-reported ITN use and 93% visual confirmation of ITN use) at α = 0.05.
As neither caregiver verbal report nor visual confirmation of an ITN at the home visit can be considered a gold standard measure of ITN use by a child, we could not calculate directly the sensitivity and specificity of either measure. For concordance in a diagnostic test, proportionate agreements in positive and negative classifications are measures analogous to sensitivity and specificity. These two indices may be more desirable to assess agreement than measures such as Cohen's kappa when marginal totals are significantly imbalanced.10 To measure the level of agreement between caregiver verbal reports of ITN use and visual confirmation of ITNs at the home visit, we calculated the proportion of positive agreement as
 |
where N is the total number of children included in each survey, a is the number of children with ITN use both reported by the caregiver and confirmed during home visit, and d is the number of children who did not use ITNs as reported by the caregiver and confirmed during home visit.11 The proportion of agreement in negative responses was similarly calculated as
 |
Bootstrap 95% confidence intervals (CIs) for both measures were constructed using 1,000 bootstrap samples. The χ2 test was used to compare proportions. Statistical analyses were conducted using R version 3.0.2 (R Core Team, Vienna, Austria).
Ethical approval for this study was obtained from the Centers for Disease Control and Prevention (Atlanta, GA; protocol no. 6227) and the University of Malawi's College of Medicine (Blantyre, Malawi; protocol no. 1151). All caregivers of the study participants provided written informed consent.
During the first survey, a median of two days passed between monthly interviews and home visits (range 0–8 days), with some outliers due to heavy rains. Caregiver-reported ITN use was 98.1%, whereas ITN use at the home visit was 98.6% (Table 1). Five responses were discordant between caregiver reports and home visits. The proportion of positive agreement was 98.8% (95% CI 97.6–99.8%) and the proportion of negative agreement was 28.6% (95% CI 0–75.0%).
Table 1.
Comparison of caregiver verbal reports of ITN use at community interviews with visual confirmation of ITNs at home visits
| First survey (wet season, December 2012–January 2013) | ||||
|---|---|---|---|---|
| Home visit | Total | |||
| Report of ITN use plus visual confirmation | Report of no ITN use or no visual confirmation | |||
| Caregiver report | Used ITN | 205 | 2 | 207 |
| Did not use ITN | 3 | 1 | 4 | |
| Total | 208 | 3 | 211 | |
| Second survey (dry season, September–November 2013) | ||||
| Home visit | Total | |||
| Report of ITN use plus visual confirmation | Report of no ITN use or no visual confirmation | |||
| Caregiver report | Used ITN | 280 | 32 | 312 |
| Did not use ITN | 8 | 5 | 13 | |
| Total | 288 | 37 | 325 | |
ITN = insecticide-treated bednet.
During the second survey, a median of one day passed between monthly interviews and home visits (range 1–3 days). Caregiver-reported ITN use was 96.0% (Table 1), which was not statistically different from the first survey (P = 0.213). Home visit ITN use was 88.6%, which was lower than in the first survey (P < 0.001). Forty responses were discordant between caregiver reports and home visits. The proportion of positive agreement in the second survey was 93.3% (95% CI 91.2–95.3%) and the proportion of negative agreement was 20.0% (95% CI 0.1–35.0%).
Reasons for the lower ITN use as confirmed by home visit during the second survey may include increased physical damage to ITNs,12 higher temperatures during the dry season,3 and/or increased time since ITN distribution.13 This analysis included only two time points, making it difficult to tease apart the effects of seasonality and time on ITN use.
The high level of agreement in positive classification of ITN use between caregiver reports and home visits was similar between surveys, suggesting that caregiver report is a reliable proxy for home visits when ITN use is reported positively. However, the proportionate agreement for negative responses was very low in both surveys suggesting that negative responses by caregivers in either setting may not be reliable. The major limitation of our study was the inability to validate caregiver-reported ITN use against a robust gold standard, such as direct nighttime observation. In addition, because only 27 of the 536 home visits (5.0%) occurred on the same day as the monthly interviews, most caregivers were asked about their child's ITN use on different nights. Although conducting home visits on the same day as the monthly interview would have been ideal, this was not possible in most cases due to resource constraints. We also note the possibility that different respondents could have been asked about ITN use at the monthly interview compared with the home visit.
ITN use by children in Machinga District, Malawi, was generally high (≥ 88.6%) in this study. There was high (> 90%) agreement for positive responses to ITN use, suggesting that caregiver report can be used in place of more intensive home visits as a measure of ITN use in epidemiologic surveys. The reasons for low agreement on negative responses are unclear but suggest that reported non-use of ITNs may not be a reliable measure.
ACKNOWLEDGMENTS
We thank Leah Gogwe, Sifiso Banda, and Exton Mtande for their assistance in the field, as well as the residents of Machinga District who participated in this study.
Disclaimer: The opinions expressed herein are those of the authors and do not necessarily reflect the views of the Centers for Disease Control and Prevention.
Footnotes
Financial support: This work was made possible through support provided by the U.S. President's Malaria Initiative, U.S. Agency for International Development, under the terms of an Interagency Agreement with the Centers for Disease Control and Prevention (CDC) and through a Cooperative Agreement (No. U01CK000135) between the CDC and the Malaria Alert Centre, College of Medicine.
Authors' addresses: Jacklyn Wong, Monica P. Shah, and John E. Gimnig, Entomology Branch and Malaria Branch, Division of Parasitic Diseases and Malaria, Centers for Disease Control and Prevention, Atlanta, GA, E-mails: vsm0@cdc.gov, mshah2@cdc.gov, and hzg1@cdc.gov. Dyson Mwandama and Don P. Mathanga, Malaria Alert Centre, College of Medicine, Blantyre, Malawi, E-mails: dmwandama@mac.medcol.mw, and dmathang@mac.medcol.mw. Kim A. Lindblade, Epidemiology and Prevention Branch, Influenza Division, Centers for Disease Control and Prevention, Bangkok, Thailand, E-mail: kil2@cdc.gov.
References
- 1.Lengeler C. Insecticide-treated bed nets and curtains for preventing malaria. Cochrane Database Syst Rev. 2004;2:CD000363. doi: 10.1002/14651858.CD000363.pub2. [DOI] [PubMed] [Google Scholar]
- 2.WHO . World Malaria Report: 2013. Geneva, Switzerland: World Health Organization; 2013. [Google Scholar]
- 3.Frey C, Traore C, De Allegri M, Kouyate B, Muller O. Compliance of young children with ITN protection in rural Burkina Faso. Malar J. 2006;5:70. doi: 10.1186/1475-2875-5-70. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Sexton JD, Ruebush TK, Brandlingbennett AD, Breman JG, Roberts JM, Odera JS, Were JBO. Permethrin-impregnated curtains and bed-nets prevent malaria in western Kenya. Am J Trop Med Hyg. 1990;43:11–18. doi: 10.4269/ajtmh.1990.43.11. [DOI] [PubMed] [Google Scholar]
- 5.Linhua T, Manderson L, Da D, Kaichen W, Xianzheng C, Changxiong L, Zhengcheng G, Ke-an W. Social aspects of malaria in Heping, Hainan. Acta Trop. 1995;59:41–53. doi: 10.1016/0001-706x(94)00086-g. [DOI] [PubMed] [Google Scholar]
- 6.Cohee L, Mills LA, Kagaayi J, Jacobs I, Galiwango R, Ludigo J, Ssekasanvu J, Reynolds SJ. High retention and appropriate use of insecticide-treated nets distributed to HIV-affected households in Rakai, Uganda: results from interviews and home visits. Malar J. 2009;8:76. doi: 10.1186/1475-2875-8-76. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Bennett A, Smith SJ, Yambasu S, Jambai A, Alemu W, Kabano A, Eisele TP. Household possession and use of insecticide-treated mosquito nets in Sierra Leone 6 months after a aational mass-distribution campaign. PLoS One. 2012;7:e37927. doi: 10.1371/journal.pone.0037927. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Choi HW, Breman JG, Teutsch SM, Liu SM, Hightower AW, Sexton JD. The effectiveness of insecticide-impregnated bed nets in reducing cases of malaria infection: a meta-analysis of published results. Am J Trop Med Hyg. 1995;52:377–382. doi: 10.4269/ajtmh.1995.52.377. [DOI] [PubMed] [Google Scholar]
- 9.Leake DW, Jr, Hii JL. Giving bednets “fair” tests in field trials against malaria: a case from Sabah, East Malaysia. Southeast Asian J Trop Med Public Health. 1989;20:379–384. [PubMed] [Google Scholar]
- 10.Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990;43:543–549. doi: 10.1016/0895-4356(90)90158-l. [DOI] [PubMed] [Google Scholar]
- 11.Cicchetti DV, Feinstein AR. High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol. 1990;43:551–558. doi: 10.1016/0895-4356(90)90159-m. [DOI] [PubMed] [Google Scholar]
- 12.Graves PM, Ngondi JM, Hwang J, Getachew A, Gebre T, Mosher AW, Patterson AE, Shargie EB, Tadesse Z, Wolkon A, Reithinger R, Emerson PM, Richards FO. Factors associated with mosquito net use by individuals in households owning nets in Ethiopia. Malar J. 2011;10:354. doi: 10.1186/1475-2875-10-354. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Alaii JA, Hawley WA, Kolczak MS, ter Kuile FO, Gimnig JE, Vulule JM, Odhacha A, Oloo AJ, Nahlen BL, Phillips-Howard PA. Factors affecting use of permethrin-treated bed nets during a randomized controlled trial in western Kenya. Am J Trop Med Hyg. 2003;68:137–141. [PubMed] [Google Scholar]