A new method to estimate mortality in crisis-affected and resource-poor settings: validation study

Abstract

Background Data on mortality rates are crucial to guide health interventions in crisis-affected and resource-poor settings. The methods currently available to collect mortality data in such settings feature important methodological limitations. We developed and validated a new method to provide near real-time mortality estimates in such settings.

Methods We selected four study sites: Kabul, Afghanistan; Mae La refugee camp, Thailand; Chiradzulu District, Malawi; and Lugufu and Mtabila refugee camps, Tanzania. We recorded information about all deaths in a 60-day period by asking key community informants and decedents’ next of kin to refer interviewers to bereaved households. We used the total number of deaths and population estimates to calculate mortality rates for 60- and 30-day periods. For validation we compared these rates with a best estimate of mortality using capture–recapture analysis with two further independent lists of deaths.

Results The population covered by the new method was 76 476 persons in Kabul, 43 794 in Mae La camp, 54 418 in Chiradzulu District and 80 136 in the Tanzania camps. The informant method showed moderate sensitivity (55.0% in Kabul, 64.0% in Mae La, 72.5% in Chiradzulu and 67.7% in Tanzania), but performed better than the active surveillance system in the Tanzania refugee camps.

Conclusions The informant method currently features moderate sensitivity for accurately assessing mortality, but warrants further development, particularly considering its advantages over current options (ease of implementation and analysis and near-real estimates of mortality rates). Strategies should be tested to improve the performance of the informant method.

Introduction

Data on the levels and causes of population mortality are critical for assessing a population’s health status and informing public health interventions in crisis-affected and resource-poor settings.^1,2 However, in such settings, routine vital registration systems are often unreliable, cease to function or do not exist. The two main alternative approaches currently used in such situations to measure population mortality are prospective demographic surveillance (regular exhaustive enumeration by home visitors of individual births and deaths) and retrospective mortality surveys (estimation of mortality over a specified recall period based on interviews with a representative sample of households). Both approaches have serious limitations. Surveillance requires a large number of closely supervised staff to collect data; it can be difficult to implement in geographically dispersed populations;² and the sensitivity of the surveillance system may be low, even in long-standing refugee camps.³ Retrospective surveys are poorly suited to generating essential operational data on current levels and causes of mortality, because, over very short recall periods (e.g. <1 month), they require unfeasibly large sample sizes to avoid meaninglessly imprecise estimates (with imprecision further compounded by the design effect inherent in cluster sampling).^4–6 Retrospective mortality surveys are also subject to various selection and reporting biases, stemming mainly from inadequate sampling frames, unequal probability of selection of households (especially in cluster designs) and deliberate or involuntary inaccuracies in questionnaire responses.⁷ Statistical analysis of retrospective survey data is also more difficult than for surveillance data. Partly as a result of these limitations, the quality of mortality surveys done in the humanitarian sector is generally low.⁵

In this article, we report on a multi-site evaluation of an alternative method (which we refer to as the informant method) designed to furnish precise and unbiased estimates of mortality over a very short recall period (i.e. almost on a real-time basis) and be simple enough so that practitioners without statistical training can implement it to rapidly measure mortality in their communities of intervention. We evaluated the validity of the informant method against a best estimate of mortality provided by capture–recapture analysis.

Methods

Ethical approvals

The study was approved by the Ethics Committee of the London School of Hygiene and Tropical Medicine; the Institutional Review Board of the Ministry of Public Health of Afghanistan; the Malawi National Health Sciences Research Committee; and the National Institute for Medical Research and also the Commission for Science and Technology in Tanzania. In Thailand, no local institutional review board has jurisdiction over the refugee camps. Household respondents and focus group discussion (FGD) members were asked to provide oral consent to participate in this study. In Thailand, written consent was sought from all study participants.

Study sites and sampling design

Recognizing that validity may be affected by local culture and types of human settlement, we aimed to document the informant method's implementation, performance and limitations in a variety of settings. Specifically, although we mainly conceived of the method as a useful option in refugee or internally displaced persons’ camps, we also wanted to document its validity and feasibility in at least one chaotic, dense urban setting and one rural, scattered community. Furthermore, we wanted to ensure that the cultural settings of the study sites would be as unrelated as possible, though we specified that half should be in Sub-Saharan Africa. Accordingly, we selected the following four sites: (i) District (nahia) 1 of Kabul city, Afghanistan, a poor urban community with chaotic layout; (ii) Mae La camp for Karen refugees, on the Thai-Burma border; (iii) Chiradzulu District in Malawi, a rural, scattered community with a high HIV/AIDS burden; and (iv) Lugufu and Mtabila camps (hereafter considered as one site) in western Tanzania, which host refugees from the Great Lakes region. The Kabul site was chosen as it featured an existing London School of Hygiene and Tropical Medicine research programme. The broad regions of the other three sites (Thai-Burma border camps; Malawi rural; East Africa camps) were determined a priori, and the actual sites were identified through investigators’ field contacts.

In each site we enquired about a recall period of 2 months (60 days), but restricted the main analysis to the most recent 1 month (30 days). Results for the full 60-day period are also presented, as they provide useful information about recall of increasingly remote dates.

We surveyed the populations of District 1, Kabul; Mae La camp; and the Tanzania camps exhaustively. Assuming a minimum crude mortality rate (CMR) of 0.2 deaths per 10 000 person-days within each of the above sites, and given that each site had an estimated population of more than 40 000, we calculated that the method's sensitivity (i.e. proportion of all deaths that are detected by the method) for a 30-day recall period could be estimated among 26 or more deaths in each site. This number of deaths would have provided a precision within ±20% for any sensitivity point estimate >80%, which we considered an acceptable performance of the method.

In Chiradzulu district, it was considered inefficient to visit all 757 villages. Instead, we surveyed a fraction, selected using a modified centric systematic area sampling design. We divided the district into 32 5 km × 5 km quadrats, with area mostly falling within the district boundaries. Using high-resolution maps, we then selected the three villages closest to the centre of each quadrat, thus yielding a non-self-weighting sample of 96 villages. Within each sampled village the search for recent deaths was exhaustive. Assuming an average population per village of 500, nearly 50 000 people would be sampled, yielding a minimum precision comparable with that of the other sites.

Implementation of the informant method

The informant method consists of an exhaustive, active search for all deaths occurring in the surveyed community over a defined recall period. Key community informants, selected after rapid qualitative work, recall households that have experienced a recent death and refer data collectors to these households; decedents’ next of kin also refer data collectors to other recently bereaved households. The number of deaths recorded from the method is then combined with population estimates to calculate a mortality rate for the recall period (see below).

We held FGDs in each study site before data collection. Each FGD obtained information for the whole study site and sought to: (i) identify which key community informants would be most knowledgeable about recent deaths; (ii) explore information sharing about recent deaths in the community; and (iii) identify alternative sources of mortality information that would provide additional lists for capture–recapture analysis, later used to validate the method (see below). Local study team members and collaborators helped to identify participants, who were selected based on their knowledge of the community (Table 1). Although elements of this method have been used previously, such as obtaining information on deaths from key-informants such as persons in charge of cemeteries⁸ and traditional birth attendants (for maternal deaths),⁹ or using snowballing techniques for referrals to households that have experienced a maternal death,^10,11 to the best of our knowledge this is the first method to combine the use of formative FGDs, information from key-informants, household referrals and population estimation to compute mortality rates.

Table 1.

Details of FGD participants, by study site

District 1, Kabul	Mae La camp	Chiradzulu District	Lugufu and Mtabila camps
Men's FGD (eight participants):  five wakil-e-guzar (sector leaders), one mullah, two education officers Women's FGD (four participants): four school teachers	33 participants: camp committee members, Mae La hospital official, section leaders, religious leaders, section health workers, home visitors, SMRU home visitors	Nine participants: village headmen (two males), headman assistant (female), graveyard chairman (male), church elder (one male, one female), Muslim representative (male), teacher (male), member of village health committee (female)	Lugufu camp (14 participants): zone leaders (seven males, one female), village leaders (four males, one female), one radio reporter (male) Mtabila camp (22 participants): zone leaders (seven males, two females), street leaders (six males, four females), pastor (one male), community security officer (male), health information team member (male)

To carry out the exhaustive search for deaths, we divided each community into sectors, corresponding to pre-existing administrative areas: guzar in Kabul (n = 24); sections in Mae La camp (n = 22); villages in Chiradzulu District (n = 96); and zones in the Tanzania camps (n = 23). The primary and secondary informants selected based on the FGD (see list 1 in Table 2) were located for each sector. The data collection then proceeded sector by sector. Sector leaders were informed in advance about the study team's upcoming visit. We asked key informants to separately list all deaths occurring among community residents during the previous 60 days and then visited the households in which those deaths had occurred.

Table 2.

Informants and sources used for implementation of the informant method and to generate additional lists for capture–recapture analysis, by site

List	District 1, Kabul	Mae La camp	Chiradzulu District	Tanzania camps
1 (Primary and secondary key informants used for the informant method)	Primary: wakil-e-guzars. Officially responsible for sector, are supposed to be informed about deaths and attend funerals. All male. Secondary: mullahs.  Theoretically responsible for funeral prayers after any death. Announce deaths and funerals during mosque prayers. Each guzar contains at least one and typically up to five mosques, both Shi’a and Sunni. All mosques were visited.	Primary: section leaders:  Responsible for official reporting of vital statistics.  Have a comprehensive coverage of the entire camp. Secondary: Karen Women's  Organization members (one representative per section).  Should have a strong knowledge of deaths of women and children. All female.	Primary: village headmen.  Responsible for allocating land in graveyards for burial. Expected to be present at all village funerals. Informed of key events in village. Role of sharing information on key events. May be male or female. Secondary: Fumukazi  Village elder women with responsibilities for pregnancies, births, and deaths of infants and children.  Work closely with village headmen. Expected to be present at all village funerals.	Primary: zone leaders. Camp residents selected by other residents. Comprehensively cover the camps through reporting system of block leaders, village leaders, and zone leaders. Secondary: secondary informants were not identified as the options were either not feasible (e.g. religious leaders were too many to consult) or needed as sources for the two additional lists.
2	Attendants at all convenience stores (small one room shops selling food, drinks and basic household items) and bakeries (community ovens where households bring their flour to be baked; separate female and male bakeries exist) in each guzar.	All Buddhist, Christian and Muslim religious leaders operating in the section. In charge of funeral rites.	Chairmen of village graveyard committees (azukuru), in charge of organizing burials. In Chiradzulu, nearly all people are said to be buried in graveyards (one or two per village).	Register of deaths maintained by the camp management agency (World Vision in Lugufu camp, International Rescue Committee in Mtabila camp).
3	Registers of the inpatient departments of all hospitals within the catchment area of District 1, including Maiwand Hospital, Ibna Sina Hospital, Indira Gandhi Hospital, Rab-e-Balkhi Maternity Hospital, the French Hospital and the Tuberculosis Hospital.	Registers maintained by the community mortality surveillance system implemented by Aide Médicale Internationale, which also captures data from the Mae La hospital inpatient department and the Shoklo Malaria Research Unit maternal health clinic.	Stabilization and maternity wards of all public health centres in Chiradzulu District (n = 10); registers of all inpatient departments in Chiradzulu District Hospital (including morgue) and St. Joseph's Nguludi Hospital; Ad hoc data collection by Health Surveillance Assistants deputized to each of the 96 villages.	Registers maintained by the community mortality surveillance system implemented by the Tanzanian Red Cross Society in Lugufu and Mtabila camps.

We administered a questionnaire to next of kin aged ≥18 years to record the date of death, which we established with the aid of a detailed calendar of local events, age and sex of the decedent and the cause and place of death. We included deaths among neonates if they were breathing at birth. In Chiradzulu only, we also administered verbal autopsy questionnaires, using the latest World Health Organization standards.¹² To uniquely identify decedents for the purpose of validation only (see below), we also collected the decedent's name; in Kabul and Mae La we collected the name of the father, and in Chiradzulu and Tanzania camps, the name of the household.

After administering the questionnaire, we asked respondents to list other deaths in their household or anywhere in the community within the previous 60 days. Once we had visited all households with deaths identified by the key informants and households of decedents, we considered the sector exhausted and moved on to the next sector. We did not visit deaths among households outside the community.

Validation

To measure the method's sensitivity, we obtained a best estimate of the true number of deaths in the recall period through capture–recapture analysis, used extensively in epidemiology to evaluate the completeness of reporting and/or the true burden of various diseases,^13,14 and recently applied to war-related killings.^15–17 This technique analyses the overlap among different lists of the events in question (in our case, recent deaths) to estimate the number of deaths that do not appear on any lists, which is then added to the ‘known’ or ascertained deaths (i.e. those that are found on one or more lists) to provide the best-estimate total.

In each site we generated three different lists of deaths, the first being that obtained from implementation of the informant method itself. We selected sources for the two additional lists (Table 2), based on criteria including sensitivity as judged by FGD participants, feasibility (i.e. how easy it would be to contact informants) and independence from the key informants already consulted (i.e. we avoided sources that would themselves obtain their knowledge from the informant method key informants, or vice versa). We collected these two additional lists only after the implementation of the informant method, to minimize investigator bias. We approached sources for the two additional lists in the same way as the informants used for our method, and asked about the same recall period. We considered deaths recorded in different sources as the same if they matched the sex, age, place of residence and name of the deceased in Kabul and Mae La, or father or household name in Chiradzulu and Tanzania camps.

Because the probability of a death appearing on a list is usually not independent from the probability of appearing on a different list (e.g. prominent individuals may be remembered by most sources; a death mentioned by one religious group might be unlikely to appear in the list of another religious group), possible interactions among lists need to be explicitly factored into the capture–recapture estimation. To do this, we fitted log-linear Poisson models¹⁸ to the three-way contingency table formed by each x_abc category, where a, b and c denote the presence (1) or absence (0) of a death within lists 1, 2 and 3, respectively (e.g. x₁₁₀ is the number of deaths on lists 1, 2 but not 3). Model terms represented the probabilities of inclusion in lists 1, 2 and 3; additional interaction terms represented possible dependencies among lists (i.e. inclusion in one list being a function of inclusion in another). Eight alternative candidate models were thus fitted (no interactions; one interaction among any two lists; two two-way interactions and three two-way interactions, i.e. saturated). Each model predicts m_abc, i.e. the quantity inside each cell of the three-list contingency table, where m is the estimator of x (the true value in the cell). From these predictions,

and N, the estimated total number of deaths, is the number of deaths appearing on one or more lists plus m_000.¹³

Instead of selecting one of the models as the most likely, we used Bayesian Model Averaging¹⁹ with uninformative priors to derive average estimates from all the models combined (in this method, the prediction is a weighted average of predictions from each model, with the weight provided by each model's posterior probability, a Bayesian statistic that indicates how likely it is that the model is the correct representation of reality). However, we first excluded any models that did not converge to a solution or had a Chi-square goodness of fit P-value >0.60 (we considered this evidence of overfitting; in practice, all these models also predicted implausibly high m₀₀₀ values). Using the remaining models, we computed

where K is the total number of models averaged over, i is one of these models and Pr is its posterior probability. Ninety-five per cent confidence intervals (CIs) were derived from the models’ likelihood profiles.

When there were a sufficient number of deaths recorded, we estimated the total number of deaths among children <5 years of age, as above; however, this was not always feasible due to small numbers.

For each site and recall period (30 and 60 days), we expressed the sensitivity of the informant method or any other source as the number of deaths captured by the source, divided by the capture–recapture estimate of the total number of deaths (N).

Calculation of mortality rates

Existing population estimates for all ages and children aged <5 years in Mae La and the Tanzanian camps were judged reliable and adopted. In Kabul and Chiradzulu, we performed our own population estimation (see Supplementary File 1 available as supplementary data at IJE Online). We expressed mortality rates as deaths per 10 000 person-days. We estimated three different mortality rates using the number of deaths (i) captured by the informant method, (ii) captured by all sources combined and (iii) estimated by the capture–recapture analysis. We computed point estimates and 95% CIs through bootstrapping programmes specific to each field site, taking into account observed design effects and sampling weights where needed (see Supplementary File 1 available as supplementary data at IJE Online).

We double-entered and cleaned all data using EpiData version 3.0 software (EpiData Association, Odense, Denmark). All analysis was done using R software.

Results

Data collection was conducted between July and October 2008, taking 11, 5, 16 and 6 working days in Kabul, Mae La, Chiradzulu and the Tanzania camps, respectively (Table 3); this includes time for collection of additional lists for capture–recapture analysis. The size of the four study populations ranged from 43 794 to 80 136 people. All community informants who were found, agreed to provide information. There were no instances in which a household was empty. One household in Tanzania did not consent.

Table 3.

Timeframe, population covered by the exhaustive search and response rate, by study site

	District 1, Kabul	Mae La camp	Chiradzulu District	Tanzania camps
Population size (age <5 years)	76 476 (13 790)	43 794 (5384)	54 418 (9462)	80 136 (16 028)
Timeframe of data collection	14–27 July 2008	11–17 July 2008a	26 August to 16 September 2008	3–9 October 2008
Number (%) of primary informants found	26/26 (100)	22/22 (100)b	91/96 (94.8)c	15/18 (83.3)d
Number (%) of secondary informants found	∼80%e	22/22 (100)	90/96 (93.8)f	no secondary informants used (see Table 2)
Response rate (%) (households found and giving consent)	100	100	100	98

^aTwo interviews were conducted on 27 July due to previous inability to contact the household.

^bIn practice, we consulted with the section leader as well as members of his/her office, who were usually present during the visit.

^cIn 5 (5.2%) of the 96 villages sampled in Chiradzulu district, the fumukazi was also the acting headman as the headman was away. In a further seven villages (7.3%) the fumukazi was also the headman on a permanent basis (these are included among the 91 found).

^dThe three remaining primary informants were contacted by other primary informants and provided the required information.

^eData were not collected systematically due to the large number of mullahs contacted: a rough estimate is provided.

^fIn six villages (6.3%) a deputy fumukazi was contacted as the fumukazi was away.

In all four sites, primary and secondary informants provided the majority of reports (Table 4). Few reports came from respondent households; of these 3/11 in Kabul, 3/9 in Mae La, 3/6 in Chiradzulu and 12/14 were also reported by community informants, considering a 60-day recall period. Several deaths were excluded from the analysis, for the following reasons: (i) stillbirth (n = 5); (ii) death outside the 60-day recall period (n = 16); and (iii) not residing in the community (n = 8).

Table 4.

Number (%) of deaths captured using the informant method, by informant type, recall period and study site

	District 1, Kabul		Mae La camp		Chiradzulu District		Tanzania camps
Deaths captured, by type of informant	60 days recall	30 days recall	60 days recall	30 days recall	60 days recall	30 days recall	60 days recall	30 days recall
Primary and secondary (%a)	55 (82.1)	11 (100.0)	20 (74.1)	11 (68.8)	90 (96.8)	34 (94.4)	42 (95.5)	20 (95.2)
Respondent households (%a)	11 (16.4)	0 (0.0)	9 (33.3)	4 (25.0)	6 (6.5)	5 (13.9)	9 (21.4)	4 (20.0)
Others (%a)	4 (6.0)	0 (0.0)	1 (3.7)	1 (6.3)	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)
Total deaths, excluding multiple reports (%b)	67	11 (16.4)	27	16 (59.2)	93	36 (38.7)	44	21 (47.7)

^aPercentages of total deaths (excluding multiple reports) in each column.

^bPercentage of all deaths reported in the site.

Of the three different lists collected in each site (Table 5), the informant method identified the largest number of deaths, except in Mae La, where the existing surveillance system identified more deaths (the overlap of the deaths recorded among the three lists is shown as Venn diagrams in Supplementary File 2 available as supplementary data at IJE Online). In each site, the age and sex profile of deaths was similar for all lists, except in Kabul, where hospital-registered deaths were all male and mostly children, and surveillance records in Mae La which had a higher proportion of children. None of the lists showed notable differences in the age and sex profile comparing the 60- and 30-day time periods.

Table 5.

Demographic profile of deaths captured by the informant method and alternative lists, by recall period and study site

	List 1		List 2		List 3		All lists combined
Site	60 days recall	30 days recall	60 days recall	30 days recall	60 days recall	30 days recall	60 days recall	30 days recall
District 1, Kabul	Informant method		Stores, bakeries		Hospital records		Total
Number of deaths (%)a	67	11 (16.4)	33	12 (36.3)	8	6 (75.0)	82	18 (22.0)
Median age, years (range)	35 (0–99)	50 (0–99)	50 (0–99)	48 (0–99)	3 (0–36)	1 (0–6)	35 (0–99)	40 (0–99)
Number of children aged <5 years (% of total deaths)	20 (29.9)	1 (9.1)	9 (27.3)	2 (16.7)	5 (62.5)	4 (66.7)	26 (31.7)	5 (27.8)
Male sex (% of total deaths)	32 (47.8)	4 (36.4)	16 (48.5)	6 (50.0)	8 (100.0)	6 (100.0)	43 (52.4)	10 (55.6)
Mae La camp	Informant method		Religious leaders		Surveillance system		Total
Number of deaths (%)a	27	16 (59.2)	23	15 (65.2)	41	18 (43.9)	52	23 (44.2)
Median age, years (range)	55 (1–87)	63 (21–87)	55 (1–85)	63 (45–85)	47 (0–87)	54 (0–87)	49 (0–87)	63 (0–87)
Number of children aged <5 years (% of total deaths)	2 (7.4)	0 (0.0)	2 (8.7)	0 (0.0)	15 (36.6)	4 (22.2)	16 (30.8)	4 (17.4)
Male sex (% of total deaths)	15 (55.6)	6 (37.5)	14 (60.9)	8 (53.3)	27 (65.9)	11 (61.1)	33 (63.4)	13 (56.5)
Chiradzulu District	Informant method		Graveyard chairmen		Health structures, HSAs		Total
Number of deaths (%)a	93	37 (39.8)	72	27 (37.5)	44	18 (40.9)	110	43 (39.0)
Median age, years (range)	35 (0–96)	35 (0–85)	34 (0–96)	34 (0–85)	35 (0–96)	29 (0–85)	35 (0–96)	34 (0–85)
Number of children aged <5 years (% of total deaths)	26 (28.0)	12 (32.4)	20 (27.8)	9 (33.3)	11 (25.0)	7 (38.9)	30 (27.3)	15 (35.7)
Male sex (% of total deaths)	47 (50.5)	16 (43.2)	34 (47.2)	9 (33.3)	25 (56.8)	9 (50.0)	55 (50.0)	19 (45.2)
Tanzania camps	Informant method		Camp register		TRCS surveillance		Total
Number of deaths (%)a	44	21 (47.7)	35	16 (45.7)	29	13 (44.8)	63	28 (44.3)
Median age, years (range)	4 (0–98)	25 (0–98)	2 (0–87)	22 (0–87)	2 (0–78)	27 (0–78)	4 (0–98)	15 (0–98)
Number of children aged <5 years (% of total deaths)	22 (50.0)	8 (38.1)	17 (48.6)	6 (37.5)	17 (58.6)	4 (30.8)	33 (52.4)	12 (42.9)
Male sex (% of total deaths)	22 (50.0)	10 (47.6)	22 (62.9)	11 (68.8)	16 (55.2)	9 (69.2)	34 (54.0)	16 (57.1)

^aPercentage of all deaths reported for the individual list in the study site.

HSAs: Health Surveillance Assistants; TRCS: Tanzania Red Cross Society.

Estimated sensitivity of the informant method and other sources

Sensitivity of the informant method ranged from 45 to 65% over a 60-day period, according to the site, and from 55 to 73% over a 30-day period (list 1, Table 6). The sensitivity of the informant method was higher than that of the other lists, except in Mae La and in Kabul over a 30-day period. The sensitivity of any source was higher over a 30-day recall period than a 60-day recall period, with the exception of the informant method in Kabul. The combination of all three sources appeared to achieve good sensitivity in all sites.

Table 6.

Estimated sensitivity of the informant method, other sources and all lists combined, by study site and recall period

	List 1		List 2		List 3		All lists combined		Total estimated deathsa
	n	% (95% CI)	n	% (95% CI)	n	% (95% CI)	n	% (95% CI)	n	(95% CI)
District 1, Kabul	Informant method		Stores and bakeries		Hospital records
60 days recall	67	62.6 (39.9–72.8)	33	30.8 (19.6–35.9)	8	7.5 (4.8–8.7)	82	76.6 (48.8–89.1)	107	(92–168)
30 days recall	11	55.0 (37.9–61.1)	12	60.0 (41.4–66.7)	6	30.0 (20.7–33.3)	18	90.0 (62.1–100.0)	20	(18–29)
Mae La camp	Informant method		Religious leaders		Surveillance system
60 days recall	27	45.0 (37.0–48.2)	23	38.3 (31.5–41.1)	41	68.3 (56.2–73.2)	52	86.7 (71.2–92.9)	60	(56–73)
30 days recall	16	64.0 (50.0–69.6)	14	56.0 (43.8–60.9)	18	72.0 (56.2–78.3)	23	92.0 (71.9–100.0)	25	(23–32)
Chiradzulu District	Informant method		Graveyard chairmen		Health structures, HSAs
60 days recall	93	65.0 (47.9–75.6)	72	50.3 (37.1–58.5)	44	30.8 (22.7–35.8)	110	76.9 (56.7–89.4)	143	(123–194)
30 days recall	36	72.5 (46.8–82.2)	27	52.9 (34.2–60.0)	18	35.3 (22.8–40.0)	43	84.3 (54.4–95.6)	51	(45–79)
Tanzania camps	Informant method		Camp register		Surveillance system
60 days recall	44	53.0 (36.4–62.9)	35	42.2 (28.9–50.0)	29	34.9 (24.0–41.4)	63	75.9 (52.1–90.0)	83	(70–121)
30 days recall	21	67.7 (51.2–72.4)	16	51.6 (39.0–55.2)	13	41.9 (31.7–44.8)	28	90.3 (68.3–96.6)	31	(29–41)

HSAs: Health Surveillance Assistants; n: number of deaths recorded by each list.

^aFrom capture–recapture analysis.

Among children <5 years of age, the method's sensitivity was 52.6% (95% CI 17.1–71.4) in Kabul (60-days period), 66.7% (95% CI 27.1–81.2) in Chiradzulu (60 days) and in Tanzania it was 53.7% (95% CI 36.1–61.1) over 60 days and 47.1% (95% CI 15.7–61.5) over 30 days. Other sensitivity estimates could not be computed due to the low number of child deaths, precluding capture–recapture analysis. However, considering ascertained deaths only, the informant method detected no more than 1/5 in Kabul (30 days), 2/16 and 0/4 in Mae La (60 and 30 days, respectively) and 12/15 in Chiradzulu (30 days).

Estimated mortality rates

The mortality rates estimated by the informant method, all the lists combined, and capture–recapture analysis are given in Table 7. The mortality rates estimated by the informant method and any other single method are substantially lower than those estimated through capture–recapture analysis over the 30- and 60-day periods. Mortality rates among children aged <5 years of age were approximately double the all-age CMR, as typically observed in developing country settings. Comparisons of our estimated mortality rates over a 60-day recall period with recent survey-based estimates of mortality from broadly comparable regions and also national-level estimates are provided in Table 8, and these show our capture–recapture estimates are broadly consistent with these other estimates.

Table 7.

Estimated crude and under 5 years mortality rates (as deaths per 10 000 person-days) based on the informant method, all lists combined and the capture–recapture estimate, by study site and recall period

	CMR (95% CI)			Under 5 years mortality rate (95% CI)
	Informant method	All lists combined	Capture– recapture analysis	Informant method	All lists combined	Capture– recapture analysis
District 1, Kabul
60 days recall	0.15 (0.12–0.19)	0.18 (0.15–0.23)	0.24 (0.19–0.34)	0.24 (0.17–0.33)	0.31 (0.22–0.43)	0.49 (0.29–1.30)
30 days recall	0.05 (0.04–0.06)	0.08 (0.07–0.10)	0.09 (0.08–0.12)	0.02 (0.02–0.03)	0.12 (0.09–0.17)	n/aa
Mae La camp
60 days recall	0.10 (0.09–0.11)	0.20 (0.18–0.22)	0.23 (0.20–0.28)	0.06 (0.06–0.07)	0.49 (0.45–0.55)	n/aa
30 days recall	0.12 (0.11–0.13)	0.18 (0.16–0.19)	0.19 (0.17–0.23)	0.00 (0.00–0.00)	0.25 (0.22–0.28)	n/aa
Chiradzulu District
60 days recall	0.30 (0.23–0.39)	0.39 (0.29–0.52)	0.51 (0.38–0.67)	0.54 (0.30–0.93)	0.61 (0.36–1.06)	0.84 (0.48–1.64)
30 days recall	0.26 (0.17–0.39)	0.32 (0.22–0.46)	0.38 (0.25–0.59)	0.55 (0.25–1.31)	0.69 (0.34–1.37)	n/aa
Tanzania camps
60 days recall	0.09 (0.09–0.10)	0.13 (0.12–0.14)	0.18 (0.15–0.24)	0.23 (0.21–0.24)	0.34 (0.32–0.37)	0.43 (0.38–0.54)
30 days recall	0.09 (0.08–0.09)	0.12 (0.11–0.13)	0.13 (0.12–0.15)	0.17 (0.15–0.18)	0.25 (0.23–0.27)	0.39 (0.30–0.71)

^aIncalculable due to small number of deaths.

Table 8.

Recent estimates of mortality (as deaths per 10 000 person-days) in the regions surrounding the four study sites and nationally

	CMR	Under 5 years mortality rate
Afghanistan	Kabul District surveys (includes residents or returnees only, and rural areas in the district): 1999–2002: 0.16 (Bartlett et al.)20 2001: 0.20 2003: 0.50 2004: 0.30 2006: 0.76 National in 2008: 0.49a, 0.55b This study: 0.24	2001: 0.54 2003: 0.59 2004: 0.63 2006: 0.73 2001–06: 0.45 (calculated based on data in Mashal et al.)21 This study: 0.49
Thailand	Tak Province in 2007: 0.17 (Thailand demographic surveillance; pers. comm., Oliver Morgan) National in 2008: 0.16a, 0.25b This study: 0.23	[no survey found] This study: ≥0.49
Malawi	Southern and Central Regions (surveys): 2005: 0.45 2006: 0.10, 0.26, 0.30, 0.40, 0.40, 0.41, 0.41, 0.80, 0.90, 1.90, 2.20, 2.50 (median: 0.41) 2007: 0.38 National in 2008: 0.41a, 0.33b This study: 0.51	2003: 1.57 2005: 1.20 2006: 0.40, 0.78, 1.60, 2.10, 3.40, 3.80 2007: 0.71, 0.82 This study: 0.84
Tanzania	[no regional surveys found] National in 2008: 0.36a, 0.30b This study: 0.18	2004: 0.72c This study: 0.43

Unless otherwise referenced, data are as reported by the Centre for Research on the Epidemiology of Disasters’ Complex Emergency Database (http://www.cedat.be)²², after excluding reports of zero mortality on plausibility grounds.

^aUS Census Bureau International Database.²³

^bUNICEF data using data from the UN Population Division's ‘United Nations World Population Prospects: The 2008 Revision’.²⁴

^cData converted from rate per 1000 live births to rate per 10 000 person-days using the following calculation: [under 5 mortality rate per 1000 live births (112) × 1000 × birth rate per 1000 persons (42.4)] × (10 000/365)/[proportion of under 5's in the population (18.2%)]. All data from the Tanzania Demographic and Health Survey 2004–05.²⁵

Discussion

To our knowledge this is the only recent study with the aim of developing and validating a new mortality estimation method based on primary data collection. The informant method potentially offers a more rapid means of estimating mortality than retrospective surveys and surveillance systems. The results show that the informant method performed better than the active surveillance system in the Tanzania refugee camps, but achieved moderate sensitivity compared with a best estimate of mortality, and was not good at identifying deaths in children <5 years of age. The informant method also cannot be used to record data on exposure to mortality risk factors, whereas retrospective surveys can potentially record such data.

There are different reasons why the informant method did not identify some deaths. The choice of key informants may have been sub-optimal and inappropriate for childhood deaths. This may have arisen because of dominance of community leaders among FGD participants, which may have masked the diversity of opinions in the community and discouraged other participants from expressing their views, resulting in over-direction by community leaders about which key informants we used. Furthermore, in both Kabul and Chiradzulu, key informants derived their information about deaths from similar sources. A greater number and diversity of key informants may have been warranted.

It may have been inherently difficult for key informants to know about all deaths in their sectors, due to: (i) large sector populations (about 3000 people in Kabul, 2000 in Mae La and 4500 in Tanzania); (ii) lack of trust and thus information sharing between key informants and community members (e.g. in Kabul many residents were short-term renters without kinship ties to the wakils); and (iii) inexperience of key informants (e.g. in Mae La many section leaders were newly appointed as part of a repatriation scheme).

Household respondents provided a minority of all referrals; given the relatively low mortality in the study sites, this may have been because households did not know about other deaths. Additionally, households may have been reticent to share such information with strangers, wary of upsetting community leaders or the bereaved families themselves or acting based on other cultural and/or religious beliefs and practices.

Both key informants and households may also have deliberately withheld information on deaths due to sensitive causes, such as suicide or drug addiction. This issue would, however, affect alternative mortality measurement methods and hence is not a particular weakness of this method. Furthermore, the main mortality rate estimate would not be affected if these deaths were mentioned, but falsely attributed to a more socially acceptable cause.

Theoretically, the informant method presents considerable advantages over both surveillance and surveys for the purposes of real-time mortality measurement. It is a one-off activity that can be repeated on a regular basis by staff with limited research skills. The small number of interviews conducted (i.e. only in households with deaths) greatly reduces data entry requirements; a simple analysis can be done with pen and paper, as there is no need for weighting and design effect adjustment; and the questionnaire can be expanded to explore the timing/location of the death (e.g. if the death occurred when residing in the study site or beforehand) and the causes and circumstances of death through methods such as verbal autopsy, since considerably more interview time can be devoted to each household than for other methods. The findings on issues of feasibility of the new method (e.g. time, financial, use of verbal autopsy questionnaires and ethical implications) will be presented elsewhere. A potential disadvantage is the need for accurate population estimation. However, population estimation is frequently required by humanitarian agencies for operational response in crisis-affected and resource-poor settings, and so is itself a useful measurement activity. Like retrospective surveys (but not surveillance), the informant method is also affected by potential survival bias (households disintegrating during the course of the recall period), though the potential effect of this bias may be small in our case due to the short recall period.

The method's sensitivity did not reach 80%, the level we aimed for when developing this method. Based upon our experience of developing and testing the method, we speculate that it may be possible to increase sensitivity by: (i) ensuring that the FGD participants represent a mix of men and women, ages, occupations and economic and social hierarchies; (ii) using multiple rounds of FGDs; (iii) using other formative research methods alongside FGDs, such as in-depth individual interviews or informal discussions; (iv) using more than two informants: in this study, we easily collected additional lists of deaths and; (v) using less formal informants (e.g. groups of people at gathering points such as water sources, shops).

Limitations of this validation study

Capture–recapture analysis may not have provided an accurate estimate of mortality. The method is highly sensitive to accuracy in matching individuals who appear in more than one list. In this study we matched on the basis of place of residence, sex, age and name: the latter two variables are often problematic in settings without vital registration. While some over- or under-matching may have occurred (resulting in under or overestimation of the true number of deaths, respectively), we did not note instances of equivocal combinations of the above characteristics, probably because the chance of repetition was very low (there were almost always less than five deaths reported on any list, for a given sector). Other problems with establishing the lists occurred, however. In Kabul, hospital records were often incomplete, leaving us with a very small list that decreased the precision of the capture–recapture estimate. In Mae La and Chiradzulu, data were collected through religious leaders and Health Surveillance Assistants, respectively, and we were not able to closely supervise data quality: if some of the deaths reported by these sources did not in fact meet our inclusion criteria, the total number of deaths would have been overestimated and sensitivity underestimated.

Establishing the correct date of death was challenging. It was apparent that many household respondents and informants had difficulties pinpointing the actual date of death. We attempted to minimize error through a series of questions about each death and a detailed calendar containing local salient events. The sensitivity of the informant method was lower for 60 days compared with 30 days, which suggests that respondents’ recall was worse over time: a well-known problem for retrospective epidemiological studies. Non-systematic misclassification of date of death, particularly around the 60-day mark, would generally have led to less overlap among lists, and thus underestimation of sensitivity and overestimation of the true number of deaths: this may explain the curious finding that, in all four sites, more deaths were estimated to have occurred during the period 60–30 days before the survey, than in the last 30 days (Table 6, last column). This finding, if not spurious, may also be due to a simultaneous seasonal effect across all four sites, though this is less plausible.

Capture–recapture analysis also entails arbitrary decisions in the choice of statistical models, though we attempted to minimize these through Bayesian averaging. Alternative choices of models did not significantly alter the main sensitivity findings (data not shown).

Despite these potential biases, the mortality rates we estimated through capture–recapture reflect expected patterns given the underlying epidemiological and demographic profile in the study sites, and were broadly consistent with recent survey data from neighbouring areas within the study countries^20–22 (Morgan, O, personal communication) and also national-level estimates.^23–25 In Afghanistan, we investigated an urban neighbourhood of Kabul that is well served by health facilities and reasonably well covered by preventive services and where mortality would be expected to be lower than the nationwide average, which includes under-served rural areas. The camp-based populations we investigated in Thailand and Tanzania enjoyed long-standing humanitarian assistance and comparatively good health services, which helps explain why their mortality rates are similar to or lower than the local and national mortality rates. In Chiradzulu district of Malawi, the CMR recorded by the informant method falls within the range of survey data from the local region and is slightly higher than the national rate, which is to be expected, as the national rate includes urban areas, which generally have lower mortality rates.

Alternative best-estimate options included prospective surveillance systems, but these either did not exist in the study sites or were suspected to have poor sensitivity, as suggested by a review of camp-based surveillance.³ We did not compare our estimates with retrospective household surveys as to our knowledge these have not been validated sufficiently to provide a reliable best-estimate comparator.^2,4,7,26 Moreover, retrospective mortality surveys typically estimate mortality rates averaged over several months and are therefore not directly comparable with mortality estimates from the informant method, whose main objective is to use very short recall periods.

Lastly, we were unable to test the method in settings of high mortality, and further studies are required to understand how well it would perform in such settings.

Conclusions

This study suggests that a new method to estimate recent mortality, based on information provided by community informants, while performing at least as well as other available methods in crisis-affected populations, still missed a substantial proportion of all deaths, particularly among children. Overall, the method's sensitivity did not reach 80%, the level we aimed for when developing this method.

However, we believe the method shows sufficient promise to warrant further development given the paramount importance of real-time mortality measurement, particularly in crisis-affected populations. Variants of our method are already being used to identify cases of maternal mortality and severe acute malnutrition in the community,^9–11,27 and further applications may be envisaged. These could cover supplementing or evaluating existing mortality surveillance systems (including cause-specific mortality such as maternal mortality, as recommended elsewhere);²⁸ monitoring cause-specific proportional mortality within the context of vertical disease programmes; rapidly identifying persons living with disabilities or investigating outbreaks of diseases with very recognizable symptoms. Improving sensitivity through strengthened methods of selecting appropriate informants and eliciting their knowledge should be the focus of future research and development of this method.

Supplementary data

Supplementary data are available at IJE online.

Funding

This study was funded by the generous support of the American people through the support of the Office of Health, Infectious Disease, and Nutrition, Bureau for Global Health, United States Agency for International Development (USAID), through both the FANTA-2 Project under the terms of Cooperative Agreement Number GHN-A-00-08-00001-00, and the FANTA Project (1998–2008) under the terms of Cooperative Agreement Number HRN-A-00-98-00046-00, managed by the Academy for Educational Development (AED). The contents do not necessarily reflect the views of USAID or the United States Government.

KEY MESSAGES.

• Current methods to estimate mortality in crisis-affected and resource-poor settings feature important methodological limitations and have not been sufficiently validated.

• We developed and validated a new method for estimating mortality that potentially offers a more rapid and timely means of estimating mortality compared with existing methods.

• The new method showed only moderate sensitivity for assessing mortality but it demonstrates sufficient promise to warrant further development given the importance of real-time mortality measurement in such settings.