Assessing the impact of microfinance programming on children: an evaluation from post-tsunami Aceh

Abstract

This paper presents an evaluation of the long-term impact of microfinance programmes on Acehnese children during the post-tsunami recovery. The study, conducted from June to August 2010, examined the impact of microfinance programming six years after the tsunami. The sample consisted of 185 microfinance participants, with a comparison group of 192 individuals who did not participate in microfinance programmes. All respondents were parents, interviewed through a structured survey. The study used four child protection indicators—diet, health, childcare and education—in contrast to traditional repayment rate indicators. The primary results were insignificant with respect to all four child protection indicators, suggesting that, with respect to these indicators, there was no long-term difference between the impact of microfinance on beneficiaries' children and non-beneficiaries' children. These findings signify a need for microfinance actors to move beyond traditional indicators of economic success to evaluate the social changes microfinance programmes are presumed to effect.

Introduction

From June to August 2010 Columbia University, the Women's Refugee Commission and the University of Indonesia conducted an evaluation study of the Group Guarantee Lending and Savings programme employed by Save the Children in Aceh, Indonesia, as part of its post-tsunami reconstruction efforts. Taking place six years after the event, the goal of the study was to examine the long-term impact of micro-finance initiatives on the well-being of Acehnese children. This study was conducted within a social protection framework—an increasing area of emphasis within both humanitarian and development contexts. UNICEF defines social protection as ‘a set of public and private policies and programmes aimed at reducing and eliminating economic and social vulnerabilities to poverty and deprivation’ (UNICEF, 2012, p. 1). Within this framework, child protection actors have emphasised the role that social protection initiatives can play in the promotion of children's long-term economic and social development, while also reducing children's risk of exposure to violence, abuse and exploitation (UNICEF, 2009b).

Increasingly, social protection initiatives have focused on strengthening the financial well-being of families in order to make additional opportunities available to children such as a greater access to education, healthcare, nutritional support and other essential services. This study builds on this premise by examining child well-being based on four primary indicators that are commonly used within the context of child-focused conditional cash transfer initiatives—diet, health, childcare and education—making these areas particularly salient for further investigation. In addition, this study contributes to the literature regarding the role of microfinance initiatives in the aftermath of complex humanitarian emergencies, both in terms of evaluating the efficacy of this approach, and highlighting the need for practitioners to place greater emphasis on more robust monitoring and evaluation practices in the early stages of programme implementation.

Background

Impact of the conflict and tsunami on Acehnese society

Even before the devastating Asian Tsunami of 2004 the province of Aceh, a region on the Indonesian island of Sumatra, was suffering both politically and economically. For decades, Aceh served as the battleground for a protracted civil conflict that began in 1976 with the Free Aceh Movement fighting for independence against the Indonesian government. This 30-year conflict resulted in an estimated 10,000 deaths and ravaged the social and economic fabric of Acehnese society (Roper et al., 2006). Many individuals and families were forced to shift from urban lifestyles and jobs to rural livelihoods, such as fishing and agriculture. Food production and distribution often proved insufficient, leaving 30 per cent of the population living below the poverty line (FAO and WFP, 2005). Due to the imposition of martial law in Aceh, few international NGOs were in the region prior to the tsunami to offer services to conflict-affected communities, thus leaving the Acehnese with very little support or aid (Boothby et al., 2010).

In addition to the hardships imposed by poverty and a long-lasting civil war, on the morning of 26 December 2004, northern Sumatra was hit by a 9.0 earthquake, which triggered a massive tsunami and flooded 11 countries around the Indian Ocean Rim. Aceh was hit the hardest, with an estimated 226,000 dead or missing and approximately 500,000 displaced or left homeless (Brennan and Rimba, 2005).

The economic impact was vast, with an estimated USD 4.45 billion in damages to property, 75 per cent of which occurred in rural areas. This damage was particularly devastating due to the fact that a significant number of Acehnese had moved to rural areas during the conflict in an attempt to create livelihoods opportunities in areas less affected by the conflict (Alexander et al., 2006). The massive waves demolished 63 per cent of fisheries, agriculture and commerce—the three largest livelihoods in the Aceh region. Additionally, much of the natural environment was destroyed, further diminishing the livelihoods of rural villagers. The environmental damage led to decreased arable land, destroyed coral reefs and rare mangrove swamps, and devastated the coastal zone (Alexander et al., 2006). Not only did it affect the livelihoods of villagers, it permanently damaged much of Aceh's infrastructure. About one third of schools, road networks and hospitals were destroyed. An estimated 115,000 houses were completely demolished while approximately 150,000 were left in poor condition. Overall, the total loss in damages was roughly equal to Aceh's GDP in 2003 (Athukorala and Resosudarmo, 2006).

Of the hundreds of thousands of people who were killed during the tsunami, it can be estimated that 100,000 of these deaths were children (Penrose and Takaki, 2006). Devastation continued after the tsunami as well. In a rapid health assessment conducted by the World Health Organization young children had one of the highest mortality rates following the tsunami, along with adults aged 70 and older (Doocy et al., 2007). This may be attributed to a lack of adequate nutrition post tsunami, which had been shown in other post-natural disaster contexts to lead to up to 50 per cent of childhood deaths (Pee et al., 2007). With respect to schooling, while most schools were running two weeks after the tsunami, only 50 to 60 per cent of children actually attended them out of fear of another tsunami (Pairojkul et al., 2010). Thus, children in post-tsunami affected areas were severely affected and were a vulnerable part of the post-tsunami population.

Economic recovery in Aceh through microfinance

In tsunami-affected regions, including Aceh, many economic recovery programmes were implemented to assist job creation to rebuild a functioning economy. One model frequently used for economic recovery in these areas was microfinance. The World Bank defines microfinance as ‘the provision of small-scale financial services to people who lack access to traditional banking services’ (World Bank, 2007, p. 3). Frequently targeting women and those with limited financial resources (World Bank, 2007), microfinance programmes cover a broad range of initiatives, including ‘loans, savings accounts, insurance products and various combinations of these services’ (Odell, 2010, p. 6).

The Grameen Bank model of microfinance, first employed in Bangladesh, focuses on market-based institutions that offer credit to poor families who do not usually have access to credit, in order to increase their income and consumption levels through self-employment (Morduch, 1999b). For the past 20 years microfinance has been used throughout the world and has been regarded as a successful method to alleviate poverty, thus increasing quality of life (CIDA, 2007). Increasingly, microfinance initiatives have also been implemented as a means of improving other well-being outcomes, such as ‘health, children's education and women's empowerment’ (Odell, 2010, p. 11).

Randomised controlled trials (RCTs) are widely recognised as the most rigorous means of evaluating the impact of microfinance initiatives, in light of their ability to compare programme participants against those who did not take part in micro-finance services (World Bank, 2007). In addition, RCTs are deemed effective due to their ability to reduce the risk of selection bias (Odell, 2010). However, as Odell (2010) notes, RCTs cannot be used to examine existing programmes as the structure for evaluation must be established before the programming cycle begins, thereby limiting the ability for RCTs to be used in certain contexts where microfinance initiatives are already underway.

Quasi-experimental studies represent another commonly-used approach to evaluating microfinance programmes, although the strength of this methodology is regarded to be less robust because of the higher risk of selection bias or non-random programme placement associated with this approach (World Bank, 2007). Quasi-experimental studies can be either prospective, by selecting the treatment and control groups prior to programme implementation; or retrospective, in which case the two groups are selected once programming has begun (World Bank, 2007).

Despite the widespread popularity of microfinance initiatives among development practitioners, a limited evidence base exists by which to examine the impact of these programmes. As Bateman (2011) notes, early research on the impact of microfinance initiatives—such as the work of Khandker (1998), Littlefield et al. (2003) and others—reported economic gains among programme participants. Bateman notes, however, that early evaluations were primarily carried out by those affiliated with microfinance institutions, raising questions of potential bias (Bateman, 2011). Bateman also notes that later research carried out by independent investigators—such as Morduch (1998) and Coleman (1999)—questioned the validity of earlier studies due to ‘data and methodological problems’ (Bateman, 2011, p. 2).

Similarly, in a systematic review of microfinance evaluations, Duvendack et al. find a lack of rigorous impact studies, and note that the majority of available evaluation reports are based on ‘weak methodologies and inadequate data’ (Duvendack et al., 2011, p. 4). They also write that four prior reviews of microfinance interventions were conducted (Gaile and Foster, 1996; Goldberg, 2005; Odell, 2010; Sebstad and Chen, 1996), and note that each of these reviews finds that evidence in support of the impact of microfinance programming is based primarily on ‘anecdotes and inspiring stories’ (Duvendack et al., 2011, p. 2) rather than scientific investigation.

While research highlighting the positive impact of microfinance programming has increasingly been called into question on methodological grounds, Bateman (2011) also highlights how new research incorporating the use of RCTs—including the work of Karlan and Zinman (2009), Banerjee et al. (2009) and Roodman and Morduch (2009)—has shown microfinance interventions to have little to no impact on the economic status or overall well-being of programme participants. Offering a more nuanced perspective, however, is Bauchet et al.'s (2011) review of randomised evaluations, which identifies the results of microfinance initiatives to be ‘modest’, producing such outcomes as altering household consumption patterns away from non-essential items, and leaving additional income available to promote new business development. In addition, Bauchet et al. (2011) find that men benefited more than women in the studies they examined, and that there was minimal impact on other outcomes such as health and access to education. Similarly, a 2011 review by the Independent Evaluation Group (IEG, 2011) suggests that there is greater evidence to support positive short-term outcomes as a result of microfinance programmes, while the evidence on long-term outcomes remains less conclusive.

Indicators and challenges of successful microfinance programmes

The Grameen Bank model has traditionally used the rate of loan repayment from the borrower as its main indicator of success. After 35 years of administering microfinance loan programmes in Bangladesh, Grameen reported a success rate of up to 98 per cent repayment (Morduch, 1999b). However, the suitability of this indicator to measure programme impact has been questioned in recent years. Critics argue that the indicator of repayment rate does not measure the social benefits these programmes are meant to foster, nor the negative repercussions that may occur from receiving microfinance loans (Banerjee et al., 2010; Morduch, 2000). Woolcock (1999), for example, noted that there was little research on the wider benefits that microfinance programming is meant to support in addition to increasing accounts of conflicting social consequences of these programmes. He provided an example of how, in order to pay a microfinance debt, a participant may take another loan from a neighbour, thus perpetuating the debt cycle.

In recent years the Grameen Bank has attempted to move beyond repayment rates to assess the impacts of microfinance on family income with the development of the Progress out of Poverty Index tool (Grameen Foundation, 2012). Additionally, USAID and the Iris Center have developed the Poverty Assessment Tools to move beyond repayment rates for understanding household well-being on a more significant level (USAID, 2012). Despite these advances, however, there is still limited evidence in understanding the impact of microfinance programming on the well-being of children and families.

Impact of microfinance with respect to children and families

With respect to the impact of microfinance on children, specifically, some programmes seem to have demonstrated positive outcomes. Holvoet (2004) evaluated the impact of microfinance programmes in south India and discovered that if a woman was a part of a women's group microfinance programme as opposed to a direct bank loan, her children were significantly more likely to be in school or even to attend private school. In fact, Holvoet (2004) found that the children of microfinance recipients in women's groups were 3.2 to 3.9 times more likely to be enrolled in private school, and about 2.7 to 3.5 times more likely to be capable of reading and writing.

Other programmes, however, seem to have shown mixed results for the children of microfinance borrowers. The Canadian International Development Agency (CIDA), for example, implemented a multi-country evaluation in India, Egypt, Tanzania and Bolivia, all of which have histories of implementing active microfinance programmes (CIDA, 2007). The purpose of the evaluation was to examine the participation of children in their parents' microfinance programmes, and to understand the overall benefit to children among family members who were actively involved in microfinance. The study concluded that when family income levels improved, so did their spending on their children's welfare. Education was a main focus when income was improved, followed by health care, housing and nutrition (CIDA, 2007). Additionally, CIDA found a significant amount of increased involvement of children in the businesses their parents had adopted because of the microfinance loan. The positive implications appeared to be that the children received practical and entrepreneurial skills that could be useful to their future careers. However, CIDA (2007) also discovered that since 80 to 85 per cent of microfinance recipients were women, their children were forced to pick up extra roles in the house, increase their time spent on business-related activities and household chores, and decrease their time spent on schooling. In some cases, children were forced to quit school completely, suggesting a negative influence on their future (CIDA, 2007).

Description of Save the Children's microfinance programme in Aceh

Following the tsunami, the international organisation Save the Children implemented a host of programmes aimed at different areas of recovery. One of these programmes, the Economic Recovery Programme (ERP), provided cash grants and microcredit loans to tsunami-affected families. Following the perceived success of the ERP programme, Save the Children supported two additional microfinance programmes intended to improve the economic security to families affected by both the tsunami and the Acehnese conflict. One of these programmes, the Group Guarantee Lending and Savings, was developed in collaboration with a local service provider, Baitul Qiradh Afdhal (hereafter referred to as ‘Afdhal’), to provide loans to women, with the hypothesis that mothers would use the extra earned income for the family unit. Afdhal used a ‘group collateral’ approach whereby participants were composed of groups of at least 10 women. The hypothesised strength of this method was that if one person was unable to make her loan payment, the entire group would be held accountable for that payment, increasing the likelihood of repayment. The individual defaulter was also expected to repay the group, creating a local form of social accountability (Save the Children, 2008).

In order to increase beneficiaries' knowledge about managing finances, Afdhal also implemented a savings programme for the borrowers. Save the Children's records indicated that as of October 2008, the programme in Aceh reached 2,583 active borrowers and distributed approximately USD 108,000 in loans. The savings programme also recorded approximately USD 53,000 in individual savings (Save the Children, 2008).

Save the Children's evaluation of the microfinance programme

In 2008, Save the Children implemented a programme evaluation of the Group Guarantee Lending and Savings programme. In its evaluation, Save the Children employed a quasi-experimental design using retrospective sampling, with a control group selected from individuals on a waiting list to participate in its microfinance programme at the time the evaluation was conducted. Although the findings of this evaluation are limited by its retrospective design, its results indicated that programme outcomes were positive. As of October 2008, Save the Children's records stated that the repayment rate of loans was 99 per cent. The other indicators included, but were not limited to, the following: total outstanding loan per month; average net income per month of microfinance clients earned from their micro enterprise or business; and percentage of microfinance borrowers who were able to support their children more than before as a result of income earned from microfinance activities.

The evaluation produced numerous positive conclusions, one of the most significant of which was that most microfinance borrowers were food secure. Another was that Afdhal was able to offer microfinance assistance to individuals and families who had no access to any form of loans beforehand. Afdhal loan recipients used 97 per cent of their loans to improve their business and were successful in increasing their savings and assets. Finally, the microfinance programme was able to increase the regular income of the borrowers (Save the Children, 2008).

The microfinance evaluation suggested that Save the Children and Afdhal's programme had a positive impact on the intervention communities, at least in terms of indicators traditionally used to evaluate microfinance programmes (for more information on traditional indicators of microfinance, see Morduch, 1999a). Save the Children, however, is an organisation devoted to the well-being of children. Save the Children did include one indicator of child impact related to economic improvement but, like the other indicators, this was based primarily on financial metrics such as increased ability to buy school books for children, as opposed to whether a child actually attends school. In other words, although Save the Children did evaluate child well-being from microfinance loans through repayment rate indicators, it was still a financial metric that lacked any social or protective indicators. As such, it is reasonable to conclude that an evaluation examining the programme's social impact on children was warranted.

The study

Purpose of the study

In June 2010, Columbia University, the Women's Refugee Commission, and the University of Indonesia were invited by Save the Children to undertake an evaluation to examine the impact of the Group Guarantee Lending and Savings programme on the well-being of children in order to fill the knowledge gap described above. A team of global advisors supported the development and oversaw the study. Fieldwork was led by one University of Indonesia faculty member, three Acehnese local researchers and two Columbia University Masters candidates. A Columbia University faculty member visited the team at the beginning of the fieldwork to provide technical support. Research for this evaluation took place from June to August 2010, thereby examining the impact of microfinance interventions six years after the tsunami.

The study aimed to measure the long-term social impacts of Save the Children's post-tsunami microfinance programme on children by analysing the effects on health, childcare, diet and education. As mentioned in the sections above, in the context of livelihoods research, it is commonly assumed that microfinance interventions will be beneficial to low-income families (Morduch, 1999a). The hypothesis of this study was that microfinance programmes would positively influence the children of the beneficiaries with respect to the four child protection indicators specified above.

Indicators of the study

The four child protection indicators selected for the evaluation—health, childcare, diet and education—were specifically chosen because they are widely recognised global indicators of the status of children (UNICEF, 2009a). Additionally, it was agreed that these indicators would allow for the evaluation of social and child protection outcomes, moving beyond the monetary influence on children. Diet was measured through evaluation of the number of daily meals and what those meals consisted of with respect to normative foods in that region. Health was measured by how often and under what financial circumstances people were able to access clinics when needed. Childcare was measured by identifying who was caring for younger children, whether parents had time to take care of their children if they were working, and how much responsibility older siblings had for childcare. Finally, schooling was evaluated by drop-out and school attendance rates.

Methods

Sample

The evaluation targeted populations on the northern coast of Aceh Province—Aceh Utara, Bireuen and Lhokseumawe—which participated in the Afdhal micro-lending programme. The complete list of recipients, which numbered 10,000, was culled to include only those who lived in areas impacted by the tsunami. Recipients from regions that were conflict-affected but had not been affected by the tsunami were excluded, thus leaving a sampling frame of 688 individuals.

Sample size was determined based on an assumed group size of 10, a desired power of 80 per cent, a confidence level of 95 per cent, an effect size between 0.2 and 0.3, and a design effect of two. The effect size estimation was determined to be representative of the level of change that psychological and social interventions generally aim to produce (Ager et al., 2011). A design effect of two was determined because of the ‘clustering’ of programme recipients into micro-lending groups of 10 or more to use the ‘group collateral approach’ (Save the Children, 2008). As a result, the original desired sample size of 75 was doubled and a sample size of 150 was targeted for the intervention group. This number was increased to 285 based on the already established groups, and the anticipation of large loss to follow up and some refusals to participate in the study.

The comparison group was selected from neighbouring villages, which were matched as closely as possible to the villages of the intervention group according to specific criteria. The matching criteria included: (1) being in the same district; (2) being similarly affected by the tsunami; (3) having a similar geographic make-up (rural versus urban); and (4) having a similar population size. Matched communities were also reviewed to ensure similar socio-economic status before the tsunami. The intervention group earned an average monthly salary of IDR 1,299,934 (USD 130), while the comparison group earned an average of IDR 1,352,485 (USD 135). For additional demographic details of the intervention and control groups, see Table 1.

Table 1. Demographic characteristics of intervention and control groups.

Characteristic	Intervention group	Control group	P-value
Average age	39.20 years (sd 9.8) n = 183	35.12 years (sd 9.4) n = 193	P < 0.0001
Female respondent	163/180 (90.6%)	166/193 (86.0%)	P = 0.174
Male respondent	17/180 (9.4%)	27/193 (14.0%)
Head of household	Yes: 53/180 (29.44%) No: 127/180 (70.56%)	Yes: 46/193 (23.83%) No: 147/193 (76.17%)	P = 0.220
Multifamily household	Yes: 59/181 (32.60%) No: 122/181 (67.40%)	Yes: 41/186 (22.00%) No: 145/186 (78.00%)	P = 0.023
Adults in household	3.13 (1.73) n = 181	2.75 (1.27) n = 192	P = 0.016
Children in household	2.72 (1.73) n = 182	2.53 (1.20) n = 193	P = 0.230
Financially responsible children in household	2.54 (1.35) n = 183	2.52 (1.20) n = 191	P = 0.928
Income earners in household	1.77 (0.77) n = 183	1.55 (0.78) n = 193	P = 0.007

Source: authors.

The national research team worked with the village leaders from the intervention villages to identify the best matches to be selected as comparison villages. A total of six communities were matched with the nine intervention communities (see Table 2).

Table 2. Sample selection for matched communities.

Intervention community	Matched comparison community
1. Gampong Pande District: Aceh Utara Tsunami impact: Slight Location: Rural Population: 1,013	Meucat District: Aceh Utara Tsunami impact: Slight Location: Rural Population: 987
2. Keude Aceh District: Lhokseumawe Tsunami impact: Slight Location: Urban Population: 2,530	Kuala Meuraksa District: Lhokseumawe Tsunami impact: Moderate Location: Rural Population: 987
3. Mon Geudong District: Lhokseumawe Tsunami impact: Slight Location: Peri-urban Population: 5,400	Kuala Meuraksa District: Lhokseumawe Tsunami impact: Moderate Location: Urban Population: 2,530
4. Pusong Baru District: Lhokseumawe Tsunami impact: Severe Location: Peri-urban Population: 5,700	Pusong Lama District: Lhokseumawe Tsunami impact: Severe Location: Peri-urban Population: 4,980
5. Cot Mamplam District: Lhokseumawe Tsunami impact: Slight Location: Rural Population: 2,800	Meunasah Manyang District: Lhokseumawe Tsunami impact: Moderate Location: Rural Population: 1,382
6. Meunasah Tengoh District: Lhokseumawe Tsunami impact: Slight Location: Rural Population: 650	Baloy District: Lhokseumawe Tsunami impact: Slight Location: Rural Population: 650
7. Aleu Mangki District: Bireun Tsunami impact: Moderate Location: Rural Population: 780	Linggong District: Bireun Tsunami impact: Moderate Location: Rural Population: 753
8. Gp Melinteung District: Bireun Tsunami impact: Moderate Location: Rural Population: 311	Linggong District: Bireun Tsunami impact: Moderate Location: Rural Population: 753
9. Alue Bie District: Bireun Tsunami impact: Moderate Location: Rural Population: 300	Linggong District: Bireun Tsunami impact: Moderate Location: Rural Population: 753

Source: authors.

Because comparison villages did not have clustered groups in the same way that the intervention communities did, the research team employed systematic sampling in each comparison village. A sampling interval was determined by dividing the total population of the village by the number of desired interviews—to match the number of interviews obtained in the corresponding intervention village. If the person selected was not home, refused an interview or had participated in a micro-finance programme, a neighbour was approached for an interview. As women were the primary targets for microfinance loans, the women who took the microfinance loan were the preferred respondents. To accommodate cultural standards, however, if the husband wanted to contribute to the responses, his feedback was also accepted.

A total of 377 individuals participated in the survey with a total response rate of 67 per cent. This included 185 intervention respondents from 19 groups (65 per cent response rate) and 192 comparison group respondents (68 per cent response rate). The primary reasons selected individuals did not participate in the study included not having children (thus making them ineligible). Additionally, in the intervention group, there was a small additional loss to the primary sample due to selected participants having moved from the area since borrowing and due to ‘fictitious borrowers’. The fictitious borrowers came largely from a small number of Afdhal borrowing groups. It was assumed that these unknown people were made up in order to borrow loans for someone else.

Tools

A structured survey was the main instrument used to evaluate the four child protection indicators in this study. The survey was quantitative and consisted of 35 questions that covered general demographic information, specific information about family composition and economic status as well as key questions targeting the four child protection indicators of interest (diet, health, childcare and schooling), based on the definitions of each indicator described above. All of the questions relied on a one-year recall period.

General demographic questions sought to understand the household structure and size, income levels prior to and following the tsunami, and what type of benefits, if any, the family received following the tsunami. The survey was administered as a face-to-face interview. Supplemental qualitative data was collected from both adults and children, the results of which have been reported elsewhere (Sparling and Gordon, 2011).

Validity and reliability of the survey instrument

An examination of secondary data on school absenteeism was undertaken halfway through the research study in order to validate a portion of the findings from the initial survey. Gaining permission from parents to examine school records also allowed the team to collect a subset of more detailed information on absenteeism, childcare and child labour.

A sub-sample of the original survey sample was selected and those participants were re-interviewed for two reasons. First, the household survey tool was developed specifically for this study and, while it was tested prior to field use, the research team wanted to ascertain the validity of certain measures. Primarily, the team desired to test whether there was a bias from mothers reporting their children's enrollment at school. The team sensed that there could be a recall bias, or the desire to appear favourably in front of foreign researchers where there might be something to gain. Second, during the course of research, the research team came to learn that some of the measures failed to capture complete data on an original indicator, and the team wanted the opportunity to clarify some trends. For example, in the original survey, the research team examined dropping out of school as the main measure of ‘missed schooling’. Following a preliminary analysis, however, the team decided that absenteeism was a better measure for missed schooling in this context, since attrition was fairly low.

In each district, one intervention and one comparison village were selected to further validate the survey by accessing school attendance records of study participants' children. The research team returned to the same villages and found the respondents from both intervention and comparison groups, and requested consent to access their children's school records. These participants were also asked where their children went to school, how many days of school their children missed in May of 2010 and in the past year, and the reasons they missed school. The information collected was then compared to the information attained in the school records.

A total of 155 respondents from both groups were visited a second time, found to be home, and agreed to participate (79 respondents from the intervention group and 76 respondents from the control group). Fifty-nine schools were visited, and 315 childrens' school records were verified from the families that were re-interviewed.

Data analysis

All data were entered and cleaned in Excel then analysed in SAS 9.2. All tests were run with a 95 per cent confidence level and all appropriate tests controlled for age, gender and level of tsunami impact. Univariate and bivariate analyses were conducted at the five per cent significance level. Continuous covariates were compared across groups using a two-sample t-test procedure and categorical covariates were compared across groups using a chi-square test procedure.

Results

Primary outcomes

The multiple logistic regression model was designed to evaluate whether receipt of a microloan (X_prime) had an impact on the four primary child well-being indicators, described above. The univariate analysis of the primary outcome measures (Y₁, Y₂, Y₃, Y₄ = enrollment at school until age 18, child attendance at health clinics, primary care-taking responsibilities for children, and diversity of available foods and number of meals per day) were all null at the five per cent alpha level, controlling for age, gender and the degree to which families self-reported the impact of the tsunami on their household (see Table 3).

Table 3. Primary outcomes.

Indicator	Test	Rates (intervention/control)	95% CI for rates	Adjusted OR	95% CI for OR	P-value
	Primary outcomes: N = 377
School enrollment (≤18 enrolled in school = yes vs. no)	Log procedure controlling for age, gender and tsunami impact level	90.17%/91.33%	(87.14%, 95.52%)/(85.73%, 94.61%)	0.880	(0.399, 1.942)	0.75
Clinic access (No money to take child to clinic = yes vs. no)	Log procedure controlling for age, gender and tsunami impact level	55.56%/43.52%	(32.99%, 54.07%)/(45.77%, 65.35%)	1.614	(0.818, 3.181)	0.17
Diet (number of meals per day = 3 vs. 2)	Log procedure controlling for age, gender and tsunami impact level	9.34%/7.84%	(4.01%, 11.61%)/(5.11%, 13.57%)	1.138	(0.484, 2.678)	0.77
Childcare (No one vs. someone)	Incidence of ‘no one’ available to take care of the child unobserved or too low to be predictive	100.00%/99.33%	(---,---)/(98.02%, 100.00%)	--	--	--

Source: authors.

Health care was measured by whether a child was financially able to attend a clinic when in need, and whether economic circumstances (such as work responsibilities or insufficient funds) prevented parents from taking the child to a clinic. Both the intervention and comparison groups reported adequate health care for their children. Similarly, no significant difference was found between the average number of meals a child ate per day or what types of foods they generally ate (for example, rice, meat, fish/egg, vegetables and dairy). Additionally, no difference was found in terms of the primary caretaker (for example, the mother, father, siblings, neighbour or nanny). Finally, there was no significant difference in parental reports of children's school attendance and drop-out rates. The research team found no difference between the households that received microfinance and comparable households, indicating that the impact of the programme on children was not detectible in this context, or that there was no salient benefit for child well-being.

Secondary outcomes

To expand on these primary results, the research team analysed underlying differences or ecological variation between the groups with varying loan sizes, using secondary indicators. In these bivariate analyses, the team tested the following secondary indicators: self-reported income change before the tsunami (T₁), immediately after the tsunami (T₂), and at the time of the survey (T₃); usage of assets for business activity; participation of other economic assistance programmes, such as cash for work, or asset replacement; and amount of livelihood training received.

For the next set of bivariate analysis, the research team tested child enrollment in school and absenteeism on parents' income level ‘now’ (T₃). The results of these tests were overall not statistically significant at a 95 per cent confidence level (see Table 4). There was one statistically relevant finding, however. Using a two-sample t-test, the average loan amount predicted whether recipients were still engaged in that particular business (p = 0.012), which provisionally suggests that a higher loan amount could make businesses more sustainable over time. Since this was not part of our primary hypothesis, the finding is purely suggestive and should be explored further.

Table 4. Secondary outcomes and demographic analysis.

Indicator	Test statistic	P-value
Demographic/secondary outcomes:
Overall income change: (income fluctuation before, after and now from tsunami between groups)	Income before tsunami vs. income now t-test = -0.1505; Income before tsunami vs. income after the tsunami t-test = 6.4845; Income now vs. income after tsunami t-test = 7.1072	P = 0.8805 P < 0.0001** P < 0.0001**
Size of loan and still engaged in business: (does X still have original business, Y vs. N)	t = 0.6734	P = 0.5016
Amount of loan and still being engaged in the business: (Y vs. N)	t = -2.5376	P = 0.012 **
Poverty and school enrollment (poor now and no school)	x2 = 0.3408	P = 0.5590
Other economic assistance: (did groups differ on amount of assistance?) N=
• cash for work	x2 = 0.1092	P = 0.741
• asset replacement	x2 = 2.9175	P = 0.088
• training	x2 = 7.3090	P = 0.007**
• all forms combined	x2 = 3.1828	P = 0.315
Poverty and school absence	t = -1.7277	P = 0.0864

Note:

^**= significance at the p < 0.05 level.

Source:authors.

In light of the finding above, the team tested how those families with higher loan amounts compared to those with lower loan amounts in terms of the primary indicators of interest. Among those that received a microfinance loan (n = 183), we dichotomised those that received a loan greater than or equal to the mean value of IDR 22.97 million (USD 2,297) into a high group versus those that received less than the mean value of IDR 22.97 million (USD 2,297) into a low group. Higher loan amount resulted in a higher likelihood trend of health, childcare and nutritional child outcomes; yet resulted in a lower likelihood trend of child attendance in school. None of these results were found to be statistically significant at the five per cent alpha level.

Survey validation outcomes with absenteeism

Paired t-tests were used to compare the average number of reported absences from the mother with the average number of reported absences from school records. The difference between the average absences reported in May was not statistically significant, suggesting that mothers could generally estimate accurately within a month. When running the same test using the average absences during the school year, there was a statistically significant difference in average absences (t = 2.174, p = 0.032). This suggests that recall bias was too great to rely on schooling information remembered for an entire year.

Using the survey validation data, the team tested whether there was a difference between group statuses in annual average absenteeism. Since the data came from a smaller sample (n = 155), the results of these tests were less robust. However, there were on average 2.5 more days missed per year in the comparison group (9.07 vs. 6.4 days per year, p = 0.0216) compared with the intervention group.

In the survey, school absence was categorised as ‘unexcused’ or ‘excused’, which included illness and family affairs. The difference in total absenteeism between groups was consistent with the results of testing ‘unexcused’ versus ‘excused’. The comparison group, on average, missed 5.7 days without parental excuse annually, compared to the intervention group missing 3.5 days. However, the difference in absences, whether they were excused or unexcused did not suggest significantly relevant issues in this situation, since two extra days of school is unlikely to have a significant impact on child well-being (see Table 5).

Table 5. Absenteeism and survey validation.

Indicator	Test statistic	P-value
Validation of survey tool: N = 155
Annual days missed (average between groups)	t = 2.3243	P = 0.02**
Unexcused absence (average between groups)	t = 2.0753	P = 0.04**
Mother-reported vs. school record-reported (comparing average number reported)
• May average	t = 1.23	P = 0.22
• annual average	t = 2.1737	P = 0.03**
Child participation after school in:
• business (y vs. n)	x2 = 0.0995	P = 0.75
• childcare (y vs. n)	x2 = 0.4636	P = 0.496

Note:

^**= significance at the p < 0.05 level.

Source: authors.

Discussion

The findings of this study indicate that there was no apparent long-term difference when comparing the intervention and comparison group, with respect to the four child protection indicators. As mentioned above, health care was measured by whether a child was able to attend a clinic when in need, and if economic circumstances prevented parents from taking the child to a clinic. The lack of a statistical difference between the two groups was attributed to the government social welfare system (Jaminan Pengamanan Sosial), which includes free health care, and was reported to address health needs adequately in both study populations.

Both groups appeared to have similar access to food. Meat was reported to be prohibitively expensive for many of the families, with the main protein source reported to be dried fish. There was some variation in the number of meals per week that included meat, vegetables and dairy, but these trends seemed to follow the poverty level of each family, and were not different based on intervention or comparison status.

Although there were no notable differences between intervention and comparison groups, findings indicate that primary care-taking of younger children is mostly undertaken by older siblings. This raises questions about the burden of responsibility that is placed on young people in this context. Studies in contexts such as sub-Saharan Africa have found that young people often feel overwhelmed by their care-taking responsibilities and feel that they lack an adult who they can turn to for advice and guidance. This research has suggested that these young caretakers and their younger siblings appear to be particularly vulnerable to negative outcomes in schooling, health and emotional well-being (Evans, 2010; Save the Children, 2010).

Certain findings of this study contradict previous evaluations of microfinance programmes. For example, unlike the CIDA (2007) report, this research found there was no increase in school drop-out rates associated with participation in a micro-finance programme. In another study, Holvoet (2004) found that children of mothers who participated in group microfinance programmes were more likely to stay in school, while this study found that children were just as likely to be in school whether their mothers participated in a microfinance programme or not. This suggests that more studies fusing economic indicators and child protection indicators such as schooling need to occur in order to find consistency.

These findings do not imply that economic rehabilitation programmes are unnecessary or unproductive. However, this evaluation suggests that it might be necessary to include other social indicators that match an organisation's mission, such as child protection indicators in the case of Save the Children. Those indicators would allow programme implementers to better understand social outcomes, in addition to financial ones. With more complete information, organisations can better adapt their micro-finance programmes to accommodate the financial and social needs of programme participants. In their programme evaluation, for example, Leatherman and Dunford (2010) identified a significant problem in health within the region they were working, and built in a health-training component to their microfinance programme. The programme was then shown to be successful in improving the livelihood and health of the participants. For example, with respect to diarrhoeal diseases, when microcredit alone was given in the Dominican Republic, there was no change in diarrhoeal disease. When health education was provided alone, the incidence of diarrhoeal diseases decreased by 29 per cent; however, when health education was coupled with microcredit, the incidence decreased by 43 per cent. Our study suggests that more research is needed to establish a positive association between microfinance and child well-being, possibly at earlier stages in a programme cycle in order to maximise desired impact.

In addition, this study highlights the need for a greater emphasis on monitoring and evaluation among humanitarian practitioners, in order to allow for more rigorous evaluations to be conducted at a later date. In the absence of randomised controlled trials or quasi-experimental studies with retrospective designs, it becomes difficult to draw strong conclusions regarding the impact of microfinance initiatives. In this way, the possibility exists that retrospective studies, such as this one, more accurately can be described as measuring the difference between intervention and control groups rather than providing evidence of a particular programme's impact. In the case of this study, since research was conducted six years after the tsunami, it could be that other factors aside from participation in microfinance programming could have contributed to the study's findings, suggesting the need to consider both economic and non-economic outcomes concurrently.

Limitations

There are several limitations that need to be considered when evaluating the study. First, the study design was retrospective, which results in disadvantages such as respondents potentially over-exaggerating or minimising the events, or recall bias.

As previously mentioned, randomised controlled trials represent the most robust method of measuring the impact of microfinance initiatives, followed by quasi-experimental studies that employ a prospective design. In the case of this study, however, these approaches were not possible in light of the fact that programming had already been implemented prior to the time that both intervention and control groups were selected.

Although the study would have been strengthened had it been prospective in nature, the enormity and complexity of the post-tsunami response led implementing agencies to prioritise programming over measurement in the initial phases of their work, so as to address the most immediate needs of survivors. In addition, the decision to delay the implementation of particular programmes in order to randomise participants post-emergency would likely have brought its own set of criticisms and negative attention. At the same time, it is the opinion of the authors that the field of emergency response is beginning to embrace an emphasis on the importance of rigorous evaluations, although a tension still remains between response and measurement, particularly in a setting as all-encompassing as the tsunami.

Additionally, a time analysis was not incorporated into the survey, meaning that while no statistically significant difference between the intervention and comparison group was found with respect to the microfinance programme's influence on child protection indicators, the research team does not know whether the intervention group was better off than the comparison group anytime before the evaluation was done. This means the intervention group could have risen out of poverty faster because of the microloans, and levelled out a few years later. Unfortunately, the study design did not allow for this measurement.

An important limitation was that some of the case definitions for this study were found to be ‘muddy’ and could have used further clarification. For example, for the education indicator, the term ‘dropped out’ was not fully defined. ‘Dropped out’ could have implied a child who had not gone to school for months but intended to return (recidivism), or a child who never intended to return to school (attrition). The impact of this limitation on the study's findings, however, is minimal: only a small number of respondents reported dropping out, making their input into the study not statistically significant. However, greater attention to this issue at the design phase would have increased the study's validity, and is an issue worthy of consideration within the context of future research.

Two different sampling methods were employed to survey the intervention group and comparison group. This was done because the research team used the Afdhal list of lending groups for the intervention group; however, there was no comparable group list for the comparison villages. This might have lead to undetected differences in the groups, either created or concealed by the different sampling methods. Finally, the tsunami response was ubiquitous.

Although the research team attempted to identify matched communities that did not receive microfinance as the comparison group, it is possible there could have been a spill-over affect of the benefits from the intervention area to the communities that did not receive those programmes.

In order to address this limitation, researchers took care in selecting participants for the comparison group, attempting to identify control villages that were similar to the intervention areas prior to the tsunami, and which were affected in a similar way. As such, it was feasible to propose that control areas could have been equally eligible for the same type of microfinance programming being provided in intervention locations. Within these villages, women for the control group were selected, although it was not possible to determine which of these women would have been selected for microfinance initiatives. The decision to select control participants in this way was made in light of an absence of naturally occurring groups of similar women in intervention areas who had not participated in microfinance programming.

As a result of the way in which the control participants were selected, it is possible that the women in these groups possessed a slightly different economic status to those in intervention areas. However, given the overall economic similarities between the intervention and control areas, significant differences in the economic status of women in each location is unlikely.

Another potential limitation is the fact that child well-being was measured from the perspective of the parents and not by the children themselves in this study using indicators that were determined based on international standards alone. Future research may look to incorporate child-centered research methods using a combination of international and locally defined indicators in order to provide insight into how children perceive and experience the benefits of microfinance on their lives. This could present the opportunity for the development of more precise and locally meaningful measurements that are better able to detect levels of child well-being. While the use of case studies could have facilitated a greater reflection of children's perspectives in the study, the time and logistical constraints of this study due to other research objectives did not make such an approach possible.

Conclusion

There are significant questions that arise from this study on microfinance programmes and how ‘success’ is currently determined. Although microfinance programmes may appear to be successful when evaluated by repayment rate indicators, they might not be highly successful when considering other important factors, such as child well-being or even the social impact on the primary borrower.

If the larger scope of a programme is to aid beneficiaries to lift themselves out of poverty, then other determinants of economic and social mobility, such as education and health care accessibility, must also be taken into account when assessing programme efficacy.

There are currently no standardised or validated approaches for measuring the impact of microfinance on children beyond economic indicators, such as repayment rates. This evaluation offers one attempt to fill this gap in the existing literature, and attempts to demonstrate that relying on traditional evaluation indicators alone is not enough to understand how livelihood programmes are affecting children. These efforts have larger implications within the field of microfinance, in terms of measuring efficacy of programmes and true social changes in the lives of the children of borrowers. The subject warrants further study to continue to build the evidence base for linkages between livelihoods and child protection.

This study also contributes to the literature regarding the role of microfinance initiatives in the aftermath of complex humanitarian emergencies, both in terms of evaluating the efficacy of this approach, and highlighting the need for practitioners to place greater emphasis on more robust monitoring and evaluation practices in the early stages of programme implementation. As this study suggests, additional rigorous evaluations of microfinance initiatives are needed in order to better understand the impact of these interventions in emergency contexts, and throughout the transition from relief to development.