Cash, COVID-19 and aid cuts: a mixed-method impact evaluation among South Sudanese refugees registered in Kiryandongo settlement, Uganda

Abstract

Objectives

In this paper, we estimate the impact after 4–8 months of a large one-off unconditional cash transfer delivered to refugees during a time of dual shocks: the COVID-19 pandemic and cuts to monthly aid. We focus on four key outcomes: (1) health-seeking behaviour; (2) COVID-19 specific preventive health practices; (3) food security and (4) psychological well-being.

Methods

We use both quantitative and qualitative data to understand the impact of a cash transfer in this context. Quantitatively, we use a baseline survey of 1200 households (Q4 2019) and follow-up with three rounds of phone surveys in Q2 and Q3 2021, capturing at least half the sample in each round. We exploit an experimental variation in the timing of the cash transfer to assess the effect of the cash transfer through ordinary least squares regressions of intention to treat. Controlling for key baseline characteristics, we analyse the effect of the cash transfer on health access, COVID-19 health practices, food security and psychological well-being. Qualitatively, we make use of a longitudinal, small-n sample of refugee respondents, each of whom we interviewed up to 15 times between February and September 2020 to understand change over time and to go deeper into key topics.

Results

We do not find a statistically significant effect (6.2%, p=0.188) of receiving the cash transfer on preventative measures against COVID-19. However, households receiving the cash transfer were more food secure, with a 14.4% (p=0.011) improvement on the food security index, have better psychological well-being (24.5%, p=0.003) and are more likely to seek healthcare in the private health facilities (10.4%, p=0.057) as compared with control households. We do not find significant results on the value of food consumption. Overall, we find stronger treatment effects for households that were the first to receive the cash transfers.

Conclusion

Taken together, we find significant support for the importance of cash transfers to refugee households mitigating against declines in food security and mental well-being in the face of shocks.

WHAT IS ALREADY KNOWN ON THIS TOPIC.

• COVID-19 pandemic might have affected already vulnerable populations such as refugees more than the general population.

• Cash transfers provide some protection against the acute effects of shocks, including those related to the COVID-19 pandemic among the general population.

• It is not well established how effective cash transfers are in the context of refugees already facing aid cuts during COVID-19.

WHAT THIS STUDY ADDS

• We show that cash transfers provided significant levels of support against food insecurity.

• Refugee households receiving cash transfers were also had better mental health.

• No strongly supportive evidence of cash transfers on COVID-19 preventive behaviour was found.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE AND/OR POLICY

• Households receiving the transfer earlier than pandemic onset have more food security and mental health protection, a finding potentially driven by transfer timing.

• Emergency social protection interventions should consider not only protective but also promotive aspects in form of timely disbursements.

Introduction

Public health emergencies, such as COVID-19, represent a widespread, negative shock on health and well-being outcomes, including food security, healthcare-seeking and mental health.1–3 The COVID-19 pandemic has perpetuated further economic disruption and global aid shortfalls 1 4 5 leaving the livelihoods of many vulnerable households, including refugees in settings of protracted displacement,6 at risk. Timely (and even anticipatory) unconditional cash transfers (UCTs) in humanitarian contexts may help to mitigate some of this vulnerability.7 Large, one-off cash transfers have improved health and well-being in rural East African contexts—in the absence of a widespread crisis.8 Smaller cash transfers in a crisis context have also lessened crisis impact,8–10 including on mental health.9 10 However, limited research has focused on how these types of transfers can help specific subpopulations, such as refugees, who may already face restricted access to services and investment opportunities. For instance, a 2017 systematic review found only five publications concerning the effects of cash transfers in humanitarian settings.11 Since the Doocy and Tappis review study, some papers have emerged with consensus on the protective potential of cash transfers in humanitarian settings across a range of outcomes.12 13 However, these studies have been only in the context of civil wars and not in situations as pandemics. Moreover, most of these studies also consider only small (monthly) cash transfers and not large one-off transfers as in our case. Only one study14 has attempted to elaborate on the possible effects of social protection on Ebola virus disease survivors in Sierra Leone. During the COVID-19 crisis, emerging evidence indicates a generally protective function of cash transfers on welfare15 16 and healthcare use17 within the general population. We, to the best of our knowledge, do not know any research assessing the effect of cash transfers in refugee settings during COVID-19. This paper intends to bridge this gap, reporting on the intersection of a large cash transfer programme to refugees, the COVID-19 pandemic and aid cuts.

We describe—quantitatively and qualitatively—the health and well-being of refugee households registered in Kiryandongo settlement, focusing on healthcare access, food security, psychological well-being and COVID-19-specific practices. We report results from a randomised evaluation of a large, one-off UCT delivered before and during the early days of the pandemic. This study was planned before the pandemic began, but the transfers in the treatment group happened to coincide with the early days of the COVID-19 lockdown.

We hypothesise that cash transfers relax refugee households’ financial constraints, enabling them to buy preventive equipment such as face masks and hand sanitisers. We further hypothesise that transfers support adherence to preventive regulations, reducing the risk of contracting the illness. Finally, we predict that transfers will improve recipient food security and psychological well-being of cash transfers recipients compared with non-recipients.

These are short-term results. Our main study relies on the phase-in of cash transfers over around 2.5 years, and will deliver results on economic outcomes after 2 years. In this analysis, we use the same randomisation to explore shorter-term health outcomes, in the context of the COVID-19 pandemic and aid cuts. We find supportive evidence that cash transfers mitigate the decline in household food security and psychological well-being amid a dual shock. We do not find strong effects on the index of COVID-19 mitigation, though mask-wearing improved.

Context: settlement, COVID-19 and aid cuts

Kiryandongo refugee settlement

Uganda is one of the world’s leading refugee-hosting countries, with 1.5 million refugees and asylum-seekers across 14 settlements. Uganda has progressive policies; these permit refugees to move freely outside the settlements, work, start businesses, engage in farming activities on rented land and access public services including education and healthcare.18–21

We focus on households registered in the Kiryandongo refugee settlement, situated next to the main Kampala-Gulu highway and the district’s commercial centre, Bweyale. Unlike some settlements, and not counting the COVID-19 lockdown, movement between the settlement and town is short (between a 10 and 40 min motorcycle ride) and with no physical barriers.

Kiryandongo is co-managed by the Office of the Prime Minister and the United Nations High Commissioner for Refugees (UNHCR) providing shelter, land and support to ~10 000 households. Established on cleared ranch land in the 1970s, Kiryandongo now hosts mostly (99%) South Sudanese refugees facing protracted displacement. Most current settlement inhabitants arrived in or after 2014, which marked South Sudanese independence and the ensuing civil war. On arrival, UNHCR registered household heads and their families—broadly defined—and issued an attestation card; households use this to access settlement services.22 Each household received around two acres of land and a few materials, such as a tarp, to build a shelter.

Most households are Nuer, Dinka and Acholi/Luo, though the settlement houses over 10 ethnicities. Many households live in makeshift or traditional huts; some have built semipermanent houses using half-burnt bricks and iron sheets. Permanent houses, using fully burned bricks and cement, are officially forbidden.

Health system and access

For health infrastructure, Bweyale has two government health centres, which serve both refugee and host communities. In the settlement are two large clinics, run jointly by the UNHCR and the Government of Uganda (GoU). Non-Governmental Organisations, with UNHCR support, provide healthcare and psychosocial support. While Uganda operates an integrated refugee policy that enables refugees and Ugandans to access the same health services, there are barriers to accessing quality care in the Ugandan public health system, such as stockouts, negative perceptions of staff attitudes, and long waiting times.23–26 In areas around Kiryandongo refugee settlement, there are several, mostly Ugandan-run, private clinics and drug retail shops. In part, private facilities serve as a (costly) buffer against the well-documented stockouts of supplies, diagnostics and tests in the Ugandan public health system.27 28

COVID-19 and lockdown

On 30 March 2020, Uganda entered a nationwide lockdown to prevent COVID-19 spread, restricting almost all movement and commerce in the country and across international borders. In June 2020, GoU, UNHCR and some NGOs started issuing free masks and small bottles of sanitisers to people in the settlement. Masks were also available for sale, ranging from UGX1000 to UGX2500 (~US$1 to US$2 PPP) (We use the World Bank’s Purchasing Power Parity (PPP) conversion factor for private consumption for the official exchange rate per international US$ in 2020, which was 1267.2. The price level ratio of PPP conversion factor (GDP) to Ugandan Shilling market exchange rate for 2020 was 0.356 (World Bank, 2021)) per mask and UGX20 000 (~US$16 PPP) per 1 L bottle of sanitiser, with smaller 100 mL bottles priced around UGX8000 (~US$6 PPP).

By October 2020—during our last round of phone surveying—GoU had announced a phased reopening of schools though schools did not open until January 2022. Additionally, non-essential businesses could operate and religious gatherings could take place under daylight curfew conditions.

Aid and aid cuts

The World Food Programme (WFP) provides households with monthly food or cash rations (their choice) for each registered household member. These constitute roughly one-quarter (23%) of baseline consumption. In April 2020, WFP hastened already-planned aid cuts as COVID-19 stretched its budget.5 WFP trimmed rations by 30%, to around US$17 PPP per registered household member. Further, as a COVID-19 precaution, WFP rescheduled ration distribution to take place every 2 months rather than monthly.29

Intervention: UCTs via mobile money

The implementing organisation, GiveDirectly (GD), aimed to give a one-off UCT via mobile money to all households registered in Kiryandongo refugee settlement (~10 000) plus ~5000 nearby Ugandan (‘host community’) households. GD chose Kiryandongo, in part, because of its size, its stability (no official new arrivals since 2016), and its relatively calm security situation. Giving to Ugandan households accords with the UN’s and GoU’s Refugee and Population Empowerment framework (ReHoPE), requiring that in allocating aid for refugees, 70% goes to refugees and 30% to host communities. Although similar numbers are not available for their Kiryandongo programme as it is ongoing, in a pilot programme in another refugee settlement in Uganda (Kyaka II), GD reported that they transferred 83% of their donor funding to beneficiaries, with the rest going to planning, management, operations and similar.

GD chose to make the transfers using mobile money based on prior experience and recipient feedback; they assessed the mobile infrastructure available in Kiryandongo as adequate. Compared with cash, mobile money is more secure for distribution and provides recipients with privacy and security. Customers can withdraw their transfers from mobile money agents or ATMs and make direct mobile money payments.

Targeting

As per its most-recent operating model, GD aimed to saturate the settlement with transfers, meaning that each household would receive a transfer. GD saturates a setting when no clear, easily explainable rationing criterion exists.

GD planned to deliver transfers to UNHCR-registered household heads. In the Kiryandongo settlement, and in our sample, about three-quarters of registered household heads are female. Official headship can mask a variety of de facto living and power arrangements.

Amount

GD aimed to provide a US$ 1000 (~3000 PPP) UCT to all households, in line with GD’s standard practice. This represents 3 months of household consumption as per average baseline results and is equivalent to nineteen months of monthly household consumption support from WFP (assuming an average household size of nine members). While the US$1000 transfer is one-off (not recurring), GD disbursed in three instalments—a smaller first instalment of UGX520 000 (~US$418 PPP) and two larger instalments of UGX1.605 million (~US$1289 PPP) each. These amounts cover mobile-agent withdrawal fees. Recipients could opt to receive a basic mobile phone from GD, in which case GD would deduct UGX68 000 (~US$55 PPP) from the final instalment to cover the cost. Around 90% of treated households chose a phone.

Transfers to our treatment group began in February 2020, and continued until August 2020. We summarise the transfer timing by cohort in table 1. Cohort 1 recsueived transfers earlier on average, starting in February 2020; cohort 2 households started receiving their first instalment in March 2020. During the first few months of the COVID-19 lockdown (April–June 2020), households in cohort 1 had received more transfers on average than households in cohort 2. By end-April, GD had distributed 39% of transfers to cohort 1 but only 18% to cohort 2. By July, operational processes allowed cohort 2 to catch up in transfer receipt and by August 2020, cohort 1 and cohort 2 had received, on average, ~80% and ~83% of their transfers, respectively.

Table 1.

Monthly transfer values

Month	Cohort 1		Cohort 2
	Share of average, cumulative total transfer value	Average, cumulative transfer value (in 2020 US$ PPP)	Share of average, cumulative total transfer value	Average, cumulative transfer value (in 2020 US$ PPP)
February	10%	US$287.28	0%	US$0.00
March	11%	US$337.05	11%	US$326.02
April	39%	US$1156.23	18%	US$528.20
May	68%	US$2031.69	52%	US$1556.06
June	72%	US$2142.28	65%	US$1921.51
July	77%	US$2290.66	81%	US$2418.32
August	80%	US$2382.75	83%	US$2454.86

Sensitisation

For refugees, their first contact with GD came in community sensitisation meetings (barazas). GD staff held at least one baraza in each settlement administrative cluster. Through barazas, GD staff, in their dark green vests, became recognisable and well known in the settlement, building trust. Staff explained the transfer and how the roll out would work. They further shared security advice to prevent fraud or theft, and warned against rumours. This meant that most people in the settlement were likely aware of the transfer, regardless of cohort assignment. GD does not impose conditions on how households can use their transfer, they suggest recipients could invest in farming activities, buy livestock, build houses, enrol children in school, buy clothes, start small businesses, buy food and seek medication among other things.

Registration and audit

To enrol a household, GD staff register each household with a mobile telecommunication company and provide them with a free SIM card. GD staff also provide basic digital and financial literacy training, covering phone use and safe-keeping for SIM cards and personal details. During later audit visits, GD staff verified details collected during registration and reiterated digital literacy and safety points.

Distribution

GD aimed to follow a 1-month cadence between enrolment and disbursement of each of the three instalments. Logistical factors disrupted this plan; we provide more information on the actual timing of the transfers for our study population in a later section.

Follow-up, mediation and ongoing support

After each instalment, GD calls customers to ensure they got the money and everything is in order. This includes addressing household conflicts or possible fraud, if applicable. In case of conflicts, GD pauses the transfer until they have investigated. The investigation may lead GD to take mitigative action, like splitting the transfer or stopping it completely in extreme cases.

GD runs a toll-free hotline in several languages to address any questions or concerns of recipients. During the COVID-19 pandemic, GD shared information about the virus and provided referrals to other organisations or service providers if requested by its customers.

During COVID-19, with the increased vulnerability of the study population, there are legitimate concerns that sticking to the planned intervention schedule would have amounted to withholding critical support from the participants during a crisis. We agree that participants’ welfare would have been most likely negatively affected and speeding up the transfer would have been helpful. However, it is important to note a few key points. First of all, even in normal circumstances, GD’s speed of enroling households was limited by manpower and logistics, which was the original reason for distributing the transfers over 24 months. With new COVID-19 preventive measures including lockdowns, these operational constraints were magnified as GD transitioned to remote operations, causing delays in the speed of transfers. Overall, operational constraints simply prevented the potential strategy of speeding up distribution once COVID-19 hit. Second and most importantly, was the suspension of GD’s licence to operate in Uganda from early September 2020. This suspension by the GoU implied that all operations, including a new COVID-19 relief package to close to 200 000 households, were halted.30 The licence was only reinstated in November 2021 with a ‘second chance’ from the president.31 These were conditions beyond which GD would have had any control. Therefore, even had GD been able to overcome operational constraints to speed up transfers during the pandemic, the suspension would have eventually prevented this.

Finally, to ensure more data security, we encrypted and uploaded all data to a secure central database. We stored back-ups on password-protected computers and folders to ensure data confidentiality. Only the research management team had access to the encrypted raw data. Finally, we provide a structured appendix32 and a reflexivity statement33 both in online supplemental file 1.

Initial design and additions

We designed the evaluation to include a baseline after randomisation, and 1-year outcome measures. As part of the initial design, we also launched a small-n, longitudinal study to connect with about 32 refugee households—both early and late transfer recipients—monthly for in-depth interviews. With the onset of COVID-19, we sought funding to examine shorter-term impacts amidst shocks. Mildmay’s Research Ethics Committee and the Uganda National Council for Science and Technology (UNCST) approved all study components, including COVID-19 alterations and additions.

Randomisation

In August 2019, around 9000 refugee households in Kiryandongo were randomised into 24 cohorts in a public lottery.34 We held the lottery during the WFP’s monthly food distribution; this event brings most households registered in the settlement to a central point to receive their rations. After receiving their food or cash, households went to tables to pick a numbered ball out of a bucket, giving them their cohort assignment. We did randomisation ‘with replacement’, so we did not expect equal numbers of households per cluster. As planned, households with Persons with Specific Needs started receiving their transfers in October 2019; these households were neither part of the randomisation nor the study. Then, each lotterised cohort would receive their transfer 1 month apart, starting with cohort 1 in February 2020. The treatment group comprises cohorts 1 and 2; the control group comprises a random sample of households from cohorts 17 to 20, initially slated to receive their transfers in mid-2022.

Unfortunately, we detected some patterns in the lottery results that suggest deviations from pure randomisation. The number of households in cohorts 1 and 2 exceeded what we would have expected by chance (9.6% realised vs 8.3% expected, difference significant at p<0.010); full histogram of cohorts in online supplemental figure S2. This suggests some ‘cheating’, likely due to the lottery facilitators occasionally recording incorrect draws (perhaps as a favour to friends). While this is disappointing, we believe it has limited consequences for interpreting our results for two reasons. First, although the difference from expected draws is statistically significant due to the large sample size, it is practically small, as it suggests around 177 households in cohorts 1 and 2 above expectation. Second, we control in our regressions for a rich set of baseline covariates (include the baseline value of our outcome variable when we have it), which should absorb any differences in the group induced by imperfect randomisation.

GD preferred disbursements by cohorts for three reasons: operational constraints in signing up households, concerns about inflationary pressure and the fact that at the start of the project, GD had not yet secured the necessary funds to saturate the settlement at once. GD needed to roll the transfers out over time and deemed, in consultation with community leaders and settlement authorities, public randomisation a transparent way to determine the timing. At baseline, 90% of respondents stated the lottery was a fair approach; 85% thought it the fairest approach possible given the constraints.

Patient and public involvement

The study did not have any clinical patients. Community members participated in the open cohort randomised allocation process. No other members of the public were involved in the design, conduct, reporting or dissemination of this research.

Samples and data collection

Quantitative sample and data collection

At baseline, we sought to reach 840 households assigned to cohorts 1–2 (treatment) and a random sample of 840 households from cohorts 17 to 20 (control). (These cohorts were chosen to balance a few factors. For treatment, we chose the earliest cohorts to have the longest exposure time before our endline survey. For control, we wanted to ensure that all participants in the study would eventually receive transfers, which was not guaranteed at the outset due to funding limitations, which eliminated the last cohorts. And we wanted to guard against anticipation effects, so we did not want to select cohorts that would receive transfer right after our endline survey around month 12. Therefore, we chose cohorts 17–20 for control to balance these concerns). During data collection, however, we discovered that a significant number of UNHCR-defined households used during randomisation did not always match the realities of household composition in the settlement. This was problematic for the study as some of our control households lived with family members who would receive a cash transfer before the follow-up survey. Additionally, finding households using the UNHCR household list was challenging, as information was often outdated. Therefore, we surveyed households where all members were exclusively listed in cohorts 1 and 2 and 17–20 with no overlaps. Our final sample comprised 1264 households.

At baseline, we conducted in-person data collection from end-September to end-November 2019. Surveys were translated and administered in Acholi, Dinka, English, Juba-Arabic, Nuer and Bari languages. We collected data using Survey CTO on tablets; the average survey length was about 2 hours. Most enumerators were South Sudanese refugees living in or near the settlement.

Between July and October 2020, we conducted three rounds of rapid phone surveys with a subset of the baseline sample. The phone surveys took about 10 days per round with interviews lasting, on average, 30 min. A group of baseline enumerators also conducted the phone surveys. Online supplemental figure S1 shows the timelines of events and data collection from initial lotterising to the third phone survey that acts as our endline.

Out of 1264 baseline households, we had active phone numbers for 1202 households. Based on experience, we predicted a response rate of 50%–60%, allowing for a meaningful minimum-detectable effect size (0.23, assuming alpha of 0.05% and 80% power). To improve response rates, we implemented a callback protocol whereby enumerators made multiple call attempts across different days. We offered respondents a small compensation for participating (approximately US$ 3 PPP mobile money transfer).

During each round, we reached and interviewed between 61%–65% of the treatment group and 57%–59% of the control group. Overall, 74% of eligible households (baseline households with a phone number) responded to at least one of the three survey rounds. In each survey round over all, our response rates were higher than other phone surveys in similar contexts during crises.35 The Consolidated Standards of Reporting Trials diagram in figure 1 shows our final sample.

Figure 1.

The Consolidated Standards of Reporting Trials (CONSORT) flow diagram for sample selection. PSN, Persons with Specific Needs.

We conducted balance checks between our treatment and control groups of key baseline variables, including baseline values of our regression outcomes (when available) and pre-specified covariates (see online supplemental table S1). Most variables were balanced but there were significant differences in some ethnicity categories. We believe this stems from the deviations from pure randomisation, mentioned above. In table 1, we show the development of the sample and phone survey response rates across all three rounds.

In online supplemental table S2, we explore differences in household characteristics between those who responded to our phone survey and those who did not. We find the responders are more educated, have spent a longer time in the settlement and are less likely to be in the Nuer ethnic group.

Qualitative sample and data collection

We randomly selected 32 refugee households for our qualitative sample, stratified on ethnicity and gender (34% of our original sample were male and 66% female respondents). Absent time to ‘soak and poke’ towards thematic saturation, we relied on random selection to determine our sample.36 A 32 household sample exceeds the saturation point in other studies.37 38 In our original plan, enacted in February and March 2020, each respondent was interviewed in-person once a month. However, when the COVID-19 pandemic reached Uganda and government authorities enacted lockdown measures, our team switched to remote data collection. This resulted in a reduction of our sample size to 17 respondents. The reduction was caused by multiple factors, including some respondents not owning a phone or living in areas with no network. We also could not rely on our interpreters during phone interviews, so we had to drop some households that could only communicate in a language not spoken by our enumerators. The reduced sample included 47% male and 53% female respondents. It also distorted ethnic representation in our sample; most ethnicities remained represented although not proportionally). Because of connectivity and attention issues when going remote, we decided to speak to respondents more frequently but for shorter durations. (Frequency increased from once to thrice per month; duration reduced from ~90 min to ~30 min per interview). Between January and September 2020, we interviewed each of these respondents up to 15 times.

Two interviewers carried out all semistructured interviews. They each spoke with the same set of respondents over time, building rapport. Both interviewers are male enumerators from baseline that we trained and coached in qualitative interviewing, note-taking and transcription. One interviewer is half-Ugandan, half-Sudanese and lives in Bweyale; the other is Ugandan. Interviews took place in a language comfortable for both respondent and interviewer and were audiorecorded.

Throughout our interviews, we covered multiple topics. In each interview, we asked about general well-being and life updates before going in-depth on topics that we regularly revisited (such as household decision-making and market behaviour) and standalone topics (such as the health and education system). We provide a more detailed overview of topics covered in online appendix supplemental table S3. (All data collection instruments for both our quantitative and qualitative work can be found at https://www.idinsight.org/project/unconditional-cash-transfers-in-kiryandongo-refugee-settlement-uganda).

At the beginning of the project, we had high acceptance into our small-n panel, including equal acceptance across the sexes. There was also similar heterogeneity across men and women in terms of the quality and depth of responses given, which we interpret as being a function of how we built our sample (through random selection) and provides reassurance that having male interviewers did not have differential effects on response quality. When we transitioned to telephonic interviews, we had a harder time reaching female respondents. Among the 17 respondents we reached at least once during the telephonic interviews, men responded to 69% and women to 63% of all call attempts. We interpret this as more likely related to competing demands on women’s time rather than the influence of having male interviewers.

Data analysis

Quantitative impact evaluation

Our analysis is a simple ordinary least squares regression of each outcome variable on treatment status, that is, the cohorts’ intended transfer receipt. As specified in the pre-analysis plan for our original study (https://www.socialscienceregistry.org/trials/6271/history/73771) on medium-term outcomes, our regressions control for various baseline characteristics including sex of household head, household size, time in the settlement and ethnic group. If we measured an outcome variable at baseline, we include the baseline value of this variable as a control. The analysis is ‘intention to treat’, in that we assign a household to the treatment group regardless of whether they actually received the transfer.

In our original preanalysis plan, we include household consumption, a food security index and psychological well-being as primary indicators. As these were included in our phone surveys, we treat them as primary indicators (Note that we only collected the food portion of household consumption). Our food security index combines several questions about hunger and meals, as used by other researchers of cash transfers.8 Our psychological well-being index combines three popular well-being measures: the World Values Survey (WVS) Happiness Questionnaire, WVS Life Satisfaction Questionnaire39 and Center for Epidemiological Studies (CES) Depression Scale. We also add an index of COVID-19-related practices as a primary indicator, as determining the impact of cash on COVID-19-related practices motivated the phone surveys. We created the index by asking about COVID-19 practices—mask-wearing, hand washing, staying home, social distancing—and combined them in an inverse-covariate-weighted index.40 We also report results from each index component separately, and related secondary outcomes. We account for the possibility of false detection due to multiple outcomes by using the Benjamin41 methods of sharpened q values.

There is considerable attrition in each phone survey round from the baseline sample. Since the non-response is not correlated with treatment, we believe our results are unbiased for the sample of respondents. As a robustness check, we include results that incorporate inverse probability weighting to correct for non-response.

Qualitative analysis

We conducted a thematic content analysis in Excel, accounting for the longitudinal nature of some of our questions.42 Before beginning analysis, we developed a deductive codebook; we added codes inductively, in consultation with one another, when unanticipated responses arose or to provide further nuance. We include direct quotes from our analysis when they represent a commonly held view.

Findings

We begin by explaining the transfer roll out, then present descriptive and impact results for our main outcomes of interest, including both quantitative and qualitative components. In table 2, we show our impact evaluation results for our primary outcomes and in table 3, we present results for index components and secondary outcomes. For qualitative results throughout, unless we explicitly note it, there was no suggestive patterning by sex or ethnicity.

Table 2.

Treatment effects of main outcomes

	(1) Control mean	(2) Treatment effect	(3) P value	(4) q value	(5) 95% CI	(6) N	(7) Data
A. Unweighted
Covid practices index	0.161	0.059	0.215	0.239	(−0.034 to 0.153)	633	R1
Food security index	−0.056	0.144**	0.010	0.022	(0.035 to 0.254)	1284	R1 and R3
Food consumption	117.819	2.575	0.676	0.511	(−9.520 to 14.670)	630	R2
Psychological index	−0.114	0.241***	0.003	0.009	(0.085 to 0.397)	632	R1
B. Weighted by inverse probability weights
Covid practices index	0.161	0.062	0.188	0.204	(−0.031 to 0.156)	633	R1
Food security index	−0.056	0.144**	0.011	0.023	(0.034 to 0.255)	1284	R1 and R3
Food consumption	117.819	2.205	0.736	0.468	(−10.621 to 15.031)	630	R2
Psychological index	−0.114	0.245***	0.003	0.008	(0.086 to 0.404)	632	R1

The table reports treatment effects of four main outcomes. Panel A reports the unadjusted treatment effects and panel B is adjusted by inverse probability weights. These indices are created using the method described in Anderson.40 All regressions control household characteristics including the gender of household head, household size, time in settlement and different ethnicities. The treatment effects of Food Consumption are calculated using ANCOVA analysis by controlling for the baseline values of the outcomes. The treatment effects of Covid Practices Index, Food Security Index and Psychological Index are calculated using OLS regression without baseline adjustment due to the unavailability of corresponding baseline measurements. Column (1) shows the means of control group. Column (2) and (3) show the treatment effects estimates and the p values, respectively. BKY (2006) sharpened two-stage q values based on the four outcomes are shown in column (4). CIs are reported in column (5). Columns (6) and (7) show the number of observations and the survey round in which the data are collected. Significance levels correspond with *** p<0.01 for 1%, ** p<0.05 or 5% and * p<0.1 for 10% respectively.

ANCOVA, analysis of covariance; OLS, ordinary least squares.

Table 3.

Extended treatment effects on COVID-19 practices, psychological well-being, food security and conflicts

Dependent variable		Treatment effect		95% CI		(6) P value	(7) N	(8) Data
	(1) Control mean	(2) Unweighted	(3) Weighted	(4) Unweighted	(5) Weighted
A. COVID-19 practices
Access sufficient water	0.799	0.018	0.018	(−0.044 to 0.079)	(−0.042 to 0.078)	0.559	633	R1
Access soap	0.871	0.008	0.007	(−0.042 to 0.059)	(−0.045 to 0.059)	0.788	633	R1
Always wearing masks	0.801	0.041**	0.043**	(0.007 to 0.076)	(0.008 to 0.079)	0.016	1819	R1 & R2 &R3
Social distancing	0.382	−0.037	−0.034	(−0.083 to 0.010)	(−0.081 to 0.013)	0.157	1707	R1 & R2 &R3
Staying at home	0.089	0.034	0.036	(−0.015 to 0.082)	(−0.013 to 0.085)	0.153	633	R1
B. Food security
Eaten less-preferred food	0.834	−0.016	−0.016	(−0.058 to 0.026)	(−0.058 to 0.026)	0.465	1283	R1 & R3
Gone to bed hungry	0.515	−0.054**	−0.056**	(−0.108 to 0.000)	(−0.111 to –0.002)	0.043	1283	R1 & R3
Skipped or cut meals	0.74	−0.048*	−0.043*	(−0.097 to 0.002)	(−0.093 to 0.007)	0.092	1284	R1 & R3
Gone whole day without food	0.45	−0.021	−0.026	(−0.073 to 0.030)	(−0.078 to 0.027)	0.335	1283	R1 & R3
Household Dietary Diversity Score	3.948	0.447***	0.459***	(0.186 to 0.708)	(0.184 to 0.734)	0.001	630	R2
C. Psychological well-being
CES-D scale	2.65	0.105*	0.103*	(−0.001 to 0.211)	(−0.004 to 0.211)	0.058	632	R1
World Value Survey’s Happiness Question	2.216	0.159**	0.167***	(0.035 to 0.284)	(0.041 to 0.293)	0.010	630	R1
World Value Survey’s Life Satisfaction Question	3.956	0.597***	0.602***	(0.261 to 0.934)	(0.262 to 0.943)	0.001	622	R1
D. Health services accessibility
Unable to access health services	0.123	0.018	0.021	(−0.044 to 0.080)	(−0.043 to 0.085)	0.520	468	R1
Visited private health facilities	0.091	0.104*	0.105*	(−0.014 to 0.222)	(−0.003 to 0.213)	0.057	149	R1

Panel A–C report treatment effects on outcomes of COVID-19 practices, Food Security and Psychological Well-being, respectively, whose summary indexes are shown in table 2. Panel D reports treatment effects on outcomes of Health Services. The treatment effects are calculated using OLS regression. All regressions control household characteristics including the gender of household head, household size, time in settlement and different ethnicities. Column (1) shows the means of the control group. Columns (2) and (4) show the unweighted treatment effect estimates and 95% CI, respectively. Columns (3), (5) are weighted by inverse probability weights. Columns (6) shows the p values (weighted). Columns (7) shows the number of observations and (8) shows the round in which the data are collected. Significance levels correspond with *** p<0.01 for 1%, ** p<0.05 or 5% and * p<0.1 for 10% respectively

OLS, ordinary least squares.

COVID-19-specific health practices

Most respondents (81%) thought it was possible to protect themselves from contracting COVID-19. A substantial section of respondents believed that prevention was possible through their actions, including hand-washing, mask use, staying home and physical distancing.

Most respondents reported having access to sufficient water (82%) and soap (87%) for regular hand-washing (about eight times) during the day. Eighty-nine per cent of respondents reported covering their mouth and nose with a face mask while leaving their homes. Although GoU started distributing free masks in June, the majority (82%) who wore masks reported buying them. (Our interview team reports that government-issued mask quality was perceived as inferior by some people). Most respondents had left the house in the last 7 days (89%); 47% of these maintained social distance while outside.

Households that received a transfer were not more or less likely to engage in protective practices, as measured by our COVID-19 practices index (see table 2). However, one index component did have significant results: receiving transfers had a positive effect on mask-wearing, as treatment households were four percentage points more likely to report wearing masks (95% CI 0.007 to 0.076) compared with control (see table 3).

Food outcomes

At baseline, food consumption expenditure among refugees in Kiryandongo was similar to the Uganda average for rural areas in 2018/2019.43 Households spent nearly half of total consumption expenditure on food, just at the threshold between medium and low vulnerability to food insecurity.44 According to WFP’s Food Consumption Score thresholds, >99% of the baseline sample had an ‘acceptable’ level of dietary diversity (>99% scored above 35 out of 112) with an average score of 73.7 out of 112. This contrasts with much of what we heard in open-ended interviews following baseline, in which many respondents expressed food security and dietary diversity concerns.

As shown in table 2, receiving a transfer had a positive and statistically significant effect on food security, increasing the index by 0.144 SDs (95% CI 0.035 to 0.254). In table 3B, we present treatment effects on the index components and show that improvement in the index is driven primarily by treatment households reporting a lower incidence of household members skipping meals (by 0.54 days in the last week) or going to bed hungry (0.48 days). We also show that treatment households have higher dietary diversity, consuming an average of 0.45 additional food groups (from a base of 3.9.), where the maximum can be 12. (Note that the Dietary Diversity Score is not part of the food security index.)

Qualitatively, only a few respondents, primarily from households that already received transfers, felt ‘there is enough food for everyone.’ Most respondents had or worried about hunger and had to ration their food, reporting sentiments like ‘hunger is too much for me.’ Some also emphasised that, with COVID-19 and aid cuts, the WFP aid was not enough, that food prices were increasing, and that they faced additional food needs from children staying at home from school during lockdown. Additional concerns revealed through our small-n work, all of which preceded COVID-19, included insufficient land to grow food, the monotony of WFP food aid, and the amount of time, and therefore fuel, required to cook the WFP-provided beans.

Psychological well-being

Respondents in our qualitative work shared considerably more feelings of stress and worry than happiness. Even before COVID-19 and the aid cuts, more respondents shared negative rather than positive feelings when asked how they were faring, including discussing ‘heartbreak’ and feeling ‘occupied’ in heart and mind. With the dual shocks, things turned even more negative and we heard about people being afraid and ‘bursting with anger.’ By far the most common worry mentioned by respondents once COVID-19 arrived is the pandemic and related lockdown measures, followed by food insecurity and lack of money.

Households that received the transfer fared better psychologically than those who did not, with treatment households scoring higher by 0.24 SDs (95% CI 0.085 to 0.397). In table 3, we break out the impacts on each component of the psychological well-being index (WVS Happiness Questionnaire, WVS Life Satisfaction Questionnaire39 and CES Depression Scale).45 Treatment households have higher levels of all components, indicating robust protection of psychological well-being from cash. Overall, happiness and life satisfaction (WVS questions) fell from baseline to the phone surveys. However, control households experienced a larger decrease in happiness and satisfaction levels than treatment households.

Health access

Most refugees seeking care make use of the public health system, either inside or outside the settlement. In July 2020, 83% of those who sought care did so in the public sector. As shown in panel D of table 3, significantly more transfer recipients accessed the private sector for needed care relative to control, suggesting that funds constrain on private care-seeking. Qualitatively, respondents highlighted long waiting times, drug stockouts and too much demand on available facilities and understaffing for public facilities. One respondent explained that ‘you will wait even the whole day, you may come at around 8am and go back at 4pm.’ While several respondents pointed out that private clinics ‘will work on you fast,’ perspectives on quality differences are mixed. For example, one respondent felt that doctors in private clinics rush their work ‘as they are after money.’

Thirteen per cent of households who needed medical attention reported being unable to access services; this did not differ by transfer status. Results after the first lockdown restrictions indicated that between 20% and 25% of individuals who needed medical help did not receive it due to supply-side disruptions.46 Our qualitative findings provide different explanations: a nurse told us that during COVID-19, public facilities did not ‘admit people without critical condition.’ Many respondents also explained that healthcare services for certain conditions or diseases are not locally available. For example, respondents highlighted that diagnosis and treatment of hepatitis B, sickle cell anaemia, cancer, or delivery complications could not be managed locally. They pointed out that such conditions or diseases would require referrals to ‘those big big hospitals’ in Gulu or Kampala. We also learnt that some respondents go back to South Sudan, Kenya, or the Democratic Republic of Congo to obtain medical care.

Heterogeneity by assigned cohorts

Our treatment households were randomly assigned to be in cohort 1 or cohort 2, and (as shown in figure 1), cohort 1 on average received their transfers before cohort 2. Therefore, exploring the treatment effect by cohort can tell us whether transfer timing matters for outcome. In table 4, we reproduce our treatment results on primary outcomes, but report them separately for cohort 1 and Cohort 2.

Table 4.

Treatment effects by cohort

	(1) Control mean	Cohort 1			Cohort 2			(8) t-test (2)-(5)	(9) N
		(2) Treatment effect	(3) P value	(4) 95% CI	(5) Treatment effect	(6) P value	(7) 95% CI
Covid practices index	0.161	0.073	0.201	(−0.039 to 0.185)	0.044	0.454	(−0.071 to 0.159)	0.029	633
Food Security Index	−0.056	0.208***	0.002	(0.077 to 0.339)	0.075	0.276	(−0.060 to 0.209)	0.133*	1284
Food consumption	117.819	4.278	0.561	(−10.167 to 18.722)	0.638	0.934	(−14.423 to 15.700)	3.640	630
Psychological Index	−0.114	0.375***	0.000	(0.188 to 0.563)	0.097	0.318	(−0.094 to 0.288)	0.278**	632

The table reports treatment effects by cohort of four main outcomes. These indices are created using the method described in Anderson.40 All regressions control household characteristics including the gender of household head, household size, time in settlement and different ethnicities. The treatment effects of Food Consumption are calculated using ANCOVA analysis by controlling for the baseline values of the outcomes. The treatment effects of Covid Practices Index, Food Security Index and Psychological Index are calculated using OLS regression without baseline adjustment due to the unavailability of corresponding baseline measurements. Column (1) shows the means of control group. Column (2)–(4) show the treatment effects estimates, the p values and CIs for cohort 1. Column (5)–(7) show the corresponding results for cohort 2. The t-test differences of treatment effects between cohort 1 and cohort 2 are reported in column (8). Columns (9) and (10) show the number of observations and the survey round in which the data are collected, respectively. Significance levels correspond with *** p<0.01 for 1%, ** p<0.05 or 5% and * p<0.1 for 10% respectively

ANCOVA, analysis of covariance; OLS, ordinary least squares.

We find that for the two outcomes for which cash had a significant effect (food security and psychological well-being), the effect is much larger among cohort 1 households. No outcome had a statistically significant result for cohort 2 households. This is somewhat surprising, as for both outcomes one might think the largest treatment effects would be present near the time of the transfer, and would fade over time (as the money gets spent).

Discussion

We find a generally positive effect of cash transfers across several of our outcomes but the story is not always consistent. Here, we focus on findings for food security and cohort effects, as these presented us with interpretation puzzles.

While we see an improvement in the food security index for households who received cash transfers, this is contrasted by the fact that we did not see increases in total food consumption, as well as the fact that many households interviewed as part of the qualitative work were still struggling with having enough to eat. We take this to mean that while cash transfers help, they have a limited effect. We believe that timing plays a big role. We measured the food security index in rounds 1 and 3 and measured consumption in round 2. Round 2 took place right after the harvest, when food was likely more abundant. It is, therefore, possible that during this time, households with cash did not spend more on food because they did not need to. Additionally, we believe that timing issues likely affected various responses in the qualitative study. Households tend to have more difficulty with food when it has been a while since they have received food distribution from UNHCR.

Cash transfers eased psychological distress in statistically significant margins in this study. Systematic review evidence of multiple studies from low-income countries shows that cash transfers generally improve psychological and mental well-being47 and are essential in humanitarian settings,48 including the COVID-19 pandemic.10

Regarding health access, COVID-19 and the consequent lockdowns negatively affected health services access for other non-COVID-19 health needs across many low-income countries49 and many countries have had to optimise their service delivery through revised priority setting.50 Service access barriers are likely to be more profound for vulnerable households such as refugees. In this study, cash transfers increased health services access, especially doubling the probability of utilising private health services. This effect, like mask usage, would have been most likely driven by the increased capacity to demand, eased by removing some liquidity bottlenecks.

Cohort effects

We found that most of the treatment effects were dominated by households in cohort 1. Recall that cohort 1 received about 11% of the transfer (the first instalment of US$410 PPP) in February 2020, before the COVID-19 lockdown. While only a fraction of the planned transfer, this amount was almost five times higher than other emergency cash transfers during COVID-19 in Uganda.51 Cohort 2, receiving their cash right in the middle of the lockdown, did not show significant effects. Food prices had already increased, while at the same time households’ inability to travel limited access to markets. While not anticipated, the importance of transfer timing has been found in other programmes as well, with notable benefits from preshock emergency support.52 53

Limitations

As with many studies, ours has some limitations. In this section, we list a few key ones.

First, one may ask questions about external validity, given that this study was conducted during a unique situation: a once-in-a-lifetime pandemic combined with aid cuts. And yet, with the climate crisis and looming antibiotic resistance, there is every reason to believe that public health emergencies will become more common. There is, further, every reason to think the existing consumption support programmes, such as UNHCR and the Nobel-winning WFP, will continue to be stretched too thin as more people are displaced, and for longer durations, across and within national borders. Therefore, we believe that the learnings from this shock may apply to future shocks.

Next, due to the COVID-19 lockdown we gathered our quantitative outcome data using phone surveys, and were only able to reach around 50% of our sample each round. Although this attrition was not correlated with treatment status, our results may not be representative of the entire population of the settlement.

Given that our study is randomised at the individual level, it is possible that our treatment had some spillover effects on control households. This could be through control households receiving support from treatment households, anticipatory effects, and economic general equilibrium effects. While we designed our study to mitigate these effects to the extent possible given constraints, we have not eliminated them completely.

Also, the baseline took place after randomisation, which was necessary since, for budgetary reasons, we could only survey a subset of cohorts. However, this means that randomisation could have affected some of our baseline outcomes, and indeed we see this through baseline imbalance on psychological well-being.

Next, the sex of our interview team for both quantitative and qualitative data collection—all male in both cases—could have affected rapport with respondents. This dynamic might have especially affected our qualitative data. However, our interviewers worked hard to build rapport and we see that, in general, our female respondents are more verbose than are our male respondents. While we cannot rule out differential openness and response quality had we been able to recruit female team members and have matched-sex interview-respondent pairs, we feel confident that we received rich, truthful responses.

Finally, as noted earlier, deviations from our desired randomisation procedure could mean that our control group does not represent an ideal counterfactual.

Conclusion

Households registered in the Kiryandongo refugee settlement faced a dual set of shocks: COVID-19 and the consequent lockdowns as well as cuts to monthly consumption support. Four to eight months after these shocks, a large cash transfer provided a modest, statistically significant corrective by supporting food security and psychological well being. Still, both our quantitative and qualitative data show that many households consistently report hunger and worry. Overall, we find that refugees registered in the Kiryandongo refugee settlement can and do take (self-reported) measures to prevent COVID-19, with limited impact of a large cash transfer on compliance. While overall healthcare utilisation declined, likely from lockdown and fear of COVID-19, the transfer allowed those seeking care to do so in the private sector. We found stronger effects among households who received transfers earlier, timed to just before and during the early days of the lockdown. Our study does not find any significant differences/effects by ethnicities, underlining the covariant nature of the COVID-19 shock.

Taken together, our findings suggest that cash transfers can be a key part of a resilience package during times of crisis but that—at least at the level provided here—it does not alone do all the work.

Data availability statement

Data are available on reasonable request. Data used in this study are still under embargo, awaiting the endline and final impact evaluation.

Ethics statements

Patient consent for publication

Consent obtained directly from patient(s)

Ethics approval

The initial protocol for the Kiryandongo impact evaluation was approved by Mildmay Uganda Research Ethics Committee MUREC-2019–0101 and the Uganda National Council for Science and Technology (UNCST) duly provided a research permit (SS281ES) to implement the study. To understand the effects of COVID-19, an amendment to the protocol was submitted and approved by MUREC in May 2020.