Using big data for evaluating development outcomes: A systematic map

Abstract

Background

Policy makers need access to reliable data to monitor and evaluate the progress of development outcomes and targets such as sustainable development outcomes (SDGs). However, significant data and evidence gaps remain. Lack of resources, limited capacity within governments and logistical difficulties in collecting data are some of the reasons for the data gaps. Big data—that is digitally generated, passively produced and automatically collected—offers a great potential for answering some of the data needs. Satellite and sensors, mobile phone call detail records, online transactions and search data, and social media are some of the examples of big data. Integrating big data with the traditional household surveys and administrative data can complement data availability, quality, granularity, accuracy and frequency, and help measure development outcomes temporally and spatially in a number of new ways.The study maps different sources of big data onto development outcomes (based on SDGs) to identify current evidence base, use and the gaps. The map provides a visual overview of existing and ongoing studies. This study also discusses the risks, biases and ethical challenges in using big data for measuring and evaluating development outcomes. The study is a valuable resource for evaluators, researchers, funders, policymakers and practitioners in their effort to contributing to evidence informed policy making and in achieving the SDGs.

Objectives

Identify and appraise rigorous impact evaluations (IEs), systematic reviews and the studies that have innovatively used big data to measure any development outcomes with special reference to difficult contexts

Search Methods

A number of general and specialised data bases and reporsitories of organisations were searched using keywords related to big data by an information specialist.

Selection Criteria

The studies were selected on basis of whether they used big data sources to measure or evaluate development outcomes.

Data Collection and Analysis

Data collection was conducted using a data extraction tool and all extracted data was entered into excel and then analysed using Stata. The data analysis involved looking at trends and descriptive statistics only.

Main Results

The search yielded over 17,000 records, which we then screened down to 437 studies which became the foundation of our systematic map. We found that overall, there is a sizable and rapidly growing number of measurement studies using big data but a much smaller number of IEs. We also see that the bulk of the big data sources are machine‐generated (mostly satellites) represented in the light blue. We find that satellite data was used in over 70% of the measurement studies and in over 80% of the IEs.

Authors' Conclusions

This map gives us a sense that there is a lot of work being done to develop appropriate measures using big data which could subsequently be used in IEs. Information on costs, ethics, transparency is lacking in the studies and more work is needed in this area to understand the efficacies related to the use of big data. There are a number of outcomes which are not being studied using big data, either due to the lack to applicability such as education or due to lack of awareness about the new methods and data sources. The map points to a number of gaps as well as opportunities where future researchers can conduct research.

Abbreviations

CRD

call record details

DFID

Department for International Development

DM

data mining

DMSP‐OLS

Defense Meteorological Satellite Program‐Operational Linescan System

EC

environmental clearance

EPPI

Evidence for Policy and Practice Information

EVI

Enhanced Vegetation Index

GPS

global positioning system

IDP

internally displaced person

IE

impact evaluation

IRB

Institutional Review Board

LMICs

low and middle income countries

ML

machine learning

MNO

mobile network operator

OECD

Organisation for Economic Co‐operation and Development

RCT

randomised controlled trial

SDG

sustainable development goal

SEDAC

Socioeconomic Data and Applications Center

SR

systematic review

SRS

satellite remote sensing

SSCI

Social Sciences Citation Index

1. PLAIN LANGUAGE SUMMARY

1.1. Big data offer unrealised potential for impact evaluations (IEs)

Traditional data collection can be costly; target populations may be inaccessible, phenomena cannot always be directly observed and interviewing people may be unethical, dangerous or impossible. In addition, budget constraints can limit data collection.

“Big data” does not require data collection in the field, and can provide insight into economic, social and political behaviour.

1.2. What is this evidence and gap map about?

Big data consist of data such as online searches, social media, citizen reporting or crowdsourced data, process‐mediated data such as mobile phone call record details (CRD), commercial transactions data and machine‐generated data from satellites, sensors or drones.

Big data can measure outcomes that could not previously be measured using household surveys. The potential of big data to answer causal attribution, however, is still not widely understood, especially in low‐ and middle‐income countries.

What is the aim of this evidence and gap map (EGM)?

This EGM of studies using big data reviews methodological, ethical and practical constraints relating to the use of big data.

1.3. What studies are included?

The map includes IEs that use big data to evaluate development outcomes, systematic reviews (SRs) of big data IEs and other measurement studies that innovatively use big data to measure and validate any development outcomes.

Sources include social networks, internet searches, mobile data, crowd sourced data, data from public agencies, data produced by business, CRD and satellite data.

The map contains 437 studies written in English published between 2005 and 2019: 48 IEs, 381 measurement studies and eight SRs.

1.4. What are the main findings of this map?

There is considerable potential for measuring various development indicators using big data. The measurement studies serve as a proof‐of‐concept for evaluators wanting innovative ways to measure development outcomes.

There is potential for more IEs on development interventions. The map shows that the number of IEs that use big data to measure outcomes or control variables is growing fast, and there is scope for greater use.

Satellite data are used the most. The use of satellite data for IEs and measurement studies has been facilitated by the availability of preprocessed satellite data, new ML techniques and increased computational capacity to process the satellite images into meaningful measures of development outcomes. Despite a number of high‐profile measurement studies, CRD data has not been used to rigorously evaluate any development outcome. Similarly, other human‐sourced data and process‐mediated data have been used only sparingly in IEs.

There are potential sectors and themes where SRs will be useful. The map highlights a few potential thematic areas where SRs will be informative, most notably those of (i) all IEs that have used satellite data; and (ii) those with reference to the data sources used in rigorously evaluating forest cover.

The number of measurement studies indicates potential for more IEs in fragile contexts.

Ethical concerns and transparency issues are substantial. Ethical issues related to informed consent, data privacy, data security and unintended exclusion are severe for some of the sources of big data. Few studies report on ethical issues related to using big data.

Some capacity constraints are acute. Computational capacity is constrained and technical expertise on large‐scale big data analysis is siloed.

1.5. What do the findings of the map mean?

This map shows that big data can contribute to the evidence base in development sectors where evaluations are often infeasible due to data issues.

One of the key “absolute gaps” that the map has identified is that there are fewer IEs than measurement studies. Given the fast‐growing availability of big data and improving computation capacity, there is great potential for the use of big data in future IEs. However, analytical, ethical and logistical challenges may hinder the use of big data in evaluations.

This report calls for standards to be set for the reporting of data quality issues, data representativeness and data transparency, and an Institutional Review Board (IRB) review specifically designed for ethical issues related to big data.

More interaction is needed between big data analysts, remote sensing scientists and evaluators.

1.6. How up‐to‐date is this EGM?

The authors searched for studies published up to December 2019.

2. BACKGROUND

2.1. Introduction

2.1.1. The problem, condition or issue

Policymakers need access to reliable data to evaluate development outcomes and decide on future resource allocation. Governments, multilateral organisations and other development players in low and middle income countries (LMICs) use censuses, nationally representative household surveys, other household surveys and administrative data to evaluate development programmes and policies. With the increasing complexity of development programmes, there is a need to collect a vast array of output, outcomes and contextual variables to robustly assess impact. However, significant data collection challenges remain. Data challenges for IEs include limitations on sample size and power due to budget constraints, inaccessible or difficult‐to‐reach sections of target populations, measurement errors due to recall bias, inadequate frequency and level of aggregation, inadequate information on controls and covariates, data collection lag times and difficulties in measuring long‐term impact‡ (Wassenich, 2007). Further, in some contexts, like conflicts and humanitarian emergency situations, data collection is often impossible. The data gaps and challenges are particularly significant for the populations and countries where the need for evidence‐informed policy decisions are perhaps the greatest (Gaarder & Annan, 2013). Another key shortcoming of survey data are inadequate aggregation at subnational administrative units such as districts, counties or villages, inhibiting evaluation of programmes with spatial attributes.

2.1.2. Data sources

Big data offers great potential for answering some of these data needs. More importantly, it answers the causal questions around which policies or interventions work, including in contexts where traditional methods of data collection are challenging. The UN Global Pulse (2013) defines big data as being digitally generated (as opposed to digitised manually), passively produced (a by‐product of digital services, transactions and interactions), automatically collected and geographically and temporally trackable. Although there is no formal definition for big data, currently the term is characterised by the three Vs: high volume, velocity and variety. Satellite images, sensors and drones, mobile phone CRDs, commercial transactions data, online searches, social media, citizen reporting or crowdsourced data are the sources of big data.

Integrating big data with traditional household surveys and administrative data can complement data availability, quality, granularity, accuracy and frequency, as well as help measure development outcomes temporally and spatially in a number of new ways (BenYishay et al., 2018; Lokanathan et al., 2017; Salganik, 2017; UN Global Pulse, 2016; York & Bamberger, 2020). For example, satellite images and mobile CRD have been used in mapping poverty (Blumenstock et al., 2015; Jean et al., 2016), disaster response (Lu et al., 2012; Wilson et al., 2016) and food security (Decuyper et al., 2014). Web searches and social media were used in predicting unemployment and crime instances (Gerber, 2014; Xu et al., 2013).

While big data is increasingly used for tracking indicators and monitoring development progress on SDGs (Lokanathan et al., 2017; UN Global Pulse, 2012; Vaitla, 2014), available data are less often utilised to address causal questions about the effects of specific policies and programmes. Big data can contribute to answering some of the causal questions around which interventions work. Big data prediction models can generate proxy estimates for key development outcomes such as wealth, human development, infrastructure quality, forest cover and more, which can be used in experimental (Jayachandran et al., 2016; Pellegrini, 2019) and quasi‐experimental studies (BenYishay et al., 2018; Jaiswal et al., 2020). Satellite images such as night light, crop intensity, water availability, land use, proximity to services and physical attributes such as elevation or slope can be used in IEs as a direct measure of outcomes or as covariates. Furthermore, big data can be used for measuring and evaluating the long‐term impacts of policies and programmes, conducting ex‐post evaluations and estimating spatial heterogeneity. For example, satellite data are available at least as far back as 1993 for all places (high‐resolution pictures are available for the entire globe at a granularity as low as 1 × 1 metre), allowing measurement of long‐term impacts. This can help fill the gaps in evidence that cannot be addressed by traditional data sources.

2.1.3. Why it is important to develop the systematic map

The potential of big data to answer causal attribution, however, is still not widely understood or used, especially in LMICs (York & Bamberger, 2020). In this context, a systematic collection of various sources of big data and ways of measuring and evaluating development outcomes will be a great value addition to the development community's contribution to evidence‐informed policymaking.

In this paper we look at IEs, SRs and measurement studies§ that use big data to evaluate development outcomes with a special focus on fragile contexts. The study highlights the new sources of data; how these new data sources can be used for measuring development outcomes innovatively; and how these new measures can be used in IEs. We map different sources of big data onto development outcomes based on SDGs to identify the current evidence base and its gaps.

3. OBJECTIVES

The overarching aim of this report is to inform policymakers and evaluators of existing evaluations based on big data and to provide a database of big data‐based IEs and studies that could inform future IEs. Specifically, the objectives of the research are to:

• Identify rigorous IEs, SRs and the studies that have innovatively used big data to measure any development outcomes, with special reference to fragile contexts;

• Summarise current understanding of potential uses, pros and cons, reliability, biases, risks and ethical issues in using big data for measurement and evaluation of development outcomes and

• Generate interest and awareness among key stakeholders (evaluators, researchers, donors, practitioners, implementers and policymakers) of the potential as well as challenges of using big data.

This systematic map addresses the following questions:

• How have different types of big data and methods been used for measuring and evaluating development outcomes?

• How dispersed or concentrated is the use of big data across development goals and geographies?

• What are the potential biases, measurement reliability issues, pros and cons, risks and ethical issues in using big data for measuring and evaluating development outcomes?

• What are some of the unexplored but promising applications of big data for IEs?

4. METHODS

4.1. Evidence and gap map: Definition and purpose

We follow 3ie's methodology and process for evidence gap maps (Snilstveit et al., 2017). To create this map, we used systematic methods to identify any completed and ongoing IEs, SRs and big data measurement studies relevant to our research objectives. We conducted systematic searches and data extraction as described in Appendices C and D. The studies identified are mapped on to the framework of big data sources and SDG outcomes to provide a visual display of the volume of and the trends in the evidence base. We also coded how the included studies have dealt with ethical and transparency related challenges. The systematic map is available through an online interactive platform on the 3ie website and allows users to explore the available evidence through different filtering options.** There are links to study summaries in the 3ie repositories (wherever applicable) and confidence ratings for the SRs.

4.2. Framework development and scope

For the purpose of this research, we define big data sources as digitally generated, passively produced and automatically collected data, as defined in UN Global Pulse (2013). The sources of big data include satellite images, sensors and drones, mobile phone CRDs, commercial transactions data, online searches, social media, citizen reporting or crowdsourced data. See Table 2 for more details on various sources of big data adapted from UN Global Pulse (2012, 2013), Yeung & Fok (2014) and Blazquez and Domenech (2018).

Table 2.

Sources of big data

Data type	Source of data
Human‐sourced information
Social networks	Text, metadata and location data from social networking sites, opinion platforms, blogs, pictures, videos, and so forth, such as Twitter, Facebook, LinkedIn, YouTube, Wiki pages
Internet searches	Internet text and internet search queries, ie Google Trends; web logs (open data)
Mobile data content	Text messages
Citizen reporting or crowdsourced data	OpenStreetMap, Humanitarian Data Exchange platform, and so forth, Collected from a large group of people who voluntarily provide information. This is a useful source for recording events and receiving people's opinion and feedback on issues. Collected through hotlines, user‐generated maps, marking of instances, events, and so forth
Process‐mediated data
Data produced by public agencies	Medical records; postal data; tax data, and so forth
Data produced by businesses	E‐commerce transaction records and credit card transaction records, ATM transactions, mobile money transfers, savings and loan repayments (collected by the service provider as a part of regular business operation and monitoring; proprietary and commercially sensitive data)
	Tolls and public transport card use data
Mobile phone CRD	Mobile CRDs that provide metadata on when the call took place, the cost, the time and the recipient of the call; location of the caller and of the recipient; and the users' mobility, social interaction and airtime transaction details; top‐up data
Machine‐generated data
Data from fixed sensors	Home automation, weather/pollution sensors, traffic sensors/webcams, security/surveillance videos/images and activity records such as electricity metres (mostly administrative data collected by the authorities as a part of regular monitoring)
Data from mobile sensors (tracking)	Community or privately owned drones (common property or privately held data)
	Mobile phone GPS (open data available from Google)
	Vehicle GPS and self‐tracking apps (proprietary data)
Data from satellites	Open data available from a number of sources, for example:
	Defense Meteorological Satellite Program‐Operational Linescan System Visible Infraref Imaging Radiometers Landsat European Space Agency Land Cover, and so forth

Abbreviation: GPS, global positioning system.

Source: Adapted from UN Global Pulse (2012, 2013), Yeung & Fok (2014), and Blazquez and Domenech (2018).

3ie evidence gap maps compile IEs and SRs. However, in this study, we include IEs and SRs as well as measurement studies: the studies that have innovatively used big data to measure and validate any development outcome. These are multidisciplinary studies that use state‐of‐the‐art methods from computer science and statistics to collect, clean and analyse big data for measuring development outcomes. For example, Jean et al. (2016) use transfer learning techniques as well as daytime and night light data from satellite images to estimate consumption expenditure at the cluster (village) level to map poverty in five African countries: Nigeria, Tanzania, Uganda, Malawi and Rwanda. While a number of such studies have used big data for measuring various development outcomes, few IEs have used these innovative big data‐based outcome measures. These measurement studies, we hope, would serve as proofs of concept for innovative use of ML and big data that can be used in future evaluations.

3ie defines an IE as a “study of the attribution of changes in the outcome to the intervention.”†† For the purpose of this systematic map, we define big data‐based IEs as any experimental or quasi‐experimental studies that use any form of big data to measure the outcomes of interest and/or the confounding variables.

We use the Organisation for Economic Co‐operation and Development (OECD) definition of fragile contexts, which includes conflicts, institutional fragility, social fragility, environmental risks, health risks and climatic risks. This list is more inclusive than the list used by the UK Department for International Development (DFID) and the World Bank, which includes conflict and institutional and social fragility (DFID, 2016). See Appendix A for more details on the classifications and country list. We use the OECD definition for classifying fragile contexts based on:

• Difficult terrain

• Natural disasters

• Conflict or humanitarian crisis

• Chemical or radio‐nuclear issues

• Disease outbreaks or epidemics.

Using big data in evaluation poses a number of analytical challenges on issues including data quality, transparency, generalisability, and privacy and ethical challenges such as consent for using data and anonymisation of the data. This report also explores how the included studies dealt with these challenges.

4.2.1. Types of study design

Table 1 summarises the criteria we used for searching, screening and including the studies for the map.

Table 1.

Selection criteria for studies

Category	Description
Population	This map includes all population from all countries but we provide breakdowns for rural areas, urban areas, conflicted‐affected persons and ethnic minorities. We also provide breakdowns for LMICs and fragile contexts separately
Sources of big data	Big data may originate from any of the following sources:
	Human‐sourced information: a. Social networks b. Internet searches c. Mobile data content d. Citizen reporting or crowdsourced data Process‐mediated data (traditional business systems and websites): a. Data produced by public agencies b. Data produced by businesses c. Mobile phone CRD Machine‐generated data (automated systems): a. Data from fixed sensors b. Data from mobile sensors (tracking) c. Data from satellites
Outcomes	1. Economic development and livelihoods 2. Agriculture and food security 3. Health and well‐being 4. Quality of education 5. Governance and human rights 6. Water and sanitation 7. Energy, industry and infrastructure provision 8. Urban development 9. Environmental sustainability 10. Partnerships for goals
Study design	IEs:
	Randomised controlled trial Regression discontinuity design Controlled before‐and‐after study using appropriate methods to control for selection bias and confounding, such as propensity score matching or other matching methods Instrumental variable estimation or other methods using an instrumental variable such as the Heckman two‐step approach Difference‐in‐differences A fixed‐effects or random‐effects model with an interaction term between time and intervention for baseline and follow‐up observations Natural experiments Other quasi‐experimental studies inducing synthetic control studies Survey, laboratory or lab‐in‐the‐field type experiments Cross‐sectional or panel studies with an intervention and comparison group using methods to control for selection bias and confounding as described above
	Measurement studies
	We included the studies that innovatively used big data to measure and validate any development outcomes. These studies use big data to measure components that would have been difficult to measure using survey data.
	SRs
	We include only the reviews that specifically looked at studies that used big data to measure development outcomes and explicitly described the search, data collection and synthesis methods according to a standard SR protocol, such as the 3ie SR protocol.

Abbreviations: CRD, call record detail; IE, impact evaluation; SR, systematic review.

4.3. Search methods and sources

Innovations in the type of devices available for measurement (satellites and sensors); daily personal use (mobiles, wearables, Internet of Things, etc.); social interaction (blogs, Facebook, Twitter, WhatsApp, etc.); and recording business transactions digitally (CRD, e‐transactions, mobile money, credit card payment, etc.) have led to an explosion of automatically collected data. However, there is no official definition of big data. McKinsey defined it broadly as data “whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze” (Manyika et al., 2011).

UN Global Pulse (2013) defines big data for the purposes of development as being digitally generated (as opposed to digitised manually), passively produced (a by‐product of digital services, transactions and interactions), automatically collected and geographically or temporally trackable. While the size, velocity and veracity are all defining characteristics of big data, the definition relevant for IE is that these are nonsampled data, passively left behind by humans using digital devices and services or automatically collected by the services providers for the purpose other than statistical inference (Letouzé, 2016; UN Global Pulse, 2016). Hence, unlike the conventional survey data where the respondents say what they do or feel, big data captures what people actually do. The implication of this is that big data is nonreactive: in other words, there is less likelihood of social desirability bias (Salganik, 2017). The other key characteristic of big data that matters for evaluation is that it is near real‐time and can be available across multiple frequencies (e.g., hourly, daily) over a long period. Table 2 provides the types of big data, subclassifications, definitions and sources.

Using these definitions, we include the following broad classification of big data‡‡:

• Human‐sourced information from social networks that is provided voluntarily by users;

• Process‐mediated data from traditional business systems and websites that includes digitally recorded business activities;

• Machine‐generated data from automated systems includes information from sensors and machines that measure and record events and situations in the physical world.

4.4. Analysis and presentation

4.4.1. Report structure

We use the SDGs as the basis for identifying the outcome categories, similar to Phillips et al. (2017).§§ We have regrouped them into smaller thematic groups based on complementarities between the sectors. Table 3 provides the definition of each of the outcomes as defined in UN (2019). Appendix B provide more details on the submaps where further breakdowns have been provided for key subclassifications where relevant.

Table 3.

Outcome categories and definitions

Category	Definition
Economic development and livelihoods (SDGs 1 and 8)	End poverty in all its forms everywhere
	Promote sustained, inclusive and sustainable economic growth, full and productive employment and decent work for all
Agriculture and food security (SDG 2)	End hunger, achieve food security and improved nutrition and promote sustainable agriculture
Health and well‐being (SDG 3)	Ensure healthy lives and promote well‐being for all ages
Quality of education (SDG 4)	Ensure inclusive and equitable quality education and promote life‐long learning opportunities for all
Governance and human rights (SDGs 5, 10 and 16)	Achieve gender equality and empower all women and girls
	Reduce inequality within and among countries
	Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels
Water and sanitation (SDG 6)	Ensure availability and sustainable management of water and sanitation for all
Energy, industry and infrastructure provision (SDGs 7 and 9)	Ensure access to affordable reliable, sustainable, and modern energy for all
	Build resilient infrastructure promote inclusive and sustainable industrialisation and foster innovation
Urban development (SDG 11)	Make cities and human settlements inclusive, safe, resilient and sustainable
Environmental sustainability (SDGs 12, 13, 14 and 15)	Ensure sustainable consumption and production patterns
	Take urgent action to combat climate change and its impacts
	Conserve and sustainable use the oceans, seas and marine resources for sustainable development
	Protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, and halt and reverse land degradation and halt biodiversity loss
Partnerships for goals (SDG 17)	Strengthen the means of implementation and revitalise the global partnership for sustainable development

Abbreviation: SDG, sustainable development goal.

Source: Adopted from UNDP SDG Official list (2016)

4.4.2. Filters for presentation

We have added filters such as population, challenging contexts, geographic coverage to provide more details about the data sources and outcomes of interest. The population filter provides information on whether the studies in the map identify subgroups such as rural, urban, refugees, conflict affected persons, ethnic minorities. With the challenging contexts filter we show if the studies were in areas which were remote, conflict affected, affected by disease outbreak, and so on. We also identified different regions where the studies were conducted.

4.4.3. Dependency

Each unit of analysis was a report, peer reviewed journal publication, working paper. We included one study per unit of analysis. In cases where we had multiple reports we included the latest one with preference given to peer review versions and/or based on ease of access to a particular version. When multiple studies were covered by a single report, we included the report and then mapped the data sources and outcomes accordingly.

4.5. Data collection and analysis

4.5.1. Screening and study selection

We first employed Evidence for Policy and Practice Information (EPPI)‐Reviewer's built‐in text mining, an ML technique, to sort the studies based on the inclusion and exclusion criteria at the title and screening stage (see Appendix D for more details on how text mining was used for screening). This reduced the number of studies to be screened at the title and abstract level to 9720. At the second stage, three researchers screened the studies for eligibility based on inclusion and exclusion criteria defined in the study protocol. At the end of stage two, we had identified 1348 studies to be screened at the full‐text level.

At stage three, four researchers coded the studies at the full‐text level. We followed a single screening with safety first approach,*** where only one reviewer screened each record and coded doubtful studies as “to discuss” to avoid excluding the studies that are potential includes. About 25% of the studies at this stage were double‐coded for consistency and any disagreements were resolved with the lead author. About 20% of the studies were spot‐checked by the lead author. At the full‐text screening stage, we excluded 1292 studies. We excluded 251 studies based on irrelevant data sources, 408 studies on outcomes and 633 studies on study designs. Studies excluded on design were primarily policy papers or computer science technical research papers that aimed at developing new algorithms rather than measuring development outcomes.

4.5.2. Data extraction and management

The three‐stage process of screening resulted in a final list of 437 studies including 48 IEs, 381 measurement studies and 8 SRs. Through a consultation process, we identified the metadata to be extracted using a standardised data extraction tool and defined them in the study protocol. During the final studies coding, we collected metadata such as outcomes studies, outcome subcategories, data sources used, geographical location of the intervention, country, evaluation design, target population, data transparency, ethics and other bibliographic information from the included studies (see Appendix D for the data extraction tool). We also critically appraised the SRs (see Appendix E for the SR appraisal tool and summary of the included SRs). Due to the size and the nature of included studies we did not conduct critical appraisal of IEs or measurement studies.

4.5.3. Tools for assessing risk of bias/study quality of included reviews

All SRs have been critically appraised using 3ie's critical appraisal checklist, adopted from the SURE checklist. Appendix F summarises all the SRs included in our sample and Table G1 provides details on the critical appraisal of the included studies.

4.5.4. Methods for mapping

The studies were mapped using 3ie's evidence gap map platform, which is organised into rows and columns. Various big data sources are placed in the rows and the development themes are placed in the columns. Any intersecting cell represents the development outcome measured or evaluated using the particular type of big data. Different colour bubbles represent the type of study: grey bubbles denote IEs, blue bubbles denote measurement studies, green bubbles denote high‐quality SRs and red bubbles denote low‐quality SRs. Hovering over the bubbles will show the links to studies. There are also filters for different regions, countries, study design, fragile context and the target population of the studies.

Development themes (such as environmental sustainability, economic development and livelihoods) contain a large number of studies. We have provided maps within the main map to show how the studies are distributed across the subthemes under these broad themes. In the submaps, the big data sources are mapped against level 2 or level 1 subclassification provided in the SDGs as relevant. For example, the submap for economic development and livelihoods has been coded against the level 2 indicators of eradicating poverty (SDG 1) and employment and economic growth (SDG 8), and the submap for environmental sustainability is coded against the level 1 classification of SDGs 12, 13, 14 and 15. The following development themes have submaps:

• Economic development and livelihoods

• Health and well‐being

• Governance and human rights

• Urban development

• Environmental sustainability.

5. RESULTS

5.1. Description of studies

5.1.1. Results of the search

We systematically searched academic databases with the help of an information specialist (see Appendix C for the list of academic databases and search strings) and manually searched specialist organisational websites and grey literature sources. The initial searches resulted in 17,393 studies, of which 17,008 studies were identified through bibliographic databases search and 385 studies were identified through hand‐searching specialist databases and IE and grey literature repositories (Figure 1). All the results were uploaded onto EPPI‐Reviewer 4 for screening and coding. Screening of the studies was done in three stages.

Figure 1.

The PRISMA flowchart. Source: Author's own calculation

5.1.2. Excluded studies

This report includes papers written in English and published between 2005 and 2019. This map does not include studies that develop algorithms or methodologies for using big data without explicit application to measuring or evaluating a development outcome. This map does not include studies that describe how big data and ML have been used in development programming to help programme implementation, coordination and management for designing and scaling new development solutions; neither does it include studies that show how big data methods are used in randomised controlled trials (RCTs) to identify the differential impact of subgroups and in improving survey data collection, such as defining sampling frames.

5.1.3. Studies awaiting classification (if applicable)

If applicable, list the characteristics of any studies that have been identified as potentially eligible but have not been incorporated into the map (ER40).

5.2. Synthesis of included studies

The map contains 437 studies, of which 48 are IEs, 381 are measurement studies and 8 are SRs. Of the 48 IEs, 8 are RCTs and the remaining are quasi‐experimental studies.

Figure 2 displays the number of studies published each year from 2005 to 2020. The light blue bar shows the measurement studies, the dark blue bar shows IEs and the line indicates the cumulative number of measurement studies, IEs and SRs. The number of measurement studies has grown gradually from 2005 to 2012, increasing substantially every year since then with maximum numbers in 2017 and 2018. The past five years alone have accounted for more than 60% of the studies, indicating the increasing availability of big data, improved computational capacity and greater interest among researchers and journals.

Figure 2.

Number of studies published per year. Source: Authors' own calculation

The figure also shows that applying big data to IEs is a new phenomenon. The first IE using big data was published in 2009. While almost all the IEs were published after 2013, more than three‐quarters of the IEs were published in the last five years. We expect that the measurement studies will be proofs of concept, leading IEs to adopt to these approaches to innovatively measure development outcomes in evaluations. The map also points to the gap between the growth of measurement studies and use of big data in IEs.

Figure 3 shows the geographical distribution of the included studies. About 50% of the studies (n = 210) are from Asia and close to 30% (n = 132) are from Sub‐Saharan Africa. The distribution of IEs and measurement studies are roughly similar to the overall distribution. One notable exception is Latin America and the Caribbean where the region accounts for 15% of total studies (n = 65), but substantially more IEs (38%, n = 18).

Figure 3.

Distribution of studies over regions. Source: Authors' own calculation

China (43) and India (34) are the most‐studied countries, and Kenya has highest number of studies (15) in Sub‐Saharan Africa (Figure 4). While East Africa is very well‐represented on the map, other African countries have fewer entries and several countries in West and North Africa have no studies at all. About 35 of the studies are multi‐country studies and 11 studies did not specify the country name primarily to conceal the identity of the data provider. The distribution of IEs and measurement studies are again roughly similar to the overall distribution but Latin America countries, particularly Mexico (5), account for a higher proportion of IEs. See Table H1 for a list of top 20 countries with the maximum number of studies and Table H2 for the geographical distribution of studies across the regions in Appendix G.

Figure 4.

Geographical distribution of studies. Source: Authors' own calculation

We find that most of the studies are concentrated in middle income countries. There are 232 studies (53%) in the middle income group, followed by 103 studies (24%) in the high income group and 71 studies (16%) in the low income group (Figure 5). Overall, about 69% (n = 303) of the total studies are from LMICs, but the IEs are distributed more in favour the LMICs as 83% of them (n = 40) are from LMICs. One of the notable features of the studies on the map is that about 82% (n = 359) of the total studies are published in peer‐reviewed journals and the remainder are working papers (18%, n = 78).

Figure 5.

Number of studies by income classification. Source: Authors' own calculation

5.2.1. Distribution of studies across data sources

As discussed in Section 2.1.2, big data can be generated by human interaction on social media, process‐mediated data recorded by governments and the business and machine‐generated data that is recorded by the automated systems. Figure 6 shows that machine‐generated data are used the most. Of the total number of studies, close to 84% (n = 380) of the studies used some form of machine‐generated data, while 12% (n = 53) of the studies used human‐generated sources and 17% of the studies (n = 77) used process‐mediated data.

Figure 6.

Number of studies per different type of big data. Source: Authors' own calculation

Table 4 below provides a detailed breakdown of the number of studies per data source.

Table 4.

Number of IEs and measurement studies across data sources

Data source	IEs	Measurement studies	SRs
Human‐sourced
Social networks	0	18	3
Internet searches	0	11	1
Mobile data content	2	10	0
Citizen reporting or crowdsourced data	2	10	0
Process‐mediated
Data produced by public agencies	1	2	0
Data produced by businesses	2	7	0
Mobile phone CRD	0	65	0
Machine‐generated data
Data from fixed sensors	7	32	0
Data from mobile sensors (tracking)	0	15	0
Data from satellites	39	270	5
Total studies	48	381

Note: Percentage of subcategory total in parentheses. Columns do not add up due to multiple entries.

Data from satellites and fixed sensors

Satellite data are the most used source of big data as it accounts for 71% of the measurement studies (n = 210) and close to 81% of the IEs (n = 39). Data from fixed sensors (such as weather and pollution sensors, traffic sensors and electricity metres that provide high‐frequency, localised measurements) could also be readily used in IEs. This is the second most used data source, with 15% of the IEs using these sources. This shows that the data from satellites and in situ sensors that help measure spatial outcomes are used most in IEs. Other big data sources have been seldom used for IEs despite measurement studies showing proof‐of‐concept.

Mobile phone CRD

A good number of measurement studies have used CRDs (17% n = 65) for measuring population movement, migration, disease spread and even to understand the literacy level of the subscribers. Surprisingly, we found no IEs that used this source of big data despite the availability of a good number of proof‐of‐concept papers in measuring key development outcomes.

Human‐sourced data

Social networks including Facebook, Twitter and Wiki pages were used to measure development outcomes in 18 measurement studies. Internet searches like Google trends and other search engine queries were used in 11 studies. Mobile data content and crowdsourced data were used in 10 studies, primarily to measure disease outbreak, price data or opinion on issues like development services. This source has not been used in IEs, with two notable exceptions using crowdsourced data (Edjekumhene et al., 2019; Van der Windt & Humphreys, 2016).

Complementarity between data sources

There are about 57 studies on the map that have combined at least two sources of data and about seven of them have combined three or more sources (Table H3 in Appendix G). Data from fixed sensors and satellites data seem to complement each other well: 13 measurement studies and 2 IEs have combined these two sources. Mobile phone CRD and satellite data are the other combination that has been used repeatedly. About 10 of the measurement studies have combined CRD data and satellite data in their analysis (see Section 5 for a discussion and example on how satellite data and CRD can be combined together for better results). However, IEs seem not to have exploited this complementarity. See Table H3 in Appendix G for a list of studies using multiple sources of big data.

5.2.2. Distribution of studies across development themes

Section 4.2 identifies 10 broad development themes based on SDGs. Figure 7 and Table 5 show the number of studies across the development themes. About 50% of studies (n = 217) focus on environmental sustainability, which includes sustainable consumption and production, climate change, underwater life, and life on land. Economic development and livelihoods accounts for about 26% of the total studies (n = 114). Urban development and health account for 16% each (n = 68). Governance and human rights (7%, n = 30) and energy, industry and infrastructure (7%, n = 30) account for the remaining studies.

Figure 7.

Number of studies against development outcomes. Source: Authors' own calculation

Table 5.

Distribution of studies across development themes

Development themes	IEs	Measurement studies	SRs	Total
Economic development and livelihood	8	105	1	114
Sustainable agriculture and food security	2	21	0	23
Health and well‐being	5	58	5	68
Education	0	4	0	0
Water and sanitation	0	28	0	28
Governance and human rights	3	26	1	30
Energy, industry and infrastructure	8	22	0	30
Urban development	3	69	3	75
Environment sustainability	25	190	2	217
Global partnership	0	2	0	2

While the distribution of IEs and measurement studies across the development themes remains the same as the overall distribution, there are a substantial number of IEs on economic development and livelihoods (17%) and energy, industry and infrastructure (17%). While most of the SRs looked at cross‐sectoral themes, health is the most‐studied sector (n = 5), followed by urban development (n = 3) and environment sustainability and economic development (n = 2). See Appendix F for critical appraisal of the SRs.

5.2.3. Units of observation

The unit of observation (or unit of analysis) is the class of elemental unit that constitutes the population and the units of measurement. Typically, in IEs, the units of observation are individuals, households, facilities (in facility surveys) or various level of administrative units such as villages, counties or districts. We have classified the unit of observation as population (including both individuals and households) or administrative units (villages, land parcels or any other units with a spatial element). The unit of observation seems to be an important element in analysing the use of big data in measuring development outcomes. Figure 8 (Panel 1) shows that about 70% of the measurement studies (n = 267) and 65% of the IEs (n = 31) have administrative units as their unit of observation. There is a clear distinction between different sources of big data, as shown in Figure 8, Panel 2. The unit of analysis for satellite data‐based studies is predominantly administrative units (n = 259, 83%), while CRD‐based studies are usually based on population units (n = 53, 82%). This difference shows that satellite data are more applicable when the outcome of interest has some spatial dimension such as local economic development, agricultural land productivity, forest cover or urban development.

Figure 8.

Units of observation. Source: Authors' own calculation

5.2.4. Studies using mixed methods

Mixed‐methods IEs that combine qualitative and quantitative analyses help assess the quality implementation and reliability of data and understand the mechanism of programme impact (Bamberger, 2012). Big data IEs can be combined with qualitative methods. However, only three IEs and five measurement studies reported using mixed methods.

5.2.5. Studies with a rural or urban focus

Figure 9 shows the proportion of studies focused on rural areas or urban areas, or both. Most studies looked at both rural and urban areas (74%, n = 325). About 9% of the studies (n = 41) focused on rural areas; 12% (n = 51) focused on urban areas. Among the remaining studies, 14 studies looked at conflict affected population, 4 were studies of ethnic minorities and 2 studied refugees.

Figure 9.

Distribution of population subgroups. Source: Authors' own calculation

5.2.6. Studies by fragile context

We used the OECD definition of fragile context that includes conflict, institutional, social fragility, environmental, health and climatic risks (OECD, 2018). Figure 10 shows that 91 studies included on the map (21%) are from countries considered to be fragile. About 39 studies were conducted in a conflict or humanitarian crisis context; 22 studies each were conducted in contexts of difficult terrain and natural disasters; and 15 studies were conducted in the context of epidemics or disease outbreaks. There was one measurement study in the context of a chemical/radio‐nuclear disaster. IEs follow the same pattern, except for one notable gap: there are no IEs in the context of epidemics or disease outbreaks despite a reasonably good number of measurement studies.

Figure 10.

Number of studies in fragile contexts. Source: Authors' own calculation

Table H4 in Appendix G shows that satellite data are the most used in fragile contexts, followed by CRD data and then the sensor data. The table also shows that almost all the big data sources have been used in one or two fragile contexts, indicating the importance of big data in fragile contexts.

5.3. Risk of bias in included reviews

Figure 11 shows that very few studies meet any of the following methodological quality markers.

• Is the construct validity explained (ie is there a discussion on how the big data‐based indicator measures what the study claims to measure)?

• Are data and codes publicly available for replication?

• Are data collection methods discussed?

• Are there data quality issues in the dataset used and how are they addressed?

• Is the data representative of the population of interest?

• Are challenges in the analysis and reporting process discussed?

• Are the results generalisable? For example, are the research findings generalisable to other situations such as other platforms (data sources) or communities, or over time?

Figure 11.

Number of IEs and MS against data quality and transparency. IE, impact evaluation. Source: Authors' own calculation

Only 95 studies (22%) have reported on at least one of the above transparency criteria. For example, only 20% (n = 91) of the total studies reported on data collection methods, 6% (n = 25) on data quality issues, 8% (n = 36) on data representativeness, 14% (n = 64) on construct validity and 7% (n = 30) on generalisability. Only 4% (n = 19) of the studies have data and codes publicly available or available upon request.

There is, however, considerable difference between IEs and measurement studies in terms of reporting on data quality issues and transparency. Table H4 in Appendix G shows that IEs report a lot better on all these parameters. Of the total 48 IEs on the map, 46 of them report at least one aspect of transparency. Almost all the IEs report on data collection methods, 90% (n = 43) report on construct validity, 60% (n = 29) discuss representativeness of data, 54% (n = 26) discuss generalisability and 45% (n = 22) discuss various data quality issues. However, only 23% (n = 11) make data and codes available and 13% (n = 6) discuss key data analysis and reporting challenges. Figure 11 shows that very few studies meet any of the following methodological quality markers.

• Is the construct validity explained (ie is there a discussion on how the big data‐based indicator measures what the study claims to measure)?

• Are data and codes publicly available for replication?

• Are data collection methods discussed?

• Are there data quality issues in the dataset used and how are they addressed?

• Is the data representative of the population of interest?

• Are challenges in the analysis and reporting process discussed?

• Are the results generalisable? For example, are the research findings generalisable to other situations such as other platforms (data sources) or communities, or over time?

Only 95 studies (22%) have reported on at least one of the above transparency criteria. For example, only 20% (n = 91) of the total studies reported on data collection methods, 6% (n = 25) on data quality issues, 8% (n = 36) on data representativeness, 14% (n = 64) on construct validity and 7% (n = 30) on generalisability. Only 4% (n = 19) of the studies have data and codes publicly available or available upon request.

There is, however, considerable difference between IEs and measurement studies in terms of reporting on data quality issues and transparency. Table H4 in Appendix G shows that IEs report a lot better on all these parameters. Of the total 48 IEs on the map, 46 of them report at least one aspect of transparency. Almost all the IEs report on data collection methods, 90% (n = 43) report on construct validity, 60% (n = 29) discuss representativeness of data, 54% (n = 26) discuss generalisability and 45% (n = 22) discuss various data quality issues. However, only 23% (n = 11) make data and codes available and 13% (n = 6) discuss key data analysis and reporting challenges.

6. DISCUSSION

6.1. Summary of main results

The use of big data in measuring development outcomes has been on the rise over the past 5 years. This rising trend is powered by the availability of (and our capacity to process) big data. In this section, we discuss the key findings, some of the notable gaps and the potential for future SRs.

6.1.1. There is a considerable potential for measuring various development indicators using big data

We identify a significant and growing evidence base of measurement studies that use some form of big data to measure a development outcome. Some outcomes are more amenable to the use of big data than others; environmental sustainability, economic development and livelihoods, health and well‐being and urban development are where the majority of studies are concentrated. Education, sanitation, governance and human rights seem to be less responsive to big data use.

Multiple entries for most development theme indicate the potential of big data in contributing to measuring development indicators. Identifying measurement studies will be a valuable addition to development evaluators who look for innovative ways to measure a development outcome that was difficult to measure at all required spatial and temporal scales using conventional data collection methods.

6.1.2. There is potential for more IEs using big data on development interventions

The map contains 48 IEs. Use of big data measures in IEs as main outcomes or for controlling key covariates is fast‐growing, but the IEs are fewer in number compared to measurement studies as well as in terms of the extent of their thematic and geographical coverage. IEs seem to be concentrated more around environmental sustainability, economic development and urban development. This complements existing efforts to build the evidence base in international development, as these sectors have much less rigorous evaluations (Sabet & Brown, 2018). The IEs also concentrate on using satellite data.

6.1.3. Satellite data are used most

The map shows that 71% of the measurement studies and 81% of the IEs used satellite data. This is also one of the sources that has been used since the early 2000s. The prominence of satellite data are primarily due to the fact that satellite images offer unique possibilities for measuring and evaluating development outcomes. Given the vast number of satellites covering almost every location on earth, it is possible to collect data at a high granularity (spatial resolution) and for multiple temporal frequencies for the past 30 years. Satellite data are freely available from several sources (such as NASA's Landsat and MODIS and the European Space Agency's Sentinel); more importantly, several preprocessed databases are available (such as AidData's Geoquery,††† Yale University's G‐Econ Project, the United Nations Environment Programme's Environmental Data Explorer,‡‡‡ NASA's Socioeconomic Data and Applications Center§§§ [SEDAC], Global Forest Change 2000–2018 [Hansen et al., 2013], and several others). This preprocessed data or the image data could then be processed and converted into meaningful outcomes to measure economic activity at local level, urban development, forest cover, land productivity, distribution of the population, and so forth. These indicators can also be used for controlling for covariates.

6.1.4. Spatial dimension matters

One of the key findings of the map is that most of the big data studies are applied in the context where the phenomenon studied has a spatial dimension, meaning the outcome and other covariates are measured on a spatial scale. Close to 70% of the studies on the map report using administrative units as their unit of measurement. This is particularly true for satellite and sensor data‐based studies, as 82% have administrative units as their unit of measurement (such as local economic development, agricultural land productivity, forest cover or urban development). This is referred to as geospatial IE (BenYishay et al., 2017). However, there is considerable difference across data sources as CRD data are used to measure changes at the population level.

6.1.5. CRD data has great potential for measuring and evaluating development outcomes but is not yet used in IEs

CRD data are one of the most widely used sources in measurement studies. This is used for measuring population movement, migration, disease spread, and so forth. Despite a number of high‐profile measurement studies, our systematic search did not find even one IE that used CRD data for rigorously evaluating a development outcome. This is a notable gap and a potential area for future exploration. It should be noted that CRD data are also fraught with multiple methodological challenges (such as nonrepresentativeness, lack of completeness, etc.) and ethical challenges (such as consent, unintended exclusion, etc.). Further, CRD data has been difficult to obtain as it is proprietary and hence it is difficult to maintain data transparency.

6.1.6. Other big data sources such as human‐sourced and process‐mediated data have good proof‐of‐concepts

Human‐sourced data (such as social networks, internet searches, mobile data content citizen reporting or crowdsourced data) and process‐mediated data (such as data produced by public agencies and by businesses) have a good number of measurement studies as proof‐of‐concept for using these sources to measure various development outcomes, but not many IEs use these sources. This also shows the possibility of potentially using these sources in future IEs. Similar caveats on methodological and ethical challenges discussed above in relation to CRD data will apply.

6.1.7. East Africa is well‐represented, but not the rest of Africa

The geographical distribution of measurement studies and IEs show that the studies are evenly spread across the continents. However, Ethiopia, Kenya, Rwanda, Tanzania and Uganda are well‐represented in terms of number of measurement studies and IEs, but the only non‐East African country that seem to have well‐represented on the map is Nigeria. There are very few studies in the rest of Africa. This gap is particularly serious given Africa's data challenges (Serajuddin et al., 2015).

6.1.8. Big data holds great potential for conducting IEs in fragile contexts, including during conflicts, humanitarian crises, epidemics and natural disasters

Conducting rigorous evaluation in fragile contexts (such as natural disasters, disease outbreaks and other crisis contexts) can be costly, risky to the beneficiaries and the evaluators, and in some cases outright infeasible. We identified 73 measurement studies, 17 IEs and one SR in such fragile contexts. Measurement studies are spread evenly across conflict or humanitarian crisis, disease outbreaks or epidemics, natural disasters and difficult‐to‐reach terrain. However, the IEs are concentrated around conflict and difficult terrain. The number of measurement studies indicate the potential for more IEs in fragile contexts.

6.1.9. There are potential sectors and themes where SRs will be useful

Though the number of IEs are fewer, the map highlights a few potentials thematic areas where SRs will help answer key questions on policy and research methods. For example, there is a concentration of IEs using satellite data, referred to as geospatial IEs, but we know little about how satellite data can help evaluate development programmes better, where it can add value, what type of interventions could be better evaluated and the technical challenges involved in using satellite data. An SR of all geospatial IEs that have used satellite data across the sectors will help understand the potential and challenges in using satellite data for IEs.

Similarly, there is a concentration of IEs on environmental sustainability and within that climate action and forest management. Though there a few SRs and evidence gap maps on forest management (Pelletier et al., 2016; Puri et al., 2016), a new review with reference to innovative, new data sources used in rigorously evaluating forest cover and the advantages and challenges thereof may be useful.

6.2. Using big data in IEs: Potentials and challenges

Rigorous IEs require a valid counterfactual. Randomising programme placement ensures preprogramme comparability of the treatment and control groups in most cases and quasi‐experimental studies employ statistical procedures to identify a valid comparison group. In either case, evaluators collect require a vast array of data on the outcomes, covariates and other contextual factors. There is almost always a trade‐off between collecting a complete array of necessary data and cost‐effectiveness, and in a few cases, it may not be feasible to collect some of the covariates and confounders.

Big data, with the help of improved ML techniques and analytical capacity, can now be manipulated to evaluate development outcomes. The potential advantages for big data are, to date, most discernible in contexts where the immediacy, scale and/or reach of data are highly prized and alternative sources of data are absent or inadequate to the task. The ability to “zoom in” on particular zones of interest, and to produce estimates for small areas, is an oft‐cited advantage of many types of big data (e.g., satellite and building footprint data, mobile phone CRD and signalling data and app‐based location data) and one with particular relevance to evaluative contexts and SDG‐related urbanisation, climate change and infrastructure. This holds particular promise for settings where census data renders small area estimation methods unsuitable. Big data has also been shown to be particularly advantageous for the analysis of disaster‐induced displacement and disease outbreaks. In each of these cases, the advantage of big data is that it can support rapid appraisal and introduction or adjustment of policies/interventions on the basis of near real‐time information.

In this section, we highlight a few examples from the map to show the steps involved in collecting, processing and using satellite and CRD data for measuring development outcomes. We draw on recent projects from 3ie and Flowminder to illustrate the processes.

6.2.1. Using satellite data in IEs

In a 3ie funded evaluation conducted by the Institute for Financial Management and Research, Pande and Sudarshan (2019) evaluated the recent environmental clearance (EC) reforms in India. Before the 2006 reform, mines of area over 25 hectares were required to hold a public hearing before approval. The new EC reform required mines of area between 5 and 25 hectares to hold a public hearing as well. This study exploits this historical discontinuity in clearance requirements to evaluate the impact of public hearings on mines' environmental compliance. Apart from rigorously evaluating the EC process in India, this study also provides a proof‐of‐concept for the use of remote sensing data and other publicly available data to monitor mines' environmental compliance. Using satellite data to assess the impact of EC process requires data on the timing of the intervention, the geographical scope of the intervention (ie the individual mines in this case), the outcome of interest (such as air pollution, land cover and water quality for the corresponding intervention) and control areas for the years before and after the intervention.

The following were the key steps involved in the big data IE.

Step 1: the researchers used web scraping techniques to collect information on the mines from their EC application for the years from 2006 to 2016, available online in a database published by Ministry of Environment, Forests and Climate Change. These are all mostly scanned PDF documents. The researchers scrapped for the information on project name and location (the tehsil and village where the mine is located); the dates of key EC stages of submission, review and approval; and mine characteristics such as minerals mined, mine production capacity and size of the mine. They also scrapped the clearing letters available in the same database for cross checking the data. They collected information about all 934 relevant mines and used 134 of them in their regression discontinuity analysis. Finally, the researchers hired ML Infomap, a local company, to geocode all the mines identified.

Step 2: Satellite data on various key environmental outcomes such as air pollution, land cover and water quality were collected from different sources.***

• The researchers used the data provided by Dalhousie University on the fine particulate matter concentration as a proxy for air pollution. This database contains average annual particulate matter concentration for every 1 km cell for the study period;

• They have used the Enhanced Vegetation Index (EVI) data from NASA's MODIS satellite to measure deforestation around the mining areas. EVI is available at a resolution of 250 metres for the entire globe and the researchers calculated annual maximum, median and mean EVI at mine sites. EVI data was used to measure the extend of and the date of beginning of deforestation (ie structural break in the time series) for each mine; and

• Data on water quality from the site monitor nearest to each mines was collected from the Central Pollution Control Board's ENVIS database. They used Biological Oxygen Demand, a measure of organics pollution, as a proxy for water pollution.

Step 3: The researchers then linked the geocoded mine sites to the corresponding cells of environmental outcomes. Of the total 934 mines in their database, they could link 889 of 1 km cells of EVI data and 882 of 250 metre cells with corresponding geocoded mine sites. They could also link 538 site monitors to the mines.

Step 4: The new EC reform required the mines of between 5 and 25 hectares in area to hold a public hearing that had not been considered big enough to hold public hearings during the previous regime (ie only the mines of above 25 hectares in area were required to hold the hearings). This study exploited the discontinuity around the 25 hectare mark and compared the mines marginally above 25 hectares with the ones marginally below 25 hectares. The final sample included 134 mines, of which 68 were treatment mines (<25 hectares) and 66 were control (>25 hectares). Using data before and after the EC applications, they estimated a difference‐in‐difference model.

This study, utilising web scraping to collect data on project characteristics and various sources of satellite data for measuring the outcomes of interest, is an excellent example of innovative data collection methods in a sector where the evidence base is very small (Rathinam et al., 2019b).

6.2.2. Using CRD analytics to inform disaster management

In this section, we briefly outline the process for undertaking CRD analytics to measure, characterise and predict population displacement and returnee/resettlement patterns in post‐disaster settings. While applications of CRD data analytics to date have lacked an evaluative component, their potential in this regard is evident. We draw on a recent project at Flowminder, which revisited three sudden‐onset disaster events to investigate drivers of displacements (individual and contextual) and the feasibility of predicting displacement locations from CRD data and data on disaster intensity and damage, on population density and on the humanitarian response. The three events were the 2010 earthquake in Haiti, the 2015 Gorkha earthquake in Nepal and the 2016 Hurricane Matthew in Haiti.

Flowminder has long‐established partnerships with the major mobile phone network operators in Haiti and Nepal. Historically, data access has been a major barrier to the scale‐up of CRD analytics for humanitarian and development applications. Mobile network operators (MNOs) are justifiably hesitant to authorise third‐party access, given the need to safeguard subscribers' personal data.††††

Prior to analysis, MNO data underwent a long series of cleaning and preprocessing steps as part of quality assurance and to support the generation of standardised metrics. A first stage of analysis was undertaken to structure the data in a usable format and to detect data anomalies. Once data was cleaned, quality assured and converted into an analysable format, a number of preliminary processing steps were undertaken, including:

• Clustering of cell tower locations

• Assessment of each subscriber's phone usage behaviours (number of events, frequency and regularity)

• Determination of a predisaster “home” location.

6.2.3. MNO data: Preprocessing steps

Here are some commonly occurring issues in MNO data. Once identified, corrections and/or accommodations can be performed prior to and/or during the preliminary processing and analysis phases.

• 1. Standard data quality issues applicable to MNO data:

• Item missing data (incomplete data records ie missing fields)

• Invalid entries for fields

• Duplicate records

• Interrupted data series' (e.g., no data for a particular time period)

• Inconsistent values (either in format, or definition) for keys that are used to join multiple datasets together

• Inconsistent entries for the “same” value (e.g., different spellings of the same place name).

• 2. Issues specific to MNO datasets, CRD:

• Inconsistencies or errors in method used for “hashing” (a form of pseudonymisation) subscribers' IDs

• Inconsistent “hashing” of sender and recipient IDs for communication events (e.g., standard SMS or phone calls).

• 3. Issues specific to MNO datasets: Cell location and coverage maps:

• Cell locations occur outside national borders

• Updated cell locations are not consistent with previous cell locations

• Small deviations in updated cell locations, possibly due to inexact global positioning system measurements

• Data, projection and coordinate system information are often missing in coverage datasets

• Inconsistent output formats

• For best server or cell‐in‐isolation maps, polygons should be labelled in a manner consistent with cell table and/or CRD dataset.

• 4. Common data anomalies, indicative of a network issue or a sudden change in subscribers' behaviour due to an event or “shock”:

• Individual cell towers have significantly more/less traffic than normal

• Overall network traffic is significantly higher/lower than expected

• Traffic from a particular region is significantly higher/lower than expected.

The processing steps undertaken to discern at individual level disaster‐induced displacements from pseudonymised, time series CRD data are presented below in Figure 13.

Figure 13.

Steps in CRD processing for displacement and return/resettlement/recovery pattern analysis. CRD, call record detail

The analysis disclosed striking commonalities in internally displaced person (IDP) return/resettlement rates, with the fraction of IDPs who remain displaced exhibiting a common rate of decay across all three post‐disaster settings studied.

In a further step, the team developed new mobility and social network metrics to permit analysis of the relationships between contextual and individual variables and displacement duration, distance and trajectories, controlling for the severity of impacts and humanitarian response. The results suggest that the dispersal of an individual's social contacts and travel history predisaster are highly predictive of their post‐disaster displacement trajectories. Individuals with localised travel patterns and social contacts were more likely to be displaced in the vicinity of their usual residence compared with those with more dispersed travel patterns and social contacts. A majority of IDPs remained within a 10 km radius of their usual place of residence. Across the three disasters, 60–70% of long‐distance displacements (in excess of 100 km) involved travel to a familiar location and/or proximate to one or more contacts discernible in the predisaster CRD data. This pattern holds controlling for the severity of impacts at local area level and is consistent across all three disasters.

Results were validated via comparisons with reports retrospectively quantifying population displacements produced by the International Organisation for Migration, as well as with reference to data on the intensity of each disaster's impact on affected areas. The results indicate that CRD data analysis can be used to predict the estimated number and spatial distribution of IDPs at different time points based on initial estimates of the number of persons displaced in the immediate wake of a disaster, as well as to predict recovery/resettlement timelines. This has important implications for post‐disaster humanitarian response and resettlement efforts. The same methods can support disaster resilience assessments and planning and provide a means to compare recovery and resettlement rates across different disaster events.

6.3. Areas of major gaps in the evidence

6.3.1. Satellite data also presents misclassification problems

Researchers, however, point to several technical challenges in using satellite images that may provide misleading conclusions. For example, Jain (2020) argues for the need for ground validation of satellite data as sometime the images could be misclassified (e.g., flood irrigation may be classified as flooded area); often this misclassification is systematic (ie forest cover is almost always misclassified as agriculture, which will bias the study results). This can be rectified with a field visit. Further, there could be differences in the data coming from different satellites and from the same satellite constellation but using different sensors (e.g., Normalised Difference Vegetation Index varies for different satellite sources and for same satellites across different versions, such as Landsat 5 and Landsat 8). Another example of what qualitative field visits could contribute to improving the interpretability of satellite data are the indicator “quality of roof construction” as a proxy for economic development. Straw roofs from satellite images are generally classified as low‐quality; zinc and other hard roofs are classified as a sign of development. However, several factors may bias this classification: in hot climates, straw roofs may be preferred for their ability to keep cool compared to other hard roofs; poor families may have received donations of high‐quality roofs; or lack of tenured security may discourage people invest in immoveable roof‐tops despite increase in income.

6.3.2. Data quality and transparency is paramount

The map also points to the need to set standards for better reporting, as about 87% of the measurement studies did not report on data quality issues, representativeness, construct validity and generalisability. This would lead to questioning the internal and external validity of the findings. There is also a need to set standards for data transparency, taking into consideration the challenges in sharing proprietary data, data storing and the capacity of the Dataverse (see Box 1 for more details on data transparency).

Box 1. Transparency in data analysis, use and sharing.

In recent times, the use of big data for in‐depth comprehension of developments in the social sector has gained traction. Transparency refers to publishing all relevant materials, including data and code, used in a research study in the public domain for independent verification. Transparency in research encompasses a number of elements that are no different while using big data as opposed to traditional data sources. Certain challenges that might arise are discussed below.

• 1. Deidentification: Deidentification is related to preserving the identity of the study subject before it is made available for any sort of analysis. A few concepts used in de‐identification of big data are K‐anonymity, L‐diversity and T‐closeness. A dataset is said to have K‐anonymity if each person in the dataset cannot be identified by information of the other K‐1 individuals in the dataset. In contrast, L‐diversity is a group‐based anonymisation technique that reduces the granularity in the dataset. T‐closeness is a refinement of L‐diversity and is used to decrease granularity over and above L‐diversity. However, there are several examples of combining different source to re‐identify the respondents in the data set (Archie et al., 2018).
• 2.Scale and storage: In today's world, storage of a high volume of data are not a challenge owing to developments in cloud computing. A fairly new system that provides solution to the scalability and storage issue is storage virtualisation. In simple terms, this is a network of storage devices that are combined to create a single storage space. A few ways of safeguarding the data storage is encrypting all processes and the usage of hybrid clouds. Data repositories such as Harvard dataverse and figshare have limited capacity to handle big data and are often restricted by the size of the data uploaded. Cloud storages such AWS and other similar such storage will be a better option.

6.3.3. Ethical concerns are substantial

Ethical challenges such as consent, data privacy, data security and unintended exclusion are well documented in the literature (Lokanathan et al., 2017; York & Bamberger, 2020). A brief analysis of the studies on the map shows that very few studies report on any of these ethical challenges. However, the challenges are different for different sources of big data. For example, satellite data that involves little human interaction may not need an IRB review but most other big data source that use human‐generated data without explicit consent for secondary use should be reviewed by IRBs. We also recommend more mixed‐method big data evaluations to mitigate the potential disconnect between development stakeholders and big data researchers. Any mixed‐method research needs to be reviewed by IRBs.

The map shows that most IEs have done well on reporting data quality issues but not on ethical issues. Since big data involves ethical issues (such as consent for secondary data use and unintended exclusion) that are new to conventional ethical standards, there is a need to update the current ethical standards practice to include big data use as well.

6.3.4. Big data may be growing in use and popularity, but the need for independent auxiliary data for “ground‐truthing” remain

Many sources of big data are partial in terms of coverage and prone to biases that are difficult to measure, control and correct for in the absence of secondary data. Despite growing awareness and acknowledgement of its limitations, the household sample survey remains the dominant source of development policymaking. Big data often require survey data as “ground‐truth” data to validate the findings. Demographic Health Surveys and Living Standards Measurement Studies are the two main surveys used in ground‐truthing. There is considerable scope for merging the income and expenditure surveys, and food surveys conducted in several developing countries with big data to assess food shortages, poverty hotspots, and so forth.

6.3.5. Some capacity constraints are acute

Development organisations need to build staff capacity in order to use big data as a strategic asset (Perera Gomez & Lokanathan, 2017). They need to build multidisciplinary teams consisting of data experts and subject matter professionals, and also compete with the private sector to recruit the staff. Other major costs involve scaling up the technical infrastructure to enable data storage and processing on a large scale and data accessibility costs. The latter can be more difficult to predict considering that big data sources that are currently public may involve licensing in the future. Besides, ensuring the sustainability of data can be a cause of concern. As suggested by Hammer et al. (2017), as most of the big data is produced as a by‐product by the private sector, continuity of data provision cannot be guaranteed in this age of evolving technology and market conditions. These concerns will call the wisdom of committing resources upfront to build capacity into question.

6.3.6. Need for better coordination between data scientists and evaluators

Big data analysts and evaluators use different framework and analytical tools. In particular, the big data measurement studies look for hidden patterns in the data with little support from theory and aim at prediction rather than causality (York & Bamberger, 2020). Further, the expertise needed to analyse big data remains largely localised and siloed. Outside of a small and highly specialised group of data scientists, there is uncertainty about how best to carry out large‐scale big data analysis. The degree of technical specialisation combined with strict access restrictions to many types of big data has hindered big data applications in development evaluation. Hence, there is a need to promote interaction between development evaluators and data scientists for better cross‐learning and adoption of big data in measuring and evaluating development outcomes.

6.3.7. The cost of collecting, analysing, storing and reporting big data is largely unknown

There is very little publicly available information on the cost of collecting, analysing and reporting big data. Blumenstock et al. (2015) reported that the phone survey for ground‐truthing the CRD data costed USD 12,000 and took 4 weeks to administer. This is, however, only the variable cost of data collection in this study. There are multiple hidden costs such as staff costs and the cost of the necessary computing infrastructure (including storage); in addition, the opportunity cost of time involved in developing partnerships with data providers in some cases is not known. BenYishay et al. (2017) report that the cost of geospatial IE is around USD 150,000. One of the 3ie funded studies using satellite data are reported to have spent USD 3300 on data collection, which is about 1% of the total study budget, but have spent USD 103,864 on data analysis and reporting (about 32% of the total cost‡‡‡‡). Similarly, another 3ie funded study that used in situ fixed sensors reported spending USD 6152 on data collection or acquisition (11%) and USD 54,444 on staff costs for analysis and reporting (55%). However, studies that have combined satellite data with household survey have reported higher costs of data collection (USD 171,582; 43%) and analysis and reporting (USD 109,495; 27%). This is, however, roughly comparable to the data collection cost of an average 3ie funded multi‐year, multi‐round survey IE, which costs about USD 176,000 (Puri & Rathinam, 2019).

6.3.8. Evaluation of potential methodological issues

While big data can help resolve many data related challenges, there are considerable methodological, analytical, logistical and ethical challenges in the way of using it in measuring development outcomes (Letouzé, 2016; Lokanathan et al., 2017; Olteanu et al., 2019; Salganik, 2017). This section briefly discusses some of the prominent methodological challenges. As big data is varied in type, quality and composition, we also discuss if the challenges are specific to any particular type of big data.

We have grouped all the big data challenges that may affect measuring development outcomes credibly and lead to questionable internal and external validity of the studies.

Nonrepresentativeness of data and selection bias

Big data may unintentionally exclude certain sections of the population or marginalised communities, thereby making the sample unrepresentative of the population being analysed. Large samples do not solve this systematic bias. This, however, is not a challenge when using satellite data that has universal coverage but, with human‐generated and CRD data, nonrepresentativeness is a serious challenge. Human‐generated data such as Twitter, Facebook or web searches, as well as mobile phone use that generate CRD data, are not representative as the usage is limited by income, education, infrastructure, and so forth. However, clarity on what is the sample frame (ie who is included and who is excluded) will help interpret big data results appropriately. Nonrepresentative sample is still useful for within‐sample comparisons, but may lead to erroneous out‐of‐sample generalisations (Olteanu et al., 2019; Salganik, 2017).

Construct validity

Construct validity is whether the proposed measure actually measures what it claims to be measuring. This becomes important when the construct is unobservable and has to be operationalised via some observed attributes (Olteanu et al., 2019). For example, does night light data truly reflect local GDP and other development outcomes such as health and education? What is it in the CRD data that reflects people's income or employment status? In many cases, the big data‐based measures may not be straightforward, and it is good practice to clearly state construct validity and provide necessary support to back the claim in the papers. Development measures based on social media are particularly challenging due to different communication styles, special usage of terms and differences in language proficiency.

Data quality issues

• Comparability of data over time: Since most of these data are collected routinely as a part of business, the nature and quality of data may change with the technology and business requirements. This may happen because the underlying technology has changed or because the people who use it have changed. For example, satellite data are not readily comparable across the years as there is a vast quality difference (Jain, 2020); good flu trends based on online searches peaked comparing to officially reported data when the underlying Google algorithm started prompting people to query more and broke the relationship between Google searches and flu prevalence (Archie et al., 2018).

• Lack of completeness: Most big data is a by‐product of peoples' everyday action and/or result of system logs of the government and businesses. It may not contain all the necessary information, such as demographic characteristics. However, combining multiple sources of data, especially big data and administrative data, can help resolve this problem (Salganik, 2017).

Generalisability

Generalisability or external validity refers to the applicability of the findings of a study to population or context other than it was produced. In the context of big data, generalisability would mean the applicability of the model to a setting different from the setting of the data that the model was trained on. For example, a model trained on satellite data from a specific geographical region may not be generalised (Head et al., 2017; Jean et al., 2016). It is good practice for studies to report on the representativeness of training data.

Data transparency

Transparency in this context refers to publishing all relevant materials, including the data and code, used in a study in the public domain for independent verification. Sharing of raw data in the public domain is often crucial for establishing confidence and reliability in the results. There are two challenges here: first, some of the major sources of big data (such as CRD) are proprietary and sharing may not be permitted beyond the closed group of researchers; and second, the data has to be de‐identified before it can be shared and it is crucial to check whether there are variables or a combination of variables that can be used to reidentify research subjects.

We assessed whether the studies included in the systematic map asked the following questions:

• Is the data representative of the population of interest?

• Is the construct validity explained (ie is there a discussion on how the big data‐based indicator measures what the study claims to measure)?

• Are there data quality issues in the dataset used and how are they addressed?

• Are the results generalisable? For example, are the research findings generalisable to other situations, such as other platforms (data source) or communities, or over time?

• Are data and codes publicly available for replication?

6.3.9. Evaluating reporting on privacy and ethical considerations

There are concerns over data access, privacy, consent and ethics in using big data. Although these are foundational issues for both small and big data studies, the challenges posed by big data have greater repercussions.

When using big data sources such as mobile data, most mobile operators have “inform and consent” policies that mandate disclosure of all relevant information to potential participants who can then evaluate this information and give explicit permission. However, these policies often contain legal language that is generally not discernible, and it is not clear if explicit consent is obtained to repurpose the data. This kind of informed consent may be completely absent in research leveraging social media data due to the impracticality of obtaining consent from millions of users.

Mobile phone user data and social media data are some of the most used sources of big data that can inform researchers about individuals' behaviour. Even if the data are de‐identified, concerns still remain over the consent and ethics of sharing such data with researchers. It is thus imperative to have an ethics approval process in place that lays down the conditions under which such research can take place. There is a need for clear ethical standards for big data research and studies should be monitored by the IRBs.

Another ethical criterion when using big data can be concerned with the assessment of risks, the most common being privacy breaches leading to identity theft or other cybersecurity risks. The possibility of the reidentification of any individual user from poorly anonymised datasets adds to the concerns over anonymity of subjects. When combined with other sources, such datasets can be used to gain detailed insights about people without their knowledge. Such precise inferences may create the capacity for discrimination or mass manipulation. Sometimes data obtained for one purpose in social data research is used for secondary analyses, but the associated risks may not be well understood. For example, Facebook data in the past has been used for ad targeting, as well as for tailoring propaganda (Horowitz et al., 2018).

Big data may also inadvertently exclude certain sections of the population. For example, this bias can be observed in the case of “Street bump,” a mobile app that notifies the Boston City Hall whenever the user hits a bump on the road (Carrera et al., 2013). The data includes information only from the app users who often use both their cars and the app; this might inadvertently exclude poorer parts of the city that app users may not frequent. Policy based on such big data sources may have unintended consequences for the people who are excluded.

We assessed the studies on the following:

• Ethical approval obtained

• Consent for secondary and other use of data discussed explicitly

• Discussions on data privacy

• Discussions on data security and governance arrangement

• Discussion of any potential unintended exclusion

• Discussion of potential unintended consequences for any group of people or individuals.

6.3.10. Reporting on privacy and ethical challenges

Figure 12 shows that most studies (81%) do not report on ethical challenges and privacy issues. Of the few that do discuss such challenges, the most frequently discussed issue is consent for data use.

Figure 12.

Number of studies reporting on ethics issues

6.4. Limitations of the systematic map

This map covers large thematic areas and outcomes corresponding to SDGs. Given the wide scope of the outcomes, the evidence is sparse and bunched around a few themes. The thematic gaps here may not be read as actual gaps, but these areas may be not readily relevant to using big data. This map rather shows what evidence or proof‐of‐concepts are available to measure and evaluate development outcomes using big data.

This map followed a systematic process of searching, screening and coding of studies based on a predefined set of criteria in the study protocol developed with inputs from key stakeholders. However, despite best efforts in searching and screening the studies, given the wide scope of big data sources and their application across all developmental themes and the pace at which the literature is growing, it is possible that some relevant studies (especially measurement studies) could have been missed out.

It was beyond the scope of the study to provide a critical quality appraisal of the IEs or the measurement studies, given the large number of studies included on the map; nor did the report look at the details of ML methods used in the included studies.

Given the wide scope of development applications, it was not possible to code the studies for all subclassifications. Though the submaps (especially for economic development and livelihoods, health and well‐being and urban development) provided coding at level 2 indicators, it was not possible to provide granular analysis of development themes corresponding to SDG indicators at level 3. Future systematic maps should aim to produce more granular classifications on the use of big data at the indicator level.

Some studies have used ML techniques for treatment effect heterogeneity in RCTs (Chernozhukov et al., 2019). However, it was beyond the scope of this report to include the role of big data analytical methods in conventional IE designs such as RCT and other quasi‐experimental designs. This is a nascent but growing body literature and could be considered for inclusion in future maps.

Several studies suggest that the key advantages of big data sources (especially satellite data) are their long‐term availability which will help evaluate the long‐term impact of development interventions. The possibility of collecting a vast array of information on several contextual factors using big data can help evaluate complex interventions (Bamberger, 2016). However, this map did not code the studies for long‐term impact or for complex interventions. Future maps may code and analyse the role of big data in measuring long‐term impact and in evaluating complex interventions.

6.5. Stakeholder engagement throughout the systemaic map process

The stakeholders in this systematic map included an advisory group comprised of sector experts, FCDO staff, CEDIL staff. All stakeholders were engaged in reviewing drafts and final reports associated with the map. The protocol for the map was developed with inputs from the advisory board, and FCDO and CEDIL staff. The advisory board played a key role in steering the search strategy and building the list of key words.

7. AUTHORS' CONCLUSIONS

Big data has great potential to help address questions of relevance to international development, including for evaluating the effects of interventions. This systematic map compiles IEs, SRs and measurement studies that incorporate big data to highlight how this innovative, new data source is being used to evaluate development outcomes and (more importantly) where there is more potential to use big data in the future evaluations. We found 437 studies, of which 48 are IEs, 381 are measurement studies and 8 are SRs. Roughly half the studies are from Asia and another 30% are from Africa; about 70% are from LMICs. Of the 48 IEs, 8 are RCTs and the remaining are quasi‐experimental studies.

Our results highlight considerable potential for using big data for measuring various development outcomes across SDG themes, but big data is more relevant to environmental sustainability, economic development and livelihoods, health and well‐being and urban development. This map also highlights that big data can contribute to the evidence base in development sectors where evaluations are not generally feasible due to a lack of data, particularly due to fragile contexts.

One of the key “absolute gaps” the map has identified is that the number of IEs is lower in comparison to measurement studies. Given the fast‐growing availability of big data and improving computation capacity, there is great potential for using big data in future IEs. This may not, however, be straightforward as there are several analytical, ethical and logistical challenges that may hinder the use of big data in evaluations. The development community that helps set standards and best practices and development stakeholders (including donors who facilitate rigorous evaluations and learning) have a strong role to play in facilitating this process. The report highlights the need for setting standards for better reporting on data quality issues, representativeness, construct validity and generalisability, as well as the need for data transparency and sharing. The report also calls for facilitating better interaction between big data analysts, remote sensing scientists and evaluators.

One of the key findings of the report is that satellite and sensor data are the most used data sources for both measurements studies and IEs. There are several sources of preprocessed satellite data that could be used directly in evaluations without the evaluators having to process them using complex ML models themselves. Satellite data seems to be particularly useful in the context where the development interventions and the outcomes studied have spatial dimension economic activity at the local level, urban development, forest cover, land productivity and distribution of the population, or where the outcome and other covariates are measured on a spatial scale (ie villages, counties, districts, plots or protected areas). CRD data, on the other hand, despite being used widely in measurement studies, is not yet used in IEs. The data deficiency in international development is partly due to fragile contexts such as diseases spread, violence, natural calamities and difficult terrain. This map highlights the potential of big data in fragile contexts: one‐quarter of the studies were conducted in such a context.

For evaluators and researchers, the report calls for better reporting on data quality, ethics and transparency. There is also an absolute gap in using mixed methods jointly with big data and cost‐effectiveness. For the donors, this report calls for more efforts on setting up best practices and ethical standards and in facilitating more interaction among remote sensing scientists, big data analysts and development evaluators.

7.1. Implications for research, practice and/or policy

• Reliable data are paramount to evaluating development outcomes and future resource allocation.

• This systematic map compiles the IEs, SRs and measurement studies to highlight how innovative, new data sources are being used in evaluating development outcomes, and more importantly where there is more potential to use big data in the future evaluations.

• This map shows that big data can contribute to evidence base in development sectors where evaluations are not generally feasible due data deficiency.

• Given the fast growing availability of big data and improving computation capacity, there is a great potential for using big data in the future IEs.

• There are several sources of preprocessed satellite data that could be used in evaluations directly without the evaluators having to process them using complex ML models themselves

• There is also an absolute gap in using mixed methods jointly with big data and cost effectiveness. This should be prioritised by donors and researchers as a mix of quantitative big data analysis and qualitative field level analysis will help strengthen the validity of the results.

• More efforts, on the donors' end, is required to set up best practices and ethical standards, and facilitating more interaction among remote sensing scientist, big data analysts and development evaluators.

CONTRIBUTIONS OF AUTHORS

• Content: Francis Rathinam, Samantha Watson, Sebastian Vollenweider, Zeba Siddiqui, Manya Mallik, Pallavi Duggal, and Sayak Khatua.

• Map methods: Francis Rathinam.

• Statistical analysis: Sayak Khatua and Francis Rathinam.

• Information retrieval: Zeba Siddiqui, Manya Mallik, Pallavi Duggal, and Sayak Khatua.

DECLARATIONS OF INTEREST

The authors declare no conflict of interest.

PLANS FOR UPDATING THE SYSTEMATIC MAP

The systematic map shows that both IEs and measurement studies have dramatically increased in the past 5 years and are continuing to grow in number. Given the potential for faster growth in the availability and computational capacity, it is very likely that the number of studies will grow faster than we have witnessed over the past 5 years. Hence, we recommend that this map be updated within the next 2 years.

The fact that more than 80% of the included studies are peer‐reviewed shows the growing number of journals interested in big data application in international development. It will be useful to include a more exhaustive grey literature search to identify the full extent of the literature.

This map shows the potential for big data to measure and evaluate various development themes. However, most of these studies are supported by universities and specialist organisations and conducted by researchers associated with these organisations. Widely disseminating the findings of the map among development researchers, evaluators, practitioners and donors will help promote the adoption of big data measures in future IEs.

APPENDIX A. LINK TO ONLINE INTERACTIVE MAP

Figure A1.

Systematic map of big data sources and outcomes. The online map can be accessed here: https://gapmaps.3ieimpact.org/evidence-maps/big-data-systematic-map

APPENDIX B. THE OECD DEFINITION OF FRAGILE STATES

OECD defines fragility as the combination of exposure to risk and insufficient coping capacity of the state, system and/or communities to manage, absorb or mitigate those risks. Fragility can lead to negative outcomes including violence, the breakdown of institutions, displacement, humanitarian crises or other emergencies. The OECD fragility framework considers not only current exposure to negative events such as natural disasters and armed conflict but also capacity to cope with likely future negative events.

The new framework considers five dimensions of fragility.

• Economic fragility is vulnerability to risks stemming from weaknesses in economic foundations and human capital including macroeconomic shocks, unequal growth and high youth unemployment;

• Environmental fragility is vulnerability to environmental, climatic and health risks that affect citizens' lives and livelihoods. Risk factors can be external or internal, including exposure to natural disasters; air, water and sanitation quality; prevalence of infectious disease; number of uprooted people; and vulnerability of household livelihoods;

• Political fragility is vulnerability to risks inherent in political processes, events or decisions; political inclusiveness (including of elites); and transparency, corruption and society's ability to accommodate change and avoid repression. Risk factors include regime persistence, state‐sponsored violence or political terror and levels of corruption;

• Security fragility is the vulnerability of overall security to violence and crime, including both political and social violence. Risks are measured by the homicide rate, level of violent organised crime, number of deaths from nonstate actors or terrorism, number of battle deaths from conventional warfare, and levels of domestic violence; and

• Societal fragility is vulnerability to risks affecting societal cohesion that stem from both vertical and horizontal inequalities, including inequality among culturally defined or constructed groups and social cleavages. Risk indicators include income inequalities (vertical) and social inequalities related to gender, growth in urbanisation and numbers of displaced people.

In the past few years, OECD has moved away from the “fragile states list” and towards measuring each of those five dimensions on a spectrum of intensity for 58 fragile contexts. This comes as a part of their effort to move towards a universal concept of fragility, recognising that it affects not only developing countries but all countries to some extent.

Table B1.

Fragile contexts provided in the OECD fragility framework 2018 (decreasing order of severity)

Economic	Environmental	Political	Security	Societal
Central African Republic	Somalia	Eritrea	Syria	South Sudan
South Sudan	Central African Republic	Sudan	Libya	Yemen
Liberia	South Sudan	DPRK	Yemen	Syria
Somalia	Chad	Yemen	Somalia	Egypt
Solomon Islands	DRC	Syria	Afghanistan	Somalia
Afghanistan	Burundi	Somalia	South Sudan	Sudan
Niger	Mozambique	South Sudan	Iraq	Burundi
Sierra Leone	Guinea‐Bissau	Burundi	Sudan	Eritrea
Comoros	Niger	Chad	Central African Republic	DPRK
Guinea‐Bissau	Liberia	Congo	Nigeria	DRC
Mozambique	Burkina Faso	Gambia	Mali	Congo
Haiti	Guinea	Ethiopia	Pakistan	Chad
Mali	Sierra Leone	Guinea	Chad	Equatorial Guinea
Congo	Mali	DRC	DRC	Pakistan
Libya	Malawi	Mauritania	West Bank & Gaza	Iran
Syria	Zambia	Bangladesh	Cameroon	Central African Republic
Yemen	Cameroon	Angola	Egypt	Afghanistan
Tajikistan	Eswatini	Djibouti	Niger	Zimbabwe
Djibouti	Côte d'Ivoire	Venezuela	Haiti	Ethiopia
Gambia	Eritrea	Iran	Ethiopia	Kenya
Mauritania	Ethiopia	West Bank & Gaza	Kenya	Eswatini
Iraq	Afghanistan	Zimbabwe	Myanmar	Mauritania
Honduras	DPRK	Madagascar	Nepal	West Bank & Gaza
Timor‐Leste	Yemen	Afghanistan	Burkina Faso	Angola
Eswatini	Haiti	Central African Republic	Congo	Iraq
Pakistan	Syria	Libya	Eritrea	Guatemala
DRC	Tanzania	Egypt	Burundi	Gambia
Eritrea	Zimbabwe	Guinea‐Bissau	Mauritania	Cameroon
Burkina Faso	Madagascar	Iraq	Iran	Haiti
Burundi	Congo	Sierra Leone	Comoros	Uganda
Zimbabwe	Rwanda	Mali	Bangladesh	Rwanda
Ethiopia	Myanmar	Haiti	Venezuela	Lao PDR
Chad	Uganda	Côte d'Ivoire	Tanzania	Myanmar
Guinea	Pakistan	Mozambique	Uganda	Venezuela
Malawi	Comoros	Myanmar	Mozambique	Tajikistan
Madagascar	Bangladesh	Rwanda	Guatemala	Bangladesh
Rwanda	Tajikistan	Liberia	Côte d'Ivoire	Guinea
DPRK	Lao PDR	Nigeria	Sierra Leone	Libya
Zambia	Djibouti	Pakistan	Liberia	Djibouti
Uganda	Mauritania	Burkina Faso	Honduras	Guinea‐Bissau
Tanzania	Kenya	Uganda	Guinea	Tanzania
Papua New Guinea	Papua New Guinea	Kenya	Guinea‐Bissau	Honduras
Nepal	Angola	Honduras	Angola	Mozambique
Equatorial Guinea	Iraq	Niger	Gambia	Côte d'Ivoire
Lao PDR	Gambia	Tajikistan	Tajikistan	Madagascar
Myanmar	Sudan	Equatorial Guinea	Equatorial Guinea	Nigeria
Cameroon	Nepal	Cameroon	Lao PDR	Papua New Guinea
Angola	Timor‐Leste	Eswatini	Solomon Islands	Zambia
Bangladesh	Nigeria	Lao PDR	Madagascar	Mali
Guatemala	Guatemala	Guatemala	Papua New Guinea	Sierra Leone
Côte d'Ivoire	Solomon Islands	Zambia	Zimbabwe	Nepal
Kenya	Venezuela	Timor‐Leste	Zambia	Burkina Faso
Venezuela	Honduras	Nepal	Timor‐Leste	Malawi
Nigeria	West Bank & Gaza	Tanzania	Malawi	Niger
Sudan	Iran	Papua New Guinea	Eswatini	Timor‐Leste
Iran	Egypt	Comoros	Djibouti	Liberia
West Bank & Gaza	Libya	Malawi	Rwanda	Solomon Islands
Egypt	Equatorial Guinea	Solomon Islands	DPRK	Comoros

APPENDIX C. DETAILS OF SUBMAPS

Table C1.

Outcome categories and definitions

Category	Definition	Subcategories
Economic development and livelihoods (SDG 1 and 8)	End poverty in all its forms everywhere	Poverty mapping/measurement
	Promote sustained, inclusive and sustainable economic growth, full and productive employment and decent work for all	Measuring wealth/GDP
		Access to basic services
		Social protection, including financial inclusion
		Vulnerability reduction
		Economic growth and productivity
		Full and productive employment, including youth unemployment
		Eradicate forced and underage labour, protect labour rights and promote safe workspaces
		Promote sustainable tourism
		Strengthen capacity of domestic financial institutions
Health and well‐being (SDG 3)	Ensuring healthy lives and promoting well‐being for all ages	Reduce mortality
		End epidemics of communicable diseases
		Strengthen prevention and treatment of substance abuse
		Decrease fatalities due to road accidents
		Universal access to sexual and reproductive healthcare services
		Achieve universal health coverage
		Reduce fatalities due to pollution
Governance and human rights (SDG 5, 10 and 16)	Achieving gender equality and empower all women and girls
	Reduce inequality within and among countries
	Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels
Urban development (SDG 11)	Make cities and human settlements inclusive, safe, resilient and sustainable	Access to affordable housing
		Access to affordable transport systems
		Enhance inclusive and sustainable urbanisation
		Strengthen efforts to protect cultural and natural heritage
		Better disaster management
		Reduce environmental impact of cities
		Universal access to green public spaces
Environmental sustainability (SDG 12, 13, 14 and 15)	Ensure sustainable consumption and production patterns
	Take urgent action to combat climate change and its impacts
	Conserve and sustainable use the oceans, seas and marine resources for sustainable development
	Protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, halt and reverse land degradation and halt biodiversity loss

APPENDIX D. SEARCH STRATEGY AND THE DATABASES SEARCHED

We developed a systematic search strategy in consultation with an information specialist after finalising the protocol.

We searched the following general databases:

• CAB Abst: www.cabi.org/publishing-products/online-information-resources/cab-abstracts/

• Econlit (Ovid): www.ovid.com/site/catalog/databases/52.jsp

• Ebsco Discovery: https://www.ebscohost.com/discovery

• Scopus: https://www.scopus.com/

• Social Sciences Citation Index (SSCI) (via Web of Science): https://library.maastrichtuniversity.nl/collections/databases/ssci/

Bilateral and multilateral agencies and general repositories of IEs in international development:

• 3ie Repository of IEs: www.3ieimpact.org/en/evidence/impactevaluations/

• 3ie RIDIE (Registry for International Development IEs): http://ridie.3ieimpact.org/

• USAID Evaluation Clearing House: https://dec.usaid.gov/dec/content/evaluations.aspx

• Innovations for Poverty Action: www.poverty-action.org/project-evaluations

• J‐Poverty Action Lab: www.povertyactionlab.org

• The World Bank: www.worldbank.org/

• AEA RCT Registry: https://www.socialscienceregistry.org/

• DFID Research for Development: http://r4d.dfid.gov.uk/

• Campbell Collaboration: www.campbellcollaboration.org

• African Development Bank: https://www.afdb.org/en/documents/publications/

• BREAD: http://ibread.org/bread/papers

• Center for Effective Global Action: http://cega.berkeley.edu/evidence/

We searched specialist organisational databases that might include big data measurement studies and IEs:

• AidData: A Research Lab at William & Mary: www.aiddata.org/

• Flowminder: https://web.flowminder.org/

• Stanford University Center on Food Security and the Environment: https://fse.fsi.stanford.edu/

• UN Global Pulse: www.unglobalpulse.org/

• Data‐Intensive Development Lab: http://didl.berkeley.edu/

• Global partnership for sustainable development: www.data4sdgs.org

• Arxiv database: (https://arxiv.org).

• Eldis: www.eldis.org

We also searched grey literature via Google Scholar and checked references of any SR that we find in our searches and met our inclusion criteria.

Given the relative recent state of the evidence base in this field and the fact that most programmes/initiatives started to flourish in the late 2000s we conducted the searches from 2005 onwards.

Web of Science (SSCI and Science Citation Index), searched 22 September 2019

# 1 34,746

TS= ("big data" or metadata or "meta data" or meta‐data or "macro data" or macrodata or "mass data" or massdata or "large data" or "bulk data")

# 2 45,418

TS= (((satellite* NEAR/3 imag*) or sensor* or surveillance or meter* or drone* or ((mobile* or cell) NEAR/3 (phone* or telephone*)) or mobiles or gps or "global positioning" or gis or "global information system*" or self‐tracking or "self tracking") NEAR/3 data)

# 3 41,524

TS= (((e‐commerce or commerc* or business* or "credit card*" or ATM* or "automated teller" or "cash machine*" or "money transfer*") and (transaction* or record or records)) or ((savings or loan or loans) NEAR/3 repay*) NEAR/3 data)

# 4 109

TS= (((((toll or tolls) NEAR/2 (road* or highway* or motorway*)) or "public transport") NEAR/3 data))

# 5 1,036

TS= ((((online or internet or web or virtual or google) NEAR/3 (search* or log or logs)) NEAR/3 data))

# 6 14,088

TS= ((("social network*" or "social media" or "opinion platform*" or blog* or Twitter or Facebook or LinkedIn or YouTube or Wiki* or Open or "text messag*") NEAR/3 data))

# 7 596

TS= ((((citizen* NEAR/2 (report* or derived or submit* or communicat* or inform or informed or informing or feedback)) or hotline* or crowdsourc*) NEAR/3 data))

# 8 133,635

#7 OR #6 OR #5 OR #4 OR #3 OR #2 OR #1

# 9 35,994

TS= (((poverty NEAR/5 (eradicat* or measur* or map*)) or ((wealth or GDP) NEAR/3 measur*) or (access* NEAR/5 (service* or financ*)) or "social protection" or (reduc* NEAR/3 vulnerab*)))

# 10 36,506

TS= ((((hunger or malnutrition or malnourished) NEAR/3 (eradicat* or decreas* or end*)) or "food security" or (improv* NEAR/3 nutrition*) or "sustain* agriculture" or (access* NEAR/3 food*) or ((productiv* or income*) NEAR/3 ("small scale" NEAR/2 produc*)) or (sustain* NEAR/3 "food produc*") or ((maintain* or maintenance) NEAR/2 "genetic diversity")))

# 11 57,651

TS= ((((health or well‐being or wellbeing or "well being") NEAR/2 (promot* or ensur*)) or (reduc* NEAR/3 (maternal or child* or "under 5*" or under‐5* or "under five*") NEAR/2 (mortality or death*)) or (("communicable disease*" or "infectious disease*") NEAR/3 (epidemic* or pandemic*)) or (reduc* NEAR/2 ("premature mortality" or "premature death*")) or ((prevent* or treatment) NEAR/3 "substance abuse") or ((reduc* or prevent*) NEAR/3 (road* or traffic or vehicle*) NEAR/2 (accident* or death* or fatal*)) or (access* NEAR/2 (reproductive or sexual) NEAR/2 service*) or (universal NEAR/2 health NEAR/2 coverage) or (pollut* NEAR/3 (death* or mortality or fatal*) NEAR/2 reduc*)))

# 12 26,746

TS= (((education or "life‐long learning" or (skill* NEAR/3 (build* or acquir* or learn*)) or literacy or numeracy) NEAR/3 (access* or complet* or graduat* or gender* or opportunit*)))

# 13 40,719

TS= (((gender* or women or girl* or female*) NEAR/3 (equalit* or inequalit* or discriminat* or empower* or violen* or harm* or (unpaid NEAR/2 (domestic or house*)) or participat* or ((sexual or reproductive) NEAR/3 right*))))

# 14 100,434

TS= (((water or WASH or sanitation) NEAR/3 (access* or quality or sustainable or manag* or ecosystem*)))

# 15 139,810

TS= (((energy NEAR/3 (access* or renewable or afford* or clean or sustainable or efficien*))))

# 16 210,410

TS= ((((econom* NEAR/2 (growth or productivity)) or employment or labour or work or job or jobs or development or "global resource*" or ((youth or "young people") NEAR/3 unemploy*) or tourism or (financ* NEAR/3 institution*) or bank or banks or banking) NEAR/3 (sustain* or increas* or promot* or efficien* or reduc* or forced or underage or "under age" or rights or safe or strengthen* or (capacity NEAR/2 build*))))

# 17 19,679

TS= (((industr* or innovation or infrastructure or (financ* NEAR/2 service*) or ((scien* or techn*) NEAR/2 research)) NEAR/3 (resilen* or sustain* or promot* or foster* or access* or upgrad* or modern*)))

# 18 97,676

TS= (((inequalit* NEAR/2 (reduc*)) or ((income NEAR/2 growth) or "equal opportunit*") NEAR/2 (sustain* or increas* or promot*)) or (inclusivity NEAR/2 (promot* or sustain* or empower*)) or (equality NEAR/2 (increas* or promot* or engender*)) or ((global NEAR/2 (financ* or market* or institution*) NEAR/3 (regulat* or monitor* or legislat* or law or laws)) or (("developing countr*" or LMIC* or "low and middle‐income" or "third world" or "global south") NEAR/3 (represent* NEAR/3 (enhanc* or ensur* or increas* or promot*)) or (migration NEAR/3 (policy or policies) NEAR/3 plan*) or (different* or special or positive) NEAR/2 (treat* or discriminat*)))

# 19 36,884

TS= (((city or cities or urban* or communit* or village* or housing or dwelling* or transport* or ((cultural or natural) NEAR/3 heritage) or ((disaster* or emergenc*) NEAR/3 manag*) or "green space*") NEAR/3 (sustain* or safe or inclusive or resilien* or afford* or protect* or ((reduc* or limit* or mitigat*) NEAR/3 "environmental impact" NEAR/3 (city or cities or urban*) or access*))))

# 20 149,994

TS= (((consum* or production or "natural resource*" or waste or wastes or procurement or (information NEAR/2 (access* or availab*))) NEAR/3 (sustain* or responsible or efficien* or environment* or reduc*)))

# 21 201,412

TS=(("climat* change*" or "global warming" or (greenhouse NEAR/2 (effect* or gas*)) NEAR/3 (action or combat* or mitigat* or impact* or resilien* or adapt* or (national NEAR/2 (policy or policies)) or ((promot* or rais* or improv*) NEAR/3 aware*))))

# 22 54,354

TS= (((ocean* or sea or seas or marine or coastal or fishing or fisher* or "small island*") NEAR/3 (sustain* or conserv* or manag* or protect* or regulat* or ecosystem* or (reduc* NEAR/2 pollut*) or acidity)))

# 23 22,000

TS= (((terrestrial or land or land‐based or forest* or mountain* or (natural NEAR/2 (habitat or ecosystem*)) or poaching or poacher* or traffick*) NEAR/3 (sustain* or conserv* or protect* or (degrad* NEAR/3 (neutral or prevent* or protect* or minimi* or reduc*)) or ("genetic resources" NEAR/2 (utilis or utiliz* or use or using or promot*)) or prohibit* or ban or banning or banned) or (("alien species" NEAR/2 (invasive or introduced or imported)) NEAR/3 (prevent* or impact*)) or ((ecosystem* or biodiversity) NEAR/3 (national or local*) NEAR/3 (plan* or policy or policies))))

# 24 2,378

TS=(((societ* NEAR/3 (peaceful or inclusive)) or justice or "rule of law") NEAR/3 (sustain* or access* or effective* or accountab* or strong or robust or promot*))

# 25 105,185

TS=((violen* or corrupt* or ((illicit or illegal) NEAR/2 financ* NEAR/2 (activit* or dealing*)) NEAR/2 (reduc* or mitigat* or eliminat*)) or (institution* NEAR/2 (accountab* or transparen* or effective*)))

# 26 17,951

TS=(((("decision making" or decision‐making or participati*) NEAR/2 (representative or democra*)) or (("global governance" NEAR/3 (participati* or involv* or includ*)) NEAR/3 ("developing countr*" or LMIC* or "low and middle‐income" or "global south" or "third world")) or ("legal identit*" NEAR/3 (provi* or promot* or regist*)) or (information NEAR/3 access*)))

# 27 141

TS= (("sustainable development goal*" NEAR/3 (partnership* or implement* or revitali* or strengthen* or sustainability or (capacity NEAR/2 build*) or ((development or financial or domestic) NEAR/2 (resource* or assistance)) or ((investment or technolog* or ("trading system*" NEAR/2 universal)) NEAR/3 (promot* or innovation*)) or (knowledge NEAR/2 shar*) or "technology bank*" or ((exports or market*) NEAR/3 (increas* or promot*) NEAR/3 ("developing countr*" or LMIC* or "low and middle‐income" or "third world" or "global south")) or "global macroeconomic" or ((policy or policies) NEAR/2 coheren*) or "global partnership*" or "public‐private partnership*" or (progress NEAR/3 (monitor* or assess* or evaluat* or review*)))))

#27 OR #26 OR #25 OR #24 OR #23 OR #22 OR #21 OR #20 OR #19 OR #18 OR #17 OR #16 OR #15 OR #14 OR #13 OR #12 OR #11 OR #10 OR #9

# 28 1,196,499

#28 AND #8

# 29 16,624

# 30 4,365

TS=(sdg or sdgs or "sustainable development goal*")

#30 AND #1

# 31 35

#31 OR #29

# 32 16,625 (Indexes=SCI‐EXPANDED, SSCI Timespan=2005‐2019)

#28 AND #8

# 33 3,519 (Indexes=SSCI Timespan=2005‐2019)

APPENDIX E. DATA EXTRACTION TOOL

Table E1.

Data extraction tool

Variable name	Variable description
Study ID	Unique ID ascribed to each record
Title name	Use only the English version of the publication's main title. If paper is not written in English and has the title translated, use the translated version of the title. If the publication does not provide an English version, include the title in its original language. Please enter title in sentence case. Ensure there are no line breaks
Language	Select full‐text language that applies: English
Open access	If the study's (full‐text) content is available, code as “Yes.” If study has paywalls, code as “No”
	Please save the PDF in the Dropbox folder called “Full‐Text PDFs” using the following format Firstauthorsurname_year_record id
	If study has multiple versions, in other words, if the study has been published as both a journal article and a working paper, both versions may be included in the IER
Type of big data	Select one or more from the list based on the type of big data being used:
	1. Human‐sourced information (social networks) 11. Social networks 12. Internet searches 13. Mobile data content 14. Citizen reporting or crowdsourced data 2. Process‐mediated data (traditional business systems and websites) 21. Data produced by public agencies 22. Data produced by businesses 23. Mobile phone CRD 3. Machine‐generated data (automated systems) 31. Data from fixed sensors 32. Data from mobile sensors (tracking) 33. Data from satellites
Outcome	Select ONE outcome that applies according to the intervention being evaluated:
	Economic development and livelihood
	Sustainable agriculture and food security
	Health and well‐being
	Education
	Water and sanitation
	Governance and human rights
	Energy, industry and infrastructure
	Urban development
	Environment sustainability
	Global partnership
Suboutcomes	Poverty mapping/measurement
	Measuring wealth/GDP
	Access to basic services
	Social protection, including financial inclusion
	Vulnerability reduction
	Economic growth and productivity
	Full and productive employment, including youth unemployment
	Eradicate forced and underage labour, protect labour rights and promote safe workspace
	Promote sustainable tourism
	Strengthen capacity of domestic financial institutions
	Reduce mortality
	End epidemics of communicable diseases
	Strengthen prevention and treatment of substance abuse
	Decrease fatalities due to road accidents
	Universal access to sexual and reproductive healthcare services
	Achieve universal health coverage
	Reduce fatalities due to pollution
	Achieve gender equality and empower all women and girls
	Reduce inequality within and among countries
	Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels
	Access to affordable housing
	Access to affordable transport systems
	Enhance inclusive and sustainable urbanisation
	Strengthen efforts to protect cultural and natural heritage
	Better disaster management
	Reduce environmental impact of cities
	Universal access to green public spaces
	Ensure sustainable consumption and production patterns
	Take urgent action to combat climate change and its impacts
	Conserve and sustainable use the oceans, seas and marine resources for sustainable development
	Protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, halt and reverse land degradation and halt biodiversity loss
Gender equity focus	Does this study consider gender and/or* equity?
	Yes
	No
Cost	Is cost data provided?
	Yes
	No
Population	What population sub‐groups does the study target?
	Rural
	Urban
	Refugees
	Conflict affected persons
	Ethnic minorities
	Sex
Geographical coverage	Select the continent/region in which the study was conducted:
	Asia
	Africa
	North America
	South America
	Europe
	Australia
	Antarctica
Fragile context	Select the type based on fragile context definition
	a. Conflict related difficulty b. Logistical difficulty c. All others
Evaluation design	Select one of three options defined as:
	1. Experimental
	a) RCT defined as prospective randomised assignment, where randomisation is implemented by researchers (or by decision makers in the context of an evaluation study)
	2. Quasi‐experimental
	a) Quasi‐random assignment: i) regression discontinuity design (sharp designs) or ii) natural experiment in which exposure to treatment is random  b) b) Nonrandom assignment: i) studies that control for unobservables (DID, FE, IV, Fuzzy RDD, ITS) or ii) studies that control for observables only (e.g., statistical matching, synth control, regression adjustment)
	3. Measurement studies
	Studies that have innovatively used big data to measure and validate an SDG indicator such as poverty mapping, food security, forest cover, and so forth
Evaluation method	If experimental, then select (only if one or two selected under evaluation design):
	RCTs
	If quasi‐experimental, then select:
	Sharp RDD
	DID
	FE estimation
	IV estimation
	Fuzzy RDD
	Statistical matching (includes PSM)
Methods for analysing big data	Select one or more methods used in the analysis of “big data”:
	ML: supervised learning, unsupervised learning, clustering, anomaly detection, random forests, artificial neural network, convolutional neural network, support vector regression
	Bayesian geostatistical models: latent Gaussian models, integrated nested Laplace approximations
	Natural language processing: vectorisation, embedding, categorisation, sentiment analysis
	Any other: please specify
Big data validation	Select one or more methods used to calibrate/validate results of big data analysis using survey data:
	Administrative/survey data used for calibration or validation? Yes/No
	Type of data
	Administrative data Survey data a. Secondary survey data b. Primary survey data
	Survey name
	Demographic Health Survey Living Standards Measurement Study Other (specify)
	Mode of data collection
	Face‐to‐face Telephone mode Mobile phone telephone mode Dual‐frame Other (specify)
	Sample size
	Sample design
	Link to survey details (if available)
Mixed methods	Select YES if study includes quantitative and qualitative analyses, otherwise select NO
Transparency in data collection, analysis and reporting	Choose if the study discusses the following:
	Are the data collection methods described? Are data quality issues such as completeness (missing or incomplete entries; empty cells) and noise discussed? Is the data representative of the population of interest? Is the construct validity explained (ie is there a discussion on how the big data‐based indicator measures what the study claims to measure)? Are the results generalisable? For example, are the research findings generalisable to other situations such as other platforms (data source) or communities, or over time? Are data and codes publicly available for replication? Any other data, analysis and reporting challenge discussed?
Ethical approval and discussion	Choose if the study discusses the following:
	Ethical approval obtained? Consent for secondary and other use of data Data privacy Data security and governance arrangement Unintended exclusion Unintended consequence on any group of people and/or individuals Others (specify)
Unit of observation	Enter all the levels of observation of the variables used for the analysis:
	Community Cohort (includes schools or clinics) Household Individual
	If more than one, include in separate rows:
	Country Districts Sub‐districts Village/city
Funding agency	What category of funding agency funded the research?
	Note: only code if reported in the study; no need to do additional research to find
	Select one of the following:
	Government agency International aid agency International financial institution Nonprofit organisation For‐profit firm Academic institution Charitable foundation or private foundation Not specified

APPENDIX F. ML AND MANUAL SCREENING

All the studies obtained from manual and automatic search were imported to EPPI‐Reviewer 4, a software facilitating management of references, identification and removal of duplicates, and screening of studies at both title and abstract and full‐text stages. The title and abstract screening was done using the ML functionality available in EPPI in order to make the process more efficient. The model built using this functionality learns from the set of manually screened studies to guess the inclusion criteria and reorganises the list of studies based on the likelihood of being included. Using a training dataset consisting of 3300 manually screened studies, the algorithm ranked studies by prioritising them in order of likelihood of inclusion and excluding more than 1500 studies in a row. Based on this likelihood, the studies are grouped into 10 groups (Figure A6), with the set of excluded studies classified in the 0–10% range.

Figure A6.

Order of relevance of studies at the title and abstract screening stage. Source: Authors' own calculation

This systematic map includes studies that are extremely diverse; hence ML may not be accurate. To undertake a more nuanced approach and to account for this diversity in studies we conducted an additional manual screening within the groups. Since this manual screening gave us only 5% includes from the set of studies within 10–40% range, those were not considered in the subsequent screening process. Once we had a set of studies that were included in the title and abstract screening stage, this was followed by full‐text screening. Here, we screened studies that had more than a 50% likelihood of inclusion based on the ML model. A random check on 20% of studies belonging to the 40–49% group provided a very limited number of relevant studies that could be included, due to which the lower likelihood studies were not screened.

APPENDIX G. SR APPRAISAL TOOL AND SUMMARY

G.1. Summary of the SRs

Yan et al. (2017). Utility and potential of rapid epidemic intelligence from internet‐based sources

The study aimed to summarise internet‐based methods that use freely accessible and unstructured data for epidemic surveillance, exploring their timeliness and accuracy outcomes. The study is based on 84 articles published between 2006 and 2016 relating to internet‐based public health surveillance methods. These studies employ search queries, social media posts and approached derived from existing internet‐based systems for early epidemic alerts and real‐time monitoring. The primary methodology used for this review is the preferred reporting items for SR and meta‐analyses. Using this method, the authors can assess the benefits and challenges of a healthcare intervention through an evidence‐based minimum set of items. The study does not clearly demonstrate the inclusion criteria for the study designs, making it difficult for the readers to interpret the findings.

Fung et al. (2016). Ebola virus disease and social media: A systematic review

The study is an SR of the existing research pertinent to the Ebola virus and social media, especially to identify the research questions and methods used to collect and analyse social media. The study searched six databases for research articles relevant to Ebola and social media. Twelve articles were included in the main analysis: seven from Twitter and one including Weibo, one from Facebook, three from YouTube and one from Instagram and Flickr. All the studies were cross‐sectional. The study uses a standardised form to extract the data. A key challenge of the review is that the methods used by the review authors to analyse the findings of the included studies are not clearly defined.

Williamson et al. (2019). Satellite remote sensing in shark and ray ecology, conservation and management

The study aims to present a summary of the existing state of knowledge on the application of satellite remote sensing (SRS). SRS is seen as a key opportunity to analyse important environmental drivers in elasmobranch ecology and to aid management decisions for the conservation of declining population. The review included 71 papers and made use of several academic tools in order to conduct the bibliographic search. These included ISI Web of Science, Scopus and Google Scholar. One of the limitations of the review is that it examines a wide variety of studies but does not combine the results of these studies explicitly.

Mehta and Pandit (2018). Concurrence of big data analytics and healthcare: A systematic review

This SR of literature aims to determine the scope of big data analytics in the field of healthcare including its applications and challenges in its adoption in healthcare. The review employs a systematic search of articles on five major scientific databases: Science Direct, PubMed, Emerald, IEEE Xplore and Taylor & Francis. Two reviewers independently extracted information on the definition of big data analytics and the sources and applications of big data. A total of 58 articles were selected. The authors did not explicitly state the methods used to analyse the quality of included studies and also did not report results for each of the studies in the review.

De Souza et al. (2019). Data mining and machine learning to promote smart cities: A systematic review from 2000 to 2018

This study aimed to present an SR regarding data mining (DM) and machine learning (ML) approaches adopted in the promotion of smart cities. The study seeks to provide, from a literature review in journals belonging to the Web of Science and Scopus databases, the different DM and ML techniques used, as well as to present the sectors most engaged in the promotion of smart cities. A total of 39 studies were included on the map, which were further analysed to assess the most commonly used DM and ML techniques to promote smart cities. While the report describes individual results of the studies clearly, it does not combine the results of all the studies.

Charles‐Smith et al. (2015). Using social media for actionable disease surveillance and outbreak management: A systematic literature review

The studies in this review demonstrate how social media may be a valuable tool in improving the ability of public health professionals to detect disease outbreaks faster than by using traditional methods and to enhance outbreak response. A social media application was defined for this review as “an internet‐based application where people can communicate and share resources and information, and where users can activate and set their own profiles, have the ability to develop and update them constantly and have the opportunity to make such profiles totally or partially public and linked with other profiles in the network.” A total of 60 articles were selected for this SR, which addressed the two research questions: can social media be integrated into disease surveillance and can it be used to improve health outcomes?

Krenn et al. (2011). Use of global positioning systems (GPSs) to study physical activity and the environment: a systematic review

The aims of this SR were to determine the capability of GPS to collect high‐quality data on the location of activities in research on the relationship between physical activity and the environment. Studies were eligible for inclusion if they were undertaken on humans, used GPS to measure the locations where physical activity occurred and included analysis of the relationship between the characteristics of the environment and any form of physical activity behaviour (including leisure‐time physical activity, sport or active travel). The capability of GPS was expressed in terms of data quality, which in turn was defined as the proportion of GPS data lost in each study. The authors do not mention the proper risk of bias assessment of the included study.

Bennett and Smith (2017). Advances in using multitemporal night‐time lights satellite imagery to detect and estimate and monitor socioeconomic dynamics

This paper reviews progress in using the multitemporal Defense Meteorological Satellite Program‐Operational Linescan System, for which digital imagery was available from 1992 to 2013. It also reviews Visible Infraref Imaging Radiometer imagery to analyse urbanisation, economic and population dynamics across range of geographic scales. The review provides an overview of data corrections and processing for a comparison of multitemporal nightlight imagery. The review includes a total of 17 studies. The results of all the studies were not properly combined and reported in the review. A risk of bias assessment of the included study has also not been reported in the review.

Table G1.

SR appraisal

Questions	Yan et al. (2017)	Fung et al. (2016)	Williamson et al. (2019)	Mehta and Pandit (2018)	De Souza et al. (2019)	Charles‐Smith et al. (2015)	Krenn et al. (2011)	Benett and Smith (2017)
Section A: Methods used to identify, include and critically appraise studies
A.1 Were the criteria used for deciding which studies to include in the review reported?	Yes	Yes	Yes	Partially	Yes	Yes	Yes	Yes
A.2 Was the search for evidence reasonably comprehensive?	No	No	Yes	No	Partially	No	Partially	No
A.3 Does the review cover an appropriate period?	Not clear	Not clear	Not clear	Not clear	Not clear	Not clear	Not clear	Not clear
A.4 Was bias in the selection of articles avoided?	No	Partially	No	No	Partially	No	No	Not clear
A.5 Did the authors use appropriate criteria to assess the quality and risk of bias in analysing the studies that are included?	No	Not clear	Not clear	No	No	No	No	No
A.6 Overall, how much confidence do you have in the methods used to identify, include and critically appraise studies?	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)
Section B Methods used to analyse the findings
B.1 Were the characteristics and results of the included studies reliably reported?	Yes	No	No	Partially	No	No	No	Partially
B.2 Are the methods used by the review authors to analyse the findings of the included studies clear, including methods for calculating effect sizes (if applicable)?	No	Yes	No	No	Yes	No	No	No
B.3 Did the review describe the extent of heterogeneity?	No	No	No	No	Yes	Yes	No	No
B.4 Were the findings of the relevant studies combined (or not combined) appropriately relative to the primary question the review addresses and the available data?	Yes	Yes	No	No	Partially	No	No	Partially
B.5 Does the review report evidence appropriately?	No	No	No	Partially	No	No	Yes	Yes
B.6 Did the review examine the extent to which specific factors might explain differences in the results of the included studies?	No	No	No	No	No	No	No	No
B.7 Overall, how much confidence do you have in the methods used to analyse the findings relative to the primary question addressed in the review?	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)
Section C: Overall assessment of the reliability of the review
C.1 Are there any other aspects of the review not mentioned before that lead you to question the results?	No	No	No	No	No	No	No	No
C.2 Are there any mitigating factors that should be taken into account in determining the review's reliability?	Not clear	Not clear	Not clear	Not clear	Not clear	Not clear	Not clear	Can't tell
C.3 Based on the above assessments of the methods, how would you rate the reliability of the review?	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)	Low (limitations are important enough that the results of the review are not reliable)

APPENDIX H. ADDITIONAL TABLES AND FIGURES

Table H1.

Top 20 countries with maximum number of studies

Country	Total	IE	Measurement study	SRs
China	43	6	37
India	34	4	30
United States	31	3	28
Multicountry	31	3	25	3
Unidentified	14	1	11	2
Kenya	13	13
Brazil	11	3	8
Mexico	10	5	5
Ethiopia	9	1	8
Bangladesh	8	1	7
Indonesia	8	1	7
Rwanda	8	2	6
Uganda	8	2	6
Japan	7	7
Nigeria	7	7
Tanzania	7	1	6
Iran	6	6
Sri Lanka	6	6

Table H2.

Distribution of studies across the region

	East Asia and Pacific	Europe and Central Asia	Latin America and the Caribbean	Middle East and North Africa	North America	South Asia	Sub‐Saharan Africa
IE	12	5	18	3	5	11	14
Measurement study	92	69	42	29	49	82	112
SRs	7	6	5	6	6	6	6
Total	111	80	65	38	60	99	132

Table H3.

Studies using multiple sources of big data

Combination of big data	IEs	Measurement study	SRs	Grand total
Mobile phone CRD || Data from fixed sensors		1		1
Mobile phone CRD || Data from satellites		10		10
Citizen reporting and crowdsourced data || Data from fixed sensors || Data from satellites		1		1
Citizen reporting and crowdsourced data || Data from mobile sensors (tracking)		1		1
Citizen reporting and crowdsourced data || Data from satellites		1		1
Data from fixed sensors || Data from satellites	2	13		15
Data from mobile sensors (tracking) || Data from satellites		5		5
Data produced by public agencies || Data from fixed sensors || Data from satellites	1	1		2
Data produced by public agencies || Data from satellites	1	3		4
Data produced by public agencies || Data produced by businesses		1		1
Mobile data content || Mobile phone CRD		2		2
Mobile data content || Citizen reporting and crowdsourced data		2		2
Mobile data content || Data from mobile sensors (tracking)		1		1
Social networks || Citizen reporting and crowdsourced data		2		2
Social networks || Data from satellites		1		1
Social networks || Data produced by public agencies || Data from fixed sensors || Data from satellites		1		1
Social networks || Internet searches		4	1	5
Social networks || Internet searches || Mobile data content || Mobile phone CRD		1		1
Social networks || Mobile data content || Mobile phone CRD		1		1
Grand total	4	52	1	57

Table H4.

Big data source wise studies in fragile context

	Chemical or radio‐nuclear	Conflict or humanitarian crisis	Difficult terrain	Disease outbreaks or epidemics	Natural disaster	Others	Total (row)
Mobile phone CRD	1	6	0	11	10	0	28
Citizen reporting and crowdsourced data	0	7	1	0	1	0	9
Data from fixed sensors	0	5	7	0	1	1	14
Data from mobile sensors (tracking)	0	0	2	0	0	0	2
Data from satellites	0	25	18	4	10	3	60
Data produced by businesses	0	0	0	0	0	0	0
Data produced by public agencies	0	0	0	1	3	0	4
Internet searches	0	0	0	1	0	1	2
Mobile data content	0	1	0	5	0	0	6
Social networks	0	2	0	2	0	1	5
Total (column)	1	46	28	24	25	6

Table H5.

How studies have dealt with data quality and transparency

	Data collection methods	Data quality issues	Data representative of the population of interest	Construct validity explained	Results are generalisable	Data and code publicly available	Data, analysis and reporting challenges discussed
IE	47	22	29	43	26	11	6
Measurement study	43	3	7	21	4	8	11
SRs	1	0	0	0	0	0	1
Grand total	91	25	36	64	30	19	18

Rathinam, F. , Khatua, S. , Siddiqui Z., Malik M., Duggal P., Watson S., Vollenweider X. Using big data for evaluating development outcomes: a systematic map. Campbell Systematic Reviews. 2021;17:e1149. 10.1002/cl2.1149