Can incentives improve survey data quality in developing countries?: results from a field experiment in India

Summary.

We report results of an experiment designed to assess whether the payment of contingent incentives to respondents in Karnataka, India, impacts the quality of survey data. Of 2276 households sampled at the city block level, 934 were randomly assigned to receive a small one-time payment at the time of the survey, whereas the remaining households did not receive this incentive. We analyse the effects of incentives across a range of questions that are common in survey research in less developed countries. Our study suggests that incentives reduced unit non-response. Conditionally on participation, we also find little impact of incentives on a broad range of sociodemographic, behavioural and attitudinal questions. In contrast, we consistently find that households that received incentives reported substantially lower consumption and income levels and fewer assets. Given random assignment and very high response rates, the most plausible interpretation of this finding is that incentivizing respondents in this setting may increase their motivation to present themselves as more needy, whether to justify the current payment or to increase the chance of receiving resources in the future. Therefore, despite early indications that contingent incentives may raise response rates, the net effect on data quality must be carefully considered.

1. Introduction

Relatively little methodological research has been conducted on survey research in less developed countries (LDCs), and almost none—notwithstanding some notable exceptions (Mensch et al., 2003; Plummer, 2004; Plummer et al., 2004)—has used the experimental standards that are now common in more developed countries (MDCs).

Our primary focus in this paper is on one such underresearched area of survey research: whether to incentivize survey participation with a financial payment. As detailed below, incentivizing respondents is common in MDC settings, and an extensive body of methodological research has reviewed this practice (Medway and Tourangeau, 2015; Mercer et al., 2015; Pforr et al., 2015; Singer and Ye, 2013; Singer, Groves and Corning, 1999). The same cannot be said about LDC settings. In fact, the long record of research on incentives is extremely limited in its geographic scope: it is concentrated in English-speaking countries and hardly extends outside western Europe (Meuleman et al., 2017). As a result, major LDC surveys—among them the demographic and health surveys (DHSs), World Bank Living Standards Measurement Study and world values surveys—do not use incentives. Some smaller projects, including some with a high profile in the scholarly literature (e.g. the Malawi ‘Diffusion and ideation change project’), do use them. But, either way, the decision either to offer LDC respondents an incentive or, more frequently, to ask them to contribute their time altruistically is not rooted in any actual methodological research that is conducted in these or similar settings.

This disjuncture between the thin body of underlying research on incentives in LDCs and current LDC incentive practices is problematic because locally conceived meanings of ‘gifting’, exchange, ownership, credit and debt have historically varied across cultures (Davies et al., 2010; Hann, 2006; Peebles, 2010). We would therefore expect the effects of incentives on response behaviour to be somewhat different in LDCs in general than in western societies, even with the rapid pace of socio-economic and cultural change in many LDC settings. For, on the one hand, on-going socio-economic and cultural change has made contemporary LDC respondents more like their MDC counterparts: they are increasingly urban and educated, increasingly functioning in a ‘monetized’ setting that is part of a transnational culture of consumption and increasingly living lives that are both more individualistic and more regulated by modern state institutions (Illouz, 2007; Timmermans and Epstein, 2010). Yet, on the other hand, aspects of the pre-existing culture of economic exchange may continue to influence contemporary behaviour. These or similar concerns have been voiced by others who are worried about what Pforr et al. (2015) referred to as the ‘cross-national transferability of incentive effects’ (Blom, 2012; Couper and De Leeuw, 2003; Meuleman et al., 2017; Singer, Groves and Corning, 1999).

To extend the geographic scope of research on incentives in survey research, we present results of a field experiment that was conducted in two cities in Karnataka State, India. Alongside this goal, we also seek to broaden the parameters within which discussions of incentives and their effects are framed. The existing literature on incentives’ effects in MDCs primarily reflects researchers’ concerns about low response rates: it is almost wholly focused on how unit or item non-response changes as researchers modify the value, timing or type of incentive across different modes of data collection (Mercer et al., 2015; Singer and Ye, 2013). We show below that low response rates are rarely a concern in contemporary LDCs. We also argue that, as much as high response rates are a standard indicator of high quality data, they may also lure researchers into a false sense of security about the quality of individual, item-specific responses (Groves et al., 2006). Our main emphasis, therefore, is on understanding whether incentives affect the content of survey responses conditional on survey participation and, if so, on what type of questions these effects are most visible. We think that this is particularly important in poor population settings where a survey is often thought to be a precursor to the distribution of development resources (Swidler and Watkins, 2017; Watkins, 1995; Weinreb, 2006), making respondents willing to distort responses even though they have agreed to participate in the research project.

Our key empirical goal in this paper is therefore to look at how incentives affect survey responses across an array of domains that are frequently covered in LDC research: household demographic characteristics, income and wealth, and individual social and political attitudes. Within each of these domains there are more or less sensitive questions, allowing us to identify whether the effects of incentives on data quality are associated more with particular domains or a given question’s sensitivity.

Overall, our results suggest that incentivizing respondents has no discernable influence on responses to questions about household characteristics and social and political attitudes, including sensitive questions for which we would expect to see signs of social desirability bias. In sharp contrast with these findings, however, we find a very substantial and consistent difference in how households that receive payments report their economic levels in comparison with households that do not receive payments. Respondents who received incentives systematically presented themselves as poorer across a range of indicators that could not easily be verified by interviewers during the survey, even as they appeared no more impoverished on items that could be verified. The pattern and consistency of these differences, despite our inability to know with certitude whether treatment or control group respondents were lying, points towards the incentives shifting respondent self-presentation.

2. Empirical evidence on incentives

A long-standing literature, beginning with Shuttleworth (1931), has highlighted the effectiveness of incentives in raising response rates in MDCs (Singer et al., 2000; Wenemark et al., 2010), especially as resistance to survey participation increases (Singer et al., 1998). Positive effects have been found across different modes of data collection, including face-to-face and mail surveys (Church, 1993; Jackle and Lynn, 2008; Singer, Van Hoewyk and Gebler, 1999; Willimack et al., 1995). Finally, the use of incentives has also increased with the growing importance and cost of panel surveys, since there has long been evidence that incentives may have a positive effect on retention (Sudman and Ferber, 1974; Zagorsky and Rhoton, 2008).

In contrast with the established influence of incentives on survey participation in MDCs, there is far less evidence about their effects on the quality of survey responses in general conditional on survey participation. By quality, we refer to the accuracy of data, which in contemporary typologies fall under response ‘bias’ (Statistics Canada, 2002), response ‘error’ (Organisation for Economic Co-operation and Development, 2002) and a combination of ‘comparability’ and ‘accuracy’ effects (Eurostat, 2003). As noted above, high response rates may lure researchers into a false sense of security about the above forms of response quality (Groves et al., 2006), while possibly neglecting other categories of quality such as relevance, timeliness, accessibility and clarity (Eurostat, 2003)

One important but early review indicated that incentives were associated with more complete and accurate responses (Sudman and Ferber, 1974). Other studies suggest that incentives had no negative effect on data quality (Singer et al., 1998; Willimack et al., 1995). More recent studies show that item non-response rates are lower among those receiving incentives (Singer et al., 2000), though clearly this says nothing about the accuracy of the actual survey responses. Medway (2012) also found thatprepaid incentives have little effect on response order or responses to open-ended questions and that, in a mail survey, incentives significantly increased reporting of highly undesirable attitudes and behaviours but had no influence on responses to non-sensitive items.

A cautious interpretation of these findings—all from MDCs—is that incentives provided in person or in advance appear to raise overall response rates with little or no indication of a decline in data quality. In fact, where there is a data quality effect, it appears to be positive.

There is no equivalent scholarly literature on the effects of incentives on data collection in LDCs. The principal reason for this lack of concern, in our view, is the extremely high levels of survey response rates that are typically obtained in LDCs relative to those currently obtained in more developed settings. In the latter, despite some notable exceptions like the Current Population Survey of the United States—where household response rates are near 90% (US Census Bureau, 2006)—response rates in general have been declining in recent decades and now typically fall below 75% (Brick and Williams, 2012). For example, the response rate in the 2014 US General Social Survey was 69.2% (Smith et al., 2016), the Canadian General Social Survey response rates have been averaging 60–65% in recent years (Statistics Canada, 2013), and the baseline response rates for the recent early baby boomer cohort of the US Health and Retirement Study were 68.7% for individuals and 71.4% for households (Weir, 2011). Likewise, in the UK’s Health Survey of England (2013), the household response rate was 64% (Craig and Mindell, 2014) and, in the British Social Attitudes Survey (2015), the response rate was 51% (Swales, 2016). Although these response rates in themselves dwarf those found in commercial research studies (Baruch, 1999; Fricker and Schonlau, 2002), or even US-based studies by the Pew Research Center (Kohut et al., 2012), they have raised concern that ‘… the validity of social science studies and their conclusions is … at stake’ (Committee on National Statistics (2013), page 7).

Response rates in developing countries are another matter entirely. For example, in nationally representative DHSs fielded across sub-Saharan Africa and Asia—DHSs are often seen as the gold standard in LDC data collection—household response rates rarely fall below 95% (ICF International, 2016). India falls squarely within this high range. Across three waves of the DHS that was fielded in India in 1992–1993, 1998–1999 and 2005–2006, household response rates varied between 97.5% and 99.0% (outright refusal occurred in 0.3–0.8% of cases, and households with no competent respondent constituted a mere 0.3–1.7% of cases) (Mishra et al., 2006; Rutstein and Bicego, 1990).

These very high response rates in LDCs have eliminated the key motive underlying research on the factors that contribute to non-response, which in MDCs includes research on incentives, because non-response is simply not a major problem in LDCs. The relevant literature is therefore a little different. The first major category is anthropological and focuses on lone researchers attempting to establish themselves in communities (Agar, 1980; Barley, 1983). Given the relatively wealthy status of western researchers in poor LDC settings, the primary way that they establish themselves is through participation in local exchange networks. This begins with researchers directing resources towards informants and their families. As they become more deeply embedded in local exchange networks, and more trusted, higher quality information—requiring more effort and more trust—is directed back at the researchers. Although this is not quite the same as a researcher–respondent relationship in a survey—trust is not built in a day in ethnographic research—it provides some basis for believing that incentives can increase response quality in LDC surveys, also.

A small secondary source of literature on the effect of incentives in LDC settings is non-experimental data discussed by LDC survey researchers, especially those who are involved in longitudinal projects. These have echoed the principal arguments that are associated with MDC researchers, asserting that incentives increase retention and respondents’ motivation to provide more valid responses (Bignami-Van Assche et al., 2003; Weinreb et al., 1998).

3. Conceptualizing the influence of incentives on survey respondents

The empirical literature from MDCs provides one basis for predicting the influence of incentives on response behaviour in LDCs, but there are reasons to be wary of transposing data collection practices across cultures. This can be seen by examining the theoretical claims underlying arguments about the supposed influence of incentives.

Survey methodologists have long been aware of how changes in the surrounding context or the internal dynamics of the interviewer–respondent interaction can alter response behaviour. This recognition builds on a long chain of research on the data effects of ‘role-independent’ interviewer characteristics (Garbarski et al., 2011; Hatchett and Schuman, 1975; Maynard et al., 2010; Schuman and Converse, 1971), the presence of a third party (Aquilino et al., 2009; Smith, 1997; Weinreb and Sana, 2010), more conversational styles of interviewing (Beatty, 1997; Schober and Conrad, 1997; Schober et al., 2004) and even the physical setting of the interview (Herzog, 2005).

Like these factors, incentives can also affect the interactional context by operating along several established routes. One is in traditional terms of exchange. As enumerated in leverage salience theory (Groves et al., 2000), an array of factors determine whether an individual will participate in a study and, by extension, the quality of that participation. An incentive is a type of narrowly focused economic exchange geared towards tipping the balance in favour of participation (Datta et al., 2001). In classical social theory, it is a type of gift that is used instrumentally to establish, signal or cement a social relationship (Darr, 2003; Hann, 2006; Mauss, 2002).

This exchange mechanism gives rise to the expectation that, as material resources flow in one direction, consent to participate in the study should flow in the other. By extension, incentives may also trigger more honesty and effort—more optimal than merely satisficing answers—if respondents who have received an incentive feel an obligation to repay the ‘gift’ in kind. In other words, the incentive attempts to accelerate the normal pace of trust building in a new social relationship between an interviewer and respondent who, until that moment, had never met. By providing an incentive, the interviewer attempts to qualify as someone who is worthy of receiving closely guarded information of the type that is usually withheld from ‘strangers’ (Weinreb, 2006).

Notwithstanding these potentially positive effects, there are reasons to expect that incentives might also harm the quality of survey data. First, even if incentives raise respondents’ motivation to participate in a survey, they may simultaneously reduce data quality where, because of their closer entanglement with interviewers, respondents provide answers that they believe will please interviewers. We can imagine how the motivation to respond with this type of social desirability bias intensifies when data are collected through face-to-face interviews (Presser and Stinson, 1998; Tourangeau and Smith, 1996). We can also imagine this being particularly problematic in LDCs, since the relative dearth of resources makes respondents more motivated to provide what they see as the ‘right’ answer, i.e. one that conforms to conventional wisdom about researchers’ goals: to document different types of health behaviour, poverty and need, to capture grey or black market activities, etc. These can be seen as an antecedent to the flow of development goods or assistance (Swidler and Watkins, 2017; Watkins, 1995).

A second concern about the effects of incentivizing respondents arises from the literature on intrinsic and extrinsic motivation. Social psychologists, using laboratory experiments, have long known that extrinsic rewards can reduce individual motivation for certain activities (Deci, 1971; Kruglanski et al., 1971). Behavioural economists argue that payments may reduce the gains that individuals receive from altruistic behaviour—this has been termed the ‘crowding-out’ effect (Frey and Oberholzer-Gee, 1997)—since it modifies the cognitive frame that is used to define a given behaviour, shifting it towards more individualistic material gain. These arguments might also lead us to expect that incentives will reduce the accuracy of survey responses.

Compounding these issues in LDC contexts is the potentially clientilist context of making payments to respondents by a large-scale project operating in a context of severe resource constraints. This is in part because interviewers in LDCs—unlike their MDC counterparts—have long been more educated and upwardly mobile than their respondents (Mitchell, 1993; Stycos, 1993; Verrall, 1987). The social distance which arises from these educational gaps makes it more difficult for interviewers to explain their work to locals (Bulmer (1993), page 211)—i.e. what interviewing is and why they are doing it. Since they often represent a government or non-governmental organization intent on ‘development’, interviewers who provide an incentive may unwittingly create the expectation that the same sponsoring organization may provide more resources in the future. Field researchers in LDCs have long asserted that these factors lead respondents to present themselves in ways that make them seem more deserving of such future assistance or gifts (Watkins, 1995), or that respondents sometimes try to forge connections with researchers and their employees as individuals (Barley, 1983), treating interviewers as potential patrons on whom—assuming some future meeting—they might call with a request. In this context, an incentive creates an additional motivation to present oneself as deserving of further assistance. We would therefore expect to see much more of an effect of incentives on questions that establish one’s economic position rather than, say, those which are directed at political attitudes or general social and household characteristics, unless there is a clear social desirability bias in one of these.

4. Research design and methods

4.1. Data and experimental design

To identify the influence of incentives on survey data quality, we use data from an experiment that was conducted in two urban centres in Karnataka State, India. Our analysis distinguishes questions about a range of household economic characteristics—where we expect the economic self-presentation motive to be strongest—from more general demographic questions, questions about the extent of shared decision making across various domains and questions that tap into confidence in the political system.

The incentives experiment was integrated into a larger household survey: part of the 2011 ‘Urban property ownership records’ (UPOR) project, administered in approximately 12000 households over a 12-month period. As part of the UPOR project, the World Bank, in partnership with the National Council of Applied Economic Research (New Delhi) and the Institute for Social and Economic Change (Bangalore), conducted a survey of 4000 households to assess the effect of the UPOR project in the south-west Indian state of Karnataka. Given the project’s emphasis on property ownership records, only households that owned their houses were included. On the basis of data from the 2011 Census of India, between 50% and 58% of households in the cities included are property owners, so it is important to note that nearly half of households—the poorer half—were excluded from the study.

A priori it is not clear to us whether this exclusion of poorer families would magnify or reduce the effects of incentives in LDCs. If incentives do influence participation more strongly among poorer households—there is evidence for the USA showing that poorer households are more likely to respond to incentives (Martin and Winters, 2001; Singer and Kulka, 2002)—then their exclusion may reduce the observed influence of incentives. In contrast, response rates are high in general. Moreover, the different meaning, magnitude and consequences of poverty in India relative to western societies make it difficult to extrapolate from findings on the effect of exclusion from one setting to another. Unintentionally, because our sample includes only relatively better-off households, the possible effect of sample selection bias on this more homogeneous sample is therefore likely to be muted.

Our experiment focused on two of the four cities that were included in the UPOR project survey in Karnataka: Davanagare and Gulberga. Both cities are small to middle sized, with populations under half a million. The sampling and treatment assignment strategy for the survey and accompanying experiment began with a listing of blocks within each city. A sample of 100 blocks was selected from within each of the two cities, with the block selection probability proportional to the number of households. Within each block, 12 households were then randomly selected for the main sample. In addition, a total of 50 replacement blocks—25 from each city—were chosen according to the same probability proportional to sample criteria, though these were not initially assigned to treatment or control categories. The selection of replacement blocks in the field was based on their physical proximity to those blocks from where replacement households were needed. The logic of this approach, while reducing costs, was also intended to maintain as much similarity between non-respondent and replacement households to reduce the threat of contamination between treatment and control groups. By the end of data collection, a total of 34 households had been sampled from 31 replacement blocks. Thus, the final sample included 231 blocks (main and replacement blocks combined) and a total of 2333 households.

The sampled blocks were randomly assigned to treatment (incentives) or control (no incentives) groups. Block level assignment was necessary to avoid treatment contamination as well as due to the exigencies of the larger survey into which the experiment was incorporated. An overall larger share of households in the sample (59%) was assigned to the control group and did not receive an incentive because of financial constraints of the incentives experiment.

During data cleaning, 57 cases were dropped because of unit non-response or unresolvable identification problems and missing information on core variables. The final analytic sample therefore included 2276 households: 934 in the treatment group and 1342 in the control group. Of these, 98.5% were in the original sample, and our analytic results are substantively identical whether we restrict our analysis to the original sample or include the extra 34 replacement households.

Interviewer training for the experiment stressed the importance of working with both types of households and treating them equally in all respects other than provision of the incentive. Protocols also made it impossible for interviewers to switch households and supervisors visited households to verify incentive payments and basic reporting accuracy. The incentive itself was valued at 250 rupees (about US $5 and roughly equivalent to the daily minimum wage in Karnataka for basic manual work such as domestic services) and it was offered at the opening introduction of the interview, conditionally on agreement to participate fully in the survey. A total of 11 respondents in the treatment group refused the incentive yet completed the survey.

Strictly speaking, ours is a conditional or ‘contingent’ incentive The interviewer declared his or her intention to give an incentive during the initial introductions, and then gave it at the end of the interview. The main respondent—86% were male and most were the household head—signed a receipt and was informed that supervisors might follow up to verify receipt of the incentive. This approach differs somewhat from normal practice in MDCs, which has shown that prepaid incentives tend to increase response rates more cost effectively than conditional incentives (Church, 1993; Singer and Ye, 2013; Singer, Van Hoewyk and Gebler, 1999). It also differs from the experiment’s original protocol which called for the incentive to be given at the start of the interview. The move away from the original protocol occurred during interviewer training after supervisors and interviewers voiced strong concerns about offering the incentive before the interview, urging payment to be made conditional on completing the interview. We acceded to their request, knowing that this modification may alter the influence of the incentive on responses, but also being a priori unclear about what the net direction of that effect might be. In at least one non-US setting, for example, researchers have suggested that prepaid incentives can cause distrust with some respondents or lead to public criticism of the survey (Börsch-Supan et al., 2013) In a three-country study of the USA, Canada and Australia, Mutti et al. (2013) showed that the effectiveness of prepaid incentives varies significantly by subpopulation. Likewise, our fieldworkers’ criticism of prepaid incentives echo those voiced in other settings. In the German wave of the 24-country ‘Programme for the international assessment of adult competencies’ (Zabal (2014), page 187), for example, a decision was also made to use conditional incentives.

4.2. Response rates

The survey achieved very high household response rates shown in Table 1: 99.9% in the treatment group and 96.0% in the control group. As noted above, these survey response rates are very similar to the rates that are obtained in the DHSs in general, and in the India DHS in particular. Consistent with the RR5 method of the American Association for Public Opinion Research and the DHS (Vaessen et al., 2005), response rates were calculated as the ratio of completed interviews to all eligible households in the sample. Although additional non-response options were included in the survey questionnaire for interviewers to report, all non-responses fell into one of only three categories: refusal (38%), no eligible person present (26%) and no one at home (36%).

Table 1.

Household response statistics by city and treatment category^†

	Combined results			Results for Davanagare		Results for Gulberga
	Overall	Treated	Control	Treated	Control	Treated	Control
Total count	2333	1114	1219	539	639	575	580
Non-response count	50	1	49	0	47	1	2
Response rate	0.979	0.999	0.960	1.000	0.926	0.998	0.997
F –statistic		14.47 (p = 0.000)		16.37 (p = 0.000)		0.20 (p = 0.656)

^†Tests account for the stratified split-panel design by using Stata complex survey data methods.

Despite the very high overall response rates across the two cities in our sample, there was one meaningful difference between them: 47 of the 50 non-respondent households were in Davanagare, and all of these were in the control group. In fact, out of the 50 non-respondent households across the two cities, only one was in a treatment block—it was one of the three non-respondent households in Gulberga. This difference in non-response between treatment and control households across the entire sample is highly significant $\left(p<0.001\right)$, suggesting that incentives effectively increased survey participation in the same way that they do in MDCs. The moderating factors here are that response rates were already very high even in the absence of incentives, and these results are driven entirely by differences in Davanagare, which is potentially a signal of some unknown difference in interviewing cultures between the two cities.

4.3. Model specifications and variables

Our primary goal is to evaluate the influence of an incentive $G$ on a range of reported behaviours and outcomes, using one of three models depending on the nature of the outcome variable. Our outcome measures include continuous or binary variables, as well as four variables that are nominal with multiple (five) categories. Our results focus on the effect of being in the incentive group $\left(G=1\right)$ versus not being in the incentive group $\left(G=0\right)$. All our models include a set of control variables to help to adjust for possible differences that might remain despite the randomization. These include dummies for gender of the head of household $\left(\text{male}\equiv 1\right)$ and city $\left(\text{Gulberga}\equiv 1\right)$, household religion $\left(\text{Hindu}\equiv 1\right)$ and caste (belonging to disadvantaged ‘other backward caste’ $\left(\text{OBC}\right)\equiv 1$), as well as the head’s age and household size. When the models are re-estimated without these controls our findings on the influence of incentives are qualitatively unchanged and the estimated effects rarely shift by more than 10%. Additional variables, such as reported education, are treated as outcomes whose reported values may be influenced by incentives and have thus been excluded from our main analyses.

Our various models identify the determinants of a function $F$ of $E\left(Y\right)$ (McCullagh and Nelder, 1989). The basic specification is of the form

$$F\left\{E\left(Y\right)\right\}=\alpha +\gamma G+\sum _{k=1}^K{\beta }_k{X}_{ik}\text{.}$$

In this context, our variable $Y$ is distributed either normally, in the case of continuous outcomes, or as a logistic distribution, in the case of binary outcomes (Hosmer and Lemeshow, 2000). In the case of nominal outcome variables, we use multinomial logistic regression models (Agresti, 2013). Across all models, the main effect, which is measured by $\gamma$, identifies the average treatment effect of the incentive variable, and the $\beta$-coefficients reflect the role of the $k$ control variables (Imbens and Wooldridge, 2009). Our main findings are consistent if we estimate the influence of incentives by using an indicator for whether each household received the incentive (treatment on the treated) rather than estimating the average treatment effect (which is available on request).

Linear regression is used for outcomes for numbers of people in the household (including by type), the household head’s reported years of education, the reported log-value of total income and subcomponents of income, the reported log-value of household expenditures and subcomponents of expenditures, and the reported log-value of assets (including by type).

Logistic regression is used on binary outcomes, which include whether the household head is Hindu, whether the household belongs to a socially disadvantaged OBC (not belonging is the reference group), whether specific household decisions are made by the wife or jointly between both spouses (the reference group is that decisions are made by the household head), whether the respondent reports having specific knowledge of the UPOR projects and their attitudes towards the project (the reference is not having heard and not having positive expectations), and whether the household owns a specific set of items (not reporting possession is the reference).

One additional set of variables involves respondents reporting their degree of confidence and trust in the government and elected officials. The possible response options ranged from 1, for the lowest level of confidence or trust, to 5 for the highest level. Despite their ordered nature, the null hypothesis of proportionality in an ordered logistic regression was rejected by the Brant log-L test $\left(p<0.001\right)$ (Brant, 1990). Instead, we use multinomial logistic regression to estimate the influence of incentives on all categories, with the reference being the middle level of confidence or trust. Multinomial logistic regression estimates separate coefficients for the influence of incentives on being in any category relative to the reference. Despite the loss of efficiency by not using information on the ordering of the outcome, the multinomial model has the added advantage of a more flexible specification and provides interesting results.

The data collection strategy involved sampling blocks from two cities in a two-stage split-panel design. To account for the design of the sample, we employ Stata’s complex survey analysis methods (svy) (StataCorp, 2013). These methods enable us to account for the stratified nature of the sample and they also avoid overly optimistic standard error estimates that are introduced through sampling of households from within clusters. Where models are estimated on a subset of our sample, the method also makes it possible to avoid miscalculations of the coefficient standard errors (Cochran, 2007; StataCorp, 2013). Model test statistics are based on the F-adjusted mean residual test (Archer and Lemeshow, 2006).

5. Analysis and results

Our analyses focus on the effects of incentives on actual survey responses across five domains: demographic and social characteristics; reported sharing of decision making within households; confidence in the political system; knowledge and attitudes regarding the public programme underlying the survey; questions regarding household income, consumption and a broad set of household assets.

5.1. Demographic and social characteristics

In Table 2, we present a comparison of reported demographic and social statistics of households across treatment and control groups. A priori, we have few expectations on why incentives would alter reported demographic responses and this turns out to be so. Reported household size is very similar across the two groups, with households in the treatment group larger by 0.158 people, but this difference is not significant. Respondents in treatment households report somewhat lower education than in non-incentivized households, but this contrast is insignificant also. Nor are there statistically significant differences between the treatment and control groups on questions pertaining to social and cultural identity. The log-odds of being Hindu or in the OBC group are higher among the treatment group, but not significantly so. Overall, we find that incentives have no notable effect on either the basic demographic or the sociocultural identities that were reported in these data.

Table 2.

Linear and logistic regression coefficient estimates of the effect of incentives on household reported demographic and social characteristics^†

	Linear regression results		Logistic regression results
	Head of household size	Head’s education	Hindu	OBC group
Incentive	0.158 (0.152)	−0.394 (0.437)	0.248 (0.270)	0.170 (0.133)
N		2276
F-statistic	20.56	87.50	1.08	2.23
R 2	0.0622	0.121

^†Tests account for the two-stage split-panel design by using Stata complex survey data analysis methods. Control variables are not shown but include male head, head’s age and city. Standard errors are in parentheses.

In Table 3, we test the influence of incentives for a more disaggregated analysis of reported household demographic composition. Given the literature on gender preferences in India—in particular, sex-selective abortion, distorted sex ratios at birth and the financial burdens that are associated with dowries (Dyson, 2012; Gupta, 1987)—we are particularly attentive to reported differences in households’ gender composition, even though son preference is less endemic in this particular region of India. The results are consistent with this expectation. Incentives have no effect on any other reported demographic characteristic, including the number of children or adults of either sex and the number of elderly.

Table 3.

Linear regression coefficient estimates of the effect of incentives on reported household composition^†

	Results for boys	Results for girls	Results for men	Results for women	Results for elderly
	b se(b)	b se(b)	b se(b)	b se(b)	b se(b)
Incentive	0.045 (0.050)	0.076 (0.052)	0.004 (0.057)	0.003 (0.049)	0.012 (0.035)
N	2276
F-statistic	9.48	7.97	22.19	14.12	0.12
R 2	0.023	0.017	0.045	0.038	0.00

^†Tests account for the two-stage split-panel design by using Stata complex survey data analysis methods. Control variables are not shown but include male head, head’s age and city. Standard errors are in parentheses.

5.2. Political attitudes

The next series of analyses presents regressions on three separate outcome variables, each of which reflects a particular type of attitude or confidence that was reported by the household head towards current political members or the political system. These variables are coded from 1 to 5, with 5 indicating the highest level of ease or freedom on each variable. The questions cover somewhat distinct dimensions: how easy is it to hold current elected officials accountable for the duties that they are supposed to perform?, how transparent is the current selection of development schemes?, and how free do respondents feel they are to vote in elections for their preferred candidate under the current government? We treat the outcome categories as nominal and use multinomial logistic regression, with the middle (value of 3) choice as the reference category in the three models. The results are shown in Table 4.

Table 4.

Multinomial logistic regression model coefficients showing the influence of incentives on reported confidence in elected government members and freedom^†

Scale	Results for accountable	Results for transparency	Results for free vote‡
	b se(b)	b se(b)	b se(b)
1 (least free, easy or transparent)	0.995§ (0.263)	0.937§ (0.262)	1.032§ (0.260)
2	−0.036 (0.181)	0.031 (0.183)	— —
3 (reference category)
4	0.189 (0.119)	−0.040 (0.129)	— —
5 (most free, easy or transparent)	1.140* (0.490)	−0.503 (0.318)	0.323§§ (0.119)
N		2271
F-statistic	12.54	13.41	15.86

^†Tests account for the two-stage split-panel design by using Stata complex survey data analysis methods. Control variables are not shown but include male head, head’s age, city, Hindu household and household size. Standard errors are in parentheses.

^‡Because of limited cases, free-vote categories 2 and 4 collapsed into the middle category, 3.

^§p < 0.001.

^§§p < 0.01.

^*p < 0.05.

The findings indicate that, when individuals receive an incentive, they appear more willing to express views that are both more positive and more negative regarding confidence in elected officials. In all three of the outcomes, the significant coefficients on value 1 relative to 3 indicate an increasing willingness to claim lower freedom and transparency associated with elected officials. In all three outcomes, we also find that incentives increase responses espousing more confidence in elected officials, although one is insignificant and points in the opposite direction (‘transparency’).

The pattern of results here indicates that incentives do not appear to push respondents systematically to respond in one particular direction, i.e. to be more or less positive about elected officials. Instead, incentives shift responses away from a more ‘mild response style’ towards a more ‘extreme responses style’ (Baumgartner and Steenkamp, 2001; Van Vaerenbergh and Thomas, 2013). Given the generally higher levels of acquiescence bias in Asian societies (Wang et al., 2008) and societies with low levels of individualism in general (Johnson et al., 2005), we suspect that this pattern more accurately captures actual variability in respondents’ attitudes, i.e. it is closer to an optimal rather than a satisficing response.

5.3. Household decision making

The survey included a module on how decisions are reportedly shared within the household across five developmentally important domains: food purchases; child education; child health; number of children desired; household savings. To ensure consistent gender effects, we focus here solely on households with male household head respondents, although our results are substantively identical when women are included along with a control for gender. In each case, the outcome variable equals ‘1’ if the decision is made either jointly or by wives and ‘0’ if the decision is made by the male head.

The results that are presented in Table 5 offer no evidence that incentives change whether decisions are reported to be made by household heads alone, or jointly with wives. This is consistent across all five domains.

Table 5.

Logistic regression coefficient estimates of the influence of incentives on reported shared decision making for male heads across various topics (reference is that the head decides alone)^†

	Results for food purchases	Results for child education	Results for child health	Results for number of children desired	Results for household savings
	b se(b)	b se(b)	b se(b)	b se(b)	b se(b)
Incentive	−0.130 (0.153)	−0.053 (0.129)	−0.201 (0.131)	0.039 (0.222)	−0.018 (0.138)
N	2183	1864	1861	1938	1862
F-statistic	7.313	0.538	2.384	1.976	1.877

^†Tests account for the two-stage split-panel design by using Stata complex survey data analysis methods. Control variables are not shown but include male head, head’s age, city, Hindu household and household size. Standard errors are in parentheses.

5.4. Knowledge and attitude towards the ‘Urban property ownership records’

The survey also included a series of questions that were aimed at gauging respondents’ prior knowledge of the UPOR project as well as their attitudes towards the changes that the project was meant to enact. These questions enable us to evaluate both whether the promise of incentives might lead respondents to report more familiarity with the programme and whether it might also lead them to speak more favourably of the actual programme.

The results shown in Table 6 clearly show that the promise of payment bears no influence on whether people report having heard of the UPOR project. Furthermore, we examined whether the promised incentive led to people being more positive about the potential effect of the project across a series of relevant outcomes. This includes whether the programme would help to facilitate future transfers of property, reduce the likelihood of conflict over land property with other individuals, reduce the potential private risk that someone would take their property and reduce the risk that government would take their property. Respondents were asked in all cases whether they believed that the programme would be successful at achieving each of these aims and then, if yes, whether they saw this as a good outcome. We considered responses that saw the programmes’ aim as achievable, and that this was a favourable outcome, as expressions of positive perspectives on the programme (outcome equals 1). The results show that, regardless of which outcome is examined, incentives had no effect on the degree to which positive attitudes were expressed about any one of the questions.

Table 6.

Logistic regression coefficient estimates of the influence of incentives on reported knowledge and attitudes towards the UPOR project^†

	Results for have heard about UPOR	Results for will UPOR facilitate property transfers?	Results for will UPOR reduce property conflict?	Results for will UPOR reduce risk private persons take property?	Results for will UPOR reduce risk government takes property?
	b se(b)	b se(b)	b se(b)	b se(b)	b se(b)
Incentive	−0.377 (0.258)	−0.106 (0.350)	0.0963 (0.249)	0.232 (0.241)	0.115 (0.175)
N	2276
F-statistic	14.43	1.605	1.240	0.557	0.637

^†Tests account for the two-stage split-panel design by using Stata complex survey data analysis methods. Control variables are not shown but include male head, head’s age, city, Hindu household and household size. Standard errors are in parentheses.

5.5. Income, consumption and assets

Earlier we raised the possibility that an incentive might alter how respondents self-present to make themselves appear more deserving of the promised incentive, or more deserving of other payments in the future. Gary Becker had first argued about a half century earlier that dishonesty should increase with higher pay-offs (Becker, 1968). An emerging literature on dishonesty emphasizes how a sizable share of people may be dishonest, depending on the context, if the gains are seen as sufficient (Jacobsen et al., 2017; Rosenbaum et al., 2014). Consistent with these assertions, it has recently been shown that when people are monitored or think that they are being monitored they are less likely to be dishonest (Rustagi, 2010). This extends to the level of lighting or visibility. Zhong et al. (2010), page 314, have shown that, even where induced by nothing more than sunglasses, darkness ‘licensed self-interested and cheating behavior’.

We test this hypothesis more directly with data on reported household income, consumption and assets. By dividing assets into those that are easily observed by the interviewer and those that cannot be observed, we can indirectly assess the magnitude of biased responses, since we assume that it is more difficult to lie about an object that is in the interviewer’s line of sight or is easily verifiable.

As above, though our discussion focuses solely on the comparison between households in treatment and control clusters, all models continue to include correction for clustering at the community level as well as controls for caste, religion, age, city and household size.

Table 7 shows the influence of incentives on the value of reported income overall and its main components. We include only households that report at least some source of income for each specific category, and all income and expenditure variables are logged. The results indicate that households in the treatment group—promised an incentive—report 11.6% lower monthly income. This is a substantial difference. The columns to the right in Table 7 shed some light on the major components of income that generate the gap between the reported income for treatment and control groups. The largest gaps are for reported pension income and reported self-employment income, with reports by treatment households lower by 28% and 14%. The effects of incentives on reported salary and wages are also negative, but these are smaller and not statistically significant.

Table 7.

Linear regression estimates of the influence of incentive on reported logged value of total income and income by source^†

	Results for total income	Results for salary	Results for self-employment	Results for pension	Results for wages
	b se(b)	b se(b)	b se(b)	b se(b)	b se(b)
Incentive	−0.116‡ (0.0419)	−0.0966 (0.0731)	−0.143‡ (0.0440)	−0.277§ (0.140)	−0.0301 (0.0375)
N	2276	772	1053	636	822
F-statistic	92.40	8.486	22.71	29.84	26.27
R 2	0.230	0.067	0.138	0.243	0.175

^†Tests account for the two-stage split-panel design by using Stata complex survey data analysis methods. Control variables are not shown but include male head, head’s age, city, Hindu household and household size. Standard errors are in parentheses.

^‡p < 0.01.

^§p < 0.05.

Reported values of expenditures or consumption on a broad range of food and non-food items are also lower in the treatment group than in the control group—6% lower and highly significant—as shown in Table 8. As noted below, the fact that this difference in reported consumption is not as large as the difference in reported income is consistent with the oft noted sensitivity of income reporting to non-sampling error (Deaton, 1992).

Table 8.

Linear regression estimates of the influence of incentives on reported logged value of total consumption, basics and luxuries consumption, and income and consumption gap (not logged)^†

	Results for total consumption	Results for basics consumption	Results for luxuries consumption	Results for income minus consumption
	b se(b)	b se(b)	b se(b)	b se(b)
Incentive	−0.057§ (0.021)	−0.012 (0.016)	−0.115‡ (0.034)	−1797.6§§ (768.5)
N	2276
F-statistic	120.22	183.42	40.43	21.23
R 2	0.400	0.591	0.154	0.140

^†Tests account for the two-stage split-panel design by using Stata complex survey data analysis methods. Control variables are not shown but include male head, head’s age, city, Hindu household and household size. Standard errors are in parentheses.

^‡p < 0.001.

^§p < 0.01.

^§§p < 0.05.

Further perspective is gained by separating the reported consumption into several categories. We divide our group into items that signal wealth and luxury versus those that are more clearly basic and essential. The basics on the list include expenditures on grains, vegetables, beverages and oil. The luxury items include meat, fuel, eating out, domestic workers and cable television. The contrast between the two categories is very strong and supports an interpretation that focuses on the types of economic self-presentation thatwere described above. Although essentials as defined here comprise more than half of overall expenditures included in our analysis, the effect of incentives is far stronger and more negative on luxury goods. Specifically, respondents who were promised an incentive reported 12% less expenditures on luxury items than those in the control group—a highly significant effect—but there was no difference in reported consumption of basic items.

The last column in Table 8 addresses the same issue from a slightly different angle. Here, the outcome is the difference between income and consumption. Although we do not necessarily expect them to be equal for any single household, a greater difference across types of interviews signals a type of ‘incoherence within a data set’, which is a standard indicator of data quality problems (Organisation of Economic Co-operation and Development, 2002). Within economics, in particular, the common tendency for income to be more strongly underreported, relative to expenditures, means that this incoherence leads to a common, mistaken, finding of dis-savings for households (Deaton, 1997; Guénard and Mesplé Somps, 2010). In our case, the estimates of both income and expenditures are based on a partial, incomplete, list. Nonetheless, it is informative that, in households that received an incentive, there is greater incoherence—i.e. a larger gap between income and expenditures—among those dimensions captured in the survey.

The next series of analyses—presented in Table 9—shows the influence of incentives on reporting the value of reported durable goods and assets owned by the household. Respondents report both on the existence of the assets and their current estimated values. In these models, as before, values are logged and the coefficients in Table 9 demonstrate very clear and powerful influences of incentives. Overall, incentives reduce the summed value of the assets by 15%. Breaking these assets into categories we find substantial variation: a 30% reduction in the value of reported appliances; a 19% reduction in recreation; a 17% reduction in furniture; a 16% reduction in jewelry. Each of these is statistically significant at the 5% level. The only category that is not significant, though still negative, is savings. It is also worth noting that the results are substantively unchanged if households reporting zero value on any one of the assets are removed from the analysis.

Table 9.

Linear regression coefficient estimates of the influence of incentives on reported logged value of total assets and separately by type of asset^†

	Results for total	Results for furniture	Results for jewelry	Results for cooking or appliances	Results for recreation	Results for savings
	b se(b)	b se(b)	b se(b)	b se(b)	b se(b)	b se(b)
Incentive	−0.151§ (0.058)	−0.170§§ (0.070)	−0.158§§ (0.074)	−0.303‡ (0.087)	−0.186‡ (0.052)	−0.073 (0.099)
N	2276	2276	2276	2276	2276	2252
F-statistic	18.66	10.59	10.30	8.49	8.14	8.82
R 2	0.067	0.033	0.032	0.036	0.043	0.034

^†Tests account for the two-stage split-panel design by using Stata complex survey data analysis methods. Control variables are not shown but include male head, head’s age, city, Hindu household and household size. Standard errors are in parentheses.

^‡p < 0.001.

^§p < 0.01.

^§§p < 0.05.

The results in Table 9 bring us back to the ‘additional advantage’ of the asset variables that was mentioned above. Depending on the specific location of the interview, most assets appearing in Table 9 would not be directly visible to the interviewer. In contrast, in most cases the physical characteristics of a respondent’s house are directly visible. The most obvious is roofing, which can be seen without entering the household. Similarly, wall material and type of floor may in many cases be easily identified by interviewers, even if they do not enter the home. And reported access to toilets and piped water are also likely to be less subject to reporting bias because both require infrastructure that may not exist in that block or section of the city and this may be known by interviewers.

This difference in the visibility of different types of assets provides us with additional leverage for understanding response behaviour. The current literature on honesty and truth-telling has shown that, in addition to those who are unconditionally honest or dishonest, many people ‘exhibit heterogenous preferences for truthfulness’ (Rosenbaum et al., 2014).

In this context, questions about visible assets are akin to increased monitoring of respondents, making them less likely to lie about ownership. The results, which are reported in Table 10, are consistent with our expectations. All the household characteristics are coded such that a value of 1 refers to a higher status material or construction whereas a value 0 refers to a lower status. We can readily see that, in contrast with the strong negative effect of incentives on reported assets that are relatively difficult to verify (see Table 9), there is no effect of incentives whatsoever on any reported characteristic of these homes that is visible to interviewers, including reported types of materials used in floors, roofs and walls, and whether there are connected toilets and piped water.

Table 10.

Logistic regression coefficient estimates of the effect of incentives on visible household structure and characteristics^†

	Results for floor	Results for roof	Results for wall	Results for connected toilet	Results for piped water
	b se(b)	b se(b)	b se(b)	b se(b)	b se(b)
Incentive	0.365 (0.380)	−0.132 (0.215)	−0.227 (0.226)	0.0892 (0.149)	−0.242 (0.201)
N	2273	2271	2273	2270	2270
F-statistic	2.301	2.105	3.995	3.693	3.709

^†Tests account for the two-stage split-panel design by using Stata complex survey data analysis methods. Control variables are not shown but include male head, head’s age, city, Hindu household and household size. Standard errors are in parentheses.

5.6. Discussion

Several important results emerge from this experiment. First, the data suggest that the promise of incentives can significantly reduce levels of unit non-response, even in places with nearly zero levels. This is consistent with the established literature on MDC settings. Moreover, although this result is testable in only one of our two study cities, Davanagare, it does suggest that as LDC societies continue to urbanize, and the high response rates of the past become more difficult to achieve, incentives also may become a standard tool in LDC survey researchers’ toolkits.

Second, the promise of payment has some effect on response quality conditional on participation in a survey, though here the results are more mixed and complex. On one hand, we find little or no systematic effect on survey reports across most domains in our experiment. This includes basic demographics as well as questions that tap important and potentially sensitive areas, such as how decisions are made in households, and knowledge regarding the project underlying the survey. Across a wide range of domains and questions within those domains, the consistent impression that is given by the data is that an incentive has no effect—positive or negative—on how people report on a range of possibly sensitive outcomes.

This non-result challenges assertions that incentives can be used to ‘buy’ more honesty through an exchange mechanism. If this was so, we would have expected to find a stronger and more consistent willingness to acknowledge being a member of an OBC or being less educated—we can indirectly infer the direction of bias for these more stigmatized responses. Although incentives potentially pushed responses to these questions towards this expected direction, the effects were not statistically significant. Likewise, if incentives could so easily trigger more accurate responses, we would also have expected to find respondents expressing lower levels of confidence in public officials and elections, and admitting higher income and expenditures. (The actual responses cannot be validated so we can only assume that lower numbers reflect more accurate responses. This assumption is supported by data from the World Value Survey (WVS). On-line tabulations (http://www.worldvaluessurvey.org/WVSOnline.jsp) of WVS data raise expectations that levels of confidence expressed about elected officials are likely to be overstated in India. The WVS collected data in India on related topics, asking respondents about confidence in a range of governmental and non-governmental institutions, including Parliament, courts, banks, environmental organizations and women’s organizations. WVS respondents in India reported much higher levels of confidence in all of these, and others, than levels reported in countries like Germany, the Netherlands, Singapore, Chile and the USA, all of which have much more trustworthy and transparent systems according to scores on Transparency International’s corruption perception index (http://www.transparency.org/cpi2015). The implication is that more accurate data from India will probably be oriented, on average, towards lower levels of confidence.) Here, also, actual empirical results provide only partial support for this hypothesis since what emerged, at least in a set of questions on attitudes towards elected officials and opinions about their effectiveness and honesty, is that respondents who were promised an incentive were more willing to express lower levels of confidence. But on a couple of variables they were also moderately more likely to express higher levels of confidence in officials. To the extent that a neutral response can be treated as a type of acquiescence to a safe response pattern, these results imply that incentives reduced acquiescence while pushing more people towards a negative view. This is likely to be a net positive contribution of incentives to data quality, but it is not a simple, one-dimensional effect.

The clearest negative effects of incentives on data quality can be found in our analysis of reported income, expenditures and assets, as presented in Tables 7–10. The values that are reported tend to be substantially lower in households that received an incentive, which is a valuable finding in itself. Not knowing what the true values are for these variables, we cannot know for sure whether incentives lead to overreporting or underreporting. However, four arguments lead us to conclude that an incentive encourages households to underreport economic status, at least in this urban Indian setting.

1. In general, evidence suggests that income and expenditures tend to be underreported in Indian survey data (Bakshi, 2008; Deaton and Kozel, 2005). Furthermore, earlier evidence argues that income is typically more prone to misreporting and underreporting than are expenditures (Deaton, 1997). This suggests that the level of correspondence between income and expenditures can be viewed as a marker of data consistency. The fact that the gap between income and expenditures is greater in treatment households suggests that incentives are leading to more underreporting of income in particular.

2. If the promise of an incentive is driving a desire to present oneself as needy and deserving of further assistance, we would expect to find that the effect is largest for reports on non-essential expenditures. This is exactly what we show in Table 8: there is little influence of incentives on essentials or basic commodities, but expenditures on luxury items are strongly affected by the incentive. They drive much of the results in terms of expenditures. The same can be said about the least visible asset categories such as appliances, jewelry and recreation (see Table 9).

3. Incentives have no influence on how visible indicators of wealth are reported (Table 10) despite the significant reduction in reported income, expenditures and non-visible assets.

4. Finally, the lack of any clear directional effect of incentives across a wide range of demographic, political and socially sensitive variables means that incentives are not generating a broadly felt increase in exchange motivations: one of the suggested mechanisms through which it should lead to more honest data reporting. This itself reduces the likelihood that exchange or obligations arising from the promise of payment are central factors in how incentives affect behaviour.

Could these effects be due to sample selection bias arising from poorer households in the treatment group remaining in the sample? This argument is difficult to support. Although the main effects of incentives on total income and consumption are in the same direction in both cities, and not significantly different across the two cities, the coefficient in Gulberga is visibly stronger for both reported income and consumption. This is true despite the nearly complete absence of non-response in Gulberga. Furthermore, it is straightforward to show, on the basis of the non-response rates in Davanagare (see Table 1), that unobserved household income levels of the unobserved, non-respondent households would have to be unrealistically low to explain the substantial gap in the mean income levels across treatment and control households. Specifically, reported mean household income in Davanagare is 15578 in the treatment group and 17101 in the control group—a gap of 1523 rupees. Given the estimated 7.4% non-response in the control group, the unobserved mean income of non-respondents would need to be minus 3480 rupees to account fully for the gap across the treatment and control groups. Thus, although incentives can increase sample selectivity in general (Curtin et al., 2005), it is not a major factor in this analysis.

In our view, a more plausible explanation for these findings is related to the resource-constrained context. As is typical in LDCs, household decision makers operate within very tight budgets and try to take advantage of any new financial resources. In this case, it is the survey programme. Whether they see the interviewer as a one-time visitor or as the vanguard of a new development initiative—whatever the interviewer tells them during the introductions has only secondary relevance to these perceptions—it affects their economic responses. For, by increasing their publicly reported level of need, respondents can justify the incentive payment and also, at no extra cost to them, plant a seed that may yet bear fruit in the future if the interviewer yet returns with his or her unknown backers: a government ministry, a development organization or some other group that is sufficiently wealthy to pay to discover how prosperous—or poor—local residents actually are.

This is the particular economic and interactional context in which respondents are reporting on a number of individual and household characteristics. It makes sense for them to provide an exaggerated impression of their poverty—at least where they can do so without entirely losing face—both through lower income and lower expenditures as well as lower asset values.

6. Conclusion

In this paper we have explored whether incentivizing participation in a survey in a relatively poor urban setting in India—i.e. providing respondents with small cash transfers at the time of the survey—can improve the quality of data. Underlying this goal is the simple fact that there is no existing experimental literature, and very little empirical literature in general, on the effect of incentivizing participation in LDC household surveys. Rather, the entire methodological edifice is based on experimental studies conducted in MDCs, and simple precedent in LDCs. This lack of empirical evidence is surprising, given the critical reliance on survey data in LDCs for identifying poverty and health conditions as well as evaluating development projects. This paper helps to fill this important gap, presenting results from a single survey-based experiment conducted in Karnataka, India.

Our findings show that there may be no easy resolution to the incentives issue. We expected to find a generally positive effect of incentives on both unit response and data quality. Instead, we obtained a mixed result. On one hand, unit non-response was small in the absence of incentives but almost non-existent with incentives. On the other hand, incentives had no effect on data quality on most questions and spheres of interest. In the most significant domain where its effects were strong and notable—respondents’ economic characteristics—incentives appear to have negative effects on data quality. Although we cannot fully validate our interpretation by using, for example, other sources of data on respondents’ economic characteristics, our findings are consistent with the idea that respondents who received an incentive were more likely to self-present in ways that would signal their suitability for more assistance. How much this matters and whether this sort of effect exists in other urban areas of India, in rural areas or other LDC settings outside India is an open question. It also remains unclear whether the specific survey instrument that was used here, which was focused so strongly on property and ownership of resources, might have contributed to this effect. We would be even more hesitant to generalize to panel surveys, where the same household is revisited over time. That is the particular type of survey design where some LDC researchers have recommended incentives to reduce sample attrition, its main function in MDCs (Bignami-Van Assche, 2003).

An additional question concerns the value of the incentive. Evidence from wealthier western contexts suggests that the value has little effect on non-response. Little is known, however, on whether varying the value of incentives matters in poorer societies. In our experiment, the first such experiment to be fielded in an LDC setting, the payment amounted to roughly a day’s wages for a labourer. Relative to the value of payments that are made to respondents in developed countries, this sounds like a large amount—payments in developed countries do not approach a day labourer’s wage. However, wages for the unskilled are low in India, even though the price of many consumer products are similar to those in MDCs. In addition, the sample frame does not include the poorest people, who would presumably be most responsive to a valuable incentive. A priori it is therefore difficult to know how the effect on data quality would change if the incentive was either reduced or increased. We leave it to future studies to consider this question. A final point about the value of the incentive is that, from a practical perspective, it is limited to amounts that can be included in survey budgets. For as interesting as it may be to incentivize different types of desired behaviours with the promise of large transfers—the approach taken in conditional cash transfer programmes—paying large amounts in surveys is unlikely to be feasible in a resource-constrained world. Over time, it may also exacerbate the extrinsic motivation problem, as people hear that, for example, in some past study, respondents received $50, but in this one they are only being offered $5.

A final question concerns the timing of the payment. It may be that, by announcing an incentive at the start of the interview but giving it only at the end, we encouraged respondents in the treatment group to accentuate their own need. Varying the timing of the actual handover would be an interesting variation on the experiment reported here: so would replicating the conventional best practice in MDCs, which is to provide the incentive unconditionally before the interview. Here, also, we would expect the timing issue to play out differently in a simple one-time-only survey than in a panel study.

Even with these limitations, however, the results of this experiment on the influence of incentives raise concerns that these types of payments should be carefully evaluated, with parallel studies replicated elsewhere, before it can be implemented on a wide scale in LDCs. Identifying desirable effects on unit non-response, but potentially undesirable effects on data quality in one critical domain, does not lend itself to clear ‘best practice’ for collecting higher quality data. In itself, this suggests that not only should there be on-going methodological research on data collection in LDCs based, in part, on revisiting some of the foundational debates about how to collect the best possible data across diverse socio-economic and interactional contexts, but we shall always also rely on econometric tools that help us to understand and, in some cases, to treat existing error. What is clear from this research is that paying respondents to participate in a survey does not, as we have shown here, necessarily eliminate or even attenuate error. It simply reduces certain types of error while adding to others.