Collecting Survey Data during Armed Conflict

Abstract

Surveys provide crucial information about the social consequences of armed conflict, but armed conflict can shape surveys in ways that limit their value. We use longitudinal survey data from throughout the recent armed conflict in Nepal to investigate the relationship between armed conflict events and survey response. The Chitwan Valley Family Study (CVFS) provides a rare window into survey data collection through intense armed conflict. The CVFS data reveal that with operational strategies tailored to the specific conflict, duration of the panel study is the main determinant of attrition from the study, just as in most longitudinal studies outside of conflict settings. Though minor relative to duration, different dimensions of armed conflict can affect survey response in opposing directions, with bombings in the local area reducing response rates but nationwide political events increasing response rates. This important finding demonstrates that survey data quality may be affected differently by various dimensions of armed conflict. Overall, CVFS response rates remained exceptionally high throughout the conflict. We use the CVFS experience to identify principles likely to produce higher quality surveys during periods of generalized violence and instability.

1. Introduction

Survey methods provide basic tools for gathering information about the human population. This information is a key source for understanding human populations and how they change and for designing policies and programs aimed at serving human populations. Because of this, collecting survey data during periods of dramatic social change, such as armed conflict, is often desirable. Such information regarding a population in conflict can serve to improve our understanding of the experiences and behavior of human populations and our understanding of programs designed to serve the needs of those in conflict. Unfortunately, conflict can pose a substantial obstacle to the collection of survey data and much of the research on conflict-affected populations is qualitative in nature, based on non-representative samples, or is limited by high levels of non-responses (Mneimneh et al. 2008). In this paper we examine the influence of armed conflict on survey non-response as a dimension of total survey error and document survey methods designed to overcome these obstacles and provide reliable measures from populations experiencing armed conflict.

The quality and reliability of surveys undertaken in conflict zones is a topic of substantial controversy. In 2002, Barakat et al. argued that scientific sampling (and thus a representative population sample) is usually not possible in conflict situations and that structured interviews might be an inappropriate means for collecting information. Concerns over data quality produce controversy regarding the basic facts surrounding armed conflict. For example, efforts to estimate the number of violent deaths during the US led invasion of Iraq vary from around 43,000 (Iraq Body Count 2011) to 654,965 (Burnham et al. 2006). Although the Burnham et al. (2006) study published in The Lancet used a household-based survey, the methods they used to obtain their unusually high estimate were subsequently widely criticized (Bohannon 2006; Johnson et al. 2008; Marker 2008). A similar whirlwind broke out around the International Rescue Committee's estimates of “excess deaths” of up to 5.4 million based on sample surveys in the Democratic Republic of Congo. The Human Security Report and Macro International found “excess deaths” to be radically lower, less than one half the IRC estimate (Coghlan et al. 2006; Coghlan et al. 2008; Human Security Report Project 2011; Macro International Inc. 2008. These contentious examples highlight the persistent concerns about whether it is possible to collect high quality and representative survey data during armed conflict (Spagat et al. 2009). Drawing from the Iraq and Democratic Republic of Congo survey problems, Marker (2008) suggests key sampling and surveying procedures that can be adapted in conflict situations to address some of the concerns with estimation. Still, it is ironic that little analytical attention has been devoted to understanding the influence of conflict on data quality and to design further strategies to address the challenges faced during data collection.

Of course there are many different dimension of survey data quality. The total survey error perspective identifies both sampling and non-sampling errors as important for total survey quality, where non-sampling errors include measurement errors due to the respondent, the interviewer and the questionnaire (Groves 1987; Groves et al. 2004). Though all the many different dimensions of survey data quality shape the value of the measures derived from surveys, many survey data users focus on survey response rates as an overall indicator of the quality of the survey. In so far as obtaining high response rates requires high effort (Groves and Couper 1998) and in general higher effort is required to produce higher quality survey data (Groves 1989), response rates may provide an overall indication of survey data quality. However, careful study clearly reveals that non-response bias varies greatly across measures within a single survey, rendering the overall survey response rate a poor indicator of the level of response bias in any particular measure (Groves 2006; Groves and Peytcheva 2008). Response rates are only one dimension of overall survey quality and the consequences of response rates for the quality of any specific measure are likely to vary. Nevertheless, because the literature on data collection during conflict repeatedly points to non-response as a crucial threat to reliability and validity of survey based measures, we focus on this dimension of survey quality. Comparison of response rates over time as armed conflict unfolds provides an important window into the relationship between armed conflict events and this dimension of survey quality.

In this article, we investigate response rates and data collection methods during armed conflict using the Chitwan Valley Family Study (CVFS), a large scale longitudinal study that was conducted before, during, and after the recent decade long armed conflict in Nepal. In the context of an on-going panel study, we argue that careful application of survey operations best-practices and management of the conflict situation make it possible for survey researchers to continue to collect reliable information from representative samples of conflict-affected populations. Conclusions about data quality and collection methods could also be relevant for new studies that begin during conflict periods.

We treat non-response and its components (refusals and non-contacts) as a summary indicator of survey quality, but explicitly recognize both the limits of this approach and other important dimensions of survey quality. By linking together newly constructed measures of specific conflict related events (such as bombings and political events) with measures of survey response (household- and individual-level response, refusal, and non-contact rates), we evaluate the magnitude of effects of specific types of conflict events on survey response. As with all longitudinal panel studies the duration of the panel study is a fundamental driver of continued participation and therefore response rates. But within this context we are able to use these uncommon measures of conflict related events to estimate their independent influence on participation in the study. This empirical analysis provides a unique window into survey data collection during dramatic armed conflict.

Because the CVFS was able to maintain exceptionally high response rates through out the Nepalese conflict, and render duration of the panel study a much more important determinant of participation than events related to the conflict, we also document the operational strategies that can be used to reduce the consequences of armed conflict on survey data collection and protect the safety of respondents and interviewers. These strategies include absolute political neutrality of the research staff, direct interaction with political leaders or warring parties on all sides of the conflict, highly transparent procedures for the protection of human subjects from disclosure risks, and tailoring of contact procedures and protocols to changing respondent and interviewer circumstances. Using evidence from Nepal, we show that the application of these strategies to field operations can minimize the consequences of violent armed conflict on the quality of survey data collected during such periods of time.

2. The Setting, the Study, and the Conflict

Our investigation focuses on survey data collected as part of the Chitwan Valley Family Study (CVFS). The Chitwan Valley lies in south central Nepal and was opened for settlement in 1955. New settlers represented a wide range of ethnic groups from the hills and mountains, turning once an isolated “death valley” into a “melting pot” creating substantial ethnic diversity in Chitwan (Gunaratne 1996; Müller-Böker 2000). The construction of two major all-weather roads in the late 1970s linked Chitwan's largest town, Narayanghat, to the capital city Kathmandu and the cities in Eastern Nepal and Northern Indian. This change turned the once the isolated valley into the transportation hub of the country, dramatically changing the lives of local people in the decades from 1980 onward (Axinn and Yabiku 2001). The area remains primarily agricultural and poor.

In 1996 the CVFS selected a systematic probability sample of 151 neighborhoods in Western Chitwan (Barber et al.1997). The CVFS defined a neighborhood as a geographic cluster of five to fifteen households. Once a neighborhood was selected, all the households and individuals aged 15 to 59 residing in the sampled neighborhood were interviewed. If any of these study respondents had a spouse living elsewhere, that spouse was interviewed as well. In 1996, a total of 4632 individuals were interviewed from those 151 neighborhoods with a 97 percent response rate.

In 1997, the CVFS launched a Monthly Demographic Event Registry to track the individuals and their households over time and collect survey data on marriage, childbearing, migration, and mortality in their households. This demographic event registry forms the backbone of the CVFS prospective panel study. Beginning in February of 1997, interviewers visited each household monthly to monitor births, deaths, marriages, divorces, contraceptive use, pregnancies, and changes in living arrangements. This monthly data collection has continued from 1997 until the present. The monthly household visit included an interview with one household member who reported on all changes in the household and a separate individual interview with each household member age 18 and older about their contraceptive use. The interviewing is conducted by the same interviewers and supervisors who collected the original individual interviews (including life histories), so they are familiar to the respondents.

The demographic event registry is conducted with high attention to quality assurance procedures. Over the first year, the CVFS maintained a response rate of 100% for the household interviews and 99.5% for the individual interviews on contraceptive use. Of course, as typical in any long-term longitudinal study, attrition has increased and response rates have declined over time (Groves and Couper 1998). The ongoing armed conflict during the period of data collection was likely an important contributing factor to this attrition. However, even during the armed conflict, the CVFS household registry and individual contraceptive use surveys were able to maintain contact with a large fraction of the original households and respondents and response rates remained quite high, even after fourteen years.

1.1 Armed Conflict in Chitwan Valley

Chitwan's population is highly heterogeneous, ethnically diverse, and politically active. Chitwan has been at the forefront of almost all of the national-level political movements as well as the recent armed conflict between the Government of Nepal and the Communist Party of Nepal Maoist.¹ The Maoists adopted Mao Tse-tung's strategy of recruiting the most oppressed groups of people (generally the rural poor) for their militant forces and turning most rural areas into their base areas for operations. They initially confined their activities to the most remote areas of the Far Western region of the country and, aside from the first wave of arrests, Chitwan remained relatively unaffected for several years. As time passed, the Maoists gained strength and the conflict spread across all parts of the country. By 2000 there were clear signs of the conflict in the Chitwan Valley, the setting for this paper. There, violence increased from 2000-2005 and ended in 2006. Bombs blasts were common throughout this period and there were periodic major gun battles in the area. In addition to the 224 conflict-related fatalities in Chitwan during 2000-2006 (Informal Sector Service Center 2006), civilians experienced 809 tortures and arrests without warrant, 358 beatings, and house raids, varying forms of intimidation, forced conscription, billeting, extortion, curfews, travel restrictions, and general strikes (South Asia Terrorism Portal 2006; Hutt 2004; Pettigrew 2004; Pettigrew and Adhikari 2009). Thus, although Chitwan was not in the most contested area of the far west, from 2000 there were continuous threats of violence and the lives of civilians were heavily affected on a daily basis.

3. Linking the Conflict to Response Rates

The violent conflict in the CVFS study area could have disrupted survey operations and increased non-response through many different potential mechanisms. In a longitudinal panel study, three factors shape non-response: (1) respondents’ refusals to participate, (2) non-contact, or difficulty locating respondents, and (3) death of respondents (Groves and Couper 1998). Fortunately, in the case of Chitwan, there were relatively few conflict-related deaths and this was not a significant factor in non-response to the surveys. Although one might imagine that conflict events increase both difficulty locating respondents and refusals to participate, there are good theoretical and empirical reasons to believe that different types of conflict-related events and the geographic location of these events may have different consequences for contacting and interviewing respondents. These differences come from differential relationships between specific conflict events and migration, local mobility, and anxiety and perceptions of threat.

With regard to migration, recent empirical work with the CVFS data from Nepal demonstrates a clear difference between specific violent and political events. Evidence shows that while violent events in the local area increased international migration, political events associated with the conflict decreased international migration (Williams et al. forthcoming). This is likely because political events are destabilizing and cause a sense of chaos throughout the country, making people less likely to undertake a migration which is also a destabilizing behavior. The geographic location of these events could also have important, and opposing, effects on internal migration. Events such as bomb blasts in the local area could increase international and internal migration by making the local area a more dangerous place to live. Alternately, events that affect the whole country and are concentrated in the capital Kathmandu, such as a State of Emergency or large protests, might decrease internal migration through making the whole country a dangerous place to travel and resettle.

Because out-migration is a key cause of inability to locate respondents and along with refusals is a component of attrition from the longitudinal study, we predict these different dimensions of conflict will also be related to survey non-response. We hypothesize that bomb blasts in the local area will increase out-migration and therefore decrease contact rates, on average, in both the household and the individual longitudinal surveys. Nationwide political events, on the other hand, will decrease internal migration and increase contact rates.

Alternately, refusals to participate in the survey will be largely driven by fear, perceptions of threat, and anxiety about the possible consequences of survey participation. Because the conflict was publicized by the Maoists as a class struggle between those who “have” and those who “do not have”, ordinary people were often caught in the middle of the two warring parties, creating great anxieties and perceptions of threat for both the poor and the more well-off. Both groups were at risk of suspicion of being a member, supporter, or spy for one or the other warring party and were often severely punished as a result. These punishments included extortion, abduction, arrest, beating, and or assassination. Furthermore, the Maoist militants, who claimed they were the “People's Army” usually moved in groups and forced ordinary people to provide shelter and food. The possibility of being forced to billet the Maoist forces induced fear throughout the population. In addition, the day after a household billeted Maoist forces, the government security forces often came and raided the house for providing shelter and food to the Maoist militia, creating even greater fear. As a result, once very welcoming households become very suspicious about even opening their doors to anyone, about the information that a study might collect, and whether the study staff would share the information they had collected (such as the number of new members in the household) with the government security forces so that they could track Maoist penetration or movement in the area.

The consequences of different conflict events on fear, perceptions of risk, and ultimately on refusals to participate in the survey are more difficult to predict because we have no direct empirical evidence on this phenomena. The mere presence of violence in the local area may tend to make respondents less likely to open their doors and less likely to invite an interviewer inside to participate in the survey, thereby increasing refusals. If this is true, then bomb blasts in the local area will increase refusals. With regard to nationwide political events, these events will also increase refusals, but to a smaller extent then violent events in the local area.

Finally, there is also good reason to expect the effects of conflict-related events on household-level and individual-level non-response to be somewhat different. This is because the CVFS household-level survey allows both multiple informants and change over time in informants, so that individual-level feelings of anxiety and threat are less likely to affect non-response to the household survey. In other words, if one individual in a family does not want to respond to the household survey, another family member can still do so on behalf of the family. Individual responses to the violent events, however, are likely to have a stronger influence on the individual survey non-response. Thus, conflict-related events will likely produce larger variations in monthly rates of non-response in the CVFS individual-level contraceptive use survey than in the household survey.

4. Empirical Evidence from the Chitwan Valley Family Study

The combination of monthly response rate measures from the CVFS for both household- and individual-level instruments and monthly measures of conflict-related events, in both the study site and nationwide, allow us to explore the empirical relationship between the two. In this section, we first describe the data and methods employed. Then we examine changes over time in the response rates of households and individuals in the CVFS. We then describe changes over time in the individual non-contact and refusal rates. Next, we proceed to results of the regression models and the specific influences of conflict-related events on individual response, non-contact, and refusal rates.

4.1 Data

Data for this study come from two sources: response rates data from the CVFS and conflict event records from the South Asia Terrorism Portal (SATP) and other news and institutional sources.

4.1.1 Measures of Survey Response

Measures of response rates come from two data sets collected by the Chitwan Valley Family Study (CVFS) since 1997: the Prospective Monthly Demographic Household Registry (household-level response rates) and the Prospective Monthly Contractive Use Survey (individual-level response rates).

Beginning in February of 1997, interviewers visited each household monthly to update demographic events such as births, deaths, marriages, divorces, contraceptive use, pregnancies, and changes in living arrangements. The 151 neighborhoods include 1,580 households and 4,632 individuals interviewed as part of the original study. All residents of these 1,580 households have been followed over time, including households and individuals from those households who have moved out of the study area. This means that the prospective panel data are maintained for all those who were interviewed in the original study, regardless of their migration behavior.

During the monthly household visit, interviewers interview one household member who is capable of reporting on all changes in the household. Of course these household informants may not be fully aware of other household members’ contraceptive use, and may misreport it as a result. Thus, to improve accuracy and safeguard confidentiality, the CVFS individually interviews each household member age 18 and older about their contraceptive use. For this study, we use 126 months of response rate data from the prospective monthly household registry and the contraceptive use survey, covering the period from February 1997 through July 2007.

4.1.2 Measures of Response Rates

Our measure of the response rate to the household registry is the percent of completed household interviews each month. The percent of completed interviews, the household response rate, is calculated by dividing the total number of completed household interviews by the total eligible² households and multiplying by one hundred. Our measure of the individual response rate is the percent of completed individual interviews. The percent of completed interviews, the individual response rate, is calculated by dividing the total number of completed individual interviews by the total eligible individuals for the contraceptive use survey and multiplying by one hundred.³

4.1.3 Measures of Refusal Rates

To measure refusal rates, we used the percent of respondents who refused to participate in the contraceptive use interview. The refusal rate is calculated by dividing the number of respondents who refused to participate in the contraceptive use interview by the total number of eligible respondents and multiplying by one hundred.

4.1.4 Measures of Non-contact Rates

To measure non-contact rates, we used the percent of respondents both within and outside of Nepal who could not be contacted after three attempts for an individual interview. This measure is calculated by dividing the total number of respondents who could not be contacted by the total number of eligible respondents (respondents residing within or outside Nepal) and multiplying by one hundred.

4.1.5 Measure of Duration

In the context of a longitudinal study like the CVFS, response rates, refusal rates, and non-contact rates are all known to vary systematically with the duration of the study (Groves and Couper 1998). In order to control for these duration effects in multivariate models, we constructed a single continuous measure of years since the study started. This duration measure is coded ‘0’ in 1997, ‘1’ in 1998, ‘2’ in 1999, etc. This measure is then used as a control variable in our multivariate models.

4.1.6 Measures of Conflict Events

The South Asia Terrorism Portal (www.satp.org), our source for records of violent events, is an India-based NGO that compiles records of all violent events in Nepal and other South Asian countries. Records include the date, place, and number of people affected (injured or killed) by each bomb blast. Measures of political events are compiled from information collected from major English and Nepali news media, situation reports of non-governmental organizations in Nepal such as the Informal Sector Service Center and the United Nations Nepal Information Platform.

Because armed conflicts are not single homogenous events or periods, quantitative measurement of a conflict is a complex task. This is especially the case for analyses of civilian responses to violence, where conflict must be measured from a civilian perspective instead of a macro-level policy or political science perspective. Instead of a single homogenous time period, conflicts are comprised of a series of discrete violent and political events that must be measured carefully. These might include gun battles, bomb blasts, abductions, kidnappings, house raids, ceasefires, states of emergency, or major demonstrations. Furthermore, these different types of events can mean different things to civilians, based on the level of violence, predictability, and frequency of each event and whether they usually occur in public or private areas (Williams et al. 2010). Thus different kinds of events can have varying, or even opposing, effects on behaviors such as migration (Williams et al. forthcoming; Williams and Pradhan 2009). As such, creating scales or indexes by adding up conflict events is not necessarily appropriate. However, individual types of events might be strongly correlated, creating problems when including measures of multiple events in one model.

To address these concerns, we use one measure of a specific kind of violent event in the local area—bomb blasts—and one measure of nationwide political events. We use bomb blasts in this analysis because these events were relatively frequent, and because they were public events that were generally recorded on a more consistent basis than abductions, kidnappings, and house raids. Our measure of bomb blasts is the number of bomb blasts per month in Chitwan and the six neighboring districts (Nawalparasi, Tanahu, Gorkha, Dhading, Makwanpur, and Parsa). Districts in Nepal are quite small, thus the combined area of these seven districts is approximately half the size of Belgium. To create this measure, we use records of bomb blasts from SATP. The data cover 67 months, from January 2002 through July 2007. For the time period that these data do not cover, from February 1997 until December 2001, we impute the number of bomb blasts to be zero. News reports and research indicate that the conflict was at a very low intensity (Hutt 2004) during this time and CVFS research staff members who are resident in the area indicate that there were very few of these violent events before 2002. Figure 1 shows the number of bomb blasts per month throughout the study period. As you can see, from mid-2003 bomb blasts occurred routinely, almost every month until 2005. Of the 67 months of data about bomb blasts, there were 27 months in which there was at least one bomb blast. The largest number of bomb blasts in any one month was 12, in July 2004.

Figure 1.

Number of bomb blasts, political events, and household and individual response rates in the Chitwan Valley Family Study

Our measure of political events is the number of major political events per month. This includes states of emergency, major strikes and protests, and major changes in government that threatened the stability or basic functionality of the central government. A ‘state of emergency’ is any period when the government officially proclaimed a state of emergency, which included proclaiming martial law, limiting constitutional rights, and instituting curfews and travel restrictions. Major changes in government include changes or depositions of the prime minister, the ruling party, the dissolution of parliament, and the 2001 Narayanhiti Palace killings. ‘Strikes and protests’ refer to any such event that involved at least several hundred people, took place nationwide, in Kathmandu and/or other urban areas, and was reported in national Nepali and English language newspapers and over FM radio broadcasts. All of these events were widely publicized to the general population and threatened the basic stability of the government, creating a sense of a vacuum of power when it was not clear who was or would be in control of the country. In addition to bomb blasts, Figure 1 also shows the number of political events that occurred in each month of the study period. Besides one long state of emergency in 2001 and 2002, political events occurred sporadically throughout.

4.2 Analytic Strategy

We use Ordinary Least Squares (OLS) regression models to predict the effects of bomb blasts, political events, and duration on individual response, refusal, and non-contact rates. Because household response rates remain very high and there is little variance, we are not able to model this outcome. Thus our regression models test the effects of these time-varying events on individual response rates, individual refusal rates, and individual non-contact rates. We lag all conflict-related measures by one month in order to assure that the results we are measuring occurred chronologically after the event. For example, the models test the effect of a bomb blast in April on response rates in May. Thus this is an examination of the immediate effects of conflict events on response rates.

Effectively, these are models of time. The units of analysis in this study are months. The outcomes of interest are response rates and the independent variables are the numbers of conflict events per month. These are all characteristics of months, not characteristics of people.

4.3 Results

Response rates of individuals and households remained very high throughout the ten-and-a-half years that we examine, from February 1997 through July 2007. As shown in Figure 1 with the solid line, household response rates slowly decreased from 100% in February 1997 to a low of 98.41% in July 2007. Individual response rates also remained high, but with more variance. Individual response rates, shown with a dashed line, started at a high of 100% in July 1997, progressively decreased until about mid-2003. Around October of 2003, individual response rates experienced an even steeper and sustained decrease until the end of the year. Then we find slightly increased response rates until the beginning of 2005 where another period of sustained decrease continued through July 2007, ending in a low of 92.51%. In Figure 1 we present these response rates laid over bomb blasts and political events to provide an opportunity for initial visual analysis of possible relationships between events and response rates. As you can see, the steepest sustained decrease in individual response rates occurs just after the first major increase in bombings in mid-2003. This could suggest a relationship between bombing and response rates and indicates the need for further statistical analysis.

One reason that the household response rates remained much higher than individual response rates throughout this period is that we allowed both multiple informants and change over time in informants for the household survey. Thus if an individual in the household migrated away or refused to participate, another household member could participate on behalf of the household. In consequence of this consistently high household response rate, disaggregating it into refusals and non-contacts and examining the impacts of specific events is not possible.

Disaggregating individual response rates into non-contact and refusal rates provides more information. These are shown in Figure 2 with the solid and dashed lines on the left-hand y-axis. Again, these are overlaid on the event data, which relate to the right-hand y-axis. Refusal rates remained low and progressively stable, increasing to a high of just 1.14%. Non-contact rates were more variable and reached a much higher peak of 6.34% in July 2007. Notably, the individual non-contact rate produces a solid line that is almost exactly symmetrical to the dashed line for the individual response rate in Figure 1, indicating that much of the variance in response rates was driven by non-contacts. Again, the steepest sustained increase in the non-contact rate appears at similar points in time as increases in bombings, around October 2003.

Figure 2.

Number of Bomb blasts, political events, and individual non-contact and refusal rates in the Chitwan Valley Family Study.

We now turn to results of regressions, using conflict events and duration to predict individual response rates. Model 1 of Table 1 shows the effect of duration since the study started on response rates. The coefficient for years since study started is -0.65 and statistically significant, indicating that for each additional year, the response rate decreased by about 0.65%. The adjusted R² is about 0.9480, indicating that about 95% of the variance in individual response rates is predicted by duration alone. The effect of duration remains consistently strong throughout Models 2-4.

Table 1.

OLS regression models, predicting individual response rates during armed conflict in the Chitwan Valley

	Model 1	Model 2	Model 3	Model 4
Local Events
Bomb blasts per month (SATP) (0-12)		- 0.01 (0.35)		- 0.01 (0.62)
Nationwide Events
Political event index (0-4)			0.25*** (3.26)	0.26*** (3.29)
Duration
Year since study started (0-10)	- 0.65*** (47.75)	- 0.65*** (45.21)	- 0.66*** (49.22)	- 0.65*** (46.87)
R2	0.9484	0.9485	0.9525	0.9527
Adjusted R2	0.9480	0.9476	0.9518	0.9515
No. of Observations (months):	126	126	126	126

T-statistics are given in parentheses.

*p<.05

**p<.01

^***p<.001 (one-tailed tests)

Model 2 shows the effects of bomb blasts on response rates, independent of duration since the study started. The coefficient for bomb blasts is small and it is not statistically significant. Thus, we fail to find evidence that bomb blasts had a significant effect on response rates.

The effect of political events on response rates, which is shown in Model 3, was positive, meaning that political events increased response rates. In other words, independent of duration, in any month after a large and destabilizing political event occurred, we find a 0.25% higher response rate. We find similar effects in Model 4, which includes all three measures for bomb blasts, political events, and duration. Note that although political events produce statistically significant effects on response rates, the coefficient is less than half the size of the duration coefficient. This suggests that while political events are important, the effect of one political event on response rates was much smaller than the effect of one year duration since the study started. In other words, we find that duration produced the strongest effect on response rates during this conflict.

Table 2 shows the results of regression models predicting refusal rates. Note that the meaning of the coefficients for refusal rates is opposite of those for response rates. When refusal rates go up, response rates go down. Similarly, a positive effect on refusal rates would be a negative effect on response rates. As shown in Model 5, duration had a small, positive, and statistically significant effect on refusals. This means that for each year of the study, refusals increased by about 0.09%. While the effect is small, this is primarily because there was little change in refusal rates throughout the study. Even with a small coefficient, the adjusted R² in Model 5 is 0.9599, meaning that about 96% of variation in refusals is a result of duration since the study started.

Table 2.

OLS regression models, predicting individual refusal rates during armed conflict in the Chitwan Valley

	Model 5	Model 6	Model 7	Model 8
Local Events
Bomb blasts per month (SATP) (0-12)		0.01*** (4.83)		0.01*** (5.02)
Nationwide Events
Political event index (0-4)			- 0.01^ (1.43)	- 0.02* (1.95)
Duration
Year since study started (0-10)	0.09*** (54.45)	0.09*** (54.87)	0.09*** (53.86)	0.09*** (55.03)
R2	0.9599	0.9663	0.9605	0.9673
Adjusted R2	0.9595	0.9657	0.9599	0.9665
No. of Observations (months):	126	126	126	126

T-statistics are given in parentheses.

^*p<.05

**p<.01

^***p<.001 (one-tailed tests)

Bomb blasts and political events had even smaller effects on refusal rates. The coefficient for bomb blasts in Model 6 and Model 8 (where political events are controlled) is 0.01, meaning that in a month after one bomb blast the refusal rate went up by 0.01%. The coefficient for political events is negative in Model 7 and in Model 8 (where bomb blasts is controlled). At -0.02, the effect of political events is similarly small, influencing a decrease in refusals of 0.02%. In other words, fewer people refused to take the survey after political events. The effect of bomb blasts on refusal rates supports our hypothesis that events in the local area would increase anxiety and make people less willing to participate in the survey. The negative effect of political events however is opposite of this prediction. This indicates that there could be an important difference between events in the local area and events that are concentrated in the capital. Further detailed research that includes qualitative interviews will be necessary to provide a better understanding of this difference. However, the key finding from this analysis of refusals is that bomb blasts and political events made very little difference in refusals and the largest influence we find was duration.

We now turn to the results predicting the non-contact rates, shown in Table 3. Duration had a positive effect throughout all models. As shown in Model 9 that includes only duration, the coefficient is 0.55, meaning that for each additional year of the survey, non-contact rates increased by about 0.55%. As with refusals and the overall individual response rates, the adjusted R² in this model is high at 0.9361. This means that duration alone accounts for about 94% of variance in non-contact rates. Bomb blasts did not have a significant influence on non-contact rates in Model 10 or in Model 12 that included all measures. Political events had a negative effect, with a coefficient of -0.24 in Model 11. This indicates that independent of duration and bomb blasts, in a month after one political event, the non-contact rate decreased by 0.24%. In other words, political events made it easier to contact people. This result is consistent in Model 12 when bomb blasts are also controlled. Thus we find similar results for non-contacts as we do for individual response rates; duration increased non-contacts and political events decreased non-contacts. Again, while the effect of political events is notable and statistically significant, it was much smaller than (less than half) the effect of duration measured in years.

Table 3.

OLS regression models, predicting individual non-contact rates during armed conflict in the Chitwan Valley

	Model 9	Model 10	Model 11	Model 12
Local Events
Bomb blasts per month (SATP) (0-12)		- 0.004 (0.18)		0.001 (0.07)
Nationwide Events
Political event index (0-4)			- 0.24*** (2.07)	- 0.24*** (3.19)
Duration
Year since study started (0-10)	0.55*** (42.62)	0.56*** (40.50)	0.56*** (43.94)	0.56*** (41.92)
R2	0.9361	0.9361	0.9410	0.9410
Adjusted R2	0.9356	0.9351	0.9401	0.9396
No. of Observations (months):	126	126	126	126

T-statistics are given in parentheses.

^p < .10

*p < .05

**p < .01

^***p < .001 (one-tailed tests)

In summary, we find that bomb blasts, political events, and duration all had significant effects on different aspects of response in the CVFS between 1997 and 2007. Bomb blasts affected only refusal rates and the effect was so small as to be almost negligible. Political events had positive effects, increasing response rates, slightly decreasing refusals, and decreasing non-contacts. These results provide evidence to support the hypothesis that migration during armed conflict might actually decrease in response to political events centered in the capital Kathmandu that cause instability and decreased migration can result in higher response rates and lower non-contact rates in Chitwan.

However, as shown in Tables 1, 2, and 3, these results clearly demonstrate that the largest effect on response rates, refusals, and non-contacts during this period was duration. In other words, the normal longitudinal attrition over time had the largest effect on each outcome and improved model fit (adjusted R²) to the largest extent. For example, in Model 4 which predicts response rates, each progressive year decreased response rates by about 0.65%. Bomb blasts had no effect and political events increased response rates by about 0.26%. Thus, during this period, attrition had the largest effect on response rates, as we would expect to find in any survey, during or not during an armed conflict. When both political events and bomb blasts are taken into account, evidence indicates that the conflict actually improved response rates (through political events) more than it decreased them (through bomb blasts).

5. Strategies for Maximizing Survey Data Quality and Safety during Conflict

Clearly, the procedures used in the Chitwan Valley Family Study (CVFS) panel data collection helped to mitigate the potential consequences of violent events on response rates. Next, we review the operational strategies used in the CVFS to reduce the consequences of armed conflict on the survey data collection and protect interviewer and respondent safety at the same time. By considering steps taken to improve the survey operation during this period of conflict, we illustrate key operational strategies that can be used to maximize survey coverage and data quality during armed conflict in other areas as well. These strategies include absolute political neutrality of the research organization and staff, direct interaction with political leaders on all sides of the conflict, highly transparent procedures for the protection of human subjects from disclosure risks, and tailoring of contact procedures and protocols to changing respondent circumstances. We describe the CVFS operational procedures linked to each of these strategies in that order below.

5.1 Absolute political neutrality of the research organization and staff

The CVFS in Nepal is conducted by a local organization, the Institute for Social and Environmental Research Nepal (ISER-N), which follows a strict practice of political neutrality. At the organizational level, this means that the ISER-N never supports any specific political parties or leader and never uses its research, educational, or outreach activities in favor of any one politician or political entity. It also means that the ISER-N does not have any institutional relationships with other non-governmental organizations which could be perceived to have specific political agendas. At the individual level, this means that the staff of the ISER-N, including interviewers, is not allowed to become leaders or activists in any specific political party or political cause. This stringent policy of absolute political neutrality proved essential in maintaining respondent cooperation as well as permission to continue operating during the period of conflict when governments were changing rapidly and sometimes informally. Moreover, ISER-N had followed this practice of political neutrality for many years before the conflict began. This long-term commitment to political neutrality helped to establish a reputation of political independence for ISER-N. As violence increased and political factions became stronger, government security became weaker, and local people's trust in either of the conflicting parties deteriorated. The commitment of ISER –N to political neutrality became a fundamental building block for maintaining high levels of respondent trust and cooperation throughout the violent period.

5.2 Direct interaction with political leaders on all sides of the conflict

As conflict began, staff of the ISER-N began systematically engaging in meetings with political leaders on all sides of the conflict. The meetings included both visits to these leaders and invitations for the leaders of the conflicting parties to visit the offices of the ISER-N to inspect them. First, these meetings were used to explain the scientific and educational objectives of the research conducted for the CVFS. Second, they were also used to explain to leaders on all sides of the conflict that the research was low risk to all of those engaged in the conflict. Third, this direct regular contact with leaders on all sides of the conflict was used to assure that both the study and the data collection organization could quickly respond to changing circumstances and demands of the conflict.

5.3 Transparent procedures for the protection of human subjects from disclosure risks

The transparency of human subjects protection protocols during the conflict became as important in the CVFS as the protocols themselves. The CVFS always followed standard protocols for the protection of human subjects from research risks, including disclosure risks. These include: a) no use of names, addresses, or other identifying information on any questionnaire materials that had respondent answers to questions, b) use of ID numbers on all respondent materials, c) control of content to avoid sensitive topics, especially topics that might be sensitive during the conflict (such as political party membership), d) computerized data entry including only ID numbers and no other identifying information, and e) secure (locked and guarded) physical separation of all study materials that did contain identifying information, such as cover sheets and control files. These routine practices always provide important safeguards against the disclosure of respondent identities, but were especially important during the period of conflict. Most important, these ongoing human subjects protection protocols were made transparent to both respondents and political leaders during the time of conflict.

5.4 Tailoring of contact procedures to changing respondent circumstances

Tailoring respondent contact procedures is a known and valuable approach to maintaining high response rates no matter what the circumstances (Groves and Couper 1998). In times of conflict this tailoring must address specific aspects of the conflict itself, as well as individual respondent and interviewer needs.

During the conflict period, CVFS respondent contact procedures were altered in several specific ways to tailor them more appropriately to the conflict circumstances. First, CVFS staff stopped writing names on contact materials and instead kept names of individual respondents in strictly confidential control files and call records. Second, interviewer calling times were altered to eliminate any calling outside of daylight hours. This was done to reduce suspicion of both the interviewers and the respondents they visited. Third, during periods of peak violence the monthly registry data collection was switched from one visit a month with monthly reporting to one visit every three months with retrospective monthly reporting across those three months. Fourth, respondent tracking procedures were changed to include institutionalized populations, including both those in the military and those in prison. Fifth, during periods of peak violence, CVFS interviewers stopped conducting follow-up interviews and tracking efforts in specific geographic locations. Although this procedure created some temporary dips in response rates, it built confidence in the interviewing staff and rapport among the respondents who could be assured that study managers were concerned for their safety and would actively avoid data collection in violent locations.

6. Conclusion

The continuation of the Chitwan Valley Family Study (CVFS) panel data collection throughout the period of the recent armed conflict in Nepal provides a relatively rare window into the dynamics of survey data collection operations through a period of intense armed conflict. This window serves both to teach us about the consequences of violent events for survey data collection and to illustrate key strategies in forming a survey operations response to circumstances of violent conflict. Though the CVFS was collecting data before the conflict in Nepal began, insights from this investigation provide an important source of information for those who seek to begin survey data collection in the context of an ongoing conflict.

In the domain of consequences of violent events for survey data collection, we use this case to demonstrate how specific violent events influence survey data quality. Faced with violent conflict in the study population, many surveys may simply cease operations and close data collection based on the assumption that quality data cannot be collected. Because the CVFS continued to collect data throughout the conflict in Nepal, we were able to analyze the links between changes over time in the frequency of violent and political events and changes over time in response rates. Non-response is only one dimension of total survey error (Groves et al. 2004) and the biases produced by non-response vary across items within a survey (Groves 2006; Groves and Peytcheva 2008). However, high response rates are a product of high effort (Groves and Couper 1998) and high effort improves survey quality (Groves 1989), so we treat response rates as some empirical insight into overall survey quality.

The analyses we present here demonstrate that with operational effort to address the violent conflict conditions, violent events themselves can be kept to a relatively modest influence on participation in the survey. As with most longitudinal surveys, duration of the CVFS was by far the greatest driver of response rates (Groves and Couper 1998). However, our results are also consistent with the conclusion that different dimensions of armed conflict can affect survey data quality in opposing directions, with higher numbers of bombings in the local area slightly increasing refusal rates but higher numbers of nationwide political events actually increasing response rates through decreasing both refusals and non-contacts. This important finding not only demonstrates that survey data quality may be affected differently by various dimensions of armed conflict, it also shows that some dimensions of armed conflict may actually increase response rates. Perhaps most important is the general result that response rates for both the household survey and the individual survey remained exceptionally high throughout the conflict period. This contradicts common assumptions about the infeasibility of collecting survey data during conflict and indicates that high quality data can indeed be collected during periods of generalized violence and instability.

In the domain of illustrating key principles in forming a survey operations response to circumstances of armed conflict, the procedures used in the CVFS panel study clearly helped to mitigate the potential consequences of those violent events on response rates. In fact, our analyses reveal that the natural decay over time in response rates likely in any panel study is a much stronger explanation of the month-to-month variation in CVFS response rates than the conflict events themselves. The operational strategies used in the CVFS to reduce the consequences of violent conflict virtually eliminated the potential negative influence of these events on response rates. The principles illustrated by those operational strategies include absolute political neutrality of the research staff, direct interaction with political leaders on all sides of the conflict, highly transparent procedures for the protection of human subjects from disclosure risks, and tailoring of contact procedures and protocols to changing respondent circumstances. We argue that these key operational principles can be used to maximize survey coverage and data quality during armed conflict in other settings too.