Does canvassing increase voter turnout? A field experiment

Abstract

This paper reports the results of a randomized field experiment involving registered voters in the city of New Haven. Nonpartisan get-out-the-vote messages were delivered through personal canvassing shortly before the November 1998 election. We find that personal canvassing increased voter turnout by ≈6. The effect of personal contact seems to be slightly smaller for voters registered with a major political party and higher for unaffiliated voters, although the hypothesis that all voters are equally affected could not be rejected. Study of several alternative political messages provided equivocal evidence suggesting the superiority of a canvassing appeal that emphasizes the closeness of the election.

Do political campaigns have important effects on voter behavior? Debate over this question has raged for several decades, generating a large literature with mixed results. Scholars remain divided on key issues. Some maintain that campaigns have only minor effects on political behavior (1, 2), whereas others argue that waging a significant campaign increases the chances of election success (3, 4). Still others claim that the level of campaign activity matters but only for some candidates (e.g., for challengers but not incumbent politicians; refs. 5 and 6). This paper focuses on a closely related question. We employ an underused research methodology, field experimentation, to measure the effectiveness of a particular kind of political activity. How effective are political mobilization efforts? To what extent does personal canvassing stimulate registered voters to go to the polls?

This question has been studied by both experimental and nonexperimental methods. The experimental tradition dates to Harold Gosnell’s studies (7) of voter registration and turnout in Chicago. Randomly assigning certain city blocks to receive mailed reminders to register and vote, Gosnell found that turnout increased by 1% in the presidential election of 1924 and by 9% in the municipal election of 1925. Canvassing and mailings also proved influential in experiments conducted by Eldersveld and Dodge (8, 9) in Ann Arbor.† Nonexperimental research (13, 14) relies mainly on survey data and examines the relationship between voter turnout and reported personal or phone contacts with political organizations or candidates. Rosenstone and Hansen (15), for example, regress reported voter turnout on reported contact with candidates or political parties and a host of control variables. Unlike the aforementioned field experiments, the Rosenstone and Hansen study examines nationally representative samples of relatively recent vintage. The drawback to this nonexperimental analysis, however, is that political contact may not be an exogenous predictor of voter turnout. If parties direct their appeals disproportionately to committed partisans, those most likely to vote will be most likely to receive political contact, and the apparent link between contact and turnout may be spurious. Moreover, the researcher has no control over, and often little knowledge of, the frequency or nature of the political contact.‡

These limitations persuaded us to launch a large-scale voter-turnout experiment that attempted to stimulate turnout by means of a personal canvass of randomly selected households. Using a variety of different nonpartisan appeals, we find that mobilization efforts raise turnout demonstrably. Canvassing door-to-door typically raises turnout by about 6%. Given the minimal nature of our experimental treatments, the magnitude of these effects is quite striking. After describing the research design and presenting the results, we return to the question of cost efficiency and what our results imply for policies designed to encourage voter turnout.

METHOD

In September 1998, we obtained a complete list of New Haven’s registered voters. Using this list, we created a data set of all households with one or two registered voters, from which we excluded all names with post office box addresses, as well as those with Yale Station addresses.§ Our experiment was designed to measure the effect of personal canvassing on voter turnout. Through a series of independent random assignments, voters were divided into control and experiment groups.¶ Data on voter turnout were obtained after the election from public records. We observed 4% sample attrition, because voter-turnout information could not be obtained for every person in the sample, the treatment group for the personal-canvassing experiment contained 4,509 people, and the control group contained 23,921.

During each Saturday and Sunday for 4 weeks before the election, we sent canvassers out to contact randomly selected registered voters. The canvassers were paid $20 per hour and were primarily graduate students. For safety reasons, all canvassers worked in pairs, and canvassing ceased at 5:00 p.m., when the sun began to set. This procedure constrained both the pool of available canvassing labor and our ability to contact people who were out during the day.

New Haven has a substantial minority population, as well as a significant non-English-speaking population. More than half of our canvassers were African-American or Spanish speakers, and, when possible, canvassers were matched to the racial and ethnic composition of the neighborhoods they worked. We divided the treatment group into three subgroups to test the relative effectiveness of different political messages. The model text for the canvassers will be discussed later in the paper and is provided in the Appendix.

RESULTS

Table 1 shows the effect of in-person contact on turnout. The upper half of Table 1 compares the turnout rates of three groups: the control group, the treatment group, and the subset of the treatment group that was contacted successfully by our canvassers. The turnout rate for the control group was 44.6%, and turnout among the treatment group was slightly under 47%. We were not able to contact everyone in the treatment group. The turnout rate among those in the treatment group who were actually contacted is slightly over 59%, and the turnout rate for those in the treatment group who were not contacted was 40%. In some previous experimental studies (e.g., refs. 8 and 9), the treatment effect was calculated by measuring the difference between the turnout rate among those actually contacted and those who were not. The group of those who were not contacted is a combination of those in the control group (whom we made no attempt to contact) and those in the treatment group whom the canvassers were unable to contact. Nonexperimental studies that use survey data implicitly make a similar comparison as well, because these studies compare the voting rates of respondents who report contact and those who do not. Aggregating the original control group and those in the treatment group who were not contacted yields a difference between the voting rates of those contacted and those not contacted that is very large, a little over 15%.

Table 1.

The effect of personal canvassing on voter turnout in New Haven

Statistics	Relationship between experimental subgroups and voter turnout
	Control group (no personal contact)	Treatment group	Treatment group (actual contact)
Percentage voting, %	44.63	46.88	59.19
Number of persons	23,921	4,509	1,605
Contact rate, %	—	—	35.60
Estimated effect of personal contact on voter turnout
Turnout differential (2.25%)/contact rate (35.60%) = 6.33% (±2.27% SEM)

This number overstates the effect of canvassing, however. If those voters who are easier to reach are also more likely to vote, which is implied by the low voting rates among those who were in the treatment group but were not reached, then the canvassing effect is partly spurious. To estimate the effect of the treatment properly, we must separate out the treatment effect from the higher probability of voting among those who are easier to contact. One way to make this separation is by augmenting the experimental design. We might have employed a control group where the canvassers make contact and deliver a nonpolitical message.‖ An alternative approach to estimating the treatment effect is outlined by Angrist et al. (18). They consider estimation of treatment effects when a nonrandom subset of those assigned to a treatment group do not receive the treatment. They show that the original random assignment of subjects to the treatment and control group, which is both positively correlated with subjects actually receiving the treatment and, by definition, statistically independent of unobserved factors that might make particular members of the treatment group less likely to receive the treatment, can be used as an instrumental variable. It follows that Eq. 1 is a consistent estimator of the treatment effect:

1

where V_E is the percentage turnout among the experiment group, V_c is the percentage turnout among the control group, N₁ is the number of voters actually contacted, and N_E is the number of voters in the treatment group.

Eq. 1 directs us to find the treatment effect by subtracting the turnout rate of the control group from the turnout rate of the experiment group and then dividing this difference by the observed “contact rate.” Using this formula, we find that the effect of a personal contact is that turnout probability rises by between 6 and 7%. The null hypothesis that canvassing has no effect on turnout can be rejected decisively by using either a one- or two-sided test. For a one-sided test, which is most appropriate given the null hypothesis that canvassing does nothing to increase turnout, the effect of personal contact is found to be significant at the 0.01 level.

We also explored whether the effect of personal contact varied with the content of the canvassers’ appeal. We used three political messages during the canvassing experiment. The messages attempted to capture ideas commonly present in political appeals to voters: (i) by voting, you provide evidence that your neighborhood is politically active, which will increase its political clout (neighborhood solidarity); (ii) voting is your civic duty (civic duty); and (iii) the election is close, and thus there is a chance that the outcome might depend on your participation (election is close). The model text for each of the three messages appears in the Appendix.

Approximately 1,500 people in the treatment group were associated with each of the three messages. Our results lend qualified support to the view that the messages varied in effectiveness. Based on the experimental results, our best guess is that an argument that emphasizes the closeness of the election is most effective. We estimate that the election is close message boosts turnout rates by nearly 10%, compared with the 5% boost observed after delivery of civic duty or neighborhood solidarity. The standard errors associated with these estimates are too large to reject the null hypothesis that the messages had equal effects, and thus these findings are properly viewed as suggestive. Nevertheless, the experimental results are consistent with the predominance of the urgent election is close message in actual get-out-the-vote drives.

Table 2 shows how the effect of canvassing voters varies across voters with different party registrations. The estimated effect of canvassing was slightly higher for unaffiliated voters, though the hypothesis that the effect was the same regardless of party registration could not be rejected. The observed difference in effectiveness of canvassing, with major party voters showing a smaller boost in turnout levels, is consistent with several political explanations. Political parties and activists target registered partisans; thus, partisans are more likely to be encouraged to vote than unaffiliated voters. If there are diminishing returns to prodding, the unaffiliated voters will show greater response. It is also the case that, because partisan voters participate at higher rates, any canvassing of registered partisans was more likely to be wasted on someone who would have voted in the absence of the contact. Finally, the unaffiliated voter may be especially receptive to a nonpartisan appeal. Thus, although the effect we observe falls short of conventional levels of statistical significance, the direction of the effect comports with prior expectations.

Table 2.

Effects of personal canvassing on voter turnout in New Haven by party registration

Statistics (by party registration)	Control group	Treatment group
Democratic voters
Percentage voting, %	48.3	50.5
Number of persons	16,511	3,078
Contact rate, %	—	37.4
Republican voters
Percentage voting, %	50.3	51.7
Number of persons	1,330	261
Contact rate, %	—	31.8
Unaffiliated voters
Percentage voting, %	33.5	36.3
Number of persons	6,080	1,170
Contact rate, %	—	31.7
Estimated effect of personal contact by party registration
Democratic voters
Turnout differential (2.2%)/contact rate (37.4%) = 5.88% (±2.62% SEM)
Republican voters
Turnout differential (1.4%)/contact rate (31.8%) = 4.40% (±10.64% SEM)
Unaffiliated voters
Turnout differential (2.8%)/contact rate (31.7%) = 8.83% (±4.77% SEM)

In addition to testing the effect of different political messages, we conducted an experiment to see whether extracting a commitment to vote enhanced the turnout effects of canvassing. A subset of those in the canvassing treatment group were randomly assigned to receive a message that included an additional component. After hearing the message, this subgroup was asked “Can I count on you to vote on November 3rd?” Note that with this design, we cannot distinguish between the effect of extracting a commitment and the effect of hearing a longer message with an additional repetition of the date of the election. Table 3 shows the results.

Table 3.

Effects of supplementary prompt on voter turnout

Statistics (by type of canvassing)	Control group	Treatment group
No supplementary prompt
Percentage voting, %	44.5	46.2
Number of persons	7,949	1,563
Contact rate, %	—	36.9
With supplementary  prompt
Percentage voting, %	44.7	47.3
Number of persons	15,972	2,946
Contact rate, %	—	34.9
Estimated effect of personal contact by voter turnout
With supplementary prompt
Turnout differential (2.6%)/contact rate (34.9%) = 7.45% (±2.85% SEM)
No supplementary prompt
Turnout differential (1.7%)/contact rate (36.9%) = 4.61% (±3.73% SEM)

There is weak evidence that the supplementary prompt increased the effectiveness of canvassing, though the difference in estimated turnout effects from canvassing with and without the additional prompt cannot pass standard tests of statistical significance. As a practical matter, given the minimal time cost associated with adding the prompt, it would be sensible for those interested in increasing turnout to include it.

Cost Effectiveness.

Using the experimental results, we can approximate the cost of raising turnout by one voter. Assuming that canvassers make 10 contacts per hour and are paid $10 per hour, each contact costs approximately $1. A contact raises the probability of voting by approximately 6.3%, implying that 16 contacts will generate an additional voter, which comes to $16 per voter. Given the brevity of the treatment message, which takes at most 30 seconds to deliver, we believe that the assumption of only 10 contacts per hour is conservative. The calculations can easily be modified by the reader to accommodate a more optimistic assumption about contact rates.

DISCUSSION

Two contrasting characterizations of political participation dominate the literature on voter turnout. One line of scholarship focuses on the institutional and political context in which participation occurs. In this tradition, primary attention is given to voter registration requirements, campaign tactics, and party and interest-group mobilization activities. Another line of work emphasizes the enduring personal traits and psychological orientations that stimulate people to engage in political action. In this line of study, scholars note that turnout levels vary with a citizen’s interest in politics, partisan attachments, feelings of political efficacy, and sense of civic obligation. These social-psychological orientations are shaped by personal characteristics such as age, education, and race. In short, one literature emphasizes personal attributes, whereas the other focuses on the importance of the political environment.

The relative importance of these two lines of explanation is critical to any discussion of why electoral turnout has declined or what to do about it. Social-psychological explanations often focus on demographic trends that have introduced large numbers of young voters with weak party attachments into the electorate, whereas more contextual explanations have stressed the decay of parties and civic organizations that formerly mobilized voters through personal canvassing. If voting is primarily a matter of individuals’ enduring propensities to vote, little can be done about declining turnout rates short of changing the ways in which children are raised by their parents or socialized in schools. On the other hand, those who stress environmental factors propose to ease registration requirements or reinvigorate parties and other mobilizing organizations.

Our findings suggest that environmental effects can make an important difference in turnout rates. The blandishments of canvassers making very brief appeals at voters’ doorsteps raised voter turnout by 6%. For midterm elections, for which citywide turnout hovers around 40%, this effect represents a significant increase in political participation.

The appeals we used were strictly nonpartisan in character, and one wonders how the effects might have differed had we advocated for a certain party or candidate.** This limitation prevents us from drawing direct parallels between our work and the nonexperimental studies of political contact and voter turnout. Nonetheless, the magnitude of the canvassing effect we observe lends credence to the thesis advanced in ref. 15 that falling rates of voter turnout reflect a decline in grass-roots political activity.

The field-experiment methodology avoids some of the problems present in previous work measuring the effect of political contact on voter behavior. Because we control the treatment effect, our analysis does not suffer from the measurement error introduced by false reports of political contact or imprecise questions about contact. The experimental design also eliminates the danger that contact is an endogenous variable, because we contact voters randomly. The major limitation of our approach is that we measure the marginal effects of contact for a specific population under a specific set of circumstances. This criticism applies to some extent to all research, because all studies are limited in scope and time. However, we readily concede that additional research is needed to assess whether our results generalize to other populations and other elections, and that the findings of any single study must be viewed as preliminary until such additional research is performed.

CANVASSING TEXTS

1. Neighborhood Solidarity.

Hi. My name is . I’m part of Vote New Haven ’98, a nonpartisan group working together with the League of Women Voters to encourage people to vote. I just wanted to remind you that the elections are being held this year on November 3rd. Politicians sometimes ignore a neighborhood’s problems if the people in that neighborhood don’t vote. When politicians see a lot of people turning out to vote, they know they should pay attention to issues important to people who live around here. We hope you’ll come out and vote. [Can I count on you to vote on November 3rd?].

2. Civic Duty.

Hi. My name is . I’m part of Vote New Haven ’98, a nonpartisan group working together with the League of Women Voters to encourage people to vote. I just wanted to remind you that the elections are being held this year on November 3rd. We want to encourage everyone to do their civic duty and exercise their right to vote. Democracy depends on the participation of our country’s citizens. We hope you’ll come out and vote. [Can I count on you to vote on November 3rd?].

3. Election Is Close.

Hi. My name is . I’m part of Vote New Haven ’98, a nonpartisan group working together with the League of Women Voters to encourage people to vote. I just wanted to remind you that the elections are being held this year on November 3rd. Each year some election is decided by only a handful of votes. Who serves in important national, state, and local offices depends on the outcome of the election, and your vote can make a difference on election day. We hope you’ll come out and vote. [Can I count on you to vote on November 3rd?].