Numeracy and the Persuasive Effect of Policy Information and Party Cues

Abstract

Numeric political appeals represent a prevalent but overlooked domain of public opinion research. When can quantitative information change political attitudes, and is this change trumped by partisan effects? We analyze how numeracy—or individual differences in citizens’ ability to process and apply numeric policy information—moderates the effectiveness of numeric political appeals on a moderately salient policy issue. Results show that those low in numeracy exhibit a strong party-cue effect, treating numeric information in a superficial and heuristic fashion. Conversely, those high in numeracy are persuaded by numeric information, even when it is sponsored by the opposing party, overcoming the party-cue effect. Our results make clear that overlooking numeric ability when analyzing quantitative political appeals can mask significant persuasion effects, and we build on recent work advancing the understanding of individual differences in public opinion.

Scholars often conclude that political messages are effective because ingroup and partisan biases dominate the effect of policy information (Bartels 2002). Yet, recent work (Bullock 2011; Nicholson 2011; Boudreau and MacKenzie 2014) shows that policy information can trump partisan cues under some circumstances. Policy appeals are often presented in numerical terms. Such appeals are a very different form of message than values-laden appeals, yet have received little attention in the literature. We aim to understand how variance in numeric ability (also called numeracy; see Peters et al. [2006]) moderates the effectiveness of quantitative political and partisan appeals.

This study presented respondents in a survey experiment with numeric messages, with or without party cues. The results indicate that highly numerate individuals appear to value numeric information, to the point that party cues lose their persuasive effect whenever combined with numeric information, while low-numeracy individuals exhibit partisan effects, regardless of the quality of the numeric policy information. This result demonstrates the importance of accounting for numeric ability when studying numeric political appeals.

Processing Political Messages

Until recently, a commonly held view in political behavior research was that party cues exerted a greater influence on the processing of political messages than more detailed, context-specific information (Cohen 2003). Scholars often concluded that the public is easily persuadable (Zaller and Feldman 1992) and strongly influenced by elites (Zaller 1992). Yet, while the strong effects of party cues are well established (Bartels 2002), recent studies demonstrate that policy information can have a similar or greater effect on attitudes than partisan information, although generally only among subgroups in the population (Bullock 2011; Nicholson 2011; Boudreau and MacKenzie 2014). The politically sophisticated are one important such subgroup: higher sophisticates engage in deeper cognitive processing, which means policy information is incorporated more than party cues (Kam 2005). However, more sophisticated citizens may be better at using different cues, and thus exhibit more of a party cue effect (Slothuus and de Vreese 2010). On the other hand, greater political sophistication might also be required to make sense of the policy information presented (Boudreau and MacKenzie 2014).

Conflicting results also abound in work on another important moderating factor—the need for cognition. Defined as “people’s tendency to engage in and enjoy thinking” (Cacioppo and Petty 1982, 130), studies have shown the need for cognition to be a key individual difference that captures individuals’ propensity to engage in deliberate and systematic cognitive processing (Cacioppo et al. 1996). Yet, studies with political appeals have generally not found significant moderating effects (Kam 2005), with the exception of Bullock (2011). These results suggest that perhaps that moderating effects in the study of political persuasion may be domain or context specific.

NUMERACY

One area that raises this possibility of domain-specific moderating effects is quantitative information. The potential moderator here is numeracy, defined as the ability to process quantitative information (Peters et al. 2006). To date, little political science research incorporates this concept (cf. Kahan et al. 2013), although prior work has controlled for the number of economic and statistical courses taken by respondents (Druckman 2004), and has investigated respondents’ ability to recall quantitative economic information (Ansolabehere, Meredith, and Snowberg 2013).

Research shows that highly numerate individuals derive more meaning and affect from numbers, while the less numerate use more narrative (nonnumerical) information and automatic reasoning when responding to information (Peters 2012). The objective measure of numeracy is moderately related to the need for cognition (Simon, Fagley, and Halleran 2004). Numeracy is also only weakly related to political knowledge and partisanship. It therefore promises to account for a different and more critical individual difference when studying the processing of numeric information. As we show below, analysts risk both over- and underestimating the persuasive effect of quantitative policy information, at least on moderately salient policy issues, if numeracy is not accounted for.

Hypotheses

If high numerates derive more value from numeric information, then they should also be more sensitive to the gradations in numeric messages. Thus, the higher a person’s numeracy, the more persuaded they should be by strong numeric messages (H1). Similarly, if high numerates are more likely to use numeric information, it stands to reason that they might prefer quantitative information to other types of information. Therefore, party cues should exert less persuasive effect among high numerates presented with numeric messages (H2). We do not suggest that highly numerate people are generally less partisan. Rather, we expect high numerates to value numeric information more than partisan information.

Conversely, we expect that low numerates should display a standard party cue effect, as they only superficially process the numeric information and do not value such information. More specifically, a political message associated with their political party should increase their level of persuasion, while a message associated with another political party should decrease their level of persuasion (H3).

Experimental Design

We administered a survey experiment using Amazon’s Mechanical Turk (MTurk) in September 2013 and February 2014. ¹ A nonprobability sample of 1,182 respondents was obtained, after self-identified Independents ² and those who failed our various checks were dropped from the sample. ³ From this final sample, 598 respondents were coded as Democrats and 584 as Republicans. ⁴

MTurk samples tend to be younger, more female and educated, and less diverse compared to the overall American population (Berinsky, Huber, and Lenz 2012). Furthermore, there is evidence that MTurk respondents tend to be more “savvy,” thus enhancing the likelihood of demand effects (Krupnikov and Levine 2014). On the other hand, studies have shown MTurk samples to be more representative than student convenience samples (Paolacci, Chandler, and Ipeirotis 2010), and have supported the validity of MTurk samples through successful replications of known experimental findings (Suri and Watts 2011). Moreover, a recent study found that MTurk subjects are less likely to miscomprehend the content of the vignettes or finish the survey too quickly, compared to a GfK sample (Weinberg, Freese, and McElhattan 2014). In short, while MTurk samples might not be suitable for capturing levels of public opinion, and might provide biased results when demand effects are plausible, they do seem to provide a useful sample when estimating treatment effects that require the attention of the respondents, without easily recognizing the intentions of the experimenter. Such a sample should serve our purposes well, as we are utilizing an experimental design, where some treatments require cognitive effort, and where our expected results run counter to the obvious “demand effect” of partisan responses. Furthermore, MTurk allows us to collect a large sample of respondents, balanced across partisanship, thus overcoming the limitations of a student sample.

The experiment consisted of a 3 (partisan cue: Democratic, Republican, or none) x 3 (type of message: strong numeric, weak (non sequitur) numeric, no numeric information) between-subjects factorial design. Respondents were exposed to information about a supposed study regarding the U.S. criminal justice system. We varied whether respondents were told that the study was commissioned by the Republican Party, Democratic Party, or omitted any party information. We also varied whether the information presented was strong or weak numeric information, or whether we omitted all information. The weaker numeric information contained irrelevant information: references to Department of Transportation spending data. This information should not persuade subjects if they are reading and processing the information carefully, as logically inconsistent messages are less persuasive (McGuire 1960). The weak condition allows us to establish whether subjects are persuaded by a message’s content or are simply impressed by a barrage of figures. ⁵ The length of the strong and weak messages was similar, as were the topics discussed (cost and incarceration rates). ⁶ All the information provided was taken from reports and briefs from think tanks and congressional sources, combining different sources for each message.

The dependent variable is respondents’ agreement with the following statement: “Probation should be used as an alternative form of punishment, instead of prison, for felons.” Subjects were provided with seven response options, ranging from “strongly agree” (= 7) to “strongly disagree” (= 1). ⁷ The mean in the control condition (where no information or party label is provided) is about 2.9, roughly equivalent to a “somewhat disagree” response. The average support in the control condition ranges from about 3.4 for Democrats to 2.4 for Republicans. ⁸ We utilize a persuasion study design, meaning that all the information provided advocates greater support for the policy alternative. A pilot study on MTurk produced strong prior attitudes against the proposal, indicating the appropriateness of the persuasion design. ⁹

Respondents also answered a battery of demographic questions, as well as a fifteen-item political knowledge scale (α = 0.7), two items capturing need for cognition (α = 0.66), and eleven items measuring numeracy (α = 0.74). We created a standardized latent measure of these three constructs using the iterated principal-factor method, and converted each latent measure into a binary variable, with a median split used to divide respondents into high and low values. ¹⁰ Dividing subjects into high- and low-numeracy subgroups for clarity of analysis is common practice in the numeracy literature (Peters et al. 2006). ¹¹

Results

Treatment effects are presented in figure 1. We estimated treatment effects using OLS regression with robust standard errors and dummy variables for each treatment condition (omitting the control). We estimated separate regressions for subjects above and below the median numeracy score. ¹²

Figure 1.

Treatment Effects, Survey Experiment.

These estimates are derived from an OLS estimation of the equation: $Support= \alpha + \beta Condition+ \epsilon$, estimated for high- and low-numeracy subjects separately, using robust standard errors. Higher values indicate greater persuasion (greater support for instituting probation for felons), compared to the control. The exact estimates that figure 1 is based on are available in the supplementary data online, table A5.

When presented with only the weak non sequitur message, devoid of any partisan frame, highly numerate subjects were no more persuaded than in the control condition. Conversely, low-numeracy subjects show a significant persuasion effect when treated with non sequitur information devoid of a partisan frame.

Highly numerate subjects exhibited a significant persuasion effect whenever presented with strong information. This effect persists even when the sponsor of the persuasive information is the subjects’ opposite party. We therefore reject the null H1 and H2. Persuasion among high numerates appears to be driven by strong numeric policy information.

Low-numeracy subjects were less persuaded by any information sponsored by the opposite party, and more persuaded when presented with their own party’s label, regardless of the policy information. As hypothesized, low numerates display party cue effects, in terms of both ingroup persuasion and outgroup backlash; we therefore reject the null H3. Moreover, whenever party cues are absent, low-numerate subjects are persuaded by both strong and weak messages.

We also estimated OLS models using alternative individual differences. ¹³ There is no meaningful pattern of differences in results between high- and low-sophistication subjects, demonstrating that a political knowledge score serves as a poor proxy for variation in the numerical ability of subjects. Further, the pattern of significant results is identical between high and low need for cognition subjects, indicating that need for cognition is also not a good measure of numeric ability. Ultimately, only numeracy reveals significant individual differences in this case.

Conclusion

Consider that the treatment effect of the strong information, opposite party condition, compared to the control, is not significant when pooling all subjects together. A scholar might conclude that policy information has a null effect on persuasion when such information is sponsored by an opposing party. Yet, when we subset the sample by numeracy, sizable and significant differences between treated and control subjects emerge. Pooling the data obscures these sizable effects because the effects for high-numeracy subjects are positive, while these effects are negative for low-numeracy subjects. Compared to the control, high-numeracy subjects in this condition were 1.24 points more persuaded, while low-numeracy subjects were 0.68 points less persuaded, by the same information.

This result constitutes strong evidence that the persuasive effect of numeric policy information is moderated by the individual’s numeric ability and preference, at least in a policy domain of low party polarization and moderately strong prior attitudes, such as criminal justice. ¹⁴ Those high in numeracy respond to the strength of numeric information, the effect of which trumps partisan cue effects. For those low in numeracy, the effect of numeric information is conditional on partisan cues. Our results contribute to the debate about the role of party cues and policy information on public opinion by showing that overlooking individuals’ numeric ability and preferences may lead to incorrect inferences when political and partisan appeals are based on numeric information.

Future work should explore the generalizability of these findings in several ways. First, similar designs should be estimated using representative national samples. Second, the design should be extended to include different policy areas, in order to understand the role of specific prior attitudes and partisan contexts. Perhaps most importantly, our results indicate that individuals might systematically differ in their preferences for different types of evidence and appeals, which has important implications for understanding political attitudes and behavior. Future work should seek to better understand this possible numeric motivation. A worthy area for further inquiry would be analyzing individual differences when presenting subjects with multiple types of political information (e.g., numeric as opposed to values-laden). Do the more numerate, for instance, trust quantitative information more than appeals to their personal values? Scholars could explore the scope conditions for when numeric motivations trump partisan motivations, and connect such numeric preference to the needs for accuracy, consistency, or certainty (Hart et al. 2009), or various dispositions (Gerber et al. 2010).

Supplementary Data

Supplementary data are freely available online at http://poq.oxfordjournals.org/.