Reply to Kalmoe and Mason: The pitfalls of using surveys to measure low-prevalence attitudes and behavior

Kalmoe and Mason (1) argues that our article’s (2) primary contribution is to replicate and extend their book’s findings (3). This is incorrect. We received a draft of their book only after posting our article. Instead, Westwood et al. (2) critiques and improves a widely adopted survey question Kalmoe and Mason previously publicized in multiple outlets (4–6) and republished in their book (3). Further, we consider additional questions by different authors (7). Other researchers building on our method have focused on the same questions because they “have attracted substantial attention and raised alarm” (8).

Our paper (2) demonstrates the pernicious effects of ignoring disengaged survey respondents, particularly when seeking to measure low-prevalence preferences (e.g., for violence) using vague survey questions. False responses from even a small number of disengaged respondents can inflate such estimates. The implications are far reaching for all the estimates in Kalmoe and Mason’s (3) book. Our results (2) suggest that every estimate of average public support for violence and trait aggression in their book (3) is biased, and the vast majority are biased upward. The error we diagnose also biases their (3) estimates of the relationship between individual’s attributes and violence support. And our critique implies their over-time comparisons are similarly biased, because those comparisons require that the share of disengaged respondents remains stable.

Political violence has so many manifestations that a survey question asking about support for violence in general has little meaning. In place of general questions, we (2) argue for questions about specific violent acts to render responses comparable. Survey evidence in their book (3) inadvertently confirms our argument (2) and invalidates their widely reported general violence questions. In two questions, Kalmoe and Mason (3) asked individuals who support political violence what violent act they had in mind. They find that respondents have a wide-ranging interpretation of “violence,” from property crime to physical assault, and some were not even thinking of violence or responded with a nonviolent crime. Their results bolster our conclusion that responses to generalized violence questions are incomparable (9).

We (2) offer statistical and design-based approaches to mitigate biases in measuring support for violence. For example, we establish statistical bounds on average attitudes based on the prevalence of disengaged respondents (1). Kalmoe and Mason (1) ignore this procedure when they assert that we discard disengaged respondents. Contra ref. 1, our findings are robust to the choice of attention check (2). An independent replication of our results using a different attention check and different survey questions finds that disengaged respondents and general questions bias estimated support for violence upward (8). Finally, we’re unsure why Kalmoe and Mason (1) assert we seek an “exact level of violence,” as we show that support hinges on the specificity and severity of violent acts (2).

Naive survey-based estimates of low-prevalence attitudes and behavior yield severely misleading inferences (2). Respondent error or inattention causes upward bias when the true prevalence is small, creating the impression that rare attitudes and behaviors are common. Credulously accepting estimates that suffer such biases conflates artifacts from poor survey design and respondent error with sincere attitudes. Our article offers tools to limit this bias and more accurately estimate public opinion (2).