Letter Regarding “Improving the Nutritional Impact of the Supplemental Nutrition Assistance Program”

In their article, “Improving the Nutritional Impact of the Supplemental Nutrition Assistance Program: Perspectives From the Participants,” Leung et al.¹ present survey data collected through an online service, Amazon Mechanical Turk (mTurk). Unfortunately, the piece does not report enough information to draw reasonable conclusions about generalizability and, more importantly, risks leading readers toward policy conclusions that are not necessarily supported by the data. This letter presents two critical concerns regarding data interpretation and participant selection.

Among participants in the study, only 67.8% of Supplemental Nutrition Assistance Program (SNAP) recipients favored item restriction with no associated benefit amount increase, compared with 83.3% of respondents not receiving SNAP. When presented with the prospect of a 10% benefit increase, non-SNAP support rises to 88.1%; notably, SNAP participants only reach this same level of support when incentivized with a 50% benefit increase. Although the authors suggest “both groups support improvements to the nutritional quality of offered foods,” their data suggest that the degree of support is highly variable. What is clear, however, is that most SNAP participants in the study (89%) support increased benefit amounts as a method of improving nutritional outcomes. No alternative proposal presented to participants in the study had such a strong valence toward support.

These significant divergences between SNAP participant and nonparticipant opinions become even more problematic in light of selection bias. The issue is given a cursory acknowledgement by the authors related to the use of the mTurk system (e.g., participants may be more technologically savvy). Although other possible sources of bias, such as the language in which the survey was administered, are not accounted for, the oversampling of whites and women are noted, yet not detailed. It should be stated that, according to the U.S. Department of Agriculture, 36.7% of all SNAP households heads are white (78.8% here), 26.1% are African American (10.2% here), and 56.7% are female (76.3% here).² Though robust group comparisons cannot be drawn from cross-tabulations of race, gender, SNAP participation, and survey responses, these calculations would have provided some assurance that undersampling of African Americans and men did not significantly influence group outcomes. Furthermore, the abstract does not clarify this non-representativeness, and instead references “SNAP participants” and their support for “limit[ing] purchases of unhealthful foods, specifically sugary beverages.”

This is not merely limited by this sampling issue, but in fact, should have driven the interpretation of results. It seems that even the title of the study ought to contain a qualifier about the participants: how they were selected and how perspectives were elicited, for example. As it is framed now, it is possible—especially considering public access to the article is often limited to the title and abstract—that these findings could be interpreted to be representative of SNAP participants writ large. When claiming to present the perspectives of a vulnerable population, especially on the topic of a highly stigmatized welfare program, researchers should take great care not to understate the limitations of their study design.