Reexamining research on motivations and perspectives of scientists relating to public engagement

Rose et al. (1) make a valuable contribution to the literature on scientists’ engagement with publics. Here, I highlight two issues that may help clarify the precise nature of this contribution.

First, we should consider whether sampling only tenure-track or tenured faculty in physical, biological, and social sciences in US land-grant universities (of which 27 were excluded) gives a representative picture of scientists’ attitudes. For example, no rationale is provided for excluding scientists on more precarious nonpermanent work contracts, even though such contracts are widespread in the United States, and may come with different incentives for and attitudes toward public communication of science.

Second, there is a limitation in how claims about statistically significant differences are communicated. It is easy to find statistically significant differences with large sample sizes like this. Therefore, researchers must apply judgment in evaluating whether such differences tell us something meaningful in order to “convey the most complete meaning of the results” (2, 3). For example, the article (1) reports gender differences in levels of agreement with the five objectives for public communication tested in the study: “Female scientists were consistently more supportive of each objective than their male colleagues, except for persuasion.” However, the effect size for this finding is limited, for example, only R² = 0.2% on the “informing” objective. This means that, if you know only the respondents’ gender, you could accurately predict just one-fifth of 1% of the variance in agreement with the informing objective for public science communication.

Rose et al. (1) overlook this distinction between statistical significance and substantive importance, with no mention in text (or in the supplementary methodological report) that the reported differences are of such tiny magnitudes. For example, the following statement about gender differences should ideally be amended: “Support for these objectives varied by gender: Female scientists were more supportive of holistic objectives, but not of persuasion.” Instead, it would have been more accurate to say, “Support for these objectives varied by gender to a very small extent: Female scientists were very slightly more supportive of the objective of listening to publics (1.6% of variance explained), but not of persuasion (less than one-third of 1% of variance explained).”

Likewise, the claim about age differences should be amended: “Scientists earlier in their careers view engagement more as a way to get people excited about science and less to persuade them.” To be more precise, it should say, “We found that, on average, scientists with more recently completed PhDs were very slightly more likely to view engagement as a way to get people excited about science and less to persuade them (less than one-half of 1% of variance explained on each attitude statement).”

In conclusion, “The quality of a survey is best judged not by its size, scope, or prominence, but by how much attention is given to dealing with all of the many important problems that can arise” (4). This letter highlights some of the limitations in how such problems have been handled in this important study.