Reply to Jern et al.: On asking the right questions

We are delighted that Jern et al. (1) share our interest in the psychopharmacological underpinnings of social behavior (2). It is a topic of growing interest and major significance. It is, however, important to correct some misunderstandings. First, we fully appreciate the concern about type I errors in multiple comparisons; it has been widely discussed in the “genome-wide” genetics literature (3–6), and we addressed it at length in our SI Appendix (2). However, it is also widely recognized that most correction procedures unhelpfully inflate type II errors (7–10). We followed professional advice and current best practice in genome-wide research (3–6). Ironically, had we reported a study on just one neurochemical, no one would have quibbled. (In our paper, we used the term “neuroendocrine” to cover the six neuropeptides/amines/steroids we considered. Biochemically, of course, they have different properties, although from the social point of view they are all equally functional. To avoid confusion, I use the term “neurochemical” here.) The results do not suddenly become nonsignificant simply because we considered several neurochemicals and their interactions simultaneously. Second, although Jern et al. obviously would not be aware of this, we have two smaller independent follow-up samples (currently under review) that show the same pattern. Our results are statistically robust. Third, more importantly, the substantive issue that we address in our paper is the fact that all studies on this topic to date invariably focus on just one social neurochemical (usually oxytocin) without making any attempt to check whether the reported effects might be confounded by the others. Only three of the six we consider actually play substantive roles, and that in itself weakens concerns about multiple comparisons: they are not all significant. Moreover, the strong signature for dopamine was completely unexpected and alone merits attention. Ours is a more nuanced story. Fourth, Jern et al. (1) question our contingency table analysis. Jern et al. test one hypothesis (essentially that dealt with our main analysis), but we test a rather different one: given that some of the associations between SNPs and social domains are stronger than might be expected, how likely is it that these “significant” effects are randomly distributed across the three social domains? That is a perfectly legitimate use of the χ² test. We used conventional significance values simply as a convenient criterion for distinguishing potentially interesting associations from the rest and, in this context, need make no particular claim about their individual statistical significances as such. This is an issue about sampling grain that has been a long-standing problem in quantitative ecology (11). We could have placed our criterion anywhere on the P value continuum. The trade-off is between samples that are too coarse-grained or too fine-grained to show anything at all and something in between; the proper question is whether there is any level of grain in between where something interesting emerges. We show that there is.