Significance
The causes of differences among groups of organisms have been a topic of conversations in and adjacent to genetics research for decades. Here, we mathematically generalize a set of arguments from more than half a century ago, showing that they have implications for methods used and conclusions drawn across diverse sub-fields of genetics. In the absence of environmental control or clear understanding of the causes of a phenotype, it is generally difficult or impossible to isolate the aggregate contribution of genetics vs. environment to an observed difference between groups.
Keywords: heritability, quantitative genetics, local adaptation, admixture
Abstract
Without the ability to control or randomize environments (or genotypes), it is difficult to determine the degree to which observed phenotypic differences between two groups of individuals are due to genetic vs. environmental differences. However, some have suggested that these concerns may be limited to pathological cases, and methods have appeared that seem to give—directly or indirectly—some support to claims that aggregate heritable variation within groups can be related to heritable variation among groups. We consider three families of approaches: the “between-group heritability” sometimes invoked in behavior genetics, the statistic used in empirical work in evolutionary quantitative genetics, and methods based on variation in ancestry in an admixed population, used in anthropological and statistical genetics. We take up these examples to show mathematically that information on within-group genetic and phenotypic information in the aggregate cannot separate among-group differences into genetic and environmental components, and we provide simulation results that support our claims. We discuss these results in terms of the long-running debate on this topic.
Understanding the causes and consequences of phenotypic variation is a central goal of many fields of research. Over the last century, researchers in multiple fields have attempted to make a study of the genetic and environmental variation within groups of organisms and use that information to understand the genetic and environmental sources of among-group differences. Such attempts have included suggestions that the explanation of phenotypic differences within groups likely extends to differences among groups (1), suggestions that observed differences may result from adaptation, for e.g., refs. 2–4, and observations that admixed individuals have phenotypes intermediate with respect to members of source populations (5–8).
Richard Lewontin critiqued the first of these arguments in a set of well-known thought experiments (9). In one example, Lewontin imagined two groups of corn plants. Within each group, genotypes vary, but environmental conditions do not. Between groups, there are no average genetic differences, but environmental conditions differ, with one group in more favorable conditions than the other. In this situation, within-group phenotypic variation is genetic (since environmental conditions are the same for all members of the same group), while between-group variation is environmental (since there are no average genetic differences between groups). Lewontin claimed that his thought experiments showed that within-group heritability is irrelevant for determining the causes of among-group differences. Lewontin’s argument was a response to Arthur Jensen, who claimed that within-group heritability estimates suggested a genetic explanation for racial differences in scholastic achievement (1).
In the decades since Lewontin wrote, several researchers have suggested that Lewontin’s thought experiment represents a pathological special case. For example, in behavioral genetics, DeFries (10), replying to both Lewontin and Jensen, built on earlier work by Lush (11) to propose the “between-group heritability,” , aiming to quantify the relative influence of genetic and environmental factors on phenotypic differences among groups. DeFries argued that Lewontin’s thought experiments correspond to specially picked edge cases and suggested that in other cases, provides a useful framework for thinking about group difference.
Here, we show that in contrast to DeFries’ suggestion, the rationale underlying Lewontin’s thought experiments is general and poses challenges for the interpretation of whenever environmental differences among groups are not controlled or understood. In particular, we show that, in line with Lewontin’s intuition, the heritabilities of phenotypic differences between groups are not constrained by heritability within groups: Perfect knowledge of within-group heritability provides no information about between-group heritability. Crucially, even if the heritability of between-group differences is estimated correctly, it leaves the direction of the genetic and environmental components of phenotypic difference unclear.
We also expand Lewontin’s critique to other approaches to similar questions. In evolutionary biology, a statistic called (12) is used to test whether phenotypic differences among groups of wild organisms are consistent with neutral evolution or instead might be better explained by natural selection. We show that a parameter used in sensitivity analyses involving can be understood as a function of . Using our results about and their relationship to , we show the sensitivity analysis that is typically undertaken when using to test for evidence of divergent selection on a phenotype may be conservative, especially if the phenotypic differentiation among groups is small. Moreover, we highlight the difficulty of interpreting and in the presence of environmental differences between groups, adding to recent arguments by Harpak and Przeworski (13).
In genetic anthropology and statistical genetics, variation in ancestry among admixed individuals is sometimes used to make claims about the sources of trait variation among groups. For example, correlations between an individual’s phenotype and ancestry proportions may be taken as evidence that an observed phenotypic difference between source populations is genetically caused. As we show and has been mentioned previously as a cautionary point, e.g., refs. 5, 7 and 14, the genetic effect of ancestry may be confounded with environmental variation affecting the phenotype. Approaches to heritability estimation based on patterns of local ancestry in admixed populations (15)—as opposed to global ancestry proportions—have also been used to calibrate claims about the sources of group differences (16). We show that the variation due to local ancestry that they leverage has no clear relationship with between-group trait differences.
We discuss our results in terms of their shared origin in gene-environment confounding and relate them to previous claims (9, 17, 18). The complications we point out arise under simple additive models, and the situation is only more complicated in models with interactions among genetic or environmental influences. We defer a brief consideration of gene-by-environment interaction to the Discussion.
Between-Group Heritability Is Unconstrained by Within-Group Heritability and Underdetermines Genetic and Environmental Differences among Groups
Lush (11) developed the between-group heritability, , to quantify the relative influence of genetic and environmental factors on a trait that differs among groups. Lush’s interest was in heritability of differences among families of organisms subject to artificial selection, with the goal of weighting family membership appropriately when selecting organisms for breeding. Lush worked in applications in which many aspects of among-group variation in the environment could be controlled or randomized.
Consider a phenotype value Y formed from additive genetic and environmental components, i.e., for individual i. It is standard to define the (narrow-sense) heritability as , where the additive genetic variance is the variance of the genetic component (), assuming no epistasis or dominance, and is the variance of the phenotype. If the sample is divided into groups, we can also define variance proportions accounted for by group membership in either traits themselves or in the genetic components of those traits. For example, with respect to the phenotypic variance, we can define , with the between-group component of the phenotypic variance (19). Call the analogous variance proportion for the genetic variance r. Then, when groups are large, the “between-group” heritability is
| [1] |
Subsequently, DeFries (10) took up this formalism to argue against the generality of Lewontin’s thought experiment. With some algebra, he showed that the between-group heritability is mathematically related to the within-group heritability, . Specifically, defining r and t as in the previous paragraph, he showed that
| [2] |
if the groups are large. DeFries claimed that Lewontin’s thought experiments—one described above, and another involving inbred lines reared in different environments—could be explained as pathological cases in which either r or t is equal to , and suggested that the relationship in Eq. 2 could be used to help interpret observed group differences in other cases. He displayed a table showing between-group heritabilities implied by various values of r, conducting a sort of sensitivity analysis.
An immediate problem with application of Eq. 2 is that in the absence of environmental controls, it is impossible to estimate r, the proportion of the variance in the genetic component of the trait accounted for by group membership, a point that DeFries conveyed (10). Despite this, some researchers in behavior genetics, including DeFries himself, took up DeFries’ formalism and argued for its relevance for understanding variation among groups of humans, generally in the context of discussions of racial differences in IQ or scholastic achievement (20–22). In particular, Jensen (23–25) repeatedly cited the definition of between-group heritability as a linear function of within-group heritability in arguments for the value of heritability estimates for understanding sources of difference among groups. Feldman and Lewontin (17) criticized the expression for in Eq. 2 as tautological, in that r is better viewed as a term that simply relates the definitions of and rather than being an independently varying ingredient of a prediction of from . Here, we concur with Feldman and Lewontin’s interpretation, showing that, despite the appearance of Eq. 2, the between-group heritability is unconstrained by the within-group heritability. Thus, within-group heritability, on its own, provides no information about between-group heritability (SI Appendix, Text and sections S1.1–S1.2). We also show that even if r is known, still underdetermines the group differences, such that opposing interpretations of among-group variation are possible under the same (known) r and .
In the specific case of two groups, the important quantities are , the difference between the mean genetic values of the two groups, and , the difference between the mean environmental values of the two groups. Then, under a standard quantitative-genetic model,
| [3] |
Eq. 3 is useful to highlight several properties of between-group heritability that generalize to cases with more than two groups. First, this expression for can be derived without reference to within-group heritability or any quantities that contribute to within-group heritability (SI Appendix, section S1.2), indicating that even perfect knowledge of heritability within groups conveys no information about genetic differences between groups without further information or assumptions.
Second, it shows that the between-group heritability is not bounded between 0 and 1: In particular, if the environmental and genetic effects point in different directions, such that the magnitude of the phenotypic deviation, is smaller than the magnitude of the genetic deviation, , then the between-group heritability will be larger than 1. We illustrate this in Fig. 1A, which shows the value of across a range of genetic and environmental deviations. One striking feature of the behavior of is that it rapidly approaches infinity when and that larger genetic deviations result in a larger range of environmental deviations over which .
Fig. 1.

Properties of . (A) The behavior of in two populations for different values of the genetic deviation and environmental deviation . Each curve corresponds to a different genetic deviation. (B) Two genetic deviations that are consistent with an observed with two populations. The circle shows all genetic deviations that hold the genetic variance constant, while the line shows all genetic deviations that sum to 0. The intersection of the line and circle shows the genetic deviations that are consistent with a given . (C) An infinite number of genetic deviations consistent with an observed with three populations. The sphere shows all genetic deviations that hold the genetic variance constant, while the plane shows all genetic deviations that result in the genetic deviations summing to 0. The intersection of the sphere and the plane, indicated by the purple circle, contains the values of genetic deviations from the grand mean that will hold constant.
Finally, the between-group heritability is uninformative about the direction of environmental and genetic differences. For the two-population case, this is reflected in the fact that given knowledge of the phenotypic deviation, δ, and the between-group heritability, there are two possible directions of genetic effects that are consistent with the data, namely . In SI Appendix, section S1.3, we show that with more than two groups, there are an infinite number of possible configurations of genetic and environmental deviations that are consistent with a given . To give some intuition for this result, we start with an alternative representation of the two-group scenario for groups of equal size (Fig. 1B). The two axes indicate genetic deviations of the two group means from the grand mean, and . Conditional on a given mean phenotypic difference between the groups, δ, obtaining a given value of requires that the sum equal a constant, solutions of which are represented by the orange circle. Additionally, because and are deviations from the grand mean, they must sum to zero, giving the green line corresponding to . The two purple points where the line and circle intersect represent the two configurations of the between-group genetic difference that lead to the same value of .
With a larger number of groups, the situation is entirely analogous, only in higher dimension. Fig. 1C shows a schematic of the three-group case. The axes again represent the genetic deviations of each group from the grand mean. The combinations of genetic deviations that result in the same (conditional on phenotypic differences) are now an orange sphere, rather than a circle. And the genetic deviations from the grand mean consistent with the condition that all genetic deviations sum to zero fall on the blue plane (rather than a line). The intersection of the sphere and the plane, shown in purple, gives a circle of genetic deviations on which all have the same while also being consistent as deviations from a grand mean. With more than three groups, the sphere becomes a hypersphere, the plane becomes a hyperplane, and the same basic relationships hold.
Within-group heritabilities, on their own, provide no information about the causes of between-group differences, in part because any configuration of group means for the genetic contributions to the trait is possible under some value of r. The value of r itself can be studied under evolutionary models, and in fact, it is closely related to the quantity studied in evolutionary quantitative genetics, which we consider in the next section.
PST Analyses Can Understate or Overstate the Possibilities for Adaptive Differentiation
The statistic is a measure of the genetic component of trait variation among populations. For diploids, in terms of the variance proportion r discussed in the previous section, it is equal to (26). When traits evolve neutrally, is expected to be equal to at putatively neutral sites, which measures the genetic variation among populations (27). Thus, by testing whether is significantly greater than , it is possible to determine whether there is more genetically explained trait variation among populations than expected under neutral evolution (28–31). Such a pattern is consistent with divergent selection causing trait differences among groups. On the other hand, lower-than-expected values of may be taken as evidence of stabilizing selection on a shared optimum.
However, to estimate requires arduous common-garden experiments that eliminate environmental confounds. Such common-garden experiments may be difficult or impossible to carry out in some organisms. Thus, it is common in the literature to use a version of calculated from phenotypes measured on wild organisms rather than organisms raised in a common garden for e.g., refs. 2–4. This statistic computed from observations of wild organisms, often labeled , is a function of the between-population phenotypic variance, , the within-population phenotypic variance, , as well as parameters a and b, which are interpreted as determining the fraction of between- and within-population phenotypic variance that is due to genetics, respectively. For diploids,
where the parameter . Because and can be estimated from purely phenotypic data but c cannot, it is common practice in the literature to explore values of c between 0 and 1, seemingly motivated by a suggestion that the genetic component of between-population differences must be proportionally smaller than the genetic component of within-population differences. A well-appreciated point is that if the choice of c is too large, then analysis can be anticonservative, leading to spurious inferences of divergent selection. In effect, environmental differences are mistaken for genetic ones, leading to a false rejection of neutral evolution. In Materials and Methods, we show that the coefficients for are , the between-group heritability, and , the within-group heritability, so that the correct value of c to obtain is
| [4] |
As shown above, is unconstrained by any measure of genetic and environmental variability within groups. Moreover, because is not constrained to be between 0 and 1, a sensitivity analysis that explores values of c between 0 and 1 may lead to extremely conservative tests for divergent selection. Finally, because both and depend on the variance of between-population means, any evolutionary scenarios that produce identical will also produce identical and , indicating that knowledge of or is not sufficient for disentangling the directions of environmental and genetic effects.
Simulated Example: Evolution in the Face of an Environmental and Optimum Shift.
To demonstrate the impact of our results on evolutionary analyses, we performed population-genetic simulations using SLiM (32). In particular, we developed a model in which very different underlying evolutionary forces result in identical and , indicating that these statistics are insufficient for characterizing evolutionary history and may be misleading in some circumstances. The scenario we considered is outlined in Fig. 2 A and B and inspired by a recent perspective by Harpak and Przeworski (13): A population evolves with a trait under stabilizing selection until a population split. In one of the populations, the environment and stabilizing selection remain constant; however, in the other population, the environment shifts, along with the optimum trait value. We explore two scenarios: one in which the environment shifts in the same direction as the fitness optimum for the trait (“concordant,” panel A), and one in which they shift in opposite directions (“discordant,” panel B). We use the same optimum shift in both cases, so that the simulations result in a phenotypic difference of , and set the environmental shift to maintain at equilibrium.
Fig. 2.

Simulation of a evolutionary scenarios. (A and B) Description of the evolutionary scenarios. A population evolves until it splits into two subpopulations of equal size. At the time of split, one of the subpopulations experiences both an environmental shift (in blue) and an optimum shift (red). The environmental shift may either be in the same direction as the optimum shift (concordant, panel A) or in an opposite direction (discordant, panel B). (C and D) Evolution of genetic and phenotypic values over time. The horizontal axis shows the number of generations since the split, and the vertical axis shows the value of either the genetic difference between the two populations or the phenotypic difference between the two populations. Concordant shifts are shown in Panel (C), discordant shifts in panel (D). (E and F) Evolution of between- and within-population heritability. The horizontal axis indicates time since the population split, and the vertical axis shows the value of different summaries of phenotypic variation, and ; median values are shown with a solid line, and the shaded band indicates the range of values between the 10th and 90th percentiles. Concordant shifts are shown in panel (E), discordant shifts in panel (F).
Fig. 2 C and D show the evolution of the difference in phenotypic and genotypic values between the two populations over the course of adaptation. In both directions of environmental shift, the phenotypic differences converge to , consistent with identical optimum shifts in both situations. However, the genetic differences display opposite dynamics, depending on the direction of the environmental shift. If the environmental shift is in the same direction as the optimum shift (panel C), then the adapting population will have a mean genetic value that is less than that of the population that did not experience an environmental shift; in the other scenario (panel D), the adapting population will have a mean genetic contribution to the trait that is greater than the population that did not experience a shift. Thus, to compare the impact of stabilizing selection with environmental change, both the common-garden genetic values and the in situ phenotypic values are necessary.
Fig. 2 E and F show that at equilibrium by construction. However, the two scenarios approach that value in different ways. When the environment and optimum shift in the same direction, the average approach to equilibrium is monotonic and smooth (panel E). However, when the optimum and environment shift in opposite directions, becomes large when the phenotypic difference between the populations is near 0, and comes back down from infinity to settle at the equilibrium value (panel F). It is also much noisier across simulation replicates, as evidenced by the middle 80% of the distribution spanning three orders of magnitude. This is mostly due to the denominator being near zero for a large portion of the simulations (SI Appendix, Fig. S1). On the other hand, the long-term expected value of is constant across the simulations and identical regardless of the direction of the environmental shift, as expected, because within-population variability will not change when there is an optimum shift. As emphasized in the previous section, knowledge of at any time during adaptation is not sufficient to characterize the direction in which selective forces are acting.
Finally, we examined calculations and their impact on the sensitivity of a test to reject neutral trait evolution in these scenarios. In each simulation, we computed across an array of values of c and also computed the true . Fig. 3 A and B show the behavior of the statistics and the test of neutral evolution, respectively, when the environment and optimum trait value shift in the same direction. Here, the behavior of for a given value of c is monotonic, but at different points during the course of adaptation, the same value of c can produce either a conservative or anti-conservative estimate of from . Early in adaptation, there should be very little genetic signal of adaptation, and rejects the null hypothesis approximately 5% of the time at the 5% significance threshold, showing that the test is well calibrated. However, because the environmental shift results in large phenotypic differences, every test initially results in high false positive detection of adaptation. Subsequently, as adaptation proceeds, detects adaptation with high power, although it loses power after approximately 300 generations, because the neutral divergence between populations increases substantially. However, for values of , as typically explored in the literature, quickly becomes smaller than , and tests using fail to reject the null hypothesis significantly earlier into adaptation. Fig. 3 C and D show that when the environment and optimum shift in opposite directions, the behavior of remains monotonic, but the behavior of for a given c becomes nonmonotonic, reaching a minimum when the phenotypic difference between populations is small and then increasing again. This results in complicated behavior of the test in comparison with the test, in which can go from anti-conservative to conservative, and then back to anti-conservative. Finally, the trajectories of are identical regardless of the direction of the environmental shift, in line with the fact that knowledge of is not informative of the direction of environmental and genetic shifts.
Fig. 3.
Evolution of and . In the Top row, the horizontal axis shows the time since the population split, and the vertical axis shows the value of either or , with the different colors indicating different values for the choice of c in the formula. In the Bottom row, the horizontal axis shows the time since the population split, and the vertical axis shows the fraction of tests in which the null hypothesis of neutral evolution is rejected at the level because . The colors are the same as the top row. (A and B) The environment and optimum shifts are concordant. (C and D) The environment and optimum shifts are discordant.
Trait Correlations with Global Ancestry or Genome-Wide Local-Ancestry Sharing Do Not reveal the Causes of Between-Group Difference
Global Ancestry.
We have seen that efforts toward quantifying the genetic contribution to differentiation among groups can be confounded by environmental variation. Some have suggested that admixed individuals present opportunities to resolve this difficulty and estimate the genetic component of between-group differences. One suggested approach is to examine the correlation between trait values and global ancestry proportions among admixed individuals (5–8, 14, 33). If trait value is positively correlated with the fraction of global ancestry from source population A, then that correlation might be taken as evidence suggesting population A is genetically predisposed to larger trait values than other source populations contributing to the admixed population.
However, this approach depends strongly on assumptions about the relationship between ancestry and environmental effects. We show in SI Appendix, section S.2.1.3 that the expected phenotype of an individual with ancestry fraction from source population 2 in a two-source admixture is
where is the mean genetic contribution to the trait in population i and is a function that expresses how the conditional expectation of the environmental effect depends on an individual’s global ancestry fraction. If the environmental and genetic effects of global ancestry are confounded in the admixed population (i.e., if the correlation of Y and is nonzero), then it may be impossible to recover . As a simple example, consider a case in which individuals with experience environmental effects with expectation , individuals with experience environmental effects with expectation , and there is a linear gradient between those extremes for individuals of intermediate admixture fraction. If we again define the difference between mean environmental effects to be, , then , and
Under this formulation, it would be possible to use linear regression to estimate the slope (in terms of ancestry proportion θ) and intercept given a sample of admixed individuals with measured admixture fractions and phenotypes, but the resulting slope would provide information only about the sum of the genetic and environmental effects of global ancestry, not their relative contribution or even their direction.
A similar approach might be applied to admixed siblings, where the correlation with trait value is computed only with respect to variation in realized global ancestry between full siblings, who have the same expected global ancestry proportions on the autosomes. Limiting to within-sibship comparisons would remove some confounds of global ancestry proportion, but it would not exclude environmentally mediated phenotypic differences caused by variation in global ancestry. For example, in African Americans, colorism (34) might lead to systematically different treatment as a function of ancestry fraction, even within sets of full siblings.
Local-Ancestry Heritability.
Although variation in global ancestry may be confounded with environmental variation, one might imagine that variation in the genome-wide sharing of local ancestry segments could provide a way forward in identifying the genetic vs. environmental basis of among-group difference.
Zaitlen et al. (15) proposed a novel estimator of heritability applicable to individuals from admixed populations. The estimator works on principles similar to SNP heritability estimators (35), such as implemented in GCTA (36). SNP heritability methods work by measuring the degree to which aggregate sharing of alleles at genotyped SNPs among people who are not closely related is correlated with similarity on a phenotype. A strength of SNP heritability methods is that they avoid potential biases stemming from environmental influences shared among closely related people. A weakness is that they cannot inform about contributions to heritability from variants that are not well tagged by SNPs included in the genetic relatedness matrix, in particular rare variants. In local-ancestry heritability estimates, rather than examining shared SNP genotypes, one examines the sharing of segments of local ancestry in an admixed population. For example, consider pairs of individuals from an admixed population formed from two source populations, where both members of all pairs have 50% of their ancestry from each source population. If a trait is heritable, then a pair whose genetic segments from source population 1 fall in exactly the same locations might be expected to be slightly more phenotypically similar than another pair who share no local ancestry segments. Heritability estimates based on local ancestry offer the potential to address one of the main issues facing SNP heritability estimates, since segments of local ancestry tag both common and rare variants. The resulting estimate has also been taken as a proxy for c in analysis (16), though Zaitlen et al. (15) made no claims about revealing the causes of phenotypic differences among source populations.
Despite the promise of the approach, heritability estimates based on local ancestry face their own challenges. Specifically, directional or stabilizing selection in the history of the source populations can lead to upward or downward biases in heritability estimates, respectively (37). In SI Appendix, section S2, we consider this problem and describe a related one that, in a pathological worst case, can lead to heritability estimates near 0 when the true heritability is 1. This problem is most pronounced when the trait under consideration has experienced selection in the source populations, a scenario that Zaitlen et al. (15) recognized as potentially problematic for their estimator.
In SI Appendix, Text and section S2, we decompose the genetic variation for a phenotype in a population having experienced several generations of admixture between two source populations into a component due to variation in global ancestry, a component due to the placement of local ancestry segments conditional on global ancestry, and a component due to randomness in genotype conditional on local ancestry. The variance component principally used by local-ancestry heritability estimation is the variance due to placement of local ancestry segments, conditional on global ancestry,
| [5] |
Here, θ is a random variable representing global ancestry fraction from source population 2, ℓ is the number of (unlinked) loci considered, is the effect size of an effect allele at locus j, and are frequencies of the effect allele in populations 1 and 2, and indicates the population variance.
Under this model, the difference in genetic means between the source populations is
where the overbar denotes the population mean. However, the expression in Eq. 5 does not provide a means to estimate , since the average of the quantity does not appear in Eq. 5; only its variance does. The mean value of is not constrained by the variance of the values. The approach of Zaitlen et al. (15) can be applied with or without a fixed effect for global ancestry fraction. However, comparing these results does not reveal the genetic component of the between-group difference if there may also be environmental effects associated with ancestry fraction, for the reasons discussed in the previous subsection (SI Appendix, section S2.3).
Discussion
We showed that knowledge of the heritability of within-population differences does not constrain the heritability of between-population differences at all, as discussed by Lewontin (9). Our results establish Lewontin’s interpretation generally.
Moreover, even if the between-group heritability were estimable, it is consistent with potentially infinitely many configurations of genetic differences among populations and cannot be used to infer the direction(s) of group differences in genetic contributions to a trait. This second point is especially important. Evolutionary models may be able to identify plausible ranges, under given assumptions, for the variance proportion of the genetic component of the trait accounted for by group membership, r, since r is a function of , which is studied in evolutionary quantitative genetics. However, even if r (and thus between-group heritability, which has r as a key ingredient) can be constrained somewhat, the causes of group difference remain ambiguous without a clear understanding of environmental differences among groups and their relevance for the trait. This point was also anticipated by a verbal argument, this one by philosopher Ned Block (18).
Although our analysis of was partially anticipated by Lewontin et al. (9, 38–41), the existence of has been invoked for decades, up to the present, to suggest that within-group heritability is informative about among-group differences (20–25). As one example, Warne (22) writes “The existence of the equation [relating to ] alone shows that the claim that ‘the genetic basis of the difference between two populations bears no logical or empirical relation to the heritability within populations’ (Lewontin, 1970, p. 7) is incorrect.” Our work clarifies that Lewontin’s claim is correct in general and not only in the special cases he used for illustration. The relationship between and is rather like the relationship between a random variable’s expectation and its variance. One can write the variance as a function of the expectation (i.e., ), but unless further assumptions are made, knowledge of the expectation alone gives no information about the variance. Roseman and Bird (42) recently arrived at a similar position, concluding that there is no relationship between and under an additive genetic model...unless evolutionarily explicit models are used to relate them.
These findings have important implications not just for quantifying the genetic component of among-group differences, but also for methods of detecting genetic adaptation using phenotypic data from multiple populations. As we showed, the phenotypic analogue of , called , is not constrained by within-population heritability. In fact, because the between-population heritability is not bounded from above by 1, common sensitivity analyses, which assume that the between-population component of genetic differentiation is smaller than the within-population component, can be very conservative. has been criticized (12, 43) and is often treated cautiously as a result. However, to our knowledge, the fact that c can be much larger than is not widely understood, nor is the fact that it can achieve very large values in realistic evolutionary scenarios.
The same fundamental problem that plagues and also prevents the separation of genetic and environmental components of differences among source populations via study of ancestry variation in admixed populations. Variation among admixed people in global ancestry proportions may be associated with both genetic and environmental effects, leading to confounding. Variation among admixed people in the sharing of local ancestry segments provides information that is sometimes useful in heritability estimation, but the resulting estimates do not provide a way forward for estimating differences in the genetic contribution to the trait among source populations, since local-ancestry heritability estimates depend principally on a variance component that is not closely related to the genetic contribution to differences among groups.
All of the phenomena we have discussed so far arise under the simplest possible models, in which phenotypes arise via additive influence of genotype and environment. In the presence of gene-by-environment interaction, the role of genetic factors in explaining group differences itself depends on the environment in which the groups exist. Even careful control of the environment then provides only a partial picture, since it tells us only about the genetic contribution to group difference in the environment studied, which may well change or reverse in a different one (38). For example, Chakraborty and Weiss (14) studied diabetes incidence in admixed populations with varying proportions of European or Native American ancestries. The increased incidence of diabetes with more Native American ancestry that they observed ran against what seemed to them the most obvious environmental risk factor, the degree to which a group’s diet was westernized. However, as they noted, “earlier prevalence studies did not reveal a high occurrence of [diabetes] in pre-World War II populations of Amerindians” suggesting the inadequacy of a purely genetic model, and a potential role for gene-environment interaction in explaining the observed group differences.
Clearly, our results do not prevent the identification of specific genetic or environmental factors that might contribute to group differences, in the sense that, if their influence were removed, the magnitude or direction of a group difference might change. For example, in domestic dogs and related canid species, a specific variant in IGF-1 appears to be an important factor in explaining body-size differences (44). This variant is associated with body size within and across species and breeds, in a broad set of different genetic backgrounds and different environments, and it plausibly has a regulatory function that relates it to body size. On the environmental side, exposure to known toxins such as air pollution can lead to health problems such as asthma or some cancers, and exposure is sometimes stratified by race or socioeconomic status, plausibly contributing to health disparities (45, 46). The problems we identify arise when simplistic assumptions are made about aggregate contributions of genetic or environmental differences to group differences, particularly when the sources of variation in the phenotype are not understood. In some cases, a single causal factor might be identified, such as a genotype that causes a Mendelian disease, although even such seemingly straightforward cases may be more complicated and susceptible to environmental modification, as in the case of phenylketonuria (47). But when the causes of variation in a phenotype are poorly understood and most phenotypic variation is unexplained, projection on the basis of known factors is particularly difficult. For example, in humans, attempts to explain group differences in terms of polygenic scores—predicted trait values calculated from genotypes, which are typically noisy predictions—are fraught with difficulties (48, 49).
When it is possible, the best solution to the problems we point out is careful control of the environment. Common-garden observations support the original uses of by Lush (11) and applications of . Departures from the common garden are unavoidable for studies of humans and of some wild organisms. At the same time, if the environment is not controlled, then the sources of among-group differences become ambiguous, and this ambiguity cannot be resolved by study of within-group variation in aggregate.
If control of the environment is not possible, it may sometimes be possible to observe that a phenotypic difference is persistent across a broad set of environments that vary in all dimensions that might be relevant for the trait. With respect to the methods discussed here, they may allow for firmer conclusions if the environmental sources of trait variation are understood and measured or modeled, even if they cannot be controlled. In practice, it is often difficult to make such conclusions—the relevant environmental variables are unknown, and it is difficult to tell whether they have been explored. In fact, Lewontin (9) contained a third thought experiment to this effect. In the same case as described in the introduction, with genetic variation within groups but no genetic variation between groups raised in different soils, Lewontin asks us to imagine a chemist making a survey of the soils. The chemist finds a large difference in nitrate concentrations, but correcting the difference only partially lessens the phenotypic difference between the groups, leading researchers to conclude incorrectly that the remaining phenotypic difference is genetic. In fact, a much subtler difference in zinc levels is “the real culprit.” This thought experiment was inspired by the long delay in the discovery of the importance of trace minerals for plant development, “because ordinary laboratory glassware will leach out enough of many trace elements to let plants grow normally.” Lewontin asked rhetorically, “Should educational psychologists study plant physiology?” We might ask today how we can come to understand the traits we care about well enough to distinguish nitrate cases from zinc cases.
Materials and Methods
A Linear Model of Quantitative Traits within and between groups.
We model the phenotype of an individual j in population i as the sum of genetic and environmental components, G and ϵ, respectively,
where and are the genetic and environmental deviations, respectively, of individual j from the mean genetic and environmental values for group i. This model is applicable to a trait regardless of genetic architecture, as long as there are no gene-by-environment interactions.
Computing the between- and Within-Group Heritability.
In terms of the linear model formulation, the definition of the between-group heritability is
and the within-group heritability is
In SI Appendix, section S1.2, we show that with these definitions, one can recover the relationship between and derived by DeFries (10).
The Parameters Used in PST.
In terms of the linear model formulation, for diploids, the definition of is
and the definition of is
In terms of the variance proportions t and r in DeFries’ expression for between-group heritability, it follows that .
For we need
and similarly
Stabilizing Selection Simulations.
We simulated phenotypes under selection using SLiM (32). In all simulations, an initial population of 500 diploid individuals with genomes of 1,000,000 positions, with a mutation rate of per generation and a recombination rate of (to approximate free recombination). New mutations have an effect on the trait drawn from a Normal (,) distribution. Environmental effects are Gaussian with variance 2. We impose Gaussian stabilizing selection with an optimum at 0 and variance of 100. After 20,000 generations, the populations split into two populations of 20,000 diploid individuals. We then consider two scenarios. In both scenarios, one subpopulation maintains an optimum at 0, while the other experiences an optimum shift to an optimum at 1. In one scenario, the environment shifts to a mean of −2, in the opposite direction of the optimum shift. In the other scenario, the environment shifts to +4, in the same direction as the optimum shift. The simulations are run for 500 generations after the population split. See Table 1 for a summary of simulation parameters. For each scenario, we conducted 500 replicate simulations.
Table 1.
Simulation parameters
| Scenario | Concordant | Discordant |
|---|---|---|
| N | 500 | 500 |
| Mutation rate | ||
| Recombination rate | ||
| Mutation distribution | N(0,.005) | N(0,.005) |
| Baseline environment distribution | N(0,2) | N(0,2) |
| Baseline selection kernel | N(0,100) | N(0,100) |
| Time of population split | 20,000 | 20,000 |
| Optimum shift | +1 | +1 |
| Environment shift | +4 | -2 |
Testing for Adaptation Using the Lewontin–Krakauer distribution.
To test for adaptation in our simulations by comparing or to , we first computed ratio-of-averages genome-wide (as implemented in SLiM’s calc_Fst function) on the basis of neutral markers that were not associated with the trait in the simulations. The Lewontin–Krakauer (50) test uses the statistic
which has distribution with degrees of freedom, where n is the number of demes (in our case, ). For the tests using , we substituted for in the definition of the Lewontin–Krakauer test statistic. We note that we computed the variance in group means without using Bessel’s correction, as shown by Weaver (51) to be the correct formula for Nei’s (52).
Supplementary Material
Appendix 01 (PDF)
Acknowledgments
We thank K. Bird, G. Coop, A. Gusev, A. Harpak, M. Pennell, and M. Przeworski for helpful comments on an earlier draft of the manuscript, and members of the Edge, Mooney, and Pennell labs for helpful conversations. We are also grateful for the helpful comments of the anonymous reviewers, which improved the manuscript. We acknowledge support from NIH grant R35GM137758 to M.D.E.
Author contributions
M.D.E. conceived the study; J.G.S. and M.D.E. designed research; J.G.S. and M.D.E. performed research; J.G.S. analyzed data; and J.G.S. and M.D.E. wrote the paper.
Competing interests
The authors declare no competing interest.
Footnotes
This article is a PNAS Direct Submission.
Data, Materials, and Software Availability
Code used in this paper can be found at https://github.com/Schraiber/heritability (53).
Supporting Information
References
- 1.Jensen A. R., How much can we boost IQ and scholastic achievement? Harv. Ed. Rev. 39, 1–123 (1969). [Google Scholar]
- 2.Salloum P. M., Lavery S. D., de Villemereuil P., Santure A. W., Local adaptation in shell shape traits of a brooding chiton with strong population genomic differentiation. Evolution 77, 210–220 (2022). [DOI] [PubMed] [Google Scholar]
- 3.Vitek N. S., McDaniel S. F., Bloch J. I., Microevolutionary variation in molar morphology of Onychomys leucogaster decoupled from genetic structure. Evolution 76, 2032–2048 (2022). [DOI] [PubMed] [Google Scholar]
- 4.M. B. Castellani et al., Genetic drift versus natural selection affecting the evolution of spectral and functional traits of two key macrophytes: Phragmites Australis and nuphar lutea. Fresh. Biol. n/a (2023).
- 5.Fernández J. R., et al. , Association of African genetic admixture with resting metabolic rate and obesity among women. Obesity Res. 11, 904–911 (2003). [DOI] [PubMed] [Google Scholar]
- 6.Molokhia M., et al. , Relation of risk of systemic lupus erythematosus to west African admixture in a Caribbean population. Hum. Genet. 112, 310–318 (2003). [DOI] [PubMed] [Google Scholar]
- 7.Kosoy R., et al. , Relationship between hypertension and admixture in post-menopausal African American and Hispanic American women. J. Hum. Hypert. 26, 365–373 (2012). [DOI] [PubMed] [Google Scholar]
- 8.Chan Y., et al. , Genome-wide analysis of body proportion classifies height-associated variants by mechanism of action and implicates genes important for skeletal development. Am. J. Hum. Genet. 96, 695–708 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Lewontin R., Race and intelligence. Bull. Atom. Sci. 26, 2–8 (1970). [Google Scholar]
- 10.J. C. DeFries, “Quantitative aspects of genetics and environment in the determination of behavior” in Genetics, Environment and Behavior: Implications for Educational Policy (Academic Press, Oxford, England) (1972), p. 324.
- 11.Lush J. L., Heritability of quantitative characters in farm animals. Hereditas 35, 356–375 (1949). [Google Scholar]
- 12.Brommer J. E., Whither Pst? The approximation of Qst by Pst in evolutionary and conservation biology J. Evol. Biol. 24, 1160–1168 (2011). [DOI] [PubMed] [Google Scholar]
- 13.Harpak A., Przeworski M., The evolution of group differences in changing environments. PLoS Biol. 19, 1–14 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Chakraborty R., Weiss K. M., Frequencies of complex diseases in hybrid populations. Am. J. Phys. Anthropol. 70, 489–503 (1986). [DOI] [PubMed] [Google Scholar]
- 15.Zaitlen N., et al. , Leveraging population admixture to characterize the heritability of complex traits. Nat. Genet. 46, 1356–1362 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Zaidi A. A., et al. , Investigating the case of human nose shape and climate adaptation. PLoS Genet. 13, 1–31 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Feldman M. W., Lewontin R. C., The heritability hang-up. Science 190, 1163–1168 (1975). [DOI] [PubMed] [Google Scholar]
- 18.Block N., How heritability misleads about race. Cognition 56, 99–128 (1995). [DOI] [PubMed] [Google Scholar]
- 19.D. Falconer, T. Mackay, Quantitative Genetics (Pearson Prentice-Hall, 4th edition, 1996).
- 20.Plomin R., DeFries J. C., Heritability of IQ. Science 194, 10–12 (1976). [DOI] [PubMed] [Google Scholar]
- 21.Warne R. T., In the Know: Debunking 35 myths about Human Intelligence (Cambridge University Press, 2020). [Google Scholar]
- 22.Warne R. T., Between-group mean differences in intelligence in the United States are 0% genetically caused: Five converging lines of evidence. Am. J. Psychol. 134, 480–501 (2021). [Google Scholar]
- 23.Jensen A. R., Interpretation of heritability. Am. Psychol. 27, 973–975 (1972). [Google Scholar]
- 24.Jensen A. R., The IQ controversy: A reply to Layzer. Cognition 1, 427–452 (1972). [Google Scholar]
- 25.Jensen A. R., The meaning of heritability in the behavioral sciences. Ed. Psychol. 11, 171–183 (1975). [Google Scholar]
- 26.Edge M. D., Rosenberg N. A., A general model of the relationship between the apportionment of human genetic diversity and the apportionment of human phenotypic diversity. Hum. Biol. 87, 313 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Whitlock M. C., Neutral additive genetic variance in a metapopulation. Genet. Res. 74, 215–221 (1999). [DOI] [PubMed] [Google Scholar]
- 28.Prout T., Barker J. S., F statistics in Drosophila buzzatii: Selection, population size and inbreeding. Genetics 134, 369–375 (1993). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Spitze K., Population structure in Daphnia obtusa: Quantitative genetic and allozymic variation. Genetics 135, 367–374 (1993). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Whitlock M. C., Evolutionary inference from QST. Mol. Ecol. 17, 1885–1896 (2008). [DOI] [PubMed] [Google Scholar]
- 31.Leinonen T., McCairns R. J. S., O’Hara R. B., Merila J., FST comparisons: Evolutionary and ecological insights from genomic heterogeneity. Nat. Rev. Genet. 14, 179–190 (2013). [DOI] [PubMed] [Google Scholar]
- 32.Haller B. C., Messer P. W., SLiM 3: Forward genetic simulations beyond the Wright-Fisher model. Mol. Biol. Evol. 36, 632–637 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Rushton J. P., Jensen A. R., Thirty years of research on race differences in cognitive ability. Psychol. Public Policy Law 11, 235–294 (2005). [Google Scholar]
- 34.Dixon A. R., Telles E. E., Skin color and colorism: Global research, concepts, and measurement. Annu. Rev. Sociol. 43, 405–424 (2017). [Google Scholar]
- 35.Yang J., Zeng J., Goddard M. E., Wray N. R., Visscher P. M., Concepts, estimation and interpretation of SNP-based heritability. Nat. Genet. 49, 1304–1310 (2017). [DOI] [PubMed] [Google Scholar]
- 36.Yang J., Lee S. H., Goddard M. E., Visscher P. M., GCTA: A tool for genome-wide complex trait analysis. Am. J. Hum. Genet. 88, 76–82 (2011). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.J. Huang, S. Basu, M. D. Shriver, A. A. Zaidi, Interpreting SNP heritability in admixed populations. bioRxiv (2023). https://www.biorxiv.org/content/10.1101/2023.08.04.551959v1 (Accessed 20 December 2023). [DOI] [PMC free article] [PubMed]
- 38.Lewontin R. C., The analysis of variance and the analysis of causes. Am. J. Hum. Genet. 26 (1974). [PMC free article] [PubMed] [Google Scholar]
- 39.Kempthorne O., A biometrics invited paper: Logical, epistemological and statistical aspects of nature-nurture data interpretation. Biometrics 34, 1–23 (1978). [PubMed] [Google Scholar]
- 40.Mackenzie B., Hypothesized genetic racial differences in IQ: A criticism of three proposed lines of evidence. Behav. Genet. 10, 225–234 (1980). [Google Scholar]
- 41.Turkheimer E., Individual and group differences in adoption studies of IQ. Psychol. Bull. 110, 392–405 (1991). [Google Scholar]
- 42.C. C. Roseman, K. A. Bird, Between group heritability and the status of hereditarianism as an evolutionary science. bioRxiv (2023). https://www.biorxiv.org/content/10.1101/2023.12.18.572247v1 (Accessed 8 August 2023).
- 43.Pujol B., Wilson A. J., Ross R. I. C., Pannell J. R., Are QST–FST comparisons for natural populations meaningful? Mol. Ecol. 17, 4782–4785 (2008). [DOI] [PubMed] [Google Scholar]
- 44.Plassais J., et al. , Natural and human-driven selection of a single non-coding body size variant in ancient and modern canids. Curr. Biol. 32, 889–897.e9 (2022). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Evans G. W., Kantrowitz E., Socioeconomic status and health: The potential role of environmental risk exposure. Annu. Rev. Public Health 23, 303–331 (2002). [DOI] [PubMed] [Google Scholar]
- 46.Liu J., et al. , Disparities in air pollution exposure in the united states by race/ethnicity and income, 1990–2010. Environ. Health Persp. 129, 127005 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Scriver C. R., The PAH gene, phenylketonuria, and a paradigm shift. Hum. Mutation 28, 831–845 (2007). [DOI] [PubMed] [Google Scholar]
- 48.G. Coop, Reading tea leaves? Polygenic scores and differences in traits among groups (2019).
- 49.Rosenberg N. A., Edge M. D., Pritchard J. K., Feldman M. W., Interpreting polygenic scores, polygenic adaptation, and human phenotypic differences. Evol. Med. Public Health 2019, 26–34 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Lewontin R. C., Krakauer J., Distribution of gene frequency as a test of the theory of the selective neutrality of polymorphisms. Genetics 74, 175–195 (1973). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Weaver T. D., Estimators for QST and coalescence times. Ecol. Evol. 6, 7783–7786 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Nei M., Analysis of gene diversity in subdivided populations. Proc. Natl. Acad. Sci. U.S.A. 70, 3321–3323 (1973). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.J. G. Schraiber, M. D. Edge, Data from “Heritability within groups is uninformative about differences among groups: Cases from behavioral, evolutionary, and statistical genetics”. Github. https://github.com/Schraiber/heritability. Deposited 15 January 2024. [DOI] [PMC free article] [PubMed]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Appendix 01 (PDF)
Data Availability Statement
Code used in this paper can be found at https://github.com/Schraiber/heritability (53).

