Interpreting population and family-based genome-wide association studies in the presence of confounding

Abstract

A central aim of genome-wide association studies (GWASs) is to estimate direct genetic effects: the causal effects on an individual’s phenotype of the alleles that they carry. However, estimates of direct effects can be subject to genetic and environmental confounding, and can also absorb the ‘indirect’ genetic effects of relatives’ genotypes. Recently, an important development in controlling for these confounds has been the use of within-family GWASs, which, because of the randomness of Mendelian segregation within pedigrees, are often interpreted as producing unbiased estimates of direct effects. Here, we present a general theoretical analysis of the influence of confounding in standard population-based and within-family GWASs. We show that, contrary to common interpretation, family-based estimates of direct effects can be biased by genetic confounding. In humans, such biases will often be small per-locus, but can be compounded when effect size estimates are used in polygenic scores. We illustrate the influence of genetic confounding on population- and family-based estimates of direct effects using models of assortative mating, population stratification, and stabilizing selection on GWAS traits. We further show how family-based estimates of indirect genetic effects, based on comparisons of parentally transmitted and untransmitted alleles, can suffer substantial genetic confounding. In addition to known biases that can arise in family-based GWASs when interactions between family members are ignored, we show that biases can also arise from gene-by-environment (G×E) interactions when parental genotypes are not distributed identically across interacting environmental and genetic backgrounds. We conclude that, while family-based studies have placed GWAS estimation on a more rigorous footing, they carry subtle issues of interpretation that arise from confounding and interactions.

1. Introduction

Genome-wide association studies (GWASs) have identified thousands of genetic variants that are associated with a wide variety of traits in humans. In the standard ‘population-based’ approach, the GWAS is conducted on a set of ‘unrelated’ individuals. The associations that are detected can arise when a variant causally affects the trait or when it is in tight physical linkage with causal variants nearby.

Central to the aims of GWASs is the estimation of variants’ effect sizes on traits of interest. These effect size estimates are important for identifying and prioritizing variants and implicated genes for functional followup, and may be used to form statistical predictors of trait values or to understand the causal or mechanistic role of genetic variation in traits. Understanding sources of error and bias in GWAS effect size estimates is therefore crucial.

The interpretation of GWAS effect size estimates is complicated by four broad factors (Vilhjálmsson and Nordborg 2013; Young et al. 2019). First, the causal pathways from an allele to phenotypic variation need not reside in the individuals who enrolled in the GWAS, but can also reflect causal effects on the individual’s environment of the genotypes of their siblings, parents, other ancestors, and neighbors (indirect genetic effects, or dynastic effects; Wolf et al. 1998). Second, a phenotypic association can result from correlations between the allele and environmental causes of trait variation (environmental confounding; Lander and Schork 1994). Third, a phenotypic association can be generated at a locus if it is genetically correlated with causal loci outside of its immediate genomic region (genetic confounding; Vilhjálmsson and Nordborg 2013). Fourth, an allele’s effect on a trait might depend on the environment and the allele’s genetic background (gene-by-environment and gene-by-gene interactions, or G×E and G×G; Freeman 1973^; Marchini et al. 2005^; Gauderman et al. 2017).

Since our primary interest here will be genetic confounding, we briefly describe some potential sources of the long-range allelic associations that drive it: population structure, assortative mating, and selection on the GWAS trait.

Population structure leads to genetic correlations across the genome when allele frequencies differ across populations or geographic regions: sampled individuals from particular populations are likely to carry, across their genomes, alleles that are common in those populations, which induces correlations among these alleles, potentially across large genomic distances. Such genetic correlations persist even after the populations mix, as alleles that were more common in a particular source population retain their association until uncoupled by recombination.

Assortative mating brings alleles with the same directional effect on a trait (or on multiple traits, in the case of cross-trait assortative mating) together in mates, and therefore bundles these alleles in offspring and subsequent generations. This bundling manifests as positive genetic correlations among alleles with the same directional effect (Wright 1921; Crow and Felsenstein 1968), which can confound effect size estimates in a GWAS on the trait.

Finally, natural selection on a GWAS trait can result in genetic correlations by favoring certain combinations of trait-increasing and trait-decreasing alleles. A form of selection that is expected to be common for many traits of interest is stabilizing selection, which penalizes deviations from an optimal trait value. By favoring compensating combinations of trait-increasing and trait-decreasing alleles, stabilizing selection generates negative correlations among alleles with the same directional effect (Bulmer 1971, 1974), and therefore can confound effect size estimates in a GWAS performed on the trait under selection or on a genetically correlated trait.

The potential for dynastic, environmental, and genetic confounds to bias GWAS effect size estimates has long been recognized (Lander and Schork 1994; Ewens and Spielman 1995), and so a major focus of the literature has been to develop methods to control for these confounds (Pritchard and Rosenberg 1999; Price et al. 2010). Standard approaches include using estimates of genetic relatedness as covariates in GWAS regressions (Price et al. 2006; Yang et al. 2014) or downstream analyses such as LD-Score regression (Bulik-Sullivan et al. 2015a,b; Bulik-Sullivan 2015). Such methods aim to control for both environmental and genetic confounding, but do so imperfectly (e.g., Berg et al. 2019^; Sohail et al. 2019). Further, it is often unclear what features of genetic stratification are being addressed (Vilhjálmsson and Nordborg 2013; Young et al. 2019): assortative mating in particular may not be well accounted for by these methods (Border et al. 2022b). Moreover, in reality, there is no bright line separating dynastic, environmental, and genetic confounding.

One promising way forward is to estimate allelic effects within families, either by comparing the separate associations of parentally transmitted and untransmitted alleles with trait values in the offspring (Spielman et al. 1993; Allison 1997; Eaves et al. 2014; Weiner et al. 2017; Kong et al. 2018), or by associating differences in siblings’ trait values with differences in the alleles they inherited from their parents (Abecasis et al. 2000; Visscher et al. 2006; Lee et al. 2018). The idea is that, by controlling for parental genotypes, within-family association studies control for both environmental stratification and indirect/dynastic effects, while Mendelian segregation randomizes alleles across genetic backgrounds. In principle, this allows the ‘direct genetic effect’ of an allele—the causal effect of an allele carried by an individual on their trait value—to be estimated. Recognizing that a variant detected in a GWAS will usually not itself be causal for the trait variation but instead will only be correlated with true causal variants, the direct effect of a genotyped variant is usually interpreted as reflecting the direct causal effects of nearby loci that are genetically correlated with the focal locus (Young et al. 2019)—but not the effects of more distant loci that might also be genetically correlated with the focal genotyped locus (e.g., because of population structure or assortative mating).

Consistent with both the presence of substantial confounds in some population-based GWASs and the mitigation of these confounds in within-family GWASs, family-based estimates of direct effect sizes and aggregate quantities based on these estimates (e.g., SNP-based heritabilities) are substantially smaller than population GWAS estimates for a number of traits, most notably social and behavioural traits (Lee et al. 2018; Selzam et al. 2019; Mostafavi et al. 2020; Howe et al. 2022; Young et al. 2022). Likewise, estimates of genetic correlations between traits are sometimes substantially reduced when calculated using direct effect estimates from within-family GWASs (e.g. Howe et al. 2022). While some of these findings could reflect the contribution of indirect genetic effects to population GWASs, it is also likely that, at least for some traits, standard controls for population stratification in population GWASs have been insufficient (Berg et al. 2019; Sohail et al. 2019; Young et al. 2022; Okbay et al. 2022; Nivard et al. 2022; Border et al. 2022a).

Our aim in this paper is to study a general model of confounding in GWASs, to generate clear intuition for its influence on estimates of effect sizes in both population- and family-based designs. A number of the issues that we analyze have previously been raised, particularly in the context of population-based GWASs (e.g., Rosenberg and Nordborg 2006^; Platt et al. 2010^; Vilhjálmsson and Nordborg 2013^; Young et al. 2019); here, we analyze them in a common framework that allows for comparison of multiple sources of confounding in both population and family-based GWASs. There is a large literature on GWASs in non-human organisms (e.g., Atwell et al. 2010^; Hayes and Goddard 2010^; Peiffer et al. 2014^; Josephs et al. 2017). However, although the results and intuition that we derive here apply equally well to human and non-human GWASs, we shall interpret them primarily from the perspective of human GWASs, in which the inability to experimentally randomize environments, together with the small effects that investigators hope to detect, makes confounding a particular concern.

Our first focus is on confounding—and genetic confounding in particular—in the absence of G×E and G×G interactions. To better understand the differences between population and within-family GWASs, we first study a general model of genetic confounding in the absence of G×E and G×G interactions. We derive expressions for estimators of direct effects in both population and within-family GWASs, as functions of the true direct and indirect effects at a locus and the genetic confounds induced by other loci. In doing so, we find that family-based estimates of direct effects are in fact susceptible to genetic confounding, contrary to standard interpretation. Reassuringly, in many of the models we consider, the resulting biases are likely to be small in humans. We also address a related case: family-based GWAS designs that consider transmitted and untransmitted parental alleles and in which the indirect (or ‘dynastic’) effect of an allele is estimated from its association with the offspring’s phenotype when carried by the parent but not transmitted to the offspring. We show that this estimator of indirect effects can be substantially biased by genetic and environmental confounds, in a similar way to population estimates of direct effects. Next, we consider various sources of genetic confounding—assortative mating, population structure, and stabilizing selection on GWAS traits— and how they influence estimates of direct effects in both population and within-family GWASs.

We then turn to sibling indirect effects, which are known to bias estimates of direct effects in sibling-based GWASs (Young et al. 2019, 2022). We characterize this bias in a simple model, and contrast it to the bias caused by sibling indirect effects in a population GWAS.

Finally, we consider G×E and G×G interactions, showing how their presence can bias population and family-based estimates of direct genetic effects in contrasting ways, complicating the interpretation of family-based estimates.

2. Effect size estimates in association study designs

Our primary focus will be on how genetic confounding can bias the estimation of direct genetic effects. These genetic confounds are due to associations between a genotyped variant at a GWAS locus and causal variants at other loci. As we will see, two kinds of association must be distinguished: cis-linkage disequilibrium (cis-LD) and trans-linkage disequilibrium (trans-LD). Genetic variants A and B are in positive cis-LD if, when an individual inherits A from a given parent, the individual is disproportionately likely to inherit B from that parent (Fig. 1A). A and B are in positive trans-LD if, when an individual inherits A from one parent, the individual is disproportionately likely to inherit B from the other parent (Fig. 1B). These covariances have also been called gametic and non-gametic LD, respectively (e.g. Weir 2008). To quantify the degrees of cis-LD and trans-LD, we denote by $D_{ij}$ and ${\tilde{D}}_{ij}$ the allelic covariances between focal variants at loci $i$ and $j$ in cis and in trans, and we denote by $r_{ij}$ and ${\tilde{r}}_{ij}$ the analogous allelic correlation coefficients. For some of our results, it will be important to distinguish the LD present in the sample on which the association study is performed and the LD present among the parents of the sample.

Figure 1:

The influence of cis- and trans-LD on effect size estimates in population-based and within-family association studies. (A) The focal allele at an association study locus (solid red square) is in positive cis-LD with trait-increasing alleles at other loci (solid black squares) if it is disproportionately likely to be found alongside them on an individual’s maternally or paternally inherited genome. (B) The focal allele at the study locus is in positive trans-LD with trait-increasing alleles at other loci if it is disproportionately likely to be found across from them on the maternally and paternally inherited genomes. (C,D) In a population association study, both positive cis- and trans-LD between the focal allele at the study locus and trait-increasing alleles anywhere else in the genome—either on the same chromosome as the study locus or on different chromosomes—generate a spuriously high effect size estimate at the study locus. (E,F) In a sibling association study, a trait-increasing allele causes a spuriously increased effect size estimate at the study locus if the parent is a coupling double heterozygote for the focal and trait-increasing alleles, having inherited them from the same parent (E), but a spuriously decreased estimate if the parent is a repulsion double heterozygote, having inherited them from different parents (F). These biases arise only if the trait-affecting locus is on the same chromosome as the focal study locus. The net bias depends on the relative frequencies of coupling and repulsion double heterozygotes in the parents, which depends on the difference in the degrees of cis- and trans-LD.

Consider a trait Y influenced by genetic variants at a set of polymorphic loci L, each of which segregates for two alleles. For ease of interpretation, and without loss of generality, we designate the ‘focal’ allele at locus l ∈ L to be the allele that directly increases the trait value, and we denote by p_l the frequency of this allele. Allelic effects are assumed to be additive within and across loci, such that the trait value of an individual can be written

$$Y=Y^{*}+\underset{\text{direct}\text{effects}}{\underbrace{\sum _{l\in L}{g}_l{\alpha }_l^{\text{d}}}}+\underset{\text{indirect}\text{effects}}{\underbrace{\sum _{l\in L}\left(g_l^{\text{m}}+g_l^{\text{f}}\right){\alpha }_l^{\text{i}}}}+ϵ.$$ (1)

Here, $g_l$, $g_l^{\text{m}}$, and $g_l^{\text{f}}$ are the numbers (0, 1, or 2) of focal alleles carried at locus $l$ by the individual, their mother, and their father respectively, ${\alpha }_l^{\text{d}}>0$ is the direct effect of the focal allele at $l$, and ${\alpha }_l^{\text{i}}$ is its indirect effect via the maternal and paternal genotypes. (For simplicity, we assume that indirect effects via the maternal and paternal genotypes are equal; this assumption is relaxed in Appendix A1.) $ϵ$ is the environmental noise, with $\mathbb{E}[ϵ]=0$, and $Y*$ is the expected trait value of the offspring of parents who carry only trait-decreasing alleles.

2.1. Population-based association studies

The variants at a genotyped locus will usually not themselves have causal effects on the trait, but will instead be in cis-LD with—and thus ‘tag’—causal variants at nearby loci. Thus, we typically think of the association at a focal genotyped locus as reflecting the direct contributions of a relatively small number of tightly linked loci, $L_{\text{local}}$, found within tens or perhaps hundreds of kb from the focal locus (Pritchard and Przeworski 2001). Under the additive model, therefore, the standard interpretation is that a population association study performed at a focal genotyped locus $\lambda$ provides an estimate of the quantity

$${\alpha }_{\lambda }=\frac{1}{p_{\lambda }\left(1-p_{\lambda }\right)}\sum _{l\in L_{\text{local}}}{D}_{\lambda l}{\alpha }_l^d,$$ (2)

where $p_{\lambda }$ is the frequency of the focal allele at $\lambda$, and $D_{\lambda l}$ is the degree of cis-LD between the focal allele at $\lambda$ and a causal allele at a nearby locus $l\in L_{\mathrm{l}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{l}}$. It is reasonable to think of this quantity as the ‘direct effect’ tagged by the focal variant at the genotyped locus $\lambda$: in the absence of confounding, it can be interpreted as the average phenotypic effect of randomly choosing a non-focal allele in the population and swapping it for a focal allele, where in this hypothetical swap, the causal alleles near the locus are included. For concreteness, we assume some fixed $L_{\text{local}}$ in our analyses, but in practice researchers seldom have a pre-defined number of ‘local’ SNPs in mind.

Effect size estimation in a population GWAS is complicated by the presence of environmental and genetic stratification. Under the model in Eq. (1), if we perform a standard population association study at locus $\lambda$, the estimated effect of the focal allele on the trait Y is

$${\widehat{\alpha }}_{\lambda }^{\text{pop}}=\frac{2}{V_{\lambda }}(\sum _{l\in L_{\text{local}}}{D}_{\lambda l}{\alpha }_l^{\text{d}}+\underset{\text{genetic}\text{confounds},\text{direct}}{\underbrace{\sum _{l\in L\backslash L_{\text{local}}}{D}_{\lambda l}{D}_{\lambda l}{\alpha }_l^{\text{d}}+\sum _{l\in L}{\tilde{D}}_{\lambda l}{\alpha }_l^{\text{d}}}}+\underset{\text{genetic}\text{confounds},\text{indirect}}{\underbrace{\sum _{l\in L}\left[D_{\lambda l}^{\prime }+{\tilde{D}}_{\lambda l}^{\prime }+2{\tilde{D}}_{\lambda l}\right]{\alpha }_l^{\text{i}}}}+\underset{\begin{array}{c}\text{environmental}\\ \text{confound}\end{array}}{\underbrace{\frac{1}{2}\mathrm{Cov}\left(g_{\lambda },ϵ\right)}}),$$ (3)

where, of the cis- and trans-LD terms, $D_{\lambda l}$ and ${\tilde{D}}_{\lambda l}$ are defined in the GWAS sample while $D_{\lambda l}^{\mathrm{\text{'}}}$ and ${\tilde{D}}_{\lambda l}$ are defined in their parents (Appendix A1.2). $V_{\lambda }$ is the genotypic variance at $\lambda$, equal to $2p_{\lambda }\left(1-p_{\lambda }\right)\left(1+F_{\lambda }\right)$ where $F_{\lambda }$ is Wright’s coefficient of inbreeding at $\lambda$.

The environmental confound is $\mathrm{C}\mathrm{o}\mathrm{v}\left(g_{\lambda },ϵ\right)/V_{\lambda }$; all non-local cis- and trans-LD terms in the study sample $\left(D_{\lambda l}\right.$ and ${\tilde{D}}_{\lambda l},l\notin L_{\text{local}}$) are direct genetic confounds (Fig. 1C,D); and all cis- and trans-LD terms among parents of sampled individuals $\left(D_{\lambda l}^{\mathrm{\text{'}}}\right.$ and ${\tilde{D}}_{\lambda l}$), together with all trans-LD terms in the study sample ${\tilde{D}}_{\lambda l}$, are indirect genetic confounds.

The direct genetic confounds arise because an allele carried by an offspring at $\lambda$ is correlated with the alleles that they carry at other loci $l\in L$ (via $D_{\lambda l}$ and ${\tilde{D}}_{\lambda l}$) that directly affect the trait value. The indirect genetic confounds arise because an allele carried by the offspring at $\lambda$—say, the maternal allele—is correlated with alleles carried by the offspring’s mother at other loci $\left(D_{\lambda l}^{\mathrm{\text{'}}}\right.$ and $({\tilde{D}}_{\lambda l}^{\mathrm{\text{'}}})$ and alleles carried by their father (as reflected by the trans-LD in the offspring, ${\tilde{D}}_{\lambda l}$). These alleles in the parents can indirectly affect the offspring’s trait value.

Thus, as is now well appreciated, population-based GWASs potentially suffer from many types of confounds (Vilhjálmsson and Nordborg 2013; Young et al. 2019). In practice, they can be reduced by including principal components—which capture genome-wide relatedness among GWAS participants—as regressors in a GWAS, or by using relatedness matrices in mixed models (Price et al. 2006; Yang et al. 2014). However, it is often unclear exactly what these methods control for in a given application (Vilhjálmsson and Nordborg 2013; Young et al. 2019), and they have been shown to be inadequate in important cases (e.g., Berg et al. 2019^; Sohail et al. 2019). When principal components (or other controls) fail to account fully for stratification, then Eq. (3) can be interpreted as a decomposition of the remaining, uncontrolled-for confounding in the GWAS.¹

2.2. Within-family association studies

The two within-family association study designs that we consider are parent-offspring GWASs and sibling GWASs. Other designs have been proposed to control for genetic and environmental confounding in the estimation of aggregate quantities such as heritability (e.g., Young et al. 2018a), but our primary focus is on the estimation of single-marker effect sizes. We do later turn to the interpretation of polygenic score regressions within families.

Estimates of direct genetic effects.

Parent-offspring studies can be used to estimate trait associations separately for parentally transmitted and untransmitted variants at a locus $\lambda ,$ ${\hat{\alpha }}_{\lambda }^{\left(\mathrm{T}\right)}$ and ${\hat{\alpha }}_{\lambda }^{\left(\mathrm{U}\right)}$, by regressing the trait value $Y$ on the transmitted and untransmitted genotypes, $g_{\lambda }^T$ and $g_{\lambda }^{\mathrm{U}}$ (Kong et al. 2018). The aim is often to estimate the direct effect of a variant, ${\hat{\alpha }}_{\lambda }^{\mathrm{d}}$, as the difference between these two estimates:

$${\hat{\alpha }}_{\lambda }^{\mathrm{d},\mathrm{T}-\mathrm{U}}={\hat{\alpha }}_{\lambda }^{\left(\mathrm{T}\right)}-{\hat{\alpha }}_{\lambda }^{\left(\mathrm{U}\right)}.$$ (4)

A second aim is to treat ${\hat{\alpha }}_{\lambda }^{\left(\mathrm{U}\right)}$ as an estimate of the indirect, or family, effect of the variant. We return to this second aim later.

In Appendix A1.4, we show that, in the absence of interactions between parental and offspring genotypes, the estimate of the direct effect of a variant at locus $\lambda$ in a parent-offspring study is

$${\widehat{\alpha }}_{\lambda }^{\text{d},\text{T}-\text{U}}={\widehat{\alpha }}_{\lambda }^{(\text{T})}-{\widehat{\alpha }}_{\lambda }^{(\text{U})}=\frac{2}{V_{\lambda }}\sum _{l\in L}\left(1-2c_{\lambda l}\right)\left(D_{\lambda l}^{\prime }-{\tilde{D}}_{\lambda l}^{\prime }\right){\alpha }_l^{\text{d}}$$ (5)

$$\approx \frac{2}{V_{\lambda }}(\sum _{l\in L_{\text{local}}}{D}_{\lambda l}^{\prime }{\alpha }_l^{\text{d}}+\underset{\text{genetic}\text{confounds},\text{direct}}{\underbrace{{\sum }_{l\in L\backslash L_{\text{local}}}\left(1-2c_{\lambda l}\right)\left(D_{\lambda l}^{\prime }-{\tilde{D}}_{\lambda l}^{\prime }\right){\alpha }_l^{\text{d}})}},$$ (6)

where $c_{\lambda l}$ is the sex-averaged recombination rate between $\lambda$ and $l$. The cis- and trans-LD terms $D_{\lambda l}^{\mathrm{\text{'}}}$ and ${\tilde{D}}_{\lambda l}^{\prime }$ are measured in the parents.

Similarly, an estimate of the direct effect can be obtained from pairs of siblings by regressing the differences in their phenotypes on the differences in their genotypes at the focal locus $\lambda$. In the presence of genetic confounds, this procedure yields a similar estimate to Eq. (6):

$${\widehat{\alpha }}_{\lambda }^{\text{d},\text{sib}}\approx \frac{2}{H_{\lambda }}\left(\sum _{l\in L_{\text{local}}}{D}_{\lambda l}^{\prime }{\alpha }_l^{\text{d}}+\underset{\text{genetic}\text{confounds,}\text{direct}}{\underbrace{\sum _{l\in L\backslash L_{\text{local}}}\left(1-2c_{\lambda l}\right)\left(D_{\lambda l}^{\prime }-{\tilde{D}}_{\lambda l}^{\prime }\right){\alpha }_l^{\text{d}}}}\right),$$ (7)

where $H_{\lambda }$ is the fraction of parents who are heterozygous at locus $\lambda$ (Appendix A1.3). An assumption in sibling GWASs is that an offspring’s phenotype is not influenced by the genotypes of their siblings — i.e., that there are no sibling indirect genetic effects. We consider violations of this assumption later.

In Eqs. (6) and (7), there is no environmental confound, because family-based GWASs successfully randomize the environments of family members with respect to within-family genetic transmission.

The derivations above further show that, while population association studies are biased by sums of trans- and cis-LD between the focal locus and all causal loci (Eq. 3), within-family association studies are instead biased by differences between trans- and cis-LD, and moreover, that the biases in within-family studies are driven only by LD between the focal locus and causal loci on the same chromosome $\left(c_{\lambda l}<1/2\right)$. To provide an intuition for this result, we focus our discussion on a sibling association study performed at $\lambda$; the intuition is identical for the analogous parent-offspring study.

Because the difference between two siblings in their maternally inherited genotypes is independent of the difference in their paternally inherited genotypes, we may consider maternal and paternal transmissions separately in studying how a locus $l\in L$ can confound effect size estimation at $\lambda$ in a sibling association study. We will phrase our discussion in terms of maternal transmission.

For effect size estimation at $\lambda$ to be genetically confounded by maternal transmission at a distant locus $l$, the mother must be heterozygous at both loci. For if she were homozygous at $l$, then maternal transmission at $l$ could not contribute to any trait differences between her offspring, while if she were homozygous at $\lambda$, maternal transmission would not result in genetic variation among her offspring at $\lambda$ with which trait variation could be associated. Therefore, we restrict our focus to mothers who are heterozygous at both $\lambda$ and $l$, or ‘double heterozygotes’. Two kinds exist (Fig. 1E,F): coupling double heterozygotes who carry the focal alleles at $\lambda$ and $l$ on the same haploid genome (‘in cis’), and repulsion double heterozygotes who carry them on separate haploid genomes (‘in trans’).

We first consider the case where the recombination rate between $\lambda$ and $l$ is small $\left(c_{\lambda l}\ll 1/2\right)$. In this case, if the mother is a coupling double heterozygote, then her offspring will tend to inherit either both or neither of the focal alleles at $\lambda$ and $l$ (Fig. 1E). Therefore, if one sibling inherits the focal allele at $\lambda$ and another does not, the first sibling will tend to inherit the focal (trait-increasing, as we have defined it) allele at $l$ and the second sibling will not, so that the effect of locus $l$ positively confounds the association between $\lambda$ and the trait (Fig. 1E). If the mother is instead a repulsion double heterozygote, then her offspring will tend to inherit either the focal allele at $\lambda$ or the focal allele at $l$, but not both (Fig. 1F). In this case, if one sibling inherits the focal allele at $\lambda$ and another does not, the second sibling will tend to inherit the focal (trait-increasing) allele at $l$ and the first sibling will not, so that the effect of locus $l$ negatively confounds the association between $\lambda$ and the trait (Fig. 1F). When $\lambda$ and $l$ are linked, therefore, the way in which $l$ genetically confounds the effect size estimate at $\lambda$ depends, positively or negatively, on whether the fraction of coupling double heterozygotes among parents is greater or smaller, respectively, than the fraction of repulsion double heterozygotes.

In contrast, if $\lambda$ and $l$ are unlinked $\left(c_{\lambda l}=1/2\right)$, then transmissions from coupling and repulsion double heterozygote parents are equal, and so $l$ cannot confound estimates at $\lambda$ (Fig. 1E,F). Put differently, meiosis in double heterozygotes fully randomizes joint allelic transmissions at $\lambda$ and $l$, with offspring equally likely to inherit any possible combination of alleles at the two loci.

Therefore, only linked loci $l$ can confound a family-based association study at $\lambda$, and they do so in proportion to (i) how small the recombination rate between $\lambda$ and $l$ is, and (ii) the difference between the fractions of parents who are coupling and repulsion double heterozygotes at $\lambda$ and $l$. Accordingly, if we write these fractions of parents as $H_{\lambda l}^{\text{coup}}$ and $H_{\lambda l}^{\text{rep}}$, then $D_{\lambda l}^{\prime }-{\tilde{D}}_{\lambda l}^{\prime }=\left(H_{\lambda l}^{\text{coup}}-H_{\lambda l}^{\text{rep}}\right)/2$, and so Eq. (7) (and Eq. 6) can be rewritten in terms of the relative frequencies of the two kinds of double-heterozygotes:

$${\hat{\alpha }}_{\lambda }^{\mathrm{d},\text{sib}}\approx \frac{2}{H_{\lambda }}\left(\sum _{l\in L_{\text{local}}} D_{\lambda l}^{\prime }{\alpha }_l^{\mathrm{d}}+\sum _{l\in L\backslash L_{\text{lacal}}} \left(\frac{1}{2}-c_{\lambda l}\right)\left(H_{\lambda l}^{\text{coup}}-H_{\lambda l}^{\text{rep}}\right){\alpha }_l^d\right).$$

In a species with many chromosomes, such as humans, for a given locus, there will be many more unlinked loci than linked loci. Therefore, the set of loci that can confound a family-based association study at a given locus will be much smaller than the set of loci that can confound a population association study at the locus. It will often be the case, therefore, that biases in the estimation of direct genetic effects will be smaller in family-based studies than in population studies, a point that we explore below when we consider sources of genetic confounding.

Estimates of indirect genetic effects.

We now return to the regression of the trait on the untransmitted genotype in parent-offspring GWASs, ${\hat{\alpha }}_{\lambda }^{\left(\mathrm{U}\right)}$, which has sometimes been treated as an estimate of the indirect effect ${\hat{\alpha }}_{\lambda }^{\mathrm{i}}$. Assuming equal indirect effects via maternal and paternal genotypes (an assumption that we relax in Appendix A1.4),

$$\begin{array}{l}{\widehat{\alpha }}_{\lambda }^{\text{i}}={\widehat{\alpha }}_{\lambda }^{(\text{U})}=\frac{2}{V_{\lambda }}(\underset{\text{local}\text{indirect}\text{effect}}{\underbrace{\sum _{l\in L_{\text{local}}}{D}_{\lambda l}^{\prime }{\alpha }_l^{\text{i}}}}+\underset{\text{genetic}\text{confounds,}\text{direct}}{\underbrace{\sum _{l\in L}\left(D_{\lambda l}^{\prime }{c}_{\lambda l}+{\tilde{D}}_{\lambda l}^{\prime }\left(1-c_{\lambda l}\right)+{\tilde{D}}_{\lambda l}\right){\alpha }_l^{\text{d}}}}\\ +\underset{\text{genetic}\text{confounds,}\text{indirect}}{\underbrace{\sum _{l\in L\backslash L_{\text{local}}}{D}_{\lambda l}^{\prime }{\alpha }_l^{\text{i}}+\sum _{l\in L}\left({\tilde{D}}_{\lambda l}^{\prime }+2{\tilde{D}}_{\lambda l}\right){\alpha }_l^{\text{i}}}}+\underset{\begin{array}{c}\text{environmental}\\ \text{confound}\end{array}}{\underbrace{\frac{1}{2}\mathrm{Cov}\left(g_{\lambda }^{\text{U}},ϵ\right)}}).\end{array}$$ (8)

The direct genetic confound reflects associations of the untransmitted alleles at the focal locus with alleles that are transmitted to the offspring at causal loci $l\in L$ and which directly affect the offspring’s trait value (via $\left.{\alpha }_l^{\mathrm{d}}\right)$. These associations are due to covariances among alleles in each parental genome $\left(D_{\lambda l}^{\mathrm{\prime }}\right.$ and ${\tilde{D}}_{\lambda l}^{\prime }$) and across the parental genomes (reflected as trans-LD in the offspring, ${\tilde{D}}_{\lambda l}$). The indirect genetic confound reflects associations of the untransmitted alleles to alleles at other loci in the parents, which can indirectly affect the offspring trait value (via ${\alpha }_l^{\mathrm{i}}$). Finally, unlike in family-based estimates of direct genetic effects (Eqs. 6 and 7), family-based estimates of indirect effects suffer from environmental confounding, in the same way that population GWASs do (Eq. 3).

Therefore, estimating the indirect effect by regressing the trait value on the untransmitted genotype is highly susceptible to environmental confounding as well as both direct and indirect genetic confounding, in a similar way to estimating the direct effect via a population-based association study (Shen and Feldman 2020). Adjustments for assortative mating in particular have been included in some PGS-based analyses of indirect effects (e.g., Kong et al. 2018^; Young et al. 2022). However, it is not clear how robust these adjustments are in the presence of multiple forms of confounding.

2.3. Polygenic scores and their phenotypic associations

A current drawback to family-based GWASs is that sample sizes are often small, limiting power to estimate direct genetic effects. Because of this limitation, instead of estimating per-locus effect sizes in family designs, investigators often measure the within-family phenotypic association of a combined linear predictor, a polygenic score (PGS), constructed using effect size estimates across many loci from a population GWAS. In the sibling-based version of this study design, the difference in siblings’ population-based PGSs is regressed on their difference in phenotypes (e.g., Lee et al. 2018^; Selzam et al. 2019). In parent-offspring designs, the population-based PGSs constructed separately for transmitted and untransmitted alleles are used as linear predictors of the offspring’s phenotype, and the difference in their slopes in this regression is estimated (e.g., Kong et al. 2018^; Okbay et al. 2022).

When such PGS regressions are used within families for the same phenotype as the population GWAS, a non-zero slope of the PGS is usually interpreted as reflecting the fact that the PGS—despite having been calculated from a population GWAS and therefore subject to many potential confounds—nevertheless does capture the direct genetic effects of alleles. When the PGS for one phenotype is regressed within families on the value of another phenotype, non-zero slopes are often interpreted as evidence that direct genetic effects on the two phenotypes are causally related, for example through pleiotropic effects of the alleles involved.

Suppose that we have performed a population GWAS for trait 1, generating effect size estimates ${\hat{\alpha }}_{\lambda }$ at a set of genotyped loci $\lambda \in \mathrm{\Lambda }$. To construct a PGS for trait 1, these effect size estimates are used as weights in a linear sum across an individual’s genotype:

$$PG{S}_1=\sum _{\lambda \in \text{Λ}}{g}_{\lambda }{\hat{\alpha }}_{\lambda }^{\mathrm{p}\mathrm{o}\mathrm{p}}.$$ (9)

In a sibling-based study (the results and intuition below will be the same for a parent-offspring study), the difference between siblings’ trait-1 PGSs, $△PG{S}_1$, is regressed on the difference in their values for trait 2, $\mathrm{\Delta }{Y}_2$ (note that trait 2 could be the same as trait 1). If $L$ is the set of loci that causally underlie variation in trait 2, and ${\beta }_l$ are the true effects of variants at these loci on trait 2, then the numerator of the slope in this regression can be written as

$$\text{Cov}\left(\text{Δ}PG{S}_1,\text{Δ}{Y}_2\right)=2\sum _{\lambda \in \text{Λ}}\sum _{l\in L}\left(1-2c_{\lambda l}\right)\left(D_{\lambda l}^{\prime }-{\tilde{D}}_{\lambda l}^{\prime }\right){\widehat{\alpha }}_{\lambda }^{\text{pop}}{\beta }_l$$ (10)

(see Appendix A2). Note that, while the population-based effect size estimates ${\hat{\alpha }}_{\lambda }$ depend on cis- and trans-LD, as detailed by Eq. (3), the patterns of LD may differ from those in the family study (the $D_{\lambda l}^{\mathrm{\prime }}-{\tilde{D}}_{\lambda l}^{\mathrm{\prime }}$ term in Eq. 10) if the population- and family-based studies differ in relevant aspects of sample composition.

The intuition for Eq. (10) is similar to that for the single-locus effect size estimate in a sibling GWAS (Eq. 7). The numerator of the difference in slopes of transmitted and untransmitted PGSs in a parent-offspring design takes a similar form to Eq. (10).

In the absence of confounding and under some simplifying assumptions, the sibling PGS covariance measures the contribution of each locus included in the PGS to the additive genic covariance between traits 1 and 2 that is tagged by the genotyped variants included in the PGS (see Eq. A.23 in Appendix A2). Under these assumptions, the sibling PGS slope therefore does provide a measure of the underlying pleiotropy between the traits.

Interpretation of the sibling PGS slopes is more complicated in the presence of genetic confounding (see Eq. A.22 in Appendix A2), which is absorbed into the effect size estimates ${\hat{\alpha }}_{\lambda }^{\text{pop}}$ (Eq. 3) so that the PGS applies a potentially strange set of weights to the genotyped loci it includes. (A related problem occurs when indirect genetic effects absorbed by the population-based PGS change the interpretation of within-family PGS slopes—see Trejo and Domingue (2018)^; Fletcher et al. (2021).) A non-zero sibling PGS slope still establishes that the trait-1 PGS loci are in systematic signed intra-chromosomal LD with loci that causally affect trait 2. However, it no longer necessarily implies that traits 1 and 2 are causally related via pleiotropy, for two reasons. To understand these reasons, suppose that the causal loci for traits 1 and 2 are distinct, i.e., that there is in fact no pleiotropy. First, a SNP included in the trait-1 PGS could tag local variants that causally affect trait 1 but which are also, via sources of confounding such as cross-trait assortative mating, in systematic long-range LD with variants on the same chromosome that causally affect trait 2. Such SNPs will be predictive of sibling differences in trait 2, even though they locally tag only trait-1 causal variants. Second, LD between variants on the same or distinct chromosomes that are causal for trait 1 and trait 2 will cause some SNPs that locally tag trait-2 causal variants to be significantly associated with trait 1 in a population GWAS, and therefore to be included in the trait-1 PGS. These SNPs, since they tag trait-2 causal variants, will be predictive of sibling differences in trait 2.

In summary, in the presence of confounding, non-zero sibling PGS slopes cannot be viewed as de facto evidence for causal relationships between traits.

3. Sources of genetic confounding in association studies

As we have seen, genetic confounding of association studies depends, in ways that vary across study designs, on levels of non-local cis- and trans-LD between the study locus and loci that influence the study trait. Below, we consider various processes that give rise to non-local cis- and trans-LD, and their likely impact on the different association study designs. We focus our attention on the potential for these sources of LD to confound measurement of several key metrics. First, the average deviation of the estimated effect size from its true value, $\mathbb{E}\left[{\hat{\alpha }}_{\lambda }-{\alpha }_{\lambda }\right]$. This measure indicates if effect sizes are systematically overestimated or underestimated because of genetic confounding. Second, the average squared effect size estimate, weighted by heterozygosity: $\mathbb{E}\left[2p_{\lambda }\left(1-p_{\lambda }\right){\hat{\alpha }}_{\lambda }^2\right]$. This quantity is related to important measures such as the genetic variance and SNP-based heritability (Bulik-Sullivan et al. 2015b b). It is also directly related to the variance of effect size estimates, and therefore captures the additional noise that genetic confounding creates in effect size estimation at a given locus. Finally, if GWASs have been performed on more than one trait, the covariance across loci of the effect size estimates for two traits may be of interest. This covariance is determined by the average heterozygosity-weighted product $\mathbb{E}\left[2p_{\lambda }\left(1-p_{\lambda }\right){\hat{\alpha }}_{\lambda }{\hat{\beta }}_{\lambda }\right]$, where ${\hat{\alpha }}_{\lambda }$ and ${\hat{\beta }}_{\lambda }$ are the effect size estimates at locus $\lambda$ for traits 1 and 2.

In what follows, for simplicity, we ignore indirect effects and assume that there is no environmental confounding (i.e., no correlation between genotypes and the environmental effects $ϵ$). For each of the sources of genetic confounding that we consider, we calculate the three measures listed above both analytically and in whole-genome simulations carried out in SLiM 4.0 (Haller and Messer 2019). In our simulations, we use two recombination maps: (i) for illustrative purposes, a simple hypothetical map where the genome lies along a single chromosome of length 1 Morgan, and (ii) the human linkage map generated by Kong et al. (2010). A more detailed description of the simulations can be found in the Methods, and code is available at github.com/cveller/confoundedGWAS.

3.1. Assortative mating

Assortative mating is the tendency for mating pairs to be correlated for particular traits—either the same trait (same-trait assortative mating) or distinct traits (cross-trait assortative mating). For example, humans are known to exhibit same-trait assortative mating for height and cross-trait assortative mating for educational attainment and height (amongst many other examples, reviewed in Horwitz and Keller 2022^; Border et al. 2022a). Assortative mating generates both cis- and trans-LD: It generates positive trans-LD among trait-increasing alleles because genetic correlations between mates translate to genetic correlations between maternally and paternally inherited genomes, and it generates positive cis-LD among trait-increasing alleles because, over generations, recombination converts trans-LD into cis-LD (Crow and Felsenstein 1968). (In some cases, assortative mating can generate cis-LD by mechanisms additional to recombination—see Veller et al. 2020.)

Constant-strength assortative mating.

If the strength of assortative mating (measured by the phenotypic correlation among mates $\rho$) is constant over time, and there are no other sources of genetic confounding such as population structure, then, for a given pair of loci $l,l^{\mathrm{\prime }}\in L$, the positive cis-LD $D_{l{l}^{\mathrm{\prime }}}$ will initially be smaller than the positive trans-LD ${\tilde{D}}_{l{l}^{\prime }}$, but will gradually grow towards an equilibrium value equal to the trans-LD $(D_{l{l}^{\prime }}^{*}={\tilde{D}}_{l{l}^{\prime }}^{*})$; in this equilibrium, assortative mating generates new cis-LD at the same rate as old cis-LD is destroyed by recombination (Crow and Felsenstein 1968, Appendix A3.1).

Therefore, in a population GWAS, effect size estimates will initially be biased upwards because of positive trans-LD, and the magnitude of the bias will grow over time as positive cis-LD too is generated from this trans-LD (Eq. 3; Fig. 2). In contrast, in a family-based GWAS, effect size estimates will initially be biased downwards because the positive trans-LD exceeds the positive cis-LD (Eqs. 6 and 7; Fig. 2). However, as the cis-LD grows over time towards the value of the trans-LD, the magnitude of the downward bias will shrink, and, in equilibrium, the family-based GWAS will not be confounded by assortative mating (Fig. 2).

Figure 2:

Assortative mating systematically biases effect size estimation in population and within-family GWASs, although the bias in within-family GWASs is expected usually to be small. Here, cross-trait assortative mating between traits 1 and 2 occurs for the first 19 generations, after which mating is random. Assortative mating is sex-asymmetric, with strength $\rho =0.2$. Distinct sets of loci underlie variation in trait 1 and 2, with effect sizes at causal loci normalized to 1. Plotted are average estimated effects of the focal alleles at loci causal for trait 1 in population and within-family GWASs on trait 2, for a hypothetical genome with one chromosome of length 1 Morgan (A) and for humans (B). Since the traits have distinct genetic bases, the true effects on trait 2 of the alleles at trait-1 loci are zero. The horizontal lines at 0.1 are a theoretical ‘first-order’ approximation of the asymptotic bias in a population GWAS (Appendix A3.1). Profiles are averages across 10,000 replicate simulation trials. Simulation details can be found in the Methods.

Under certain simplifying assumptions, we can calculate the average bias that assortative mating induces in a population GWAS in equilibrium, in the absence of other sources of genetic confounding such as population structure (Appendix A3.1). In the case of same-trait assortative mating, effect size estimates are inflated by an average factor of approximately $h^2\rho /\left(1-h^2\rho \right)$, where $\rho$ is the phenotypic correlation among mates and $h^2$ is the trait heritability (for similar calculations, see Yengo et al. 2018; Border et al. 2022b). In the case of cross-trait assortative mating, if assortative mating is directional/asymmetric with respect to sex—i.e., the correlation $\rho$ is between female trait 1 and male trait 2—then assortative mating generates spurious associations between trait 1 and alleles that affect trait 2 (and vice versa). If the loci underlying the two traits are distinct, then, in equilibrium, the spurious effect size estimate at non-causal loci is approximately $h^2\rho /2$ times the effect at causal loci, assuming the traits to have the same heritabilities and genetic architectures (horizontal dahsed line in Fig. 2). If cross-trait assortative mating is bi-directional/symmetric with respect to sex, then, in equilibrium, the average spurious effect size estimate at non-causal loci is approximately $h^2\rho$ times the effect at causal loci. Upward biases in effect size estimates at causal loci are also expected under cross-trait assortative mating, but these are second-order relative to the biases at non-causal loci (Fig. S1).

The systematic over- and under-estimation of effect sizes that assortative mating induces in population and family-based GWASs, respectively, will also affect our second measure of interest, the heterozygosity-weighted average squared effect size estimate $\mathbb{E}\left[2p_{\lambda }\left(1-p_{\lambda }\right){\hat{\alpha }}_{\lambda }^2\right]$ (and therefore also downstream quantities such as SNP heritabilities). In a population GWAS, the presence of trans-LD and the gradual creation of cis-LD under assortative mating will increase the biases in effect size estimates over time (Fig. 2), which will concomitantly increase the average value of ${\hat{\alpha }}_{\lambda }^2$ (Fig. 3; also see Border et al. 2022b). Moreover, cross-trait assortative mating will generate signals of genetic correlations among traits even in the absence of any pleiotropic effects of underlying variants (Border et al. 2022a). In a family-based GWAS, the temporary attenuation of effect size estimates owing to a transient excess of trans-LD over cis-LD under assortative mating will lead to a similar attenuation in the average squared effect size estimate (Fig. 3), although, like the bias in effect size estimates themselves, this attenuation is expected to be small in humans (Fig. 3B).

Figure 3:

The impact of assortative mating on the average squared effect size estimate in population and within-family GWASs. Same-trait assortative mating of strength ρ = 0.2 occurs in generations 0–19; mating is random before and after this period. Under random mating, the average squared effect size estimates exceed the true average squared effect size (yellow line) because random drift generates chance local LD with causal alleles that inflates the variance of effect size estimation (e.g. Bulik-Sullivan et al. 2015b). The magnitude of this variance inflation depends on the GWAS design and sample size, and the effect of assortative mating and its cessation should be judged in reference to it. To guide the eye in this judgment, the faint horizontal lines show the average squared effect size estimate in the last 20 generations of the initial burn-in period of random mating. Profiles are averages across 10,000 replicate simulation trials. Simulation details can be found in the Methods.

As shown by Border et al. (2022a^,b), the effects of assortative mating on estimates of heritability and genetic correlations described above are not well controlled for by LD Score regression (Bulik-Sullivan et al. 2015a,b). The LD score of a variant proxies the amount of local causal variation the SNP tags, but because assortative mating generates long-range signed LD among causal variants, it causes local causal variants to be in long-range signed LD with other causal variants throughout the genome. Therefore, the slope of the LD score regression absorbs the effects of assortative mating, causing its estimates of heritability and of the degree of pleiotropy to be inflated.

Historical assortative mating.

If, at some point in time, assortative mating for traits ceases and mating becomes random with respect to those traits, the positive trans-LD that was present under assortative mating will immediately disappear, leaving only the positive cis-LD that had built up; this cis-LD will then be gradually eroded by recombination. If equilibrium had been attained under assortative mating, the cis-LD would have grown to match the per-generation trans-LD. Therefore, in the first generation after assortative mating ceases, the upward bias in population GWAS effect size estimates would halve as the trans-LD disappears (Eq. 3); the bias would then shrink gradually to zero as the cis-LD erodes (Fig. 2). A similar pattern will be observed for the heterozygosity-weighted average value of ${\hat{\alpha }}_{\lambda }^2$ in the population GWAS, which eventually returns to its equilibrium level under random mating (Fig. 3).

In contrast, with the disappearance of the positive trans-LD but the persistence of positive cis-LD, the bias in family-based effect size estimates will suddenly become positive once assortative mating ceases (having temporarily been negative under assortative mating before equilibrium was attained); this bias too will then gradually shrink to zero as recombination erodes the remaining cis-LD (Fig. 2). Concomitantly, the average squared effect size estimate in the family GWAS will suddenly increase when assortative mating ceases, after which it too will gradually return to its equilibrium value under random mating (Fig. 3).

Assortative mating between traits with different genetic architectures.

An important practical question is how genetic confounding affects the GWAS loci we prioritize for functional follow-up and for use in the construction of polygenic scores. SNPs are usually prioritized on the basis of their GWAS p-value, which is proportional to the estimated variance explained by a SNP, $2p_{\lambda }\left(1-p_{\lambda }\right){\hat{\alpha }}_{\lambda }^2$ (where $p_{\lambda }$ is the minor allele frequency). The results above assume, in the case of cross-trait assortative mating, that the traits involved have similar genetic architectures (distribution of $p_l$ and ${\alpha }_l$ at causal loci, and the total number of causal loci). In that case, if there is no pleiotropy between the traits, then while SNPs that tag trait-1 causal loci are predictive of the value of trait 2 owing to LD between trait-1 and trait-2 causal loci, we nonetheless expect the SNPs that tag trait-2 causal loci to be better predictors of trait 2, such that GWAS investigators would primarily pick out SNPs tagging trait-2 causal loci for prioritization and use in polygenic scores.

However, analysis of human GWASs suggests that quantitative traits can have widely different genetic architectures, with, in particular, substantial differences in the effective numbers of causal loci involved and in the distribution of minor allele frequencies (Simons et al. 2022, and references therein). If two traits with distinct genetic bases show cross-trait assortative mating, but trait 1 has a denser genetic architecture (fewer causal loci) than trait 2, then the genetic signal of assortative mating—systematic LD between trait-1 and trait-2 causal loci—will be more heavily loaded per-locus onto trait-1 loci than onto trait-2 loci. In a GWAS on trait 2, this will inflate the magnitude of spurious effect size estimates at SNPs that tag trait-1 loci relative to effect size estimates at SNPs that tag causal trait-2 loci. In Appendix A3.1, we quantify this effect, showing that, in a population GWAS for trait 2, the average magnitude of spurious effect size estimates at trait-1 loci is proportional to $\sqrt{\left|L_2\right|/\left|L_1\right|}$, where $\left|L_1\right|$ and $\left|L_2\right|$ are the numbers of loci underlying variation in traits 1 and 2 respectively. Thus, when trait 1 has a denser genetic architecture than trait 2 (| $L_2\left|/|L_1\right|$ is large), the magnitudes of effect size estimates at non-causal trait-1 loci could substantially overlap with those at causal trait-2 loci (as illustrated in Fig. 4), potentially causing part of the apparent, mappable genetic architecture of the trait-2 GWAS to actually tag trait-1 loci.

Figure 4:

Cross trait assortative mating for traits with different genetic architectures can generate large spurious effect size estimates in population GWASs. Here, the traits have equal heritability, but the number of loci contributing variation to trait 1 is ten-fold smaller than that for trait 2. Shown, for a population GWAS on trait 2, are estimated distributions of the magnitude of effect size estimates at loci causal for trait 2 (grey) and at loci causal for trait 1 (greens), under random mating (light green) and after 20 generations of cross-trait assortative mating (sex-asymmetric, of strength ρ = 0.2) for traits 1 and 2 (dark green). Although the true effects of trait-1 loci on trait 2 are zero in these simulations (no pleiotropy), there is sampling noise in effect size estimation at trait-1 loci under random mating (light green line), so that the mean magnitude of effect size estimates is shifted away from zero (light green dot; dashed line displays 95th percentile under random mating). Under assortative mating, the magnitudes of the spurious effect size estimates at trait-1 loci shift significantly rightward (dark green line), coming to overlap substantially with the distribution of effect size estimate magnitudes at causal trait-2 loci (grey line; the distribution for trait-2 loci does not appreciably differ under random and assortative mating). Densities are estimated from pooled effect size estimates from 1,000 replicate simulations. Simulation details in Methods.

3.2. Population structure

When a population GWAS draws samples from individuals of dissimilar ancestries, differences in the distribution of causal genotypes, and potentially of environmental exposures, can confound the association study (Lander and Schork 1994; Vilhjálmsson and Nordborg 2013). Correcting for confounds due to population structure has therefore been an important pursuit in the GWAS literature (Spielman et al. 1993; Pritchard et al. 2000; Price et al. 2010).

For concreteness, consider a simple model where two populations diverged recently, with no subsequent gene flow between them. Genetic drift—and possibly selection—in the two populations will have led to allele frequency differences between them at individual loci. If allele frequencies have diverged at both a genotyped study locus and at loci that cansally influence the study trait, these frequency differences will manifest as linkage disequilibria between the study locus and the causal loci in a sample taken across both populations, even if the loci are not in LD within either population. Specifically, if the frequencies of the focal allele at a given locus $k$ are $p_k^{\left(1\right)}$ and $p_k^{\left(2\right)}$ in populations 1 and 2, then the cis-LD between the focal alleles at the association study locus $\lambda$ and a causal locus $l$ is

$$D_{\lambda l}^{\left(S\right)}=\frac{1}{4}\left(p_{\lambda }^{\left(1\right)}-p_{\lambda }^{\left(2\right)}\right)\left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right)$$ (11)

in a sample that weights the two populations equally, with the superscript $\left(S\right)$ denoting that this LD is due to stratification. The trans-LD takes exactly the same form: ${\tilde{D}}_{\lambda l}^{(S)}=D_{\lambda l}^{(S)}$. From Eq. (3), locus $l$ therefore confounds estimation of the direct effect at $\lambda$ in a population GWAS, by an amount proportional to

$$2\left(D_{\lambda l}^{\left(S\right)}+{\tilde{D}}_{\lambda l}^{\left(S\right)}\right){\alpha }_l^{\text{d}}=\left(p_{\lambda }^{\left(1\right)}-p_{\lambda }^{\left(2\right)}\right)\left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right){\alpha }_l^{\text{d}}.$$ (12)

These genetic confounds are in addition to environmental confounding that would arise if the environments of the two populations alter their average trait values by different amounts.

In contrast, estimates of direct effects obtained from within-family association studies are not genetically confounded, because cis- and trans-LD are equal (Eqs. 6 and 7). Another way of seeing this is to consider that, by controlling for family, within-family GWASs control for the population, and in the scenario considered, by construction, there are no within-population LDs to confound effect size estimation.

Allele frequency divergence due to drift.

How do the confounds introduced by population structure affect the first of our measures of interest, the average deviation of effect size estimates from their true values? The answer depends on the source of allele frequency differences between the two populations. If the differences are due to neutral genetic drift, they will be independent of each other (assuming causal loci are sufficiently widely spaced) and independent of the direction and size of effects at individual loci. Therefore, the LD induced by these allele frequency differences will, on average, not bias effect size estimates in a population GWAS:

$$\mathbb{E}\left[\left(p_{\lambda }^{\left(1\right)}-p_{\lambda }^{\left(2\right)}\right)\left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right){\alpha }_l^{\mathrm{d}}\right]=\mathbb{E}\left[p_{\lambda }^{\left(1\right)}-p_{\lambda }^{\left(2\right)}\right]\mathbb{E}\left[p_l^{\left(1\right)}-p_l^{\left(2\right)}\right]\mathbb{E}\left[{\alpha }_l^{\mathrm{d}}\right]=0\text{,}$$ (13)

since $\mathbb{E}\left[p_k^{\left(1\right)}-p_k^{\left(2\right)}\right]=0$ at any locus $k$.

However, the LD induced by population structure will inflate the average squared effect size estimate, and by extension the variance of effect size estimates (Fig. 5). In Appendix A3.2, we quantify this effect for the same simple case of two separate populations. We find that the average squared effect size estimate in a population GWAS is an increasing function of the divergence between the two populations (as measured by $\left.F_{ST}\right)$, the number of loci contributing variation to the study trait, and the true average squared effect size per locus (see also Rosenbery and Nordborg 2006^; Lee and Lee 2023a).

Figure 5:

The impact of population structure and admixture on the average squared effect size estimate in population and within-family GWASs. Here, two populations are isolated until generation 0, at which point they mix in equal proportions. Initial allele frequencies are chosen independently for the two populations, such that allele frequency differences between the populations resemble those that would accumulate over time via random drift. As in Fig. 3, the equilibrium value of the mean squared effect size estimate under random mating is greater than the true mean squared effect size, in both the population and within-family GWAS, owing to linkage disequilibria among causal alleles that arise due to drift. This explains why, in the insets, the blue (population) and red (within-family) profiles do not shrink all the way down to the yellow (true) line after admixture, when mating is random. Note too the difference in scale of the y-axes in the insets: the return to equilibrium is much more rapid under the human genetic map than for a hypothetical genome of one chromosome of length 1 Morgan, since, with more recombination, the ancestry-based linkage disequilibria are broken down more rapidly. Profiles are averages across 10,000 replicate simulation trials. Simulation details can be found in the Methods.

In contrast, because effect size estimates from within-family GWASs are not confounded in this model of isolated populations, the average squared effect size estimate will not differ substantially from its expectation in an unstructured population (Fig. 5).

While we have focused on a simple model of two isolated populations, the result that within-family association studies are not confounded holds for other kinds of population structure as well. Specifically, we may be concerned that a population GWAS suffers from genetic confounding along some given axis of population stratification. However, the family-based estimates will be unbiased by confounding along such an axis if the maternal and paternal genotypes at each locus are exchangeable with respect to each other along this axis (Appendix A3.2). This requirement will be met in expectation under many models of local genetic drift in discrete populations or along geographic gradients. However, as we will shortly argue, migration and admixture introduce further complications.

Allele frequency divergence due to selection or phenotype-biased migration.

Selection and phenotype-biased migration can also generate allele frequency differences among populations (for a review of phenotype-biased migration, see Edelaar and Bolnick 2012). Unlike genetic drift, both of these forces can lead to systematic directional associations between effect sizes and changes in allele frequencies between populations. For example, if selection has favored alleles that increase the trait in population 1 but not in population 2, then

$$\mathrm{E}\left[\left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right){\alpha }_l^{\mathrm{d}}\right]>0.$$ (14)

as directional selection causes systematic changes in allele frequencies across the loci $l$ underlying variation in the trait under selection (e.g., Hayward and Sella 2022). Importantly, this form of selection can occur even if the mean phenotype of the two populations does not change (Harpak and Przeworski 2021; Yair and Coop 2022). Similarly, phenotype-biased migration, where, say, individuals with a higher value of the phenotype tend to migrate from population 2 to population 1, can also create a positive association between effect sizes and allele frequency differences (Eq. 14).

Unlike the case of neutral genetic drift in the two populations, where the sign of the LD between two alleles is independent of their effect sizes, the effect-size-correlated associations driven by selection or phenotype-biased migration can add up across loci, and thus lead to substantial, systematic biases in estimates of allelic effect sizes. This systematic genetic confounding would also substantially inflate the average squared effect size estimate and thus measures of the genetic variance tagged by SNPs.

In addition, these systematic sources of genetic confounding can generate genetic correlations between traits with no overlap in their sets of casual loci—i.e., with no pleiotropic relationship. This will occur if two traits have both experienced selection or biased migration along the same axis. To take a concrete example, if people tend to migrate to cities in part based on traits 1 and 2, then these traits will become genetically correlated. If this axis is explicitly included as a covariate in the GWAS, then its influence on estimates of heritability and genetic correlations will be removed. However, its influence will not be removed by inclusion of genetic principal components or the relatedness matrix, if this axis (here, city vs. non-city) is not a major determinant of genome-wide relatedness at non-causal loci (Vilhjálmsson and Nordborg 2013). Nor will LD score regression control for this influence, as the selection- or migration-driven differentiation of a variant along the axis will be correlated with the extent to which it tags long-range causal variants involved in either trait. This effect on LD score regression is similar to that discussed above for assortative mating (Border et al. 2022a,b). Thus, like assortative mating, selection and phenotype-biased migration along unaccounted-for axes of population stratification can generate genetic correlations between traits. These selection- and migration-driven correlations should not necessarily be viewed as spurious, since genetic correlations should include those that arise from systematic long-range LD, but they complicate the interpretation of population-level genetic correlations as evidence for pleiotropy.

Again, these issues largely vanish in family-based studies, although phenotype-biased migration can cause transient differences in cis- and trans-LD that lead to biases in family-based estimates of direct effects (Eqs. 6 and 7).

3.3. Admixture

When populations that have previously been separated come into contact, alleles from the same ancestral population remain associated with each other in the admixed population until they are dissociated by recombination. If allele frequencies had diverged between the ancestral populations, this ‘ancestry disequilibrium’ can translate to cis-LD between loci affecting a trait (Nei and Li 1973), potentially confounding GWASs performed in the admixed population. More generally, long range LD will be an issue when there is genetic stratification and ongoing migration between somewhat genetically distinct groups.

For concreteness, we again consider a simple model where two populations have been separated for some time, allowing allele frequencies to diverge between them. The populations then come into contact and admix in the proportions $A$ and $1-A$. We assume that mating is random with respect to ancestry in the admixed population.

Suppose that, just before admixture, the frequencies of the focal allele at a given locus $k$ were $p_k^{\left(1\right)}$ and $p_k^{\left(2\right)}$ in the two populations. Then the initial degree of cis-LD between loci $\lambda$ and $l$ in the admixed population is given by Eq. (11), weighted by the proportions in which the populations admix:

$$D_{\lambda l,0}^{\left(A\right)}=A\left(1-A\right)\left(p_{\lambda }^{\left(1\right)}-p_{\lambda }^{\left(2\right)}\right)\left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right);$$ (15)

see, e.g., Pfaff et al. (2001). This cis-LD subsequently decays at a rate c_λl per generation, so that, $t$ generations after admixture,

$$D_{\lambda l,t}^{\left(A\right)}=D_{\lambda l,0}^{\left(A\right)}{\left(1-c_{\lambda l}\right)}^t=A\left(1-A\right)\left(p_{\lambda }^{\left(1\right)}-p_{\lambda }^{\left(2\right)}\right)\left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right){\left(1-c_{\lambda }\right)}^t.$$ (16)

Because we assume that mating is random in the admixed population, the trans-LD is zero in every generation after admixture: ${\tilde{D}}_{\lambda l,t}^{(A)}=0$. Note that the decay of cis-LD in an admixed population will be slowed if individuals mate assortatively by ancestry, because the trans-LD generated by assortative mating is contimuslly converted by recombination to new cis-LD (as in our assortative mating model above; see Zaitlen et al. (2017) for more discussion of this point in the context of population admixture).

Allele frequency divergence due to drift.

How do these patterns of LD affect a population GWAS? If allele frequency differences between populations arose from neutral drift, they will be independent of effect sizes at causal loci and across loci, and therefore will not contribute, on average, a systematic directional bias to effect size estimates. However, they will inflate the average squared effect size estimate, by a smaller amount than for a population GWAS performed when the populations were still separated (because of the elimination of trans-LD under random mating in the admixed population). Moreover, this amount will decline in the generations after admixture as the remaining cis-LD is eroded by recombination (Eq. 16; Fig. 5). We quantify these effects in Appendix A3.3 (see also Pfaff et al. 2001^; Rosenberg and Nordborg 2006^; Zaitlen et al. 2014^; Lee and Lee 2023b).

Although within-family GWASs were not genetically confounded when the populations were separate (because cis- and trans-LDs were equal, as discussed above), they become genetically confounded in the admixed population, as all trans-LD is eliminated by random mating in the admixed population, leaving an excess of cis-LD relative to trans-LD that biases effect size estimates (Eqs. 6 and 7). As in the case of the population GWAS, these biases will be zero on average if allele frequency differences between the ancestral populations were due to drift. However, after admixture, they will still inflate the average squared effect size estimate (and thus the variance of effect size estimates), which will thereafter decline in subsequent generations as the cis-LD is gradually broken down by recombination (Eq. 16; Fig. 5).

In comparing the average squared effect size estimate in a population and a family-based GWAS, we observe that the value in the population GWAS rapidly declines to approximately the same level as the value in the within-family GWAS, despite the former having started at a much higher level in the initial admixed population (Fig. 5). The explanation is that LD between unlinked loci confounds effect size estimation in the population GWAS but not the within-family GWAS, such that (i) the average squared effect size estimate from the population GWAS is initially much higher than that from a within-family GWAS, because it is inflated by LD between many more pairs of loci, and (ii) the average squared effect size estimate from the population GWAS declines more rapidly, because LD between unlinked loci is broken down more rapidly than LD between linked loci.

Allele frequency divergence due to selection or phenotype-biased migration.

In addition to drift, and as discussed above, selection and phenotype-biased migration can generate systematic, signed (effect-size correlated) LD, which would lead to systematic cis-LD in the descendent admixed population. These would lead to larger inflations of genetic variance and genetic correlations than would be expected had allele frequency divergence between the ancestral populations been due to drift alone, and would complicate interpretations of genetic correlations as being due to pleiotropy. Moreoever, if the admixed population is more than a few generations old such that LD between unlinked loci but not linked loci has largely been broken down, then population- and family-based estimates of these quantities might be similar.

Spurious genetic correlations due to confounding in population-based PGSs.

Factors other than selection and phenotype-biased migration can also generate non-pleiotropic genetic correlation signals in family-based studies of admixed populations. In fact, the use of confounded population GWAS effect sizes can be sufficient. As an example of the confounding of genetic correlations in admixed populations due to a confounded GWAS for one trait, consider the GIANT-GWAS height polygenic score. Owing to confounding within Europe (Berg et al. 2019; Sohail et al. 2019), the height PGS showed large differences between Northern Europeans and sets of individuals sampled in other locations, such as the African 1000 genomes samples (Martin et al. 2017). This confounding generated a spurious, systematic correlation between height effect sizes and allele frequency differences across populations, with height-increasing alleles that are more common among Northern Europeans being assigned larger effects (Berg et al. 2019). As a result, in a PGS constructed from these effect size estimates, larger PGS values are predictive of greater North European ancestry. Now imagine a sibling-based study performed in a sample with recently admixed ‘European’ and ‘non-European’ ancestry—African Americans, for example. An individual with a larger value than their sibling for the GIANT height PGS will, on average, carry more ‘European’ ancestry. In African Americans, there will also be a systematic association of lighter skin pigmentation with recent ‘European’ ancestry, and selection on skin pigmentation will have driven a signed difference in allele frequencies between European and West African ancestors. Putting these observations together, the GIANT height PGS, being predictive of the degree of European ancestry, may well be predictive of skin pigmentation differences between African American sibling pairs (Eq. 10), leading to the naive and incorrect conclusion that height and skin colour are causally linked. In reality, this result would reflect the fact that alleles predicted to increase height and alleles that affect skin color are in systematic effect-signed admixture LD, as in Eq. (15), as a consequence of stratification-biased effect size estimates from the GIANT European GWAS.

3.4. Stabilizing selection

Stabilizing selection—selection against deviations from an optimal phenotypic value—is thought to be common (Sella and Barton 2019), and has recently been argued to be consistent with the genetic architectures of many human traits (Simons et al. 2022). By disfavoring individuals with too many or too few trait-increasing alleles, stabilizing selection generates negative cis-LD among alleles with the same directional effect on the trait (Bulmer 1971). Thus, stabilizing selection will attenuate GWAS effect size estimates at genotyped loci that tag these causal loci.

To quantify these biases, we consider the model of Bulmer (1971^{, 1974}), in which a large number of loci contribute to variation in a trait under stabilizing selection, with the population having adapted such that the mean trait value is equal to the optimum. Under this model, stabilizing selection rapidly reduces variance in the trait by generating negative cis-LD among trait-increasing alleles. If we make the simplifying assumption that all loci have equal effect sizes, then the equilibrium reduction in trait variance, $-d^{\mathrm{*}}$ (where $d^{\mathrm{*}}<0$), can be calculated as a function of the genic variance $V_g$, the environmental noise $V_E$, the strength of stabilizing selection $V_S/V_P$ (scaled according to the phenotypic variance $V_P$), and the harmonic mean recombination rate, ${\bar{c}}_h$, among loci underlying variation in the trait (Bulmer 1974; Appendix A.3.4).

Under these same assumptions, we calculate in Appendix A3.4 the average per-locus attenuation bias in effect size estimates induced by stabilizing selection, $\left({\alpha }_l-{\hat{\alpha }}_l\right)/{\alpha }_l$. In a population GWAS, this attenuation bias is approximately

$$\frac{{\alpha }_l-{\hat{\alpha }}_l^{\mathrm{p}\mathrm{o}\mathrm{p}}}{{\alpha }_l}=-\frac{d^{\mathrm{*}}}{V_g}.$$

In a within-family GWAS, the average proportionate bias is approximately

$$\frac{{\alpha }_l-{\hat{\alpha }}_l^{\mathrm{f}\mathrm{a}\mathrm{r}\mathrm{m}}}{{\alpha }_l}=-\frac{d^{\mathrm{*}}\left(1-2{\bar{c}}_h\right)}{V_g};$$

i.e., smaller than in a population GWAS by a factor of $1-2{\overline{c}}_h$

Thus, the bias in effect size estimation can be calculated given estimates of the phenotypic variance and heritability of the trait, the harmonic mean recombination rate, and the strength of stabilizing selection (Appendix A3.4). In the Methods, making some simplifying assumptions about the genetic architecture of the trait in question, we calculate an approximate value ${\bar{c}}_h\approx 0.464$ for humsns. Using this value, Fig. 6 shows the average proportionate reduction in GWAS effect size estimates for various strengths of stabilizing selection and heritabilities of the trait. The range of selection strengths was chosen to match that inferred for human traits by Sanjak et al. (2018).

Figure 6:

Stabilizing selection attenutates GWAS effect size estimates. The calculations displayed here assume that genetic variation in the trait is contributed by 1,000 loci of equal effect spaced evenly along the human genome. Stabilizing selection is stronger if the width of the selection function scaled by the phenotypic variance, $V_S/V_P$, is smaller. The placement of the point for human height assumes a heritability of 0.8 and a strength of stabilizing selection of $V_S/V_P=30$, as estimated by Sanjak et al. (2018). Details of these calculations can be found in Appendix A3.4. Note the different scales of the $y$ -axes in $A$ and B.

Attenuation of effect size estimates is larger if stabilizing selection is stronger or if the trait is more heritable. Taking height as an example, heritability is ~0.8, $V_P\approx 7{\mathrm{c}\mathrm{m}}^2$, and Sanjak et al. (2018) estimate a sex-averaged strength of stabilizing selection of $V_S/V_P\approx 30$. From these values, we calculate that a population GWAS would systematically underestimate effect sizes at loci that causally influence height by about 3% on average, in the absence of other sources of LD (Fig. 6A). More generally, within the range of reasonable strengths of stabilizing selection inferred by Sanjak et al. (2018), we calculate average attenutations of population-based effect size estimates of up to 5% for highly heritable traits ${(h}^2\approx 1)$ under strong stabilizing selection $\left(V_S/V_P\approx 20\right)$, down to 0.25% for less heritable traits $\left(h^2\approx 0.4\right)$ under weak stabilizing selection $\left(V_S/V_P\approx 170\right)$ (Fig. 6A).

Given the estimate ${\bar{c}}_h~0.464$, the proportionate bias that stabilizing selection induces in within-family GWASs is expected to be a fraction $1-2{\bar{c}}_h\approx 7\mathrm{\%}$ that in population-based GWASs. Thus, for height, a within-family GWAS would underestimate effect sizes by only about 0.2% on average (Fig. 6B).

The quantitative importance of these biases will vary by application. In situations where the goal is gene discovery, for example, 5% reductions in effect size estimates are unlikely to flip the statistical significance of variants with large effects on a trait. However, the attenuations in effect size estimates caused by stabilizing selection are systematic across loci, and therefore could substantially affect aggregate quantities based on these estimates. For example, the range of average reductions in population effect size estimates calculated above for human traits would translate to reductions in naive estimates of SNP-based heritabilities of between 0.5% and 10% (~6% in the case of height). If effect sizes are estimated by within-family GWAS, on the other hand, the reductions in these SNP-based heritability estimates would be much smaller.

As a further example, by generating negative LD between alleles with the same directional effect on the trait, the impact of stabilizing selection opposes, and therefore masks, the genetic impact of assortative mating (Brown et al. 2016). A practical consequence is that stabilizing selection will tend to attenuate estimates of the strength of assortative mating based on GWAS effect sizes, which often use cross-chromosome correlations of polygenic scores (e.g., Yengo et al. 2018^; Yamamoto et al. 2023). In humans, the phenotypic correlation among mates for height has been measured at about ~0.25 (Stulp et al. 2017). In Appendix A3.4, we calculate that estimates of this correlation based on cross-chromosome correlations in PGSs will be biased downwards by about 20%, to ~0.20, because of stabilizing selection on height. Were assortative mating weaker, or stabilizing selection stronger, the genetic impact of assortative mating would be masked to an even greater extent (Appendix A3.4).

As in our analysis of assortative mating above, if stabilizing selection ceases in some generation, the negative LD that built up during the period of stabilizing selection will decay over subsequent generations, rapidly for pairs of loci on different chromosomes and more slowly for linked pairs of loci. Patterns of selection on human traits have changed over time—for example, the strength of stabilizing selection on birth weight has relaxed (Ulizzi and Terrenato 1987). In general, therefore, patterns of confounding reflect a composite of contemporary and historic processes.

3.5. Sibling indirect effects

Indirect effects of siblings’ genotypes on each other’s phenotypes are known to be a potential source of bias in sibling-based GWASs (Fletcher et al. 2021; Young et al. 2022), and can be measured and corrected only if, in addition to sibling genotypes, parental genotypes are also available (Kong et al. 2018; Young et al. 2022). To generate intuition for their impact on GWASs, we consider a simple model of indirect sibling effects in the absence of $\mathrm{G}\times \mathrm{E}$ interactions and other confounding effects, focusing on a single-locus model for simplicity. We suppose that the indirect effect of an individual’s phenotype on their sibling’s phenotype is $\beta$, so that the phenotypes of two siblings $i$ and $j$ can be written

$$\begin{array}{rr}& Y_i=Y^{\mathrm{*}}+\alpha g_i+\beta Y_j+{ϵ}_i,\\ & Y_j=Y^{\mathrm{*}}+\alpha g_j+\beta Y_i+{ϵ}_j.\end{array}$$ (17)

Taking their difference and rearranging, we find that

$$\mathrm{\Delta }Y=\frac{\alpha }{1+\beta }\mathrm{\Delta }g+\frac{1}{1+\beta }\mathrm{\Delta }ϵ.$$ (18)

Therefore, in the absence of genetic confounding and $\mathrm{G}\times \mathrm{E}$ interactions, a sibling-based association study would return an effect size estimate of

$${\hat{\alpha }}^{\mathrm{s}\mathrm{i}\mathrm{b}}=\frac{\alpha }{1+\beta }$$ (19)

on average. Thus, if sibling indirect effects are synergistic $(\beta >0)$, they lead to underestimation of the direct genetic effect at the locus. In contrast, if sibling indirect effects are antagonistic $(\beta <0)$, they lead to overestimation of the direct genetic effect.

How would a population GWAS be affected by the same sibling indirect effects? Sibling i’s phenotype can be written

$$\begin{array}{l}{Y}_i=Y^{\mathrm{*}}+\alpha g_i+\beta Y_j+{ϵ}_i,\\ =Y^{\mathrm{*}}+\alpha g_i+\beta (Y^{\mathrm{*}}+\alpha g_j+\beta Y_i+{ϵ}_j)+{ϵ}_i\\ \Rightarrow Y_i=\frac{1}{1-{\beta }^2}(Y^{\mathrm{*}\mathrm{*}}+\alpha g_i+\alpha \beta g_j+{ϵ}_i+\beta {ϵ}_j),\end{array}$$ (20)

where $Y^{\mathrm{*}\mathrm{*}}=(1+\beta )Y^{\mathrm{*}}$. Therefore, if we were to randomly choose one sibling from each sibship and estimate the effect size at the locus using a population association study across families, we would obtain

$${\hat{\alpha }}^{\text{pop}}=\frac{\mathrm{Cov}\left(g_i,Y_i\right)}{\mathrm{Var}\left(g_i\right)}=\frac{1}{1-{\beta }^2}\left(\alpha +\alpha \beta r_g^{\mathrm{s}\mathrm{i}\mathrm{b}\mathrm{s}}\right),$$ (21)

where $r_g^{\text{sibs}}=\mathrm{C}\mathrm{o}\mathrm{v}\left(g_i,g_j\right)/\mathrm{V}\mathrm{a}\mathrm{r}\left(g_i\right)$ is the genotypic correlation between sibs at the locus. Sibling indirect effects alter the effect size estimate in a population GWAS via two channels. The first is through the factor $1/\left(1-{\beta }^2\right)$ in Eq. (21), which reflects second-order feedbacks of an individual’s phenotype on itself, via the sibling. Since $1/\left(1-{\beta }^2\right)>1$, these feedbacks act to exacerbate the effects of causal alleles. For example, if sibling indirect effects are antagonistic $(\beta <0)$, then a sibling with a large trait value will tend to indirectly reduce the trait value of their sibling, which in turn will indirectly further increase the trait value of the focal individual. This channel therefore pushes population GWASs towards overestimating the magnitude of direct genetic effects.

The other channel by which sibling indirect effects can influence a population GWAS is driven by the genotypic correlation among siblings, and is easiest to understand if we assume that sibling indirect effects are weak $\left({\beta }^2\ll 1\right)$. In this case, ${\hat{\alpha }}^{\text{Pop}}\approx \alpha +\alpha \beta r_g^{\text{sibs}}$. Since the genotypic correlation $r_g^{\text{sibs}}>0$, this channel of sibling indirect effects has the opposite effect to the one it has on a sibling GWAS: if sibling indirect effects are synergistic $(\beta >0)$, the population GWAS overestimates the direct genetic effect at the locus, while if sibling indirect effects are antagonistic $(\beta <0)$, the population GWAS underestimates the direct genetic effect. The reason for this difference is that a sibling GWAS is based on siblings whose genotypes differ at the focal locus, and whose genotypic values are therefore anticorrelated. If sibling indirect effects are synergistic $(\beta >0)$, they will tend to attenuate the phenotypic differences between such siblings, and therefore attenuate effect size estimates. In contrast, because siblings’ genotypes are positively correlated across the entire population, synergistic sibling indirect effects $(\beta >0)$ will tend to exacerbate phenotypic differences across families, leading a population GWAS to overestimate effect sizes.

3.6. Gene-environment (GxE) and gene-gene (GxG) interactions

Up to this point, we have assumed that alleles’ direct effects do not vary across environments or genetic backgrounds. To generate intuition for the influence of $\mathrm{G}\times \mathrm{E}$ (and $\mathrm{G}\times \mathrm{G}$) interactions on population and family-based GWAS designs, we restrict our focus to a single causal locus, assuming no genetic confounding and no indirect effects of siblings. To incorporate $\mathrm{G}\times \mathrm{E}$ interactions, we allow the effect size of the alleles at the locus to depend on the family environment. The phenotype of individual $i$ in family $f$ is

$$Y_i=Y^{\mathrm{*}}+\left(\alpha +{\alpha }_f+{\alpha }_i\right)g_i+{ϵ}_f+{ϵ}_i,$$ (22)

where we arbitrarily define ${\alpha }_f$ and ${\alpha }_i$ such that their population means are zero: $\mathbb{E}\left[{\alpha }_f\right]=\mathbb{E}\left[{\alpha }_i\right]=0$. $\alpha$ is then the average causal effect of the allele were it randomized across individuals from different families in our sample. ${\alpha }_f$ is the deviation of this effect in family $f$ due to their environment, and ${\alpha }_i$ is an individual deviation which we assume to be independent of $i$ ‘s genotype; both ${\alpha }_f$ and ${\alpha }_i$ can be thought of as random slopes in a mixed model. Note that ${\alpha }_f$ can reflect the interaction of alleles with the family’s external environment (as we have framed it here) or with the family’s genetic background (a $\mathrm{G}\times \mathrm{G}$ interaction).

If we perform a sibling GWAS by taking pairs of full siblings $i$ and $j$ in family $f$ and regressing the difference in their phenotypes $\mathrm{\Delta }{Y}_f=Y_i-Y_j$ on the difference in their genotypes at the focal locus $\mathrm{\Delta }{g}_f=g_i-g_j$, we obtain an effect size estimate

$${\hat{\alpha }}^{\text{sib}}=\alpha +\mathbb{E}\left[{\alpha }_f\mid \text{parent}\text{heterozygous}\right],$$ (23)

where the second term—the deviation of the family-based estimate ${\hat{\alpha }}^{\text{sib}}$ from $\alpha$—is the average family deviation conditional on a parent being heterozygous at the focal locus (Appendix A4). The intuition is that, because only heterozygous parents contribute the genetic variation among siblings on which our effect size estimate is based, if these heterozygous parents are non-randomly distributed across environments, then the family-based GWAS samples values from a distribution of family effects ${\alpha }_f$ that is different to the overall population distribution.

We can compare this estimate from a sibling GWAS to one from a population GWAS, again under the assumption of no genetic confounding or indirect effects from siblings:

$${\hat{\alpha }}^{\mathrm{p}\mathrm{o}\mathrm{p}}\approx \alpha +(1-2p)(1-2F)\mathbb{E}\left[{\alpha }_f\mid g_i=1\right]+2(p+(1-2p\left)F\right)\mathbb{E}\left[{\alpha }_f\mid g_i=2\right]$$ (24)

where $p$ is the frequency of the focal variant and $F$ is the inbreeding coefficient at the locus (Appendix A4). The approximation holds if $F$ is small. Note that Eq. (24) conditions on the number of focal alleles carried by the sampled individual, whereas Eq. (23) conditions instead on the parental genotype.

Like the family-based estimate, the population-based effect size estimate is distorted when heterozygotes are not randomly distributed over family backgrounds $\left(E\left[{\alpha }_f\mid g_i=1\right]\ne 0\right)$ as well as when homozygotes are not randomly distributed across family backgrounds (when $E\left[{\alpha }_f\mid g_i=2\right]\ne 0$). Thus, effect size estimates from both family- and population-GWAS can differ from the genetic effects that would be estimated if genotypes were randomly distributed across interacting family backgrounds, and these distortions will in general not be the same aross population and family-based study designs.

As noted above, because of current sample size constraints in family-based studies, a common strategy is to calculate the association of population-based PGSs and phenotypic differences among family members. In the absence of confounding, it is clear from Eq. (10) that the influence of $\mathrm{G}\times \mathrm{E}$ interactions on the covariance of sibling differences in PGSs and trait values would depend on the average value of ${\hat{\alpha }}^{\text{sib}}{\hat{\alpha }}^{\text{pop}}$ across loci. Thus the slope of the PGS in this regression could be affected if, on average, the alleles at casual loci tagged by genotyped variants in the PGS are more often found in environments that suppress (or enhance) their effects. $\mathrm{G}\times \mathrm{E}$ interactions across many loci have been suggested by some recent studies (Mostafavi et al. 2020; Zhu et al. 2022), but their quantitative impact on differences between population and family-based GWASs remains unknown.

An allele’s effect could also systematically differ across families $\left({\alpha }_f\right)$ if it is involved in epistatic interactions with alleles at other loci in the genome $(\mathrm{G}\times \mathrm{G})$. By analogy to our $\mathrm{G}\times \mathrm{E}$ model above, epistatic interactions would lead to biases in family-based GWASs if parents who are heterozygous at the focal study locus tend to have systematically different genotypes at loci that interact epistatically with the focal locus, relative to the population distribution of such genetic backgrounds.

Up to this point, we have also ignored parent-offspring interactions as a possible source of bias in family-based studies. Following the the same logic above, interactions between parents’ and offsprings’ alleles will result in family GWAS estimates that are the average effect of the focal allele in an offspring conditional on the genetic background of a heterozygous parent. Thus, again, a non-random distribution of genetic backgrounds in heterozygous parents is a potential source of bias.

One way that heterozygous parents might exhibit a non-random distribution of genetic backgrounds is via trait-based assortative mating, which could therefore modify the way that epistasis and parent-offspring interactions influence effect size estimation in a family-based GWAS relative to a population GWAS and relative to the true average population effect.

A final, overarching complication is that the individuals participating in a population GWAS are not a random subset of the population(s) from which they are drawn (Fry et al. 2017; Pirastu et al. 2021; Tyrrell et al. 2021), and families enrolled in GWASs can be even less representative of the population as a whole (Mostafavi et al. 2020; Benonisdottir and Kong 2022). These participation biases can potentially lead to systematic differences between the distributions of genotypes and interacting environments experienced by the population, the GWAS sample, and participants in a family-based study.

4. Discussion

It has long been recognized that population GWASs in humans can be biased by environmental and genetic confounding (Lander and Schork 1994; Vilhjálmsson and Nordborg 2013). Currently, population GWASs attempt to control for these confounds by focusing on sets of individuals that are genetically more similar and by controlling for population stratification. However, these controls are imperfect and are not always well defined. For example, controlling for genome-wide patterns of population stratification based on common alleles does not control for the genetic and environmental confounding of rare variants (Zaidi and Mathieson 2020). Work on genetic confounding has uncovered increasing evidence that assortative mating may be leading to large biases in estimates of direct genetic effects and to large genetic correlations for a number of traits (Yengo et al. 2018; Border et al. 2022b,a); moreover, it can often be unclear whether genetic signals of assortative mating are due to trait-based mate choice or some other more general form of genetic confounding (e.g., Haworth et al. 2019). Additionally, while we have focused primarily on genetic confounding, for a number of traits there are also signals of residual environmental confounding in GWAS signals (Selzam et al. 2019; Mostafavi et al. 2020; Okbay et al. 2022; Abdellaoui et al. 2022). Thus, subtle and often interwoven forms of genetic and environmental confounding remain a major issue in many GWASs (Young et al. 2022), compromising the interpretation of GWAS effect size estimates and downstream quantities such as SNP heritabilities and genetic correlations.

Effect size estimates from within-family GWASs are less affected by these various confounds. In the absence of $\text{G}\times \text{E}$ interactions, they are not subject to environmental confounding across families, because the environments of family members are effectively randomized with respect to within-family genetic transmission. As we have shown, family-based estimates should also suffer substantially less from genetic confounding, because genetic transmission at unlinked loci (but not linked loci) is randomized by independent assortment of chromosomes in meiosis. Nonetheless, family-based GWASs can suffer from residual genetic confounding as well as sibling indirect effects and $\text{G}\times \text{E/G}\times \text{G}$ interactions; they also raise a number of conceptual problems that we discuss below.

Sources of genetic confounding.

Genetic confounding is caused by long-range LD between loci that affect the trait or traits under study. To illustrate the potential for genetic confounds to bias GWAS effect size estimates, we have considered several sources of long-range LD. Some of these—assortative mating, selection on GWAS traits, and phenotype-biased migration—can cause systematic directional biases in GWAS effect size estimates. Others, such as neutral population structure, cause random biases that influence the variance of effect size estimates and related quantities. Assortative mating and neutral population structure have received considerable theoretical attention in the GWAS literature (e.g., Rosenberg and Nordborg 2006^; Yengo et al. 2018^; Border et al. 2022a^,b). Here, we have further outlined how both selection and phenotyped-biased migration can drive systematic genetic confounding that may not be well accounted for by current methods of controlling for stratification.

We wish to emphasize stabilizing selection in particular as a potential source of systematic confounding in GWASs. Stabilizing selection has been well studied in the quantitative genetics literature but less so in the context of GWASs, despite its expected ubiquity. By selecting for compensating combinations of trait-increasing and trait-decreasing alleles, stabilizing selection generates negative LD between alleles with the same directional effect on the trait (Bulmer 1971, 1974), and can therefore bias GWAS effect size estimates downwards. While the potential for stabilizing selection to confound effect size estimation has been noted (e.g., Brown et al. 2016^; Yair and Coop 2022^; Li et al. 2023), the resulting biases have not, to our knowledge, been quantified. Our calculations suggest that these downward biases could, for some human traits, be as large as 5% systematically across all causal loci in population GWASs. While biases of this magnitude are unlikely to compromise some goals of GWASs, such as gene discovery, they could be quantitatively problematic for other GWAS aims, such as estimation of SNP heritabilities and the strength of assortative mating. Moreover, while our results pertain to (a particular model of) stabilizing selection, many kinds of selection generate LD between genetically distant loci—in fact, only multiplicative selection among loci does not (Bürger 2000, pgs. 50 and 177). Therefore, the general result that selection can generate genetic confounding will hold more broadly.

For a given genotyped locus in a GWAS, there is no bright line between local ‘tagged’ LD and long-range confounding LD, and one reasonable objection to the approach taken here is that that we have used an arbitrary definition of the causal loci that are locally tagged by a genotyped locus (L_local in Eq. 2). All of the sources of genetic confounding that we have considered generate LD among causal loci both within and across chromosomes. Under these models, the within-chromosome LD that is generated is, in a sense, a continuation of the LD generated across chromsomes (moving from a recombination rate = 0.5 to ≤ 0.5). Thus, while investigators may prefer some looser definition of ‘local’ when thinking about genotyped GWAS loci as tag SNPs, to extend that definition to include all loci on the same chromosome as the SNP would, by reasonable interpretation, be to include confounding into the desired estimator.

The extent to which the absorption of genetic confounding in estimated effect sizes is a problem depends on the application. In the case of polygenic prediction, absorbing environmental effects, indirect effects, the effects of untyped loci throughout the genome can help to improve prediction accuracy, although this does come at a cost to interpretability. For GWAS applications focused on understanding genetic causes and mechanisms, the biases in effect size estimates and spurious signals of pleiotropy among traits generated by genetic confounding will be more problematic.

Indirect genetic effects.

Family GWASs are often interpreted as providing the opportunity to ask to what extent parental genotypes (or other family genotypes) causally affect a child’s phenotype (‘genetic nurture’; Kong et al. 2018). Viewed in this way, the association between untransmitted parental alleles and the child’s phenotype would seem, at first, a natural estimate of indirect genetic effects.

In practice, however, if the population GWAS suffers from genetic and environmental confounds, then the estimated effects of untransmitted alleles will absorb that confounding in much the same way that estimates of direct genetic effects from a population GWAS do (Eq. 8; Shen and Feldman 2020). For example, in the case of assortative mating, a given untransmitted allele is correlated with alleles that were transmitted both by this parent and by their mate, and these transmitted alleles can directly affect the offspring’s phenotype. Thus, while family-based estimates of direct genetic effects benefit from the randomization of meiosis and from controlling for the environment, family-based estimates of indirect genetic effects lack both of these features and should be interpreted with caution. Indeed, recent work using parental siblings to control for grandparental genotypes has shown that little of the estimated ‘indirect genetic effect’ may be causally situated in parents (Nivard et al. 2022). With empirical estimates of indirect genetic effects potentially absorbing a broad set of confounds (Demange et al. 2022; Young et al. 2022), and few current studies of indirect effects having designs that allow such confounding to be disentangled, it is premature—and potentially invalid—to interpret associations of untransmitted alleles causally in terms of indirect genetic effects (Wolf et al. 1998). Rather, they should be treated agnostically in terms of ‘non-direct’ effects.

Direct genetic effects.

Mendelian segregation provides a natural randomization experiment within families (Fisher 1952), and so crosses in experimental organisms and family designs have long been an indispensable tool to geneticists in exploring genetic effects and causation. Growing concerns about GWAS confounding and the increasing availability of genotyped family members have led to a return of family-based studies to the association study toolkit (Young et al. 2019). Family-based estimates of direct genetic effects are often interpreted as being unbiased and discussed in terms of the counterfactual effect of experimentally substituting one allele for another (Morris et al. 2020; Brumpton et al. 2020; Young et al. 2022).

As we have shown, family-based GWASs are indeed less subject to confounding than population-based GWASs: in the presence of genetic and environmental confounding, the family-based estimate of the effect size at a given locus provides a much closer approximation to the true effects of tightly linked causal loci than a population-based estimate does. The family-based estimate is not biased by environmental variation across families and avoids the correlated effects of the many causal loci that lie on other chromosomes. Still, the family-based estimate does absorb the effects of non-local causal loci on the same chromosome, and so cannot truly be said to be free of genetic confounding. Rather than considering a single allele being substituted between individuals, a better experimental analogy for the effect size estimate would be to say that we are measuring the mean effect of transmission of a large chunk of chromosome surrounding the focal locus, potentially carrying many causal loci.

In addition, while within-family GWASs offer these advantages, in other ways, they move us further away from the questions about the sources and causes of variation among unrelated individuals that motivate population GWASs in the first place. Indeed, the presence of confounding introduces a number of conceptual issues in moving from within-family GWAS to the interpretation of differences among individuals from different families (Coop and Przeworski 2022a,b). For example, in the presence of genetic confounding, the effect of a causal allele of interest will depend on a set of weights: its LD to many other causal alleles. In estimating the direct effect of the allele, family-based approaches weight these LD terms differently to population-based approaches, which, we argue, can complicate the interpretation of these estimates. For example, when previously isolated populations admix, same-ancestry alleles will be held together in long genomic blocks until these are broken up by recombination, which will happen very quickly for alleles on different chromosomes but more slowly for alleles on the same chromosome. A few generations after admixture, therefore, cross-chromosome ancestry LD will largely have dissipated, but contiguous ancestry tracts will still span substantial portions of chromosome lengths. Since both population and within-family GWASs are similarly confounded by the same-chromosome LD, their mean squared effect sizes will be similar in this case (Fig. 5). Bearing in mind that the LD resulting from admixture is not present in the source populations, it becomes unclear which weighting of ancestry LD is appropriate if we want to interpret the resulting effect size estimates as direct effects. As this example illustrates, while family-based GWASs are a useful device for dealing with confounding, it is not always obvious how to interpret the quantities that they measure.

A number of additional complications arise when, to compensate for the small effect sizes of individual loci, researchers combine many SNPs into a polygenic score (PGS) and study the effects of PGSs within families (or use them as instruments in Mendelian randomization analyses). For one, SNPs are usually chosen for inclusion in the PGS on the basis of their statistical significance in a population GWAS. This approach prioritizes SNPs whose effect size estimates are amplified (or even wholly generated) by confounding (for an example of how this leads to residual environmental confounding in applications of sibling-based effect size estimates, see Zaidi and Mathieson 2020). Second, the weights given to SNPs that are included in the PGS absorb the effects of confounding, and this confounding is heterogeneous across SNPs. Thus, when we study the correlates of trait-A PGS differences between siblings in the presence of GWAS confounding, we are not observing the average phenotypic outcomes of varying the genetic component of trait A between siblings. Rather, we are varying a potentially strangely-weighted set of genetic correlates of trait A.

An observation that a population GWAS PGS is predictive of phenotypic differences among siblings demonstrates that the PGS SNPs tag nearby causal loci, but beyond that, interpretation is difficult. Notably, if there is cross-trait assortative mating for traits A and B, but no pleiotropic link between the traits, then some of the SNPs identified as significant in a GWAS on trait A may be tightly linked to loci that causally affect trait B but not trait A. If these loci are included in the trait-A PGS, then when we study the effect of variation in the trait-A PGS on sibling differences, we are accidentally absorbing some components of the variation in trait B across siblings. Thus, we might observe a correlation between the trait-A PGS and differences in trait B between siblings, and this correlation may be lower than is observed at the population level, without there existing any pleiotropic (or causal) link between A and B. These effects can be exacerbated if the two traits have different genetic architectures (Figure 4). Instead of using a set of SNPs and weights from a population GWAS, genetic correlations between traits due to pleiotropy could be estimated from the correlation of effect sizes estimated within families (Howe et al. 2022). Given current sample size constraints in family-based studies, the confidence intervals on these estimates are large. Moreover, significant family-based correlations need not reflect pure pleiotropy, since, as we have shown, they are not completely free of genetic confounding due to intra-chromosomal LD.

Also complicating the interpretation of family-based effect size estimates are various types of interactions. Indirect effects between siblings can bias family estimates of direct genetic effects (Eq. 19; Young et al. 2019^; Fletcher et al. 2021^; Young et al. 2022) in ways that are conceptually different from the biases they introduce to population-based estimates (Eq. 21). These sibling effects can potentially be addressed with fuller family information (e.g., parental genotypes in addition to sibling genotypes; Kong et al. 2018^; Young et al. 2022).

As we have further shown here, $\text{G}\times \text{E}$ (and $\text{G}\times \text{G}$) interactions can also complicate the interpretation of family-based effect size estimates. The reason is that, even if we were to know the causal alleles for a trait of interest, what we estimate by measuring their associations with phenotypic differences within families is not analogous to the counterfactual effects of experimentally substituting alleles in random individuals. Instead, we are necessarily restricting our focus to the effect of their transmission from heterozygous parents. If heterozygous parents tend to experience different environments or carry different genetic backgrounds than homozygotes do, within-family designs will tell us about direct effects in these particular environments or genetic backgrounds, rather than in the population as a whole. Thus, although the ongoing shift towards family-based studies is motivated by concerns about confounding, with different alleles experiencing different environmental and genetic backgrounds, family-based studies can be influenced by conceptually similar issues of confounding in the presence of $\text{G}\times \text{E}$ and $\text{G}\times \text{G}$ interactions. Such interactions are difficult to reliably identify and measure, but there are a growing number of potential examples from GWASs (Tropf et al. 2017; Barcellos et al. 2018; Young et al. 2018b; Mostafavi et al. 2020; Patel et al. 2022). The interaction issues raised here echo a set of conceptually distinct concerns about the interpretation of average treatment effects in other contexts (Słoczyński 2022), reinforcing the need for care in interpreting such estimates as informative about causes across heterogeneous groups.

In summary, family-based studies are a clear step forward towards quantifying genetic effects, with large-scale family studies carrying the potential to resolve long-standing issues in human genetics. However, these designs come with their own sets of caveats, which will be important to understand and acknowledge as family-based genetic studies become a key tool in the exploration of causal effects across disparate fields of study.

Methods

All simulations were carried out in SLiM 4.0 (Haller and Messer 2019). Code is available at github.com/cveller/confoundedGWAS.

For the purpose of carrying out sibling association studies in our simulations, we assumed a simple, monogamous mating structure: each generation, each female and each male is involved in a single mating pair, and each mating pair produces exactly two offspring (who are therefore full siblings). To maintain the precisely even sex ratio required by this scheme, we assumed that a quarter of mating pairs produce two daughters, a quarter produce two sons, and half produce a son and a daughter. Population sizes were chosen to ensure that these numbers of mating pairs were whole numbers, and mating pairs were permuted randomly each generation before assigning brood sex ratios (to ensure that no artifact was introduced by SLiM’s indexing of individuals).

Each generation, per-locus effect size estimates were calculated for both population-wide and sibling GWASs. The former were calculated as the regression of trait values on per-locus genotypes, while the latter were calculated as the regression of sibling differences in trait values on sibling differences in per-locus genotypes.

In all simulations, the total population size was N = 40,000.

Assortative mating.

For our general cross-trait assortative mating setup, traits 1 and 2 are influenced by variation at sets of bi-allelic loci $L_1$ and $L_2$ respectively. The effect sizes of the reference allele at locus $l$ on traits 1 and 2 are ${\alpha }_l$ and ${\beta }_l$ respectively. An individual’s polygenic score (PGS) is then $P_1={\sum }_{l\in L_1} g_l{\alpha }_l$ for trait 1 and $P_2={\sum }_{l\in L_2} g_l{\beta }_l$ for trait 2. In all the scenarios we simulated, traits had heritability 1, so that individuals’ trait values are the same as their PGSs.

Our aim is to simulate a scenario where assortative mating is based on females’ values for trait 1 and males’ values for trait 2, such that, across mating pairs, the correlation of the mother’s PGS for trait 1, $P_1^m$, and the father’s PGS for trait 2, $P_2^f$, is a constant value $\rho$ (in all of our simulations, $\rho =0.2$). To achieve this, we use an algorithm suggested by Zaitlen et al. (2017): At the outset, we choose an accuracy tolerance $\epsilon$ such that, if by some assignment of mates the correlation of their PGSs falls within $\epsilon$ of the target value $\rho$, we accept that assigment. Each generation in which assortative mating occurs, we rank females in order of their PGSs for trait 1, and males in order of their PGSs for trait 2. We then calculate the PGS correlation across mating pairs, ${\rho }_0$, if females and males were matched according to this ranking. If this (maximal) correlation is smaller than the upper bound of our target window $\left({\rho }_0<\rho +\epsilon \right.$, which very seldom occurred in our simulations), then females and males mate precisely according to their PGS rankings and we move on to the next generation. If, instead, ${\rho }_0$ exceeds $\rho +\epsilon$, then we follow the following iterative procedure until we have found a mating structure under which the correlation of PGSs falls within $\epsilon$ of the target value $\rho$.

First, we choose initial ‘perturbation sizes’ ${\xi }_0$ and ${\xi }_1=2{\xi }_0$. Suppose that, in iteration $k$ of the procedure, the perturbation size is ${\xi }_k$ and the chosen mating structure leads to a correlation among mates of ${\rho }_k$. If $\left|{\rho }_k-\rho \right|<\epsilon$, we accept the mating structure and move on to the next generation. Otherwise, we choose a new perturbation size ${\xi }_{k+1}$: (i) if ${\rho }_{k-1},{\rho }_k>\rho$, then ${\xi }_{k+1}=2{\xi }_k$; (ii) if ${\rho }_{k-1}>\rho >{\rho }_k$ or ${\rho }_{k-1}<\rho <{\rho }_k$, then ${\xi }_{k+1}=\left({\xi }_{k-1}+{\xi }_k\right)/2$; (iii) if ${\rho }_{k-1},{\rho }_k<\rho$, then ${\xi }_{k+1}={\xi }_k/2$. Once we have chosen ${\xi }_{k+1}$, for each individual we perturb their PGS (trait 1 for females; trait 2 for males) by a value chosen from a normal distribution with mean 0 and standard deviation ${\xi }_{k+1}$, independently across individuals. We then rank females and males according to their perturbed PGSs, and calculate the correlation ${\rho }_{k+1}$ of their true PGSs if they mate according to this ranking. (Since, in our experience, there can be substantial variance in the ${\rho }_{k+1}$ values that result from this procedure, we repeat it 5 times and choose the mating structure that produces the value of ${\rho }_{k+1}$ closest to the target value $\rho$.) We then decide if another iteration—i.e., another perturbation size ${\xi }_{k+2}$—is required.

A1. Genetic confounding in population and family-based GWAS designs

A1.1. The model

Under the general additive model we have studied, an individual’s value for trait Y is

$$Y=Y^{\text{*}}+\sum _{l\in L}{\alpha }_l^{\text{d}}{g}_l+\sum _{l\in L}{\alpha }_l^{\text{i},\text{m}}{g}_l^{\text{m}}+\sum _{l\in L}{\alpha }_l^{\text{i},\text{f}}{g}_l^{\text{f}}+ϵ,$$ (A.1)

where $g_l$ is the number of focal alleles at locus $l$ carried by the individual, ${\alpha }_l^{\mathrm{d}}$ is the direct genetic effect on the trait value of the focal allele at $l$ (which we assume to be positive, without loss of generality), $g_l^{\mathrm{m}}$ and $g_l^{\mathrm{f}}$ are the numbers of copies of the focal allele at locus $l$ carried by the individual’s mother and father respectively, and ${\alpha }_l^{\mathrm{i},\mathrm{m}}$ and ${\alpha }_l^{\text{i,f}}$ are the indirect genetic effects of the focal allele at $l$ vis the mother’s and father’s genotype respectively. $ϵ$ is the environmental disturbance, with mean zero, and $Y^{\mathrm{*}}$ is the expected trait value of the offspring of parents who carry only trait-decreasing alleles.

It will be useful to expand Eq. (A.1) in terms of the individual’s and the individual’s parents’ maternally and paternally inherited genotypes:

$$Y=Y^{\text{*}}+\sum _{l\in L}{\alpha }_l^{\text{d}}\left(g_l^{\text{mat}}+g_l^{\text{pat}}\right)+\sum _{l\in L}{\alpha }_l^{\text{i},\text{m}}\left(g_l^{\text{m},\text{mat}}+g_l^{\text{m},\text{pat}}\right)+\sum _{l\in L}{\alpha }_l^{\text{i},\text{f}}\left(g_l^{\text{f},\text{mat}}+g_l^{\text{f,pat}}\right)+ϵ,$$ (A.2)

where $g_l^{\text{mat}}$ is the number of focal alleles at locus $l$ that the individual inherited maternally, $g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}$ is the number of focal alleles at $l$ that the individual’s mother inherited maternally, etc.

A1.2. Population GWAS

If we perform a standard population GWAS at a genotyped locus $\lambda$, the estimated effect of the focal allele at $\lambda$ on the trait Y is

$${\hat{\alpha }}_{\lambda }^{\text{pop}}=\frac{\mathrm{Cov}\left(g_{\lambda },Y\right)}{\mathrm{Var}\left(g_{\lambda }\right)}.$$ (A.3)

Here, $\mathrm{V}\mathrm{a}\mathrm{r}\left(g_{\lambda }\right)$ is the genotypic variance at $\lambda$ among sampled individuals, equal to $2p_{\lambda }\left(1-p_{\lambda }\right)\left(1+F_{\lambda }\right)$, where $p_{\lambda }$ is the frequency of the focal allele at $\lambda$ and $F_{\lambda }$ is the coefficient of inbreeding at $\lambda$. For example, if $\lambda$ is at Hardy-Weinberg equilibrium, then $\mathrm{V}\mathrm{a}\mathrm{r}\left(g_{\lambda }\right)=2p_{\lambda }\left(1-p_{\lambda }\right)$; if, instead, the population is divided into several populations, in each of which Hardy-Weinberg equilibrium obtains at $\lambda$ but between which the frequency of the focal variant differs, then $\mathrm{V}\mathrm{a}\mathrm{r}\left(g_{\lambda }\right)=2p_{\lambda }\left(1-p_{\lambda }\right)\left(1+F_{\left.ST,\lambda \right)}\right)$, where $F_{ST,\lambda }$ is the value of $F_{ST}$ at locus $\lambda$.

The covariance term in Eq. (A.3) expands out to

$$\begin{array}{l}\text{Cov}\left(g_{\lambda },Y\right)=\text{Cov}(g_{\lambda }^{\text{mat}}+g_{\lambda }^{\text{pat}},Y^{\text{*}}+\sum _{l\in L}{\alpha }_l^{\text{d}}\left(g_l^{\text{mat}}+g_l^{\text{pat}}\right)\\ +\sum _{l\in L}{\alpha }_l^{\text{i},\text{m}}\left(g_l^{\text{m},\text{mat}}+g_l^{\text{m},\text{pat}}\right)+\sum _{l\in L}{\alpha }_l^{\text{i},\text{f}}\left(g_l^{\text{f},\text{mat}}+g_l^{\text{f},\text{pat}}\right)+ϵ)\\ =\text{Cov}\left(g_{\lambda }^{\text{mat}}+g_{\lambda }^{\text{pat}},\sum _{l\in L}{\alpha }_l^{\text{d}}\left(g_l^{\text{mat}}+g_l^{\text{pat}}\right)\right)\\ +\text{Cov}\left(g_{\lambda }^{\text{mas}},\sum _{l\in L}{\alpha }_l^{\text{i},\text{m}}\left(g_l^{\text{m},\text{mat}}+g_l^{\text{m},\text{pat}}\right)+\sum _{l\in L}{\alpha }_l^{\text{i},\text{f}}\left(g_l^{\text{f},\text{mat}}+g_l^{\text{f},\text{pat}}\right)\right)\\ +\text{Cov}\left(g_{\lambda }^{\text{pat}},\sum _{l\in L}{\alpha }_l^{\text{i},\text{m}}\left(g_l^{\text{m},\text{mat}}+g_l^{\text{m},\text{pat}}\right)+\sum _{l\in L}{\alpha }_l^{\text{i},\text{f}}\left(g_l^{\text{f},\text{mant}}+g_l^{\text{f},\text{pat}}\right)\right)+\text{Cov}\left(g_{\lambda },ϵ\right)\\ =\sum _{l\in L}(\left[\text{Cov}\left(g_{\lambda }^{\text{mat}},g_l^{\text{mat}}\right)+\text{Cov}\left(g_{\lambda }^{\text{mat}},g_l^{\text{pat}}\right)+\text{Cov}\left(g_{\lambda }^{\text{pat}},g_l^{\text{mat}}\right)+\text{Cov}\left(g_{\lambda }^{\text{pat}},g_l^{\text{pat}}\right)\right]{\alpha }_l^{\text{d}}\\ +\left[\text{Cov}\left(g_{\lambda }^{\text{mat}},g_l^{\text{m},\text{mat}}+g_l^{\text{m},\text{pat}}\right)\right]{\alpha }_l^{\text{i},\text{m}}+\left[\text{Cov}\left(g_{\lambda }^{\text{pat}},g_l^{\text{f},\text{mat}}+g_l^{\text{f},\text{pat}}\right)\right]{\alpha }_l^{\text{i},\text{f}}\\ +\left[\text{Cov}\left(g_{\lambda }^{\text{mat}},g_l^{\text{f}}\right)\right]{\alpha }_l^{\text{i},\text{f}}+\left[\text{Cov}\left(g_{\lambda }^{\text{pat}},g_l^{\text{m}}\right)\right]{\alpha }_l^{\text{i},\text{m}})+\text{Cov}\left(g_{\lambda },ϵ\right)\\ =\sum _{l\in L}(2\left(D_M+{\tilde{D}}_M\right){\alpha }_l^{\text{d}}\\ +\left[\text{Cov}\left(g_{\lambda }^{\text{mat}},g_l^{\text{mm},\text{mat}}+g_l^{\text{mm},\text{pat}}\right)\right]{\alpha }_l^{\text{i},\text{m}}+\left[\text{Cov}\left(g_{\lambda }^{\text{pat}},g_l^{\text{f},\text{mmat}}+g_l^{\text{f},\text{pat}}\right)\right]{\alpha }_l^{\text{i},\text{f}}\\ +\left[\text{Cov}\left(g_{\lambda }^{\text{mat}},g_l^{\text{f}}\right)\right]{\alpha }_l^{\text{i},\text{f}}+\left[\text{Cov}\left(g_{\lambda }^{\text{pat}},g_l^{\text{m}}\right)\right]{\alpha }_l^{\text{i},\text{m}})+\text{Cov}\left(g_{\lambda },ϵ\right),\end{array}$$ (A.4)

where $D_{\lambda l}$ and ${\tilde{D}}_{\lambda l}$ are the degrees of cis- and trans-linkage disequilibrium between the focal alleles at loci $\lambda$ and $l$ in the GWAS sample. Since $g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}}$ equals $g_{\lambda }^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}$ or $g_{\lambda }^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}}$ with equal probability, $\mathrm{Cov}\left(g_{\lambda }^{\text{mat}},g_l^{\text{m},\text{mat}}+g_l^{\text{m},\text{pat}}\right)=D_{\lambda l}^{\prime }+{\tilde{D}}_{\lambda l}^{\prime }$, and similarly, $\mathrm{Cov}\left(g_{\lambda }^{\text{pat}},g_l^{\text{f},\text{mat}}+g_l^{\text{f},\text{pat}}\right)=D_{\lambda l}^{\prime }+{\tilde{D}}_{\lambda l}^{\prime }$ (here, $D_{\lambda l}^{\mathrm{\prime }}$ and ${\tilde{D}}_{\lambda l}^{\text{′}}$ are the LDs in the parents of the sample, assumed to be equal across mothers and fathers). Since maternal transmission is independent of paternal genotype, and vice versa, Cow $\left(g_{\lambda }^{\text{mat}},g_l^{\mathrm{f}}\right)=\mathrm{C}\mathrm{o}\mathrm{v}\left(g_{\lambda }^{\mathrm{m}},g_l^{\mathrm{f}}\right)/2$ and $\mathrm{C}\mathrm{o}\mathrm{v}\left(g_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}},g_l^{\mathrm{m}}\right)=\mathrm{C}\mathrm{o}\mathrm{v}\left(g_{\lambda }^{\mathrm{f}},g_l^{\mathrm{m}}\right)/2$. So

$$\begin{array}{l}\mathrm{Cov}\left(g_{\lambda },Y\right)=\sum _{l\in L}(2\left(D_{\lambda l}+{\tilde{D}}_{\lambda l}\right){\alpha }_l^{\text{d}}+\left(D_{\lambda l}^{\prime }+{\tilde{D}}_{\lambda l}^{\prime }\right)\left({\alpha }_l^{\text{i},\text{m}}+{\alpha }_l^{\text{i},\text{f}}\right)\\ +\frac{1}{2}\left[\mathrm{Cov}\left(g_{\lambda }^{\text{m}},g_l^{\text{f}}\right)\right]{\alpha }_l^{\text{i},\text{f}}+\frac{1}{2}\left[\mathrm{Cov}\left(g_{\lambda }^{\text{f}},g_l^{\text{m}}\right)\right]{\alpha }_l^{\text{i},\text{m}})+\mathrm{Cov}\left(g_{\lambda },ϵ\right).\end{array}$$ (A.5)

If ${\alpha }_l^{\mathrm{i},\mathrm{m}}={\alpha }_l^{\mathrm{i},\mathrm{f}}={\alpha }_l^{\mathrm{i}}$, then

$$\begin{array}{l}\text{Cov}\left(g_{\lambda },Y\right)=\sum _{l\in L}\left(2\left(D_{\lambda l}+{\tilde{D}}_{\lambda l}\right){\alpha }_l^{\text{d}}+\left(2\left(D_{\lambda l}^{\prime }+{\tilde{D}}_{\lambda l}^{\prime }\right)+\frac{1}{2}\left[\text{Cov}\left(g_{\lambda }^{\text{m}},g_l^{\text{f}}\right)+\text{Cov}\left(g_{\lambda }^{\text{f}},g_l^{\text{m}}\right)\right]\right){\alpha }_l^{\text{i}}\right)+\text{Cov}\left(g_{\lambda },ϵ\right)\\ =\sum _{l\in L}\left(2\left(D_{\lambda l}+{\tilde{D}}_{\lambda l}\right){\alpha }_l^{\text{d}}+\left(2\left(D_{\lambda l}^{\prime }+{\tilde{D}}_{\lambda l}^{\prime }\right)+\frac{1}{2}\left[8{\tilde{D}}_{\lambda l}\right]\right){\alpha }_l^{\text{i}}\right)+\text{Cov}\left(g_{\lambda },ϵ\right)\\ =2\sum _{l\in L}\left(\left(D_{\lambda l}+{\tilde{D}}_{\lambda l}\right){\alpha }_l^{\text{d}}+\left(\mid D_{\lambda l}^{\prime }+{\tilde{D}}_{\lambda l}^{\prime }+2{\tilde{D}}_{\lambda l}\right){\alpha }_l^{\text{i}}\right)+\mathrm{Cov}\left(g_{\lambda },ϵ\right).\end{array}$$ (A.6)

In the second line of Eq. (A.6), we have used the fact that covariances across parents translate to covariances across maternal and paternal genomes in the offspring. Note, however, that $\mathrm{C}\mathrm{o}\mathrm{v}\left(g_{\lambda }^{\mathrm{m}},g_l\right)$ and $\mathrm{C}\mathrm{o}\mathrm{v}\left(g_{\lambda }^{\mathrm{f}},g_l^{\mathrm{m}}\right)$ need not, in general be equal—e.g., they will not be so under sex-besed cross-trait assortative mating—which is why we could not apply a similar simplification to Eq. (A.5).

Dividing Eq. (A.6) by $\mathrm{V}\mathrm{a}\mathrm{r}\left(g_{\lambda }\right)$, and recognizing that, for $l\in L_{\text{local}},$ $c_{\lambda }\approx 0$, we recover Eq. (3) in the Main Text.

A1.3. Sibling GWAS

Consider two full siblings. Let $g_l^{\mathrm{m}\mathrm{a}\mathrm{t},1}$ and $g_l^{\mathrm{m}\mathrm{a}\mathrm{t},2}$ indicate whether sib 1 and sib 2 respectively inherited the focal (trait-increasing) allele from their mother at locus $l$. Let $g_l^{\mathrm{p}\mathrm{a}\mathrm{t},1}$ and $g_l^{\text{pat,}2}$ be analogous indicators for paternal transmission. Write $\mathrm{\Delta }{g}_l^{\mathrm{m}\mathrm{a}\mathrm{t}}=g_l^{\mathrm{m}\mathrm{a}\mathrm{t},1}-g_l^{\text{mat,}2}$ and $\mathrm{\Delta }{g}_l^{\text{pat}}=g_l^{\text{pat,}1}-g_l^{\text{pat,}2}$. Since maternal and paternal transmission are independent, $\mathrm{\Delta }{g}_l^{\mathrm{m}\mathrm{a}\mathrm{t}}$ and $\mathrm{\Delta }{g}_l^{\text{pat}}$ are independent for all pairs of loci $l$ and $l^{\mathrm{\prime }}$ (including $l=l^{\mathrm{\prime }}$). The difference in the two siblings’ genotypic values at locus $l$ is $\mathrm{\Delta }gl=\mathrm{\Delta }{g}_l^{\text{mat}}+\mathrm{\Delta }{g}_l^{\text{pat}}$. From Eq. (A.1), the difference in their trait values is

$$\text{Δ}Y=\sum _{l\in L}\text{Δ}{g}_l{\alpha }_l^d+\text{Δ}ϵ,$$ (A.7)

where $\mathrm{\Delta }ϵ$ is the difference in the environmental disturbances experienced by the two siblings. Notice that the indirect effects cancel out of Eq. (A.7), since the parental genotypes are the same for the two siblings. So, in a sib-GWAS for trait Y, the estimated effect size at $\lambda$ is

$$\begin{array}{l}{\hat{\alpha }}_{\lambda }^{\mathrm{s}\mathrm{i}\mathrm{b}}=\frac{\mathrm{Cov}\left(\mathrm{\Delta }{g}_{\lambda },\mathrm{\Delta }Y\right)}{\mathrm{Var}\left(\mathrm{\Delta }{g}_{\lambda }\right)}=\frac{\mathrm{Cov}\left(\mathrm{\Delta }{g}_{\lambda },\sum _{l\in L} \mathrm{\Delta }{g}_l{\alpha }_l^{\mathrm{d}}+\mathrm{\Delta }ϵ\right)}{\mathrm{Var}\left(\mathrm{\Delta }{g}_{\lambda }\right)}\\ =\frac{\mathrm{Cov}\left(\mathrm{\Delta }{g}_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}}+\mathrm{\Delta }{g}_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}},\sum _{l\in L} \left(\mathrm{\Delta }{g}_l^{\mathrm{m}\mathrm{a}\mathrm{t}}+\mathrm{\Delta }{g}_i^{\mathrm{p}\mathrm{a}\mathrm{t}}\right){\alpha }_l^{\mathrm{d}}\right)+\mathrm{Cov}\left(\mathrm{\Delta }{g}_{\lambda },\mathrm{\Delta }ϵ\right)}{\mathrm{Var}\left(\mathrm{\Delta }{g}_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}}+\mathrm{\Delta }{g}_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}}\right)}\\ =\frac{\sum _{l\in L} \left[\mathrm{Cov}\left(\mathrm{\Delta }{g}_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{n}},\mathrm{\Delta }{g}_l^{\mathrm{m}\mathrm{a}\mathrm{t}}\right)+\mathrm{Cov}\left(\mathrm{\Delta }{g}_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}},\mathrm{\Delta }{g}_l^{\mathrm{p}\mathrm{a}\mathrm{t}}\right)\right]{\alpha }_l^{\mathrm{d}}+\mathrm{Cov}\left(\mathrm{\Delta }{g}_{\lambda },\mathrm{\Delta }ϵ\right)}{\mathrm{Var}\left(\mathrm{\Delta }{g}_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}}\right)+\mathrm{Var}\left(\mathrm{\Delta }{g}_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}}\right)}\\ =\frac{\sum _{l\in L} \left(\mathbb{E}\left[\mathrm{\Delta }{g}_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}}\mathrm{\Delta }{g}_l^{\mathrm{m}\mathrm{a}\mathrm{t}}\right]+\mathbb{E}\left[\mathrm{\Delta }{g}_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}}\mathrm{\Delta }{g}_l^{\mathrm{p}\mathrm{a}\mathrm{t}}\right]\right){\alpha }_l^{\mathrm{d}}+\mathrm{Cov}\left(\mathrm{\Delta }{g}_{\lambda },\mathrm{\Delta }ϵ\right)}{\mathbb{E}\left[{\left(\mathrm{\Delta }{g}_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}}\right)}^2\right]+\mathbb{E}\left[{\left(\mathrm{\Delta }{g}_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}}\right)}^2\right]},\end{array}$$

since $\mathrm{C}\mathrm{o}\mathrm{v}\left(\mathrm{\Delta }{g}_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}},\mathrm{\Delta }{g}_l^{\mathrm{p}\mathrm{a}\mathrm{t}}\right)=\mathrm{C}\mathrm{o}\mathrm{v}\left(\mathrm{\Delta }{g}_l^{\mathrm{m}\mathrm{a}\mathrm{t}},\mathrm{\Delta }{g}_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}}\right)=0$ (line 3) and $\mathbb{E}\left[\mathrm{\Delta }{g}_k^{\mathrm{m}\mathrm{a}\mathrm{t}}\right]=\mathbb{E}\left[\mathrm{\Delta }{g}_k^{\mathrm{p}\mathrm{a}\mathrm{t}}\right]=0$ for all loci $k$ (line 4). The denominator $\mathbb{E}\left[{\left(\mathrm{\Delta }{g}_{\lambda }^{\text{mat}}\right)}^2\right]+\mathbb{E}\left[{\left(\mathrm{\Delta }{g}_{\lambda }^{\text{pat}}\right)}^2\right]=H_{\lambda }$, the fraction of parents in the family GWAS sample who are heterozygous at locus $\lambda$. The only non-zero contributions to $\mathbb{E}\left[\mathrm{\Delta }{g}_{\lambda }^{\text{mat}}\mathrm{\Delta }{g}_l^{\text{mat}}\right]$ and $\mathbb{E}\left[\mathrm{\Delta }{g}_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}}\mathrm{\Delta }{g}_l^{\mathrm{p}\mathrm{a}\mathrm{t}}\right]$ come from parents who are heterozygous at both $\lambda$ and $l$. Such parents are either ‘coupling’ double-heterozygotes carrying the focal alleles at $\lambda$ and $l$ in coupling phase (i.e., inherited from the same parent), or ‘repulsion’ double-heterozygotes carrying the focal alleles at $\lambda$ and $l$ in repulsion phase (inherited from different parents). Among parents, let the fractions of coupling and repulsion double-hets for loci $\lambda$ and $l$ be $H_{\lambda l}^{\text{coup}}$ and $H_{\lambda l}^{\mathrm{r}}$ respectively. If the recombination rate between the loci is $c_{\lambda l}^{♀}$ in females and $c_{\lambda l}^{♂}$ in males, then

$$\begin{array}{l}\mathbb{E}\left[\mathrm{\Delta }{g}_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}}\mathrm{\Delta }{g}_l^{\mathrm{m}\mathrm{a}\mathrm{t}}\right]=\mathbb{E}\left[\mathrm{\Delta }{g}_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}}\mathrm{\Delta }{g}_l^{\mathrm{m}\mathrm{a}\mathrm{t}}\mid \text{mother}\text{is}\text{coupling}\text{double-het}\right]H_{\lambda l}^{\text{coup}}\\ +\mathbb{E}\left[\mathrm{\Delta }{g}_{\lambda }^{\text{mat}}\mathrm{\Delta }{g}_l^{\text{mat}}\mid \text{mother}\text{is}\text{repulsion}\text{double-het}\right]H_{\lambda l}^{\text{rep}}\\ =\left(\frac{1}{2}-c_{\lambda l}^{\mathrm{♀}}\right)\left(H_{\lambda l}^{\text{coup}}-H_{\lambda l}^{\mathrm{r}}\right)\\ =\left(1-2c_{\lambda l}^{\mathrm{♀}}\right)(D_{\lambda l}^{\mathrm{\prime }}-{\tilde{D}}_{\lambda l}^{\mathrm{\prime }}),\end{array}$$

since $H_{\lambda l}^{\text{coup}}-H_{\lambda l}^{\mathrm{r}}=2(D_{\lambda l}^{\mathrm{\prime }}-{\tilde{D}}_{\lambda l}^{\mathrm{\prime }})$, where $D_{\lambda l}^{\mathrm{\prime }}$ and ${\tilde{D}}_{\lambda l}^{\mathrm{\prime }}$ are the cis- and trans-LD between the focal/traitincreasing alleles at $\lambda$ and $l$ among parents.. Similarly,

$$\mathbb{E}\left[\mathrm{\Delta }{g}_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}}\mathrm{\Delta }{g}_l^{\mathrm{p}\mathrm{a}\mathrm{t}}\right]=\left(1-2c_{\lambda l}^{♂}\right)(D_{\lambda l}^{\mathrm{\prime }}-{\tilde{D}}_{\lambda l}^{\mathrm{\prime }})$$

So

$${\hat{\alpha }}_{\lambda }^{\mathrm{d},\text{sib}}=\frac{2\sum _{l\in L} \left(1-2c_{\lambda l}\right)(D_{\lambda l}^{\prime }-{\tilde{D}}_{\lambda l}^{\prime }){\alpha }_l^{\mathrm{d}}+\mathrm{C}\mathrm{o}\mathrm{v}\left(\mathrm{\Delta }{g}_l,\mathrm{\Delta }ϵ\right)}{H_{\lambda }},$$ (A.8)

where $c_{\lambda l}$ is the sex-averaged recombination fraction between $\lambda$ and $l$. $\mathrm{S}\mathrm{i}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{C}\mathrm{o}\mathrm{v}\left(\mathrm{\Delta }{g}_l,\mathrm{\Delta }ϵ\right)=0$, and recognizing that, for $l\in L_{\text{local}},$ $c_{\lambda l}\approx 0$ and $\left|{\tilde{D}}_{\lambda l}\right|\ll \left|D_{\lambda l}\right|$ in expectation, we recower Eq. (7) in the Main Text.

A1.4. Indirect effects: transmitted vs. untransmitted alleles

In Eq. (A.2), $g_l^{\text{mat}}$ represents the allele that was transmitted maternally from among the set of maternal alleles $\left\{g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},g_l^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}}\right\}$. Thus, if the maternally transmitted allele was the grandmaternal allele (with probability 1/2, and in which case $g_l^{\mathrm{m}\mathrm{a}\mathrm{t}}=g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}$), then the untransmitted allele at locus $l$ is the grandpaternal allele, with genotypic value $g_l^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}}$. To make this distinction clear, we write $g_l^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{T}}$ for the genotypic value of the maternally transmitted allele at locus $l$, and $g_l^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{U}}$ for the maternally untransmitted allele at locus $l$. Similarly, $g_l^{\text{patT}}$ and $g_l^{\text{patU}}$ represent the paternally transmitted and untransmitted alleles at $l$. The transmitted and untransmitted genotypes are $g_l^{\mathrm{T}}=g_l^{\mathrm{m}\mathrm{a}\mathrm{n}\mathrm{T}}+g_l^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{T}}$ and $g_l^{\mathrm{U}}=g_l^{\text{matU}}+g_l^{\text{patU}}$ respectively.

Estimating direct effects

The regressions of the trait value on the transmitted and untransmitted genotypes are

$${\hat{\alpha }}_{\lambda }^{\mathrm{T}}=\frac{\mathrm{Cov}\left(g_{\lambda }^{\mathrm{T}},Y\right)}{\mathrm{Var}\left(g_{\lambda }^{\mathrm{T}}\right)}=\frac{\mathrm{Cov}\left(g_{\lambda }^{\mathrm{T}},Y\right)}{\mathrm{Var}\left(g_{\lambda }\right)}\text{and}{\hat{\alpha }}_{\lambda }^{\mathrm{U}}=\frac{\mathrm{Cov}\left(g_{\lambda }^{\mathrm{U}},Y\right)}{\mathrm{Var}\left(g_{\lambda }^{\mathrm{U}}\right)}=\frac{\mathrm{Cov}\left(g_{\lambda }^{\mathrm{U}},Y\right)}{\mathrm{Var}\left(g_{\lambda }\right)},$$

where we have used the fact that, since transmission at $\lambda$ is random, $\mathrm{Var}\left(g_{\lambda }^{\mathrm{T}}\right)=\mathrm{Var}\left(g_{\lambda }^{\mathrm{U}}\right)=\mathrm{Var}\left(g_{\lambda }\right)$.

The estimate of the direct effect of the focal variant at $\lambda$ is then

$${\hat{\alpha }}_{\lambda }^{\mathrm{d}}={\hat{\alpha }}_{\lambda }^{\mathrm{T}}-{\hat{\alpha }}_{\lambda }^{\mathrm{U}}=\frac{\mathrm{Cov}\left(g_{\lambda }^{\mathrm{T}},Y\right)-\mathrm{Cov}\left(g_{\lambda }^{\mathrm{U}},Y\right)}{\mathrm{Var}\left(g_{\lambda }\right)}.$$

We have

$$\begin{array}{l}\text{Cov}\left(g_{\lambda }^{\text{mast}},Y\right)=\text{Cov}(g_{\lambda }^{\text{mat}},Y^{\text{*}}+\sum _{l\in L}\left(g_l^{\text{mat}}{}^{\text{T}}+g_l^{\text{patT}}\right){\alpha }_l^{\text{d}}\\ +\sum _{l\in L}\left(g_l^{\text{m},\text{mat}}+g_l^{\text{m},\text{pat}}\right){\alpha }_l^{\text{i},\text{mm}}+\sum _{l\in L}\left(g_l^{\text{f},\text{mat}}+g_l^{\text{f},\text{pat}}\right){\alpha }_l^{\text{i},\text{f}}+ϵ)\\ =\sum _{l\in L}\left[\text{Cov}\left(g_{\lambda }^{\text{matT}},g_l^{\text{matT}}\right)+\text{Cov}\left(g_{\lambda }^{\text{matT}},g_l^{\text{pat}\text{T}}\right)\right]{\alpha }_l^{\text{d}}\\ +\sum _{l\in L}\left[\text{Cov}\left(g_{\lambda }^{\text{matT}},g_l^{\text{m},\text{mat}}\right)+\text{Cov}\left(g_{\lambda }^{\text{matT}},g_l^{\text{m},\text{pat}}\right)\right]{\alpha }_l^{\text{i},\text{m}}\\ +\sum _{l\in L}\left[\text{Cov}\left(g_{\lambda }^{\text{matT}},g_l^{\text{f},\text{mat}}+g_l^{\text{f},\text{pat}}\right)\right]{\alpha }_l^{\text{i,f}}+\text{Cov}\left(g_{\lambda }^{\text{matT}},ϵ\right)\\ =\sum _{l\in L}\left[D_{\lambda l}^{\prime }\left(1-c_{\lambda l}^{\text{♀}}\right)+{\tilde{D}}_{\lambda l}^{\prime }{c}_{\lambda l}^{\text{♀}}+\text{Cov}\left(g_{\lambda }^{\text{matT}},g_l^{\text{patT}}\right)\right]{\alpha }_l^{\text{d}}\\ +\sum _{l\in L}\left(D_{\lambda l}^{\prime }+{\tilde{D}}_{\lambda l}^{\prime }\right){\alpha }_l^{\text{i},\text{m}}+\sum _{l\in L}\left[\text{Cov}\left(g_{\lambda }^{\text{matT}},g_l^{\text{f},\text{mat}}+g_l^{\text{f},\text{pat}}\right)\right]{\alpha }_l^{\text{i},\text{f}}+\text{Cov}\left(g_{\lambda }^{\text{manT}},ϵ\right),\end{array}$$ (A.9)

and

$$\begin{array}{l}\text{Cov}\left(g_{\lambda }^{\text{matU}},Y\right)=\text{Cov}(g_{\lambda }^{\text{matU}},Y^{\text{*}}+\sum _{l\in L}\left(g_l^{\text{matT}}+g_l^{\text{pat}\text{T}}\right){\alpha }_l^{\text{d}}\\ +\sum _{l\in L}\left(g_l^{\text{m},\text{mat}}+g_l^{\text{m},\text{pat}}\right){\alpha }_l^{\text{i},\text{m}}+\sum _{l\in L}\left(g_l^{\text{f},\text{mat}}+g_l^{\text{f},\text{pat}}\right){\alpha }_l^{\text{i},\text{f}}+ϵ)\\ =\sum _{l\in L}\left[\text{Cov}\left(g_{\lambda }^{\text{matU}},g_l^{\text{matT}}\right)+\text{Cov}\left(g_{\lambda }^{\text{matU}},g_l^{\text{patT}}\right)\right]{\alpha }_l^{\text{d}}\\ +\sum _{l\in L}\left[\text{Cov}\left(g_{\lambda }^{\text{matU}},g_l^{\text{m},\text{mat}}\right)+\text{Cov}\left(g_{\lambda }^{\text{matU}},g_l^{\text{m},\text{pat}}\right)\right]{\alpha }_l^{\text{i,m}}\\ +\sum _{l\in L}\left[\text{Cov}\left(g_{\lambda }^{\text{matU}},g_l^{\text{f},\text{mat}}+g_l^{\text{f},\text{pat}}\right)\right]{\alpha }_l^{\text{i},\text{f}}+\text{Cov}\left(g_{\lambda }^{\text{matU}},ϵ\right)\\ =\sum _{l\in L}\left[D_{\lambda l}^{\prime }{c}_{\lambda l}^{\text{♀}}+{\tilde{D}}_{\lambda l}^{\prime }\left(1-\mid c_{\lambda l}^{\text{♀}}\right)+\text{Cov}\left(g_{\lambda }^{\text{matU}},g_l^{\text{pat}\text{T}}\right)\right]{\alpha }_l^{\text{d}}\\ +\sum _{l\in L}\left(D_{\lambda l}^{\prime }+{\tilde{D}}_{\lambda l}^{\prime }\right){\alpha }_l^{\text{i},\text{m}}+\sum _{l\in L}\left[\text{Cov}\left(g_{\lambda }^{\text{matU}},g_l^{\text{f},\text{mat}}+g_l^{\text{f},\text{pat}}\right)\right]{\alpha }_l^{\text{i},\text{f}}+\text{Cov}\left(g_{\lambda }^{\text{matU}},ϵ\right).\end{array}$$ (A.10)

Since $\mathrm{Cov}\left(g_{\lambda }^{\text{matT}},g_l^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{T}}\right)=\mathrm{Cov}\left(g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{U}},g_l^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{T}}\right)$ and $\mathrm{Cov}\left(g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{T}},g_i^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}+g_l^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}\right)=\mathrm{Cov}\left(g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{U}},g_l^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}+g_l^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}\right)$,

$$\begin{array}{l}\mathrm{Cov}\left(g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{T}},Y\right)-\mathrm{Cov}\left(g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{U}},Y\right)=\sum _{l\in L} \left[D_{\lambda }^{\mathrm{\prime }}(1-c_{\lambda l}^{\mathrm{♀}})+{\tilde{D}}_{\lambda l}^{\mathrm{\prime }}{c}_{\lambda l}^{\mathrm{♀}}\right]{\alpha }_l^{\mathrm{d}}-\sum _{l\in L} \left[D_{\lambda l}^{\mathrm{\prime }}{c}_{\lambda l}^{\mathrm{♀}}+{\tilde{D}}_{\lambda l}^{\mathrm{\prime }}(1-c_{\lambda l}^{\mathrm{♀}})\right]{\alpha }_l^{\mathrm{d}}\\ +\mathrm{Cov}\left(g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{T}}-g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{U}},ϵ\right)\\ =\sum _{l\in L} (1-2c_{\lambda l}^{\mathrm{♀}})\left(D_{\lambda l}^{\mathrm{\prime }}-{\tilde{D}}_{\lambda l}^{\mathrm{\prime }}\right){\alpha }_l^{\mathrm{d}}+\mathrm{Cov}\left(g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{T}}-g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}},ϵ\right).\end{array}$$

Similarly,

$$\mathrm{Cov}\left(g_{\lambda }^{\text{pat}\mathrm{T}},Y\right)-\mathrm{Cov}\left(g_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{U}},Y\right)=\sum _{l\in L} \left(1-2c_{\lambda }^{♂}\right)\left(D_{\lambda l}^{\mathrm{\prime }}-{\tilde{D}}_{\lambda l}^r\right){\alpha }_l^{\mathrm{d}}+\mathrm{Cov}\left(g_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{T}}-g_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{U}},ϵ\right).$$

Since $g_{\lambda }^{\mathrm{T}}=g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{T}}+g_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{T}}$ and $g_{\lambda }^{\mathrm{U}}=g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{n}\mathrm{U}}+g_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{U}}$,

$$\mathrm{cov}\left(g_{\lambda }^{\text{T}},Y\right)-\mathrm{cov}\left(g_{\lambda }^{\text{U}},Y\right)=2\sum _{l\in L}\left(1-2c_{\lambda l}\right)\left(D_{\lambda l}^{\text{′}}-{\tilde{D}}_{\lambda l}^{\text{′}}\right){\alpha }_l^{\text{d}}+\mathrm{cov}\left(g_{\lambda }^{\text{T}}-g_{\lambda }^{\text{U}},ϵ\right),$$

where $c_{\lambda l}$ is the sex-averaged recombination fraction between $\lambda$ and $l$. Therefore, the transmitted-untransmitted regression coefficient at locus $\lambda$ is

$${\hat{\alpha }}_{\lambda }^{\mathrm{d},\mathrm{T}-\mathrm{U}}=\frac{\mathrm{Cov}\left(g_{\lambda }^{\mathrm{T}},Y\right)-\mathrm{Cov}\left(g_{\lambda }^{\mathrm{U}},Y\right)}{\mathrm{Var}\left(g_{\lambda }\right)}=\frac{2\sum _{l\in L} (1-2c\lambda l)\left(D_{\lambda l}^{\mathrm{\prime }}-{\tilde{D}}_{\lambda l}^{\mathrm{\prime }}\right){\alpha }_l^{\mathrm{d}}+\mathrm{Cov}\left(g_{\lambda }^{\mathrm{T}}-g_{\lambda }^{\mathrm{U}},ϵ\right)}{V_{\lambda }}.$$

Estimating indirect effects

The coefficient in the regression of the trait value $Y$ on the untransmitted genotype $g_{\lambda }^{\mathrm{U}}$ at locus $,$ ${\hat{\alpha }}_{\lambda }^{\mathrm{U}}$, has sometimes been considered to provide an estimate of the indirect ‘family’ effect of the focal variant at $\lambda :{\hat{\alpha }}_{\lambda }^{\mathrm{i}}={\hat{\alpha }}_{\lambda }^U$. From Eq. (A.10) and its analog for the paternally untransmitted allele,

$$\begin{array}{l}\mathrm{Cov}\left(g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{U}}+g_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{U}},Y\right)=\sum _{l\in L} \left(\left[2D_{\lambda l^{\mathrm{\prime }}{c}_{\lambda }}^{\mathrm{\prime }}+2{\tilde{D}}_{\lambda l}^{\mathrm{\prime }}\left(1-c_{\lambda l}\right)+\mathrm{Cov}\left(g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{U}},g_l^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{T}}\right)+\mathrm{Cov}\left(g_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{U}},g_l^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{T}}\right)\right]{\alpha }_l^{\mathrm{d}}\right.\\ +\left(D_{\lambda l}^{\mathrm{\prime }}+{\tilde{D}}_{\lambda l}^{\mathrm{\prime }}\right){\alpha }_l^{\mathrm{i},\mathrm{m}}+\sum _{l\in L} \left[\mathrm{Cov}\left(g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{U}},g_l^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}+g_l^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}\right)\right]{\alpha }_l^{\mathrm{i},\mathrm{f}}\\ \left.+\left(D_{\lambda l}^{\mathrm{\prime }}+{\tilde{D}}_{\lambda l}^{\mathrm{\prime }}\right){\alpha }_l^{\mathrm{i},\mathrm{f}}+\sum _{l\in L} \left[\mathrm{Cov}\left(g_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{U}},g_l^{\mathrm{m}\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}+g_l^{\mathrm{m}\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}}\right)\right]{\alpha }_l^{\mathrm{i},\mathrm{m}}\right)\\ +\mathrm{Cov}\left(g_{\lambda }^{\text{matU}},ϵ\right)+\mathrm{Cov}\left(g_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{U}},ϵ\right),\end{array}$$

where $c_{\lambda l}$ is the sex-averaged recombination fraction. In this expression,

$$\mathrm{Cov}\left(g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{U}},g_l^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{T}}\right)+\mathrm{Cov}\left(g_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{U}},g_l^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{T}}\right)=\mathrm{Cov}\left(g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{T}},g_l^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{T}}\right)+\mathrm{Cov}\left(g_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{T}},g_l^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{T}}\right)=2{\tilde{D}}_{\lambda l},$$

while

$$\mathrm{Cov}\left(g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{U}},g_l^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}+g_l^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}\right)=\mathrm{Cov}\left(g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{U}},g_l^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{U}}+g_l^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{T}}\right)=\mathrm{Cov}\left(g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{U}},g_l^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{U}}\right)+\mathrm{Cov}\left(g_{\lambda }^{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{U}},g_l^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{T}}\right)=2{\tilde{D}}_{\lambda l},$$

and, similarly, $\mathrm{Cov}\left(g_{\lambda }^{\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{U}},g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}+g_l^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}}\right)=2{\tilde{D}}_{\lambda l}$. So

$$\begin{array}{l}\mathrm{Cov}\left(g_{\lambda }^{\text{U}},Y\right)=\mathrm{Cov}\left(g_{\lambda }^{\text{matU}}+g_{\lambda }^{\text{patU}},Y\right)=\sum _{l\in L}[2\left(D_{\lambda l}^{\prime }{c}_{\lambda l}+{\tilde{D}}_{\lambda l}^{\prime }\left(1-c_{\lambda l}\right)+{\tilde{D}}_{\lambda l}\right){\alpha }_l^{\text{d}}\\ +\left(D_{\lambda l}^{\prime }+{\tilde{D}}_{\lambda l}^{\prime }+2{\tilde{D}}_{\lambda l}\right)\left({\alpha }_l^{\text{i},\text{m}}+{\alpha }_l^{\text{i},\text{f}}\right)]+\mathrm{Cov}\left(g_{\lambda }^{\text{U}},ϵ\right).\end{array}$$ (A.12)

If we assume that indirect effects via the maternal and paternal families are equal $({\alpha }_l^{\mathrm{i},\mathrm{m}}={\alpha }_l^{\mathrm{i},\mathrm{f}}={\alpha }_l^{\mathrm{i}})$, then Eq. (A.12) simplifies further to

$$\text{Cov}\left(g_{\lambda }^{\text{U}},Y\right)=2\sum _{l\in L}\left[\left(D_{\lambda l^{\prime }}^{\text{′}}{c}_{\lambda l}+{\tilde{D}}_{\lambda l}^{\text{′}}\left(1-c_{\lambda l}\right)+{\tilde{D}}_{\lambda l}\right){\alpha }_l^{\text{d}}+\left(D_M^{\text{′}}+{\tilde{D}}_{Ml}^{\text{′}}+2{\tilde{D}}_{\lambda l}\right){\alpha }_l^{\text{i}}\right]+\text{Cov}\left(g_{\lambda }^{\text{U}},ϵ\right).$$ (A.13)

In this case, the estimate of the indirect effect of the focal allele at $\lambda$ is

$${\hat{\alpha }}_{\lambda }^{\mathrm{i}}=\frac{\mathrm{Cov}\left(g_{\lambda }^{\mathrm{U}},Y\right)}{\mathrm{Var}\left(g_{\lambda }\right)}=\frac{2\sum _{l\in L} \left[\left(D_{\lambda l}^{\mathrm{\prime }}{c}_{\lambda l}+{\tilde{D}}_{\lambda l}^{\mathrm{\prime }}\left(1-c_{\lambda l}\right)+{\tilde{D}}_{\lambda }\right){\alpha }_l^{\mathrm{d}}+\left(D_{\lambda l}^{\mathrm{\prime }}+{\tilde{D}}_{\lambda l}^{\mathrm{\prime }}+2{\tilde{D}}_{\lambda l}\right){\alpha }_l^{\mathrm{i}}\right]+\mathrm{Cov}\left(g_{\lambda }^{\mathrm{U}},ϵ\right)}{\mathrm{Var}\left(g_{\lambda }\right)}.$$ (A.14)

A2. Polygenic scores and their phenotypic correlations

Suppose that we have estimated effect sizes ${\hat{\alpha }}_{\lambda }$ at a set of genotyped loci $\lambda \in \mathrm{\Lambda }$ using a population GWAS for trait 1. For each individual, we can then compute a polygenic score:

$$PG{S}_1=\sum _{\lambda \in \text{Λ}}{g}_{\lambda }{\hat{\alpha }}_{\lambda }^{\text{pop}}.$$ (A.15)

PGSs are often treated as predictions of individuals’ genetic values for traits. In this regard, we might therefore be interested in the covariance across the population between the PGS for a trait and individuals’ values for that trait: $\mathrm{Cov}\left(PG{S}_1,Y_1\right)$. Additionally, if PGSs are treated as predictions of genetic values of traits, then we might be interested in how the PGS calculated for one trait covaries with the value of another trait: $\mathrm{Cov}\left(PG{S}_1,Y_2\right)$. Such covariances might be informative of genetic correlations between traits, or pleiotropy of the alleles underlying genetic variation in the traits. We focus on the two-trait covariance, since it nests the single-trait covariance as a special case. If the total set of loci causally underlying variation in traits 1 and 2 is $L$, then the population covariance between the PGS for trait 1 and the value of trait 2 is

$$\begin{array}{l}\text{Cov}\left(PG{S}_1,Y_2\right)=\text{Cov}\left(\sum _{\lambda \in \text{Λ}}{g}_{\lambda }{\hat{\alpha }}_{\lambda }^{\text{pop}},\sum _{l\in L}{g}_l{\beta }_l\right)\\ =\text{Cov}\left(\sum _{\lambda \in \text{Λ}}\left(g_{\lambda }^{\text{m}}+g_{\lambda }^{\text{p}}\right){\hat{\alpha }}_{\lambda }^{\text{pop}},\sum _{l\in L}\left(g_l^{\text{m}}+g_l^{\text{P}}\right){\beta }_l\right)\\ =2\sum _{\lambda \in \text{Λ}}\sum _{l\in L}\left(D_{\lambda l}+{\tilde{D}}_{\lambda l}\right){\hat{\alpha }}_{\lambda }^{\text{pop}}{\beta }_l.\end{array}$$ (A.16)

The effect size estimates from the population GWAS for trait 1 are

$${\hat{\alpha }}_{\lambda }^{\text{pop}}=\frac{2}{V_{\lambda }}\sum _{l^{\mathrm{\prime }}\in L} (D_{{\lambda }^{\mathrm{\prime }}}+{\tilde{D}}_{\lambda l^{\mathrm{\prime }}}){\alpha }_{l^{\mathrm{\prime }}}\approx {\alpha }_{\lambda }+\frac{2}{V_{\lambda }}\sum _{\begin{array}{c}{l}^{\mathrm{\prime }}\in L\\ l^{\mathrm{\prime }}\ne \lambda \end{array}} (D_{{\lambda }^{\mathrm{\prime }}}+{\tilde{D}}_{\lambda l^{\mathrm{\prime }}}){\alpha }_{l^{\mathrm{\prime }}},$$

and so Eq. (A.16) is, in general,

$$\mathrm{Cov}\left(PG{S}_1,Y_2\right)=\sum _{\lambda \in \Lambda }2p_{\lambda }\left(1-p_{\lambda }\right){\alpha }_{\lambda }{\beta }_{\lambda }+2\sum _{\lambda \in \Lambda }\sum _{\begin{array}{c}l\in L\\ l\ne \lambda \end{array}}\left(D_{\lambda l}+{\tilde{D}}_{\lambda l}\right){\alpha }_{\lambda }{\beta }_l$$ (A.17)

$$+4\sum _{\lambda \in \Lambda }\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne \lambda \end{array}}\sum _{\begin{array}{c}l\in L\\ l\ne \lambda \end{array}}\frac{1}{V_{\lambda }}\left(D_{\lambda l^{\prime }}+{\tilde{D}}_{\lambda l^{\prime }}\right)\left(D_{\lambda l}+{\tilde{D}}_{\lambda l}\right){\alpha }_{l^{\prime }}{\beta }_l.$$ (A.18)

In a family-based study, we might instead be interested in the covariance between siblings’ differences in the trait-1 population PGS and their differences in trait 2. We can write this covariance in our model as

$$\begin{array}{l}\text{Cov}\left(\text{Δ}PG{S}_1,\text{Δ}{Y}_2\right)=\text{Cov}\left(\sum _{\lambda \in \text{Λ}}\left(\text{Δ}{g}_{\lambda }^{\text{m}}+\text{Δ}{g}_{\lambda }^{\text{p}}\right){\hat{\alpha }}_{\lambda }^{\text{pop}},\sum _{l\in L}\left(\text{Δ}{g}_l^{\text{m}}+\text{Δ}{g}_l^{\text{p}}\right){\beta }_l\right)\\ =\mathbb{E}\left[\left(\sum _{l\in \text{Λ}}\left(\text{Δ}{g}_{\lambda }^{\text{m}}+\text{Δ}{g}_{\lambda }^{\text{p}}\right){\hat{\alpha }}_{\lambda }^{\text{pop}}\right)\left(\sum _{l\in L}\left(\text{Δ}{g}_l^{\text{m}}+\text{Δ}{g}_l^{\text{p}}\right){\beta }_l\right)\right]\\ =\sum _{\lambda \in \text{Λ}}\sum _{l\in L}\mathbb{E}\left[\left(\text{Δ}{g}_{\lambda }^{\text{m}}+\text{Δ}{g}_{\lambda }^{\text{p}}\right)\left(\text{Δ}{g}_l^{\text{m}}+\text{Δ}{g}_l^{\text{p}}\right){\hat{\alpha }}_{\lambda }^{\text{pop}}{\beta }_l\right]\\ =\sum _{\lambda \in \text{Λ}}\sum _{l\in L}\left(\mathbb{E}\left[\text{Δ}{g}_{\lambda }^{\text{m}}\text{Δ}{g}_l^{\text{m}}{\hat{\alpha }}_{\lambda }^{\text{pop}}{\beta }_l\right]+\mathbb{E}\left[\text{Δ}{g}_{\lambda }^{\text{p}}{\text{Δ}}_l^{\text{p}}{\hat{\alpha }}_{\lambda }^{\text{pog}}{\beta }_l\right]\right),\end{array}$$ (A.19)

since maternal and paternal transmission are conditionally independent. Focusing on maternal transmission, and writing $h_{\lambda l}^{\mathrm{c},\mathrm{m}}$ and $h_{\lambda l}^{\mathrm{r},\mathrm{m}}$ for the events that the mother is respectively a coupling and a repulsion heterozygote at loci $\lambda$ and $l$, with $H_{\lambda l}^{\text{coup}}$ and $H_{\lambda l}^{\text{rep}}$ their associated probabilities (which are assumed to be the same for mothers and fathers),

$$\begin{array}{l}\mathbb{E}\left[\mathrm{\Delta }{g}_{\lambda }^{\text{m}}\mathrm{\Delta }{g}_l^{\text{m}}{\widehat{\alpha }}_{\lambda }^{\text{pop}}{\beta }_l\right]=\mathbb{E}\left[\mathrm{\Delta }{g}_{\lambda }^{\text{m}}\mathrm{\Delta }{g}_l^{\text{m}}{\widehat{\alpha }}_{\lambda }^{\text{pop}}{\beta }_l\mid h_{\lambda l}^{\text{c},\text{m}}\right]H_{\lambda l}^{\text{coup}}+\mathbb{E}\left[\mathrm{\Delta }{g}_{\lambda }^{\text{m}}\mathrm{\Delta }{g}_l^{\text{m}}{\widehat{\alpha }}_{\lambda }^{\text{pop}}{\beta }_l\mid h_{\lambda l}^{\text{r},\text{m}}\right]H_{\lambda l}^{\text{rep}}\\ =\left(\mathbb{E}\left[\mathrm{\Delta }{g}_{\lambda }^{\text{m}}\mathrm{\Delta }{g}_l^{\text{m}}\mid h_{\lambda l}^{\text{c},\text{m}}\right]H_{\lambda l}^{\text{coup}}+\mathbb{E}\left[\mathrm{\Delta }{g}_{\lambda }^{\text{m}}\mathrm{\Delta }{g}_l^{\text{m}}\mid h_{\lambda l}^{\text{r},\text{m}}\right]H_{\lambda l}^{\text{rep}}\right){\widehat{\alpha }}_{\lambda }^{\text{pop}}{\beta }_l\\ =\left(\frac{1}{2}-c_{\lambda l}^{\text{♀}}\right)\left(H_{\lambda l}^{\text{coup}}-H_{\lambda l}^{\text{rep}}\right){\widehat{\alpha }}_{\lambda }^{\text{pop}}{\beta }_l\\ =\left(1-2c_{\lambda l}^{\text{♀}}\right)\left(D_{\lambda l}^{\prime }-{\tilde{D}}_{\lambda l}^{\prime }\right){\widehat{\alpha }}_{\lambda }^{\text{pop}}{\beta }_l,\end{array}$$

with $D_{\lambda l}^{\mathrm{\prime }}$ and ${\tilde{D}}_{\lambda l}^{\mathrm{\prime }}$ measured in the parents. Similarly,

$$\mathbb{E}\left[\mathrm{\Delta }{g}_{\lambda }^{\mathrm{P}}\mathrm{\Delta }{g}_l^{\mathrm{P}}{\hat{\alpha }}_{\lambda }^{\mathrm{p}\mathrm{o}\mathrm{g}}{\beta }_l\right]=\left(1-2c_{\lambda l}^{♂}\right)\left(D_{\lambda l}^{\mathrm{\prime }}-{\tilde{D}}_{\lambda l}^{\alpha }\right){\hat{\alpha }}_{\lambda }^{\mathrm{p}\mathrm{o}\mathrm{p}}{\beta }_l,$$

and so Eq. (A.19) becomes

$$\text{Cov}\left(\text{Δ}PG{S}_1,\text{Δ}{Y}_2\right)=2\sum _{\lambda \in \text{Λ}}\sum _{l\in L}\left(1-2c_{\lambda l}\right)\left(D_{\lambda l}^{\prime }-{\tilde{D}}_{\lambda l}^{\prime }\right){\hat{\alpha }}_{\lambda }^{\text{Pop}}{\beta }_l,$$ (A.20)

where $c_{\lambda l}$ is the sex-averaged recombination fraction between $\lambda$ and $l$.

Before we substitute the population GWAS estimates ${\hat{\alpha }}_{\lambda }^{\text{pop}}$ into Eq. (A.20), it is worth considering what value this expression would take if effect sizes were correctly estimated at every study locus, ${\hat{\alpha }}_{\lambda }^{\text{pop}}={\alpha }_{\lambda }$. In this case, Eq. (A.20) becomes

$$\begin{array}{l}\mathrm{Cov}\left(\mathrm{\Delta }PG{S}_1,\mathrm{\Delta }{Y}_2\right)=2\sum _{\lambda \in \mathrm{\Lambda }}\sum _{l\in L}\left(1-2c_{\lambda l}\right)\left(D_{\lambda l}-{\tilde{D}}_{\lambda l}\right){\alpha }_{\lambda }{\beta }_l\\ =\sum _{\lambda \in \mathrm{\Lambda }}2p_{\lambda }\left(1-p_{\lambda }\right){\alpha }_{\lambda }{\beta }_{\lambda }+2\sum _{\lambda \in \mathrm{\Lambda }}\sum _{\begin{array}{c}l\in L\\ l\ne \lambda \end{array}}\left(1-2c_{\lambda l}\right)\left(D_{\lambda l}^{\prime }-{\tilde{D}}_{\lambda l}^{\prime }\right){\alpha }_{\lambda }{\beta }_l.\end{array}$$ (A.21)

If the two traits are distinct, then the first term in Eq. (A.21) is the genic covariance of traits 1 and 2 across the set of study loci (more precisely, tagged locally by the study loci), and reflects systematic pleiotropy at these loci; this term would, for example, be positive if alleles tend to have same-direction effects on traits 1 and 2. If we were studying only one trait, then ${\alpha }_{\lambda }={\beta }_{\lambda }$, and the first term would be the genic variance of the trait across study loci, ${\sum }_{\lambda \in \mathrm{\Lambda }} 2p_{\lambda }\left(1-p_{\lambda }\right){\alpha }_{\lambda }^2$. The socond term in Eq. (A.21) is an effect of linkage disequilibria between study loci and the loci that are causal for trait 2; these LDs are absorbed by the PGS because the PGS is a sum across loci. In the absence of such LDs, or in cases where the cis- and trans-LDs are equal so that $D_{\lambda l}^{\mathrm{\prime }}-{\tilde{D}}_{\lambda l}^{\mathrm{\prime }}=0$, Eq. (A.21) would equal the genic variance in the single-trait case and the genic covariance in the two-trait case.

The effect size estimates from a population GWAS are in fact

$${\hat{\alpha }}_{\lambda }^{\text{pop}}=\frac{2}{V_{\lambda }}\sum _{l\in L} (D_{\lambda l^{\mathrm{\prime }}}+{\tilde{D}}_{\lambda l^{\mathrm{\prime }}}){\alpha }_{l^{\mathrm{\prime }}}\approx {\alpha }_{\lambda }+\frac{2}{V_{\lambda }}\sum _{\begin{array}{c}{l}^{\mathrm{\prime }}\in L\\ l^{\mathrm{\prime }}\ne \lambda \end{array}} (D_{{\lambda }^{\mathrm{\prime }}}+{\tilde{D}}_{\lambda l^{\mathrm{\prime }}}){\alpha }_{l^{\mathrm{\prime }}},$$

$D_{{\lambda }^{\mathrm{\prime }}}$ and ${\tilde{D}}_{\lambda l^{\mathrm{\prime }}}$ are measured in the sample. We assume these to be equal to the values in parents in the family-based GWAS, ${\tilde{D}}_{\lambda l}^{\mathrm{\prime }}$ and ${\tilde{D}}_{\lambda l}^{\mathrm{\prime }}$, and so the value taken by Eq. (A.20) is

$$\begin{gathered} \mathrm{Cov}\left(\mathrm{\Delta }PG{S}_1,\mathrm{\Delta }{Y}_2\right)=2\sum _{\lambda \in \mathrm{\Lambda }}\sum _{l\in L}\left(1-2c_{\lambda l}\right)\left(D_{\lambda l}-{\tilde{D}}_{\lambda l}\right){\widehat{\alpha }}_{\lambda }^{\text{pop}}{\beta }_l \\ =2\sum _{\lambda \in \mathrm{\Lambda }}\sum _{l\in L}\left(1-2c_{\lambda l}\right)\left(D_{\lambda l}-{\tilde{D}}_{\lambda l}\right)\left({\alpha }_{\lambda }+\frac{2}{V_{\lambda }}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne \lambda \end{array}}\left(D_{\lambda l^{\prime }}+{\tilde{D}}_{\lambda l^{\prime }}\right){\alpha }_{l^{\prime }}\right){\beta }_l \\ =\underset{\text{pleiotropy}}{\underbrace{\sum _{\lambda \in \mathrm{\Lambda }}2p_{\lambda }\left(1-p_{\lambda }\right){\alpha }_{\lambda }{\beta }_{\lambda }}}+\underset{\begin{array}{c}\text{covariance}\text{from}\text{LD}\text{absorbed}\text{by}\text{PGS}\\ \text{because}\text{it}\text{is}\text{a}\text{sum}\text{across}\text{loci}\end{array}}{\underbrace{2\sum _{\lambda \in \mathrm{\Lambda }}\sum _{\begin{array}{c}l\in L\\ l\ne \lambda \end{array}}\left(1-2c_{\lambda l}\right)\left(D_{\lambda l}-{\tilde{D}}_{\lambda l}\right){\alpha }_{\lambda }{\beta }_l}} \\ +\underset{\begin{array}{c}\text{covariance}\text{from}\text{LD}\text{absorbed}\text{by}\text{PGS}\\ \text{because}\text{effect}\text{size}\text{estimates}\text{absorb}\text{LD}\end{array}}{\underbrace{4\sum _{\lambda \in \mathrm{\Lambda }}\sum _{\begin{array}{c}l\in L\\ l\ne \lambda \end{array}}\left(1-2c_{\lambda l}\right)\left(D_{\lambda l}^2-{\tilde{D}}_{\lambda l}^2\right){\alpha }_l{\beta }_l/V_{\lambda }}} \\ +\underset{\begin{array}{l}\text{covariance}\text{from}\text{systematic}\text{LD}\text{between}\text{variants}\\ \text{with}\text{same}\text{directional}\text{effect}\text{on}\text{trait}\end{array}}{\underbrace{4\sum _{\lambda \in \mathrm{\Lambda }}\sum _{\begin{array}{c}l\in L\\ l\ne \lambda \end{array}}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne \lambda ,l\end{array}}\left(1-2c_{\lambda l}\right)\left(D_{\lambda l^{\prime }}+{\tilde{D}}_{\lambda l^{\prime }}\right){\alpha }_{l^{\prime }}\left(D_{\lambda l}-{\tilde{D}}_{\lambda l}\right){\beta }_l/V_{\lambda }.}} \end{gathered}$$ (A.22)

In the absence of genetic confounding $\left(D_{\lambda l}={\tilde{D}}_{\lambda l}=0\right)$ or, more generally, if genetic stratification is such that the cis- and trans-LDs are equal $\left(D_{\lambda l}-{\tilde{D}}_{\lambda l}=0\right.$), then Eq- (A.22) simplifies to the SNP-tagged genic covariance between traits 1 and 2:

$$\mathrm{cov}\left(\text{Δ}PG{S}_1,\text{Δ}{Y}_2\right)=\sum _{\lambda \in \text{Λ}}2p_{\lambda }\left(1-p_{\lambda }\right){\alpha }_{\lambda }{\beta }_{\lambda }.$$ (A.23)

If traits 1 and 2 are the same, then this is simply the SNP-tagged genic variance of the trait: $\mathrm{Cov}(△PGS,△Y)=$ ${\sum }_{\lambda \in \mathrm{\Lambda }} 2p_{\lambda }\left(1-p_{\lambda }\right){\alpha }_{\lambda }^2$.

Eq. (A.22) simplifies somewhat if we focus on a single trait $\left({\alpha }_l={\beta }_l\right)$ and assume that there is no trans-LD $\left({\tilde{D}}_{\mathrm{X}}=0\right)$; in this case,

$$\begin{array}{l}\mathrm{Cov}(\mathrm{\Delta }PGS,\mathrm{\Delta }Y)=\underset{\text{SNP-tagged}\text{genic}\text{variance}}{\underbrace{\sum _{\lambda \in \mathrm{\Lambda }}2p_{\lambda }\left(1-p_{\lambda }\right){\alpha }_{\lambda }^2}}+\underset{\begin{array}{c}\text{variance}\text{from}\text{LD}\text{absorbed}\text{by}\text{PGS}\\ \text{because}\text{it}\text{is}\text{a}\text{sum}\text{across}\text{loci}\end{array}}{\underbrace{2\sum _{\begin{array}{c}\lambda \in \mathrm{\Lambda }\\ l\ne L\end{array}}\sum _{l\in L}\left(1-2c_{\lambda l}\right)D_{\lambda l}{\alpha }_{\lambda }{\alpha }_l}}\\ +\underset{\begin{array}{l}\text{variance}\text{from}\text{LD}\text{absorbed}\text{by}\text{PGS}\\ \text{because}\text{effect}\text{size}\text{estimates}\text{absorb}\text{LD}\end{array}}{\underbrace{4\sum _{\lambda \in \mathrm{\Lambda }}\sum _{\begin{array}{c}l\in L\\ l\ne \lambda \end{array}}\left(1-2c_{\lambda l}\right)D_{\lambda l}^2{\alpha }_l^2/V_{\lambda }}}+\underset{\begin{array}{c}\text{variance}\text{from}\text{systematic}\text{LD}\text{between}\text{variants}\\ \text{with}\text{same}\text{directional}\text{effect}\text{on}\text{trait}\end{array}}{\underbrace{4\sum _{\lambda \in \mathrm{\Lambda }}\sum _{\begin{array}{c}l\in L\\ l\ne \lambda \end{array}}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne \lambda ,l\end{array}}\left(1-2c_{\lambda l}\right)D_{\lambda l}{\alpha }_l{D}_{\lambda l^{\prime }}{\alpha }_{l^{\prime }}/V_{\lambda }}}.\end{array}$$ (A.24)

A3. Sources of genetic confounding

The calculations above reveal that genetic confounds in GWAS designs can depend on long-range LD in the sample and among parents of the sample. Here, we consider several possible sources of long-range LD.

A3.1. Assortative mating

If there is a constant correlation among mates for their values of two traits, then a genetic equilibrium will eventually be achieved. In this equilibrium, for any pair of loci $l$ and $l^{\mathrm{\prime }}$, the trans-LD ${\tilde{D}}_{{ll}^{\mathrm{\prime }}}$ will be constant. Call this constant value $D_{l{l}^{\prime }}^{\mathrm{*}}$, and suppose that the recombination fraction between the loci is $c_{l{l}^{\prime }}$. With ${\tilde{D}}_{l{l}^{\prime }}$, constant across generations, the balance of its conversion into cis-LD (at rate c cur per generation) and the destruction of cis-LD by recombination (at rate $c_{l{l}^{\prime }}$ per generation) will result in an equilibrium level of cis-LD equal to the degree of trans-LD: $D_{l{l}^{\prime }}={\tilde{D}}_{l{l}^{\prime }}=D_{l{l}^{\prime }}^{\mathrm{*}}$ (e.g., Crow and Felsenstein 1968).

The value of $D_{l{l}^{\prime }}^{\mathrm{*}}$ will, in general, depend in a complicated way on the strength of effects of $l$ and $l^{\mathrm{\prime }}$ on the traits upon which assortative mating is based and on the linkage relations of these loci to one another and to other causal loci. However, while it is therefore difficult to calculate the individual equilibrium LD terms $D_{l{l}^{\prime }}^{\mathrm{*}}$, we can in some cases calculate weighted sums of these terms across locus pairs.

Let the set of loci that influence one or both traits be $L$, and let ${\alpha }_l$ be the effect size of the focal variant at locus $l$ on trait 1 and ${\beta }_l$ its effect on trait 2 (the analyses below also apply to same-trait assortative mating, setting ${\alpha }_l={\beta }_l$). Recall the notation $g_l^{\mathrm{m}\text{,mat}}$ and $g_l^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}}$ for a mother’s maternally and paternally inherited genotype at locus $l$, with $g_l^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}$ and $g_l^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}$ a father’s analogs. The mother’s breeding value for trait 1 is

$$G_1^{\mathrm{m}\mathrm{n}}=\sum _{l\in L} g_l^{\mathrm{m}}{\alpha }_l=\sum _{l\in L} \left(g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}+g_l^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}}\right){\alpha }_l=\sum _{l\in L} g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}{\alpha }_l+\sum _{l\in L} g_l^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}}{\alpha }_l=G_1^{\mathrm{m},\mathrm{m}\mathrm{a}t}+G_1^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}},$$

and, similarly, her breeding value for trait 2 is

$$G_2^{\mathrm{m}}=\sum _{l\in L} g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}{\beta }_l+\sum _{l\in L} g_l^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}}{\beta }_l=G_2^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}+G_2^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}}.$$

The father’s breeding values for the two traits are

$$G_1^{\mathrm{f}}=\sum _{l\in L} g_l^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}{\alpha }_l+\sum _{l\in L} g_l^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}{\alpha }_l=G_1^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}+G_1^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}$$

and

$$G_2^{\mathrm{f}}=\sum _{l\in L} g_l^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}{\beta }_l+\sum _{l\in L} g_l^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}{\beta }_l=G_2^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}+G_2^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}.$$

We assume that individual trait values equal the breeding values plus environmental disturbances that are uncorrelated with the breeding values:

$$Y_1^{\mathrm{m}\mathrm{n}}=G_1^{\mathrm{m}}+{ϵ}_1^{\mathrm{m}};Y_2^{\mathrm{m}}=G_2^{\mathrm{m}}+{ϵ}_2^{\mathrm{m}};Y_1^{\mathrm{f}}=G_1^{\mathrm{f}}+{ϵ}_1^{\mathrm{f}};Y_2^{\mathrm{f}}=G_2^{\mathrm{f}}+{ϵ}_2^{\mathrm{f}};$$

where

$$\mathrm{Var}\left({ϵ}_1^{\mathrm{m}}\right)=\mathrm{Var}\left({ϵ}_1^{\mathrm{f}}\right)=V_E^1,\mathrm{Var}\left({ϵ}_2^{\mathrm{m}}\right)=\mathrm{Var}\left({ϵ}_2^{\mathrm{f}}\right)=V_E^2,$$

and

$$\mathrm{Cov}\left({ϵ}_i^{\mathrm{m}},G_i^{\mathrm{m}}\right)=\mathrm{Cov}({ϵ}_i^{\mathrm{f}},G_i^{\mathrm{f}})=0\text{for}i\in \left\{1,2\right\}.$$

A3.1.1. Same-trait assortative mating, or cross-trait assortative mating that is symmetric with respect to sex

We first consider the case where the strength of assortative mating between two traits, ass measured by their correlation coefficient across mating pairs, is equal in the female-male and male-female directions. Notice that this scenario covers same-trait assortative mating. In the case of cross-trait assortative mating, it could occur if assortative mating arises by mechanisms other than direct female (or male) mating preferences.

We assume that there is a constant correlation $\rho$ among mating pairs for their phenotypic values of traits 1 and 2. In equilibrium, this will translate to a constant correlation ${\rho }_G$ between their breeding values as well (e.g., Felsenstein 1981). To calculate ${\rho }_G$, we first note that, because assortative mating is bssed on phenotypic values and not breeding values per se, if we know the phenotypes of a pair of mates, we obtain no further information about the similarity of their breeding values; that is,

$$\mathrm{Cov}\left(G_1^{\mathrm{m}},G_2^{\mathrm{f}}\mid \left\{Y_1^{\mathrm{m}},Y_2^{\mathrm{f}}\right\}\right)=\mathrm{Cov}\left(G_2^{\mathrm{m}},G_1^{\mathrm{f}}\mid \left\{Y_2^{\mathrm{m}},Y_1^{\mathrm{f}}\right\}\right)=0.$$ (A.25)

For the same reason, if we know the phenotypic values of two mates, then the trait-2 value of the male does not offer any information on the female’s trait-1 breeding value beyond that already offered by the female’s trait-1 phenotype, and vice versa; that is,

$$\begin{array}{r}\mathbb{E}\left[G_1^{\mathrm{m}}\mid \left\{Y_1^{\mathrm{m}},Y_2^{\mathrm{f}}\right\}\right]=\mathbb{E}\left[G_1^{\mathrm{m}}\mid Y_1^{\mathrm{m}}\right];\mathbb{E}\left[G_2^{\mathrm{f}}\mid \left\{Y_1^{\mathrm{m}},Y_2^{\mathrm{f}}\right\}\right]=\mathbb{E}\left[G_2^{\mathrm{f}}\mid Y_2^{\mathrm{f}}\right];\\ \mathbb{E}\left[G_2^{\mathrm{m}}\mid \left\{Y_2^{\mathrm{m}},Y_1^{\mathrm{f}}\right\}\right]=\mathbb{E}\left[G_2^{\mathrm{m}}\mid Y_2^{\mathrm{m}}\right];\mathbb{E}\left[G_1^{\mathrm{f}}\mid \left\{Y_2^{\mathrm{m}},Y_1^{\mathrm{f}}\right\}\right]=\mathbb{E}\left[G_1^{\mathrm{f}}\mid Y_1^{\mathrm{f}}\right].\end{array}$$

If $Y_1$ and $G_1$, and similarly $Y_2$ and $G_2$, are bivariate normal, then

$$\mathbb{E}\left[G_1\mid Y_1\right]=\mathbb{E}\left[G_1\right]+h_1^2\left(Y_1-\mathbb{E}\left[Y_1\right]\right)\text{and}\mathbb{E}\left[G_2\mid Y_2\right]=\mathbb{E}\left[G_2\right]+h_2^2\left(Y_2-\mathbb{E}\left[Y_2\right]\right)$$ (A.27)

where $h_1^2$ and $h_2^2$ are the heritabilities of traits 1 and 2, respectively.

From the law of total covariance,

$$\begin{array}{l}\mathrm{cov}\left(G_1^{\text{m}},G_2^{\text{f}}\right)={\mathrm{Cov}}_{\left\{Y_1^{\text{m}},Y_2^{\text{f}}\right\}}\left(\mathbb{E}\left[G_1^{\text{m}}\mid \left\{Y_1^{\text{m}},Y_2^{\text{f}}\right\}\right],\mathbb{E}\left[G_2^{\text{f}}\mid \left\{Y_1^{\text{m}},Y_2^{\text{f}}\right\}\right]\right)\\ +{\mathbb{E}}_{\left\{Y_1^{\text{m}},Y_2^{\text{f}}\right\}}\left[\mathrm{Cov}\left(G_1^{\text{m}},G_2^{\text{f}}\mid \left\{Y_1^{\text{m}},Y_2^{\text{f}}\right\}\right)\right]\\ ={\mathrm{Cov}}_{\left\{Y_1^{\text{m}},Y_2^{\text{f}}\right\}}\left(\mathbb{E}\left[G_1^{\text{m}}\mid Y_1^{\text{m}}\right],\mathbb{E}\left[G_2^{\text{f}}\mid Y_2^{\text{f}}\right]\right)[\text{from}\text{Eqs}.\text{A}.25\text{and}\text{A}.26]\\ =\mathrm{Cov}\left(h_1^2{Y}_1^{\text{m}},h_2^2{Y}_2^{\text{f}}\right)[\text{from}\text{Eq}.\text{(A}.27)]\\ =h_1^2{h}_2^2\mathrm{Cov}\left(Y_1^{\text{m}},Y_2^{\text{f}}\right).\end{array}$$ (A.28)

Similarly, $\mathrm{Cov}(G_2^{\mathrm{m}},G_1^{\mathrm{f}})=h_1^2{h}_2^2\mathrm{Cov}\left(Y_2^{\mathrm{m}},Y_1^{\mathrm{f}}\right)$.

Let $V^1$ and $V^2$ be the phenotypic variances of traits 1 and 2, and $V_G^1$ and $V_G^2$ their additive genetic variances, assumed to be the same across the sexes. Given the calculations above, the correlation among mates for their breeding values of traits 1 and 2, ${\rho }_G$, can be written

$$\begin{gathered} {\rho }_G=\frac{\frac{1}{2}\left[\text{Cov}\left(G_1^{\text{m}},G_2^{\text{f}}\right)+\text{Cov}\left(G_2^{\text{m}},G_1^{\text{f}}\right)\right]}{\sqrt{V_G^1{V}_G^2}} \\ =\frac{\frac{h_1^2{h}_2^2}{2}\left[\text{Cov}\left(Y_1^{\text{m}},Y_2^{\text{f}}\right)+\text{Cov}\left(Y_2^{\text{m}},Y_1^{\text{f}}\right)\right]}{\sqrt{h_1^2{V}^1{h}_2^2{V}^2}} \end{gathered}$$ (A.29)

$$=h_1{h}_2\frac{\frac{1}{2}\left[\text{Cov}\left(Y_1^{\text{m}},Y_2^{\text{f}}\right)+\text{Cov}\left(Y_2^{\text{m}},Y_1^{\text{f}}\right)\right]}{\sqrt{V^1{V}^2}}=h_1{h}_2\rho .$$ (A.30)

When traits 1 and 2 are the same, we have ${\rho }_G=h^2\rho$, a standard result (e.g., Wright 1921; Felsenstein 1981).

Expanding the numerator of Eq. (A.29),

$$\begin{array}{l}\frac{1}{2}\left[\mathrm{Cov}\left(G_1^{\mathrm{m}},G_2^{\mathrm{f}}\right)+\mathrm{Cov}\left(G_2^{\mathrm{m}},G_1^{\mathrm{f}}\right)\right]=\frac{1}{2}\left[\mathrm{Cov}\left(G_1^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}+G_1^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}},G_2^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}+G_2^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}\right)\right.\\ \left.+\mathrm{Cov}\left(G_2^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}+G_2^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}},G_1^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}+G_1^{f,\mathrm{p}\mathrm{a}\mathrm{t}}\right)\right]\\ =\frac{1}{2}\left[\mathrm{Cov}\left(G_1^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},G_2^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}\right)+\mathrm{Cov}\left(G_1^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},G_2^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}\right)\right]+\frac{1}{2}\left[\mathrm{Cov}\left(G_1^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}},G_2^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}\right)+\mathrm{Cov}\left(G_1^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}},G_2^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}\right)\right]\\ +\frac{1}{2}\left[\mathrm{Cov}\left(G_2^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},G_1^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}\right)+\mathrm{Cov}\left(G_2^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},G_1^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}\right)\right]+\frac{1}{2}\left[\mathrm{Cov}\left(G_2^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}},G_1^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}\right)+\mathrm{Cov}\left(G_2^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}},G_1^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}\right)\right]\\ =\frac{1}{2}\left[\mathrm{Cov}\left(G_1^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},G_2^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}\right)+\mathrm{Cov}\left(G_2^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},G_1^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}\right)\right]+\frac{1}{2}\left[\mathrm{Cov}\left(G_1^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},G_2^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}\right)+\mathrm{Cov}\left(G_2^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},G_1^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}\right)\right]\\ +\frac{1}{2}\left[\mathrm{Cov}\left(G_1^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}},G_2^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}\right)+\mathrm{Cov}\left(G_2^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}},G_1^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}\right)\right]+\frac{1}{2}\left[\mathrm{Cov}\left(G_1^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}},G_2^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}\right)+\mathrm{Cov}\left(G_2^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}},G_1^{\mathrm{f},\mathrm{p}\mathrm{a}\mathrm{t}}\right)\right].\\ \end{array}$$ (A.31)

But

$$\begin{array}{l}\frac{1}{2}\left[\mathrm{Cov}\left(G_1^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},G_2^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}\right)+\mathrm{Cov}\left(G_2^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},G_1^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}\right)\right]=\frac{1}{2}\left[\mathrm{Cov}\left(\sum _{l\in L} g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}{\alpha }_l,\sum _{l^{\mathrm{\prime }}\in L} g_{l^{\mathrm{\prime }}}^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}{\beta }_{l^{\mathrm{\prime }}}\right)\right.\\ \left.+\mathrm{Cov}\left(\sum _{l\in L} g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}{\beta }_l,\sum _{l,\in L} g_V^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}{\alpha }_{l^{\mathrm{\prime }}}\right)\right]\\ =\frac{1}{2}\left[\sum _{l\in L} \sum _{l^{\mathrm{\prime }}\in L} \mathrm{Cov}\left(g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},g_{l^{\mathrm{\prime }}}^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}\right){\alpha }_l{\beta }_{l^{\mathrm{\prime }}}+\sum _{l\in L} \sum _{l^{\mathrm{\prime }}\in L} \mathrm{Cov}\left(g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},g_{l^{\mathrm{\prime }}}^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}\right){\alpha }_{l^{\mathrm{\prime }}}{\beta }_l\right]\\ =\frac{1}{2}\left[\sum _{l\in L} \sum _{l^{\mathrm{\prime }}\in L} \mathrm{Cov}\left(g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},g_{l^{\mathrm{\prime }}}^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}\right){\alpha }_l{\beta }_{l^{\mathrm{\prime }}}+\sum _{l\in L} \sum _{l^{\mathrm{\prime }}\in L} \mathrm{Cov}\left(g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},g_l^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}\right){\alpha }_{l^{\mathrm{\prime }}}{\beta }_l\right]\\ =\sum _{l\in L} \sum _{l^{\mathrm{\prime }}\in L} \frac{1}{2}\left[\mathrm{Cov}\left(g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},g_{l^{\mathrm{\prime }}}^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}\right)+\mathrm{Cov}\left(g_{l^{\mathrm{\prime }}}^{\mathrm{m}\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},g_l^{\mathrm{f},\mathrm{m}\mathrm{a}\mathrm{t}}\right)\right]{\alpha }_l{\beta }_{l^{\mathrm{\prime }}}\\ =\sum _{l\in L} \sum _{l^{\mathrm{\prime }}\in L} {\tilde{D}}_{l{l}^{\mathrm{\prime }}}{\alpha }_l{\beta }_{l^{\mathrm{\prime }}},\\ \end{array}$$

since grandmaternal and grandpaternal alleles are transmitted to the offspring with equal probability, independently across maternal and paternal transmission. The three additional terms in Eq. (A.31) likewise each amount to ${\sum }_{l\in L} {\sum }_{l^{\prime }\in L} {\tilde{D}}_{l{l}^{\mathrm{\prime }}}{\alpha }_l{\beta }_{l^{\mathrm{\prime }}}$, and so

$$\frac{1}{2}\left[\text{Cov}\left(G_1^{\text{m}},G_2^{\text{f}}\right)+\text{Cov}\left(G_2^{\text{m}},G_1^{\text{f}}\right)\right]=4\sum _{l\in L}\sum _{l^{\prime }\in L}{\tilde{D}}_{l{l}^{\prime }}{\alpha }_l{\beta }_{l^{\prime }}.$$ (A.32)

Noting that the trans-covariance at a given locus ${\tilde{D}}_l=p_l\left(1-p_l\right){\tilde{r}}_{ll}$, where ${\tilde{r}}_{ll}$ is the within-locus correlation (equal to the inbreeding coefficient at the locus), we can split Eq. (A.32) into within- and between-locus terms:

$$\frac{1}{2}\left[\mathrm{Cov}\left(G_1^{\text{m}},G_2^{\text{f}}\right)+\mathrm{Cov}\left(G_2^{\text{m}},G_1^{\text{f}}\right)\right]=4\sum _{l\in L}{p}_l\left(1-p_l\right){\tilde{r}}_{ll}{\alpha }_l{\beta }_l+4\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}{\tilde{D}}_{l{l}^{\prime }}{\alpha }_l\mid {\beta }_{l^{\prime }}.$$ (A.33)

In the denominator of Eq. (A.29),

$$V_G^1=\mathrm{Var}\left(G_1^{\mathrm{m}\mathrm{m}}\right)=\mathrm{Var}\left(G_1^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}+G_1^{\mathrm{m}\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}}\right)=\mathrm{Var}\left(G_1^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}\right)+\mathrm{Var}\left(G_1^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}}\right)+2\mathrm{Cov}\left(G_1^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},G_1^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}}\right),$$ (A.34)

Expanding the first term,

$$\begin{array}{l}\mathrm{Var}\left(G_1^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}\right)=\mathrm{Var}\left(\sum _{l\in L} g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}{\alpha }_l\right)=\sum _{l\in L} \mathrm{Var}\left(g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}\right){\alpha }_l^2+\sum _{l\in L} \sum _{\begin{array}{c}{l}^{\mathrm{\prime }}\in L\\ l^{\mathrm{\prime }}\ne l\end{array}} \mathrm{Cov}\left(g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},g_{l^{\mathrm{\prime }}}^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}\right){\alpha }_l{\alpha }_l\prime \\ =\sum _{l\in L} p_l\left(1-p_l\right){\alpha }_l^2+\sum _{l\in L} \sum _{\begin{array}{c}{l}^{\mathrm{\prime }}\in L\\ l^{\mathrm{\prime }}\ne l\end{array}} D_{l{l}^{\mathrm{\prime }}}^{\mathrm{\prime }}{\alpha }_l{\alpha }_{l^{\mathrm{\prime }}}.\\ \end{array}$$

Similarly, the second term is

$$\mathrm{Var}\left(G_1^{\text{m},\text{pat}}\right)=\sum _{l\in L}{p}_l\left(1-p_l\right){\alpha }_l^2+\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}{D}_{l{l}^{\prime }}^{\prime }{\alpha }_l{\alpha }_{l^{\prime }}.$$

The third, covariance term in Eq. (A.34) is

$$\begin{array}{l}\mathrm{Cov}(G_1^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},G_1^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}})=\mathrm{Cov}\left(\sum _{l\in L} g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}}{\alpha }_l,\sum _{l^{\mathrm{\prime }}\in L} g_{l^{\mathrm{\prime }}}^{\mathrm{m}\mathrm{p}\mathrm{a}\mathrm{t}}{\alpha }_{l^{\mathrm{\prime }}}\right)=\sum _{l\in L} \sum _{l^{\mathrm{\prime }}\in L} \mathrm{Cov}\left(g_l^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},g_{l^{\mathrm{\prime }}}^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}}\right){\alpha }_l{\alpha }_{l^{\mathrm{\prime }}}\\ =\sum _{l\in L} \sum _{l^{\mathrm{\prime }}\in L} \frac{1}{2}\left[\mathrm{Cov}\left(g_l^{\mathrm{m}\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},g_{l^{\mathrm{\prime }}}^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}}\right)+\mathrm{Cov}\left(g_{l^{\mathrm{\prime }}}^{\mathrm{m},\mathrm{m}\mathrm{a}\mathrm{t}},g_l^{\mathrm{m},\mathrm{p}\mathrm{a}\mathrm{t}}\right)\right]{\alpha }_l{\alpha }_{l^{\mathrm{\prime }}}\\ =\sum _{l\in L} \sum _{l^{\mathrm{\prime }}\in L} {\tilde{D}}_{{ll}^{\mathrm{\prime }}}^{\mathrm{\prime }},{\alpha }_l{\alpha }_{l^{\mathrm{\prime }}}=\sum _{l\in L} p_l\left(1-p_l\right){\tilde{r}}_{ll}^{\mathrm{\prime }}{\alpha }_l^2+\sum _{l\in L} \sum _{l^{\mathrm{\prime }}\in L} {\tilde{D}}_{{ll}^{\mathrm{\prime }}}^{\mathrm{\prime }}{\tilde{D}}^{\mathrm{\prime }}{\alpha }_l{\alpha }_{l^{\mathrm{\prime }}}.\end{array}$$

Putting these together in Eq. (A.34),

$$V_G^1=\mathrm{Var}\left(G_1^{\text{m}}\right)=2\sum _{l\in L}{p}_l\left(1-p_l\right)\left(1+{\tilde{r}}_{ll}^{\prime }\right){\alpha }_l^2+2\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}\left(D_{l{l}^{\prime }}^{\prime }+{\tilde{D}}_{l{l}^{\prime }}^{\prime }\right){\alpha }_l{\alpha }_{l^{\prime }}.$$

Similarly,

$$V_G^2=\mathrm{Var}\left(G_2^{\text{m}}\right)=2\sum _{l\in L}{p}_l\left(1-p_l\right)\left(1+{\tilde{r}}_{ll}^{\prime }\right){\beta }_l^2+2\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}\left(D_{l{l}^{\prime }}^{\prime }+{\tilde{D}}_{l{l}^{\prime }}^{\prime }\right){\beta }_l{\beta }_{l^{\prime }}.$$

In equilibrium, $D_{{ll}^{\mathrm{\prime }}}^{\prime }={\tilde{D}}_{{ll}^{\mathrm{\prime }}}^{\mathrm{\prime }}={\tilde{D}}_{{ll}^{\mathrm{\prime }}}=D_{{ll}^{\mathrm{\prime }}}^{\mathrm{*}}$, for $l\ne l^{\mathrm{\prime }}$, and ${\bar{r}}_{ll}^{\mathrm{\prime }}={\tilde{r}}_{ll}={\tilde{r}}_{ll}^{\mathrm{*}}$, so

$$\frac{1}{2}\left[\mathrm{Cov}\left(G_1^{\text{m}},G_2^{\text{f}}\right)+\mathrm{Cov}\left(G_2^{\text{m}},G_1^{\text{f}}\right)\right]=4\sum _{l\in L}{p}_l\left(1-p_l\right)r_{ll}^{*}{\alpha }_l{\beta }_l+4\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}{D}_{l{l}^{\prime }}^{*}{\alpha }_l{\beta }_{l^{\prime }},$$ (A.35)

$$V_G^1=2\sum _{l\in L}{p}_l\left(1-p_l\right)\left(1+{\tilde{r}}_{ll}^{*}\right){\alpha }_l^2+4\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}{D}_{l{l}^{\prime }}^{*}{\alpha }_l{\alpha }_{l^{\prime }}+V_E^1\approx V_g^1++4\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}{D}_{l{l}^{\prime }}^{*}{\alpha }_l{\alpha }_{l^{\prime }},$$ (A.36)

$$V_G^2=2\sum _{l\in L}{p}_l\left(1-p_l\right)\left(1+{\tilde{r}}_{ll}^{*}\right){\beta }_l^2+4\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}{D}_{l{l}^{\prime }}^{*}{\beta }_l{\beta }_{l^{\prime }}+V_E^2\approx V_g^2+4\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}^{*}{D}_{l{l}^{\prime }}^{*}{\beta }_l{\beta }_{l^{\prime }},$$ (A.37)

where $V_g^1$ and $V_g^2$ are the genic variances of traits 1 and 2, and the approximations come from the fact that, under assortative mating for a polygenic trait, the sum of the $~|L{|}^2$ cross-locus trans-LD terms ${\tilde{D}}_{{ll}^{\prime }}^{\mathrm{*}}$ dominates the sum of the $\left|L\right|$ within-locus trans-LD terms ${\tilde{D}}_{ll}^{\mathrm{*}}=p_l\left(1-p_l\right){\tilde{r}}_{ll}^{\mathrm{*}}$ (Crow and Kimura 1970, Ch. 4). Eq. (A.29) in equilibrium is therefore

$$\begin{array}{l}{\rho }_G=\frac{4\sum _{l\in L}{p}_l\left(1-p_l\right){\tilde{r}}_{ll}^{*}{\alpha }_l{\beta }_l+4\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}{D}_{l^{\prime }}^{*}{\alpha }_l{\beta }_{l^{\prime }}}{\sqrt{V_G^1{V}_G^2}}\\ \approx \frac{4\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}{D}_{l{l}^{\prime }}^{*}{\alpha }_l{\beta }_{l^{\prime }}}{\sqrt{\left(V_g^1+4\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}{D}_{l{l}^{\prime }}^{*}{\alpha }_l{\alpha }_{l^{\prime }}\right)\left(V_g^2+4\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}{D}_{l{l}^{\prime }}^{*}{\beta }_l{\beta }_{l^{\prime }}\right)}}.\end{array}$$ (A.38)

We now consider some special cases.

Same-trait assortative mating with equal effect sizes.

In the case of same-trait assortative mating, $\alpha l={\beta }_l$, so Eq. (A.38) simplifies to

$${\rho }_G=\frac{4\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}{D}_{l{l}^{\prime }}^{*}{\alpha }_l{\alpha }_{l^{\prime }}}{V_g+4\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}{D}_{l{l}^{\prime }}^{*}{\alpha }_l{\alpha }_{l^{\prime }}},$$ (A.39)

from which

$$4\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}{D}_{l^{\prime }}^{*}{\alpha }_l{\alpha }_{l^{\prime }}\approx \frac{{\rho }_G}{1-{\rho }_G}{V}_g\left(=\frac{h^2\rho }{1-h^2\rho }{V}_g\right).$$ (A.40)

Since, in equilibrium, $D_{l{l}^{\mathrm{\prime }}}={\tilde{D}}_{l{l}^{\mathrm{\prime }}}$, this expression can also be written

$$2\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}\left(D_{l{l}^{\prime }}^{*}+{\tilde{D}}_{l{l}^{\prime }}^{*}\right){\alpha }_l{\alpha }_{l^{\prime }}\approx \frac{{\rho }_G}{1-{\rho }_G}{V}_g.$$ (A.41)

Because the additive genetic variance $V_G=V_g+2{\sum }_{l\in L} {\sum }_{\begin{array}{c}{l}^{\mathrm{\prime }}\in L\\ l^{\mathrm{\prime }}\ne l\end{array}} (D_{l{l}^{\mathrm{\prime }}}^{\mathrm{*}}+{\tilde{D}}_{l{l}^{\mathrm{\prime }}}^{\mathrm{*}}){\alpha }_l{\alpha }_{l^{\mathrm{\prime }}}$, Eq. (A.41) can also be written

$$V_G=V_g/\left(1-{\rho }_G\right),$$ (A.42)

which is a classic result (e.g., Wright 1921; Crow and Kimura 1970, Ch. 4).

If we make the further assumption that effect sizes are the same across loci $(\alpha l=\alpha$ for all $l\in L)$, then Eq. (A.41) becomes

$$2\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}\left(D_{l{l}^{\prime }}^{*}+{\tilde{D}}_{l{l}^{\prime }}^{*}\right)\approx \frac{1}{{\alpha }^2}\frac{{\rho }_G}{1-{\rho }_G}{V}_g.$$ (A.43)

In a population association study at locus l, assuming no indirect effects and no sources of genetic confounding other than assortative mating, the effect size estimate is

$$\hat{\alpha }l=\alpha +\frac{2}{V_l}\sum _{\begin{array}{c}{l}^{\mathrm{\prime }}\in L\\ l^{\mathrm{\prime }}\ne l\end{array}} \left(D_{{ll}^{\mathrm{\prime }}}^{\mathrm{*}}+{\tilde{D}}_{{ll}^{\mathrm{\prime }}}^{\mathrm{*}}\right){\alpha }_{l^{\mathrm{\prime }}},$$

so that the proportionate bias in the effect size estimate at $l$ is

$$\frac{{\hat{\alpha }}_l-{\alpha }_l}{{\alpha }_l}=\frac{2}{V_l}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}\left(D_{l{l}^{\text{'}}}^{\text{*}}+{\tilde{D}}_{l{l}^{\text{′}}}^{\text{*}}\right)\frac{{\alpha }_{l^{\prime }}}{\alpha l}=\frac{2}{H_l}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}\left(D_{l{l}^{\text{'}}}^{\text{*}}+{\tilde{D}}_{l{l}^{\text{′}}}^{\text{*}}\right),$$ (A.44)

since ${\alpha }_l={\alpha }_l$ by assumption and $V_l\approx H_l=2p_l\left(1-p_l\right)$ because assortative mating does not substantially increase within-locus homozygosity (Crow and Kimura 1970, Ch. 4). The average proportionate bias across loci is then

$$\begin{array}{l}\frac{1}{\left|L\right|}\sum _{l\in L}\frac{{\hat{\alpha }}_l-{\alpha }_l}{{\alpha }_l}=\frac{1}{\left|L\right|}\sum _{l\in L}\frac{2}{H_l}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}\left(D_{l{l}^{\prime }}^{*}+{\tilde{D}}_{l{l}^{\prime }}^{*}\right)\\ \approx \frac{2}{\left|L\right|\overline{H}}\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}\left(D_{l{l}^{\prime }}^{*}+{\tilde{D}}_{l{l}^{\prime }}^{*}\right)\\ \approx \frac{1}{\left|L\right|\overline{H}{\alpha }^2}\frac{{\rho }_G}{1-{\rho }_G}{V}_g\\ =\frac{1}{V_g}\frac{{\rho }_G}{1-{\rho }_G}{V}_g\\ =\frac{{\rho }_G}{1-{\rho }_G},\end{array}$$ (A.45)

where we have used Eq. (A.43) and have assumed that minor allele frequencies do not differ widely across loci. Since ${\rho }_G=h^2\rho$, where $\rho$ is the phenotypic correlation among mates and $h^2=V_G/V_p$ is the heritability of the trait, Eq. (A.45) can also be written

$$\frac{1}{\left|L\right|}\sum _{l\in L}\frac{{\widehat{\alpha }}_l-{\alpha }_l}{{\alpha }_l}=\frac{h^2\rho }{1-h^2\rho }.$$ (A.46)

Sex-symmetric cross-trait assortative mating with distinct genetic bases and equal effect sizes.

In the case of cross-trait assortative mating, if the sets of loci underlying the two traits, $L_1$ and $L_2$, are distinct, then ${\alpha }_l\ne 0\Rightarrow {\beta }_l=0$ and ${\beta }_l\ne 0\Rightarrow {\alpha }_l=0$. In this case, Eq. (A.38) becomes

$${\rho }_G=\frac{4\sum _{l\in L_1}\sum _{l^{\prime }\in L_2}{D}_{l{l}^{\prime }}^{*}{\alpha }_l{\beta }_{l^{\prime }}}{\sqrt{V_G^1{V}_G^2}},$$ (A.47)

from which

$${\rho }_G\sqrt{V_G^1{V}_G^2}=4\sum _{l\in L_1}\sum _{l^{\prime }\in L_2}{D}_{l{l}^{\prime }}^{*}{\alpha }_l{\beta }_{l^{\prime }}=2\sum _{l\in L_1}\sum _{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}^{*}+{\tilde{D}}_{l{l}^{\prime }}^{*}\right){\alpha }_l{\beta }_{l^{\prime }}.$$ (A.48)

Because assortative mating is cross-trait, the LDs that assortative mating induces across $L_1$ and $L_2$ will dominate the second-order LDs induced within $L_1$ and within $L_2$. Therefore, $V_G^1\approx V_g^1$ and $V_G^2\approx V_g^2$.

The effect size estimate at a locus $l\in L_1$ in a population GWAS on trait 2 is

$${\widehat{\beta }}_l\approx \frac{2}{V_l}\sum _{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right){\beta }_{l^{\prime }}\approx \frac{2}{H_l}\sum _{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right){\beta }_{l^{\prime }},$$ (A.49)

while the true effect size ${\beta }_l$ is zero, since $l\notin L_2$. In equilibrium, the average effect size estimate, and thus the average deviation of these estimates from the true values, is therefore

$$\frac{1}{\left|L_1\right|}\sum _{l\in L_1}{\widehat{\beta }}_l\approx \frac{1}{\left|L_1\right|}\sum _{l\in L_1}\frac{2}{H_l}\sum _{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right){\beta }_{l^{\prime }}\approx \frac{2}{\left|L_1\right|{\overline{H}}_1}\sum _{l\in L_1}\sum _{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right){\beta }_{l^{\prime }},$$ (A.50)

where we have assumed that minor allele frequencies are not very different across $L_1$ $\left({\bar{H}}_1\right.$ is the average heterozygosity in $L_1$). If we further assume that effect sizes at causal loci are equal for each trait ( ${\alpha }_l=\alpha$ for all $l\in L_1$ and ${\beta }_{l^{\mathrm{\prime }}}=\beta$ for all $l^{\mathrm{\prime }}\in L_2$), then Eq. (A.50) can be written

$$\begin{array}{l}\frac{1}{\left|L_1\right|}\sum _{l\in L_1}{\widehat{\beta }}_l\approx \frac{2}{\left|L_1\right|{\overline{H}}_1}\sum _{l\in L_1}\sum _{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right)\beta \\ =\frac{\alpha }{\left|L_1\right|{\overline{H}}_1{\alpha }^2}\times 2\sum _{l\in L_1}\sum _{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right)\alpha \beta \\ =\frac{\alpha }{V_g^1}\times {\rho }_G\sqrt{V_G^1{V}_G^2}\text{[from}\text{Eq}.\text{A}.48]\\ \approx {\rho }_G\sqrt{\frac{V_G^1}{V_G^2}}\alpha =\sqrt{\frac{V_G^1}{V_P^1}\cdot \frac{V_G^2}{V_P^2}}\rho \sqrt{\frac{V_G^1}{V_G^2}}\alpha =\rho \frac{V_G^1}{\sqrt{V_P^1{V}_P^2}}\alpha ,\end{array}$$ (A.51)

recalling from Eq. (A.30) that ${\rho }_G=\sqrt{h_1{h}_2}\rho$.

In the further special case where both the genetic and the phenotypic variances of the two traits are equal, then so are the heritabilities of the two traits. In this case, Eq. (A.51) simplifies to

$$\frac{1}{\left|L_1\right|}\sum _{l\in L_1}{\widehat{\beta }}_l\approx \frac{V_G}{V_P}\rho \alpha =h^2\rho \alpha ,$$ (A.52)

where $h^2$ is the common heritability of the two traits.

Sex-symmetric cross-trait assortative mating for traits with different genetic architectures.

Eq. (A.52) reveals an interesting role for genetic architecture in the bias that cross-trait assortative mating can generate in population association studies performed at non-causal loci. Suppose, as we did in deriving Eq. (A.52), that the two traits on which assortative mating is based have the same genetic and phenotypic variances, $V_G$ and $V$, and therefore also the same heritabilities, $h^2$. We shall make the further assumption that the traits have the same genic variance, $V_g$. Assume further that the sets of loci underlying traits 1 and 2, $L_1$ and $L_2$, have similar mean heterozygocities $\approx \bar{H}$. Normalize the effect size sizes at loci causal for trait 2 to $\beta =1$, so that the traits’ common genic variance is $V_g=\left|L_2\right|\bar{H}$.

Suppose that we now perform a population GWAS for trait 2. At loci that are causal for trait 2 $\left(l\in L_2\right.$), we will estimate effect sizes accurately: ${\hat{\beta }}_l\approx 1$ (there will be a small positive second-order bias, of order ${\rho }^2$, since the locus $l\in L_2$ comes into positive LD with loci $l^{\mathrm{\prime }}\in L_1$, which in turn have come into positive LD with loci $l^{\mathrm{″}}\in L_2$).

At loci that are causal for trait $1\left(l\in L_1\right)$, and which therefore have no effect on trait 2, we will estimate effect sizes on average as given by Eq. (A.52): ${\hat{\beta }}_l=h^2\rho \alpha$.

How does the number of loci underlying variation in trait 1, $\left|L_1\right|$, affect this biased estimate of their effect on trait 2? For the genic variance of trait 1 to be the same as that of trait 2, $V_g=\left|L_1\right|\bar{H}{\alpha }^2=\left.\left|L_2\right|\bar{H}{\beta }^2=\left|L_2\right|\bar{H}\right)$, and so we must have ${\alpha }^2=\left|L_2\right|/\left|L_1\right|$. Substituting this into the average effect size estimate at non-causal loci, ${\hat{\beta }}_l=h^2\rho \sqrt{\left|L_2\right|/\left|L_1\right|}$.

So, the average effect size estimate at causal loci $l\in L_2$ is ${\hat{\beta }}_l\approx 1$, while the average effect size estimate at non-causal loci $l\in L_1$ is ${\hat{\beta }}_l=h^2\rho \sqrt{\left|L_2\right|/\left|L_1\right|}$. How do these two quantities compare? If the number of loci underlying the two traits is the same, $L_1=L_2$, and effect size estimates at non-causal loci are smaller than those at causal loci by a factor of about $h^2\rho$. However, if there are more loci underlying trait 2 than underlying trait 1—i.e., if trait 1 has a more concentrated genetic architecture $L_1<L_2$—then the effect size estimates at non-causal loci will be closer to those at causal loci. Indeed, if trait 1 has a sufficiently concentrated architecture relative to trait 2, specifically, if $L_1<h^4{\rho }^2{L}_2$, then the effect size estimates at non-causal loci will, on average, be larger in magnitude than effect size estimates at causal loci.

More generally, the calculations above suggest that, in a more realistic scenario where effect sizes vary across loci, the trait-2 GWAS distribution of magnitudes of effect size estimates at trait-1 loci (non-causal) will overlap more with the distribution of magnitudes of effect size estimates at trait-2 loci (causal) if the genetic architecture of trait 1 is more concentrated (Fig. 4). This will lead to a greater number of trait 1 loci being identified as statistically significantly associated with trait 2 in the trait-2 GWAS.

A3.1.2. Cross-trait assortative mating that is asymmetric with respect to sex

We now consider the case where the strength of assortative mating between two traits, as measured by their correlation coefficient across mating pairs, is not equal in the female-male and male-female directions. This is clearest in the case of an active mate preference exhibited by one sex for some phenotype exhibited by the other sex.

To study this case, we make several simplifying assumptions. First, we assume that the genetic bases of variation in the two traits are distinct: ${\alpha }_l\ne 0\iff {\beta }_l=0$. Second we assume that there is only one active direction of assortative mating: female trait 1 and male trait 2. That is, conditional on the mother’s breeding value for trait 1 and the father’s breeding value for trait 2, there is no correlation between the mother’s breeding value for trait 2 and the father’s breeding value for trait 1:

$$\mathrm{Cov}\left(G_2^{\mathrm{m}},G_1^{\mathrm{f}}\mid \{G_1^{\mathrm{m}},G_2^{\mathrm{f}}\}\right)=0.$$

Suppose that there is a constant correlation ${\rho }_G$ between mothers’ breeding values for trait 1 and fathers’ breeding values for trait 2:

$${\rho }_G=\frac{\mathrm{Cov}\left(G_1^{\mathrm{m}},G_2^{\mathrm{f}}\right)}{\sqrt{V_G^1{V}_G^2}}.$$ (A.53)

To study the genetic consequences of this assortment, we need to know the average bi-directional correlation among mates for traits 1 and 2 (Eq. A.29). Since traits 1 and 2 will come into a positive genetic correlation via assortative mating of female trait 1 and male trait 2, there will also be a positive covariance between mothers’ breeding values for trait 2 and fathers’ breeding values for trait 1, which we can express using the law of total covariance:

$$\begin{array}{l}\mathrm{Cov}\left(G_2^{\mathrm{m}},G_1^{\mathrm{f}}\right)={\mathrm{Cov}}_{\{G_1^{\mathrm{m}},G_2^{\mathrm{f}}\}}\left(\mathbb{E}\left[G_2^{\mathrm{m}}\mid \{G_1^{\mathrm{m}},G_2^{\mathrm{f}}\}\right],\mathbb{E}\left[G_1^{\mathrm{f}}\mid \{G_1^{\mathrm{m}},G_2^{\mathrm{f}}\}\right]\right)\\ +{\mathbb{E}}_{\}{G}_1^{\mathrm{m}},G_2^{\mathrm{f}}\}}\left[\mathrm{Cov}\left(G_2^{\mathrm{m}},G_1^{\mathrm{f}}\mid \{G_1^{\mathrm{m}},G_2^{\mathrm{f}}\}\right)\right]\\ ={\mathrm{Cov}}_{\{G_1^{\mathrm{m}},G_2^{\mathrm{f}}\}}\left(\mathbb{E}\left[G_2^{\mathrm{m}}\mid G_1^{\mathrm{m}}\right],\mathbb{E}\left[G_1^{\mathrm{f}}\mid G_2^{\mathrm{f}}\right]\right).\end{array}$$ (A.54)

If $G_1^{\mathrm{m}}$ and $G_2^{\mathrm{m}}$ are bivariate normal (more generally, if $G_2^{\mathrm{m}}=a+b{G}_1^{\mathrm{m}}+\epsilon$, with $\mathbb{E}\left[\epsilon \right]=\mathbb{E}\left[\epsilon G_1^{\mathrm{m}}\right]=0$), then

$$\begin{array}{l}\mathbb{E}\left[G_2^{\mathrm{m}}\mid G_1^{\mathrm{m}}\right]=\mathbb{E}\left[G_2^{\mathrm{m}}\right]+{\rho }_{\mathrm{m}1,\mathrm{m}2}\sqrt{V_G^2/V_G^1}\left(G_1^{\mathrm{m}}-\mathbb{E}\left[G_1^{\mathrm{m}}\right]\right)\\ =\mathbb{E}\left[G_2^{\mathrm{m}\mathrm{m}}\right]+{\rho }_{\mathrm{m}1,\mathrm{m}2}\left(G_1^{\mathrm{m}}-\mathbb{E}\left[G_1^{\mathrm{m}\mathrm{m}}\right]\right),\end{array}$$

where ${\rho }_{\mathrm{m}1,\mathrm{m}2}=\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\left(G_1^{\mathrm{m}},G_2^{\mathrm{m}}\right)$ is the genetic correlation between traits 1 and 2 in mothers, and where we have assumed that the two traits have equal variance. Similarly, if $G_1^{\mathrm{f}}$ and $G_2^{\mathrm{f}}$ are bivariate normal, then

$$\mathbb{E}\left[G_1^{\mathrm{f}}\mid G_2^{\mathrm{f}}\right]=\mathbb{E}\left[G_1^{\mathrm{f}}\right]+{\rho }_{\mathrm{f}\mathrm{i},\mathrm{f}2}\left(G_2^{\mathrm{f}}-\mathbb{E}\left[G_2^{\mathrm{f}}\right]\right).$$

Substituting these expressions into Eq. (A.54),

$$\mathrm{Cov}\left(G_2^{\mathrm{m}},G_1^{\mathrm{f}}\right)={\rho }_{\mathrm{m}1,\mathrm{m}2}{\rho }_{\mathrm{f}1,\mathrm{f}2}\mathrm{Cov}\left(G_1^{\mathrm{m}},G_2^{\mathrm{f}}\right).$$ (A.55)

But, in our case, ${\rho }_{\mathrm{m}1,\mathrm{m}2}={\rho }_{\mathrm{f}1,\mathrm{f}2}$, the common value of which we shall call ${\rho }_{12}$, and so the average bi-directional correlation is

$$\frac{\frac{1}{2}\left[\mathrm{Cov}\left(G_1^{\mathrm{m}},G_2^{\mathrm{f}}\right)+\mathrm{Cov}\left(G_2^{\mathrm{m}},G_1^{\mathrm{f}}\right)\right]}{\sqrt{V_G^1{V}_G^2}}=\frac{\frac{1}{2}\left(1+{\rho }_{12}^2\right)\mathrm{Cov}\left(G_1^{\mathrm{m}},G_2^{\mathrm{f}}\right)}{\sqrt{V_G^1{V}_G^2}}=\frac{{\rho }_G}{2}\left(1+{\rho }_{12}^2\right).$$ (A.56)

Given this value, the calculations of the effect of assortative mating on the weighted sums of cis- and trans-covariances, and thus on the additive genetic variance, proceed as for the case of symmetric assortative mating above.

Assuming the genetic bases of the two traits to be distinct, we may substitute the average bi-directional correlation, ${\rho }_G\left(1+{\rho }_{12}^2\right)/2$, into Eq. (A.48) to find

$${\rho }_G\left(1+{\rho }_{12}^2\right)=\frac{4\sum _{l\in L_1}\sum _{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right){\alpha }_l{\beta }_{l^{\prime }}}{\sqrt{V_G^1{V}_G^2}}.$$ (A.57)

But

$${\rho }_{12}=\frac{2{\sum }_{l\in L_1}{\sum }_{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right){\alpha }_l{\beta }_{l^{\prime }}}{\sqrt{V_G^1{V}_G^2}},$$

and so Eq. (A.57) can be written as the quadratic equation ${\rho }_G\left(1+{\rho }_{12}^2\right)=2{\rho }_{12}$, the relevant solution to which is ${\rho }_{12}=\left(1-\sqrt{1-{\rho }_G^2}\right)/\rho G$. If $\rho G$ is small, we use the first-order Taylor approximation $\sqrt{1-{\rho }_G^2}\approx 1-{\rho }_G^2/2$ to find

$$\frac{{\rho }_G}{2}\approx {\rho }_{12}=\frac{2\sum _{l\in L_1}\sum _{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right){\alpha }_l{\beta }_{l^{\prime }}}{\sqrt{V_G^1{V}_G^2}}\approx \frac{2\sum _{l\in L_1}\sum _{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right){\alpha }_l{\beta }_{l^{\prime }}}{\sqrt{V_g^1{V}_g^2}}.$$ (A.58)

In the particular scenario we have simulated in Fig. 2, $V_g^1=V_g^2,{\alpha }_l=1$ for all $l\in L_1$, and ${\beta }_l=1$ for all $l\in L_2$, so Eq. (A.58) further simplifies to

$$4\sum _{l\in L_1}\sum _{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right)=\rho V_g^1$$ (A.59)

In a population association study for trait 2 performed at a locus $l\in L_1$ (so that ${\beta }_l=0$),

$${\widehat{\beta }}_l={\beta }_l+\frac{2}{V_l}\sum _{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right){\beta }_{l^{\prime }}=\frac{2}{V_l}\sum _{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right).$$ (A.60)

Across loci in $L_1$, the average estimate is

$$\overline{{\widehat{\beta }}_l}=\frac{1}{\left|L_1\right|}\sum _{l\in L_1}\frac{2}{V_l}\sum _{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right).$$ (A.61)

In our simulations, $p_l\approx 1/2$ for all $l$ so that $V_l\approx 2p_l(1-p_l)=1/2$, and $\left|L_1\right|=\left|L_2\right|=500$, so $V_g^1=V_g^2=250$. Under this configuration,

$$\begin{array}{l}{\overline{\widehat{\beta }}}_l=\frac{1}{\left|L_1\right|}\sum _{l\in L_1}\frac{2}{V_l}\sum _{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right)=\frac{1}{500}\sum _{l\in L_1}\frac{2}{1/2}\sum _{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right)\\ =\frac{4}{500}\sum _{l\in L_1}\sum _{l^{\prime }\in L_2}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right)=\frac{{\rho }_G{V}_g^1}{500}={\rho }_G/2.\end{array}$$

The trait we simulated is genetic, with heritability 1, and so ${\rho }_G=\rho$, the phenotypic correlation among mates. We chose a strength of assortative mating of $\rho =0.2$, and so, in equilibrium, the average effect size estimate at non-causal loci should be approximately 0.1, which is indeed the case in Fig. 2.

Sex-asymmetric cross-trait assortative mating for traits with different genetic architectures.

For the case where the numbers of loci underlying traits 1 and 2 differ, and noting that the ‘effective’ correlation among mates in the sex-asymmetric case is approximately half that in the sex-symmetric case (Eq. A.58), we can perform a similar back-of-the-envelope calculation as in the sex-symmetric cross-trait assortative mating case above to find that, when effect sizes are constant across trait-1 loci and constant across trait-2 loci (though differing across traits 1 and 2), the effect size estimates at trait-1 (non-causal) loci in a trait-2 population GWAS is, on average, a fraction $\frac{h^2\rho }{2}\sqrt{\left|L_2\right|/\left|L_1\right|}$ of the estimates at trait-2 (causal) loci.

Thus, more generally, when the number of loci underlying trait 1 is small relative to the number of loci underlying trait 2, the distribution of magnitudes of effect size estimates at trait-1 loci in a trait-2 GWAS can overlap substantially with the distribution of magnitudes of effect size estimates at trait-2 loci (Fig. 4), causing variants at these non-causal trait-1 loci to show up as significant in the trait-2 GWAS.

A3.2. Population structure

In the model we have considered, with results displayed in Fig. 5, there are initially two isolated populations of equal size. The frequency of the focal variant at locus $l$ is $p_l^{\left(1\right)}$ in population 1 and $p_l^{\left(2\right)}$ in population 2, so that its overall frequency is $p_l=\left(p_l^{\left(1\right)}+p_l^{\left(2\right)}\right)/2$. A population GWAS at locus $\lambda$ returns an effect size estimate

$${\hat{\alpha }}_{\lambda }^{\text{pop}}=\frac{2}{V_{\lambda }}\sum _{l\in L} \left(D_{\lambda l}+{\tilde{D}}_{\lambda l}\right){\alpha }_l,$$

where $D_{\lambda l}$ and ${\tilde{D}}_{\lambda l}$ are calculated across both populations and are generally nonzero because of allele frequency differences between the two populations at loci $\lambda$ and $l$ (Nei and Li 1973). In our case,

$$V_{\lambda }=2p_{\lambda }\left(1-p_{\lambda }\right)\left(1+F_{\lambda }\right),$$

and

$$D_{\lambda l}={\tilde{D}}_{\lambda l}=\frac{1}{4}\left(p_{\lambda }^{\left(1\right)}-p_{\lambda }^{\left(2\right)}\right)\left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right),$$

so

$${\hat{\alpha }}_{\lambda }^{\mathrm{p}\mathrm{o}\mathrm{p}}=\frac{p_{\lambda }^{\left(1\right)}-p_{\lambda }^{\left(2\right)}}{2p_{\lambda }\left(1-p_{\lambda }\right)\left(1+F_{\lambda }\right)}\sum _{l\in L} \left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right)\alpha l.$$

Squaring this and multiplying by $2p_{\lambda }\left(1-p_{\lambda }\right)$,

$$2p_{\lambda }\left(1-p_{\lambda }\right){\left({\hat{\alpha }}_{\lambda }^{\text{pop}}\right)}^2=\frac{{\left(p_{\lambda }^{\left(1\right)}-p_{\lambda }^{\left(2\right)}\right)}^2}{2p_{\lambda }\left(1-p_{\lambda }\right){\left(1+F_{\lambda }\right)}^2}\left[\sum _{l\in L}{\left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right)}^2{\alpha }_l^2+\sum _{l\ne l^{\prime }}\left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right)\left(p_{l^{\prime }}^{\left(1\right)}-p_{l^{\prime }}^{\left(2\right)}\right){\alpha }_l{\alpha }_{l^{\prime }}\right].$$ (A.62)

Neutral allele frequency divergence.

If allele frequency divergence between the two populations is neutral, frequency changes at different loci are independent of one another and of effect sizes, so the second term in square brackets above is zero in expectation. In addition, because Hardy-Weinberg equilibrium obtains within each population, non-zero expected values of $F_{\lambda }$ derive only from allele frequency differences between the populations, so that $F_{\lambda }=F_{ST,\lambda }$ in expectation. Therefore,

$$\begin{array}{l}\mathbb{E}\left[2p_{\lambda }\left(1-p_{\lambda }\right){\left({\widehat{\alpha }}_{\lambda }^{\text{pop}}\right)}^2\right]=\frac{1}{{\left(1+F_{ST}\right)}^2}\mathbb{E}\left[\frac{{\left(p_{\lambda }^{(1)}-p_{\lambda }^{(2)}\right)}^2}{2p_{\lambda }\left(1-p_{\lambda }\right)}\right]|L|\mathbb{E}\left[{\left(p_l^{(1)}-p_l^{(2)}\right)}^2\right]\mathbb{E}\left[{\alpha }_l^2\right]\\ =\frac{1}{{\left(1+F_{ST}\right)}^2}\mathbb{E}\left[2F_{ST,\lambda }\right]|L|\mathbb{E}\left[2F_{ST,l}{H}_l\right]\mathbb{E}\left[{\alpha }_l^2\right]\\ \approx \frac{4|L|}{{\left(1+F_{ST}\right)}^2}{\left(\mathbb{E}\left[F_{ST,l}\right]\right)}^2\mathbb{E}\left[H_l\right]\mathbb{E}\left[{\alpha }_l^2\right]\\ =4|L|{\left(\frac{F_{ST}}{1+F_{ST}}\right)}^2\mathbb{E}\left[H_l\right]\mathbb{E}\left[{\alpha }_l^2\right],\end{array}$$

where $H_l=2p_l\left(1-p_l\right)$. If the ancestral allele frequency at $l$ was $p_l^{\mathrm{a}}$, then $\mathbb{E}\left[H_l\mid p_l^{\mathrm{a}}\right]=2p_l^{\mathrm{a}}\left(1-p_l^{\mathrm{a}}\right)\left(1-F_{ST,l}\right)$, and so $\mathbb{E}\left[H_l\right]$ is calculated using the law of iterated expectations by averaging this quantity over the ancestral distribution of allele frequencies: $\mathbb{E}\left[H_l\right]\approx \mathbb{E}\left[H_l^a\right]\left(1-F_{ST}\right)$, where $H_l^a=2p_l^a\left(1-p_l^a\right)$. So

$$\mathbb{E}[2p_{\lambda }\left(1-p_{\lambda }\right){\left({\hat{\alpha }}_{\lambda }^{\mathrm{p}\mathrm{o}\mathrm{p}}\right)}^2]\approx 4|L|{\left(\frac{F_{ST}}{1+F_{ST}}\right)}^2\left(1-F_{ST}\right)\mathbb{E}\left[H_l^{\mathrm{a}}\right]\mathbb{E}\left[{\alpha }_l^2\right].$$ (A.63)

Selection and phenotype-biased migration.

Above, in calculating the mean heterozygosity-weighted value of ${\left({\hat{\alpha }}_{\lambda }\right)}^2$ under neutral frequency divergence between populations, we assumed that in Eq. (A.62) the second term in the square brackets was zero, i.e., that the effect-size-signed population allele frequency difference was uncorrelated across loci. Howevever, when selection or phenotype-biased migration acts, this will no longer be true. For example, if higher genetic values of the trait were favoured in population 1 relative to population 2, then selection will on average have driven a mean shift such that $\mathbb{E}\left[\left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right){\alpha }_l\right]>0$. This in turn will drive systematic positive covariances between terms $\left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right)\alpha l$ and $\left(p_{l\prime }^{\left(1\right)}-p_{l\prime }^{\left(2\right)}\right){\alpha }_{l\prime }$, and as these covariances are summed over all pairs of loci in Eq. (A.62), the resulting inflation of the average squared effect size estimate (and other genome-wide summaries) could be quantitatively substantial.

More general population stratification.

Given a sample of $N$ individuals, the sample cis-LD between two markers $\lambda$ and $l$ can be written generally as

$$D_{\lambda l}=\frac{1}{N-1}\sum _{i=1}^N\left(\mathrm{\Delta }{g}_{i,\lambda }^{\text{m}}\mathrm{\Delta }{g}_{i,l}^{\text{m}}+\mathrm{\Delta }{g}_{i,\lambda }^{\text{p}}\mathrm{\Delta }{g}_{i,l}^{\text{p}}\right),$$ (A.64)

where $\mathrm{\Delta }{g}_{i,k}^{\mathrm{m}}$ and $\mathrm{\Delta }{g}_{i,k}^{\mathrm{P}}$ are the deviations of individual $i$ ‘s maternal and paternal focal allele count at locus $k$ from their mean frequencies. The trans-LD between $\lambda$ and $l$ is

$${\tilde{D}}_{\lambda l}=\frac{1}{N-1}\sum _{i=1}^N\left(\mathrm{\Delta }{g}_{i,\lambda }^{\text{m}}\mathrm{\Delta }{g}_{i,l}^{\text{p}}+\mathrm{\Delta }{g}_{i,l}^{\text{m}}\mathrm{\Delta }{g}_{i,\lambda }^{\text{p}}\right).$$ (A.65)

These cis- and trans-LD terms are equal only if

$$D_{\lambda l}-{\tilde{D}}_{\lambda l}=\frac{1}{N-1}\sum _{i=1}^N\left(\mathrm{\Delta }{g}_{i,\lambda }^{\text{m}}-\mathrm{\Delta }{g}_{i,\lambda }^{\text{p}}\right)\left(\mathrm{\Delta }{g}_{i,l}^{\text{m}}-\mathrm{\Delta }{g}_{i,l}^{\text{p}}\right)=0,$$ (A.66)

i.e., if the maternal and paternal alleles at the one locus are exchangeable with respect to deviations of the allelic state at the other locus.

We might often be concerned with stratification along some specific axis of variation in our sample. Call this axis $v$, with every individual having a value along $v$, with mean zero across individuals (for example, in our two population case above, the vector $v$ could be 1 for population 1 and −1 for population 2). The covariance of the maternal allele at locus $l$ with the vector $v$ is proportional to $a_l^{\mathrm{m}}\cdot v={\sum }_i a_{i,l}^{\mathrm{m}}{v}_i$. So the contribution of LD along this axis to the difference in cis- and trans-LD is

$$D_{\lambda l}^{\left(v\right)}-{\tilde{D}}_{\lambda l}^{\left(v\right)}=\left(\left(\mathrm{\Delta }{g}_{\lambda }^{\mathrm{m}}-\mathrm{\Delta }{g}_{\lambda }^{\mathrm{P}}\right)\cdot v\right)\left(\left(\mathrm{\Delta }{g}_l^{\mathrm{m}}-\mathrm{\Delta }{g}_l^{\mathrm{P}}\right)\cdot v\right),$$ (A.67)

which is zero only if the maternal and paternal genotypes at the two loci are exchangeable with respect to each other along the axis $v$.

A3.3. Admixture

Suppose that two previously isolated populations admix in proportions $A$ and $1-A$, with subsequent random mating in the admixed population. Following the notation in the Section A3.2 above, before admixture, the frequency of the focal variant at locus $l$ was $p_l^{\left(1\right)}$ in population 1 and $p_l^{\left(2\right)}$ in population 2, so that its overall frequency in the admixed population is $p_l=A{p}_l^{\left(1\right)}+(1-A)p_l^{\left(2\right)}$.

When the two populations admix, trans-LD between all pairs of loci disappears in expectation, owing to random mating in the admixed population: ${\tilde{D}}_{\lambda l}^t=0$ for any pairs of loci $\lambda$ and $l$ and for any number of generations $t$ after admixture. However, cis-associations between alleles that were more prevalent in one ancestral population than in the other will be retained as cis-LD in the admixed population until these associations are eroded by recombination. The initial degree of cis-LD between loci $\lambda$ and $l$ in the admixed population is

$$D_{\lambda l}^0=A\left(1-A\right)\left(p_{\lambda }^{\left(1\right)}-p_{\lambda }^{\left(2\right)}\right)\left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right).$$

When $t$ generations have elapsed since admixture, this cis-LD will have been eroded by recombination to

$$D_{\lambda l}^t=D_{\lambda l}^0{\left(1-c_{\lambda l}\right)}^t=A(1-A)\left(p_{\lambda }^{\left(1\right)}-p_{\lambda }^{\left(2\right)}\right)\left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right){\left(1-c_{\lambda l}\right)}^t,$$

where $c_{\lambda l}$ is the sex-averaged recombination rate between $\lambda$ and $l$. Therefore, $t$ generations after admixture, a population association study at $\lambda$ returns an effect size estimate

$${\hat{\alpha }}_{\lambda }^{\mathrm{p}\mathrm{o}\mathrm{p},t}=\frac{2}{V_{\lambda }}\sum _{l\in L} D_{\lambda l}^t{\alpha }_l=A(1-A)\frac{p_{\lambda }^{\left(1\right)}-p_{\lambda }^{\left(2\right)}}{p_{\lambda }\left(1-p_{\lambda }\right)}\sum _{l\in L} \left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right){\left(1-c_{\lambda l}\right)}^t{\alpha }_l,$$

while a sibling-based association study at $\lambda$ returns

$${\hat{\alpha }}_{\lambda }^{\mathrm{s}\mathrm{i}\mathrm{b},t}=\frac{2}{H_{\lambda }}\sum _{l\in L} \left(1-2c_{\lambda l}\right)D_{\lambda l}^t{\alpha }_l=A(1-A)\frac{p_{\lambda }^{\left(1\right)}-p_{\lambda }^{\left(2\right)}}{p_{\lambda }\left(1-p_{\lambda }\right)}\sum _{l\in L} \left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right){\left(1-c_{\lambda l}\right)}^t\left(1-2c_{\lambda l}\right){\alpha }_l,$$

where we have substituted $V_{\lambda }=H_{\lambda }=2p_{\lambda }\left(1-p_{\lambda }\right)$ owing to random mating in the admixed population.

Squaring the population estimate and multiplying by $2p_{\lambda }\left(1-p_{\lambda }\right)$,

$$\begin{array}{l}2p_{\lambda }\left(1-p_{\lambda }\right){({\widehat{\alpha }}_{\lambda }^{\text{pop},t})}^2=2A^2{(1-A)}^2\frac{{\left(p_{\lambda }^{(1)}-p_{\lambda }^{(2)}\right)}^2}{p_{\lambda }\left(1-p_{\lambda }\right)}[\sum _{l\in L}{\left(p_l^{(1)}-p_l^{(2)}\right)}^2{\left(1-c_{\lambda l}\right)}^{2t}{\alpha }_l^2\\ +\sum _{l\ne l^{\prime }}\left(p_l^{(1)}-p_l^{(2)}\right)\left(p_{l^{\prime }}^{(1)}-p_{l^{\prime }}^{(2)}\right){\left(1-c_{\lambda l}\right)}^t{\left(1-c_{\lambda l^{\prime }}\right)}^t{\alpha }_l{\alpha }_{l^{\prime }}],\end{array}$$ (A.68)

while the heterozygosity-weighted squared sibling effect size is

$$\begin{array}{l}2p_{\lambda }\left(1-p_{\lambda }\right){({\widehat{\alpha }}_{\lambda }^{\text{sib},t})}^2=2A^2{(1-A)}^2\frac{{\left(p_{\lambda }^{(1)}-p_{\lambda }^{(2)}\right)}^2}{p_{\lambda }\left(1-p_{\lambda }\right)}[\sum _{l\in L}{\left(p_l^{(1)}-p_l^{(2)}\right)}^2{\left(1-c_{\lambda l}\right)}^{2t}{\left(1-2c_{\lambda l}\right)}^2{\alpha }_l^2\\ +\sum _{l\ne l^{\prime }}\left(p_l^{(1)}-p_l^{(2)}\right)\left(p_{l^{\prime }}^{(1)}-p_{l^{\prime }}^{(2)}\right){\left(1-c_{\lambda l}\right)}^t{\left(1-c_{\lambda l^{\prime }}\right)}^t\left(1-2c_{\lambda l}\right)\left(1-2c_{\lambda l^{\prime }}\right){\alpha }_l{\alpha }_{l^{\prime }}].\end{array}$$ (A.69)

Neutral allele frequency divergence.

If allele frequency divergence between the two populations was neutral, then frequency changes at different loci are independent of one another, of effect sizes, and of recombination rates (assuming the loci are sufficiently far apart), so the second terms in square brackets in Eqs. (A.68) above is zero in expectation, so that

$$\begin{array}{l}\mathbb{E}\left[2p_{\lambda }\left(1-p_{\lambda }\right){({\hat{\alpha }}_{\lambda }^{\text{pop,}t})}^2\right]=4A^2(1-A{)}^2\mathbb{E}\left[\frac{{\left(p_{\lambda }^{\left(1\right)}-p_{\lambda }^{\left(2\right)}\right)}^2}{2p_{\lambda }\left(1-p_{\lambda }\right)}\right]|L|\mathbb{E}\left[{\left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right)}^2\right]\frac{}{(1-c{)}^{2t}}\mathbb{E}\left[{\alpha }_l^2\right]\\ =4A^2(1-A{)}^2\overline{(1-c{)}^{2t}}\mathbb{E}\left[2F_{ST,\lambda }\right]|L|\mathbb{E}\left[2F_{ST,l}{H}_l\right]\mathbb{E}\left[{\alpha }_l^2\right]\\ \approx 16A^2(1-A{)}^2\overline{(1-c{)}^{2t}}|L|F_{ST}^2\mathbb{E}\left[H_l\right]\mathbb{E}\left[{\alpha }_l^2\right],\end{array}$$

where $\overline{(1-c{)}^{2t}}$ is the average value of ${\left(1-c_l\right)}^{2t}$ taken across all pairs of loci $l,l^{\mathrm{\prime }}$.

Similarly, under drift in the ancestral populations, the average squared sibling-based effect size estimate can be simplified to

$$\mathbb{E}\left[2p_{\lambda }\left(1-p_{\lambda }\right){\left({\hat{\alpha }}_{\lambda }^{\mathrm{s}\mathrm{i}\mathrm{b},t}\right)}^2\right]\approx 16A^2(1-A{)}^2\overline{(1-c{)}^{2t}(1-2c{)}^2}|L|F_{ST}^2\mathbb{E}\left[H_l\right]\mathbb{E}\left[{\alpha }_l^2\right],$$

where $\overline{(1-c{)}^{2t}(1-2c{)}^2}$ is the average value of ${\left(1-c{l}^{\mathrm{\prime }}\right)}^{2t}\left(1-2cl{l}^{\mathrm{\prime }}\right)$ taken across all pairs of loci $l,l^{\mathrm{\prime }}$.

Selection and phenotype-biased migration.

As in the case of population structure, selection and phenotype-biased migration in the ancestral populations can drive systematic positive covariances between the terms $\left(p_l^{\left(1\right)}-p_l^{\left(2\right)}\right){\alpha }_l$ and $\left(p_{l^{\mathrm{\prime }}}^{\left(1\right)}-p_{l^{\mathrm{\prime }}}^{\left(2\right)}\right){\alpha }_{l^{\mathrm{\prime }}}$ in Eqs. (A.68) and (A.69) above, so that the second terms in square brackets in these equations do not cancel in expectation as they did under neutral divergence between the ancestral populations. Again, as these covariances are summed over all pairs of loci in Eqs. (A.68) and (A.69), the resulting inflation of the average squared effect size estimate and other genome-wide summaries could be substantial.

A3.4. Stabilizing selection

We consider the model of Bulmer (1971, ¹⁹⁷⁴), in which a very large number of loci contribute variation to a trait under stabilizing selection. We assume that the distribution of trait values is centered on the optimal value $Y^{\mathrm{*}}$, and that the relative fitness of an individual with trait value $Y$ is $\mathrm{e}\mathrm{x}\mathrm{p}(-{(Y-Y^{\mathrm{*}})}^2/2V_S)$, where $V_S$, the width or ‘variance’ of this gaussian selection function, governs the strength of stabilizing selection, with larger $V_S$ values implying weaker selection. Under this model, selection acts to reduce the phenotypic variation each generation; if the trait value is normally distributed with variance $V_P$, then selection reduces the within-generation phenotypic variance by an amount

$$\mathrm{\Delta }{V}_P=\frac{-V_p^2}{V_S+V_p}.$$ (A.70)

How much of this reduction carries over to the offspring generation then depends on the heritability of the trait.

Owing to the large number of loci in this model, the buildup of LD among them occurs on a faster timescale than the change in allele frequencies at individual loci. Assuming the loci to have equal effect sizes, Bulmer (1974) showed that the overall reduction in the phenotypic variance due to stabilizing selection, $d$, rapidly approaches a quasi-equilibrium value that approximately satisifes

$d^{\mathrm{*}}=\frac{1}{2}{h}^{\mathrm{*}4}\mathrm{\Delta }{V}_p^{\mathrm{*}}/{\bar{c}}_h,$ (A.71)

where $h^{\mathrm{*}2}$ is the heritability of the trait in this equilibrium and ${\bar{c}}_h$ is the harmonic mean of the recombination rates amongst all pairs of loci. On this rapid timescale, the reduction in variance is due to LD among the loci underlying the trait; in fact,

$$d=2{\alpha }^2\sum _{l\in L}\sum _{l^{\prime }\in L}{D}_{l{l}^{\prime }},$$ (A.72)

where $\alpha$ is the common per-locus effect size and $D_{l{l}^{\prime }}$ is defined with respect to the trait-increasing alleles at $l$ and $l^{\mathrm{\prime }}$. The individual linkage disequilibria $D_{l{l}^{\prime }}$, in expectation, are proportional to the inverse recombination rates $1/c_{l{l}^{\mathrm{\prime }}}$. Writing

$$2{\alpha }^2\sum _{l\in L}\sum _{l^{\prime }\in L}{D}_{l{l}^{\prime }}^{*}=d^{*}=\frac{1}{2}{h}^{*4}\mathrm{\Delta }{V}_P^{*}/{\overline{c}}_h=\frac{1}{2}{h}^{*4}\mathrm{\Delta }{V}_P^{*}\frac{\sum _l\sum _{l^{\prime }\ne l}1/c_{l{l}^{\prime }}}{\left(\begin{array}{c}|L|\\ 2\end{array}\right)},$$ (A.73)

where $\left(\begin{array}{c}|L|\\ 2\end{array}\right)=|L|(|L|-1)/2$ is the number of pairs of distinct loci in $L$, it is apparent that

$$\mathbb{E}[D_{l{l}^{\mathrm{\prime }}}^{\mathrm{*}}]=\frac{1}{4{\alpha }^2}{h}^{\mathrm{*}4}\mathrm{\Delta }{V}_p\frac{1/c_{l{l}^{\mathrm{\prime }}}}{\left(\begin{array}{c}\left|{}^L\right|\\ 2\end{array}\right)}.$$ (A.74)

Henceforth we deal only with equilibriuj quantities and therefore drop the star superscript for neatness. The phenotypic variance $V_p$ can be written $V_p=V_G+V_E=V_g+d+V_E$, where $V_G$ is the additive genetic variance, $V_g$ is the genic variance, and $V_E$ is the variance due to the environment. Eqs. (A.70) and (A.71), together with the definition of heritability $h^2=V_G/V_P$, define a quadratic equation in $d$:

$$(1+2H)d^2+2\left[\left(V_S+V_g+V_E\right){\bar{c}}_h+V_g\right]d+V_g^2=0.$$ (A.75)

Eq. (A.75) matches Eq. (10) in Bulmer (1974), with Bulmer’s parameter $c$ replaced by $1/2V_S$. For ease of reference in what follows, we write Eq. (A.75) in the standard form $a{d}^2+bd+c=0$. The roots are

$$d_{+,-}=\frac{-b\pm \sqrt{b^2-4ac}}{2a}=\frac{-\left[\left(V_S+V_g+V_E\right){\bar{c}}_h+V_g\right]\pm \sqrt{{\left[\left(V_S+V_g+V_E\right){\bar{c}}_h+V_g\right]}^2-(1+2H)V_g^2}}{1+2{\bar{c}}_h}.$$ (A.76)

To see which of these roots is the relevant one, we first note that the roots are both real, since the requirement for this is

$$\begin{array}{l}{\left[\left(V_S+V_g+V_E\right){\bar{c}}_h+V_g\right]}^2\ge \left(1+2{\bar{c}}_h\right)V_g^2\iff \left(V_S+V_g+V_E\right){\bar{c}}_h+V_g\ge \sqrt{1+2{\bar{c}}_h}{V}_g\\ \iff V_S+V_E\ge \frac{\sqrt{1+2{\bar{c}}_h}-1-{\bar{c}}_h}{{\bar{c}}_h}{V}_{g,}\end{array}$$

and $\sqrt{1+2{\bar{c}}_h}<1+{\bar{c}}_h$ for ${\bar{c}}_h>0$, while $V_S+V_E>0$. Furthermore, since $b>0$ and $4ac>0$, both roots are in fact negative, with $d_{-}<d_{+}<0$. Now note that

$$\begin{array}{l}2d_{-}<d_{+}+d_{-}=-\frac{b}{a}=-\frac{2\left[\left(V_S+V_g+V_E\right){\bar{c}}_h+V_g\right]}{1+2{\bar{c}}_h}\\ <-\frac{2\left[\left(V_g+V_g+V_E\right){\bar{c}}_h+V_g\right]}{1+2c_h}\text{(since}{V}_g<V_S)\\ <-\frac{2\left[\left(V_g+V_g\right){\bar{c}}_h+V_g\right]}{1+2{\bar{c}}_h}\text{(since}\left.V_E>0\right)\\ =-2V_g,\end{array}$$

i.e., $V_g+d_{-}<0$. But then if the relevant root were $d=d_{-},0\le V_G=V_g+d_{-}<0$, a contradiction. So the relevant root is in fact

$$d=d_{+}=\frac{-\left[\left(V_S+V_g+V_E\right){\bar{c}}_h+V_g\right]+\sqrt{{\left[\left(V_S+V_g+V_E\right){\bar{c}}_h+V_g\right]}^2-\left(1+2{\bar{c}}_h\right)V_g^2}}{1+2{\bar{c}}_h},$$ (A.77)

from which

$$-\frac{d}{V_g}=\frac{1-{\bar{c}}_h\left(\sqrt{1+2\left(1+\frac{1}{{\bar{c}}_h}\right)X+X^2}-\left(1+X\right)\right)}{1+2{\bar{c}}_h},$$ (A.78)

where $X=\frac{V_S+V_E}{V_g}$. Since, in the absence of selection, $V_G=V_g$, Eq. (A.78) gives the proportionate reduction in the additive genetic variance due to selection.

From Eq. (A.72), $d=2{\alpha }^2{\sum }_l {\sum }_{l^{\mathrm{\prime }}\ne l} D_{{ll}^{\mathrm{\prime }}}$, and, since $V_g={\sum }_l 2p_l\left(1-p_l\right){\alpha }^2={\alpha }^2\bar{H}|L|$, with $|L|$ the number of loci and $H$ the average heterozygosity across them, we have

$$\frac{d}{V_g}=\frac{2\sum _l \sum _{l^{\mathrm{\prime }}\ne l} D_{{ll}^{\mathrm{\prime }}}}{\bar{H}L}.$$ (A.79)

In a population association study performed at locus $l$, the effect size estimate is

$${\widehat{\alpha }}_l^{\text{pop}}={\alpha }_l+\frac{2}{2p_l\left(1-p_l\right)}\sum _{l^{\prime }\ne l}{D}_{l{l}^{\prime }}{\alpha }_{l^{\prime }}=\alpha \left(1+\frac{2}{2p_l\left(1-p_l\right)}\sum _{l^{\prime }\ne l}{D}_{l{l}^{\prime }}\right),$$ (A.80)

so that the proportionate error is

$$\frac{2}{2p_l\left(1-p_l\right)}\sum _{l^{\prime }\ne l}{D}_{l{l}^{\prime }}.$$ (A.81)

The mean proportionate error across loci is therefore

$$\frac{1}{|L|}\sum _{l\in L}\left(\frac{2}{2p_l\left(1-p_l\right)}\sum _{l^{\prime }\ne l}{D}_{l{l}^{\prime }}\right)\approx \frac{2\sum _l\sum _{l^{\prime }\ne l}{D}_{l{l}^{\prime }}}{\overline{H}|L|}=\frac{d}{V_g},$$ (A.82)

from Eq. (A.79), and assuming that the heterozygosities do not vary much across loci. That is, the average proportionate bias to effect size estimation that stabilizing selection induces is approximately equal to the proportionate reduction in the additive genetic variance, which is given in general form by Eq. (A.78).

In a within-family association study performed at locus $l$, the effect size estimate is

$${\widehat{\alpha }}_l^{\text{fam}}={\alpha }_l+\frac{2}{2p_l\left(1-p_l\right)}\sum _{l^{\prime }\ne l}\left(1-2c_{l{l}^{\prime }}\right)D_{l{l}^{\prime }}{\alpha }_{l^{\prime }}=\alpha \left(1+\frac{2}{2p_l\left(1-p_l\right)}\sum _{l^{\prime }\ne l}\left(1-2c_{l{l}^{\prime }}\right)D_{l{l}^{\prime }}\right),$$ (A.83)

so that the proportionate error is

$$\frac{2}{2p_l\left(1-p_l\right)}\sum _{l^{\prime }\ne l}\left(1-2c_{l{l}^{\prime }}\right)D_{l{l}^{\prime }}.$$ (A.84)

The mean proportionate error across loci is therefore

$$\begin{array}{l}\frac{1}{|L|}\sum _{l\in L}\left(\frac{2}{2p_l\left(1-p_l\right)}\sum _{l^{\prime }\ne l}\left(1-2c_{l{l}^{\prime }}\right)D_{l{l}^{\prime }}\right)\approx \frac{2{\sum }_l{\sum }_{l^{\prime }\ne l}\left(1-2c_{l{l}^{\prime }}\right)D_{l{l}^{\prime }}}{\overline{H}|L|}\\ \approx \frac{2{\sum }_l{\sum }_{l^{\prime }\ne l}\left(1-2c_{l{l}^{\prime }}\right)\frac{d{\overline{c}}_h}{2{\alpha }^2\left(\begin{array}{c}L\mid \\ 2\end{array}\right)c_{l{l}^{\prime }}}}{\overline{H}|L|}\\ =\frac{d{\overline{c}}_h}{{\alpha }^2\overline{H}|L|\left(\begin{array}{c}|L|\\ 2\end{array}\right)}\sum _l\sum _{l^{\prime }}\left(\frac{1}{c_{l{l}^{\prime }}}-2\right)\\ =\frac{d{\overline{c}}_h}{V_g\left(\begin{array}{c}|L|\\ 2\end{array}\right)}\left(\frac{\left(\begin{array}{c}|L|\\ 2\end{array}\right)}{{\overline{c}}_h}-2\left(\begin{array}{c}|L|\\ 2\end{array}\right)\right)\\ =\frac{d}{V_g}\left(1-2{\overline{c}}_h\right),\end{array}$$ (A.85)

where we have used Eq. (A.74) in the second line. Therefore, the mean error in the within-family GWAS is smaller in magnitude than that in a population GWAS by a factor $1-2{\bar{c}}_h$.

If ~1,000 loci underlie variation in the trait (and all contribute approximately the same variation), ${\bar{c}}_h\approx 0.4640$ in humans (see Methods), and so the average bias that stabilizing selection induces in within-family GWASs will be about $1-2{\bar{c}}_h\approx 7\mathrm{\%}$ that in population GWASs. If ~10,000 loci underlie variation in the trait, ${\bar{c}}_h\approx 0.4346$, and so the bias in within-family GWASs will be about $13\mathrm{\%}$ that in population GWASs.

The calculations above give the average proportionate bias to GWAS estimates in terms of the basic parameters of the model, $V_g,V_E,V_S$, and ${\bar{c}}_h$. Often, however, not all of these parameters will be measurable. For example, human height appears to be under stabilizing selection (Sanjak et al. 2018), is highly heritable, and this heritability is believed to be underlain largely by direct genetic effects (Lee et al. 2018). However, it is difficult to directly measure the genic variance in height $V_g$ because not all causal loci will be assayed in association studies—and, moreover, even if they were, effect size estimation at these causal loci would be biased by the genetic confounds that we have studied in this paper. However, the phenotypic variance in height $V_p$ can obviously be measured, and the heritability of height $h^2$ can also be measured using classical methods rather than effect size estimation in association studies. The strength of stabilizing selection on height can also be measured (Sanjak et al. 2018). From $V_p$ and $h^2$, the additive genetic variance $V_G$ can be estimated $\left(V_G=h^2{V}_p\right)$.

This example suggests that, in many applications, it might be useful to be able to estimate the equilibrium value of $d$ using $V_G$ (or $VP$), $V_E,V_S$, and ${\bar{c}}_h$, even though $V_G$ (and $V_P$), in the model we have considered, is a state variable influenced by the state variable of primary interest, $d$. This is straightforward: returing to our use of a star superscript to denote equilibrium values, if we treat $V_G$ and $V_P$ as their equilibrium values $V_G^{\mathrm{*}}$ and $V_P^{\mathrm{*}}$, Eq. (A.71) can be estimated directly, and also simplifies to

$$d^{\mathrm{*}}=-\frac{1}{2{\bar{c}}_h}\cdot \frac{V_G^{\mathrm{*}2}}{V_S+V_G^{\mathrm{*}}+V_E}=-\frac{1}{2{\bar{c}}_h}\cdot \frac{V_G^{\mathrm{*}2}}{V_S+V_p^{\mathrm{*}}}=\frac{1}{2{\bar{c}}_h}\cdot \frac{h^{\mathrm{*}4}{V}_p^{\mathrm{*}2}}{V_S+V_P^{\mathrm{*}}}.$$ (A.86)

The proportionate bias in a population GWAS, given by Eq. (A.82), can similarly be estimated from $h^2$, $V_p,V_S$, and ${\bar{c}}_h$, by first observing that

$$V_g=V_G^{\mathrm{*}}-d^{\mathrm{*}}=V_G^{\mathrm{*}}+\frac{1}{2{\bar{c}}_h}\frac{V_G^{\mathrm{*}2}}{V_P^{\mathrm{*}}}=V_G^{\mathrm{*}}\left(1+\frac{1}{2{\bar{c}}_h}\cdot \frac{V_G^{\mathrm{*}}}{V_S+V_P^{\mathrm{*}}}\right)$$

, so that Eq. (A.82) can be written

$$\frac{d^{*}}{V_g}=\frac{-\frac{1}{2{\overline{c}}_h}\cdot \frac{V_G^{*2}}{V_S+V_P^{*}}}{V_G^{*}\left(1+\frac{1}{2{\overline{c}}_h}\cdot \frac{V_G^{*}}{V_S+V_P^{*}}\right)}=\frac{-\frac{1}{2{\overline{c}}_h}\cdot \frac{V_G^{*}}{V_S+V_P^{*}}}{1+\frac{1}{2{\overline{c}}_h}\cdot \frac{V_G^{*}}{V_S+V_P^{*}}}=\frac{-\frac{1}{2{\overline{c}}_h}\cdot \frac{h^{*2}{V}_P^{*}}{V_S+V_P^{*}}}{1+\frac{1}{2{\overline{c}}_h}\cdot \frac{h^{*2}{V}^{*}}{V_S+V_P^{*}}}=-\frac{1}{2{\overline{c}}_h\left(\frac{1+V_S/V_P^{*}}{h^{*2}}\right)+1},$$ (A.87)

which reveals that the proportionate bias depends only on $c_h,h^{\mathrm{*}2}$ and the scaled inverse strength of selection, $V_S/V_p$.

From Eq. (A.85), the proportionate bias in a within-family GWAS is then approximately

$$\frac{d^{\mathrm{*}}}{V_g}\left(1-2{\bar{c}}_h\right)=-\frac{1-2{\bar{c}}_h}{2{\bar{c}}_h\left(\frac{1+V_S/V_P^{\mathrm{*}}}{h^{\mathrm{*}}}\right)+1}.$$ (A.88)

Stabilizing selection attenuates estimates of the strength of assortative mating based on cross-chromosome PGS correlations

Recently, the strength of assortative mating has been estimated based on measurement of the correlation of polygenic scores across distinct sets of chromosomes (e.g., Yengo et al. 2018; Yamamoto et al. 2023). Were assortative mating acting in isolation, such correlations would be due entirely to the positive cis- and trans-LDs among same-effect alleles created by assortative mating. Since stabilizing selection, acting in isolation, generates negative cis-LDs among same-effect alleles, it will attenuate the positive cis-LDs generated by assortative mating, and therefore reduce the correlation in PGSs among distinct sets of chromosomes, leading to underestimates of the strength of assortative mating if this effect is not taken into account.

To quantify this attenuation, we first calculate the strength of (positive) cross-chromosome LDs expected under assortative mating alone; then we calculate the strength of (negative) cross-chromosome LDs expected under stabilizing selection alone; then, assuming these LDs to be generated independently of one another—so that the LDs generated under the joint action of assortative mating and stabilizing selection are the sums of the LDs expected under these forces alone—we calculate how much stabilizing selection attenuates the correlation in PGSs across distinct sets of chromosomes.

Cross-chromosome correlations in PGSs.

The number of autosomes in the haploid set is $n$ (=22 in humans). Label the set of loci on chromosome $k$ that contribute variation to our trait of interest $L_k$; the overall set of loci underlying variation in the trait is $L=\left\{L_1,L_2,\dots ,L_k\right\}$. We divide the chromosomes into distinct sets $K_1$ and $K_2$ (e.g., $K_1$ could be the set of odd numbered chromosomes and $K_2$ the even). Let $L^{\left(1\right)}$ and $L^{\left(2\right)}$ be the sets of causal loci on the chromosomes in $K_1$ and $K_2$ respectively (i.e., $L^{\left(i\right)}={\cup }_{k\in K_1}{L}_k)$.

Suppose that we have accurately estimated effect sizes at all loci $l\in L$. For each individual, we then calculate a polygenic score for $K_1$ and for $K_2$:

$$P_1=\sum _{l\in L^{\left(1\right)}} g_l{\alpha }_l;P_2=\sum _{l^{\mathrm{\prime }}\in L^{\left(2\right)}} g_{l^{\mathrm{\prime }}}{\alpha }_{l^{\mathrm{\prime }}.}$$

We are interested in the correlation in the population between $P_1$ and $P_2$, and in particular, how this correlation is affected by assortative mating and stabilizing selection for the focal trait. The correlation can be written

$$\mathrm{Corr}\left(P_1,P_2\right)=\frac{\mathrm{Cov}\left(P_1,P_2\right)}{\mathrm{Var}\left(P_1\right)\mathrm{Var}\left(P_2\right)},$$

with

$$\begin{array}{l}\mathrm{Cov}\left(P_1,P_2\right)=\mathrm{Cov}\left(\sum _{l\in L^{(1)}}{g}_l{\alpha }_l,\sum _{l^{\prime }\in L^{(2)}}{g}_{l^{\prime }}{\alpha }_{l^{\prime }}\right)\\ =\sum _{l\in L^{(1)}}\sum _{l^{\prime }\in L^{(2)}}\mathrm{Cov}\left(g_l,g_{l^{\prime }}\right){\alpha }_l{\alpha }_{l^{\prime }}\\ =2\sum _{l\in L^{(1)}}\sum _{l^{\prime }\in L^{(2)}}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right){\alpha }_l{\alpha }_{l^{\prime }}.\end{array}$$ (A.89)

Since, to make progress in the case of stabilizing selection, we will assume effect sizes to be equal across loci, we make that assumption now, so that

$$\mathrm{Cov}\left(P_1,P_2\right)=2{\alpha }^2\sum _{l\in L^{(1)}}\sum _{l^{\prime }\in L^{(2)}}\left(D_{l{l}^{\prime }}+{\tilde{D}}_{l{l}^{\prime }}\right).$$ (A.90)

Since every pair of loci $\left(l,l^{\mathrm{\prime }}\right)$ across $L^{\left(1\right)}$ and $L^{\left(2\right)}$ are by definition unlinked, under many processes (including assortative mating and stabilizing selection), the values of $D_{l{l}^{\prime }}$ , and ${\tilde{D}}_{l{l}^{\prime }}$ will not differ much in expectation across locus pairs, in equilibrium. Therefore, we may approximate $D_{l{l}^{\prime }}=D^{*}$ and ${\tilde{D}}_{l{l}^{\prime }}={\tilde{D}}^{*}$ for all $l\in L^{\left(1\right)}$ and $l^{\mathrm{\prime }}\in L^{(2}$, so that Eq. (A.90) simplifies further:

$$\mathrm{Cov}\left(P_1,P_2\right)=2\left|L^{\left(1\right)}\right|\left|L^{\left(2\right)}\right|\left(D^{\mathrm{*}}+{\tilde{D}}^{\mathrm{*}}\right){\alpha }^2.$$ (A.91)

Assortative mating alone.

Under assortative mating with equal effect sizes across loci, in equilibrium, LDs are approximately equal across locus pairs, regardless of the recombination rate between them; moreover, cis- and trans-LDs are equal (see above). Therefore, to calculate $D^{\mathrm{*}}(={\tilde{D}}^{\mathrm{*}})$, we simply apportion the total LD given by Eq. (A.40) among individual locus pairs:

$$\begin{array}{l}\frac{h^2\rho }{1-h^2\rho }{V}_g\approx 4\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in L\\ l^{\prime }\ne l\end{array}}{D}_{l^{\prime }}^{*}{\alpha }_l{\alpha }_{l^{\prime }}=4|L|(|L|-1){\alpha }^2{D}^{*}\\ \Rightarrow D^{*}\approx \frac{\frac{h^2\rho }{1-h^2\rho }{V}_g}{4|L|(|L|-1){\alpha }^2}=\frac{\frac{h^2\rho }{1\mid -h^2\rho }|L|\overline{H}{\alpha }^2}{4|L|(|L|-1){\alpha }^2}=\frac{\frac{h^2\rho }{1-h^2\rho }\overline{H}}{4(|L|-1)}\approx \frac{1}{4}\cdot \frac{h^2\rho }{1-h^2\rho }\cdot \frac{\overline{H}}{|L|},\end{array}$$ (A.92)

when $|L|$ is large. Similarly,

$${\tilde{D}}^{*}\approx \frac{1}{4}\cdot \frac{h^2\rho }{1-h^2\rho }\cdot \frac{\overline{H}}{|L|},$$ (A.93)

so that the overall contribution of assortative mating to the covariance in Eq. (A.91) is proportional to

$$D^{*}+{\tilde{D}}^{*}\approx \frac{1}{2}\cdot \frac{h^2\rho }{1-h^2\rho }\cdot \frac{\overline{H}}{|L|}.$$ (A.94)

Stabilizing selection alone.

Under stabilizing selection, the total amount of negative cis-LD is given by Eq. (A.87):

$$2{\alpha }^2\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in l\\ l^{\prime }\ne l\end{array}}{D}_{l{l}^{\prime }}=d=-\frac{V_g}{2{\overline{c}}_h\left(\frac{1+V_S/V_P}{h^2}\right)+1},$$ (A.95)

where we have dropped the equilibrium ‘$\mathrm{*}$’ markers. This expression does not easily decompose into terms from individual locus pairs. However, if we assume that stabilizing selection is relatively weak $(V_S/V_P^{\mathrm{*}}\gg 1)$ and that the recombination process is such that the harmonic mean recombination rate ${\bar{c}}_h~1/2$ (as is the case in humans), Eq. (A.95) can be approximated by

$$2{\alpha }^2\sum _{l\in L}\sum _{\begin{array}{c}{l}^{\prime }\in l\\ l^{\prime }\ne l\end{array}}{D}_{l{l}^{\prime }}=d\approx -\frac{V_g}{2{\overline{c}}_h\left(\frac{1+V_S/V_P}{h^2}\right)}=-\frac{1}{2}\cdot \frac{h^2{V}_g}{1+V_S/V_P}\cdot \frac{1}{{\overline{c}}_h}=-\frac{1}{2}\cdot \frac{h^2{V}_g}{1+V_S/V_P}\cdot \frac{2\sum _{l,l^{\prime }}1/c_{l{l}^{\prime }}}{|L|(|L|-1)},$$

from which we infer that, in expectation,

$$2{\alpha }^2{D}_{l{l}^{\prime }}\approx -\frac{h^2{V}_g}{1+V_S/V_P}\cdot \frac{1/c_{l{l}^{\prime }}}{|L|(|L|-1)}.$$

Therefore, for unlinked $l$ and $l^{\mathrm{\prime }}\left(c_{l{l}^{\mathrm{\prime }}}=1/2\right)$, in expectation,

$$\begin{array}{l}{D}_{l{l}^{\prime }}\approx -\frac{1}{{\alpha }^2|L|(|L|-1)}\cdot \frac{h^2{V}_g}{1+V_S/V_P}=-\frac{\overline{H}}{{\alpha }^2\overline{H}|L|(|L|-1)}\cdot \frac{h^2{V}_g}{1+V_S/V_P}=-\frac{\overline{H}}{(|L|-1)V_g}\cdot \frac{h^2{V}_g}{1+V_S/V_P}\\ =-\frac{\overline{H}}{|L|-1}\cdot \frac{h^2}{1+V_S/V_P}\approx -\frac{\overline{H}}{|L|}\cdot \frac{h^2}{1+V_S/V_P}.\end{array}$$ (A.96)

Stabilizing selection does not systematically generate trans-LD, so, in expectation, ${\tilde{D}}_{l{l}^{\prime }}=0$. Therefore, under stabilizing selection alone, the contribution of an unlinked locus pair to the covariance in Eq. (A.91) is

$$D^{*}+{\tilde{D}}^{*}=D^{*}\approx -\frac{\overline{H}}{|L|}\cdot \frac{h^2}{1+V_S/V_P}.$$ (A.97)

How much does stabilizing selection attenuate the signal of assortative mating?

Comparing Eqs. (A.94) and (A.97), we find that the proportionate attenuation of assortative mating’s effect (in isolation) by the action of stabilizing selection is

$$\frac{-\frac{\overline{H}}{|L|}\cdot \frac{h^2}{1+V_S/V_P}}{\frac{1}{2}\cdot \frac{h^2\rho }{1-h^2\rho }\cdot \frac{\overline{H}}{|L|}}=\frac{-2}{1+V_S/V_P}\cdot \frac{1-h^2\rho }{\rho }.$$ (A.98)

For example, in the case of human height $\left(h^2~0.8\right)$, the signal of assortative mating (strength $\left.\rho ~0.25\right)$ is attenuated by stabilizing selection (strength $V_S/V_P~30$) by a proportionate amount of approximately $20\mathrm{\%}$. That is, one might measure by other means (e.g., the phenotypic correlation among mates, together with an estimate of the heritability of height) that the strength of assortative mating is $\rho =0.25$, but estimating this strength from cross-chromosome PGS correlations without accounting or correcting for stabilizing selection on height would yield $\hat{\rho }\approx \mathrm{0.2,20}\mathrm{\%}$ smaller than the true value.

A4. One-locus GxE

We study the phenotypic model in Eq. (22), with the phenotype of individual $i$ in family $f$ given by

$$Y_i=Y^{\mathrm{*}}+\left(\alpha +{\alpha }_f+{\alpha }_i\right)g_i+{ϵ}_f+{ϵ}_i,$$ (A.99)

where, across the population, $\mathbb{E}\left[{\alpha }_f\right]=\mathbb{E}\left[{\alpha }_i\right]=\mathbb{E}\left[{ϵ}_f\right]=\mathbb{E}\left[{ϵ}_i\right]=0$, and ${\alpha }_i,{ϵ}_f$, and ${ϵ}_i$ are all independent of $g_i$

Sibling GWAS.

Let $i$ and $j$ be siblings in family $f$, and define $\mathrm{\Delta }{Y}_f=Y_i-Y_j,\mathrm{\Delta }{g}_f=g_i-g_j$, and $\mathrm{\Delta }{ϵ}_f={ϵ}_i-{ϵ}_j$. A sibling association study returns an effect size estimate

$$\begin{array}{l}{\hat{\alpha }}^{\mathrm{s}\mathrm{i}\mathrm{b}}=\frac{\mathrm{Cov}\left(\mathrm{\Delta }{Y}_f,\mathrm{\Delta }{g}_f\right)}{\mathrm{Var}\left(\mathrm{\Delta }{g}_f\right)}=\frac{\mathrm{Cov}\left(\left(\alpha +{\alpha }_f\right)\mathrm{\Delta }{g}_f+\left({\alpha }_i{g}_i-{\alpha }_j{g}_j\right)+\mathrm{\Delta }{ϵ}_f,\mathrm{\Delta }{g}_f\right)}{\mathrm{Var}\left(\mathrm{\Delta }{g}_f\right)}\\ =\frac{\mathbb{E}\left[\left(\alpha +{\alpha }_f\right){\left(\mathrm{\Delta }{g}_f\right)}^2\right]+\mathbb{E}\left[\left({\alpha }_i{g}_i-{\alpha }_j{g}_j\right)\mathrm{\Delta }{g}_f\right]+\mathbb{E}\left[\mathrm{\Delta }{ϵ}_f\mathrm{\Delta }{g}_f\right]}{H},\end{array}$$

where $H$ is the fraction of parents who are heterozygous at the focal locus. Since ${\alpha }_i,{\alpha }_j,{ϵ}_i$, and ${ϵ}_j$ are genotype-independent perturbations, $\mathbb{E}\left[\left({\alpha }_i{g}_i-{\alpha }_j{g}_j\right)\mathrm{\Delta }{g}_f\right]=\mathbb{E}\left[\mathrm{\Delta }{ϵ}_f\mathrm{\Delta }{g}_f\right]=0$, and so

$$\hat{\alpha }=\frac{\mathbb{E}\left[\alpha {\left(\mathrm{\Delta }{g}_f\right)}^2\right]+\mathbb{E}\left[{\alpha }_f{\left(\mathrm{\Delta }{g}_f\right)}^2\right]}{H}=\alpha +\frac{\mathbb{E}\left[{\alpha }_f{\left(\mathrm{\Delta }{g}_f\right)}^2\right]}{H},$$ (A.100)

which deviates from $\alpha$ by an amount $\mathrm{E}\left[{\alpha }_f{\left(\mathrm{\Delta }{g}_f\right)}^2\right]/H$.

Let $\mathrm{\Delta }{g}_f^{\text{mat}}$ and $\mathrm{\Delta }{g}_f^{\text{pat}}$ be the difference in the genotypes of the siblings in family $f$ due to maternal and paternal transmission. Because of the independence of maternal and paternal transmission in a given family, the term additional to $\alpha$ in Eq. (A.100) can be split into $\mathbb{E}\left[{\alpha }_f{\left(\mathrm{\Delta }{g}_f^{\text{mat}}\right)}^2\right]/H$ and $\mathbb{E}\left[{\alpha }_f{\left(\mathrm{\Delta }{g}_f^{\text{pat}}\right)}^2\right]/H$, which we can analyze separately.

If the mother is heterozygous, then ${\left(\mathrm{\Delta }{g}_f^{\text{mat}}\right)}^2$ equals 1 with probability $1/2$ and 0 with probability $1/2$; if the mother is homozygous, then ${\left(\mathrm{\Delta }{g}_f^{\text{mat}}\right)}^2$ is 0. Therefore, denoting by $h^{\mathrm{m}}$ the event that the mother is heterozygous,

$$\frac{\mathbb{E}\left[{\alpha }_f{\left(\mathrm{\Delta }{g}_f^{\mathrm{m}\mathrm{a}\mathrm{t}}\right)}^2\right]}{H}=\frac{\frac{1}{2}\mathbb{E}\left[{\alpha }_f\mid h^{\mathrm{m}}\right]\mathrm{P}\mathrm{r}\mathrm{o}\mathrm{b}\left(h^{\mathrm{m}}\right)}{H}=\frac{1}{2}\mathbb{E}\left[{\alpha }_f\mid h^{\mathrm{m}}\right].$$

The same holds for paternal transmission, and so the deviation of the family-based estimate $\hat{\alpha }$ from $\alpha$ is

$$\hat{\alpha }-\alpha =\frac{\mathbb{E}\left[{\alpha }_f{\left(\mathrm{\Delta }{g}_f\right)}^2\right]}{H}=\mathbb{E}\left[{\alpha }_f\mid h\right].$$ (A.101)

That is, quite intuitively, if the average $\mathrm{G}\times \mathrm{E}$ effect ${\alpha }_f$ is different in the families of heterozygous parents than in the population as a whole, then limiting estimation to the offspring of heterozygous parents will be problematic.

Population GWAS.

Under the same one-locus model, a population association study returns an effect size estimate of

$$\begin{array}{l}{\hat{\alpha }}^{\mathrm{p}\mathrm{o}\mathrm{p}}=\frac{\mathrm{Cov}\left(Y_i,g_i\right)}{\mathrm{Var}\left(g_i\right)}=\frac{\mathrm{Cov}\left(\left(\alpha +{\alpha }_f+{\alpha }_i\right)g_i+{ϵ}_f+{ϵ}_i,g_i\right)}{\mathrm{Var}\left(g_i\right)}\\ =\alpha +\frac{\mathrm{Cov}\left({\alpha }_f{g}_i,g_i\right)}{\mathrm{Var}\left(g_i\right)}.\end{array}$$ (A.102)

We can immediately see from Eq. (A.102) that if the family environments are randomized across genotypes, such that ${\alpha }_f$ and $g_i$ are independent (implying $\mathrm{Cov}\left({\alpha }_f{g}_i,g_i\right)=0$), then the population estimate will coincide with $\alpha$.

To calculate the deviation of the population estimate from $\alpha$ in the general case, let $F$ be the inbreeding coefficient at the locus. Then $\mathrm{Var}\left(g_i\right)=2p(1-p)(1+F)$, where $p$ is the frequency of the focal variant, and the frequency of heterozygotes is $f_1=2p(1-p)(1-F)$ while the frequencies of the two homozygotes are $f_0=(1-p{)}^2+p(1-p)F$ (zero focal alleles) and $f_2=p^2+p(1-p)F$ (two focal alleles). The covariance term in Eq. (A.102) can then be written

$$\begin{array}{l}\mathrm{Cov}\left({\alpha }_f{g}_i,g_i\right)=\mathbb{E}\left[{\alpha }_f{g}_i^2\right]-\mathbb{E}\left[{\alpha }_f{g}_i\right]\mathbb{E}\left[g_i\right]=\mathbb{E}\left[{\alpha }_f{g}_i^2\right]-2p\mathbb{E}\left[{\alpha }_f{g}_i\right]\\ =\left(0\times \mathbb{E}\left[{\alpha }_f\mid g_i=0\right]f_0+1\times \mathbb{E}\left[{\alpha }_f\mid g_i=1\right]f_1+4\times \mathbb{E}\left[{\alpha }_f\mid g_i=2\right]f_2\right)\\ -2p\left(0\times \mathbb{E}\left[{\alpha }_f\mid g_i=0\right]f_0+1\times \mathbb{E}\left[{\alpha }_f\mid g_i=1\right]f_1+2\times \mathbb{E}\left[{\alpha }_f\mid g_i=2\right]f_2\right)\\ =\mathbb{E}\left[{\alpha }_f\mid g_i=1\right]f_1(1-2p)+4\mathbb{E}\left[{\alpha }_f\mid g_i=2\right]f_2(1-p)\\ =2\mathbb{E}\left[{\alpha }_f\mid g_i=1\right]p(1-p)(1-2p)(1-F)+4\mathbb{E}\left[{\alpha }_f\mid g_i=2\right]\left(p^2(1-p)+p(1-p{)}^{2}F\right).\end{array}$$

The deviation of the population-based estimate from $\alpha$ is therefore

$$\begin{gathered} {\hat{\alpha }}^{\mathrm{p}\mathrm{o}\mathrm{p}}-\alpha =\frac{\mathrm{Cov}\left({\alpha }_f{g}_i,g_i\right)}{\mathrm{Var}\left(g_i\right)} \\ =\frac{2\mathbb{E}\left[{\alpha }_f\mid g_i=1\right]p(1-p)(1-2p)(1-F)+4\mathbb{E}\left[{\alpha }_f\mid g_i=2\right]\left(p^2(1-p)+p(1-p{)}^{2}F\right)}{2p(1-p)(1+F)} \\ =\mathbb{E}\left[{\alpha }_f\mid g_i=1\right](1-2p)\frac{1-F}{1+F}+2\mathbb{E}\left[{\alpha }_f\mid g_i=2\right](p+(1-p)F)\frac{1}{1+F} \end{gathered}$$ (A.103)

$$\approx \mathbb{E}\left[{\alpha }_f\mid g_i=1\right](1-2p)(1-2F)+2\mathbb{E}\left[{\alpha }_f\mid g_i=2\right](p+(1-2p)F).$$ (A.104)

The approximation holds when $F$ is small.

An interesting special case is where homozygotes for the focal allele and heterozygotes have the same distribution of environments, so that $\mathbb{E}\left[{\alpha }_f\mid g_i=1\right]=\mathbb{E}\left[{\alpha }_f\mid g_i=2\right]=\mathbb{E}\left[{\alpha }_f\mid g_i>0\right]$. In this case, Eq. (A.103) simplifies to

$${\hat{\alpha }}^{\mathrm{p}\mathrm{o}\mathrm{p}}-\alpha =\mathbb{E}\left[{\alpha }_f\mid g_i>0\right],$$ (A.105)

which reveals that, if individuals who carry the focal allele tend to experience different environments to individuals who do not carry the focal allele, then the population GWAS estimate will deviate from the average effect under true randomization, $\alpha$. Moreover, in this case, if $\mathrm{E}\left[{\alpha }_f\mid g_i=1\right]$ and $\mathbb{E}\left[{\alpha }_f\mid h\right]$ are the same—that is, if the mean environment of heterozygous offspring is the same as that for heterozygous parents—then the sibling and population-based effect size estimates are the same.