The Ecology of Hybrid Incompatibilities

Abstract

Ecologically mediated selection against hybrids, caused by hybrid phenotypes fitting poorly into available niches, is typically viewed as distinct from selection caused by epistatic Dobzhansky–Muller hybrid incompatibilities. Here, we show how selection against transgressive phenotypes in hybrids manifests as incompatibility. After outlining our logic, we summarize current approaches for studying ecology-based selection on hybrids. We then quantitatively review QTL-mapping studies and find traits differing between parent taxa are typically polygenic. Next, we describe how verbal models of selection on hybrids translate to phenotypic and genetic fitness landscapes, highlighting emerging approaches for detecting polygenic incompatibilities. Finally, in a synthesis of published data, we report that trait transgression—and thus possibly extrinsic hybrid incompatibility in hybrids—escalates with the phenotypic divergence between parents. We discuss conceptual implications and conclude that studying the ecological basis of hybrid incompatibility will facilitate new discoveries about mechanisms of speciation.

Speciation occurs in an explicitly ecological context (Mayr 1942; Schluter 2000; Sobel et al. 2010; Germain et al. 2021). Hybrids, when they form, find themselves amid a tangled bank of biotic and abiotic stresses; in addition to completing development and gametogenesis, they must obtain food, avoid predators, mitigate stressors, and find mates, among other challenges. Although hybrid fitness is a key determinant of gene flow between lineages (Coyne and Orr 2004; Roux et al. 2016; Irwin 2020; Westram et al. 2022), research into the genetic basis of selection against hybrids—termed “postzygotic isolation”—has primarily focused on barriers affecting viability and gametogenesis that manifest in the laboratory (Reifová et al. 2023). As a result, less is known about the genetic basis of postzygotic isolating barriers caused by ecology (Schluter and Rieseberg 2022).

The primary genic model of postzygotic isolation is the (Bateson–)Dobzhansky–Muller Incompatibility (DMI) model (Bateson 1909; Dobzhansky 1937; Muller 1942). Until the mid-twentieth century, hybrid sterility and inviability were puzzling—how could evolutionary divergence between lineages proceed in a manner that leads to unfit hybrids? The DMI model posits that postzygotic isolation will typically involve substitutions at more than one locus. Low-fitness allele combinations (“DMIs”) come together for the first time in a hybrid, where they interact epistatically to reduce fitness (Fig. 1A; Orr 1995). To date, several studies have identified genes or loci underlying negative epistasis for fitness—where the fitness effects of an allele depend on the genotype at one or more other loci (Guerrero et al. 2017)—in hybrids (Presgraves et al. 2003; Bomblies et al. 2007; Phadnis and Orr 2009; Chae et al. 2014; Zuellig and Sweigart 2018; Powell et al. 2020; Moran et al. 2021). This empirical support has generated a strong consensus that DMIs are critical for postzygotic isolation.

Figure 1.

Conceptual overview of the Dobzhansky–Muller incompatibility (DMI) model and an example of a trait-based pollinator-mediated hybrid incompatibility. (A) A two-locus, recessive DMI. One of the two opposite-ancestry homozygous genotypes (aaBB) has low fitness. (B) Asymmetric selection against opposite-ancestry allele combinations in monkeyflowers, Mimulus lewisii, and Mimulus cardinalis. The two species have divergent floral colors and shapes (“wild-type”). The YUP locus encodes the flower color difference. Visitation by bumblebees decreases when the cardinalis allele is introgressed into the lewisii background, while flower color variation has little effect on hummingbird pollination in the cardinalis background. Patterns are a simplified presentation of those reported by Bradshaw and Schemske (2003).

In the field of speciation genetics, DMIs are typically discussed in the context of isolating barriers categorized as “intrinsic” and not those that are “extrinsic” (Price 2008). These categories relate to expectations about the role of the ecological niche and/or the degree of environment dependence in causing postzygotic isolation. Intrinsic barriers are often defined as being caused by “inherent fitness problems” (Coyne and Orr 2004) and typically interpreted to mean “unconditional … with respect to the environment” (Anderson et al. 2023). By contrast, extrinsic barriers are defined as hybrids having a poor phenotypic fit in available niches, and are (by definition) environment dependent (Schluter and Conte 2009; Nosil 2012). Exemplifying the convention treating DMIs as nonecological, Coyne and Orr (2004) introduce the DMI model within the section, “Genetic Modes of Intrinsic Postzygotic Isolation.” In some cases, the distinction between intrinsic and extrinsic barriers can be blurry because exogenous factors are sometimes required to reveal an apparently “intrinsic” incompatibility (Coyne and Orr 2004). In this article, we aim to clarify how negative epistasis for fitness between opposite-ancestry alleles can result from selection against quantitative hybrid phenotypes mediated by the ecological niche. It is well established that divergent ecological selection can generate DMIs that are not mediated by the niche (Moyle et al. 2012; Wright et al. 2013; Wilkinson et al. 2021). Here, we are not concerned with the question of whether or not negative epistasis for fitness is environment dependent (Fuller 2008), but rather with why alleles are incompatible in contemporary environments. In other words, what is the context linking hybrid genotypes to fitness?

In a largely separate line of inquiry from work on DMIs, the study of speciation by natural selection investigates how adaptive phenotypic evolution drives the evolution of reproductive isolation (Schluter 1996, 2000, 2001, 2009; Moyle 2004; Nosil 2012; Langerhans and Riesch 2013). Many studies have demonstrated that ecology mediates hybrid fitness (Hatfield and Schluter 1999; Rundle 2002; El Nagar and MacColl 2016; Best et al. 2017; Zhang et al. 2021; Thompson and Schluter 2022), and when parents are adapted to different habitats, convention holds that hybrids are selected against because their intermediate phenotypes “fall between” parental niches. While many studies have discovered potential ecological barrier loci using genome scans (Kulmuni et al. 2020), such approaches can say little about how selection acts on phenotypes. Because most work linking adaptive phenotypic divergence with hybrid incompatibility has focused on nonecological DMIs, we presently lack a conceptual foundation upon which to integrate phenotypic fitness landscapes with epistatic hybrid incompatibilities (Funk et al. 2006; Kulmuni and Westram 2017; Satokangas et al. 2020).

Here, we propose that conceptual barriers between the study of ecological speciation and DMIs can be overcome by considering how opposite-ancestry alleles combine to generate transgressive phenotypes in hybrids (Rieseberg et al. 1999). Just as the historic view of DMIs considers previously untested genotype combinations at multiple loci (Gavrilets 2004), the ecological view of DMIs considers phenotypes that occupy a non-parent-like and non-intermediate region of multivariate trait space—which we refer to as “transgressive” phenotypes. We first make the case that the DMI model should be interpreted more broadly than is convention. Next, we quantitatively review the genetic mapping literature (see Box 1 for an overview of quantitative components of this article), which supports the view that transgression-based incompatibilities will often be polygenic. After reviewing existing approaches for studying ecologically mediated DMIs, we develop intuition about how phenotypic transgression manifests as incompatibility on phenotypic fitness landscapes and via selection on genetic ancestry. Last, we conduct a quantitative review showing that the magnitude of transgression in hybrids increases with the magnitude of phenotypic divergence between their parents. We conclude that embracing the ecological basis of hybrid incompatibility will facilitate research into the importance of DMIs for speciation-by-selection and clarify new hypotheses about mechanisms of speciation.

BOX 1. OVERVIEW OF QUANTITATIVE VIGNETTES.

Summary

The article contains three quantitative vignettes that use various types of data and analysis methods. This box briefly summarizes the nature of the three vignettes and describes how their results can be reproduced. Full methods and results can be seen in the more substantive “Supplemental Material” document, located in the associated Dryad repository (doi:10.5061/dryad.qfttdz0mr).

Data and code accessibility

All data, metadata, simulation input files, and analysis code can be found on Dryad (doi:10.5061/dryad.qfttdz0mr). Data are all in the “data_dryad” folder, and scripts are in the “scripts” folder. Additional details are provided in readme files.

Minimally sufficient data sets underlying figures—containing just the plotted data with no metadata—can be found in data_dryad/figure_data/main/ for main text figures and data_dryad/figure_data/supp/ for supplementary figures. Each column in the data set has a prefix, “x_” or “y_” indicating its axis in the corresponding figure—some plots (e.g., histograms) only have a single variable. If grouping or coloring variables are used, these are indicated with intuitive prefixes, such as “grp_” or “clr_”.

Key details of quantitative vignettes

Vignette 1: Analysis of quantitative trait loci.

• Data source: Systematic review of studies performing QTL-mapping in hybrid crosses.

• Analysis: Qualitative reporting of patterns across studies.

• Key results are shown in Figure 2 of the main text.

• Additional methods and results can be found in Supplemental Material S1.

Vignette 2: Genetically explicit simulations of selection on hybrid incompatibilities

• Data source: individual-based simulations performed in Admix'em (Cui et al. 2016).

• Analysis: Qualitative reporting of patterns evident in the simulations.

• Key results are shown in Figure 4 of the main text.

• Additional methods and results can be found in Supplemental Material S2.

Vignette 3: Analysis of phenotypic variation in hybrids.

• Data source: Systematic review of hybrid phenotypes, as well as parental (nonhybrid) individuals, from controlled crosses.

• Analysis:

  • Processing of data to quantify transgression in hybrids and phenotypic divergence between parents, and regression analysis of their correlation.
  • Formal meta-analysis of transgression and phenotypic variation in hybrids.

• Key results are shown in Figure 5 of the main text.

• Additional methods and results can be found in Supplemental Material S3.

THE DOBZHANSKY–MULLER MODEL AS A GENERAL MECHANISM OF POSTZYGOTIC ISOLATION

Speciation researchers often make two assumptions about DMIs, which in order of their pervasiveness are that (1) DMIs are context-independent, and (2) DMIs typically involve few loci (i.e., “oligogenic” [Orr and Turelli 2001; Matute et al. 2010; Bank et al. 2012; Lindtke and Buerkle 2015; Li et al. 2022; Xiong and Mallet 2022; but see Orr 1995; Palmer and Feldman 2009; Livingstone et al. 2012]). Incompatibilities mediated by maladaptive transgressive phenotypes will likely be context dependent and underpinned by more than two loci—we address this latter claim quantitatively below. It is therefore worth briefly discussing why DMIs can have context-dependent effects on fitness and be underpinned by an arbitrarily large number of loci.

At its core, the DMI model has two essential components. First, assuming diploidy and bi-allelic loci, two or more loci must be involved in a DMI; second, alleles must interact epistatically wherein at least one hybrid allele combination confers lower fitness than the parental configurations (Fig. 1A). So long as selection against hybrid phenotypes results from epistatic selection against opposite-ancestry allele combinations, such interactions possess the two key components of a DMI (Muller 1942). We therefore view context-dependent and genetically complex DMIs to fit well under the definition of hybrid incompatibility. Importantly, while the fitness effects of DMIs are generally thought of as being static through space and time, the fitness landscapes underlying ecological DMIs are likely dynamic (Bordenstein and Drapeau 2001). Henceforth, we consider only the contemporary relative fitness landscape when discussing ecologically mediated DMIs.

The presently held assumptions about DMIs—that they are “intrinsic” and simple in architecture—likely established for myriad reasons. Truly unconditional hybrid inviability and/or sterility are sufficiently explained via DMIs, whereas DMIs are not necessary to explain conditional inviability and/or sterility. Moreover, DMIs are most amenable to study if they appear in the laboratory and are more genetically tractable if they involve few loci (Sweigart et al. 2006); similarly, two-locus DMIs are the simplest case for theorists to model (Orr 1995), and for outlining the basic principles of the DM model. Ecologically mediated and genetically complex DMIs do not lend themselves to precise inference: ecological mechanisms of selection on DMIs are typically speculative and derived from natural history knowledge (discussed below), and evidence for complex “intrinsic” DMIs, even in genetic model systems, is usually indirect (Cabot et al. 1994; Maside et al. 1998; Tao et al. 2003; Kao et al. 2010; Lollar et al. 2023). (Some empirical approaches commonly used to study incompatibilities, such as the analysis of introgression lines [Masly and Presgraves 2007; Moyle and Nakazato 2010], make no assumptions about the number of loci involved in DMIs and only ask whether the focal region is involved in a DMI.) However, although they might be difficult to observe and characterize, DMIs that manifest only in the field and involve many genes are still DMIs.

Ecology-mediated DMIs represent a case of environment-specific fitness epistasis. That is, the fitness effects of a particular allele depend on both the genotype at other loci and the environment where fitness is measured (Costanzo et al. 2021; Bakerlee et al. 2022), constituting a gene-by-gene-by-environment interaction (i.e., G × G × E). For example, Ono et al. (2017) evolved independent lines of yeast in a medium containing dilute fungicide and found that lines acquired different single-step mutations conferring tolerance. When these alleles—each beneficial as “single-mutants”—were brought together as “double-mutants” in dilute fungicide, they reduced fitness. However, in concentrated fungicide, many previously incompatible alleles became high-fitness double-mutants (Ono et al. 2017). Note that fitness epistasis implies something different than trait epistasis, which occurs when the phenotypic effect of one locus is affected by the genotype at other loci (Fierst and Hansen 2010). Fitness epistasis emerges naturally as a consequence of stabilizing or disruptive selection on one or more traits, or correlational selection between traits (see Whitlock et al. 1995 for discussion). Existing research suggests that fitness epistasis can be environment dependent (Nosil et al. 2020; Costanzo et al. 2021; Bank 2022), although this topic remains underexplored empirically (Domingo et al. 2019). Since trait variation is typically higher in recombinant hybrids (East 1916; also see below), and trait combinations are often disrupted in hybrids (Rieseberg 1995; Rosenthal et al. 2003), we expect that environment-specific fitness epistasis will commonly occur between opposite-ancestry alleles in hybrids.

A simple example of an ecologically mediated hybrid incompatibility between opposite-ancestry alleles—only apparent under field conditions—is evident in data from Bradshaw and Schemske (2003) (Fig. 1B). Bradshaw and Schemske (2003) quantified how an allele swap influenced pollinator visitation in Mimulus monkeyflowers. In this system, flowers attract and interact with a specific pollinator—bumblebees for pink-flowered Mimulus lewisii and hummingbirds for red-flowered Mimulus cardinalis. Bradshaw and Schemske (2003) reciprocally introgressed the YUP locus, which contains a large-effect flower color allele (Bradshaw et al. 1998; Schemske and Bradshaw 1999; Liang et al. 2023). The authors found that bumblebees largely ignored red lewisii-like flowers, whereas hummingbirds were little-deterred by pink cardinalis-like flowers (Bradshaw and Schemske 2003). The (recessive) M. cardinalis YUP allele is therefore incompatible within the lewisii genetic background, whereas the M. lewisii YUP allele is mostly compatible within the M. cardinalis genome, and this incompatibility is mediated by an ecological mechanism of selection: pollinator visitation. We use this simple example of a large-effect allele swap because it is intuitive, but below we primarily focus on more polygenic incompatibilities.

CURRENT APPROACHES FOR STUDYING ECOLOGICALLY MEDIATED HYBRID INCOMPATIBILITIES

Most work investigating how ecology mediates the fitness consequences of DMIs has approached the topic without a view to the ecological niche and phenotypic fitness landscapes. These studies generally either (1) compare the relative fitness of crosses between environments, or (2) observe a specific hybrid incompatibility phenotype affected by a selective mechanism absent from standard laboratory conditions; examples of both are described below. This work complements our goals because it supports the view that hybrid incompatibilities can have a critical ecological context—that is, it provides evidence for G × G × E interactions between opposite-ancestry alleles. However, because they do not consider phenotypic fitness landscapes, such approaches for studying ecologically mediated fitness epistasis are conceptually distinct from mechanisms of speciation-by-natural-selection. Therefore, we subsequently focus on how “ordinary” (Orr 2001) quantitative traits, which are expected to be under species- or population-specific stabilizing selection (e.g., bill shape in birds) rather than universal directional selection (e.g., pollen viability in plants), can underlie fitness epistasis in hybrids. Phenotype values for ordinary traits do not inherently transmit information about how they affect fitness—a particular bill shape can be adaptive or maladaptive depending on how it is used. By contrast, phenotype values for fitness traits, such as the fraction of gametes that are viable, do transmit information about fitness.

Environmental Effects on Relative Hybrid Fitness

If incompatibility between opposite-ancestry alleles is only exposed under particular ecological/environmental conditions, the relative fitness of incompatibility-afflicted hybrids should be reduced in the presence of the mediating ecological mechanism(s). In an interspecific cross in the plant genus, Silene, hybrid breakdown—a reduction in fitness in the F₂ compared to the average of parents and the F₁ (Edmands 2002)—is observed in the field (Favre et al. 2017; Karrenberg et al. 2019; Gramlich et al. 2022) but not in a relatively benign botanical garden (Liu and Karrenberg 2018). Critically, hybrid breakdown is direct evidence of negative epistasis for fitness (Whitlock et al. 1995; Fenster et al. 1997) and therefore DMIs. Similar patterns of ecological mechanisms mediating fitness epistasis in hybrids have been observed in several systems (Fitzpatrick and Shaffer 2004; Campbell and Waser 2007; Sambatti et al. 2008; Schumer et al. 2014; Schumer and Brandvain 2016; Kulmuni et al. 2020; Thompson and Schluter 2022). Studies manipulating the hypothesized mechanism(s) of selection in the field are needed to confirm the role of niche-based selection—such approaches will be highly informative for understanding the causes of postzygotic isolation (Schluter 1998).

Many studies have observed that the fitness effects of apparently “intrinsic” DMIs are affected by the experimental environment (Bordenstein and Drapeau 2001). For example, the magnitude of hybrid inviability often varies markedly with temperature (Muller 1942; Willett and Burton 2003; Bomblies et al. 2007; Demuth and Wade 2007; Willett 2011; Bundus et al. 2015; Miller and Matute 2017), and even nutrient conditions (Grant 1953). Most studies documenting such phenomena have attributed their findings to broad mechanisms such as “stress,” which is typically interpreted to mean that “hybrids suffer inherent fitness problems that are exacerbated (in stressful environments)” (Coyne and Orr 2004, p. 250). As such, a stressful (or “harsh”) environment might be one where the rank-order of fitness of hybrid cross types (e.g., F₂ < F₁) is the same as in a benign environment, but where a decline in overall survival probability (i.e., reduction in absolute fitness) renders the difference detectable. For instance, if F₂s have ½ the survival probability of F₁s due to DMIs, this difference will be highly detectable if many individuals experience mortality and relatively undetectable if few do. This field of research usefully highlights that experimental conditions can greatly influence our conclusions about the presence and/or strength of incompatibility.

Hybrid Incompatibility Phenotypes under Ecological Selection

Some studies report DMI phenotypes with indirect, ecologically mediated links to viability. For example, Fitzpatrick (2008a) and Powell et al. (2020) inferred that particular combinations of opposite-ancestry alleles reduced burst-speed performance—a predator-avoidance behavior—in hybrid salamanders and fish, respectively. These incompatibilities are explicitly mediated by ecology because their fitness consequences are fully evident only in the presence of predators. Other studies have identified DMI phenotypes affecting cognition. Many Drosophila hybrids fail to locate a nearby food source (Turissini et al. 2017), and hybrid chickadees have relatively poor learning and memory compared to parents, which could reduce their ability to relocate stored food (McQuillan et al. 2018; Rice and McQuillan 2018); in both cases, negative fitness epistasis could be ameliorated by providing food to hybrids—similar to how cytoplasmic incompatibility can be ameliorated with antibiotics (Turelli and Hoffmann 1991; Bordenstein et al. 2001). Other studies have found that hybrids are more susceptible to parasites and herbivores, which are only found in the field (Sage et al. 1986; Strauss 1994). Other studies have found evidence of disrupted mating traits in hybrids (Buckley 1969), which would only be evident if hybrids are subject to realistic sexual selection. Clearly, some allele combinations are only incompatible when exposed to selection in complex natural environments.

THE GENETIC ARCHITECTURE OF PHENOTYPIC VARIATION IN HYBRIDS

DMIs are typically conceived of as interactions between a small number of opposite-ancestry alleles of individually large effects (Maheshwari and Barbash 2011; Li et al. 2013). When opposite-ancestry alleles are incompatible because they produce low-fitness phenotypes in hybrids, the genetic architecture of phenotypic divergence between parental lineages determines the genetic architecture of postzygotic isolation (Yamaguchi and Otto 2020). While many studies have documented large-effect alleles underlying phenotypic divergence (Schemske and Bradshaw 1999; Colosimo et al. 2005; Protas et al. 2006; Chan et al. 2010), others suggest that smaller-effect alleles should predominate (Orr 1998; Rockman 2012; Barton 2022). Reviews of the quantitative trait locus (QTL) mapping literature suggest most traits are underpinned by QTL of small-to-moderate effects but these studies tend to be restricted to a single taxon and/or include domesticated or laboratory populations (Louthan and Kay 2011; Hall et al. 2016; Peichel and Marques 2017). It is therefore unclear what general patterns about the genetic architecture of phenotypic variation in hybrids might emerge when considering a taxonomically broad set of lineages that diverged under natural conditions.

To establish generalities about the genetic architecture of phenotypic variation in hybrids, we conducted a systematic review of studies that mapped QTL in hybrid crosses (Supplemental Material S1; 91 studies and 2676 QTL). Crosses were either between different species (interspecific), or between different populations of the same species (intraspecific). Our effect size metric is “proportion (or percent) of phenotypic variance explained” by a QTL, and mean effect size did not differ between intraspecific and interspecific crosses (mixed model P = 0.64). The median broad-sense heritability (estimated by few studies, typically using ratios of genetic and environmental variation) was 0.57, indicating that most phenotypic variation is heritable in these studies.

We found that individual QTL explained a median of 8.3% of the phenotypic variation (Fig. 2A), and the median total explained phenotypic variation for each trait was 22.8% (Fig. 2B). Considering only the single largest-effect QTL within each study, the median variance explained was 30.8%. This suggests that while detected QTL have reasonably large effects and most published studies detect at least one QTL of very large effect, the majority of QTL went undetected. These results imply that phenotypic variation in hybrids is polygenic; note that detected QTL effect sizes are often overestimates (Beavis 1998; Xu 2003). As a result, selection against transgressive hybrid phenotypes will typically involve many loci with modest phenotypic effects. We expect that the most productive approaches for studying ecologically mediated fitness epistasis in hybrids will therefore recognize that hybrids vary continuously in the expression of incompatible phenotypes—that is, by degree and not kind. Given the challenges of detecting additive phenotypic effects of most QTL, it is reasonable to assume that individual epistatic fitness effects of QTL will be even more difficult to resolve. Therefore, we expect that genetic approaches for studying ecologically mediated DMIs will generally not seek to resolve genetic interaction networks but rather aim to identify broader patterns.

Figure 2.

A systematic review of 91 studies reveals a polygenic genetic architecture of phenotypic variation in hybrids. (A) Distribution of effect sizes (proportion variance explained [PVE]) for all QTL (n = 2676; median = 0.083). (B) Total explained phenotypic variance per trait (summed PVE for each trait; n = 1141; median = 0.228). Values that would sum > 1 are shown as 1 (see Supplemental Material S1 for additional details).

HYBRID INCOMPATIBILITIES ON PHENOTYPIC AND GENOTYPIC FITNESS LANDSCAPES

In this section, we bridge concepts from research into DMIs with those from speciation-by-natural-selection (Langerhans and Riesch 2013) by examining hybrids on phenotypic fitness landscapes (Fragata et al. 2019) and considering their underlying genetics (Schneemann et al. 2020; De Sanctis et al. 2023). We first build intuition by focusing on a scenario where hybridizing populations have each adapted to the same phenotypic optimum (Anderson and Weir 2022), referred to as “mutation-order” speciation (Schluter 2009). We then address the scenario where nonhybrid parent lineages are adapted to different phenotypic optima because of divergent natural selection—known as “ecological” speciation (Schluter 2001). In both scenarios, selection against hybrids occurs because hybrid phenotypes are poorly suited to the available niche(s) (Nosil 2012), and DMIs manifest as low-fitness transgressive phenotypes in hybrids.

Our working model follows the assumptions of Fisher's (1930) geometric model. In the geometric model, an individual's fitness is determined by the Euclidean distance between its phenotype—which can have any number of traits—and a phenotypic fitness optimum (see Schneemann et al. 2023 for further discussion and quantitative exploration). We assume that hybrid phenotypes are underpinned by additive genetic variation and that all alleles are fixed in parents. We also assume that the only niches available are those of the cross parents, and do not consider novel adaptive niches (Rieseberg et al. 2007; Dittrich-Reed and Fitzpatrick 2013). Although nonadditive phenotype expression, likely due to dominance, is common in F₁ hybrids (Thompson et al. 2021), the additive model nevertheless makes informative, testable, and increasingly well-supported (discussed below) predictions about hybrid phenotypes and genotypes; how these predictions are affected by nonadditive phenotype expression has been explored elsewhere (Barton and Gale 1993; Turelli and Orr 2000; Simon et al. 2018; Schneemann et al. 2022). Under additivity, F₁ hybrids are phenotypically intermediate between parents and lack phenotypic variation. F₂ hybrids, formed by intercrossing F₁s, exhibit segregation variance that is proportional to the magnitude of genetic differences accumulated between parents (Thompson et al. 2019; Thompson 2020). Investigations of hybridization after “system drift” (True and Haag 2001) have explored similar concepts (Schiffman and Ralph 2022).

Critically, Fisher's geometric model and models of pairwise (as in Fig. 1A) incompatibilities make identical predictions about cross mean fitness and selection on ancestry under most biologically plausible parameter values (Simon et al. 2018). Specifically, both models predict lower fitness of the F₂ compared to the F₁ and nonhybrids—hybrid breakdown—and predict identical patterns of selection on genetic ancestry, which we discuss below. Thus, although phenotypes are additive, there can still be substantial epistasis for fitness.

Ecological Hybrid Incompatibility under Parallel Selection

The logic of ecological DMIs—hybrid incompatibilities where the fitness consequences are mediated by ecological agents of selection on phenotypic fitness landscapes—is simplest when considering a single phenotypic optimum and trait (Fig. 3A). Under parallel natural selection, DMIs can accumulate because of populations fixing alternative alleles by chance (Schluter 2009; Unckless and Orr 2009). While originally formulated in terms of intrinsic and oligogenic DMIs, such “mutation-order” speciation is easily interpreted via a polygenic trait on a phenotypic fitness landscape (Barton 2001).

Figure 3.

Conceptual overview of mechanisms of extrinsic postzygotic isolation. Each panel contains the implied phenotype distributions of genetic crosses, phenotypic mechanism of selection, expected cross fitness, and expected pattern of selection on genomic ancestry (from left to right). Formal models underlying genetic fitness landscapes are described in Supplemental Material S4; landscapes here are qualitative and meant to highlight the differences between scenarios. (A) Under mutation-order speciation, parent populations and F₁ hybrids have the same mean and variance for a focal trait (trait axis 1). If parent populations used the same alleles for adaptation, segregation variance will not be observed in hybrids—the parent genomes are fully compatible. However, if parent populations used different loci for adaptation to the same phenotypic optimum (mutation-order speciation), segregation variance in the F₂ will cause hybrid breakdown—the parent genomes are, to some degree, incompatible. In F₂ hybrids, selection acts against individuals with low ancestry heterozygosity relative to their ancestry proportion. (B) Under ecological speciation, parents are adapted to different optima. A one-axis view, where hybrids “fall between niches,” does not predict hybrid breakdown nor selection on ancestry heterozygosity, but rather selection on ancestry proportion. “Best” environment implies that individual hybrids are measured in the environment to which their phenotype is better suited. (C) Fitness epistasis, causing hybrid breakdown, emerges under ecological speciation when considering two or more axes. When recombinant hybrids have transgressive phenotypes, hybrid breakdown results and selection acts both on ancestry proportion and ancestry heterozygosity in the F₂.

Consider an ancestral population that splits into two derived populations that each undergo adaptation for increased body size. Under additivity, F₁ hybrids and parental populations have similar phenotype means and variances and high fitness. F₂ hybrids will resemble F₁s if parents used the same alleles for adaptation, but will have more variable phenotypes if parents used different alleles (Thompson et al. 2019). In the latter case, some F₂s will be too large (“overshooting”) and some will be too small (“undershooting”) relative to the optimum. Because selection on body size is imposed by the niche, and maladaptive phenotypes result from the inheritance of opposite-ancestry alleles at different genetic loci, this represents a case of ecologically mediated negative fitness epistasis between opposite-ancestry alleles. This genetic variation in F₂s would generate hybrid breakdown—a phenomenon caused by DMIs (Turelli and Orr 2000; Fishman and Willis 2001).

Phenotypic models of selection can have clear genetic underpinnings. When discussing genetic fitness landscapes in hybrids, we favor a continuous summary of ancestry along two axes (Fitzpatrick 2012; Simon et al. 2018): (1) ancestry proportion, and (2) ancestry heterozygosity. Ancestry proportion is the fraction of a hybrid's genome that is inherited from either one of the parent taxa. Ancestry heterozygosity is the fraction of a hybrid's genome that is heterozygous for ancestry from both parent taxa. We prefer these terms and recommend their adoption because alternatives for ancestry proportion—simply “ancestry” (Fitzpatrick 2012) or “hybrid index” (Buerkle 2005)—are either insufficiently precise (“ancestry”) or often used for phenotypic classification of hybrids (“hybrid index” [Wang et al. 2019]). Similarly, alternatives for ancestry heterozygosity—simply “heterozygosity” (Simon et al. 2018), or with preceding terms such as “interclass” or “interspecific” heterozygosity (Fitzpatrick 2012; Larson et al. 2013)—lack precision or are overly restrictive for a specific taxonomy. The terms “ancestry proportion” and “ancestry heterozygosity” are precise, generally applicable, and clearly related by their invocation of “ancestry.”

Selection against F₂ hybrids with transgressive phenotypes has predictable consequences with respect to ancestry proportion and ancestry heterozygosity (Fig. 3A). Specifically, the most transgressive individuals will generally have low ancestry heterozygosity relative to their ancestry proportion (at QTL and linked regions). As a result, the model predicts directional selection for increased ancestry heterozygosity at such loci (Simon et al. 2018). This example, conceptually similar to the empirical results of Ono et al. (2017), described above, has been described theoretically (Barton 1989; Slatkin and Lande 1994), and its predictions hold for any number of (uncorrelated) traits (Chevin et al. 2014; Yamaguchi and Otto 2020).

Ecological Hybrid Incompatibility under Divergent Selection

When speciation is caused by divergent natural selection, parent populations occupy different phenotypic optima (Schluter 2001; Nosil et al. 2002). The mechanism often understood to underlie selection against hybrid phenotypes when parents are ecologically divergent is that their intermediate phenotypes “fall between” the niches of parents (Hatfield and Schluter 1999; Schluter 2000, 2001; Rundle and Whitlock 2001; Nosil 2012), and robust tests documenting this mechanism have been completed in diverse systems (Rundle 2002; Campbell et al. 2008; Egan and Funk 2009; Kuwajima et al. 2010; Richards et al. 2016; Soudi et al. 2016; Bendall et al. 2017).

Like the example of parallel selection discussed above, ecological speciation is generally seen one-dimensionally: via a single “axis of divergence” (Fig. 3B; Thibert-Plante and Hendry 2009). This axis might be a single trait, or a combination of traits that “load” onto a single axis (Nosil 2012). Such selection is epistatic because the fitness consequences of any allele substitution depend on the genetic background (if hybrids can access the niche where their phenotype is better suited). However, this epistasis does not generate patterns expected of hybrid incompatibilities. In particular, hybrid breakdown need not occur because F₂ hybrids deviating in “bad” directions are balanced by hybrids deviating in “good” directions. If we grant that hybrids can optimize their fitness given their phenotype by “choosing” the best habitat, F₂s might even show improved mean fitness compared to F₁s. For instance, consider a scenario where selection favors large and small body sizes and acts against intermediate values. F₁ hybrids will be intermediate and have poor fitness. F₂ hybrids will exhibit phenotypic variation, but all deviations will make the F₂ either smaller- or larger-bodied, and the worst F₂ will be F₁-like. Moreover, selection acts only against intermediate values of ancestry proportion and not against individuals with more opposite-ancestry homozygous loci (Fig. 3B; Gow et al. 2007; Taylor et al. 2012).

Negative epistasis for fitness between opposite-ancestry alleles emerges when the ecological speciation model is extended into two or more dimensions (e.g., traits or PC axes) (Fig. 3C). For Pythagorean reasons, selection acts against opposite-ancestry trait combinations more than intermediacy. Consider a simple two trait system, [z₁, z₂], with each trait governed by a single, additive, bi-allelic locus, A/a (affecting z₁) and B/b (affecting z₂). Non-hybrid parents—genotypes aabb and AABB—have trait values [0, 0] and [1, 1], respectively. An F₂ hybrid with genotype AaBb has an intermediate phenotype of [0.5, 0.5], and its distance (d) from either parental phenotype is $d=\sqrt{{0.5}^2+{0.5}^2}=\sqrt{0.5}\approx 0.71$. By contrast, an F₂ hybrid with genotype AAbb will have a transgressive phenotype of [1, 0] and a distance of $d=\sqrt{1^2+0^2}=\sqrt{1}=1$. We define the latter phenotype as transgressive because it occupies an area of phenotype space that is neither parent-like or geometrically intermediate (Lamichhaney et al. 2017). Therefore, although the two hypothetical F₂ hybrids have identical ancestry proportions (both 0.5), the hybrid with entirely homozygous ancestry has lower fitness (greater distance to optimum) than the relatively heterozygous hybrid.

In addition to novel combinations of divergent traits, and like the “mutation-order” scenario, hybrids between divergent parents can exhibit transgression in traits that do not differ between parents (Thompson 2020; Schiffman and Ralph 2022). Barton (2001) investigated such selection in a model where divergent selection acted on one trait, while nine traits were under stabilizing selection favoring the ancestral trait value. Because populations fixed alternative alleles with deleterious pleiotropy and compensatory mutations, recombinant hybrids had substantial segregating phenotypic variation (Rieseberg et al. 2003) in the traits where parents did not differ. As with transgression resulting from novel combinations of parent-like traits, transgression in traits under stabilizing selection results from hybrids being homozygous for opposite-ancestry alleles at different loci. Because variation in F₂ hybrids will primarily occur along axes other than the primary axis of divergence, some hybrid breakdown is expected. With (molecular) genotype data, one should expect to see both selection against intermediate ancestry values and selection favoring high ancestry heterozygosity (Fig. 3C).

A Holistic View of Selection on Hybrids

We have outlined how negative epistasis for fitness between opposite-ancestry alleles emerges on phenotypic fitness landscapes in cases of speciation-by-selection. Under mutation-order speciation, phenotypic transgression in hybrids results from the inheritance of opposite-ancestry homozygous loci, causing hybrid breakdown and selection favoring F₂ hybrids that are more heterozygous for ancestry. Under ecological speciation, “falling between niches,” as typically interpreted via a single axis of divergence, causes neither hybrid breakdown nor selection on ancestry heterozygosity, and causes disruptive selection on ancestry proportion. Selection against transgressive phenotypes, by contrast, causes hybrid breakdown and selection for increased ancestry heterozygosity. Importantly, the number of directions (or “dimensions”) where recombinant hybrid phenotypes can “go wrong” can be great (Orr 2000); the magnitude of hybrid breakdown, and the strength of selection against DMIs are expected to grow with the number of traits under divergent or stabilizing selection (Chevin et al. 2014).

Estimates of selection on phenotypes are biased by what we choose to measure and how we interpret our measurements (Houle et al. 2011). By contrast, estimates of fitness and selection on genetic ancestry capture the effect of selection across all traits, but have a coarse connection to phenotypes. Future empirical work into speciation-by-selection should quantify the relative strength of selection caused by “falling between the niches” (ancestry proportion) versus DMIs (ancestry heterozygosity). Following Rundle and Whitlock (2001), it would be prudent to explore how genetic crosses can be compared experimentally via estimates of fitness and patterns of selection on genetic ancestry to clarify mechanisms of selection on hybrids (Rhode and Cruzan 2005; Johansen-Morris and Latta 2006). By conducting experimental manipulations of proposed agents of selection (Rennison et al. 2019), researchers can quantify how particular ecological mechanisms affect selection on ancestry. Experiments quantifying selection on ancestry in hybrid crosses have great promise to illuminate the genetic architecture of postzygotic isolation and its underlying causes.

The geometric model assumes that selection is predictable entirely by the phenotypic distance of a hybrid's phenotype to the parental optima—that is, symmetrical around an optimum. Of course, it is entirely possible for trait combinations that are all equidistant from an optimum (e.g., [0, 0.71] vs. [0.5, 0.5]) to have very different fitness values depending on how traits function together. Such patterns can result, for example, when there are nonlinear relationships between form and function (Wainwright et al. 2005). Unfortunately, little is known about how selection acts on transgressive phenotypes in hybrids. Below, we briefly review what is known about the extent and fitness consequences of phenotypic transgression in hybrids, and conduct an original test of a hypothesis about how transgression changes over the course of divergence.

INCIDENCE, CONSEQUENCES, AND EVOLUTION OF PHENOTYPIC TRANSGRESSION IN HYBRIDS

Evidence of Selection against Transgressive Phenotypes in Hybrids

There is increasing evidence that phenotypic transgression in hybrids is common and maladaptive. Reviews report that transgression caused by nonadditive trait expression is often substantial in F₁s (Rieseberg and Ellstrand 1993; Rieseberg 1995; Thompson et al. 2021), which might reduce fitness. Indeed, several studies have now shown that hybrids between insect lineages adapted to divergent host-plants exhibit a preference for the host-plant upon which they have lower fitness (Matsubayashi et al. 2010; McBride and Singer 2010; Bendall et al. 2017), and studies have started to quantify the fitness consequences of such patterns in natural hybrids (Lingle 1993; Cooper et al. 2018; de Zwaan et al. 2022).

Transgression in recombinant hybrids has also been subject to increasing study. In F₂ hybrids between benthic and limnetic ecotypes of threespine stickleback fish (Gasterosteus aculeatus), individuals with transgressive jaw morphology that combined different features of the two parents had lower fitness than individuals with parent-like or intermediate phenotypes, presumably because they could capture neither evasive nor attached prey (Arnegard et al. 2014). Similarly, a study of BC₁ hybrid sunflowers (Helianthus annuus × Helianthus debilis) found that more transgressive plants had lower reproductive fitness than less transgressive plants (Thompson et al. 2021). Although research is progressing, general patterns about the incidence and fitness consequences of trait transgression in hybrids are unresolved. Studies quantifying phenotypic fitness landscapes under natural conditions (Martin and Wainwright 2013; Arnegard et al. 2014; Keagy et al. 2016; Martin and Gould 2020) represent a promising way to make progress.

Only one study has tested for ecologically mediated DMIs using molecular genotype data (Thompson et al. 2022). Most approaches used to study oligogenic incompatibilities with genotype data aim to locate specific outlier regions (Fig. 4A; Sotola et al. 2023) or correlations in ancestry between loci (Schumer et al. 2014). However, when negative epistasis for fitness is underpinned by complex interactions among many small-effect loci, they might not appear as significant outliers (Fig. 4B) or generate detectable ancestry correlations; detecting such interactions therefore requires alternative approaches. Using F₂ hybrid crosses between benthic and limnetic ecotypes of threespine stickleback fish, Thompson et al. (2022) found that mean genome-wide ancestry heterozygosity was elevated in surviving individuals retrieved from replicate seminatural experimental ponds, but not in aquarium-raised fish. This is consistent with the hypothesis that DMI loci are distributed throughout the genome and are caused by ecologically mediated natural selection (Fig. 3C)—such a pattern would not occur if selection acted only on ancestry proportion (Fig. 3B).

Figure 4.

Simulations showing the key differences between the genetics of large-effect and simple versus small-effect and complex incompatibilities. (A) With large-effect incompatibilities, observed ancestry heterozygosity at incompatibility loci (“×” symbols) in surviving F₂ hybrids typically exceeds the standard deviation in neutral simulations (gray bands). Selection is implemented as in Figure 1A, where the low-fitness allele combination is lethal. (B) When selection acts against transgression caused by small-effect (pleiotropic) QTL, this generates epistatic selection across many loci and individual loci are less likely to appear as significant outliers. The model follows Barton (2001), with the strength of selection matching the mortality in A. Data are from 1000 replicate simulations (Supplemental Material S2).

Does Transgression Change as a Function of Phenotypic Divergence?

Reproductive isolating barriers are more predictable causes of speciation when they increase with divergence between populations (Coyne and Orr 1989, 1997; Matute et al. 2010; Moyle and Nakazato 2010). Thus, we asked: does phenotypic transgression in hybrids—presumably resulting from the expression of opposite-ancestry alleles—grow with phenotypic divergence between cross parents?

Theory predicts that the magnitude of phenotypic transgression in hybrids should increase with the degree of phenotypic evolution that has occurred since parents shared a common ancestor (Barton 1989, 2001; Slatkin and Lande 1994; Chevin et al. 2014). This occurs because the phenotypic variation unlocked by recombination—the segregation variance—is expected to increase as populations diverge. If traits are additive, transgression is expected to increase with phenotypic divergence in the F₂ cross generation but not the F₁. Such a relationship implies that extrinsic selection against hybrids might increase with phenotypic divergence. Using 12 crosses between stickleback populations differing in phenotypic divergence, Chhina et al. (2022) found support for this prediction in both F₂ and (unexpectedly) F₁ hybrids. However, stickleback are extremely young species that have experienced strong divergent natural selection; many allopatric lineages experience parallel or stabilizing selection for hundreds of thousands of years (Anderson and Weir 2022), which might render it difficult to predict transgression from phenotypic divergence (Chevin et al. 2014). It is therefore pressing to test whether the divergence–transgression relationship is general because this would imply that the evolution of extrinsic hybrid fitness follows patterns analogous to those described elsewhere as a “speciation clock” for hybrid viability and fertility (Coyne and Orr 1989, 1997; Edmands 2002; Price and Bouvier 2002; Dagilis et al. 2019; Coughlan and Matute 2020; Matute and Cooper 2021).

We undertook an original quantitative review that collated data from 62 studies (93 crosses and 30,925 individuals) with individual-level phenotype data for both parent lineages, F₁ hybrids, and BC₁ and/or F₂ hybrids. The data are a roughly even mix of insects, plants, and vertebrates. We collapsed data onto principal components, and for each cross we computed phenotypic divergence between parents as the ratio of between-population to within-population phenotypic variation (Fig. 5A). We computed the magnitude of transgression in multivariate space for each hybrid as the distance between its phenotype and the axis connecting parent mean phenotypes (Fig. 5B), standardized according to the amount of apparent “transgression” caused by phenotypic variation in parents. This metric captures both transgression caused by “mismatched” combinations of traits and transgression in traits that do not differ between parents—indeed these two forms generate identical patterns on principal components (Fig. 5C). In addition to testing whether the magnitude of phenotypic divergence between parents predicts the magnitude of phenotypic transgression in hybrids, we used formal meta-analysis to test whether transgression (principle component analysis [PCA] of all traits) and phenotypic variation (trait-by-trait) differ among hybrid cross types (methods and detailed analysis in Supplemental Material S3).

Figure 5.

Systematic review of phenotypic transgression and variation in hybrids. (A) Phenotypic divergence was calculated as the ratio of mean between-species to mean within-species phenotypic distances (length of black lines). (B) Transgression was calculated as the phenotypic distance between hybrids and the line connecting parent mean phenotypes—hybrid values are analyzed as the hybrid:parent transgression ratio. (C) Transgression caused by “mismatched” traits is geometrically equivalent to transgression in a single nondivergent trait; the two phenomena are indistinguishable on principal components. (D) Cross parents that are more phenotypically divergent beget increasingly transgressive hybrids (all P < 0.001; note that axes differ across plots due to differences in parental phenotypic divergence between studies making BC₁ vs. F₂ crosses). (E, upper) F₁, BC₁, and F₂ hybrids exhibit significant transgression (all P < 0.0001; meta-analysis of log[ratio-of-means]), although cross types all differ at P < 0.05. (E, lower) Trait variation (random effects meta-analysis of log[ratio-of-coefficients of variation]) does not differ between parents and F₁ hybrids (P = 0.44), but is significantly elevated in the BC₁ (P = 0.0002) and F₂ (P < 0.0001). Red points are means ± 95% CI, gray points are study means (see Supplemental Material S3). Negative values indicate that hybrids exhibit less transgression (upper) or less variation (lower) than expected from parent values. (Data in A and B based on data in Bradshaw et al. 1998.)

Our analysis provides new insight into the phenomenon of phenotypic transgression in hybrids. The magnitude of transgression was positively associated with the magnitude of phenotypic divergence between cross parents in F₁, BC₁, and F₂ hybrids (all P < 0.0001; Fig. 5D). For every unit increase in the between-population to within-population phenotypic variation, transgression increases by ∼25.7% (95% CI [23.3%–28.1%]; linear model). For F₂s and BC₁s, this pattern is consistent with theoretical expectations of the segregation variance in recombinant hybrid crosses (Slatkin and Lande 1994; Barton 2001; Chevin et al. 2014). Formal meta-analysis revealed that the average magnitude of transgression in F₁ hybrids was four-fifths as great as in BC₁ and/or F₂ hybrids from the same cross, although all hybrid cross types differ significantly in mean transgression (all P < 0.05; Fig. 5E). Trait variation in BC₁ and F₂ hybrids was significantly greater—1.3× and 1.49×, respectively—than what is observed in parents (both P < 0.001), while F₁ hybrids do not exhibit more variation than parents (P = 0.44) (Fig. 5E). This pattern is expected under most quantitative genetic models, and indicates that nonadditive trait expression (whether caused by genetic dominance or uniparental effects) causes the transgression observed in the F₁ (Fig. 5D).

The pattern that we have documented—a positive relationship between phenotypic divergence between nonhybrid parent populations and transgression in hybrids—is evidence of a potentially general mechanism that could link extrinsic postzygotic isolation with phenotypic divergence. Such links between phenotypic divergence and reproductive isolation are necessary, although not sufficient, for demonstrating ecological speciation (Funk et al. 2006; Nosil 2012). The divergence–transgression relationship is similar to the well-documented relationship between laboratory-based estimates of postzygotic isolation and genetic divergence (Coyne and Orr 1989; Coughlan and Matute 2020). Studies quantifying phenotypic fitness landscapes in hybrids are needed to evaluate the fitness consequences of transgression and to determine whether the divergence–transgression relationship underlies an “extrinsic speciation clock.”

CONCLUDING REMARKS

In this article, we sought to clarify how selection against maladaptive transgressive phenotypes in hybrids generates epistasis for fitness between opposite-ancestry alleles, that is: genic hybrid incompatibility. In a similar theme to how previous work has aimed to reconcile DMI theory with the statistical theory of quantitative genetics (Demuth and Wade 2005; Fitzpatrick 2008b), we aim to reconcile DMI theory with concepts in speciation-by-natural-selection. Our perspective is that divergent alleles that combine to generate maladaptive transgressive phenotypes represent hybrid incompatibility alleles. By integrating speciation genetics with concepts from speciation-by-selection (i.e., mutation-order and ecological speciation), we suggest that studying phenotypic transgression and selection on genetic ancestry represent powerful and currently underused phenotypic and genetic approaches for studying mechanisms of postzygotic isolation.

Dobzhansky (1937) wrote that “the genotype of a species is an integrated system adapted to the ecological niche in which the species lives. Gene recombination in the offspring of species hybrids may lead to formation of discordant gene patterns.” Although Dobzhansky appears to have viewed DMIs quite broadly, the conventional view of DMIs has narrowed over time to focus mostly on laboratory-estimated viability and sterility. This focus is not entirely unwarranted—truly environment-independent genic inviability or sterility can only evolve via the DMI model, whereas environment-dependent inviability or sterility does not necessarily require interactions between multiple loci (Coyne and Orr 2004). However, this focus might exclude a significant fraction of DMI loci and we favor a more holistic view in which DMIs have myriad possible phenotypic consequences with genetic architectures ranging from mono- or oligogenic to highly polygenic (Abbott et al. 2013). Modern genomic studies indicate that loci underlying hybrid incompatibilities in the wild could be numerous and spread throughout the genome (Langdon et al. 2022; Xiong et al. 2023). We view it as a major goal of twenty-first century speciation research to make progress toward understanding generalities about the genetic architecture of incompatibilities and their underlying causes. To accomplish this, in addition to the refinement of methods aimed at identifying barrier loci (Laetsch et al. 2023), continued development and validation of methods to detect epistatic selection in hybrid populations (Schumer and Brandvain 2016) and genetic crosses (Simon et al. 2018) will be critical.

Clarifying that DMIs can have underlying ecological causes will improve inferences about mechanisms of speciation. At present, researchers who detect evidence of incompatibilities with genotype data often conclude that the patterns are caused by “intrinsic” selection acting on hybrids (Pulido-Santacruz et al. 2018; Cronemberger et al. 2020; Nikolakis et al. 2022). While this could be correct, we argue that the label of “intrinsic” is conjecture. In fact, whether an incompatibility is environment-independent is unfalsifiable—there will always be an environment yet to be tested. Instead of being the end of a line of inquiry, researchers should remain agnostic about the mechanism(s) underlying DMIs and instead aim to generate and test mechanistic hypotheses. Said differently, genetic evidence of DMIs on its own reveals little about why alleles are incompatible.

As currently used, the “intrinsic versus extrinsic” dichotomy seems to encourage the conflation of environment dependence with particular components of fitness (e.g., germination or hatching), even though Coyne and Orr (2004) explicitly discourage this. Researchers must recognize that the laboratory is an environment, and whether it is more or less benign than field conditions should not be assumed a priori. Studies in killifish (Kozak et al. 2012) and hemiparasitic plants (Wesselingh et al. 2019) detected strong hybrid inviability in the laboratory, concluded that F₁ inviability was an intrinsic barrier to gene flow, then later found that inviability was alleviated in the field. Given that the word “intrinsic” seems to inspire such premature conclusions, we suggest that refraining from using it in the empirical literature would be a productive change in the language of speciation (Harrison 2012). Simply referencing the genetic mechanism and the focal phenotype—for instance, “an incompatibility reducing viability”—would be more accurate.

Many exciting questions about the ecology of hybrid incompatibilities await further study. Are complex traits, such as suction feeding in fishes (McGee et al. 2013, 2015; Arnegard et al. 2014; Higham et al. 2016) or locomotion (Lingle 1992a,b), particularly likely to underlie hybrid breakdown? Do incompatibilities under ecological selection experience negative frequency-dependent selection and might this maintain barriers to gene flow (Moran et al. 2021; Xiong and Mallet 2022)? To what degree is ecologically mediated postzygotic isolation caused by “falling between niches” versus transgression? What are general patterns about the genetic architecture of ecologically mediated incompatibilities? In sum, by embracing the ecology of hybrid incompatibilities, we stand to make great progress toward clarifying ecology's role in speciation.

AUTHOR CONTRIBUTIONS

K.A.T. drafted the paper, and all authors contributed to revisions. K.A.T. collected the phenotype data and collected the QTL data with J.M.C., H.J., and H.S. K.A.T. conducted simulations with input from M.S. and Y.B. K.A.T. analyzed all data. H.S. produced fitness landscape models.