Hybridization in human evolution: Insights from other organisms

Abstract

During the late Pleistocene, isolated lineages of hominins exchanged genes thus influencing genomic variation in humans in both the past and present. However, the dynamics of this genetic exchange and associated phenotypic consequences through time remain poorly understood. Gene exchange across divergent lineages can result in myriad outcomes arising from these dynamics and the environmental conditions under which it occurs. Here we draw from our collective research across various organisms, illustrating some of the ways in which gene exchange can structure genomic/phenotypic diversity within/among species. We present a range of examples relevant to questions about the evolution of hominins. These examples are not meant to be exhaustive, but rather illustrative of the diverse evolutionary causes/consequences of hybridization, highlighting potential drivers of human evolution in the context of hybridization including: influences on adaptive evolution, climate change, developmental systems, sex-differences in behavior, Haldane’s rule and the large X-effect, and transgressive phenotypic variation.

1 |. A SHIFTING PARADIGM: INTROGRESSIVE HYBRIDIZATION AND HUMAN EVOLUTION

Traditionally, the evolution of hominins–humans and our closest relatives since we collectively diverged from our common ancestor with chimpanzees/bonobos–has been represented in two distinct ways. One depicts evolutionary relationships among purported hominin species as a tree describing ancestor–descendant relationships, with evolutionary relatedness depicted by branches that split as one travels upward from the trunk to the leaves, ending in extinction for all those branches not leading to our leaf. Within this set of phylogenetic models lies “out of Africa with replacement” accounts of recent human evolution.¹ The out of Africa with replacement models propose that the entirety of the recent non-African components of individuals’ ancestry (before world empires and colonialism) traces its origins back to Africa sometime between 50 and 100 thousand years ago (ka). As such, non-African human variation in the recent past is a subset of variation that was present in humans in Middle Pleistocene Africa. Under this view, the recent (precolonial) genetic variation was initially established by founder effects associated with modern human populations moving out of Africa, and then influenced by recurring gene flow within and among the various regions of the world over tens of millennia. It posits further that these early modern populations did not interbreed with resident groups of archaic humans such as Neanderthals, and that differentiation between them was generated by mutation, drift and natural selection acting in more or less independently evolving lineages.

In contrast, multiregional continuity evolutionary models emphasize sustained gene flow, as opposed to replacement and extinction. In the case of recent human evolution, this includes sustained gene flow across the entirety of the human inhabited world at all points throughout the Pleistocene and into the present day.² The neutral component of the multiregional model came to be identified as isolation by distance.^3–5 Under isolation by distance, the local reduction of genomic variation by random genetic drift is counteracted by the addition of new variation to local groups via gene flow. Left unperturbed by major evolutionary events such as range expansions and local replacement of populations, this balance between gene flow and random genetic drift leads to an equilibrium between within group and among group genetic variation. In this treatment, adaptation to regional conditions by natural selection leads to more pronounced differences among groups. A key distinguishing factor between the multiregional continuity model of recent human evolution and the out of Africa with replacement model is that the former postulates that Homo sapiens populations outside of Africa became established far earlier (~1 million years ago [Ma] or more) than the latter model (~100 ka or less).

These two models have occupied the extreme ends of a continuum of possible accounts of recent human evolution. Other models of recent human evolution combined features of both renditions and allowed for a combination of long periods of isolation interrupted by bouts of gene flow between groups.^6,7 These models were often subsumed under the general multiregional view in the 1990s and 2000s,⁸ although sometimes Bräuer’s model (Afro European sapiens hypothesis) was lumped together with out of Africa.⁹

A striking realization in the last decade is that neither of the scenarios at the extreme poles of this continuum is in fact supported by new genetic and genomic evidence.^10–15 Clear evidence for reticulation among diverged lineages/populations (Neanderthals, Denisovans, H. sapiens) falsifies the predictions of branching phylogenetic models without reticulation, while evidence for periods of substantial isolation among these same groups is inconsistent with the multiregional continuity model. Further studies of the phenotypic consequences of hybridization across taxa^16–22 have shown that admixture of different lineages may have provided genetic variation that was then favored by natural selection to drive evolutionary innovation in our hominin ancestors, and perhaps contributed to bursts of exceptional phenotypic diversification, not only in our recent past (e.g., emergence of H. sapiens²³), but also deeper in time.

This shifting narrative is perhaps unsurprising when considered in a broader context. For some time we have known that hybridization, resulting in gene flow and subsequent introgression, together with natural selection and drift, plays an important role in speciation and the evolution of diversity in animal taxa.^24–27 Recent research validates the work of earlier evolutionary biologists (e.g., Anderson²⁸; Anderson and Stebbins²⁹), who emphasized the significance of introgressive hybridization, and the phenotypic variation it can produce, to adaptive evolution and biodiversification (e.g., “divergence-with-gene-flow”³⁰). Evolutionary biologists are now moving past the question of whether lineages can diverge while undergoing gene exchange with other lineages, toward investigating how such exchange affects the interacting groups of organisms (Figure 1).³¹ Hybridization can also play a role in extinction.³² There are many examples of morphologically distinct species whose representation in the fossil record seems to wane or cease over time. In such cases, some genetic characteristics of a lineage may continue even when anatomical morphs disappear, as is arguably the case in recent human evolution.

FIGURE 1.

Illustration of the gradation from gene flow to introgression along a continuum of speciation, resulting in a reticulated pattern of speciation at the population level. Genetic distance refers to distance between the hybridizing parents (increasing as you move through time). Changing phenotypic effects with increasing genetic distance are shown to the right. Phenotypic variance refers to within population variance of the hybrid population. The shapes of the effect curves are qualitative and so not to any scale; though based on available evidence to date, they remain working hypotheses [Color figure can be viewed at wileyonlinelibrary.com]

Here we draw from studies of key organisms, chosen for their potential as models for considering processes relevant to human evolution. We use these examples to argue four points germane to understanding the relevance and potential influence of hybridization and introgression in human evolution.

1. The human genome shows ample signs of introgression via hybridization, which is also not unusual in other organisms, including amphibians, birds, and myriad terrestrial mammals, including primates.

2. As informative as the human genome has been, it is insufficient to answer questions about the entire span of human evolution. Phenotypic approaches are needed, as are genomic data from other extant and extinct groups for comparison and context.

3. Developing phenotypic approaches to hybridization requires a better understanding of the relationships of genes, development, and phenotype for which experimental and comparative approaches are necessary.

4. Introgressive hybridization is especially relevant to the evolution of humans and other organisms because it provides an important source of genetic variation on which natural selection may act.

Demonstrating the accuracy of these points establishes both the occurrence and theoretical importance of hybridization and introgression. Our goal is not merely to highlight means for identifying hybridization in human evolution, but to provide examples that will point the way toward an exciting new set of approaches and questions, encouraging exploration of a wide range of models for understanding the potential dynamics of genetic exchange in shaping the evolutionary trajectories of our ancestors.

2 |. THE HUMAN GENOME SHOWS EXTENSIVE EVIDENCE OF INTROGRESSION THAT IS COMPARABLE TO WHAT IS KNOWN IN MANY OTHER ORGANISMS

2.1 |. Introgression in human evolution

New genetic and genomic evidence for introgression in human evolution establish that hybridization and introgression contributed to shaping genomic variation.^10–15 If there is a history of repeated hybridization and introgression, then the “species tree” itself may become reticulate. These new results show that networks or braided streams,²³ as opposed to trees, are a more useful way of depicting species and population relationships as inferred from molecular genetic data,³³ thus invalidating division of our fossil ancestors into strictly discrete units or species (at least biologically, although such division may be heuristically useful). Splitting and subsequent contact and mixture of lineages is a part of human evolution that cannot be ignored.

Recent human evolutionary genomic and genetic studies (reviewed by Smith and colleagues³⁴) point to a complex picture of lineage divergence and re-merging. Most of the genetic studies providing evidence for introgression have focused on the contact between lineages in Eurasia in the Late Pleistocene. These lineages diverged recently, over the course of the past 1 Ma or so, and repeated signtures of past hybridization events have been demonstrated among them (Figure 2). The most recent estimates indicate that ancient H. sapiens and Neanderthals diverged from each other 520–630 ka while the divergence of Neanderthals and Denisovans has been dated to 390–440 ka.¹³ The current genetic evidence for gene exchange includes movement from these ancient H. sapiens into Neanderthals ca. 100 ka, and from Neanderthals into ancient H. sapiens ca. 50 ka.³⁵ Gene flow between ancient H. sapiens and Denisovans has also been demonstrated, as represented in the genomes of recent Asians and Australasians/Melanesians,^14,15,36 which harbor Denisovan ancestry. Recently, the genome of a first generation descendent of a Neanderthal mother and a Denisovan father living ca. 90 ka was reported.³⁷ That study, along with previous analyses, also demonstrates ongoing gene flow from Neanderthals into Denisovans.^14,37 There is also possible evidence that the Denisovans may have acquired genes from an ancient hominin that diverged from the ancestor of humans more than a million years ago.¹⁴ Other studies provide evidence for gene flow between ancient and recent lineages within Africa and for a recent influx of Neanderthal genes by way of Eurasian back migration into at least East Africa within the last 4,000 years.^38,39

FIGURE 2.

Hypothesized directions and magnitudes of introgression in hominin evolution. Genetic evidence for known introgression indicated by black arrows between lineages. The green region indicates the time period of purported fossil (phenotypic) evidence for introgression between Neanderthals and H. sapiens (fossil/phenotypic evidence for introgression between Denisovans and other taxa is currently unknown, due to our limited understanding of the morphology of Denisovans). Genetic evidence of a first generation hybrid (Denisova 11)³⁷ between Neanderthals and Denisovans is shown with a dotted line, though the introgressive effect of this on future lineages is currently unknown. Presumably, evidence (both genotypic and fossil/phenotypic) for gene flow earlier in the separation of lineages would be more difficult to detect, as would phenotypic/genotypic differences among the hybridizing taxa. This is consistent with the hominin fossil record of the middle Pleistocene [Color figure can be viewed at wileyonlinelibrary.com]

More than simply establishing that gene flow took place, genomic studies also contribute insight into the dynamics of genetic exchange across populations during times of hybridization and introgression, and their effects on people living today. For example, current evidence indicates a 1.8–2.6% genomic contribution from Neanderthals to living Eurasians,¹³ and 3–6% from Denisovans to living Melanesians.¹⁴ It is estimated that ~20% of the Neanderthal genome is represented in the combined genomes of living human populations, although only 2% is typically found in any particular non-African person.⁴⁰

2.2 |. Hybridization in nonprimate mammals

A look toward large bodied terrestrial mammals shows that humans are not an outlier when it comes to hybridization (see also Table 1). Hybridization and introgression are common in many groups, including between distantly-related species, and these systems can provide referential models for human hybridization. Large canids, for example, can disperse over long distances, and hence individuals can encounter both divergent conspecific populations and related species as well as variation in habitat. Ancient H. sapiens similarly dispersed throughout the six inhabited continents, coming into contact with other hominin taxa across a broad spectrum of ecological settings in Eurasia and Africa. Like hominin ancestors, many wolf-like canids have only recently diverged, and hence have not evolved strong reproductive isolating mechanisms, resulting in a high prevalence of hybridization. Examples include admixture between the gray wolf (Canis lupus) and coyote (C. latrans), that have only diverged in the last ~50 ka⁴¹; the domestic dog (C. familaris) and the gray wolf, which diverged about 25 ka⁴² (see review in Freedman and Wayne⁴³; Thalmann et al.⁴⁴); as well as between gray wolves and even more divergent species, such as the golden wolf (C. anthus) and golden jackals (C. aureus; Freedman and Wayne⁴³; Koepfli et al.⁴⁵). In fact, the amount of gray wolf introgression into these two taxa is strikingly similar to that seen between Neanderthals and H. sapiens.

TABLE 1.

Factors relating to hybridization in nonhominin organisms included in this discussion and their importance to understanding hybridization’s role in hominin evolution

	Greater frequency of Introgression than expected	Evidence for Introgression of positively selected genes (adaptation)	Hybrids exhibiting increased variation or Heterosis	Hybridization and its relationship to climate and environment	Differential contribution of sexes in hybridization–Including Haldane’s rule and large X effect	Direct association between genetic and morphological traits relating to hybridization	Specific applications to hybridization in hominin evolution
Galapagos finches	X	X	X	X		X	Demonstration of combined effects of introgression and natural selection, evidence for long-term introgression in closely related species, evidence of critical role of climate, ecology and “cultural mechanisms” in successful hybridization
Clawed frogs	X	Speculated	Speculated	Speculated		X	Traits that appear identical in phenotype may actually have different patterns of genetic control; allopolyploidization may be a good model for understanding ancient polyplodization in jawed vertebrates
Mice	X		X		X		Demonstrate greater variability in hybrids and backcrossed samples than in parental groups, show that relatively high amounts of hybridization (≥30% in mice) are necessary to accurately detect hybridization in a sample
Wolf-like canids		X	X			X	Impact of wide geographic dispersions, examples of relatively recent divergence in many lineages and impact of this on establishment of reproductive barriers, role of skin/hair characteristics
Recent and ancient bears				X	X		Impact of climate, especially bringing species/populations together that normally are fully allopatric
Marmosets	X		X	X			Nature of hybrid zones critical in determining outcomes of hybridization
Howler monkeys				X	X		Both Haldane’s rule and large X effects can be critical factors in primate hybridization
Baboons	X		X	X	X		Closely-related, relatively young species that commonly hybridize at species borders; hybrids can exhibit heterosis and high frequencies of skeletal anomalies. Extensive, ancient “swamping” hybridization by male introgression is attested by patterns of mitochondrial variation among populations

Similarly, a number of recent studies suggest that admixture between bear species is widespread.⁴⁶ There is evidence for gene introgression from extinct cave bears (Ursus spelaeus complex) into living brown bears (Ursus arctos), as well as some gene flow from brown bears into cave bears.⁴⁷ Although cave bears have been extinct for 25,000 years, modern brown bears still have between 0.9 and 2.4% of cave bear DNA remaining in their genomes. This is comparable to what is seen with Neanderthals and H. sapiens. Significant admixture has been well-established between polar bears (U. maritimus) and brown bears,^48–51 which form the sister clade to cave bears. These species diverged some 479–343 ka.⁵⁰ However, brown bears from several populations carry mitochondrial DNA (mtDNA) haplotypes that are more closely related to polar bear haplotypes than to other brown bear haplotypes.^49,52 The very recent common ancestry of the mitochondria carried by these bears, as recent as 37–10 ka in some cases, suggests a history of admixture between these species.⁴⁹ The extant brown bears that have the closest mitochondrial relationship with polar bears live on Admiralty, Baranof, and Chichagof (ABC) Islands in southeast Alaska.^49,52 Likewise, ABC Islands brown bears show evidence of polar bear ancestry in their nuclear genomes, which include at least 6–9% genetic material of polar bear origin.⁴⁸ MtDNA evidence suggests that another, now extinct, admixed population of brown bears lived in Ireland around the time of the Last Glacial Maximum.⁴⁹ The distribution of polar bear ancestry within the ABC Islands brown bears’ genomes is best explained by a maternal polar bear origin of the population with substantial male brown bear immigration.⁴⁸ This evidence suggests that in the past the ABC Islands populations had a much higher frequency of polar bear derived alleles than it does today.⁴⁸

This alternative, population conversion, or “nuclear swamping,” model of admixture envisions a much greater impact on the genetic diversity of populations experiencing admixture than the classic model of low frequency admixture. Preliminary analyses of the Irish brown bears’ nuclear genomes suggest that a similar phenomenon may have taken place in those populations.⁵³ For closely related species that rarely come into contact because of their different ecological requirements, disruptive climatic events may temporarily create the opportunity for admixture that substantially exceeds the amount observed in more stable conditions. Again, this scenario is relevant to hominin evolution, and especially Late Pleistocene Europe, where climatic fluctuations would have affected ranges of ancient hominins, and potentially both the prevalence and intensity of contact.

2.3 |. Hybridization and introgression in nonhuman primates

Among nonhuman primates (NHPs), mammals often used as models for human ancestors due to their close phylogenetic relatedness, hybridization has been most extensively studied in baboons. The six recognized baboon species have parapatric ranges with contact between neighboring species. Since natural hybridization has been recorded between the most morphologically distinct (chacmas, Papio ursinus vs. Kinda baboons, P. kindae), the most behaviorally different (hamadryas, P. hamadryas vs. olive, P. anubis) and two of the most phylogenetically distant (olive vs. yellow, P. cynocephalus) species, it is likely that hybridization occurs at all the geographic regions where baboon ranges are in contact.⁵⁴ The best-known examples of ongoing hybridization in baboons are from hybrid-zones between hamadryas and olive baboons in Awash National Park, Ethiopia,^55,56 and between olive and yellow baboons in Kenya.^57,58 The earliest studies were confined to morphological and behavioral traits, and showed, at least in the case of the hamadryas-olive hybrids, that species-characteristic behaviors (such as the male hamadryas’ herding, a behavior that is not expressed by olive baboons) most likely have a genetic basis. Hybrid males show an outer appearance (e.g., coat color and pattern) and behavior that is intermediate between the two parental species.^56,59–61

Baboon hybrid zones are a manifestation of a long-standing evolutionary process. Results of whole genome analyses of all six Papio species reveal a history of population differentiation and genetic divergence among lineages that includes multiple episodes of gene flow (admixture or introgression) among distinct lineages (phylogenetic species) and the presence of ghost lineages.⁶² Like the genus Homo, baboons are the products of a radiation that began in non-forest, tropical Africa around 2 Ma. Along with the rest of the fauna of tropical Africa (e.g., Dolotovskaya et al.⁶³; Grubb⁶⁴; Kingdon⁶⁵), they have been subject to climatic fluctuations and geological events that caused repeated fragmentation and reconnection of populations, producing successive intervals of genetic divergence due to geographical isolation and secondary contact with admixture. The ecological and habitat parallels make them a prime analog for hominin evolution.^21,23,66 They are especially relevant for early African savanna-adapted hominins with possibly similar patterns of range expansion and contraction, population isolation, secondary contact, and admixture.

Recent genetic work has revealed a complex history of natural gene flow among baboon species lineages, and even between Papio and another, closely-related but distinct genus, Rungwecebus (the kipunji). It suggests that we entertain the possibility of similar episodes in human history, with gene flow not only among Pleistocene species of Homo, but also, at an earlier time, between emerging Homo populations and other hominins, such as Paranthropus, that were morphologically and adaptively distinct. Phylogeographic analyses (mostly mitochondrial-based) show that in baboons the distribution of mitochondrial haplogroups is not reciprocally monophyletic among phenotypic lineages, and geographic co-occurrence with phenotypic characters that define species is weak. For example, northern chacma baboons carry mitochondrial haplotypes that are closely related to those of southern yellow baboons, northern yellow baboons carry mitochondrial haplotypes related to those of hamadryas baboons, and ancient mitochondrial lineages exist within western olive baboons, most likely derived from baboon populations that no longer exist.^67,68 These incongruences between mitochondrial haplogroup and external phenotypes (and the species they define) suggest past episodes of gene flow among diverging baboon lineages.^69–71 Mitochondrial-nuclear discordance might result from “mitochondrial capture” or “nuclear swamping.”^70,72,73 These examples are relevant to hominin evolution; for example, genetic affinities between mitochondrial and autosomal DNA are discordant, with closer mitochondrial DNA relationships between Neandertals and modern humans, and closer autosomal relationships between Neandertals and Denisovans.¹⁵

South American Callithrix marmosets (Figure 3) also demonstrate extensive hybridization between subspecies and species, in variable environments and with different levels of reproductive isolation, therefore making them an additional comparative primate model for considering hominin hybridization. Speciation of current Callithrix marmosets began approximately 3.7 Ma, with the most recent divergence event taking place less than 1 Ma between C. jacchus and C. penicillata.^74,75 Experimental interspecific crosses revealed different levels of reproductive isolation between different species pairs, as captive C. jacchus × C. penicillata hybridized with relative ease, but physical anomalies such as blindness were common when other species pairs were crossed.^76–78 The genus originated in the humid Atlantic Forest biome of southeast Brazil and migrated north to also occur in Brazil’s savannah- and desert-like semi-arid biomes,⁷⁹ with the six Callithrix species settling into allopatric distributions,⁸⁰ and hybridizing naturally at distribution borders under secondary contact.⁸¹ Genetic and phenotypic accounts of natural Callithrix hybridization exist between four different Callithrix species pairings occurring along river barriers and in areas of ecological transition.^81–84 Relationships among Callithrix genomes are better represented by reticulating species trees as opposed to bifurcating ones, with divergence occurring under a scenario of speciation with gene flow.⁸¹ Thus, marmoset speciation also serves as a referential model of the splitting, subsequent contact and admixture of closely related lineages, a dynamic that was also important for recent human evolution.

FIGURE 3.

A comparison of facial variation among two Callithrix species and their hybrid. Left: Anthropogenic Callthrix penicillata x C. geoffroyi hybrid from Viçosa, MG, Brazil. Top right: C. geoffroyi. Bottom right: C. penicillata [Color figure can be viewed at wileyonlinelibrary.com]

Anthropogenic hybridization as a result of the illegal pet trade introduced Callithrix species from northern and central Brazil, particularly C. jacchus and C. penicillata, into other regions of the country and created artificial secondary contact with other exotic and native Callithrix populations (e.g., Malukiewicz et al.;⁸¹ Malukiewicz et al.;⁸² Mendes;⁸³ Passamani et al.⁸⁴). Although illegal and undesirable from a conservation biology perspective, the artificial introduction of marmoset into the ranges of existing species may provide unique insights into the effects of demographic differences in the hybridization process of ancient hominins, especially those that involved long-range (and possibly rapid) movements of hominin groups into occupied territories, including the geographic radiation of Neanderthals into Asia (and probably North Africa) and the movement of ancient H. sapiens both within and out of Africa. Moreover, anthropogenic Callithrix hybridization may also be particularly useful for considering shifts in demographic and genomic composition of admixed populations whose parental taxa are closely related but display different degrees of relatedness (as in Neanderthals, Denisovans, and H. sapiens). For example, hybridization between more divergent species (e.g., C. aurita × C. jacchus; C. aurita × C. penicillata) often results in mixed social groups where individuals possess both hybrid and parental phenotypes.^85,86 In both cases, male C. aurita hybrids (with ancestry from C. penicillata or C. jacchus) tend to possess the Y chromosome of the former^85,87 and mtDNA of the latter,⁸⁵ suggesting that some hybrid allele combinations may be less favorable. In contrast, hybrid swarms, or highly variable populations, whose genomes consist of products of subsequent segregation and recombination, backcrossing, and crossing between the hybrids, are common among more closely related Callithrix species (C. jacchus, C. penicillata, C. geoffroyi, and C. kuhlii) hybridizing under artificial secondary contact.⁸¹ Overall, anthropogenic movement of C. jacchus and C. penicillata is driving them to dominate modern Callithrix hybridization–and in some cases possibly causing genetic swamping of (if not only introgression into) other marmoset populations. Further study of this process could provide some insight into the dynamics of hominin dispersal(s) out of Africa.

Natural hybridization has also been genetically confirmed among more divergent primate species pairs. In particular, the study of a hybrid zone between two howler monkey species (Alouatta pigra × A. palliata)^88,89 has provided interesting parallels to the observed outcomes of hybridization between Neanderthals and H. sapiens. Despite the much deeper divergence (~3 Ma)⁹⁰ between the parental howler monkey species than between H. sapiens and now-extinct congeners, in both cases there is evidence of reduced or null introgression of sex chromosomes. In particular, studies that mapped Neanderthal⁹¹ and Denisovan⁹² ancestry in modern human genomes found reduced ancestry of these species’ alleles on the H. sapiens X chromosome. This pattern has been observed among other organisms that hybridize and is referred to as the “large X-effect,”⁹³ for which the prevailing hypothesis is that the X chromosome is enriched for genes associated with reproductive barriers (e.g., hybrid sterility or inviability)⁹⁴ and therefore does not introgress during hybridization. Analyses of introgression of autosomal (microsatellite) and X chromosome markers in the howler monkey hybrid zone show that the X chromosome has reduced introgression when compared to the autosomes.⁹⁵ Thus, despite differences in divergence, studying the mechanisms that are responsible for this pattern in howler monkeys may allow us to contribute to the understanding of possible scenarios that led to the observed reduction of Neanderthal and Denisovan ancestry on the X chromosome in the genome of our own lineage.⁹¹

With regards to the Y chromosome, analyses of 91 admixed males in the howler monkey hybrid zone showed a pattern consistent with a lack of introgression of SRY gene variants that are diagnostic for each species.⁹⁵ This pattern contrasts with the observed bi-directional introgression of mtDNA haplotypes (exclusively transmitted by the mother). Together these patterns are consistent with Haldane’s rule, where hybrids of the heterogametic sex (XY males in this case) are often inviable or infertile.⁹⁶ If males from the first (F1) generation were fertile, we would expect that Y chromosome variants would be passed through backcrossing to later generation hybrids in a similar fashion as the mtDNA. Similarly, Mendez et al.⁹⁷ analyzed and compared Neanderthal Y chromosome sequence data to those of humans and chimpanzees and found no evidence of Neanderthal Y chromosome haplotypes in modern humans. The authors speculate that this may be a consequence of hybrid incompatibilities. However, the ancient nature of this admixture coupled with the extinction of Neanderthals makes it difficult to directly test this hypothesis. Thus, the current study of the patterns of introgression in the howler monkey hybrid zone may add to our understanding of the general mechanisms that led to the lack of Y chromosome introgression, a topic that is also of great interest in the study of speciation genetics. Comparative analyses of the evolutionary mechanisms leading to these patterns of restricted introgression in sex chromosomes among model and non-model species in natural and laboratory conditions will shed light on the evolutionary mechanisms that produced similar patterns during the ancient hybridization among Homo species.

Although the patterns on sex chromosome introgression in the howler monkey hybrid zone are similar in some respects to what is known for ancestral populations of anatomically modern humans, Neanderthals, and Denisovans, the deeper divergence between hybridizing howler monkey species make this system a better analogue for general patterns of admixture between more divergent hominin lineages (e.g., Paranthropus and Homo), comparable to what is seen between Papio and Rungwecebus (above), and may suggest that admixture between more divergent hominin taxa that overlapped in time and space could have been possible. Thus, this hybrid system also provides us with a living model to understand the types of processes and mechanisms that may have played a role in the hybridization events at different stages during our evolutionary past.

To sum, although we have chosen to focus here only on a select group of organisms, they illustrate how hybridization and introgression are important for shaping mammal genomes, comparable to what has been observed across myriad organisms living today (e.g., Arnold and Kunte³¹), including primates,^98,99 but also other organisms more distantly related to us. The influence of hybridization on the genetic and phenotypic variation observed in the present day suggests that the magnitude of gene exchange, the loci most affected, and whether parental species persist after hybridization, are influenced by levels of divergence between hybridizing taxa, as well as by climate, biogeography, and demography. These results also suggest an important role for stochastic factors. This is important to keep in mind when interpreting the results of hybridization between different hominin lineages. Comparative models in the form of other hybridizing organisms illustrating the importance of climate (bears), dispersal (canids), population dynamics, and behavior (baboons; howler monkeys), intrinsic hybrid fitness (howler monkeys), and more particular ecological circumstances (Callithrix) will be vital for building a better understanding of the role of hybridization among hominin taxa.

3 |. GENETIC DATA ARE LIMITED IN THEIR ABILITY TO INTERPRET THE FOSSIL RECORD, NECESSITATING A PHENOTYPIC APPROACH

We cannot understand the details of evolutionary process from genetic material alone. There is growing evidence that DNA can only be recovered from a small fraction of human fossils, even among those from the most recent period of inter-species contact in the Late Pleistocene. Furthermore, genomic studies to detect gene exchange among taxa such as early Homo, Australopithecus, and Paranthropus are currently impossible due to the advanced degradation of DNA in all remains of these more ancient lineages. Genomic studies also cannot tell us much about the phenotypes of our archaic ancestors because of the limits in our understanding of how genotypes might influence phenotypes. Features with a simple genetic basis or those with detailed and generalizable genotype–phenotype maps (e.g., pigmentation¹⁰⁰) may allow reconstruction of fossil phenotypes, but complex characteristics and/or characteristics with the potential for complex interactive bases (e.g., skeletal morphology¹⁰¹) are not likely candidates.

The kinds of perturbations that hybridization imposes on morphology, however, can serve as a guide to identifying admixture in the fossil record using phenotypes alone, a necessary precursor to understanding the dynamics of past gene exchange. This possibility exists because morphological variation within species and among closely related species arises from the channeling of genetic and environmental influences into phenotypic variation through conserved common developmental pathways.¹⁰² The kinds of perturbations that stem from epistatic interactions in hybrids might lead to characteristic phenotypic changes even if the underlying genotypes are different.

Most of the morphological studies purporting to identify fossil hominin hybrids are uninformed by genetic and comparative perspectives on the morphological effects of hybridization (but see Ackermann¹⁸). In these cases, evidence for admixture is largely based on the presence of mixed (mosaic) or intermediate morphology, rather than the application of theory bound analyses, resulting in the identification of several candidate hybrid hominin fossils. These include Lagar Velho 1,¹⁰³ Mladeč 5 and 6,¹⁰⁴ Cioclovina 1,¹⁰⁵ Peştera cu Oase 2,¹⁰⁶ Vindija,¹⁰⁷ Klasies River Mouth,¹⁰⁸ Jebel Iroud and Mugharet el ‘Aliya in North Africa,^108,109 Lingjing (Xuchang crania) in China,¹¹⁰ Middle Pleistocene hominins,^111,112 and possibly others.^113,114 However, none of these specimens have been widely accepted as hybrids, and hybrid status has specifically been disputed for several of them (e.g., Grine et al.¹¹⁵; Harvati et al. ¹¹⁶; Stringer¹¹⁷; Tattersall and Schwartz¹¹⁸).

Ackermann (2010)¹⁸ used a systematic and theoretically informed approach based on comparative mammalian data to identify fossil hybrids, and concluded that the fossils Skhul IV and V and PesLtera cu Oase 2, and other specimens from Qafzeh, Krapina and Amud, possibly showed many of the characteristic signs of hybridization identified in genetic and comparative studies of mammals¹⁸ (see Section 4). The Peştera cu Oase 1 individual, which, like associated Peştera cu Oase 2, shows admixture based on morphology,^106,119 was confirmed to be a hybrid using ancient DNA (ca. 4–6 generations prior).¹⁰ At a population level, the recent evidence for admixture at 100 ka³⁵ supports proposals that specimens like Skhul are admixed, while genetic evidence of Neanderthal influence in North Africa¹²⁰ support earlier morphology-based suggestions.^108,109 This consilience between morphological and genomic approaches at the individual and population levels suggests that a general biologically informed perspective on hybridization may yield further insights into the dynamics of gene exchange in the fossil record.

The limited genomic reach into the past poses important challenges for identifying hybridization in the fossil record and understanding its evolutionary consequences. A unified comparative and experimental approach will afford a clearer picture of the ways in which hybridization drives perturbations of development that may allow for a general set of diagnostics for identifying hybrid individuals and populations in the fossil record. Moreover, from a comparative perspective, we would like a better sense of the variational (phenotypic, but also genetic) and evolutionary consequences of hybridization in the wild. A more thorough understanding of the phenotypic consequences of hybridization, particularly in the skeleton, is a pressing concern for the study of human evolution.

4 |. COMPARATIVE AND EXPERIMENTAL APPROACHES TO STUDYING THE PHENOTYPIC EFFECTS OF HYBRIDIZATION

4.1 |. Baboons as phenotypic models for hybridization

Currently our best comparative basis for identifying the morphological effects of hybridization on the skeleton in primates comes from research on hybrids of Papio cynocephalus (the yellow baboon) and P. anubis (the olive baboon), one of the hybridizing moieties described above. Studies of captive, known-pedigree, purebred yellow and olive baboons and their hybrids have shown that these hybrids exhibit high frequencies of skeletal and dental traits that are rare in parents, including small and/or supernumerary canine teeth, rotated molars, and sutural anomalies.^20,21 These traits suggest that hybridization results in a breakdown in the coordination of early development, reflected in visible perturbations in dental and sutural formation.^17,18 In addition to these rare nonmetric features, baboon hybrids also show a greater degree of overall size variation (Figure 4),^20,21 including in the interior of the nasal cavity.¹²¹ On average, baboon hybrid crania tend to be large, with many individuals at or outside the range of the largest parent species.

FIGURE 4.

Hybrid and purebred baboon crania, illustrating the wide variation and relatively large size of hybrids relative to their parent taxa. Top: Female baboons, with parent taxa on left (Papio anubis, top, P. cynocephalus, bottom), and three F1 hybrids on right. Bottom: male baboons, from left to right, F1 hybrid, F1 hybrid, P. cynocephalus, and P. anubis. Images reprinted from: Ackermann RR, Rogers J, and Cheverud JM. Identifying the morphological signatures of hybridization in primate and human evolution. J Hum Evol 51: 632–645, copyright (2006), with permission from Elsevier

Transgressive phenotypes are particularly marked in the craniofacial skeleton and dentition.^20,21 This is a stroke of good fortune as arguments over taxonomy and phylogeny in the hominin fossil record mostly revolve around dental and craniofacial characteristics. Both atypical traits and extreme size are transgressive relative to the parents. Moreover, close examination of specific aspects of the cranium that exhibit heterosis and dysgenesis in hybrids suggest that the overall form of hybrid baboon crania differs from that of the parents.^20,21 Based on overall covariation of metric features, comparisons of parents and first generation (F₁) hybrids indicate that the hybrids exhibit lower overall trait covariation than either parent (Ritzman & Ackermann, unpublished data). While these differences are small, the fact that they occur with increased phenotypic variance suggests a decrease in integration in hybrids.^102,122

Results from baboons are consistent with what we know about the skeletal anatomy of hybrids in other mammalian lineages. Features such as the presence of supernumerary teeth, rotated teeth, and sutural anomalies in the neurocranium and face, appear in ungulates,¹⁶ rodents,¹²³ and primates,^17,19–21 although they manifest somewhat differently in each taxon. Moreover, specific patterns in supernumerary teeth (i.e., the presence of mandibular vs. maxillary supernumerary teeth and/or unilateral vs. bilateral expression of supernumerary teeth) have also been observed to differ between hybrids and parents.^20,123 The consistency of these findings across taxa strongly suggests that the presence of such nonmetric traits in relatively high frequencies is a general indicator of hybridization. Moreover, second generation hybrids and backcrossed individuals show that these traits persist in later generation hybrids²¹ (see also Ackermann and Bishop¹⁹).

Although less well-studied, signatures of hybridization in the size and shape of the skull have also been identified. As discussed above for baboons, the most noteworthy signature of hybridization is large cranial size, with hybrids either (and statistically significantly): (a) exceeding the midpoint value of the means of the parental taxa (e.g., Cheverud et al.¹²⁴), or more strictly (b) exceeding the means of both parents. Using these definitions, extreme cranial size in hybrids has been identified in mice^125–127 and in primates.^20,124 Heterotic phenotypes are also evident beyond the F₁ generation. Taken together, these results suggest that, in addition to the presence of rare nonmetric traits (as above), large cranial size may be a more general feature of mammalian hybrids. Further research needs to be done into the ways in which hybridization causes the genomic interactions that underlie these changes in conserved developmental pathways. Finally, the effects of hybridization on the postcranial skeleton are largely unstudied, except insofar as many previous studies in mice and primates have found that hybrids generally exhibit longer limbs and increased body size relative to parents.^{18,20,21,125,127–133} Furthermore, the apparent effect of hybridization on overall morphology may be diminished after multiple generations of backcrossing, making backcrossed individuals physically undistinguishable from the parental species with whom they share most of their genome.¹³⁰

4.2 |. Mice as phenotypic models for hybridization

Outside of primates, mice stand out as a powerful mammalian model for investigating the effects of hybridization on phenotypic variation because they can be bred in controlled conditions and have short generation times, making it feasible to perform large, multigenerational experiments with replication. This makes them excellent model organisms for understanding the morphology of mammalian hybrids, including hominins. There is also a broad literature regarding hybridization and introgression among subspecies of Mus musculus,^134–137 as well as some research focusing on hybridization among more divergent species of the genus Mus.^138–140 In addition to comparative analysis of wild and captive organisms, an ongoing experimental research project aims to determine whether hybrids show any relationships between variation in coat color and variation in craniomandibular/skeletal traits, as a result of shared developmental processes being affected by hybridization.^22,141,142 Breeding experiments have produced multi-generation mouse recombinants of three closely related subspecies of M. musculus, as well as two species (M. musculus and M. spretus), with various degrees of introgression.

Morphometric analyses of variation in craniomandibular size and shape have been performed among the subspecies, first generation (F₁), second generation (F₂), and first generation backcrossed (B₁) hybrids.^22,141 Important patterns are emerging that might shed light on human evolution. In particular, the results indicate that hybrids are typically as large or larger than the parent taxon, with mean shape generally intermediate to the parents but more closely resembling the smaller parent.^22,141 Features such as molar length are relatively large in hybrids, while later generation (F₂, B₂) hybrids more variable that F₁s, and backcrosses more closely approximating the shape of the parental groups they are backcrossing into. When combined with unusual nonmetric traits,^20,21 these features can be considered suggestive of mixed ancestry in the fossil record (see also Ackermann¹⁸). For the mice, this pattern appears to hold across very different scenarios of hybridization, for example, (a) between species that hybridize in nature but gene-flow and fertility rates of hybrids are low; (b) between species that hybridize in nature and the hybrids are successful; and (c) between species that do not hybridize in nature (due to geographic separation), but where hybrids can be produced under laboratory conditions.²² This suggests that these results are fairly robust and can be applied to a wide range of contact scenarios in different sets of lineages.

Analyses of coat color variation, focused on comparisons of dorsal-ventral (DV) patterning in F₁ animals and parental strains,¹⁴² indicate that hybrids have highly variable coat morphologies.^143,144 Given that the DV pattern is established during prenatal development,¹⁴⁵ observed differences in the ventral coat morphology of the F₁ mouse hybrids relative to the parents, as well as new combinations of dorsal and ventral coat colors, could be the result of two differently co-adapted genomes and developmental patterns coming together. Interestingly, the development of DV patterning can be tied to the development of the skeleton, with mutations in the Tbx15 gene affecting both DV patterning and skeletal morphology.¹⁴⁶ This has interesting potential implications for human evolution. We know some of the major changes in human evolution are soft tissue changes, such as changes in hair and sweat gland structure, function and distribution,^147,148 and that genes related to skin/hair phenotypes have introgressed from other ancient lineages into our own lineage nd remain in people today (e.g.,Vernot and Akey⁴⁰; Sankararaman et al.⁹¹). A variant of the TBX15 gene which is derived from a hominin closely related to the Denisovans is found at high frequencies in some modern human populations.¹⁴⁹ The results from the mice suggest that some of the skeletal changes/differences we observe in hominins could result from introgression of gene variants that were retained due to the adaptive value of other phenotypes. Further understanding of the links between these systems has the potential to provide information into the underlying target of selection and could allow us to extrapolate from the fossil skeletal phenotype to such soft-tissue features.

Some suggested hominin hybrid samples express patterns similar to what the mouse and baboon studies indicate for hybrids (although it is not possible to know what generation–F₁, F₂, B_1, and so on–the fossil samples represent). For example, the Vindija Neanderthals (Figure 5) have an intermediate facial anatomy compared to most Neanderthals and early modern humans; facial size is more like modern humans but morphological features of the mandible and supraorbital torus are more like the Neanderthals.^107,150 At Klasies River Mouth, there is evidence of extensive intra-sample variation with indications of a very large face (for one individual) and mandibles that exhibit a mosaic of both modern and archaic features.¹⁰⁸ These and other individuals/samples are generally more like the smaller hominin on the landscape (H. sapiens) in terms of morphology, if not gracility (e.g., Cioclovina 1, Mladeč 5 and 6, Oase 1 and 2, Jebel Irhoud, and Lagar Velho1). Atypical traits are also present in hominins,¹⁵¹ including a number of anomalies of the maxilla and dentition in Upper Paleolithic samples. Strikingly, Abri Pataud 1 has two supernumerary teeth adjacent to the right upper second molar, while the isolated tooth Dolní Vĕstonice 33 has been suggested to be a distomolar.¹⁵¹ Although these patterns are commensurate with hybridization, clearly more work is necessary to identify conclusively the prevalence, demographics, and lingering morphological effects of hybridization in past humans (but see^152–154 for recent attempts at the latter).

FIGURE 5.

Intermediate features of the Vindija Neandertals. Details and data are available in Smith, et al.¹⁰⁷ clockwise from top left: (1) Vindija 259 left adult maxilla. Reconstructed nasal breadth falls almost three standard deviations (SDs) below the Neandertal mean. (2) Vindija mandible 206 (lingual view). An incipient incurvatio mandibulae and mentum osseum are visible on the external mandibular symphysis. (3) Vindija mandible 231, oriented to the alveolar plane. The almost vertical symphysis reflects reduction of facial prognathism. The symphyseal angles of vi 206 and vi 231 fall more than three SD below the Neandertal mean. (4) Vindija 231 mandible. The symphysis exhibits a weak outline of the mental trigone, a modern human feature [Color figure can be viewed at wileyonlinelibrary.com]

5 |. THE INTERPLAY BETWEEN HYBRIDIZATION AND ADAPTATION

The burgeoning literature on the phenotypic and adaptive consequences of hybridization in the wild provides several further insights relevant to problems we encounter in understanding human evolution. Hybridization and introgression can facilitate adaptation in several distinct ways. By introducing alleles into a new population that have already been a part of the evolution of an adaptation to a local condition in a different population, a new population borrows fitness-enhancing variants. Hybridized populations will, in most cases, have higher heterozygosity than either parent population. This can lead to an increase in the genetic variance underlying individual differences in a phenotypic characteristic and a concomitant increase in its propensity to respond to directional selection. Likewise, interactions arising from novel combinations of alleles at loci (dominance) and genotypes across loci (epistasis) can cause increases in the genetic variance of phenotypic characteristics out of proportion to any increase in genomic variation. The generation of new variation can thereby aid in the adaptation of a population to novel conditions.

Adaptive radiations are canonical examples of divergence through natural selection resulting in species with diverse phenotypes associated with ecological variation in their habitats.¹⁵⁵ For instance, recent (postaustralopithecine) hominins and Darwin’s finches both diversified over the last 1–2 Ma, and both diversifications were influenced by glacial cycles of changing climate. Continental habitats and Galápagos Islands fluctuated in size and connectivity along with the availability of conspecific mates, with periods of isolation. A feature common to both systems was the exchange of genes between previously isolated populations. In some cases, the genes were adaptive, for example, genes affecting beak size in finches, and genes affecting properties of the immune system in hominins.^91,156

A long-term study of one adaptive radiation, Darwin’s finches (Subfamily: Geospizinae) in the Galápagos archipelago, provides an example of how hybridization and its consequences can facilitate adaptation to new environments and the origins of new species. Two features make this group suitable for intensive study. Most of the islands are in, or close to, the natural state, and no finch species has become extinct through human activity.¹⁵⁷ An uninterrupted 40-year study of finches on the small (0.34 ha) island of Daphne Major has demonstrated instances of evolution by natural selection and identified the circumstances under which it occurs.¹⁵⁸ However, it has also revealed the unsuspected prevalence of introgressive hybridization.¹⁵⁹

A central concern in the study of hybridization in the human fossil record is the degree to which hominin hybrids might have been viable and capable of reproduction.^91,160,161 Darwin’s finches add some perspective to this issue. Hybridization in the finches was observed in the first year of the long-term breeding study on the island.¹⁵⁸ A few breeding pairs of Geospiza fortis, the medium ground finch, bred successfully with G. fuliginosa, the small ground finch, and with G. scandens, the cactus finch. Offspring fledged from nests of both types of heterospecific pairs, thus demonstrating the lack of strong genetic barriers to reproduction in the early stages of offspring development. Unanswered, however, was the question of whether the hybrids were fertile. Another 7 years elapsed before there was an (affirmative) answer: when ecological conditions were favorable for their survival in dry seasons they bred successfully in the following wet seasons. They did not, however, breed with each other but backcrossed to one or other of the parental species according to the song of their fathers that they learned when in the nest and during the period of parental dependence as fledglings. The comparisons demonstrated that hybrids and backcrosses survived as well as, and perhaps better than, contemporary nonhybrids in each of the three cohorts. Moreover, they were as successful in obtaining mates and raising offspring as were G. fortis and G. scandens (G. fuliginosa does not have an independent breeding population on the island). Thus, in terms of survival and reproduction they were as fit as their respective parental species.¹⁵⁹ Additionally, the importance of song in this scenario highlights the degree to which the dynamics of hybridization and introgression are behaviorally/culturally mediated, a particularly relevant and relatively unexplored aspect in hominin contact scenarios.

Hybridization has the potential to lead to the formation of a new species through introgression of genes from one species to another, and hybrids then evolve along a new trajectory in a different environment.¹⁵⁸ Again Darwin’s finches on Daphne provide an example. An immigrant G. conirostris arrived in 1981, bred with G. fortis and initiated a new lineage. After two generations a severe drought in 2003–04 caused heavy mortality among all finches leaving two survivors. They were a brother and a sister, and they bred with each other. Their offspring and grand-offspring also bred only with other members of the lineage and not with G. fortis or G. scandens. Their beaks were larger than the beaks of either species, and the males sang a unique song and, coupled with apparent reproductive isolation, the lineage behaves as a new species.^158,162 This example differs from the standard allopatric model of speciation by incorporating introgressive hybridization as an important factor in the development of reproductive isolation on a small island. It also once again highlights the importance of altered behavior as a phenotypic outcome of hybridization, and something on which selection can act, a phenomenon also seen in primates. For example, in the case of hybridization between olive and hamadryas baboons, behavior can be strongly affected by hybridization. Hybrid males between these two species show intermediate forms of mating behavior. Whereas olive baboon males form temporary consortships with estrous females, hamadryas baboon males monopolize females irrespective of their reproductive status. Hybrid males show less monopolization tendencies than hamadryas males and also show a weak form of the hamadryas typical herding behavior.^56,59,163 The possibility that hybridization-induced behavioral novelty might have played a role in human evolution has not been well-explored (but see Ackermann et al.²³).

Another example of hybrid speciation from even more phylogenetically diverged organisms are African clawed frogs of the genus Xenopus. About 30 allo-tetraploid, allo-octoploid, and allo-dodecaploid species exist in this genus, and each was formed by speciation of hybrid individuals (the progeny of two diploids, two tetraploids, and tetraploid and octoploid species, respectively).¹⁶⁴ Potential adaptive consequences of these hybrid speciation events include increased resistance to parasites^165,166 and increased tolerance to an acidic embryonic environment.¹⁶⁷ Allopolyploid Xenopus provide an interesting perspective into early genomic evolution of ancient polyploid ancestors of hominins;¹⁶⁸ also similar to hominins, some hybrid Xenopus are not polyploid (e.g., Furman et al.¹⁶⁹). Moreover, hominins are themselves descended from ancient ancestors that underwent two rounds of polyplopidization before jawed vertebrates diversified over 500 Ma.¹⁷⁰ Some of the duplicated genes that arose from these ancient events persist today, and small scale gene duplication contributes to copy number variation between closely related primate species.¹⁷¹ Gene copy number variation thus may contribute to adaptation and reproductive isolation in modern primates, including humans. Whether the ancient (>500 million years old) whole genome duplication events occurred via spontaneous genome duplication within a species (autopolyploidization) or in association with hybridization among species (allopolyploidization)¹⁷² may never be known, though the later possibility has several intriguing implications. Similar to the benefits associated with hybridization without genome duplication, allopolyploidization brings together (a) beneficial alleles that evolved independently in two diverged parental species, (b) deleterious alleles from each parental species, and perhaps most significantly, (c) genetic variants that function well in each parental species but that are incompatible when combined.

Returning to our previous example of gray wolves, the transfer of an allelic variant that causes black coat color in most dogs to North American gray wolves provides another insight into the adaptive consequences of hybridization.¹⁷³ In North America, black wolves are common in many populations and are nearly as frequent as gray (wild type) wolves. Genetic analysis has demonstrated that the genetic variant, called the K locus, originated in domestic dogs, and was likely transferred to wolves from pre-Columbian dogs of Native Americans.¹⁷³ This dominant mutation causing black coat color enhances the fitness of heterozygote black individuals, but decreases the fitness of homozygous black wolves^174,175 suggesting coat color alone is not the primary object of selection. Black wolves may have higher survivorship during viral disease epidemics, such as distemper and mange, given that the K locus is a beta-defensin gene that responds to viral infections.¹⁷³ Moreover, selection has favored disassortative mating, where individuals have a preference for mates of a different color.¹⁷⁶ One possible model for the persistence of the black gene is that in areas where canine disease is common and persists in carnivore reservoirs such as domestic dogs or other wild carnivore species, the coat color polymorphism is maintained. In other areas where this is not the case, such as the High Arctic, the disease burden is lower, and observed black wolves are rare. Such adaptive introgression can occur from wolves into dogs as well. Hypoxia adaptation in high altitude Tibetan wolves and dogs involves selection at the EPAS1 gene, for which some alleles may have been transferred to dogs from wolves as the former have recently arrived in the Tibetan Plateau. Living Tibetans also obtained new allelic variation for the same gene via ancient admixture between H. sapiens and Denisovans.^36,177,178

In the wild, coyotes and wolves hybridize, most commonly in the Great Lakes area of the United States and Canada.^179–182 A genome-wide analysis showed that all genomic segments did not introgress equivalently, as some fragments had a more extensive distribution and were more frequent than predicted by the effects of background levels of admixture alone.⁴¹ The regions that were differentially introgressed from gray wolves into coyotes, and from coyotes into gray wolves, contained genes that affected morphology and physiology, which are gene categories that might be expected to have important functions in hybrids, as body size and morphology are essential components of successful prey capture. Moreover, the prey size in the Great Lakes is smaller than elsewhere consisting of deer rather than larger prey such as elk or moose. In this situation, an intermediate sized canid (between coyote and gray wolf) may have higher fitness than the parental forms. Interestingly, this intermediate morphology was detected in preliminary studies of the morphology of the Eastern coyote or “coywolf,”¹⁷⁹ and an ongoing study suggests they have atypical cranio-dental traits (Schroeder, unpublished data). As with Darwin’s finches, should the nature of available food change, a different sized canid may be selected. In fact, body size has changed in coyote-sized canids in North America throughout the Pleistocene,¹⁸³ and some of these body size changes may reflect admixture between coyote and wolf-sized canids as observed today. Moreover, many of the North American wolf species (including C. lupus, C. rufus, and C. lupus lycaon) are hybrids themselves, with these populations displaying between 30 and 50% coyote ancestry.⁴¹

The canid example is useful for understanding the dynamics and potential adaptive consequences of introgression in hominins. Like canids, hominins dispersed widely resulting in gene introgression between lineages that diverged recently in some cases and more deeply in others. Introgressed DNA linked to phenotypic characteristics in living people has been subject to the effects of natural selection after introgression. Jolly (2001) and Garrigan and Kingan (2007) hypothesized that introgression from now-extinct species of Homo may have provided H. sapiens with locally adapted allelic variation;^66,184 this hypothesis is now well substantiated.^{36,40,91,154,185–187} The persistence of ancient gene variants in extant human populations, especially those associated with skin and hair characteristics (including pigmentation, toughness of the skin (perhaps tied to better cold adaptation), and other factors related to keratin (e.g., Vernot and Akey⁴⁰; Sankararaman et al.⁹¹; Dannemann and Kelso¹⁸⁵)), immunity (specifically the human leucocyte antigen [HLA] Class 1 genes),^156,188 and high-altitude adaptations,^36,186 is in striking parallel to what we see in canids. Genes affecting morphology and physiology have also diverged in hominin ancestral groups and could have been affected by hybridization.

Returning to the EPAS1 example, strong selection at the EPAS1 locus in Tibetan populations living at extreme elevation led to changes in oxygen metabolism, which allowed individuals in this group to better function and reproduce in hypoxic environment.^36,186 The allele subject to the resulting selective sweep is derived from EPAS1 alleles introgressed from the Denisovan lineage.³⁶ Hackinger et al. (2016) showed a similar effect of EPAS1 was present in other high elevation parts of the Himalayan region.¹⁸⁶ The Denisovan “Core haplotype” in human populations living above 2,000 m in elevation was enriched. Introgressed Denisovan alleles were absent in human samples from lowland populations.¹⁸⁶ Variants apparently conferring adaptive advantages for cold adaptation,¹⁸⁹ immune responses,¹⁸⁸ and sundry other functions^185,190 introgressed from ancient hominins are also evident in the genome today.

Adaptation to combinations of local and novel conditions is not the entire story, however. Genetic drift and neutral mutations drive human genome evolution to the extent that expectations of neutrality form the null hypothesis of tests for natural selection in the genome.¹⁹¹ Additionally, some of the genomic evidence for hybridization observed in living humans suggests that there may have been purifying selection against variants formerly in Neanderthal populations,^192–194 although recent research suggests that negative selection against Neandertal ancestry did not play as strong a role in recent human evolution as previously suggested.¹⁹⁵

Taken together, these different pieces of evidence point to the ways in which hybridization and introgression form an indispensable part of the understanding of the role of natural selection and adaptation in recent human evolution. They provide a path to produce novel genotypic combinations that would not readily arise through mutation alone. Some of these novel genotypic combinations influence fitness, sometimes for ill, but often to adaptive ends.

6 |. CONCLUSION: THE (EXCITING!) WAY FORWARD

The question of what makes humans different from other species has captured the imagination of both scientists and nonscientists. At a certain level, we know the answer: our DNA is different from that of other species, and this allows us to abstract, create, and destroy like no other. So how did our species acquire its unique genome? Genetic and fossil evidence demonstrate that the ancestors of humans split into distinct branches as they spread and diversified, adapting to local conditions. Archaic human “cousins,” often recognized as species separate from H. sapiens, were similar to us in many ways–they wore clothes, buried their dead, and made art (e.g., Klein¹⁶⁰; Hoffmann et al.¹⁹⁶; Rodríguez-Vidal et al.¹⁹⁷). Before they became extinct, they exchanged genetic material with humans by hybridizing on multiple occasions (e.g., Green et al.¹²; Kuhlwilm et al.³⁵; Vernot and Akey⁴⁰; Sankararaman⁹²; Vernot et al.¹⁹⁸). Which bits of our DNA did we get from these archaic “cousins”, and what are the biological consequences? We are just starting to learn this. We know, for example, that several disease-related genes were affected, not all people got the same bits of DNA, and not all parts of our genomes received the same amount of DNA from these extinct species. The question now is why and how this happened.

Beyond documenting the quantity and results of gene exchange, this exciting avenue of research also challenges much of the existing framework for considering hominin diversity. Yes, it is causing us to rethink the details of modern human origins and accept that neither a strict replacement scenario nor a multiregional gene flow model can explain recent human evolution. But more profoundly, it is expanding the model of divergence through reproductive isolation and adaptation, to one that recognizes gene exchange as a key player in the speciation process. We now know that gene exchange has played a central role in providing new diversity, injecting new variation from one population into another, and thereby facilitating the rapid (adaptive) diversification of lineages in well studied and evolutionarily relevant taxa. We have detailed much of this evidence in the preceding sections of this article and argue that human evolution provides yet another example of this critical evolutionary process.

The interspecific exchange of gene variants also complicates our concepts of species. This is a real challenge for palaeoanthropologists, for whom finding and categorizing new fossils–or even new species–is sometimes a primary (and exciting) goal. How do we recognize and appreciate diversity when it has been shaped by gene exchange? How do we rethink and visualize our phylogenies as a reticulating network rather than a branching bush? How do we continue to improve our models of population dynamics through time to reflect this? Part of that will involve a much better grasp of within-taxon variation and the effects interacting evolutionary forces have on complex developmental systems and on the phenotype. Paleoanthropologists can benefit from considering other organisms and disciplines, where there has been a growing acceptance that boundaries of species are imprecise and more porous than previously believed, and that the dynamics of gene exchange complicate relationships. These issues provide conceptual and practical challenges for our field, but they also provide opportunities for fresh exploration into the dynamics of human evolution. More broadly, gene flow between related species is one of several ways the generation of variation produces macroevolutionary implications.¹⁹⁹

The research presented here provides some exciting material for commencing such exploration. These organismal models can help us consider new explanations, and develop new explanatory hypotheses, for the patterns we observe in the fossil record. For example, the majority of closely related hybridizing taxa discussed here have been shown to have a greater frequency of hybridization and amount of introgression than originally thought, as well as differential contribution of sexes in hybridization, a model which is likely to fit the Middle Pleistocene hominin situation. It is also quite plausible that–like in finches–the combined effects of introgression and natural selection were important for hominin gene exchange, but that cultural mechanisms like language were critical factors in the dynamics of this exchange. Climate fluctuations are also likely to have played a crucial role in these dynamics, bringing taxa together that are normally allopatric (as seen in bears), but also moving the geographic boundaries between taxa and shifting which taxa are favored over time. Wide geographic dispersion (e.g., wolf-like canids) is also a characteristic of later hominins and the differentiation of skeletal phenotypes and skin/hair characteristics surely played a role in the emergent morphology of their hybrids (as suggested in canids, baboons and mice). All of these applications (and more; see Table 1) provide a rich opportuntiy for better understanding how, when and why human complexity emerged.

GLOSSARY

Adaptive introgression

When the incorporation of new genetic variants through introgression leads to an increase of the fitness of the recipient gene pool.

Backcross

The breeding of a hybrid with a member of one of the parental species.

Cohort

A group of individuals born in the same year (or reproductive cycle).

Dysgenesis

Hybrid dysfunction, including sterility or lowered function. For analyses of morphology, dysgenesis can refer to measured features that are smaller or reduced in the hybrids relative to parents and/or to the mid-parental value.

Enriched gene/haplotype

Increased presence or expression of a gene/haplotype in a sample.

Epistasis/Epistatic interactions

Interactions between alleles at different loci, where the effects of an allele at one locus is modified by alleles at other loci in an interacting complex.

Gene flow

The transfer of genetic variants (or alleles) from one population to another.

Genome duplication

Production of cells/organisms with additional copies of entire genome; resulting cells and organisms are polyploid.

Ghost lineages

Lineages that are now phenotypically extinct but whose presence can be detected by their genomic contribution to extant lineages.

Haldane’s rule

In the case where hybrids between taxa produce only one sex that is inviable or sterile, that sex is more likely to be the heterogametic sex, for example, in mammals males with XY chromosomes (as opposed to the homogametic, e.g., in mammals females with XX chromosomes).

Heterosis

Hybrid vigor that can result from heterozygote advantage or recombinant hybrid vigor. For analyses of morphology, heterosis is often used to describe measured features, which are larger in hybrids relative to parents and/or to the mid-parental value.

Heterospecific

Other species, in contrast to same species (conspecific).

Heterozygosity

The probability that two alleles at a locus drawn from a population will differ. High heterozygosity indicates large amounts of genetic variation.

Hybrid

Offspring of a cross between genetically/phylogenetically divergent taxa (verb: hybridize).

Hybridization

Interbreeding between individuals from genetically differentiated lineages (typically at or above the rank of subspecies).

Hybrid speciation

A form of speciation where hybridization between two distinct species leads to the formation of a new (third) species.

Hybrid vigor

Increased fitness in the hybrid relative to the parents, often due to increased heterozygosity.

Introgression

Incorporation of gene variants (i.e., alleles) from one lineage into the gene pool of another.

Large X-effect

The disproportionately large involvement of the X chromosome in the reproductive isolation of closely related species and the process of speciation.

Mitochondrial capture

Complete mitochondrial introgression, for example, the replacement of the original mitochondrial genome of a population/taxon with that of another as a result of female movement/invasion and interbreeding.

Nuclear swamping

Replacement of the nuclear genome in one population by the nuclear genome of another by repeated unilateral introgression, while leaving the mitochondrial genome in place, for example, as a result of the repeat movement/invasion of males.

Parapatric

Spatial relationship between populations/taxa whose ranges do not significantly overlap but are immediately adjacent.

Phylogeographic

Relationship between the structure of gene genealogies and geography.

Reticulation/Reticulate evolution

The fusing of previously separated branches on an evolutionary tree, through repeated hybridization events, or other forms of horizontal gene transfer.

Selective sweep

A reduction or elimination of allelic variation due to strong positive selection on and subsequent fixation of (an)other allele(s).

Transgressive phenotypes

When traits in hybrids are outside of the range of the parents.

AUTHOR BIOGRAPHIES

Rebecca Rogers Ackermann is a biological anthropologist, Professor in the Department of Archaeology, Director of the Human Evolution Research Institute, and Deputy Dean of Transformation in the Faculty of Science at the University of Cape Town. Her research focuses on evolutionary process, and specifically how gene flow, drift and selection interact to produce skeletal diversity through time, with a particular focus on human evolution.

Michael L. Arnold is an evolutionary biologist and Distinguished Research Professor in the Department of Genetics at the University of Georgia. His research focusses on the effects caused by genetic exchange among divergent lineages, particularly adaptation and biodiversification.

Marcella D. Baiz is a molecular ecologist, and a postdoctoral researcher in the Department of Biology at The Pennsylvania State University. She uses genetics and genomics to understand evolutionary processes, particularly the causes and consequences of introgression in hybridization and speciation.

James A. Cahill is a postdoctoral researcher at Rockefeller University. His interests are population genomics, evolutionary biology, and ancient DNA, in particular of bears. He is currently researching the evolution of vocal communication.

Liliana Cortés-Ortiz is a Research Associate Professor in the Department of Ecology and Evolutionary at the University of Michigan. Her research focuses on evolution and systematics of Neotropical primates, with an emphasis in understating the role of hybridization in primate diversification.

Ben Jonathan Evans is an evolutionary biologist and Professor in the Department of Biology at McMaster University, Canada. His research explores sex chromosomes and sexual differentiation, genome evolution, and speciation.

B. Rosemary Grant and Peter R. Grant are evolutionary ecologists and emeritus Professors in the Department of Ecology and Evolutionary Biology at Princeton University. Their joint research seeks to understand the process of speciation in groups of organisms that have adaptively radiated into many ecological niches, and the particular contributions of interspecific hybridization.

Benedikt Hallgrímsson is a biological anthropologist, Professor and Head of the Department of Cell Biology and Anatomy and Scientific Director for the Alberta Children’s Research Institute at the University of Calgary. His research addresses the developmental and genetic basis of morphological variation as it relates to evolution and disease.

Robyn A. Humphreys is a PhD student in the Department of Archeology and the Human Evolution Research Institute at the University of Cape Town. Her Master’s research focused on the effects of hybridization on coat color variation in hybrid mice; she is currently researching issues at the interface of archeology and public engagement.

Clifford J. Jolly is Professor Emeritus in the Department of Anthropology, New York University. His research has focused on the evolution and diversification of papionin monkeys, and their relevance to human evolution, integrating behavioral, genetic and morphological approaches.

Joanna Malukiewicz is a postdoctoral researcher associated with CEMAFAUNA, Universidade Federal do Vale do Rio São Francisco and the German Primate Center. Her research focuses on genomic, health, and evolutionary consequences of hybridization in closely related species. Her model organisms are the primates of the Callithrix genus.

Christopher J. Percival is an Assistant Professor in the Department of Anthropology at Stony Brook University. He studies how modifications to early developmental processes produce evolutionary changes in craniofacial morphology and birth defects.

Terrence B. Ritzman is an Assistant Professor in the Department of Neuroscience at the Washington University School of Medicine. His research focuses on the comparative anatomy of primate crania as it relates to human evolution.

Christian Roos is a geneticist, senior scientist in the Primate Genetics Laboratory and head of the Gene Bank of Primates of the German Primate Center in Göttingen, Germany. His interests are genomics, evolution, and conservation of primates, in particular of Asian catarrhines.

Charles C. Roseman is an Associate Professor in in the Department of Animal Biology at the University of Illinois. His research focuses on the genetic basis and evolution of complex traits and synthesizing quantitative genetics and evolutionary developmental biology.

Lauren Schroeder is a palaeoanthropologist, and Assistant Professor in the Department of Anthropology at the University of Toronto. Her research focuses on the application of evolutionary quantitative genetics to understand the evolution of the cranial and mandibular diversity within hominins.

Fred H. Smith is University Professor of Anthropology and Biological Sciences Emeritus at Illinois State University and adjunct Professor of Anthropology at the University of Colorado. His research focuses on the paleobiology of Neanderthals and the emergence of modern humans.

Kerryn A. Warren is a postdoctoral research fellow at the Human Evolution Research Institute at the University of Cape Town, and an Underground Astronaut for the Lee R Berger Foundation. Her research looks at the effects of hybridization on mammalian morphology, and the effectiveness of evolution education in the South African school curriculum.

Robert K. Wayne is a Distinguished Professor and HHMI Professor in the Department of Ecology and Evolutionary Biology at UCLA. He applies new molecular genetic techniques to questions in ecology, evolution, behavior, and conservation of animals.

Dietmar Zinner is a zoologist, and Senior Scientist in the Cognitive Ethology Laboratory of the German Primate Center in Göttingen, Germany. His research focusses on evolution, ecology, and behavior of primates, in particular on the phylogeography of African papionins.