Partial genomic survival of cave bears in living brown bears

Abstract

Although many large mammal species went extinct at the end of the Pleistocene epoch, their DNA may persist due to past episodes of interspecies admixture. However, direct empirical evidence of the persistence of ancient alleles remains scarce. Here, we present multifold coverage genomic data from four Late Pleistocene cave bears (Ursus spelaeus complex) and show that cave bears hybridized with brown bears (Ursus arctos) during the Pleistocene. We develop an approach to assess both the directionality and relative timing of gene flow. We find that segments of cave bear DNA still persist in the genomes of living brown bears, with cave bears contributing 0.9 to 2.4% of the genomes of all brown bears investigated. Our results show that even though extinction is typically considered as absolute, following admixture, fragments of the gene pool of extinct species can survive for tens of thousands of years in the genomes of extant recipient species.

Reporting Summary

Further information on experimental design is available in the Nature Research Reporting Summary linked to this article.

It is increasingly apparent that admixture among closely related mammalian species may have occurred frequently over the course of their evolution¹. Many extant Holarctic mammals existed in widespread sympatry with now-extinct megafauna species during the Pleistocene, providing the opportunity for admixture. Palaeogenomic studies have shown evidence of gene flow from two archaic hominins into modern humans^2,3, but it remains debated whether these represent distinct species, or early archaic populations within the broader human radiation⁴. Thus, a genetic contribution of ecologically and morphologically divergent Pleistocene megafauna to living mammal populations represents a plausible hypothesis that is largely untested by empirical evidence.

Cave bears are an iconic component of the Pleistocene megafauna. Cave bears went extinct around 25,000 years ago⁵, following a protracted period of population decline, with interactions with humans being a likely contributing factor^6,7. Admixture between brown bears (Ursus arctos) and polar bears (Ursus maritimus), which form the sister clade to cave bears^8,9, is well documented^10–12, and recent studies suggest that interspecies admixture may be widespread among representatives of the Ursidae¹³. However, the genetic contribution, if any, of extinct bear species to their living congeners is largely unknown. Specifically, it is unknown whether admixture occurred between brown bears and cave bears, which coexisted in widespread sympatry and local syntopy in Eurasia for hundreds of thousands of years^14,15 before extinction of the cave bear.

Results

Sampling of bear genomes.

To investigate whether brown and cave bears admixed during the Pleistocene, we extracted and sequenced nuclear genomic DNA from the petrous bones of four Late Pleistocene cave bears.

These samples were assigned to recognized cave bear taxa based on morphology and geographic location, and subsequently verified by analysis of mitochondrial sequences¹⁶. Although we refrain from forming any taxonomic conclusions based on our genomic datasets, we retain these assigned names for consistency with the published cave bear literature. Three of the cave bear samples are from Europe: an individual from the Gamssulzen cave, Austria, which is assigned to the taxon ingressus and has been ¹⁴C dated to 35,062 ± 966 yr BP⁷; an individual from the Eirós cave, Spain, which is assigned to the taxon spelaeus and has been ¹⁴C dated to 34,806 ± 931 yr BP⁷; and a third individual from the Windischkopf cave, Austria, assigned to the taxon eremus, which has been ¹⁴C dated to >49,000 yr BP and dated by phylogenetic tip dating analysis to 71,992 yr BP (95% credibility interval 54,640–91,860 yr BP)⁷. The fourth cave bear is from the Hovk-1 cave in the southern Caucasus (Armenia), and is assigned to the taxon kudarensis. The Caucasus cave bear is ¹⁴C dated to >49,000 yr BP¹⁷ and stratigraphic unit 6, from which the bone was excavated, is optically stimulated luminescence dated to 54,600 ± 5,700 yr BP¹⁸. We also sequenced DNA extracted from the petrous bone of a Late Pleistocene brown bear from the Winden cave, Austria, which has been ¹⁴C dated to 41,201 ± 895 yr BP⁷. This individual was contemporaneous with cave bears and is located in geographic proximity to the two sequenced Austrian cave bears, ingressus and eremus. We additionally sequenced brown bears from modern Georgian, Slovenian, Russian and Spanish populations, which, when combined with published datasets, provides representative sampling of the complete Holarctic distribution of the brown bear. Moreover, our sampling design provides approximate geographic pairing of each sequenced cave bear with a modern brown bear individual. Our analyses also included published datasets of three polar bears, an American black bear (Ursus americanus)¹², an Asiatic black bear (Ursus thibetanus), a spectacled bear (Tremarctos ornatus)¹³ and the genome assembly of the giant panda (Ailuropoda melanoleuca)¹⁹. Full details of samples and datasets are provided in Supplementary Tables 1–3.

Cave bear relationships.

Phylogenetic analysis of aligned genomic sequences supported the monophyly of cave bears and their position as sister to the brown and polar bear clade (Fig. 1a). Within cave bears, the Caucasus cave bear kudarensis is sister to a clade containing the European cave bears eremus, ingressus and spelaeus, with the latter two forming sister taxa within that clade. This result is consistent with relationships inferred from morphology²⁰ but contradicts those inferred using mitochondrial DNA, which instead indicate spelaeus and eremus as sister clades^7,16 (Supplementary Fig. l). This incongruence may reflect either incomplete sorting of mitochondrial lineages in the population ancestral to ingressus and spelaeus, or transfer and fixation of introgressed mitochondrial lineages as a result of admixture between spelaeus and eremus.

Fig. 1 |. Phylogenetic relationships of the sequenced brown, polar and cave bear genomes.

a, The maximum-likelihood phylogeny of individuals used in this study based on whole-genome transversion differences, rooted using the American black bear outgroup. b, Measures of clade differentiation based on D statistic tests inconsistent with the species tree. The tested topologies are shown on the left, with points showing the calculated clade differentiation (D values) for all possible combinations of the sampled individuals. This measure of clade differentiation scales between 0 (indicating a phylogenetic trifurcation) and 1 (indicating complete lineage sorting and absence of post-divergence gene flow). Measures of differentiation for the brown bear clade relative to polar bears, and for the polar bear clade relative to brown bears, are shown for comparison with values obtained for cave bears.

We then examined the extent to which cave bear clades are differentiated from one another by calculating the proportion of derived alleles an individual shares with members of the same clade relative to those shared with their sister clade, expressed as D statistics^2,21 for topologies that are inconsistent with the species tree (Fig. 1b). This measure of clade differentiation scales between 0 and l, with low values indicating much incomplete lineage sorting and/or admixture among clades, and high values indicating that lineage sorting is approaching completion. This analysis revealed relatively low differentiation of the European cave bears ingressus and spelaeus clade from their sister eremus. In contrast, the European cave bear clade as a whole is highly differentiated from the Caucasus cave bear kudarensis lineage, suggesting deep and temporally sustained structuring among these respective populations.

Admixture between cave bears and brown bears.

We investigated interspecies admixture using D statistics^2,21. All brown bears exhibited a significant excess of shared derived alleles with cave bears relative to polar bears (Supplementary Fig. 2), supporting admixture between cave bears and brown bears. We then used the related f̂ statistic²¹ to estimate the genomic proportion shared between brown bears and cave bears as a consequence of admixture following the divergence of brown and polar bears. This admixture proportion corresponds to the fraction of the cave bear genome that has introgressed into the brown bear genome assuming unidirectional gene flow. Values of f̂ were variable among brown bears, and highest in the Late Pleistocene Austrian brown bear (at least 2.4%, averaged across all comparisons), which was contemporaneous with cave bears. Among modern brown bears, the cave bear admixture proportion is highest in the Georgian brown bear (at least 1.8%), intermediate in Western European brown bears (at least 1.3 to l.4%), and lowest in the Russian and American brown bears (at least 0.9 to 1.0%). Since American brown bears exist outside of the Pleistocene distribution of cave bears, the variation in admixture proportions among brown bears shows some correspondence with their geographic and temporal proximity to the sampled Late Pleistocene cave bears. However, we did not find any obvious localized pattern of increased admixture between geographically paired brown bears and cave bears (Supplementary Fig. 2). Variable post-admixture diffusion of cave bear alleles via gene flow among brown bear populations may therefore be responsible for the observed patterns at a more geographically localized scale.

We also investigated variability in the proportion of brown bear admixture among pairs of cave bears. All comparisons between European cave bears fell below the Z > 3 threshold for strong significance, probably because these individuals were admixed with brown bears in similar proportions. However, comparisons involving the Caucasus cave bear kudarensis and European cave bears suggested an increased proportion of admixture with brown bears (around 1%) for European cave bears, with moderate statistical support (Z > 2) for many comparisons (Fig. 2). This suggests that some admixture with brown bears may have postdated the basal divergence of the sampled Caucasian and European cave bear lineages.

Fig. 2 |.

a, The proportion of the genomes of individual brown bears derived from admixture with cave bears, following the divergence of brown bears and polar bears (f̂results). Eighteen admixture estimates per brown bear represent all possible combinations of European cave bear introgressor, relative to three polar bears. Almost all values are strongly significant (Z > 3). A small number of comparisons involving the Russian and American bears were more moderately supported (Z > 2.4). ABC, Admiralty, Baranof and Chichagof Islands; Den., Denali; LP, Late Pleistocene. b, The proportion of the genomes of individual European cave bears derived from admixture with brown bears, following the divergence of the European cave bear clade and the Caucasus cave bear kudarensis (f̂results). Twenty dots per cave bear sample represent all possible combinations of modern Eurasian brown bear introgressors. All comparisons involving eremus are at least moderately supported (Z > 2), but 30% and 70% of comparisons involving ingressus and spelaeus, respectively, fell below this threshold (indicated by open circles).

Models of unidirectional gene flow do not explain the observed patterns.

The D statistic does not provide explicit information on the direction of gene flow. To explore this, we first investigated the possibility of unidirectional gene flow from cave bears to brown bears, and vice versa. For both scenarios, we calculated the expected D statistic values under a variety of demographic scenarios. Using a previously described approach²¹, the expected value of D was calculated for a given underlying demographic model, assuming a range of parameter values for the phylogeny, divergence times, population sizes, gene flow direction, the time and strength of gene flow, and generation times. After obtaining a single value of D expected for each combination of parameter values, we then used approximate Bayesian computation to compare the expected D with the value of D we observed from the sampled genomes. The closer the observed and expected D values are (based on the Euclidean distance), the higher the chance of the particular underlying model(s) generating those D values being close to the true demographic model. However, comparisons did not yield conclusive evidence for gene flow in either direction or for a specific range of demographic parameter values of the underlying model, because the parameter combinations that generated the closest D value covered a range of models that was too wide to point to a specific conclusive subset of scenarios (see Methods). Failure to identify demographic models consistent with the observed patterns may result from several confounding factors, including the accumulation of errors in parameter value estimates, population structure and other model misspecifications (such as bidirectional gene flow) that our model does not take into account.

A phylogenetic test of directional admixture.

To further investigate the direction of gene flow, we implemented a test based on the distribution of rooted tree topologies along a non-overlapping sliding genomic window. This approach is similar in principle to recently developed DFOIL statistics²², but involves comparisons of phylogenetic trees rather than diagnostic nucleotide positions. Our analyses used five individuals: a polar bear, a brown bear, the least admixed Caucasus cave bear (kudarensis), a more admixed European cave bear (spelaeus), and an Asiatic black bear as an outgroup. The ingroup species tree has a symmetrical topology: ((polar, brown),(least admixed cave, more admixed cave)). We divided the aligned haploidized genome sequences into ~50,000 non-overlapping blocks of 25 kb length, and computed the rooted maximum-likelihood phylogeny of each block. Conversion of diploid genome data into a single pseudohaploid sequence complicates phylogenetic analysis of regions where an individual is heterozygous for both admixed and unadmixed alleles. To overcome this problem (see Methods and Supplementary Fig. 3), we grouped each of the twelve possible non-symmetrical topologies into one of four topology classes: class 1, polar bear basal; class 2, brown bear basal; class 3, least admixed cave bear basal; and class 4, more admixed cave bear basal (Fig. 3a). The null hypothesis of no admixture would result in equal representation of classes 1 and 2, and of classes 3 and 4. In the absence of gene flow, the absolute numbers of blocks in each of these sets would reflect the degree of incomplete lineage sorting, which is dependent on effective population size and divergence times within each clade. In contrast, gene flow from cave bears into brown bears would result in an overrepresentation of class 1 relative to class 2. Gene flow in the opposite direction, from the brown/polar bear lineage into the more admixed cave bear, would result in an overrepresentation of class 3 relative to class 4. An overrepresentation in both cases is indicative of bidirectional gene flow.

Fig. 3 |. Test of gene flow direction based on the distribution of rooted tree topologies along a non-overlapping 25 kb sliding window.

Five individuals are used in the test: a brown bear, a polar bear, the least admixed Caucasus cave bear (kudarensis, ‘cave –’), a more admixed European cave bear (spelaeus, ‘cave +’) and the Asiatic black bear outgroup. a, The symmetrical rooted ingroup species tree and the four alternative classes of topology that are informative on gene flow directionality. b, The proportion of 25 kb genomic blocks returning each topology class for eight brown bears, each represented by individual numbered bars: 1, Alaska (ABC Islands), North America; 2, Alaska (Denali), North America; 3, Russia; 4, Slovenia; 5, Sweden; 6, Georgia; 7, Spain; 8, Late Pleistocene Austria. The results support bidirectional gene flow among cave bears and brown bears: overrepresentation of topology class 1 relative to class 2 for all brown bears investigated indicates gene flow from cave bears into brown bears, and overrepresentation of topology class 3 relative to class 4 indicates gene flow in the opposite direction, from the brown/polar bear lineage into the more admixed cave bear. c, The cumulative size distributions of regions returning each topology class, determined by counting the number of consecutive 25 kb blocks returning the same topology. Absolute counts are provided in Supplementary Table 5. Individual points represent the cumulative proportions for individual brown bears. For the purpose of visualization, average cumulative proportions for each topology class are shown as coloured lines centred on the mean proportion observed for each block size and linked by linear interpolation.

We first evaluated the ability of our approach to detect directional admixture using a well characterized system: recent gene flow from polar bears into the brown bear populations inhabiting the Alaskan Admiralty, Baranof and Chichagof^10,12 (ABC) Islands. We divided the aligned pseudohaploid genome sequences of two polar bears, an ABC islands brown bear, a European brown bear and the American black bear outgroup into non-overlapping genomic blocks and computed the maximum-likelihood phylogeny based on transversion sites. After the species tree, the next most abundant topology class comprised non-symmetrical trees in which the ABC Islands brown bear and polar bear formed a clade and the European brown bear occupied the basal ingroup position, which is consistent with gene flow from polar bears into the ABC Islands brown bear population (see Methods and Supplementary Table 4). Based on this result, we concluded that the distribution of rooted tree topologies along a non-overlapping sliding window is a reliable method to test for directional admixture.

Applying this test to the newly generated cave bear and brown bear genome data revealed clear and consistent evidence of gene flow from cave bears into brown bears. Topology class 1 outnumbered class 2 by a ratio of 1.2–1.5 for all brown bears investigated (Fig. 3b), representing an absolute difference of 1,226–3,052 blocks (Supplementary Fig. 4). The overrepresentation was greatest in the Late Pleistocene Austrian brown bear, with a rank order of diminishing levels of cave bear introgression broadly consistent with that inferred from D statistic analyses (Supplementary Fig. 2). Assuming introgressed blocks exist in a heterozygous state, the total genomic overrepresentation of topology class 1 suggests that between 1.2 to 3.1% of the genomes of the brown bears investigated is derived from cave bear introgression (Supplementary Fig. 4), which exceeds f̂estimates. This proportion may still be an underestimate, because some introgressed blocks are likely to exist in a homozygous state. We also found evidence of gene flow from the brown/polar bear lineage into the more admixed cave bear, with topology class 3 outnumbering class 4 for all brown bears by a ratio of approximately 1.1, representing an absolute difference of 196–448 blocks (Fig. 3b and Supplementary Fig. 4).

The length distribution of introgressed genomic blocks in admixed individuals reflects the number of intervening generations (recombination events) separating them from their F1 hybrid ancestor, and can therefore be used to infer the relative timings of different admixture events²³. Of the four topology classes, the largest block sizes were associated with class 1 (13.7 to 15.6% of occurrences spanning more than one 25 kb block), which suggests that the most recent episodes of admixture involved gene flow from cave bears into brown bears (Fig. 3c and Supplementary Table 5). Length distributions are remarkably similar among modern and ancient brown bears, suggesting that all introgressed blocks derive from a similar time period or from the same admixed population. Thus, the reduced admixture proportions observed in modern brown bears relative to the Late Pleistocene Austrian individual probably result from diffusion into less admixed populations, rather than from an additional gene flow event into the ancient population. We also found a marginally larger length distribution for topology class 3 relative to class 4 (6.4 to 8.0% versus 5.4 to 7.2% of occurrences spanning more than one 25 kb block, respectively; see Fig. 3c and Supplementary Table 5). This is consistent with the low-level gene flow from the brown/polar bear lineage into cave bears inferred from the relative abundance of topology classes. Interestingly, we found the length distribution of topology class 2 (11.9 to 13.0% of occurrences spanning more than one 25 kb block; see Fig. 3c and Supplementary Table 5) to be intermediate between class 1 and classes 3 and 4, suggesting that class 2 fragments are not solely the result of lineage sorting, but may reflect earlier gene flow from cave bears into either the common ancestor of polar and brown bears or the population ancestral to polar bears after their divergence from the lineage leading to modern brown bears.

Introgressed cave bear genes potentially under selection in brown bears.

Introgressed cave bear genes in brown bears may increase fitness and undergo positive selection, as suggested for Neanderthal and Denisovan alleles in modern humans^24–26. Identifying such genes is, however, challenging, given our low coverage unphased genomic datasets and fragmented reference genome assembly. With this in mind, we undertook a search for candidate introgressed genes by identifying 25 kb genomic blocks returning a topology consistent with being homozygous for introgressed cave bear alleles (brown bear nested within the cave bear clade) for all eight investigated brown bears. This uncovered 34 blocks which were compared against all UniProt vertebrate sequences in the Swissprot division. Five blocks were found to contain notable matches to UniProt sequences: matches to zinc finger protein PLAGL1 gene in pig, human and chicken; matches to the Titin gene in human; matches to the Tigger transposable element-derived protein 1 gene in human; and two separate blocks with matches to a LINE-1 reverse transcriptase homologue in slow loris and LINE-1 retrotransposable element ORF2 protein in human and mouse. These observations may result from selection on introgressed cave bear genes in brown bears, incomplete lineage sorting or selection on ancestral polymorphism. Although our analysis does not allow rigorous testing of these alternative explanations, it does provide pertinent evolutionary hypotheses for future investigation.

Discussion

Our results provide conclusive evidence of admixture between brown bears and extinct cave bears. This admixture involved gene flow in both directions, but at least the most recent episode of admixture transferred cave bear alleles into recipient brown bear populations. The evolutionary consequences of such admixture are difficult to assess. Introgression of novel alleles may facilitate adaptation when exposed to certain selection regimes^24–27. Equally, such alleles may prove maladaptive, providing short-term fitness reductions and possibly leading to expulsion from the gene pool²⁸. Brown bears are a widespread and successful species that has been recipient of alleles from two ecologically and morphologically divergent lineages in their recent evolutionary history: cave bears and polar bears^10–12. Brown bears thus represent an excellent opportunity to study the biological implications of admixture in a dynamic multispecies system, with the five potentially introgressed loci identified by our functional analysis providing a logical starting point. Substantial progress in Ursid genetics will be required before this opportunity can be exploited fully.

Determining the direction of gene flow is of key importance in the study of admixture since the evolutionary implications of admixture processes strongly depend on the donor–recipient relationship. Previous inferences have generally relied on either high coverage, phased genomic sequence data^2,29, or allele frequency estimates^30,31, requiring either high genomic coverage or prior knowledge of the site frequency spectrum. Inference based on D statistics has been possible when one of the admixing species has extremely low genetic diversity¹², but such approaches cannot be generalized to all species. Thus, none of these approaches are particularly well suited for studies of extinct Pleistocene megafauna, where genomic coverage and the number of individuals sampled are likely to be low. Recently developed DFOIL statistics²²—which, similarly to the D statistic, summarize allelic patterns as continuously distributed statistics—may provide a suitable approach in such cases. However, identification of specific introgressed regions of the genomes of recipient species may be more straightforward using the phylogenetic approach described here, which instead relies on a finite number of diagnostic tree topologies. Furthermore, the phylogenetic approach has the potential to make use of a broad array of existing nucleotide substitution, molecular clock and coalescent-based population models, allowing further analytical refinement and development.

Here, we have shown that studying the distribution of rooted tree topologies along a non-overlapping sliding genomic window is a simple but powerful approach to determine gene flow directionality, which is robust to low genomic coverage and requires data from as few as four individuals. Using this method, we have shown that a fraction of the cave bear gene pool survives in the genomes of living brown bears, mirroring the Neanderthal ancestry found in non-African humans², but at much deeper phylogenetic divergence. This result forces a reevaluation of the very concept of species extinction. At the genetic level, species may survive and participate in the theatre of evolution for tens of thousands of years after their disappearance from the fossil record.

Methods

Samples.

Ancient DNA data was obtained from cave bear and brown bear petrous bones. Data from modern brown bears was obtained from either skin or soft tissue samples. Complete details of all samples are provided in Supplementary Tables 1 and 2. No sample randomization or investigator blinding was undertaken.

Laboratory methods for ancient samples.

All laboratory work preceding library amplification was carried out in dedicated ancient DNA facilities at the University of York or the University of Potsdam, following established guidelines³². The ancient data were generated over a series of experiments, with the aim of optimizing DNA extraction and library preparation. The outcome of these experiments has been described previously¹⁷. The contribution of sequence data from each experiment is reported in Supplementary Table 6. For the majority of data, DNA was obtained from 50 mg of bone, sampled from the densest regions of the petrous bones, then ground to a fine powder using ceramic mortar and pestles, and digested overnight in 1 ml EDTA/proteinase K extraction buffer³³. DNA was then isolated using a published method based on silica columns³³, and eluted in 25 μl TET buffer. 20 μl of each DNA extract was used for preparation of Illumina sequencing libraries. First, DNA extracts were treated with uracil-DNA glycosylase and endonuclease VIII to remove uracil residues resulting from cytosine deamination, which are typically prevalent in ancient DNA fragments³⁴, and then converted into libraries using a published protocol based on singlestranded DNA³⁵. A unique eight base-pair index sequence was incorporated within the P7 adapter sequence of each library during amplification to facilitate data demultiplexing, with the optimal number of PCR cycles applied to each library determined in advance using a qPCR approach³⁵. Amplified libraries were purified using commercial silica spin columns (Qiagen MinElute) and quantified using a Qubit 2.0 fluorometer (ThermoFisher Scientific) and either the Agilent 2100 Bioanalyzer or the 2200 TapeStation Instrument, before paired-end sequencing on Illumina platforms. Full details of alternative laboratory methods and the data obtained are provided in Supplementary Table 6.

The authenticity of the sequences obtained from the ancient samples was verified by mapping sequences to the polar bear reference genome assembly³⁶ and then calculating fragment length distributions and checking for an excess of C to T substitutions at the fragment ends, which is indicative of ancient DNA damage (Supplementary Fig. 5), using the program mapDamage³⁷.

Laboratory methods for modern samples.

DNA was extracted from modern tissue samples using a commercial kit (Qiagen DNeasy). Extract concentrations were measured using Qubit and DNA quality assessed using TapeStation genomic DNA assay. 500 ng of DNA in a volume of 50 μl was then sheared by sonication to an average fragment length of 500 bp using a Covaris S220 System. Sheared DNA was converted into Illumina sequencing libraries using a published protocol based on double-stranded DNA³⁸, with modifications³⁹. Library amplification and indexing was carried out as described previously for single stranded libraries. Library molecules corresponding to insert sizes <300 bp and >1,000 bp were removed prior to sequencing using a PippinPrep instrument (Sage Science). Modern libraries were sequenced on an Illumina NextSeq 500 platform generating approximately 80–200 million 150 bp read pairs for each library.

Processing of sequencing data.

We merged overlapping paired end reads with SeqPrep⁴⁰ and mapped either both merged and unmerged reads (modern samples), or merged reads only (ancient samples), to the reference genome assembly of the giant panda¹⁹ with bwa aln v0.7.7⁴¹. To account for the evolutionary divergence between the reference genome and the samples, we increased the allowed mismatch rate by setting the −n flag to 0.01 rather than the default of 0.04. We excluded reads with a MapQuality score less than 30 and removed duplicate reads with samtools v0.1.19⁴².

We selected the giant panda reference genome rather than the less evolutionarily distant and more contiguous polar bear reference genome³⁶ for evolutionary analyses. As an ingroup to the group of samples being studied, the polar bear reference genome might introduce bias into our mappings that would disproportionately impact admixture inference. Cave bear reads from regions of the cave bear genome potentially introgressed from the polar/brown bear lineage would have a greater probability of mapping to the polar bear genome than reads from other parts of the genome because of their lower divergence. This biased assembly would produce the artefact of an inflated frequency of shared derived alleles between polar bears and cave bears in the cave bear assemblies. By contrast, mapping to the outgroup giant panda reference would have no bias for or against mapping introgressed regions making it a more suitable reference genome for this study.

For analysis, we generated haploidized sequences for each individual by randomly selecting a single high quality base call (BaseQuality ≥30, read MapQuality ≥30) at each site in the panda reference genome. This method better represents non-reference alleles for low coverage samples than genotype calling, which tends to be biased toward the reference allele, potentially confounding downstream analyses². To avoid inclusion of repetitive or duplicated genomic elements, we masked sites where an individual’s coverage was above the 95th percentile of genome wide coverage.

Phylogenetic analysis of aligned nuclear and mitochondrial sequences.

The aligned pseudohaploid nuclear sequences were recoded into binary characters to only score transversions (Rs: 0, Ys: 1) and all aligned columns containing missing data were removed. The maximum-likelihood phylogeny was then computed under the BINGAMMA model with the American black bear as outgroup using RaxML⁴³.

The mitochondrial sequences of the investigated European cave bears have been published previously⁷. A consensus mitochondrial sequence for the Caucasus cave bear kudarensis was generated from the shotgun sequencing data using an iterative mapping approach. First, merged reads were mapped to a published spelaeus mitogenome (Genbank EU327344) using the ‘mem’ algorithm in bwa v0.7.8⁴¹. Reads with MapQuality score less than 30 were removed with samtools, and duplicate reads removed using MarkReadsByStartEnd.jar (https://github.com/dariober/Java-cafe/tree/master/MarkDupsByStartEnd). From this alignment, an alternative reference sequence was created using FastaAlternateReferenceMaker from the GenomeAnalysisToolKit v3.1⁴⁴ based on the variants detected using GATK’s UnifiedGenotyper. Then, merged reads were again mapped to this newly generated reference with the more stringent ‘aln’ algorithm in bwa, and reads with low mapping quality and duplicates removed as described previously. A final consensus was generated from this alignment in Geneious v7.0, using a minimum sequence depth of 3× and a 90% majority rule for base calling.

Mitochondrial sequences were aligned using MUSCLE⁴⁵ and a problematic section of the control region removed, as described previously⁷. The final alignment contained 16,361 aligned nucleotide positions. Phylogenetic analysis was conducted using maximum-likelihood under the GTR-GAMMA model using RAxML-HPC2 8.2.3⁴³ on the CIPRES Portal⁴⁶ using the American black bear as outgroup. Clade support was assessed by 500 bootstrap replicates using the GTR-CAT substitution model.

D statistic tests of lineage differentiation and admixture.

We tested for admixture between cave bears and their nearest extant relatives, polar bears and brown bears, with the D statistic (ABBA, BABA test)^2,21. All D statistics calculated during this study are reported in the Supplementary information (Supplementary Data 1–4). To avoid bias resulting from ancient DNA cytosine deamination (C- > T error) damage, we restricted our analysis to transversion sites. To test for significance, we applied the weighted block jackknife². Because of the low contiguity of the giant panda genome, we used 1 Mb non-overlapping blocks, rather than the 5 Mb non-overlapping blocks used in previous studies^2,10,12, which would exclude most of the panda scaffolds (N50 = 1,281,781)¹⁹. Despite their smaller size, 1 Mb non-overlapping bins are adequate for testing cave bear introgression into brown bears because they are substantially longer than the longest estimated length of introgressed blocks of cave bear ancestry which is 175 kb (Supplementary Table 5). We consider results more than three standard errors different from zero (Z > 3) as strong evidence of admixture, and more than two standard errors different from zero (Z > 2) as providing moderate evidence of admixture.

We investigated four alternative outgroups for admixture tests. These were, in order of increasing phylogenetic distance from the focal clade¹³: American black bear, Asiatic black bear, spectacled bear and giant panda. All outgroups resulted in a significant signal of admixture between all cave bears and all modern brown bears, following the divergence of polar bears and brown bears (Supplementary Fig. 6). In contrast, tests involving the Late Pleistocene brown bear similarly supported admixture between brown bears and cave bears when the less divergent outgroups were used, but the two most divergent outgroups resulted in a reversal of the admixture signal (Supplementary Fig. 6). We attribute this effect to accumulated errors in the ancient pseudohaploid sequences as a result of both sequencing error and spurious read mapping, both of which tend to occur at higher rates in ancient compared to modern DNA datasets. Specifically, at sites where the outgroup has a private allele, accumulated errors in an ancient sample occupying the P2 position would convert a proportion of these BBBA sites to BABA sites, causing that individual to appear unadmixed relative to P1. This effect is amplified with more divergent outgroups since they will posses more private alleles relative to the ingroup. For the American black bear outgroup, we also observed consistently elevated D values relative to those generated using other outgroups, suggesting differential allele sharing between American black bear and P1 and P2 ingroup lineages (polar and brown bear, in this case). We calculated D statistics to assess this imbalance using the giant panda as outgroup, and found a significant excess of derived alleles shared between American black bear and polar bears, relative to brown bears, in almost all comparisons (Supplementary Fig. 7). This may reflect admixture between American black bear and polar bear, admixture with a ghost lineage, ancestral population structure, or the transfer of archaic alleles into brown bears by a cave bear vector, but we did not investigate these alternative explanations further. All other outgroups did not show any clear or consistent imbalance in allele sharing with either ingroup lineage. Overall, we selected Asiatic black bear as the most appropriate outgroup, being sufficiently closely related to the ingroup to avoid artefacts associated with ancient datasets (Supplementary Fig. 6), and showing no consistent pattern of differential allele sharing with either polar bears or brown bears (Supplementary Fig. 7). Asiatic black bear was thus used as outgroup for all subsequent admixture tests.

To quantify the amount of admixture, we used the f̂statistic²¹, which is the excess of shared derived alleles between the admixed individual and candidate introgressor standardized by the maximum excess of shared derived alleles expected in an entirely (100%) introgressed individual. All f̂values calculated during this study are reported in the Supplementary Data 5 and 6. The f̂expected value is best calculated by using individuals that we hypothesize best approximate the diversity within the introgressing populations. We considered the European cave bears to best represent the diversity within a potentially introgressing cave bear lineage. For brown bear introgressors, we selected Eurasian brown bears as best representing diversity in a potential brown bear introgressor. As with the D statistic, we determined significance based on weighted block jackknife with 1 Mb blocks.

Inferring the direction of gene flow based on observed values of D.

In order to obtain more insights into the underlying demographic model, we followed the methodology of a previous study²¹ by calculating the expected counts of ABBA and BABA generated under the model of instantaneous unidirectional admixture (IUA). Populations P1, P2 and P3 represent polar, brown and cave bear, respectively. We assumed two versions of this model, a single admixture event from P3 into P2, and the same model but with the direction of gene flow from P2 into P3. The range of parameter values tested were as follows (following the notation described elsewhere²¹): a constant population size of N_e 10,000 to 30,000 in steps of 5,000 in all populations; a generation time of 13 years; t_P2 800 ka (thousand years ago) to 1.4 Ma (million years ago); t_P3 1.5 Ma to 2 Ma, both in steps of 50 ka; f̂ 0.01 to 0.1 in steps of 0.01 and t_gf 150 ka to t_P2 – 50 ka in steps of 50 ka. For each of the 135.850 combinations of parameter values, we then calculated the expected counts of ABBA and BABA.

We then applied a procedure based on approximate Bayesian computation (described in detail previously⁴⁷) to identify combinations of parameter values that generate expected D values closely matching the observed D. We used the grid of parameter values as discrete uniform prior distributions and the expected D as summary statistics. We kept the closest 1% parameter combinations based on the Euclidean distance between observed and expected D. However, this approach proved inconclusive, because too many different parameter values generated D values that closely matched the observed D. We could not identify a specific conclusive subset of parameter values that would support specific demographic models. The distributions with the kept parameter values were broadly uniform and, therefore, uninformative. Based on these observations, we did not investigate this approach in more detail. In summary, we did not obtain any clear evidence to identify the direction of gene flow or a specific range of demographic parameter values from the IUA model.

Phylogenetic test of directional admixture.

This test is based on the assertion that if an admixed individual of a recipient species possesses an introgressed allele at a given locus, then phylogenetic analysis of that locus will cluster the introgressed allele within the diversity of the donor species and not within the diversity of the recipient species. Such patterns can also result from incomplete lineage sorting; however, using the same principles upon which the D statistic is based²¹, this can be differentiated from admixture by looking for an imbalance in the frequencies of specific topologies across a large number of loci. We devised an implementation of this test using a sample of four individuals plus a suitable outgroup. There are fifteen possible rooted tree topologies. Three of these rooted trees are symmetrical, and we selected individuals such that one of these symmetrical trees represented the species tree. As such, rooted trees that may reflect the possession of introgressed alleles in one individual are non-symmetrical.

Precise phylogenetic placement of putatively introgressed alleles is confounded by the use of unphased pseudohaploid sequences when individuals are heterozygous for both introgressed and non-introgressed alleles (here, allele refers to a linked haplotype sequence spanning a particular region of the genome). Random selection of nucleotides from both introgressed and non-introgressed alleles from such a heterozygous individual would be likely to result in a phylogeny with branch lengths representing an average between the symmetrical species tree and the non-symmetrical admixed tree (Supplementary Fig. 3). In this averaged tree, the position of the admixed individual within the donor species clade may shift, but the overall topology will remain non-symmetrical. For this reason, we grouped the twelve possible non-symmetrical rooted topologies into four topology classes (three topologies each) according to the individual occupying the basal ingroup position, which formed the basis of the directional admixture test (Fig. 3a). This procedure accounts for the variable positioning of introgressed individuals within the donor clade resulting from heterozygosity by exploiting the preservation of tree non-symmetry.

A further consideration for implementing the test is the selection of an appropriate block size. Phylogenetic analysis of whole-genomic alignments returns the reciprocal monophyly of cave bears, brown bears and polar bears (Fig. 1a). Thus, it is likely that phylogenetic analysis of very large genomic blocks would, in general, also return this species tree. A suitable block size is therefore sufficiently small to be able to detect regions resulting from admixture and incomplete lineage sorting, while also being of sufficient size for accurate phylogenetic estimation. We therefore investigated a range of block sizes to identify an appropriate size providing good sampling of topological variation along the aligned genome sequences.

Evaluation of the method using the example of recent gene flow from polar bears into the brown bear populations inhabiting the ABC Islands involved genomic data from two polar bears, an ABC Islands brown bear, a European brown bear, and the American black bear outgroup. Details of read processing, mapping and pseudohaploidisation were as described above. Introgressed polar bear alleles in the ABC Islands brown bear should be detectable as a relative increase in the frequency of genomic blocks returning the ABC islands brown bear and polar bears as monophyletic relative to the number returning the European brown bear and the polar bears as monophyletic. To test this hypothesis, we divided aligned pseudohaploid sequences into non-overlapping blocks of equal sizes and computed the maximum-likelihood phylogeny based on transversion sites as described previously. Blocks that contained >50% missing data for any of the individuals were recorded as having insufficient data and not used for phylogenetic analysis. The topology of each phylogeny was evaluated using a custom Perl script that made use of the ETE3 software⁴⁸. Analysis of both 0.25 Mb and 0.1 Mb block sizes produced topology frequencies consistent with the hypothesized pattern: following the species tree, the next most abundant topology class comprised non-symmetrical trees in which the ABC Islands brown bear and polar bears formed a clade, with the European brown bear occupying the basal ingroup position, which is consistent with gene flow from polar bears into the ABC Islands brown bear population (Supplementary Table 4). Based on this result, we concluded that the distribution of rooted tree topologies along a non-overlapping sliding window is a reliable method to test for directional admixture.

We then applied the test to investigate directional gene flow between cave bears and brown bears. We tested a range of block sizes and found that a block size of 25 kb provided good sampling of topological variation along the aligned sequences. The estimated abundance of topology classes were also consistent using larger block sizes (Supplementary Table 7). At these block sizes, ingroup lineages may not have sorted with respect to the outgroup, leading to erroneous outgroup rooted tree topologies. In the four taxon framework of the admixture test, if the assumed outgroup is allowed to occupy all positions within the tree, there are 105 possible rooted topologies representing three alternative tree shapes. We considered the effect of rooting each of these tree shapes when the assumed outgroup occupies an internal position within the tree (Supplementary Fig. 8). We identified four possible outcomes of this outgroup misspecification. In some cases, both incomplete and complete lineage sorting of ingroup taxa with respect to their sister clade was correctly identified, but with incorrect outgroup positioning. In other cases, we predicted artefactual grouping of taxa into both admixture-informative and admixture-uninformative topologies, but importantly in all cases equilibrium of topology class frequencies are maintained. Thus, although outgroup misspecification may mislead absolute measures of lineage sorting, we predict no systematic bias that may lead to erroneous inference of admixture. We empirically tested this hypothesis by replicating the analysis with the addition of the more distantly related panda, which served as outgroup for the rooting of phylogenetic trees. Aligned sequences from the panda genome assembly were included in each 25 kb block containing the four ingroup taxa and the assumed Asiatic black bear outgroup. A rooted maximum-likelihood phylogenetic tree was computed as described previously for each block, using the panda as outgroup. These were then filtered for only those trees where the ingroup were monophyletic with respect to the Asiatic black bear, using BioPerl⁴⁹ functions. Phylogenies for each of these blocks were then recomputed using the Asiatic black bear as outgroup, and the abundance of alternative topologies counted as described previously. Although the absolute abundance of topology classes was reduced in comparison to the complete dataset, differences in abundance between topology classes were consistent and congruent with admixture patterns inferred from the complete dataset (Supplementary Fig. 9).

Finally, we investigated the potential of biases in the estimation of topology frequencies introduced by missing data by comparing the distribution of missing data among blocks returning each topology (Supplementary Fig. 10). The occurrence of missing data was found to be highly consistent among topologies, indicating that the presence or absence of missing data is not influencing our frequency estimates.

The lengths of potentially admixed genomic regions were estimated from the complete dataset by counting the number of consecutive blocks returning the respective tree topology. Due to the presence of blocks with insufficient data, these measurements are likely to be underestimates. Length comparisons were made by calculating empirical cumulative densities and plotted in R⁵⁰, with all other parameters set at default.

Identification of introgressed genes potentially under selection.

We identified genomic blocks where all eight investigated brown bears had sufficient data for phylogenetic reconstruction, and where phylogenetic analysis indicated that each was nested within the cave bear clade (topologies 5 and 6, Supplementary Table 7). For each block identified, the corresponding sequence of the panda was retrieved and compared against all UniProt vertebrate sequences in the swissprot division using blastx, requiring an E value of 10⁻⁴⁰ or better, and requiring that at least 50% of the UniProt sequence was covered in one or more of the resulting alignments.