Teasing apart the sources of phylogenetic tree discordance across three genomes in the oak family (Fagaceae)

Abstract

Background

Gene tree incongruence is a well-documented, but the biological and analytical factors driving phylogenetic discordance remains incompletely understood. In this study, we investigated how different factors contribute to incongruence among gene trees in Fagaceae.

Results

Each dataset produced highly supported topologies, with Fagus and Trigonobalanus consistently placed as early-diverging lineages within the Fagaceae family. However, the cpDNA and mtDNA divided the remaining Fagaceae species into New World and Old World clades, a pattern that sharply contrasted with the phylogenetic relationships inferred from nuclear genome data. These discrepancies between the cytoplasmic and nuclear gene trees likely result from ancient interspecific hybridization within Fagaceae. The decomposition analyses revealed that gene tree estimation error, incomplete lineage sorting, and gene flow accounted for 21.19%, 9.84%, and 7.76% of gene tree variation, respectively. We further revealed that 58.1–59.5% of genes exhibited consistent phylogenetic signals (“consistent genes”), while 40.5–41.9% of genes displayed conflicting signals (“inconsistent genes”). Consistent genes showed stronger phylogenetic signals and were more likely to recover the species tree topology than inconsistent genes. However, consistent and inconsistent genes did not significantly differ in terms of sequence- and tree-based characteristics. By excluding a subset of inconsistent genes, the study significantly reduced inconsistencies between concatenation- and coalescent-based approaches.

Conclusions

This study illustrates how diverse factors contribute to gene tree incongruence, offering new insights into the evolutionary history of Fagaceae.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12870-025-06963-3.

Background

Advances in next-generation sequencing technology have significantly enhanced the use of genomic data to reconstruct the phylogenies of species [1]. These innovations in data acquisition and phylogenomic methodologies have proven effective in resolving complex evolutionary relationships across diverse taxa, such as birds [2, 3], primates [4], and green plants [5]. However, increasing evidence reveals widespread conflict among phylogenetic trees across the tree of life [6–9], complicating our understanding of species evolution. These incongruities can be attributed to various biological processes, such as gene flow, incomplete lineage sorting (ILS), and horizontal gene transfer, as well as gene tree estimation error (GTEE) generated during data analyses [9]. Although numerous studies have explored the underlying causes of gene tree conflict [10–12], the relative contributions of different biological and analytical factors to phylogenetic tree discordance remain poorly understood.

Gene flow is prevalent in plants [13], which can lead to chloroplast genome (cpDNA) capture between hybridizing species, resulting in cytoplasmic-nuclear gene tree conflicts [14–17]. While cpDNA has been widely used to infer phylogenetic relationships and reticulate evolutionary histories in plants, the mitochondrial genome (mtDNA) has been largely neglected due to the low substitution rate and complex structure variation [18, 19]. Although both chloroplast and mitochondrial genomes are typically maternally inherited in flowering plants [20], recent studies have revealed incongruities between cpDNA- and mtDNA-based phylogenetic trees [15, 21, 22], suggesting that these two genomes do not always share a common evolutionary history. Therefore, mtDNA is potentially a crucial phylogenetic tool in plant evolutionary studies.

Hybridization can also lead to conflict among nuclear gene trees. The extent of gene flow is shaped by natural selection, recombination rate, and gene density, resulting in a heterogenous landscape of introgression [23]. Phylogenetic analyses of genomic regions with varying introgression patterns may therefore result in gene tree incongruence [24, 25]. In addition, ILS and GTEE can contribute to gene tree discordance, particularly during rapid speciation events. In such cases, ancient polymorphisms are randomly sorted and preserved across multiple speciation events, resulting in coalescence occurs more recently with non-sister species, than with the sister species [26, 27]. Moreover, limited accumulation of substitutions during a short speciation interval provides minimal phylogenetic signal [9]. Consequently, gene trees may fail to accurately track the true speciation history, leading to gene tree-species tree conflict. To address these issues, concatenation- and quartet-based methods have been employed to infer the species trees based on phylogenomic data containing conflicting phylogenetic signals. The concatenation-based approach combines all single-gene alignments into a supermatrix to increase phylogenetic information [10]. However, this method assumes a shared evolutionary history across genes, which may be violated in some cases like ILS and gene flow [28, 29]. On the other hand, the quartet-based approach infers species trees that accounts for ILS but ignores the influence of GTEE arising from low-quality data [30, 31]. As gene flow, ILS, and GTEE may simultaneously drive gene tree variation [9], distinguishing their relative contributions remains challenging.

The Fagaceae family includes eight recognized genera and approximately 900 species, which are ecologically and economically important forest trees in the Northern Hemisphere [32–37]. Phylogenomic analyses have revealed that Fagaceae experienced rapid radiation following the K-Pg boundary and during the Oligocene to early Miocene, generating the majority of extant lineages [17, 33, 38–40]. Hybridization is common within Fagaceae, and ancient hybridization events have caused cytoplasmic-nuclear discordance as well as incongruities between nuclear gene trees [17, 38, 40, 41]. ILS and GTEE may have also generated conflict between gene trees in Fagaceae, but the relative contributions of these processes remain underexplored [40]. In addition, previous studies employing concatenation- and quartet-based approaches have identified a significant conflict node regarding the phylogenetic relationships among the genera Quercus, Notholithocarpus, Chrysolepis, and Lithocarpus (hereafter referred to as the “QNCL” node) [17, 40], but the causes of this incongruence remain unclear.

In this study, we investigated how different factors contribute to incongruence among gene trees. We first called SNPs from 90 Fagaceae species using a de novo assembled mitochondrial genome of Castanopsis eyrei as a reference and constructed a phylogeny based on the mtDNA dataset. We then examined incongruities among the mitochondrial, chloroplast, and nuclear gene trees. Subsequently, we performed decomposition analysis to quantify the relative contributions of gene flow, ILS, and GTEE to nuclear gene tree variations. Finally, we identified consistent and inconsistent genes based on their likelihood- and quartet-based phylogenetic signals, and assessed whether removing inconsistent genes reduces incongruence between the concatenation- and quartet-based approaches.

Methods

Taxon sampling, mitochondrial genome assembly, and SNP calling

The nuclear DNA data were retrieved from Zhou et al. [17] and consisted of 2124 nuclear loci from 122 individuals representing 90 species across eight genera in the Fagaceae family. All species utilized in this study were identified by Dr. Paul S. Manos and Dr. Baosheng Wang. All voucher specimens have been deposited in South China Botanical Garden Herbarium. Detailed collection information and voucher specimens for each species are summarized in Table S1. The mtDNA data included 115 individuals from 86 Fagaceae species, with short-read sequencing data available in Zhou et al. [17] (Table S1). To obtain mtDNA sequences of Fagaceae species, we extracted Illumina reads of C. eyrei from Zhou et al. [17] and assembled the mitochondrial genome using GetOrganelle v1.7.1 [42]. Contigs with a depth < 25 × were discarded to eliminate contamination from nuclear genome fragments. We further excluded short contigs (< 100 bp) and retained 25 contigs with a total length of 532,260 bp, ranging from 112 bp to 50,731 bp. To further improve the assembly, we aligned Illumina reads to the 25 contigs using Bowtie2 [43] and extracted the relevant reads with the F4 flag using SAMtools [44]. We then assembled these reads with Unicycler [45]. The final assembled mitochondrial genome of C. eyrei consists of four scaffolds with a total length of 568,352 bp. The mitochondrial genome was annotated using the online tool IPMGA (http://www.1kmpg.cn/ipmga/), and 56 annotated mitochondrial genes were obtained.

To call mitochondrial genome SNPs, we randomly sampled three million paired-end reads for each individual using Seqtk v1.4-r122 (https://github.com/lh3/seqtk) and mapped them to the reference mitochondrial genome of C. eyrei using BWA v0.7.17 [46]. The mapped reads were sorted using SAMtools v1.8 [44], and SNPs were called using “HaplotypeCaller” implemented in GATK v4.2 [47]. Low-quality data was filtered using the thresholds “–min-base-quality-score 30” and “–minimum-mapping-quality 30”. In addition, SNPs with a mean depth higher than 300 or lower than 10 were removed. All heterozygous sites were excluded because the mitochondrial genome in plants is haploid. To mitigate the influence of nuclear and chloroplast-derived sequences in the phylogenetic analyses, we blasted the mitochondrial genome of C. eyrei against the nuclear and chloroplast genomes of the same species using BLASTN with a cutoff E-value < 1E⁻⁵. A total of 223 mitochondrial fragments with identity ≥ 95% and length ≥ 150 bp were considered to be nuclear or chloroplast-derived sequences (Table S2) and excluded. Since the phylogenetic analyses include both variant and invariant sites, we retained 8,540 sites (SNPs and invariants) from the mitochondrial genome for subsequent analyses.

We selected C. eyrei as the reference genome for mitochondrial read mapping and SNP calling, because we had sufficient data to assemble a high-quality mitochondrial genome for this species. Moreover, Castanopsis eyrei, belonging to the genus Castanopsis, is closely related to major lineages within the Fagaceae [17, 40], which help to avoid potential bias that could arise from mapping reads to a more distantly related reference genome. It is worth noting that all individuals included in this study exhibited relatively low missing rate (ranging from 0.49% to 15.9%), except for three individuals from the genus Fagus, which showed higher missing rates (43.1–43.9%; Table S1). This high missing rate is likely due to the long divergence time of Fagus from other Fagaceae species. These results suggest that reference bias likely had a minimal impact on mitochondrial SNP calling in this study.

Phylogenetic analyses

Phylogenetic analyses of mtDNA data were performed using two concatenation-based methods: Maximum Likelihood (ML) implemented in IQ-TREE v2.3.6 [48] and Bayesian inference (BI) implemented in MrBayes v3.2.6 [49]. For ML analysis, the best-scoring tree was selected from 1000 ML trees, with topological robustness assessed using 1000 non-parametric bootstrap replicates. For BI, 10,000 trees were sampled from 10 million generations of Markov chain Monte Carlo runs, with one tree sampled every 1000 generations. The first 2500 (25%) trees were discarded as burn-in to ensure stationarity of the chains. The remaining trees were used to generate a strict consensus tree and calculate the posterior probability for each node. PartitionFinder2 [50] was used to identify the optimal partitioning strategy and evolutionary model for each partition based on the Akaike Information Criterion [51]. In ML analyses, the GTRGAMMA model was applied to all DNA sequence partitions, and the models determined by PartitonFinder2 were used for Bayesian analyses.

Phylogenetic analyses of nuclear DNA data were conducted using IQ-TREE and two quartet-based approaches: ASTRAL-III v5.7.3 [52] and SVDquartets v1.0 [53]. ML analysis was performed following the same procedure as described for mtDNA data. For ASTRAL-III analysis, gene trees were generated under the GTRGAMMA model with 1000 fast bootstrap replicates using IQ-TREE, and summarized into a species tree with 1000 coalescent bootstrap replicates. To improve the accuracy of tree inference, branches with supports less than 50% were collapsed. In SVDquartets analysis, all possible quartets were evaluated, and node support was estimated using 500 bootstrap replicates. To compare the tree topologies of Fagaceae constructured based on cpDNA, mtDNA, and nuclear data, we extracted the cpDNA tree from Zhou et al. [17]. This cpDNA tree was constructed using 72 chloroplast genes and includes 115 individuals representing 86 species of Fagaceae.

Quantification of the relative effects of ILS, GTEE, and gene flow on gene tree variation

To assess the relative importance of ILS, GTEE, and gene flow in contributing to nuclear gene tree variation, we conducted a decomposition analysis following the approach developed by Cai et al. [11]. We estimated the levels of ILS, GTEE, and gene flow for each internal node on our ASTRAL-III species tree. First, to simulate the levels of ILS, we modified the key population mutation parameter “theta” when generating gene trees under the coalescent model using the function “sim.coaltree.sp. mu” in the R package Phybase [54]. The parameter “theta” was calculated as the ratio of ML tree branch lengths (mutation units) to ASTRAL branch lengths for each node (coalescent units) [11]. Second, gene flow was quantified using the “reticulation index” statistic developed by Cai et al. [11]. To do that, we defined introgression branches based on the triplet frequencies. For each asymmetric triplet topology, we inferred the two species involved in allele exchange by analyzing triplet frequency distributions. Specifically, the second most frequent triplet configuration was interpreted as evidence of gene flow, with the two phylogenetically closer species within this triplet identified as the introgression donor-recipient pair. All nodes along the two introgression-associated branches on the species tree were annotated as participating in the introgression process. Subsequently, we counted the number of introgression branches for each node in the species tree. This count was then normalized by the total number of triplets associated with the node, resulting in the reticulation index. The index was calculated and visualized using a script developed by Cai et al. [11]. Third, to estimate the level of GTEE, we simulated 2124 gene alignments using Seq-Gen.v1.3.2 [55]. Following Cai et al. [11], we assume a general reversible process (GTR) model and simulate alighmnets of uniform length (800 bp). The GTR model is widely used for phylogenetic analyses based on nuclear genes [48], and the length of the simulated fragments was chosen to approximate the average length of our empirical data. The same strategy of alignment simulation has also been applied in previous studies for estimating the GTEE [11, 56, 57]. Gene trees for each simulated fragment were constructed using IQ-TREE and then summarized on the ASTRAL-III species tree to obtain a bootstrap probability (BP) value for each node, representing gene tree variation caused by GTEE [11]. Finally, empirical gene tree variation was measured as the gene concordance factor using the “-gcf” option in IQ-TREE [48].

To decompose the relative contributions of ILS, GTEE, and gene flow to gene tree variation, we applied multiple linear regression models [58] implemented in the R package “relaimpo”. The regressions were log-transformed, and R² values were used to assess the goodness of fit. We tested four methods: ‘lmg’, ‘last’, ‘first’, and ‘pratt’ [59, 60]. The relative importance of each factor and the associated confidence intervals were calculated using the functions ‘boot.relimp’ and ‘booteval.relimp’ with 100 bootstraps.

Identification of consistent and inconsistent genes between concatenation- and quartet-based approaches

To define consistent and inconsistent genes between different phylogenetic approaches, we focused on a striking conflict node (“QNCL” node) concerning the relationships among the genera Quercus, Notholithocarpus, Chrysolepis, and Lithocarpus. Previous phylogenetic analyses using the concatenation-based IQ-TREE identified Chrysolepis as a sister group to the clade formed by genera Quercus and Notholithocarpus [17]. By contrast, two quartet-based methods yielded different topologies with moderate support: ASTRAL-III placed Lithocarpus as a sister group to Chrysolepis, while SVDquartets positioned Lithocarpus as a sister group to the clade formed by genera Quercus and Notholithocarpus [17]. An approximately unbiased test [61] indicated that these three topologies were significantly different (P < 0.01). We designated the three topologies recovered by IQ-TREE, ASTRAL-III, and SVDquartets analyses as T1, T2, and T3, respectively.

To assess the consistency of phylogenetic signals supporting these topologies, we calculated two measurements (ΔGLS and ΔGQS) developed by Shen et al. [12]. ΔGLS represents the difference in gene-wise log-likelihood scores between two conflicting topologies, whereas ΔGQS represents the difference in gene-wise quartet scores between two topologies [12, 62]. For each pair of conflicting topologies (i.e., T1 vs. T2, T1 vs. T3, and T2 vs. T3), we calculated ΔGLS and ΔGQS for each gene using the Perl script from Shen et al. [12]. Genes were considered as “consistent” when ΔGLS > 0 and ΔGQS > 0 (both supporting the primary topology), or ΔGLS < 0 and ΔGQS < 0 (both supporting the alternative topology). Conversely, genes were considered as “inconsistent” when ΔGLS ≥ 0 and ΔGQS ≤ 0, or ΔGLS ≤ 0 and ΔGQS ≥ 0.

Evaluation of the difference between consistent and inconsistent genes

To investigate which factors may contribute to the inconsistency of concatenation- and quartet-based phylogenetic signals, we compared multiple characteristics between consistent and inconsistent genes. First, we counted the number of consistent and inconsistent genes supporting the three distinct topologies in each comparison using PhyParts [7]. Each gene tree was mapped to the three topologies in order to evaluate gene tree inconsistency. Considering the high error rate during the construction of single gene trees, we set different thresholds of bootstrap support (specifically 0, 10, 30, and 50) to remove unstable branches with low support, thereby reducing noise introduced by randomness.

Second, we calculated four parameters derived from ΔGLS and ΔGQS: absolute ΔGLS, normalized absolute ΔGLS, absolute ΔGQS, and normalized absolute ΔGQS. Normalized absolute ΔGLS was estimated by using the length of gene alignment, and normalized absolute ΔGQS was estimated by using the total number of quarters (Nq). For each gene, Nq was estimated as n × (n-1) × (n-2) × (n-3)/24, where n is the number of tips in the gene tree [12].

Third, we estimated ten sequence- and tree-based parameters to detect potential causes of inconsistent genes. Five of these parameters were estimated based on gene alignment: length of alignment, percentage of parsimony-informative sites, GC content, evolutionary rate determined by pairwise sequence similarity, and relative composition frequency variability (RCFV) [63]. Five parameters were estimated based on gene trees: average bootstrap support, proportion of internal branch lengths to total branch lengths (Treeness) [63], degree of violation of a molecular clock (DVMC) [64], signal-to-noise ratio (Treeness divided by RCFV), and average length of internal branches (IBL) with respect to T1 (or T2 for T2 vs. T3). IBL was calculated using the “compute.brlen” function of the ape package in R [65], while all other parameters were calculated using PhyKIT [66]. For IBL, we constrained each gene tree to recover each alternative topology. When we constrained and optimized a single gene tree to recover T1 (or T2/T3), we calculated the length of its corresponding internal branch relative to T1 (or T2/T3) and observed that the IBL is similar between the inconsistent and consistent genes. Finally, we quantified gene tree-species tree inconsistencies by calculating the normalized Robinson-Foulds (RF) distance between gene trees and species trees inferred from concatenation-based IQ-TREE, quartet-based ASTRAL-III and SVDquartets.

Results

Phylogeny construction based on the mitochondrial genome

Phylogenetic analyses based on mtDNA data yielded a topology, with strong support for major nodes along the backbone (BS > 80% and BI > 95%; Fig. S1). In the mtDNA tree, the genera Fagus and Trigonobalanus were positioned as early-diverging lineages within the Fagaceae family (Fig. 1; Fig. S1 and S2). The remaining species with hypogeous seeds (hereafter referred to the “HS clade”) were clustered into the Old World (OW) and New World (NW) clades (Fig. 1; Fig. S1 and S2). The OW clade included all species found in Eurasia, such as the genera Lithocarpus, Castanea, and Castanopsis, as well as the subgenus Cerris of the genus Quercus. The NW clade comprised all species found in North America, including the genera Notholithocarpus and Chrysolepis, along with the subgenus Quercus of the genus Quercus. Notably, Eurasian white oak species were grouped together with their North American counterparts from the section Quercus.

Fig. 1.

Conflicts between cytoplasmic (mtDNA/cpDNA, left) and nuclear (nuDNA, right) phylogenetic trees. Left panel: topology of the maximum likelihood (ML) tree inferred from mtDNA data. In the mtDNA tree, the HS clade consists of six genera divided into two major clades: NW and OW. Color (red and blue) nodes indicate consistent relationships between mtDNA and cpDNA, with phylogenetic support ≥ 95% in both ML and BI analyses on mtDNA tree (red) and support < 95% in any one of the methods (blue). Black nodes indicate inconsistent relationships between mtDNA and cpDNA. Right panel: topology of the ASTRAL-III species trees derived from nuclear genes (adapted from Zhou et al., 2022). Nodes showing consistent relationships between ASTRAL-III, SVDquartets, maximum likelihood, and MrBayes are marked in red (phylogenetic support ≥ 95% in all four analyses) or blue (support < 95% in any one of the four analyses). Nodes displaying conflicting relationships across analyses are marked with black dots. Different colors represent distinct genera and various groups of Quercus

Further division within the OW and NW clades revealed three non-monophyletic genera (Quercus, Trigonobalanus, and Notholithocarpus) and six non-monophyletic sections (Quercus, Virentes, Ponticae, Protobalanus, Cerris, and Ilex) of the genus Quercus (Fig. 1; Fig. S1 and S2). Specifically, the OW clade was split into six major subclades with unresolved relationships. Two species from the section Ilex formed an early-diverging subclade, while the remaining Ilex species were either grouped with species from the sections Cerris or Cyclobalanopsis. The other three subclades were each composed of species from the genera Castanea, Castanopsis, and Lithocarpus. The NW clade was divided into two major subclades. One subclade contained Eurasian white oak species (section Quercus), Q. pontica (section Ponticae), and two individuals of the genus Notholithocarpus (Fig. 1; Fig. S1 and S2). The other subclade included species from the genera Notholithocarpus and Chrysolepis, as well as species from five sections (Quercus, Virentes, Ponticae, Protobalanus, and Lobatae) of the genus Quercus (Fig. 1; Fig. S1 and S2). Species from the sect. Lobatae formed a strongly supported clade (BS = 100, BI = 100), while the remaining species clustered into three clades, each containing 2–4 species from different sections (Fig. 1; Fig. S1 and S2).

The topology of the mtDNA tree was largely congruent with that of the cpDNA tree (Fig. 1; Fig. S2), but both were notably incongruent with the nuclear gene trees (Fig. 1). This discrepancy resulted in the identification of nine non-monophyletic genera and sections in the cpDNA and mtDNA trees, as described above. Additionally, we observed several conflicts between the mtDNA and cpDNA trees regarding the placements of 3–4 species within the genera Castanopsis and Lithocarpus, with middle to strong supports for the conflicting nodes (Fig. S2). Other conflicts were found in the genera Castanea and Quercus, where the mtDNA or cpDNA trees generally exhibited low supports for the conflicting nodes (Fig. S2), likely due to stochastic error related within population variation for the two organelle genomes.

Contributions of ILS, GTEE, and gene flow to nuclear gene tree variation

Decomposition analysis revealed that these three factors collectively explained 38.79% (R² = 0.3879) of total gene tree variation across internal nodes using the “lmg” algorithm (Fig. 2). GTEE was the most dominant factor, accounting for 21.19% of variation (Fig. 2B). ILS and gene flow explained 9.84% and 7.76% of the variation, respectively (Fig. 2A and 2 C). The relative contribution of these three factors was consistent across regression methods and bootstrap replicates, expect that gene flow explained more variation than ILS when using the “last” algorithm (Fig. S3A). In addition, we found no significant correlation between ILS and gene flow or between GTEE and gene flow (P > 0.05; Pearson’s correlation test; Fig. S3B), suggesting that the effect of gene flow was distinguished from those of the other two factors. However, ILS and GTEE were positively correlated (Pearson’s r = 0.410, P < 0.001; Fig. S3B), suggesting that the influence of these two factors on gene tree variation cannot be fully separated.

Fig. 2.

Relative contributions of ILS, GTEE, and gene flow to nuclear gene tree variation across Fagaceae. A ILS. Nodes are colored according to the inferred population mutation parameter theta. B GTEE. Nodes are colored based on BP values, which represent the percentage of recovered nodes from simulations. C Gene flow. Nodes are colored according to the reticulation index. D Gene tree variation. Nodal BP values reflect the recovery of nodes in gene trees. The percentages of gene tree variation attributed to ILS, GTEE, and gene flow are indicated by red numbers

Consistent and inconsistent genes between the concatenation- and quartet-based approaches

For each pairwise comparison of topologies (i.e., T1 vs. T2, T1 vs. T3, and T2 vs. T3), we estimated ΔGLS and ΔGQS for each gene. The distribution of both ΔGLS and ΔGQS revealed that the three topologies were supported by a comparable number of genes, although the number of genes supporting T3 was slightly lower than the number of genes supporting T1 and T2 (Fig. 3; Tables S3–S5). Specifically, in the comparison of T1 vs. T2, 50.4% and 50.8% of genes supported T1 based on ΔGLS and ΔGQS, respectively (Fig. 3B; Table S3). In the comparison of T1 vs. T3, 55.2% and 56.4% of genes supported T1 based on ΔGLS and ΔGQS, respectively (Fig. 3B; Table S4). Finally, in the comparison of T2 vs. T3, 55.1% and 53.8% of genes supported T2 based on ΔGLS and ΔGQS, respectively (Fig. 3B; Table S5).

Fig. 3.

Dissecting incongruence between three topologies in the phylogenomic data matrix. A Schematic representation of the relationships among the genera Chrysolepis and Lithocarpus and the clade formed by the genera Quercus and Notholithocarpus (QN), as recovered by concatenation-based IQ-TREE (T1), quartet-based ASTRAL-III (T2), and SVDquartets (T3) analyses. B Distributions of ΔGLS and ΔGQS across 2124 genes in three comparisons: T1 vs. T2 (upper), T1 vs. T3 (middle), and T2 vs. T3 (lower). ΔGLS (above the y-axis) and ΔGQS (below the y-axis) values were calculated by measuring the difference in gene-wise log-likelihood scores and the difference in gene-wise quartet scores for each pair of conflicting topologies. In each comparison, red and green bars represent genes supporting the two conflicting topologies (e.g., T1 and T2), respectively. The numbers of consistent and inconsistent genes in each comparison are provided below the plot

We further examined the proportions of consistent and inconsistent genes based on ΔGLS and ΔGQS in each comparison. In the comparisons of T1 vs. T2, T1 vs. T3, and T2 vs. T3, 58.8%, 59.5%, and 58.1% of genes were consistent, respectively (Fig. 3B; Tables S3–S5). Among these consistent genes, 30–35.6% had ΔGLS > 0 and ΔGQS > 0, which was slightly higher than the proportion (23.9–28.8%) of genes with ΔGLS < 0 and ΔGQS < 0. Among inconsistent genes, the proportion (19.7–21.6%) of genes with ΔGLS > 0 and ΔGQS < 0 was comparable with the proportion (20.3–20.8%) of genes with ΔGLS < 0 and ΔGQS > 0 (Fig. 3B).

To investigate the factors contributing to the phylogenetic signal inconsistencies, we compared characteristics between consistent and inconsistent genes, yielding several significant findings. First, consistent genes exhibited higher values of absolute ΔGLS, normalized absolute ΔGLS, absolute ΔGQS, and normalized absolute ΔGQS in all three comparisons (Fig. 4A; Tables S3–5). Second, the proportion of consistent genes recovering T1, T2, or T3 was generally higher than the proportion of inconsistent genes recovering the three topologies (Fig. 4B). This observation held true even when different bootstrap thresholds were applied to filter out low-support genes, although the proportion of both consistent and inconsistent genes supporting the three topologies decreased as the threshold increased. These results suggested that inconsistent genes are less likely to recover the T1, T2, or T3 topology. We further compared consistent and inconsistent genes using ten sequence- and tree-based parameters, but found no significant differences between them (P > 0.05, Wilcoxon-Mann Whitney U-test; Fig. S4; Tables S6 and S7). Moreover, when performing comparisons with concatenation- or quartet-based trees, we found a similar level of gene tree discordance between consistent and inconsistent genes (Fig. S4; Table S8).

Fig. 4.

Comparison of characteristics between consistent and inconsistent genes. A Differences between consistent and inconsistent genes in terms of four parameters (absolute ΔGLS, normalized absolute ΔGLS, absolute ΔGQS, and normalized absolute ΔGQS) for each pair of topologies. All four parameters significantly differed (P < 0.01, Wilcoxon-Mann Whitney U-test) between consistent and inconsistent genes. B The numbers of consistent and inconsistent genes supporting the three distinct topologies (T1, T2, and T3) under different bootstrap thresholds (i.e., 0, 10, 30, and 50). Upper panel: T1 vs. T2; middle panel: T1 vs. T3; lower panel: T2 vs. T3. The three topologies are presented in Fig. 3A

We tested whether the removal of inconsistent genes reduced incongruities between concatenation- and quartet-based trees. By excluding 208 (9.79% of all genes) inconsistent genes identified in all comparisons (Fig. S5), phylogenetic analyses of IQ-TREE, ASTRAL-III, and SVDquartets produced a congruent topology based on the reduced dataset. All three methods consistently resolved the genus Chrysolepis as a sister group to the clade formed by the genera Notholithocarpus and Quercus, with support values of 99, 82 and 67 in IQ-TREE, ASTRAL-III, and SVDquartets, respectively (Fig. 5).

Fig. 5.

Phylogenetic relationships of Fagaceae species inferred using IQ-TREE based on nuclear DNA data, excluding the 208 inconsistent genes. The topology is congruent with the original maximum likelihood analysis based on the full dataset (T1 in Fig. 3A). Nodes showing consistent relationships between maximum likelihood, ASTRAL-III, and SVDquartets are marked in red (phylogenetic support ≥ 95% in all three analyses) or blue (support < 95% in any one of the three analyses). The red arrow indicates the node placing the genus Chrysolepsis as a sister group to the clade formed by the genera Notholithocarpus and Quercus. The color scheme for lineages is consistent with that used in Fig. 1

Discussion

Conflicts between cytoplasmic and nuclear gene trees

Phylogenetic analyses based on the mitochondrial genome yielded a topology, which closely mirrored that of the cpDNA tree but significantly differed from that of the nuclear gene trees in terms of the relationships among major lineages within the HS clade (Fig. 1). Both the mtDNA and cpDNA trees divided the HS clade into NW and OW sub-clades, reflecting their geographic distribution patterns [33, 36]. In contrast, nuclear gene trees divided the HS clade into six monophyletic genera, representing their current taxonomic status [17, 40, 67]. The discrepancies between the cpDNA and nuclear gene trees have been attributed to interspecific hybridization in Fagaceae [17, 40], which may also explain the conflict between the mtDNA and nuclear gene trees. The chloroplast and mitochondrial genomes are materially inherited and transmitted via seed, while the nuclear genome is biparentally inherited and transmitted via both pollen and seed in angiosperms [19]. The higher transmission capability of pollen compared with seed often leads to capture of both chloroplast and mitochondrial genomes between hybridizing species, resulting in discordance between cytoplasmic (cpDNA and mtDNA) and nuclear gene trees [68, 69]. Horizontal gene transfer (HGT) has been also proposed as a contributing factor to discrepancies between cytoplasmic and nuclear gene trees [70, 71]. However, HGT generally transmits small genomic fragments rather than the simultaneous transfer of entire chloroplast and mitochondrial genomes between incompatible species [72–75]. Therefore, we concluded that the major conflicts between cytoplasmic and nuclear gene trees are primarily due to hybridization events among Fagaceae species.

We also identified several conflicting nodes between the mtDNA and cpDNA trees. These discordances may be attributed to the differences in substitution rates between the mitochondrial and chloroplast genomes [1, 76, 77], or occasional paternal leakage of mtDNA or cpDNA fragments in hybrids [21, 78]. Conflicts between mtDNA and cpDNA trees have also been documented in other plants such as Heuchera [15] and pea [22]. These findings suggest that the chloroplast and mitochondrial genomes may not share a strict common evolutionary history. With the advent of third-generation sequencing technologies, an increasing number of mitochondrial genomes are being assembled [19]. Future studies of mtDNA trees will complement cpDNA trees, collectively elucidating the full evolutionary history of cytoplasmic genomes.

The roles of ILS, GTEE, and gene flow in shaping nuclear gene tree discordance

Despite the strong support for topologies inferred by concatenation and quartet-based approaches, a large number of conflicting signals were observed among nuclear gene trees [17, 40]. Decomposition analyses revealed that GTEE was the most significant factor leading to gene tree discordance, which is consistent with the results of other studies [11, 56, 57, 79]. This prominence of GTEE is likely due to the short sequence fragments used in these studies, which contain fewer informative sites, thereby increasing GTEE [80, 81]. Our hypothesis is supported by simulation analysis showing a negative correlation between GTEE and fragment length [11]. Moreover, a recent phylogenetic study of Fagaceae that employed long fragments (mean length of ca. 1,000 bp) demonstrated that GTEE contributes less to gene tree discordance than ILS and gene flow [40]. GTEE can also arise from incorrectly assumed nucleotide substitution models in phylogenetic inference [1]. Although multiple approaches have been applied to mitigate the effect of GTEE when constructing the phylogeny of Fagaceae [17], these methods may have not fully captured the complex evolutionary processes occurring across different sites and lineages. Future studies utilizing more accurate evolutionary models for long fragments or genome-scale datasets will likely reduce GTEE, thereby minimizing gene tree conflict [11, 82, 83].

ILS and gene flow were both identified as significant factors contributing to gene tree discordances in the Fagaceae. ILS is more likely to occur during rapid successive speciation events, especially when new lineages descend from ancestors with large effective population sizes (N_e) [84, 85], leading to inconsistencies between gene trees and species trees [28, 86]. The Fagaceae family underwent multiple episodes of rapid diversification during the Cenozoic era [17, 40], and many of these species have large N_e during their evolutionary history [87–89]. Therefore, ILS that occurred during these periods likely contributes to the observed conflicts among gene trees. Moreover, there is a strong correlation between ILS and GTEE, making it challenging to distinguish between their effects. Consequently, the contribution of ILS to gene tree conflicts may be higher than currently estimated. The correlation between ILS and GTEE has been reported in previous study [11], and may be explained as short internal branches and large Ne lead to strong ILS and elevated estimation error.

The role of gene flow in gene tree discordance may also be underestimated. For example, widespread gene flow has been detected within the subgenus Quercus [17, 38, 40, 41, 90]. However, relatively low reticulation index values were estimated for internal nodes within this subgenus (Fig. 2), suggesting that many hybridization signals might not be accurately detected. Moreover, decomposition analyses rely on the estimation of parameters for each factor and the selection of regression models, both of which inherently carry limitations [11]. For instance, our simulation dataset was generated using a GTR model and alignments of 800 bp. These assumptions that do not capture the full variation in empirical datasets, which may introduce bias in GTEE estimation. Future improvements in simulation methods, such as incorporating gene-specific parameters or varying sequence lengths and evolutionary models, could lead to more accurate and realistic estimates of GTEE. Consequently, errors from both the simulation and regression processes may make these percentage estimates less precise. These uncertainties may explain why only 38.79% of gene tree variation was explained by ILS, GTEE, and gene flow. Similarly, previous studies have shown that substantial portion (26.46 − 33.80%) of gene tree variation remain unexplained in decomposition analyses. Nevertheless, the relative impact of biological and analytical factors on gene tree variation, as inferred in this study, can provide valuable insights to understand the conflicts among gene trees and to develop improved methods for species tree construction.

Removing inconsistent genes reduces incongruities between concatenation- and quartet-based trees

Approximately 40% of genes exhibited inconsistent phylogenetic signals between concatenation- and quartet-based analyses. In comparison with consistent genes, inconsistent genes showed lower levels of phylogenetic signals and were less likely to recover the species tree topologies. Previous studies have indicated that removing problematic sequences can mitigate errors to some extent and enhance the stability of phylogenetic outcomes [62, 91]. After excluding a subset of the most inconsistent genes, we significantly eliminated inconsistencies among concatenation- and quartet-based approaches, achieving a topology congruent with the original concatenation tree. All three methods consistently placed the genus Chrysolepis as closely related to the genera Quercus and Notholithocarpus. This finding is in line with the current geographic distribution of these species. The genera Chrysolepis and Notholithocarpus are relics of western North American, while Lithocarpus is widely distributed in Southeast Asia [33]. Thus, the close relationship of the two North American genera likely reflect their shared biogeographic history.

It is worth noting that the results obtained after removing inconsistent genes should be interpreted with caution. First, earlier studies have suggested that concatenation trees are more reliable than coalescent trees under conditions of low ILS and high GTEE, whereas coalescent trees are more accurate than concatenation trees under conditions of high ILS and low GTEE [12, 30, 31, 92]. Given that our data exhibited high levels of both ILS and GTEE, we speculated that both methods might have their limitations, potentially leading to reduced reliability in accurately resolving phylogenetic relationships. Second, simulation analyses have suggested that trees obtained by removing inconsistent genes do not necessarily represent the true species tree under the condition of high levels of ILS and GTEE [81]. Furthermore, we found no significant differences between consistent and inconsistent genes in terms of internal branch lengths and gene tree conflicts, indicating that levels of ILS and GTEE are similar between them. The Fagaceae family has undergone ancient rapid speciation and gene flow [17, 40], making it extremely challenging to resolve relationships among certain taxa. Future studies employing novel methods, such as synteny-based phylogenetic inferences [93], may shed light on the diversification history of these species. However, it is important to note that these approaches rely heavily on high-quality and complete genome assemblies, along with accurate gene annotations [93]. The availability of such genome assemblies, particularly for key lineages in Fagaceae, is crucial for facilitating further research in this area.

Conclusions

This study utilized mtDNA data to corroborate that nuclear-cytoplasmic discordance in Fagaceae stems from ancient interspecific hybridization and highlighted the complexity of gene tree incongruence in Fagaceae, demonstrating that gene tree estimation error, incomplete lineage sorting, and gene flow jointly drive phylogenetic heterogeneity. Although consistent and inconsistent genes do not differ functionally or structurally, filtering subsets of genes can effectively reconcile conflicts between analytical methods. These findings provide new insights into the evolutionary history of Fagaceae and emphasize the importance of integrating multiple factors in phylogenetic studies.

Abbreviations

ILS

Incomplete lineage sorting

GTEE

Gene tree estimation error

cpDNA

Chloroplast genome

mtDNA

Mitochondrial genome

HGT

Horizontal gene transfer

RF

Robinson-Foulds

OW

Old World

NW

New World

ML

Maximum Likelihood

BI

Bayesian inference

BS

Bootstrap support

RCFV

Relative composition frequency variability

DVMC

Degree of violation of a molecular clock

Authors’ contributions

B.W. designed the research. B.F.Z., Z.S. and Y.S. collected the experiment materials. Y.Y.L, J.S.W and R.X.Y ran the experiments and performed the statistical analysis. S.J.L, W.J.L, Q.Q.L, J.W.N and L.J.Q conducted the data analysis. Z.S. and B.F.Z performed the data analysis and wrote the manuscript. B.W. and P.S.M revised the manuscript. All authors read and approved the final manuscript.

Funding

This work was supported by the National Key R&D Program of China (2024YFF1306601), Guangdong Flagship Project of Basic and Applied Basic Research (2023B0303050001), Guangdong Basic and Applied Basic Research Foundation (2023A1515110098), Guangdong Science and Technology Plan Project (2023B1212060046), National Natural Science Foundation of China (NSFC 32161123003).

Data availability

The short-read whole-genome sequencing data analyzed in this study have been deposited in Genbank under accession No. PRJNA773751. The alignments of nuclear genes and plastomes analyzed in this study have been deposited in the Dryad digital data repository (https://doi.org/10.5061/dryad.vq83bk3tc). The assembled mitochondrial genome of Castanopsis eyrei and the alignments of mitochondrial DNA sequences generated in this study have been deposited in the figshare database (https://doi.org/10.6084/m9.figshare.28014950).

Declarations

Ethics approval and consent to participate

The collection of all samples fully complied with national and local legislation. In accordance with national and local laws and regulations, special permission was granted for the collection of plant samples in this study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.