Severe bottleneck of ancient Homo populations: insights from computational modeling and relevant fossil evidence

Abstract

Reconstructing ancient population size history is essential for understanding the evolutionary origin of Homo sapiens. We recently developed the fast infinitesimal time coalescent process (FitCoal) and detected a severe population bottleneck occurring approximately 930 thousand years ago. However, two recent studies compared FitCoal and mushi and concluded that the severe bottleneck is a statistical artifact. In this study, we compared the two methods against a benchmark of 10 billion msprime coalescent simulations. We demonstrate that FitCoal achieves both superior speed and accuracy in expected site frequency spectrum (SFS) estimation. Analyses of simulated datasets confirmed that FitCoal reliably recovers the bottleneck, whereas mushi fails under identical conditions. Independent fossil and paleoclimate evidence is consistent with the timing and evolutionary impact of this bottleneck, including associations with hominin dispersals, speciation events, and a subsequent increase in brain size. These findings refine the demographic history of Homo during the Pleistocene and highlight the importance of high-precision SFS computation for revealing critical evolutionary transitions that shaped modern human ancestry.

Introduction

Over the past 4 million years, at least 22 hominin taxa have emerged, yet only modern humans (Homo sapiens) have persisted to the present day (de Sousa et al. 2023). The Early and Middle Pleistocene epochs represent a pivotal period in Homo evolution, during which African H. erectus declined (Di Vincenzo and Manzi 2023) and gave rise to a new species ancestral to modern humans, Neanderthals, and Denisovans (Prüfer et al. 2014). Understanding the evolutionary trajectory leading to modern humans therefore requires reconstructing population size dynamics throughout the Pleistocene.

Site frequency spectrum (SFS)-based methods provide powerful tools for reconstructing population histories (Li and Stephan 2006; DeWitt et al. 2021; Excoffier et al. 2021). We consider the unfolded SFS based on derived allele counts within a single population. It comprises (n−1) allele frequency categories, where n is the number of sampled sequences. Previous studies developed methods to estimate the expected branch lengths for each SFS category and to calculate the likelihoods of empirical SFS under various demographic scenarios (Polanski and Kimmel 2003; Živković and Wiehe 2008). However, it was previously difficult to precisely calculate these expected values. To address this challenge, the fast infinitesimal time coalescent process (FitCoal) was recently introduced to precisely infer both recent and ancient population size histories using the SFS (Hu et al. 2023). FitCoal accurately calculates the expected SFS and associated likelihood. Moreover, high-frequency alleles within the SFS are particularly susceptible to confounding factors, such as positive selection (Li 2011; Vy and Kim 2015), introgression (Tagore and Akey 2025), population structure (Kern and Hey 2017), variance in reproductive success (Freund et al. 2023), and misclassification of ancestral and derived variants (Baudry and Depaulis 2003). Consequently, excluding such overrepresented alleles reduces biases and enhances the robustness and reliability of inferred demographic histories (Bhaskar et al. 2015; Liu and Fu 2015; Hu et al. 2023). Building on previous studies (Excoffier et al. 2013; Bhaskar et al. 2015; Liu and Fu 2015, 2020), FitCoal identifies the most appropriate demographic model by progressively increasing the number of inference time intervals and iteratively maximizing the likelihood. Because the likelihood is expected to increase with model complexity, FitCoal employs a heuristic criterion, using a default threshold of 20% likelihood gain to balance model fit and complexity. Using this approach, present-day human genomic sequences from 3,154 individuals in the 1000 Genomes Project (The 1000 Genomes Project Consortium 2015) and the Human Genome Diversity Project—Centre d’Etude du Polymorphisme Humain panel (Bergström et al. 2020) were analyzed, suggesting the presence of an ancient severe bottleneck that drove human ancestors close to extinction.

Two recent studies have challenged this finding (Cousins and Durvasula 2025; Deng et al. 2025), arguing that the inferred severe bottleneck is a statistical artifact, based on their interpretation that the signal was present only in African populations and absent in non-African populations, as well as on the possibility that distinct demographic models can generate similar SFS patterns (Myers et al. 2008). However, the issue of model nonidentifiability (Myers et al. 2008) is well known, and its effect on FitCoal inference has been evaluated using simulated data when true models are known. Given the same number of inference time intervals, if different demographic models generate identical SFSs, these models will have identical likelihoods and may result in multiple maximum likelihood peaks, referred to as global attractors, which cause the issue of nonidentifiability. After analyzing massive simulated data sets under 173 different demographic models/conditions, we did not observe such an effect within the explored model space (Hu et al. 2023), indicating that SFSs can be used to infer demographic histories (Bhaskar and Song 2014). Therefore, in this study, we compared FitCoal against mushi, the backbone analysis used in the two studies (Cousins and Durvasula 2025; Deng et al. 2025). Overall, our results demonstrate the importance of combining high-precision computation with fossil evidence.

Results and discussion

It has been widely recognized for several decades that current African and non-African human populations share a common evolutionary history prior to their divergence. Our previous study identified a severe population bottleneck occurring approximately between 930 and 813 thousand years ago (Ka) in 12 African and 19 non-African populations (Hu et al. 2023), which is inconsistent with the conclusion of the two studies (Cousins and Durvasula 2025; Deng et al. 2025). To examine this discrepancy, the bottleneck estimates were extracted from our previous study (Hu et al. 2023) without performing additional analyses. We confirmed that both the timing and severity of the bottleneck were virtually identical between African and non-African populations (Fig. 1 and Figure S1).

Figure 1.

The ancient human severe bottleneck (Hu et al. 2023). a) The severe bottleneck inferred in African and non-African populations. Mean values are shown, the time axis is linearly scaled, and a generation time is assumed to be 24 years. b) Estimated bottleneck parameters for 10 African and 19 non-African populations. Two South African populations (San and Bantu) were excluded due to small sample sizes (six and eight individuals, respectively). Estimates are based on previously published results (Hu et al. 2023). Wilcoxon rank-sum test; NS, not significant. P-values are 0.6583, 0.1644, 0.6038, 0.6038, and 0.6638, respectively.

The ability to directly detect the severe bottleneck is largely determined by the expected number of ancestral lineages ($\overline{k}$) tracing back to the time of the severe bottleneck (about 813 Ka). The empirical threshold for $\overline{k}$ is approximately 2.5 (see Figure S58; Hu et al. 2023). If the number of sampled African genomes is six or more, the severe bottleneck is directly detected because $\overline{k}$ exceeded 2.5. Non-African populations underwent the out-of-Africa dispersal, resulting in $\overline{k}<2.5$, which accounts for why the severe bottleneck could not be detected in the non-African populations in the initial stage of our previous study. We further noticed a discrepancy in the estimated effective population size between African and non-African populations between about 800 and 500 Ka, after the bottleneck was relieved. The results of simulations demonstrated that this discrepancy is due to a hidden effect of the severe bottleneck on non-African populations. Therefore, we extended FitCoal to improve its sensitivity, especially for ancestral demographic changes, and subsequently identified the severe bottleneck in non-African populations. Since the signal for the existence of the severe bottleneck is weaker in non-African populations, we observed a broader range of bottleneck estimates in non-African populations (Fig. 1b and Figure S1b).

To further validate the efficacy of FitCoal, we compared population histories inferred using the complete SFS and restricted subsets of the SFS under two bottleneck models, one recent and one ancient. The recent bottleneck model was based on two previous studies (Liu and Fu 2015; Hu et al. 2023), whereas the ancient bottleneck was derived from the African YRI model. Remarkably, the recent bottleneck was accurately inferred even when analyzing only 10% of the SFS categories, corresponding to derived allele frequencies ≤ 0.1 (Fig. 2a). FitCoal effectively captures subtle yet widespread signals in the SFS generated by the recent bottleneck because each coalescent state retains a nonzero probability when traced back to this event (Hu et al. 2023). This inference was obtained under a modified analysis setting, in which the demographic history was inferred when conditioned on three inference time intervals. In contrast, under the standard FitCoal inference procedure, the bottleneck cannot be reliably inferred using only rare derived alleles because the bottleneck signature is weak. For the ancient bottleneck, rare derived alleles failed to capture its signatures (Fig. 2b), because they do not trace back far enough to reach the ancient bottleneck. The ancient bottleneck was inferred when 40% or more of the SFS categories were included. This feature reduces bias from confounding factors, which often have a stronger impact on high-frequency alleles. Although recent work partially addressed this issue by simulating ancestral-derived allele misclassification (Cousins and Durvasula 2025), we took a different approach. Following two previous studies (Bhaskar et al. 2015; Liu and Fu 2015), FitCoal mitigates these confounding influences by specifically excluding excess high-frequency alleles. These results highlight the substantial demographic information contained in different categories of the SFS and support the robustness of FitCoal in reconstructing complex evolutionary histories.

Figure 2.

Implications of FitCoal's high-precision computation. a) FitCoal-inferred population size histories using the complete SFS and minor alleles ($\mathrm{freq}\le 0.1$) under a recent bottleneck model. Shaded bands represent estimated ranges spanning the 2.5th to the 97.5th percentiles. b) FitCoal-inferred population size histories (only medians are shown) under an ancient bottleneck, based on the complete SFS, 10% SFS, and 40% SFS. c) Impact of the ancient severe bottleneck on the SFS. Each SFS was normalized by the total number of derived mutations (ie the sum of the SFS entries). Solid lines show the normalized expected SFS under African and non-African demographic histories, respectively, while dashed lines represent cases without the ancient severe bottleneck. Arrows indicate the onset of allele deficiency in the bottleneck models compared to the models without bottlenecks. d) Population size histories inferred by FitCoal and mushi. Shaded areas denote estimated ranges between the 2.5th and 97.5th percentiles. The black line represents the true model, mimicking the demographic history of African populations. e) Upper panel: expected branch lengths for each SFS category calculated by FitCoal, mushi, and $1\times {10}^{10}$  msprime simulations under the synthetic model. Branch length refers to the expected branch length for each SFS category. Lower panel: deviations from FitCoal estimates. f) Effect of simulation number on the accuracy of averaged branch lengths, compared with FitCoal and mushi estimates under the synthetic model. g) FitCoal-inferred population size histories from simulated SFSs under the synthetic model. Red lines indicate histories inferred from 1,000 simulated datasets (n = 216, L = 100 Mb). h) FitCoal-inferred population size histories using the expected SFS of the synthetic model as input, shown for different inference time intervals. For each interval, the history with the maximum likelihood among 10 independent runs is displayed. i) The maximum log-likelihood values from 10 independent runs, conditional on the number of inference time intervals. The dashed horizontal line indicates the theoretical maximum log-likelihood ($\log (L_{\max }^{\mathrm{global}})=$−1,182.17). The solid triangle marks the best results. The inset shows the maximum log-likelihood values from 10 runs for intervals of 5 and 6. A generation time is assumed to be 24 years. The time axis in (b) is linearly scaled, while the remaining time axes are log-scaled.

It has been reported that SFS truncation was applied to infer demographic histories when no excess of high-frequency alleles was found (Deng et al. 2025). Two earlier studies (Bhaskar et al. 2015; Liu and Fu 2015) first introduced this SFS-truncation approach, which removes excess high-frequency alleles while retaining as many SFS categories as possible. Here, we clarify that it is not necessary to exclude high-frequency alleles if no such excess is present. When an excess of high-frequency alleles is detected, the truncation boundary can be determined using a script released with FitCoal (Hu et al. 2023). The truncation boundary may be flexibly determined when the goal is to examine the impact of different truncation thresholds (eg Fig. 2a, b in this study, and Figure S6 in Cousins and Durvasula 2025).

We then investigated the changes in the SFS caused by the presence or absence of the ancient severe bottleneck event. The bottleneck-free models were constructed from representative models for African and non-African populations, in which the severe bottleneck was manually removed (Figure S2). We found that this bottleneck significantly reduced the proportions of mid- and high-frequency alleles in both African and non-African populations (Fig. 2c). Notably, the reduction was more pronounced in African populations, becoming evident from the 20th SFS category onward, but only from the 50th category onward in non-African populations. This finding is consistent with the broader range of bottleneck estimates observed in non-African populations (Fig. 1b).

We then compared the performance of FitCoal with mushi (DeWitt et al. 2021), as the latter was used to replicate the results of FitCoal (Cousins and Durvasula 2025; Deng et al. 2025). One of their main arguments was that mushi does not detect the severe bottleneck after analyzing datasets from the 1000 Genomes Project, the same datasets used in the FitCoal analysis. To evaluate the detectability of the bottleneck, we used a simulation-based approach, since the true model is known. We simulated genome-wide polymorphic datasets based on the demographic history of African populations. The complete simulated SFSs were used for both methods. FitCoal successfully recovered the bottleneck (Fig. 2d), whereas mushi failed under identical conditions. The log-likelihood values of FitCoal-inferred demographic histories were significantly higher than those inferred by mushi (−1,211.63 vs −2,849.65, two-tailed t-test, P-value < $2.2\times {10}^{-16}$, Figure S3). These outcomes reinforce FitCoal's effectiveness and reliability in identifying the ancient severe bottleneck from genomic data.

To further explore the differences between FitCoal and mushi in inferring population history, we compared their estimations of expected branch lengths, a critical component in calculating the expected SFS. It is known that mushi relies on the joint probability density function of coalescent times, making it susceptible to accumulated numerical errors (Polanski et al. 2003; Polanski and Kimmel 2003; Živković and Wiehe 2008). FitCoal overcomes this limitation by introducing a novel framework, the fast infinitesimal time coalescent process. For a fair comparison, we examined a synthetic demographic model previously associated with misidentification of a severe bottleneck (Deng et al. 2025). The expected branch lengths estimated by FitCoal and mushi, under identical conditions, differed by a mean of 46.42 generations (min = 0, max = 2,071, SD = 161.39), amounting to a total discrepancy of 9,981 generations across SFS categories (Fig. 2e). To assess which method more closely reflects true values, we conducted $1\times {10}^{10}$  msprime coalescent simulations (Baumdicker et al. 2022) as a benchmark. FitCoal's estimates matched the averaged simulation values in 194 of 215 SFS categories (90.2%), with only single-generation deviations in the remaining categories (mean = 0.098, min = 0, max = 1, SD = 0.298, sum = 21) (Table S1). Conversely, mushi exhibited deviations that were two orders of magnitude larger (Fig. 2f), underscoring FitCoal's superior accuracy. Two recent studies reported no differences between mushi estimates and msprime simulations (Cousins and Durvasula 2025), nor between the expected SFSs generated by two distinct demographic models (Deng et al. 2025). This is likely because their analyses were limited to comparisons analogous to the upper panel of Fig. 2e, where the log-scaled axes minimize the differences. Similarly, Pearson's correlation also failed to show differences between the two methods ($r=0.999$, and P-value $<2.2\times {10}^{-16}$). We also compared FitCoal and mushi under the demographic history of African populations (the true model in Fig. 2d). Smaller differences were found (mean = 0.75, min = 0, max = 85, SD = 6.33, sum = 141; Table S2). Furthermore, FitCoal demonstrated remarkable computational efficiency, completing the calculation in 0.00013 CPU seconds. In contrast, mushi and msprime required 1.29 CPU seconds and 4.90 CPU years, respectively; msprime appears slower because the number of simulations performed here is substantially larger than in traditional simulation approaches ($1\times {10}^{10}$ vs ${10}^5$–${10}^6$) (Li and Stephan 2006; Excoffier et al. 2013). These results highlight FitCoal's advantages in both accuracy and computational speed.

Deng et al. (2025) generated an expected SFS under the synthetic demographic model and used it as input for FitCoal, leading to the detection of a spurious severe bottleneck event. However, in our previous study, we used simulated and empirical SFSs, not expected SFSs, as input for FitCoal. We used expected SFSs only for likelihood calculations (Hu et al. 2023). Here, we evaluated the performance of FitCoal under the same synthetic model. The model assumes a very slow population decline to half its original size over 1.3 million years, followed by a recovery within about 0.5 million years. We simulated 1,000 genome-wide polymorphic datasets and found that FitCoal successfully identified the recovery stage, although the slow population decline was not detected due to its weak signal (Fig. 2g). In all 1,000 simulated cases, we did not detect any spurious severe bottleneck events, in contrast to the findings of the previous study (Deng et al. 2025). These results were consistent across all simulated datasets, highlighting the importance of using simulated data to validate conclusions in demographic analyses.

When an expected SFS is used as input for FitCoal, a theoretical global maximum likelihood $(L_{\max }^{\mathrm{global}})$ can be calculated analytically (see Materials and methods). In this case, users should manually increase the number of inference time intervals (m) until $L_{\max }^m$ converges to $L_{\max }^{\mathrm{global}}$ and the inferred demographic histories stabilize. We then applied this approach to the expected SFS generated under the synthetic model. When $m=4$, a spurious severe bottleneck event was observed before convergence to $L_{\max }^{\mathrm{global}}$ (Fig. 2h, i), indicating insufficient model resolution when m is small. When $m\ge 5$, $L_{\max }^m$ converged to $L_{\max }^{\mathrm{global}}$, and the inferred demographic histories stabilized and became consistent with the synthetic model. These results underscore the importance of understanding the distinct statistical properties of expected versus simulated or empirical SFSs when inferring demographic history.

Cousins and Durvasula (2025) used a deep ancestral structure model to explain the severe bottleneck. This model assumes that two large ancestral populations diverged 1.5 Ma and then admixed 300 Ka. However, it remains unclear how these two hominin populations could have remained isolated for 1.2 million years, and why the admixture event was immediately followed by the emergence of modern humans, which has been dated to approximately 300 to 200 Ka in Africa (Galway-Witham and Stringer 2018; Meneganzin et al. 2022). Importantly, they did not note that the severe bottleneck was directly identified in African populations, whereas a hidden effect of the bottleneck was detected in non-African populations. This hidden effect is consistent with coalescent theory and the out-of-Africa dispersal. It also argues against the possibility that the ancient severe bottleneck was falsely inferred due to positive selection, population structure, or sequencing errors. If the inferred demographic histories of African populations had been affected by these confounding factors, the bottleneck would also have been similarly and directly inferred in non-African populations. We also investigated 18 parameter sets for six human population structure models and found that population structure is unlikely to explain the severe bottleneck (Hu et al. 2023).

To further investigate the effect of population structure, we examined a 10-island model (Figure S4a). Since the small effective size during the bottleneck increases the coalescent rate (Mazet et al. 2016), we set an extremely high connectivity among populations between 930 and 800 Ka. We considered two different sampling strategies: 200 sequences from a single population (single sampling) and 20 sequences from each population (balanced sampling). Although the simulated SFSs deviated from the typical L-shape (Figure S4b, c), FitCoal inferred a nearly constant population size over the past ∼1.5 million years, with a large ancestral population in both cases (Figure S4d, e). No ancient severe bottlenecks were detected under these simulated conditions.

The existence of the severe bottleneck is supported by multiple independent lines of evidence, including the gap in the African hominin fossil record (Profico et al. 2016), the timing of ancestral chromosome fusion events (Dreszer et al. 2007; Poszewiecka et al. 2022; Poszewiecka et al. 2023), major cultural and behavioral shifts (Clark and Linares-Matás 2024), and climatic transitions during the Early-to-Middle Pleistocene (Jouzel et al. 2007). The timing of the bottleneck coincides with major episodes of hominin dispersal from Africa to Eurasia (Micarelli et al. 2024; Muttoni and Kent 2024; Li et al. 2026), potentially enabling populations to evade extinction driven by climatic changes. Accumulating evidence also suggests that the evolutionary lineage leading to our species may have begun to emerge shortly after the end of the bottleneck, as indicated by fossil remains dated to ∼773 Ka from Morocco (Hublin et al. 2026). The evolutionary context inferred from the earliest known human facial fossil in Western Europe (Huguet et al. 2025) is also consistent with the demographic decline scenario associated with this bottleneck. Therefore, independent fossil evidence is valuable for distinguishing between different demographic scenarios.

Our findings suggest that this severe bottleneck may have significantly influenced phenotype and behavior during later human evolution. Multiple macroevolutionary events occurred after the rapid recovery from the bottleneck (Fig. 3a), followed by the “muddle in the middle”, a period characterized by prolonged hominin coexistence and pronounced morphological diversity (Bergström et al. 2021; Ni et al. 2021; Fu et al. 2025). In particular, the fixation of chromosome 2 fusion and its possible contribution to reproductive isolation remains a critical unresolved issue in Homo evolution. To further explore the role of bottleneck in adaptation and speciation, we analyzed endocranial volumes and encephalization quotient data from African fossil specimens. Our analysis revealed a significant increase in brain size following the bottleneck (Fig. 3b, Figure S5, and Table 1), suggesting a relationship between population contraction and accelerated cognitive and social evolution. Additionally, recent evidence indicates that the last populations of African H. erectus experienced extreme aridity at the onset of the bottleneck, potentially driving behavioral adaptations critical for improved survival and resource utilization (Mercader et al. 2025).

Figure 3.

Integration of genetic evidence and fossil record. a) Evolutionary history of hominins over the past 1 million years. LCA refers to the last common ancestor shared by Denisovans, Neanderthals, and modern humans. Triangles indicate speciation events. Q1–Q4 denote the Early Pleistocene, Middle Pleistocene, Late Pleistocene, and Holocene, respectively. The time axis is linearly scaled. b) Endocranial volumes and the encephalization quotients of African fossil specimens before and after the severe bottleneck. Central lines of boxplots represent the medians and box limits denote the 25th and 75th quantiles. Statistical significance was determined by a two-tailed Student's t-test.

Table 1.

Endocranial volume and the encephalization quotient of African Homo erectus and the LCA.

Taxon	Specimen	Temporal range	Endocranial volume (cm3)	Encephalization quotient	References
H. erectus (Africa)	KNM-ER 3732	1.90 Ma	750	4.83	Jerison (1973), Holloway et al. (2004), Beaudet and de Jager (2023)
	KNM-ER 3733	1.63 Ma	848	5.46	Jerison (1973), Holloway et al. (2004), Lepre and Kent (2015)
	KNM-ER 3883	1.57 Ma	804	5.18	Jerison (1973), Holloway et al. (2004), Lepre and Kent (2015)
	Daka	1.00 Ma	995	6.41	Jerison (1973), Asfaw et al. (2002), Holloway et al. (2004)
	Buia	1.00 Ma	995	6.41	Jerison (1973), Abbate et al. (1998), Bruner et al. (2016)
	KNM-ER 42700	1.55 Ma	721	4.64	Jerison (1973), Spoor et al. (2007), Neubauer et al. (2018)
	KNM-WT 15000	1.60 Ma	900	5.79	Jerison (1973), Brown et al. (1985), Holloway et al. (2004)
	OH 9	1.48 Ma	1,067	6.87	Jerison (1973), Swisher et al. (1994), Holloway et al. (2004)
	OH 12	1.00 Ma	727	4.68	Jerison (1973), Anton (2004), Holloway et al. (2004)
LCA	Bodo	0.60 Ma	1,250	6.80	Jerison (1973), Clark et al. (1994), Holloway et al. (2004)
	Broken Hill 1	0.30 Ma	1,325	7.21	Jerison (1973), Holloway et al. (2004), Grün et al. (2020)
	Saldanha	0.80 Ma	1,225	6.66	Jerison (1973), Holloway et al. (2004), Braun et al. (2013)
	Omo-Kibish 2	0.23 Ma	1,430	7.77	Jerison (1973), Holloway et al. (2004), Vidal et al. (2022)
	Ndutu	0.40 Ma	1,100	5.98	Jerison (1973), Rightmire (1983), Holloway et al. (2004)

In conclusion, high-precision calculation of expected SFSs is critical for minimizing artifacts in demographic inference as emphasized by recent studies (Hilgers et al. 2025; Ishigohoka and Liedvogel 2025). Collectively, advanced demographic inference and extensive human genomic data promise deeper insights into hominin speciation, the genetic impacts of the severe population bottleneck, and the evolutionary forces shaping modern humans. Our findings contribute to a better understanding of critical evolutionary events within the genus Homo and highlight the importance of integrating genetic, paleontological, and environmental evidence.

Materials and methods

The parameters of the severe bottleneck estimated from African and non-African populations were obtained from our previously published results (Hu et al. 2023). In this study, 12 African (ACB, ASW, ESN, GWD, LWK, MSL, YRI, Biaka, Mandenka, Yoruba, San, and Bantu) and 19 non-African populations (BEB, CDX, CEU, CHB, CHS, CLM, FIN, GBR, GIH, IBS, ITU, JPT, KHV, MXL, PEL, PJL, PUR, STU, and TSI) were included. The Wilcoxon rank-sum test was used to examine whether there was a significant difference between African and non-African estimates in Fig. 1b and Figure S1.

To illustrate the rich demographic information embedded within the SFS, we simulated 200 SFS datasets using msprime (Baumdicker et al. 2022) under a three-stage recent bottleneck model (Fig. 2a), which is the same as that in Fig. 2c of Hu et al. (2023). We also implemented a three-stage ancient bottleneck model (Fig. 2b), the parameters of which are close to the demographic history of African populations (shown in Fig. 2d). Mutation and recombination rates were set at $1.2\times {10}^{-8}$ and $0.96\times {10}^{-8}$ per base per generation, respectively, across 200 sequences of 100 Mb each. The generation time was 24 years. Demographic inference was performed with FitCoal. Ten repeats were performed to maximize the likelihood.

Four models were employed to investigate the effects of the severe bottleneck on present-day human populations (Figure S3). Specifically, bottleneck models I and III from Hu et al. were utilized, with bottleneck I representing a demographic history analogous to the true population history of African populations and bottleneck III analogous to that of non-African populations. Two additional models, modified versions excluding the severe bottleneck, were also derived from these original models. FitCoal calculated expected SFSs for all four scenarios, assuming n = 200 sequences (Fig. 2c).

We employed bottleneck model I as the true model, corresponding to the “severe bottleneck model” described by Deng et al. Using this model, we simulated 1,000 SFS datasets with msprime. We simulated 188 sequences and the lengths of simulated sequences were 100 Mb each. These simulated datasets were then analyzed using mushi (DeWitt et al. 2021) with parameters specified by Deng et al., yielding 1,000 inferred population histories. FitCoal was also used to infer population histories from the same simulated SFS datasets. Following our previous study, 10 repeats were performed to maximize the likelihood and analyze the simulated SFSs. Given a simulated or empirical SFS, FitCoal and mushi may yield two different demographic histories. Their output maximum likelihoods cannot be compared because the expected SFSs calculated by the two methods are substantially different. Therefore, we first calculated the expected SFSs using FitCoal for both demographic histories, as FitCoal is consistent with the results of ten billion coalescent simulations (Table S1). Next, the likelihoods were calculated and compared between the two histories.

The synthetic demographic model was obtained using the scripts from Deng et al. (2025). Initially, this model was generated using an expected SFS calculated by mushi based on a severe bottleneck scenario for an African population. The expected SFS was derived by multiplying branch lengths by the total mutation rate across the analyzed sequence fragments. To assess this synthetic model, we compared expected branch lengths for each SFS category using three independent methods. We assumed that n = 216 sequences. First, we calculated the expected branch lengths using mushi, following the approach described in recent studies (Cousins and Durvasula 2025; Deng et al. 2025). Second, expected branch lengths were calculated using FitCoal version 1.3, freely available at https://doi.org/10.5281/zenodo.4765446 and released alongside this manuscript. Third, we performed simulations with msprime, generating $1\times {10}^{10}$ coalescent trees for a single locus based on the synthetic model and calculating the mean branch length per SFS category. The number of simulations $1\times {10}^{10}$ was selected after evaluating expected branch length accuracy (Fig. 2f). Due to rounding precision considerations, at least $1\times {10}^{10}$ coalescent simulations were necessary for accurate inference of expected branch lengths. These simulations were conducted on a CPU server equipped with an Intel Xeon Gold 6248R processor utilizing 60 parallel threads, with total runtime reported in core-hours. We loaded a precalculated table merely once into FitCoal, and FitCoal's runtime for calculating expected branch lengths was determined as the average of 5,000 repetitions on the same server, using a single thread. Similarly, we calculated the expected branch lengths of FitCoal and mushi (Table S2) under the demographic history of African populations.

To examine the robustness of FitCoal in inferring population demography, we simulated 1,000 SFS datasets under the synthetic model using msprime. Simulation parameters matched those used by Deng et al. (2025), setting mutation and recombination rates at $1.2\times {10}^{-8}$ per base per generation across 216 sequences, each 100 Mb in length. Because the simulated data followed a neutral single-population model, exclusion of high-frequency mutations was unnecessary. FitCoal was run with default parameters, using the default logLPRate of 20 (indicating a 20% likelihood gain) to infer 1,000 population size histories.

When an expected SFS $\overrightarrow{\lambda }=({\lambda }_1,\cdots ,{\lambda }_{n-1})$ of n sequences is rounded and input into FitCoal, the theoretical global maximum likelihood can be analytically calculated as $L_{\max }^{\mathrm{global}}=\prod _{i=1}^{n-1}\frac{{\lambda }_i^{{\xi }_i}{e}^{-{\lambda }_i}}{{\xi }_i!}$, where ${\xi }_i=$ round(${\lambda }_i$). We used the expected SFS, which is 826.65 Mb in length and derived from the synthetic model to perform inference with FitCoal (Fig. 2h, i). The standard FitCoal inference procedure cannot be used to determine the best model because the statistical properties of expected SFSs differ from those of empirical or simulated SFSs. We observed that the likelihood gain between five and six inference intervals is only 0.022% ($\log (L)=$ −1,185.93 and −1,185.67, respectively). Moreover, the two maximum likelihoods converged to $L_{\max }^{\mathrm{global}}$ ($\log (L)=$ −1,182.17). Therefore, the best model was obtained when the number of inference intervals was five. Overall, since expected SFSs cannot be derived directly from samples and are typically used to calculate likelihoods for empirical or simulated SFSs, we recommend against using expected SFSs as input to FitCoal.

We considered a 10-island model, with each island having an effective population size of $1\times {10}^4$ (Figure S4a). The effective population size of each island was assumed to be constant over time. The migration rate was set to 0.1 during 80 to 93 Ka and 1 × 10⁻¹⁰ for other periods. The migration rate refers to the proportion of individuals in population i that migrate from population i to population j per generation. Two sampling strategies were examined. Under single sampling, 200 sequences were sampled from a single population. To speed up the simulation, we simulated 1,000 independent 100 Kb fragments, with the mutation and recombination rates set to $1.2\times {10}^{-8}$ and $1.0\times {10}^{-8}$ per base per generation, respectively. Under balanced sampling, 20 sequences were sampled from each population, and 10,000 independent 10 Kb fragments were simulated. In both cases, the total simulated genome length was 100 Mb. The simulated SFSs were then used as input for demographic inference using FitCoal with default parameters.

The endocranial volume data for the Buia specimen were sourced from Bruner et al. (2016), KNM-ER 42700 from Neubauer et al. (2018), and all remaining specimens from Holloway, Broadfield and Yuan (2004). Encephalization quotients for each specimen were calculated following Jerison (1973). Body mass estimates (in kilograms) used in these calculations were obtained from Carretero et al. (2004), Grabowski et al. (2015), and Jungers et al. (2016).

Author contributions

Conceptualization: S.Z., Y.Z., F.D.V., G.M., W.H., Z.H., Z.S., Y.H.P., and H.L.; Investigation: S.Z., Y.Z., and F.D.V.; Supervision: G.M., Y.H.P., and H.L.; Writing: S.Z., Y.Z., F.D.V., G.M., Z.S., Y.H.P., and H.L.

Supplementary material

Supplementary material is available at Molecular Biology and Evolution online.

Funding

This work was supported by grants of National Natural Science Foundation of China to Y.H.P. (no. 31100273) and to H.L. (no. 32270674), Eastern Talent Plan Leading Project (to H.L.), and CAS Youth Interdisciplinary Team (to Z.S.).

Data availability

FitCoal version 1.3 is freely available at https://doi.org/10.5281/zenodo.4765446. Raw data and code are available at https://doi.org/10.5281/zenodo.19435195.