Lessons Learned from Bugs in Models of Human History

Abstract

Simulation plays a central role in population genomics studies. Recent years have seen rapid improvements in software efficiency that make it possible to simulate large genomic regions for many individuals sampled from large numbers of populations. As the complexity of the demographic models we study grows, however, there is an ever-increasing opportunity to introduce bugs in their implementation. Here, we describe two errors made in defining population genetic models using the msprime coalescent simulator that have found their way into the published record. We discuss how these errors have affected downstream analyses and give recommendations for software developers and users to reduce the risk of such errors.

Simulation plays a central role in population genomics studies. Recent years have seen rapid improvements in software efficiency that make it possible to simulate large genomic regions for many individuals sampled from large numbers of populations. As the complexity of the demographic models we study grows, however, there is an ever-increasing opportunity to introduce bugs in their implementation. Here, we describe two errors made in defining population genetic models using the msprime coalescent simulator that have found their way into the published record. We discuss how these errors have affected downstream analyses and give recommendations for software developers and users to reduce the risk of such errors.

Main Text

In the effort to build more realistic simulations of genetic diversity, scientific software developers often focus on computational speed and biological realism. As the models simulated become more realistic, however, they also become more complex and difficult to specify. The interface through which users define their models is therefore increasingly important. Without an intuitive and thoroughly documented interface, it is very difficult to simulate complex population models correctly.

The msprime coalescent simulator1, 2, 3 is now widely used in genetics studies. Much of its appeal is the large increase in efficiency over the classical ms program,⁴ which makes it feasible to simulate large samples of whole chromosomes for the first time. Another distinct advantage of msprime is its Python application programming interface (API), which greatly increases the flexibility and ease of use over the standard approach of text-based command line interfaces. In particular, programs like ms require users to specify cryptic command line options to describe demographic models. For example, the Gutenkunst et al.⁵ demographic model (which is the subject of this note), as written in ms syntax, is shown in Figure 1A. This model is relatively simple, and models with many more populations and parameters are increasingly common. Descriptions of such models in ms syntax are not easy to comprehend. The Python interface for msprime, by contrast, allows the user to state models in a more human-readable and programmatic manner and has many advantages over ms’ command line interface.

Figure 1.

Implementations of Demography in Coalescent Simulations

(A) The Gutenkunst et al. (2009)⁵ human expansion model as written in ms syntax.

(B) The same human expansion model with demographic events written in msprime syntax. In earlier versions of the msprime documentation, the highlighted line 17 was missing, so population structure erroneously continued into the past.

(C) A simulation of this model in Martin et al. (2017)⁶ omitted demographic events in the simulation function (highlight). This omission meant that populations remained distinct in the distant past and had excess $F_{ST}$.

(D) In msprime version 1.0, models will be specified so that population configurations and demographic events are coupled, reducing the possibility of making such errors of omission.

Even when using a high-level programming language like Python, however, implementing multi-population models of demographic history is difficult and prone to error. In this note, we discuss two implementation errors that arose through unfortunate design decisions in msprime’s demography API and that then found their way into the scientific record. The first error has relatively mild effects on genetic diversity but was used in many publications, whereas the second error was used only once but had a large impact on the simulation results. In light of these implementation errors, we discuss improvements to msprime’s API motivated by these discoveries and, more generally, best practices for implementing and simulating complex multi-population demography.

Case 1: A Misspecified Model in msprime’s Documentation

To illustrate the demography API, msprime included a description of a widely used three-population Out-of-Africa model⁵ as part of its tutorial documentation. In this model (Figure 2A), Eurasian (CEU and CHB) and African (YRI) populations split from each other in the deep past, followed by a more recent split of European and Asian populations with variable rates of continuous migration between each of the populations. Regrettably, the implementation in the msprime tutorial was incorrect. Before the time of the split of African and Eurasian populations, when there should have been just a single randomly mating population, migration was allowed to occur between the ancestral population and a second population with size equal to the Eurasian bottleneck size for all time into the past (Figure 2B). This incorrect model was introduced into the tutorial for msprime version 0.3.0 and remained in the documentation for around 4 years.

Figure 2.

Expected Diversity Statistics under the Gutenkunst et al. Model

(A) The correctly implemented model. Dashed arrows depict continuous migration.

(B) The incorrectly implemented model from the msprime tutorial with migration continuing into the past beyond the mass migration event with proportion 1 from the ancestral population to the bottleneck population.

(C) Marginal allele frequency spectra under the two models. Heterozygosity in the incorrect model is inflated by $\sim 3.5\text{\%}$, although the general shapes of the distributions are qualitatively similar.

(D) Similarly, the increased heterozygosity leads to excess $D^2$, although the LD-decay is qualitatively similar between models.

(E–G) We compared the true size history for each population under this model to their expected inverse coalescent rates, which are often interpreted as $N_e$. The correct and incorrect models (A and B) are equivalent in the recent history, so the inverse coalescence rates only differ in the more distant past. For multi-population demographic history, the inverse coalescence rates are not expected to match the “true” historical size along each branch because population structure and migration between populations change coalescent probabilities over time.

Fortunately, the effects of this error are subtle. Population sizes and structure since the time of the earliest split are unaffected, so differences in expected $F_{ST}$ are negligible between the correct and incorrect models. However, the ancient structure distorts the distribution of early TMRCAs (times to most recent common ancestor) (Figures 2E–2G). The extraneous ancient population increases the long-term effective population size, resulting in roughly $4\text{\%}$ excess heterozygosity in contemporary populations compared to the intended model, but the overall effects on patterns of diversity are minimal (Figures 2C and 2D).

Even though the error has a limited effect on simulated data, the tutorial code has been copied many times and used in publications. By searching for some identifying strings from the model definition on GitHub, we found 32 repositories containing either direct copies of the erroneous model code or code that was obviously derived from it (we have opened issues on each of these repositories to alert the authors). In most cases, the publications used simulations to test a non-demographic inference method, such as inferring gene genealogies from sequencing data,⁷ determining the age of a genetic variant,⁸ or studying the power of variant association tests.⁹ In each of these studies, this model was used as an example of a complex population history. Zhou et al.¹⁰ used the incorrect model as an example of how their method for visualizing demographic models can support msprime input. Finally, Pfaffelhuber et al.¹¹ used simulations of the incorrect model demography to evaluate their method for choosing ancestry-informative markers. Given the very subtle effect of the incorrect model on demography (and the fact the method was evaluated with other simulations and real data), it seems unlikely that the model details had any qualitative effect on their conclusions.

This long-standing error could have been prevented by better API design. To model a population split currently in msprime, a user must specify a “MassMigration” event that moves lineages from one population to another and then must also remember to turn off migration between those populations at the same time. The demographic events for the correct model are given in Figure 1B. Note that the migration rate change on line 17 was missing in the incorrect model. The release of msprime version 1.0 will introduce a “PopulationSplit” event, which more intuitively links the movement of lineages with appropriate changes in migration rates at the time of the split.

Case 2: Incorrect Model Parameters in an Analysis Pipeline

In another publication using this model,⁶ a separate error was introduced: the model itself was defined as suggested in the documentation (with updated parameters from Gravel et al.¹²) and inspected with the msprime debugging tools. Despite these initial checks, the simulation was performed without passing the list of demographic events, such as historical changes in size or population splits, so that the three populations never merged and remained separated with low levels of migration (Figure 3A). This leads to a vast overestimate of the divergence across human populations. Although the correct model predicts a mean $F_{ST}$ of $0.05-0.10$ across the three populations, the simulated model generated $F_{ST}$ ranging between $0.3-0.6$, depending on the populations considered. Overall diversity was also strongly affected: heterozygosity was more than doubled in African populations and reduced by more than half in Eurasian populations relative to the correct model.

Figure 3.

The Transferability of Polygenic Risk Scores (PRSs) under Neutrality

(A) In Martin et al.,⁶ the simulated demographic model did not apply demographic events in the past, so continental populations were simulated as isolated with low levels of migration for all time. The correct model has historical events as shown in Figure 2A.

(B–G) We repeated the simulation experiment in Martin et al.⁶ by using the correct demographic model. GWAS summary statistics were computed from 10,000 case and control subjects in the European population, and the distribution of inferred PRSs was compared across the three simulated populations. Correlations between true and inferred polygenic scores were computed over 100 simulation replicates. Unlike the original study, we do not observe large differences in mean inferred PRS across the three populations (C and D), and although risk prediction in the African and East Asian populations is still reduced compared to the European population, the reduction in prediction accuracy is not as large as reported in the original study (E–G). (B) and (C) show the distribution of true and inferred PRSs after standardizing PRSs across all individuals in the simulation (this is in contrast to Figure 5 in Martin et al., which plotted unstandardized PRS⁶). Because the polygenic trait was simulated as a threshold trait, it is the relative distribution of PRSs across individuals that determined cases and controls instead of absolute true or inferred PRS. (D) shows a comparison of PRS distributions for a simulation replicate with 1,000 causal variants. For each population in the violin plots, the original correlations from Martin et al.⁶ are shown on the left, and correlations via the correct model are shown on the right. For direct comparison to the original study, see Figure 5 in Martin et al.⁶ Box plots show median, first and third quartiles, and 95% intervals.

This simulation was performed to assess the transferability of polygenic risk scores across human populations. In other words, it sought to explore how human demographic history and population structure affect our ability to predict genetic risk in diverse populations given the well-documented unequal representation in medical genetic studies.¹³ The resulting publication has been influential in the discussion of health inequalities and genomics: it has been cited over 350 times since 2017. The large excess in divergence under the incorrect model here exaggerated the role of demography and genetic drift in limiting the transferability of genetic risk scores across populations.

Difficulties in transferability remain in the corrected model (Figure 3), although risk prediction in each population is significantly improved (compare to Figure 5 in Martin et al.⁶). Simulations under the correct model indicate that the accuracy of genetic risk scores is still substantially reduced in understudied populations (Figures 3E–3G), supporting one of the main conclusions of Martin et al.⁶ However, the reduction is much less pronounced than reported. In particular, we do not observe large differences in mean predicted risk across populations (Figures 3C and 3D) that were present in data and simulations from Martin et al.⁶ Thus, a model with continental-scale demographic structure, highly polygenic architecture, and neutral evolution does not appear to explain the large directional biases in mean predicted risk identified in Martin et al.⁶

From a software perspective, this error was easy to make and could have been prevented by better API design. The original msprime API requires the user to pass three separate parameters to specify a demographic model (see example in Figure 1C). Note that the same three parameters must be passed to both the debugger and the simulate function. To help prevent such errors, msprime version 1.0 will introduce a demography class. The above snippet would then be rewritten (approximately) as shown in Figure 1D. This simple change to the interface makes it much less likely that different models are passed to the debugger and simulator, reducing the potential for error.

Conclusions

The implementation of complex demographic models is error prone, and such errors can have a large impact on downstream analyses and interpretation. The discovery and correction of the demographic models discussed here underscore how API design choice can lead to the propagation of mistakes that are difficult to notice. There are myriad other ways to get models slightly or very wrong. Migration rates can be set in the wrong direction, for example, and parameters can be rescaled incorrectly when a new mutation rate is used. We therefore recommend the following steps to ensure more robust simulations.

First, if possible, we recommend using the stdpopsim¹⁴ library, which removes the need to re-implement demographic models from scratch. This is a “standard library” of quality-controlled models and simulation resources for a growing number of commonly studied species. The resource is built around an open-source community development model with rigorous code review and quality control (QC) procedures. For example, when a contributor adds a new model, it is not fully integrated into the catalog until a second, entirely independent, implementation of the model is provided by another developer. If the original and “QC” versions of the model are precisely equal, then we can have a reasonable degree of confidence in correctness. It is through this QC process that we discovered the misspecified model in msprime’s documentation. However, stdpopsim has QC-verified models available for only a handful of published models and species, so it might not meet the needs of a particular simulation study. If the required model is not present in stdpopsim, then extra care should be taken to validate the implementation. This could include verification through code review or comparison with an independent implementation of the model. If the model is of wider interest to the community, we would encourage developers to contribute it to stdpopsim; at the very least, the QC procedures in place provide a strong reassurance of correctness.

Second, regardless of whether the model has been implemented locally or obtained from a resource like stdpopsim, basic statistical validation should always be performed on the results. Errors are all too easy to make, and an analysis of the basic statistical properties of the simulations is essential due diligence. For example, a brief analysis of $F_{ST}$ values most likely would have prevented the error in Martin et al.⁶ It is understandable that such analyses were not undertaken in this case because processing such a huge dataset (600,000 samples) was highly challenging. Recent progress has made statistical analysis at this scale much easier¹⁵ so that $F_{ST}$, heterozygosity, the allele frequency spectrum, or the distribution of coalescence times can be rapidly computed. However, the problems would have been apparent if pilot simulations with smaller sample sizes were undertaken and analyzed because simulations and analysis of small samples sizes are very efficient. Graphical inspection of demographic models can also help identify issues, especially if visualization can be automated. The demography debugging tool in msprime summarizes demographic events occurring over time, and various efforts are underway to facilitate the specification and visualization of population genetics models, such as Zhou et al.¹⁰ and the demography package used in this paper.

Finally, openness is essential to the self-correcting nature of science. We only know about these errors because of open code and open-source development processes. By making their entire pipeline available, Martin et al.⁶ not only enabled other research teams to build upon their findings, but they also made it possible for errors to be found and corrected. There must be many, many more mistakes out there, and we need both pre- and post-publication vigilance from users and developers to ensure the soundness of the large body of simulation-based analyses.

Data and Code Availability

We computed the expected allele frequency spectrum (AFS) by using moments version 1.0.3¹⁶ and LD-decay curves by using moments.LD.¹⁷ $F_{ST}$ and other diversity statistics were computed from the expected AFS and verified with branch statistics from the output of msprime simulations using tskit version 0.2.3.¹⁵ Demographic models were plotted with the demography package written for Python (https://github.com/apragsdale/demography, version 0.0.3). The demography package allows users to define a demographic model as a directed acyclic graph using networkx,¹⁸ which can be plotted as in Figures 2A, 2B, and 3A. Demography also can translate the graph-based demography into msprime input commands or simulate the AFS or two-locus statistics by using moments^16,17 or dadi.⁵ For the analysis of polygenic risk, we used the original pipeline from Martin et al.⁶ available from https://github.com/armartin/ancestry_pipeline/blob/master/simulate_prs.py. We updated the pipeline to run with the correct demographic parameters and more recent versions of msprime and tskit, and the updated pipeline is available at https://github.com/apragsdale/PRS. Data and Python scripts to recreate Figures 2 and 3 can be found at https://github.com/jeromekelleher/msprime-model-errors. A full list of the GitHub repositories containing copies of the erroneous model are also given here.