hmmibd-rs: an enhanced hmmIBD implementation for parallelizable identity-by-descent detection from large-scale Plasmodium genomic data

Abstract

Background

Identity-by-descent (IBD), which describes recent genetic co-ancestry between pairs of genomes, is a fundamental concept in population genomics. It has been used to estimate genetic relatedness, detect selection signals, and understand population demography. The IBD detection method hmmIBD demonstrates high accuracy in inferring IBD segments between haploid genomes, including Plasmodium falciparum, and is widely used in malaria genomic surveillance. However, the current single-threaded implementation of hmmIBD does not utilize the full capacity of multi-processor computers, making it difficult to apply to large data sets, and does not accommodate non-uniform recombination rates across the genome.

Methods

We developed an enhanced implementation of hmmIBD, named hmmibd-rs, which leverages multi-threaded computing to parallelize IBD inference over genome pairs and which supports optional, user-defined recombination rate maps for more accurate IBD detection and filtration from genomes with non-uniform recombination. We further streamlined large-scale IBD detection by incorporating auxiliary built-in functionalities to preprocess input directly from the standard binary variant call format (BCF) and filter IBD output to reduce disk usage.

Results

Our new implementation significantly reduces IBD detection computation time nearly linearly with the increased number of CPU threads used; using 128 threads shortens IBD detection time from 5.2 days to 1.3 h for 220 million pairs of simulated Plasmodium falciparum-like chromosomes, increasing computational speed by approximately 100 × over the single-threaded hmmIBD algorithm. Incorporating non-uniform recombination rates in hmmibd-rs enhances the accuracy of IBD inference by mitigating the overestimation of IBD breakpoints in recombination cold spots and their underestimation in hot spots. Non-uniform rates also improve length filtration of IBD segments, dramatically reducing the rate of false positive in recombination cold spots. When applied to empirical data sets, hmmibd-rs completes the detection of IBD from MalariaGEN Pf7 (n ≈ 10,000 monoclonal samples) within hours, enabling a single-day IBD analysis pipeline for large genomic data sets.

Conclusion

hmmibd-rs builds upon, accelerates, and enhances hmmIBD for efficient and accurate IBD detection, serving as a crucial tool for advancing large-scale malaria genomic surveillance.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12936-026-05814-2.

Background

Identity-by-descent (IBD) refers to alleles or genomic regions (segments) that are identical between two individuals/genomes due to shared ancestry. For species with high recombination rates relative to mutation rates, such as malaria parasites, metrics that leverage recombination, e.g., IBD, capture finer-scale and higher-resolution dynamics in population demography [1–3]. IBD, inferred from malaria parasite genomic data, conveys important information about the recent history of populations, including genetic relatedness within and between populations, loci under natural selection, and time-specific demography (effective population size and population structure), thus playing a crucial role in malaria genomic surveillance [4–14].

Accurate detection of IBD segments often requires genotype data with sufficient marker density [14–16]. Species with a high ratio of recombination rate to mutation rate, such as the malaria parasite Plasmodium falciparum, tend to have a (common variant) marker density (per-genetic unit) two orders of magnitude lower than that of humans [13, 16–20]. Thus, an IBD detection algorithm robust to low marker density is crucial for high-recombining species. Our accompanying project suggests that hmmIBD stands out among many other IBD detection methods, uniquely providing high-quality IBD segment calls, including shorter segments, that allow for the generation of accurate results even for quality-sensitive inferences [15]. Despite its high accuracy and wide adoption in malaria research, hmmIBD could be improved in several areas: it uses only a single thread of multi-core CPUs, it assumes a uniform recombination rate across the genome, and it requires substantial preprocessing of input data into a specific format prior to analysis [12]. These limitations may prevent its wider application to scenarios that require shorter run times, such as IBD inference from large data sets, sensitivity analysis, and the generation of simulation-based null for real-data analysis, or that rely on a non-uniform genetic map [19, 21]. This caveat is particularly relevant given the demand for analysis of large-scale whole-genome sequence (WGS) data [19, 22, 23] and opportunities to construct high-resolution recombination rate maps based on recent genetic crosses [24–26].

In this work, we addressed these limitations by reimplementing and enhancing the Hidden Markov model (HMM) described in the original paper [12]. Our new implementation, hmmibd-rs, offers three key features: parallelized IBD inference, support for non-uniform recombination rates, and streamlined data management.

Methods

We enabled parallelization for the HMM inference process at the level of single haploid genome pairs or groups of haploid genome pairs. In our new implementation in the Rust programing language, we first modularized the original algorithm into multiple components, including data processing modules and different subcomponents of the HMM inference process. Relying on the modular structure, we then isolated the HMM inference process for a genome pair as the basic unit for parallelization. The original sequential HMM inference process, iterated over genome pairs, was converted into parallelizable tasks, utilizing the Rayon crate, a library designed for data parallelism [27]. In addition, we provide options to optimize memory usage based on parameters such as the maximum number of alternative alleles per locus and the output file buffer sizes.

We enabled non-uniform recombination rates for HMM inferences and IBD segment filtration using user-provided genetic maps. For HMM inference, we updated the term $e^{-k\rho d_t}$ in the transition probabilities matrix [12] to the term $e^{-k{c}_t}$, where ρ is the recombination rate per generation per bp, $d_t$ is the physical distance between the $t$ th marker and the $t-1$ th marker, and $c_t$ is the genetic distance between the two markers given by the user. As $c_t=\rho d_t$, our new implementation maps physical distances between markers to genetic distance, and directly uses genetic distance $c_t$ for HMM inference, thus removing the assumption of a uniform recombination rate along the genome. Besides the HMM inference steps, we also allow the usage of a recombination rate map for post-HMM inference, built-in IBD segment length filtration in genetic units, which is usually needed for analyses based on IBD segments since short IBD segment estimates are more error-prone and often filtered out before downstream analyses [15, 16, 28].

We improved the efficiency and ergonomics of input and output data management. We developed a simple, cross-platform auxiliary library bcf_reader in Rust and used it to process input data directly from the common genotype file format, binary variant call format (BCF). Based on this library, we implemented two main built-in functions. Before describing the first main function, it is important to state that samples input into hmmibd-rs are assumed monoclonal by design, and that all heteroallelic calls in data input are errors. The first main function corrects erroneous heteroallelic calls by replacing them with dominant alleles if the total allele depths (via the parameter min_depth) and the fraction of reads supporting the dominant alleles (via the parameters min_ratio and min_r1_r2) are high; otherwise, these are set to missing (as detailed in Supplementary Table 1). We note that the criteria for determining whether to use dominant alleles or missing data are somewhat subjective: users may opt for stringent thresholds, such as setting all heteroallelic calls to missing data, which comes with the caveat of removing more sites and samples during the subsequent genotype filtering step, or more permissive ones to include all heteroallelic calls by using the allele with the highest read support, which may introduce substantial genotyping errors. The second built-in function is to iteratively filter samples and genetic sites (based on the missingness of genotype calls) to obtain high-quality genotype data while retaining balanced numbers of markers and samples (Supplementary Table 1). Genotype correction and filtering are optional, with all related parameters specified  in a configuration file that is disabled by default to be compatible with hmmIBD. The dominant-allele-based input pre-processing from BCF files is a heuristic strategy for working with samples that are assumed monoclonal. Haploid genomes from polyclonal infections may be inferred via external deconvolution programs like DEPloid or DEPloidIBD [29, 30] and provided to hmmibd-rs in a traditional table format used in hmmIBD.

We have included additional options, made available via the command line interface, to customize HMM parameters and data management parameters (e.g., BCF processing, IBD filtering, output buffering, and optional suppressing) to allow the user to balance analytical needs and computational/storage efficiency. Additionally, hmmibd-rs is designed to be fully compatible with hmmIBD to facilitate the transition to hmmibd-rs, and, by default, generates both files for IBD segments and files for the fraction of sites IBD (estimates of genetic relatedness) like hmmIBD.

Methods used for simulation, measurement of computation time, and downstream analysis of detected IBD segments are similar to our previous analysis [9, 15], with further description provided in the Supplementary Methods. Details of these analyses can be found in the related pipeline and source code listed under Data Availability.

Results

Our new implementation, hmmibd-rs, reduces computation time of HMM IBD inference both by increasing single-thread performance and by enabling multithreading. When both hmmibd-rs and hmmIBD are forced to use a single thread, run times with hmmibd-rs were about 40% shorter than those of hmmIBD (Fig. 1a), due to the more compact memory representation of the genotype matrix and reduced disk read and write operations in hmmibd-rs. When multithreading is enabled, the performance of hmmibd-rs is nearly linear with respect to the number of threads and the number of genome pairs (Fig. 1b); better scaling is observed for larger sample sizes where the reduction in time used in IBD inference outweighs the overhead of multithreading (Amdahl’s Law) [31]. To test the performance on a large data set, we ran hmmIBD and hmmibd-rs on simulated P. falciparum-like genomic data with a sample size of up to n = 30,000, which is the same order of magnitude as the MalariaGEN Pf7 data set (n > 21,000). Our new implementation completed the IBD detection from the simulated data set in 1.3 h using 128 threads with the AMD EPYC 9654 CPU model, whereas the single-threaded hmmIBD took an estimated 5.2 days to complete IBD detection (Fig. 1c). Additionally, when IBD segment length filtering options were used, the resulting file sizes were largely reduced (Supplementary Table 2). Thus, this new implementation, in this example, can accelerate the process by two orders of magnitude.

Fig. 1.

hmmibd-rs significantly reduces the computation time for IBD detection in simulated P. falciparum-like chromosomes when compared to hmmIBD. a Comparison of single-thread runtime of hmmibd-rs and hmmIBD for simulated data with different sample sizes. b Multithreading performance of hmmibd-rs. c Runtime for detecting IBD from simulated data on a MalariaGEN-scale using hmmibd-rs with 128 threads. Note that in a and c sample sizes are indicated as both the number of chromosomes (top horizontal axis ticks) and the number of chromosome pairs (bottom horizontal axis ticks). Dotted line in c indicates that the top right data point (star) is an estimate based on the extrapolation of data points involving smaller sample sizes

To assess how recombination rate misspecification affects IBD detection, we simulated genomes with a non-uniform recombination rate map (true rate, T) that had the same mean recombination rate (average rate, A) as the P. falciparum genome (Fig. 2a and b). We used either true or average rates in the HMM inference and IBD length filtering steps, representing each combination with a two-letter code, [T/A][T/A/X], where the first letter denotes the rate used for HMM inference and the second for length filtering (e.g., "AT" uses the average rate for HMM inference and the true rate for length filtering, while "X" indicates no filtering). Supplementary Fig. 1 illustrates that, without IBD length filtering, hmmIBD (AX) underestimates and overestimates IBD breakpoints in recombination hotspots and coldspots, respectively, compared to hmmibd-rs (TX). Uniform length filtering (i.e., using average rate to calculate IBD segment length in genetic units) of hmmIBD-inferred IBD segments (AA, Fig. 2c, red line) magnifies this trend, which is accompanied by high rates of both false negative (Fig. 2d) and false positive (Fig 2e) IBD segments (see definition in Supplementary Methods), and a deviation in their chromosomal coverage (Fig. 2f) from the true IBD segments (tskibd [9], TT, black dotted line). However, map-guided filtering (i.e., using true rate to calculate IBD segment length in genetic units) of hmmIBD-inferred IBD segments (AT, Fig. 2c–f, red dotted line) yields results comparable to hmmibd-rs (TT, blue line). This improvement is due to map-guided length calculation elongating inferred IBD segments (in centimorgans) in hotspots, thus reducing their removal by filtering, and shortening them in coldspots, increasing their removal. Regardless of the software used (hmmibd-rs, TT, and hmmIBD, AT), IBD detection performs poorly in hotspots, exhibiting a high rate of false negative IBD segments even when the true rate is used (Fig. 2d). This is likely because marker density per genetic distance is extremely low in hotspots, resulting in fewer inferred short segments (both IBD and non-IBD) and, consequently, fewer breakpoints (Fig. 2c). False positives arise when marker density is too low to detect a non-IBD segment within an IBD background. Conversely, false negatives occur when marker density is too low to detect an IBD segment within a non-IBD background, causing segments to merge. Generally, false negative IBD segments are more frequent than false positive ones because the non-IBD state (background) is marginally more probable in population samples where most pairs have relatedness (genome wide IBD probability) < 0.5. These results are generally mirrored when using realistic, genetic-cross-based genetic map [24] for data simulation and IBD detection (Supplementary Fig. 2). These findings underscore the importance of accurately characterizing local recombination rate variation, leveraging this information for accurate detection and filtering of IBD segments, and interpreting IBD-based analyses cautiously in recombination hotspots.

Fig. 2.

hmmibd-rs improves the accuracy of reported IBD segments by incorporating a non-uniform recombination rate map in the HMM algorithm and the IBD length filtering step. a, b the true, non-uniform recombination rate map (blue line) and the constant chromosome-wide average rates (black dotted line). c Number of detected IBD breakpoints (two ends of IBD segments over 2 centimorgans, cM) in 15 kb windows along the simulated P. falciparum-like genomes. d, e False negative rate (d) and false positive rate (e) of detected IBD segments (over 2 cM) calculated using IBD segment overlapping analysis (see Supplementary Methods). f Coverage of detected IBD segments (over 2 cM), normalized to the coverage of true IBD segments (over 2 cM, determined by tskibd [9]). Note: (1) The true (non-uniform) rate was used to simulate the genotype data; (2) Either true or average rates were used in the Hidden Markov model (HMM) inference and IBD length filtering steps, with each combination represented with a two-letter code, [T/A][T/A/X], where the first letter denotes the rate used for HMM inference and the second for length filtering (e.g., "AT" uses the average rate for HMM inference and the true rate for length filtering, while "X" indicates no filtering). Also, see Supplementary Fig. 1 for the IBD breakpoint analysis when all IBD segments estimated by hmmibd-rs and hmmIBD are included (without length filtering)

Another obstacle in the hmmIBD-based analytical pipeline is the need for an hmmIBD-specific format for the input data, which adds a data formatting step to IBD detection and downstream analysis, and is particularly cumbersome if iterative filtering of samples and variants is done for large genotype datasets. We mitigated these issues by implementing optional, built-in, all-in-memory functions for iterative sample and site filtering, and for heteroallelic call correction using dominant alleles (Supplementary Table 1). We presented a simple pipeline based on these features implemented in hmmibd-rs, to demonstrate its applicability to large-scale WGS data sets, from VCF/BCF files to IBD-based estimates, which includes genotype filtering and heteroallelic call correction (bcftools [32] and hmmibd-rs), IBD calling (hmmibd-rs), and IBD coverage calculation. We were able to finish IBD calling from the raw genotype call files within a single day using 64 threads CPU (Fig. 3a), with most of the time spent on bcftools for the initial genotype filtering step. As proof of concept, the resulting IBD data show signals of positive selection (Fig. 3b) consistent with previous reports using either IBD-based methods [5, 9, 10, 33], or non-IBD-based methods, including signals associated with the increased frequency of drug resistance mutations in pfaat1 and pfcrt in populations over time [33].

Fig. 3.

The hmmibd-rs-based, VCF-to-IBD pipeline enables fast and streamlined detection of IBD segments from the MalariaGEN Pf7 data set within a single day. a Descriptions, tools involved, numbers of samples and records analyzed, computation time, and output file sizes over the five steps of the VCF-to-IBD pipeline. b Example analysis (IBD coverage) of the detected IBD segments from the MalariaGEN Pf7 dataset. Genes of interest (potentially under positive selection) were labeled above the plot. IBD estimates here were based on a constant average recombination rate (6.67 × 10^–7 per bp per generation)

Discussion

This study presents an improved implementation of hmmIBD with three important features for large-scale population genomics: significantly reduced computation run time via multithreading, optional recombination rate map specification, and improved data management. Compared to other probabilistic IBD (segment) detection methods popular in malaria research, including hmmIBD [12], isoRelate [10] and DEploidIBD [30], hmmibd-rs is the first attempt to leverage the memory-safe language Rust and its rich ecosystem to embrace the era of large-scale genomics by enabling computational parallelization, employing a standard input format and lowering difficulty of long-term software maintainability and further development to incorporate more complex models. Assuming constant recombination rates, hmmIBD has already been applied to P. falciparum, non-falciparum malaria parasites (e.g. P. vivax [34]), and many other species, such as Cryptosporidium hominis [35], honeybees (Apis mellifera) [36], liverwort (Marchantia polymorpha) [37], and green alga (Chlamydomonas reinhardtii) [38]. hmmibd-rs is designed to be compatible with hmmIBD and will remain useful for these applications. For new species, context-specific benchmarking and parameter optimization are recommended to ensure optimal performance [15]. The new features of hmmibd-rs including its parallelizability and support for a non-uniform genetic map may allow the detection of IBD segments from larger, more complex datasets, including those with more samples, larger genomes and variable recombination rates. For example, our recent work has used the community detection algorithm Infomap [39, 40] to analyze genome-wide pairwise total IBD sharing within networks containing WGS data from around 1,000 P. falciparum isolates to understand population structure and how it changes over time [9]. Further development of IBD-based surveillance using large-scale datasets, such as MalariaGEN Pf8 [23], may allow the capture of fine-scale spatiotemporal patterns of parasite population demographics, crucial for understanding malaria transmission and intervention impacts [1–3, 41, 42].

One caveat of our analyses of hmmibd-rs is the lack of reliable non-uniform recombination rates for empirical data sets of high-recombining species like P. falciparum, despite initial efforts to estimate either the average rate or high-resolution rate maps based on limited samples [21, 24–26, 43]. Ongoing work is needed to estimate high-resolution recombination rate maps by integrating recombination events observed in various genetic crosses, and linkage-disequilibrium (LD) or IBD patterns observed in WGS data from large-scale population samples [21, 26, 43–47]. This will allow further evaluation of biases in IBD-based analysis due to the use of a simple average rate or a map estimated based on LD, or IBD patterns in small empirical datasets. Currently, hmmibd-rs lacks a built-in function for estimating recombination rate maps from empirical data. Therefore, users must supply a pre-existing genetic map generated by external tools or, if unavailable or unreliable, employ a constant recombination rate. Given the higher error rates in recombination hotspots (even with the use of true rate maps for IBD inference and filtering), we recommend exercising caution when interpreting results in known or suspected regions of high recombination.

We stress that the optional built-in function that corrects heteroallelic calls is intended for monoclonal samples. Advanced genotype deconvolution tools, such as DEploid and DEploidIBD [29, 30], can be used to pre-process genotype calls of polyclonal infections.

Conclusion

Hmmibd-rs enhances the original IBD detection algorithm hmmIBD with key features that significantly accelerate IBD detection from large-scale genomic data and enable the incorporation of a genetic map for improved accuracy in genomes with non-uniform recombination. The new implementation allows for more efficient, accurate, and streamlined IBD-based analysis of Plasmodium genomes, contributing to more timely malaria genomic surveillance.

Author contributions

BG: Led algorithm enhancement, developed and tested the program, and led paper writing; SFS: Helped improve paper; ART: Helped improve the paper; TDO: Supervised work; STH: Supervised work, and helped improve the paper.

Funding

This work was supported by NIH 1R01AI145852 granted to ST-H and TDO by the U.S. National Institutes of Health.

Data availability

Source code for hmmibd-rs in Rust is available at https://github.com/bguo068/hmmibd-rs; version v0.1.5 was used in this work; this repository also includes a version of hmmIBD in C, modified to allow users to specify the average recombination rate via a command line option. Source code for the cross-platform library for reading BCF file format, bcf-reader is available at https://github.com/bguo068/bcf-reader. The pipeline that simulates data and benchmarks and characterizes hmmibd-rs and hmmIBD is accessible at https://github.com/bguo068/hmmibd-rs-bench. The pipeline for a full demonstration of hmmibd-rs’s application to MalariaGEN Pf7 is provided at https://github.com/bguo068/hmmibd-rs-bench-empirical. The genotype data of the MalariaGEN Pf7 data set [22] and its sample meta information, including F_WS estimates, are publicly available at https://www.malariagen.net/resource/34/.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.