A Python-based programming language for high-performance computational genomics

To the Editor:

The vast growth of next-generation sequencing data has provided us with a new understanding of many biological phenomena. As sequencing technologies evolve, sequencing datatypes (such as standard Illumina short reads, PacBio long reads or 10X barcoded reads) typically require new implementations of corresponding computational analysis techniques, necessitating software that is not only computationally efficient, but also quick to develop and easy to maintain so as to enable rapid adaptations to new kinds of data.

However, developing an efficient software tool requires domain expertise in performance engineering, computational modeling, and the ability to translate biological assumptions into algorithm and software optimizations¹. As a result, most high-performance genomics software is highly application-specific, difficult to understand, and difficult to maintain. These factors collectively have sparked a replication crisis in the fields of computational biology and life sciences in general^2–4. Ultimately, researchers in the field lack good tools for developing and updating software; one typically has to choose between a software ecosystem that allows rapid development at the expense of performance and scalability (e.g., Python or R) or low-level languages that have higher performance but are hard to develop and maintain (e.g., C, C++ or Rust)⁵. Existing solutions that try to fill the void between these extremes (e.g., MATLAB or Julia) are primarily geared towards numerical computing rather than computational genomics, and the few attempts that have been made at simplifying the development of bioinformatics software (e.g., Rust-Bio⁶, SeqAn^7,8, BioJulia⁹ or Biopython¹⁰) are based on languages that are either unfamiliar¹¹, complex and hard to program, or too high level to attain good performance.

Here, we introduce Seq (Fig. 1a), a high-performance, Python-based, compiled programming language geared towards biology that combines the ease-of-use of high-level languages like Python or MATLAB with the runtime performance of low-level languages like C or C++. Seq shares the syntax and semantics of the widely used Python language, to which it adds genomics-specific language constructs, types, and optimizations invisible to the user. We chose Python as a base language due to its succinct, clear syntax and widespread adoption; it is one of the most commonly used languages in bioinformatics, and in programming in general¹¹. Moreover, Seq’s first-class support for C and Python interoperability makes it easy to integrate into existing pipelines. Unlike compilers for more general-purpose languages, Seq’s compiler uses fundamental genomics building blocks like sequences and k-mers, as well as operations on them, such as hashing, indexing, and alignment, to increase performance. These domain-specific optimizations and program transformations are performed automatically by the Seq compiler (Supplementary Fig. S5) and in Seq, they often require just a single additional line of code (Fig. 1b shows example Seq code for a simple k-mer based read mapper), whereas implementation by hand would require large-scale code changes and substantial testing time to incorporate into existing tools. Many of these optimizations require a global view of the program and are thus out of reach for software libraries, several of which have been developed for several languages^6,10,12. Thus, Seq enables users to write high-level, Pythonic code without worrying about low-level implementation details or optimizations, which the compiler handles internally (see ‘A short, practical guide to Seq’ in Supplementary Information). Seq aims to fill the same role in computational biology as MATLAB has in numerical computing by becoming an accessible portal to the world of high-performance genomics.

Figure 1.

(a) Conceptual comparison of Seq, Python and C++. Seq combines the high-performance of C++ with the programming ease and clarity of Python, by virtue of domain-specific compiler optimizations that are hidden from the user. (b) Example Seq code for a simple k-mer based read mapper. (c) Schematic of standard genomics pipeline and those state-of-the-art tools implemented in Seq.

To demonstrate Seq’s versatility, we re-implemented 8 popular genomics tools in Seq, spanning key tasks in the genomics analysis pipeline (Fig. 1c), such as finding super-maximal exact matches, or SMEMs (BWA-MEM¹³), genome homology table construction (CORA¹⁴), Hamming distance-based all-mapping (mrsFAST¹⁵), long-read alignment (minimap2¹⁶), single-cell data pre-processing (UMI-tools¹⁷), SAM/BAM post-processing (GATK¹⁸), global sequence alignment (AVID¹⁹), and haplotype phasing (HapTree20^20,21). We observe substantial runtime improvements over the original, hand-optimized C/C++ implementations while using less code than, and maintaining identical accuracy to, the original versions. Specifically, Seq achieved up to an order of magnitude runtime improvement in data pre- and post-processing, genome index construction and numerical downstream analysis, a two-fold improvement in FM-index querying, and up to three-fold improvements in Smith-Waterman alignment and Hamming distance-based read mapping, with similar memory usages to the hand-optimized, original versions (Supplementary Fig. S1 and Tables S1–S8). The Seq implementations had up to an order of magnitude fewer lines of code than their C or C++ counterparts, making them easier to develop and less prone to bugs²². We also compared Seq to several high-performance genomics libraries — including SeqAn, Rust-Bio and BioJulia — and observed substantial speedups in every case (over an order of magnitude in some cases; Supplementary Note 2).

Seq is open source and freely available (https://seq-lang.org) and can be tried online (https://seq-lang.org/demo), with no downloads required. The implementations mentioned above, as well as the experimental notebook, are available on GitHub (https://github.com/seq-lang/seq-benchmarks). A practical Seq tutorial is also included in the Supplementary Information (Supplementary Note 1). A preliminary conference version that showcased Seq and focused on specifics of the programming language itself — without genomics applications — appeared in OOPSLA²³.

Seq is the first programming language tailored for bioinformatics that combines high-performance and scalability with a familiar, high-level and easy-to-maintain programming interface. It is the first step towards a rich ecosystem providing software development environments, visualization tools and debugging support, all designed with the domain of bioinformatics in mind. By offering biologists, bioinformaticians, and other researchers a scalable way to prototype, experiment and analyze large biological datasets through a familiar, high-performance language, we hope that Seq will act as a catalyst for scientific discovery and innovation.