Fostering discoveries in the era of exascale computing: How the next generation of supercomputers empowers computational and experimental biophysics alike

Abstract

Over a century ago, physicists started broadly relying on theoretical models to guide new experiments. Soon thereafter, chemists began doing the same. Now, biological research enters a new era when experiment and theory walk hand in hand. Novel software and specialized hardware became essential to understand experimental data and propose new models. In fact, current petascale computing resources already allow researchers to reach unprecedented levels of simulation throughput to connect in silico and in vitro experiments. The reduction in cost and improved access allowed a large number of research groups to adopt supercomputing resources and techniques. Here, we outline how large-scale computing has evolved to expand decades-old research, spark new research efforts, and continuously connect simulation and observation. For instance, multiple publicly and privately funded groups have dedicated extensive resources to develop artificial intelligence tools for computational biophysics, from accelerating quantum chemistry calculations to proposing protein structure models. Moreover, advances in computer hardware have accelerated data processing from single-molecule experimental observations and simulations of chemical reactions occurring throughout entire cells. The combination of software and hardware has opened the way for exascale computing and the production of the first public exascale supercomputer, Frontier, inaugurated by the Oak Ridge National Laboratory in 2022. Ultimately, the popularization and development of computational techniques and the training of researchers to use them will only accelerate the diversification of tools and learning resources for future generations.

Significance

From the integration of artificial intelligence tools to the development of specialized hardware, computational biophysics has evolved to encompass state-of-the-art technologies in every stage of its scientific research efforts. The development of new computational tools had implications across all fields, leading public funding agencies to support the creation and use of supercomputers for biological research and private sector investments to support numerous in silico drug development startups. The next generation of exascale supercomputers are already creating new opportunities for integrating computational modeling and experiments, pushing computational biophysics beyond explaining experimental results, fostering discoveries with an unprecedented level of detail.

Introduction

The history of biophysics is paved by cross-disciplinary innovation. From Erwin Schrödinger’s take on genetics using an information transfer perspective (1) to Hodgkin and Huxley’s description of electric currents in cells (2), biology and mathematics and physiology and physics have continuously worked together. With the advent of computers, new opportunities emerged for combining disciplines, as exemplified by the 2013 Nobel Prize awarded 10 years ago to Karplus, Levitt, and Warshel for their contributions to theoretical chemistry. Software and hardware have become essential to many aspects of biophysical research, not only to make sense of experimental data but to propose and validate new models.

In this perspective, we will outline how large-scale computing has provided novel insights to classic research efforts and can continue to open new roads for the exploration of biophysical phenomena. While we attempt to cover important computational techniques, detailing their many differences would be outside the scope of this perspective. To the readers that are interested in details of the computational techniques, we suggest recent reviews in quantum mechanics/molecular mechanics, coarse-grained simulations, and Brownian dynamics, among other techniques that benefit from exascale computing (3,4,5,6,7,8). Here, we will focus on the interplay between methodological advances and biological breakthroughs.

From molecules to cells

Protein structure prediction, one of the oldest biochemistry challenges tackled by computational biophysics, was also the focus of recent broadspread media attention due to the performance of artificial intelligence (AI)-based algorithms. The notion that a primary sequence of amino acid residues contained all the necessary information to determine a protein’s three-dimensional structure captivated researchers for decades. Many attempted to use computers to impose physical and chemical laws onto unsuspecting bits representing atoms and bonds, and despite documented examples of sequence-similar and structure-dissimilar proteins in public databases (9), homology-based protein structure prediction enjoyed continuous successes over the years, with prominent examples being Modeller (10), Rosetta (11), and SWISS-MODEL (12). The new AI-era models such as AlphaFold (13), RoseTTAFold (14), and, more recently, OmegaFold (15) also rely heavily on previously determined primary and tertiary structures of known proteins, either explicitly using sequence alignments or implicitly storing that information on millions of neural network parameters. They have now reached the point where almost 200 million structures, including the entire human proteome, have been computationally predicted (16). The unifying thread in this diverse set of approaches is the extensive use of state-of-the-art computer hardware and software to optimize models, execute algorithms, and analyze results. Specifically, the development of new graphic processing units (GPUs) and tensor processing units over the past decade, along with specialized machine-learning packages such as Keras, TensorFlow, and PyTorch, among others, now provides unprecedented training potential for new neural network models and applications.

Beyond predicting structures, the study of protein dynamics and properties has benefited from large-scale computing in many ways. Dedicated hardware built exclusively for molecular dynamics (MD) simulations has allowed unprecedented insight into small proteins’ folding pathways (17). With the advance of GPUs and tensor processing units, training neural networks is becoming faster and easier, opening new doors for academic research. For example, AI models trained on protein sequences to produce learned representations can now be used to predict features such as fold stability and mutation impact on protein function (18). The same infrastructure powers efforts to integrate AI into analysis of protein dynamics (19), such as the dimensionality reduction of an MD trajectory using autoencoders (20). These and other developments were only possible due to the creation of advanced neural network architectures, including case-specific autoencoders and recurrent neural networks. This type of novel insight into protein sequence and dynamics can significantly help AI-based drug development pipelines (21) and computer-aided protein design and evolution efforts (22).

A particularly interesting research endeavor where computational power has bridged the gap between theory and experiments is single-molecule force spectroscopy (SMFS), where unbinding or unfolding forces are used as probes for molecular interactions and energy landscapes (25,26). The computational study of mechanostable molecular interactions started decades ago. The streptavidin:biotin complex, for example, widely used in experimental research, was simulated 25 years ago (27) using the most advanced hardware available at the time to achieve picosecond-level sampling of eight replicas per system. Almost 20 years later, the same molecular complex was revisited to elucidate the origins of conflicting experimental observations: In 2018 (28), a monovalent streptavidin tetramer was simulated to reveal different unbinding pathways for biotin when pulling the complex from the C- or N-terminus. In 2020 (29), in a study that demanded 400 80-ns-long replicas for a total of 32 $\mu$s of simulation time, the four-subunit streptavidin complex bound to a single biotin molecule was simulated to reveal how the tethering geometry affects experimental measurements. Exploring more complicated interfaces, in silico SMFS reached hundreds of replicas totaling dozens of $\mu$s of MD simulated time when modeling protein complexes, providing an unprecedented level of detail and explaining the origins of mechanostability in protein:protein interactions (23,30,31,32). More recently, combining this in silico SMFS approach with AI-based structure prediction has allowed for the first screening of mechanical properties of dozens of Staphylococcus’ adhesins during initial steps of infection (24) (see Fig. 1). These efforts were made possible largely by the use of National Science Foundation-funded supercomputers, which allowed parallel broad sampling of in silico SMFS experiments at multiple pulling speeds, tethering geometries, and interface variants. Despite technological advances, there is still a large gap between timescales accessible through experimental and in silico SMFS. While state-of-the-art resources have significantly broadened researchers’ access to all-atom MD simulations that span millisecond timescales, few groups can reliably obtain enough replicas of independent simulations as to fit the models used to explain experimental observations. A recent example of this limitation was explored by using coarse-grained MD simulations to connect all-atom and experimental SMFS data for Staphylococcus’ adhesins (33).

Figure 1.

Modern supercomputers are powering discoveries with an in silico SMFS approach. Staphylococcus aureus is a pathogen that can form biofilms on implants and medical devices. A critical step to biofilm formation is the formation of a tight interaction between microbial surface proteins called adhesins and components of the extracellular matrix of the host. A combination of in silico and in vitro single-molecule techniques has revealed how the bond between staphylococcal adhesins and their human targets can withstand forces in the same order of that of covalent bonds (23,24). To see this figure in color, go online.

In recent decades, supercomputers have been used to simulate increasingly complicated molecular complexes (34). The first all-atom simulation of the entire HIV-1 viral capsid (35), for example, relied on cryoelectron microscopy (cryo-EM) data to build a 64-million-atom system. In 2013, simulating this system required 128,000 cores of the National Science Foundation-funded Blue Waters supercomputer to achieve 200 ns of total simulation time for the entire capsid. Almost 10 years later, the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) viral envelope was simulated using a combination of AI-powered methods and high-performance computing resources from the Department of Energy-funded Summit supercomputer. This research led to the creation of a 305-million-atom simulation that elucidated how elements of the SARS-CoV-2 viral envelope interact (36), which still achieved 68 ns per day of simulation performance. Large-scale simulation efforts have also targeted realistic compositions of cell membranes (37), and the electron transfer process in mitochondrial complexes (38).

More than broadening the reach of simulations to increasingly larger systems, high-performance computing has also been used to extend the range of enhanced sampling and free-energy calculation in MD simulations (39). Many macromolecular complexes would require seconds of simulated time in order to spontaneously visit relevant biological states multiple times. That is why the continuous improvement of enhanced sampling methods is crucial for computational biophysics. For example, researchers used a weighted ensemble strategy to simulate the spike protein trimer of SARS-CoV-2 in over 130 $\mu$s of trajectories, obtaining hundreds of spontaneous opening and closing events (40) that showed how glycans control the motion of this macromolecular complex. One particular challenge for the exascale era that is already being addressed today is the integration of quantum mechanical/molecular mechanical (QM/MM) calculations to classical simulations, which will require efficient communication as both simulation size and computer clusters grow over time (41,42). QM/MM simulations have been used to investigate not only chemical reactions in biological molecules (43) but also to understand how these molecules behave in highly polarizable environments, such as biological membranes (44). In a hybrid QM/MM approach, the charging of a tRNA molecule was examined using a massively parallel computation strategy that included a string method with swarms of trajectories to enhance the sampling of possible chemical reactions paths (45).

While exascale computing achieved by large supercomputer clusters will revolutionize computational biophysics research in the coming years, petascale research is already accessible today. The combination of drastic increase in computing power within similar price ranges for central processing units, the development of new architectures for GPUs, and improvements in supercomputer infrastructure and usability allows researchers to reach unprecedented levels of simulation throughput to connect in silico and in vitro experiments.

Improvements made to speed up MD software packages (46,47), coupled with novel hardware, allowed researchers to accumulate collections of simulation data of unprecedented size. In fact, the development of specialized GPU kernels and the accelerated rate of inter-GPU communication have greatly accelerated MD packages, both in small and large systems. Now, a major part of computational biophysics efforts are focused on the development of analysis methods that can accelerate data processing and extraction of information from simulations. Researchers are engaging this analysis challenge on many fronts, from graphical user interface-based packages like QwikMD (48,49) to coding-oriented packages such as MDAnalysis (50), PyTraj (51,52), MDtraj (53), and Bio3D (54). Specialized analysis methods have also been developed to detect large-scale, low-frequency movements in macromolecular complexes (55,56) or to represent biomolecules as a graph and detect cooperative motions (57,58). The common theme among these and other methods is their ability to represent macromolecular dynamics in low-dimensional spaces that are amenable to both visualization and analysis. Thankfully, the origin of this problem also provides solutions. Computer clusters are regularly used to accelerate the analysis of results they created and can even help visualize systems too large to leave their massive storage devices (59,60).

Experimental biology advances have been equally important in pushing the boundaries of computational biophysics research. Cryo-EM has occupied a prominent role in such studies given its ability to provide large-scale architectures for macromolecular complexes and even detailed structures for their building blocks. From the structure of the eukaryotic translational initiation complex (61) to the structure of the mitochondrial complex I (62,63) or the organization of the entire nuclear pore complex of Saccharomyces cerevisiae (64), cryo-EM has continued to provide a macroscopic view for many research efforts, opening the doors for the detailed examination of dynamics that all-atom simulations can provide. A notable example was the determination of the water-mediated proton transport through the $V_0$ proton channel of a yeast ATPase (65), which revealed how the transport mechanism relies on specific protonation states of amino acid side chains throughout the channel. The cryo-EM and MD techniques outlined here can be combined for even more synergistic research, providing hybrid approaches for structure determination (66,67), such as MD flexible fitting (68) and damped-dynamics flexible fitting (69). The exploration of macromolecular dynamics can also be accelerated by combining experiments and simulations (70), including methods such as CryoFold (71) and Metainference (72), where short-lived intermediate states can be computationally inferred to describe large-scale molecular rearrangements.

Moving toward cell-scale models, different simulation and analysis techniques are combined to reach length scales and timescales that would be impossible to achieve only a few years before. Already in 2008, researchers were exploring how crowding in the Escherichia coli cytoplasm affected molecular diffusion and bimolecular association reactions using coarse-grained reaction-diffusion simulations (73). In 2010, a landmark study used the same biological target, the E. coli cytoplasm, and its 50 most abundant macromolecules to perform a Brownian dynamics simulation that assessed crowding effects on protein folding, molecular association, and thermodynamic properties in the cytoplasm (74). Recently, another group used an all-atom approach to build a model of the E. coli cytoplasm composed of 1.5 million atoms and submitted it to a total of 3 $\mu$s of MD simulation (75), focusing on the technical aspects of preparing and validating all-atom models for future larger-scale models of bacterial cytoplasm.

A tour de force that exemplifies the union of experimental and computational biophysics was the exploration of principles of cellular energy metabolism in Rhodobacter sphaeroides (76) (see Fig. 2). While all individual molecules and protein complexes involved in the photosynthetic energy conversion mechanism of the bacteria’s chromatophore were previously resolved, their structure and pairwise interactions are not sufficient to provide a unified model for this mechanism. It was necessary to combine atomic force microscopy, electron microscopy, crystallography, mass spectrometry, proteomics, and optical spectroscopy data to build the model for the chromatophore. Beyond protein structures and the lipid composition of the membrane in which proteins were immersed, the very arrangement of all components was mapped to build a 100-million-atom model for the organelle. With all this information, researchers could explore multiple aspects of the photosynthetic energy conversion mechanism. Simulations ranged from QM/MM, to obtain excitation energies of pigments, to all-atom, coarse-grained, and Brownian dynamics, reaching hundreds of microseconds of sampling to track the internal movements of the electron carrier cytochrome $c_2$. Evidently, the effort relied on supercomputing resources and a variety of specialized algorithms and software packages made available to the community.

Figure 2.

Model of a 100-million-atom chromatophore, an organelle responsible for light harvesting in Rhodobacter sphaeroides. The model was built using state-of-the-art supercomputers to combine atomic force microscopy, electron microscopy, crystallography, mass spectrometry, proteomics, and optical spectroscopy data. Figure adapted from Singharoy et al. (76). To see this figure in color, go online.

Much of the work described so far focuses on studying a protein or complex within its local biological environment. Similarly, the study of chemical reactions can benefit from contextualizing them within the greater network to which they belong. The field of chemical kinetics, and in particular the discreteness and stochasticity inherent to chemical reactions within living cells, has seen great advances with the development of theory and computational methods for time-resolved simulations of reaction networks (77). Such simulations allow researchers to explore a variety of behaviors observed during cell growth, proliferation, and differentiation, for example with the modeling of self-regulated feedback loops in gene expression circuits (78).

While many reactions in a cell can be approximated using deterministic models, such as ordinary differential equations or flux balance analysis, in order to reach whole-cell coverage, one must integrate biochemical systems that rely on species with low copy numbers, such as mRNAs or transcriptions factors. In such cases, stochasticity is unavoidable, and researchers have developed software packages to describe reactions kinetics that can even incorporate spatial inhomogeneity. A prominent example was the recent whole-cell model of the model organism JCVI-syn3A, a minimal cell with an artificially reduced genome that allowed for an unprecedented model coverage of a living organism (79). This effort counted with a variety of modeling approaches and integrated ordinary differential equations with reaction-diffusion master equation models to describe an entire cell cycle of this cell, including energy metabolism, DNA replication, and gene expression. The same research also counted with an extensive array of experimental input, from qPCR to cryoelectron tomograms.

In the context of whole-cell modeling, the development of large-scale computing resources was doubly important. First, it allowed for rare events to be extensively sampled, which is essential for the study of stochastic systems, and for validating models using experimental observations. Second, it allowed researchers to meet the parameter optimization demands from numeric approximations of complex reaction networks. Cell-scale models are inevitably underinformed, and researchers depend on parameter estimation and careful sensitivity analysis to properly analyze model results. StochSS (80), for example, is a stochastic simulation package already integrated into cloud computing environments, allowing researchers to access extensive computational power with ease. Lattice Microbes (81), on the other hand, focused on GPU-specialized code to extend the reach of reaction-diffusion master equation simulations into larger spatial coverage of a cell and longer simulation times, leveraging the impressive advances seen in GPU hardware over the past decade.

Conclusion

Since all software packages described here are freely available for academic use, and many are easily accessible through web interfaces, it is easy to forget about the costs associated with creating and providing such services. AlphaFold, for example, reportedly took many days of training using the most advanced hardware to reach the published model and is distributed with over two TB of databases. Training the final version of this model (and ignoring intermediate versions) was estimated to require dozens of thousands of dollars in computing costs, resources largely unavailable to the vast majority of academic researchers in the world. Thinking about the big picture, the need for extensive computing resources in many fields justifies investments of hundreds of millions of dollars made by governments to build the fastest supercomputers in the world. Universities also recognize the need for computational infrastructure by funding new computing-oriented faculty with larger start-up packages that approach the ones offered to experimental counterparts. This hidden cost of large-scale computing comes with an environmental impact: Australian astronomers estimated that their supercomputer usage is responsible for emitting the equivalent of approximately 15 kilotons of CO₂ per year (82). This is prompting supercomputing centers and cloud computing providers to be more transparent about energy expenditure, helping researchers to balance computational power and environmental impact (83). Hopefully, a combination of more efficient software and renewable energy sources will soon power all our research needs.

Between state-of-the-art high-performance computing infrastructure easily accessible through cloud instances and breakthroughs in quantum computing already being used in drug development efforts (84), computational biophysics continues to benefit from private sector investments. NVIDIA routinely makes its hardware available for universities, so academic researchers can develop new algorithms using the latest advances in GPU architectures, and AMD’s GPUs currently power some of the largest supercomputers in the world. Microsoft, Amazon, and other cloud service providers offer free resources for students and academic rates for laboratories, not to mention tech giants such as Google and Facebook who are continuously funding algorithmic developments in their AI-specialized divisions. In fact, both basic and applied research in computational biophysics now enjoy the neural network architectures originally designed for computer vision or text processing. While there are notable case-specific developments, such as AlphaFold, recurrent neural networks originally designed for language processing now power DNA sequence analysis, and facial recognition algorithms now process microscopy images.

While computational research in biophysics has always been driven by improvements in experimental methodologies, much of the past research has focused on establishing computational methods themselves. Now, with the advances made in computer hardware, in silico models can capitalize on the algorithms and theories developed over the past decades to literally reach the time- and length scales observed experimentally. As a result, computational research can routinely and accurately create new proteins, propose atomic-level modifications to small molecules, and design regulatory circuits using molecular biology building blocks. More than simply recapitulating experimental observations, the current range of possibilities in computational biophysics allows us bring detail and provide missing information that is out of reach for in vitro models. Due to these and other advances, in silico research has gained both the attention and trust of experimentalists, which is reflected in major combined computational-experimental efforts now frequently seen in high-impact publications. Our path forward will include new combinations of computational and experimental efforts, with unprecedented precision and reach, ultimately allowing us to challenge biological assumptions and propose new models for biophysical phenomena.

Author contributions

All authors wrote and reviewed the article.

Editor: Meyer Jackson.