GPU-accelerated modeling of biological regulatory networks

Abstract

The complex regulatory dynamics of biological networks can be succinctly captured using discrete logic models. Previous work has shown that global optimization schemes are well suited for finding state transition logic and kinetic model parameters, in what are typically large and irregular solution spaces where even relatively small networks represent search spaces exceeding 10⁴⁰ possible solutions. Powerful computing strategies are required to make this modeling practical for in silico pharmaceutical research. Here, we present a benchmark study quantifying the speedup achieved using a GPU framework in the regulatory logic modeling of two biological networks spanning an order of magnitude increase in complexity and several orders of magnitude in search space size. GPU implementation resulted in a 33%–57% reduction in wall time over multi-thread CPU and a 33%–1866% increase over serial CPU while also delivering better quality solutions in many cases. Evaluation on an even larger pathogen-host immune regulatory network suggests that migration to GPU may support convergence in less than 12 h for established models of immune signaling and less than 6 days for networks describing neural transmission in animals. These results offer a first scalability benchmark for regulatory logic modeling of biological networks, making the latter a potentially attractive approach for practical in silico hypothesis generation.

Introduction

The capacity of high throughput measurement platforms in fields such as genomics, proteomics, and metabolomics to produce large amounts of biological data continues to gain momentum and has supported the reverse engineering of an abundance of undirected interaction networks^1–3. While such association networks provide essential clues to the regulatory functionality at play in biological systems; they lack additional information such as source-target direction, regulatory mode of action and kinetics that are necessary to predict and study the dynamic behavior of a system under varying conditions. An impediment to the further parameterization of these networks lies in the complexities of conducting longitudinal as opposed to cross-sectional studies—the costs of conducting time-course experiments are high and obtaining an adequate number of sequential samples can be logistically challenging⁴. These challenges are further compounded when attempting to model the progression of infection with pathogens requiring high levels of containment, due to the labor-intensive and costly measures that must be taken to safely handle such organisms. On the computational side, fitting detailed time-series data using conventional methods of differential equations is an expensive optimization process and the result can be highly context-specific; the set of differential equations that describes one system’s dynamics in one experimental context is not necessarily transferable to even the same system’s behavior under slightly different conditions. One solution to these issues of generalization and data sparsity initially proposed by Thomas and d’Ari⁴ and that has gained traction^5–10 consists in applying a hypothesis-driven approach rooted in mechanistic prior knowledge. Indeed, the latter suggest the assembly of a logical model of known regulatory mechanisms and subsequently validating or invalidating such models based on their ability to explain experimental observations under different conditions; in essence using what experimental data might be available for model selection rather than model assembly. A discrete logical framework, e.g. binary “On/Off”⁵ or ternary “Low/Medium/ High”⁶ activation of a network element, is used to formally describe the system’s decisional rules and direct its evolution in time from one state to another¹⁰. Available data is then used to constrain the search space of discrete logic parameter values to a finite number of combinations⁷. This method has found reasonable success in a broad range of situations, where a qualitative solution is sufficient e.g. when the goal is to assist in generating hypotheses^8,9 or direct experimental design¹¹. Still, even moderately sized protein-protein interaction networks with fewer than 50 interaction edges can give rise to discrete logic parameter search spaces on the order of 10⁴⁰ or larger, and this number balloons exponentially with even slight increases to network size. Because even simple discrete logic networks can support complex dynamic landscapes, classical gradient-based optimization methods for parameter identification will converge prematurely on what are often only locally optimal solutions. To avoid such local minima, it is often necessary to use more exhaustive but also more computationally expensive global optimization methods such as genetic algorithms or simulated annealing. In these settings, CPU implementations for finding global optima quickly become inefficient even for moderately sized biological networks. GPU platforms offer an attractive alternative however the migration of optimization algorithms from CPU requires the use of specialized libraries such as NVIDIA’s CUDA¹² or ARM’s ROCm¹³. Moreover, the increase in performance obtained from a GPU implementation can vary broadly and be highly dependent on the algorithm and specific problem¹⁴.

In this study, we describe the reformulation and migration of a simulated annealing algorithm from CPU to GPU with the objective of quantifying the increase in performance achievable as it applies to the identification of decisional logic and kinetic parameters governing the dynamic behavior of regulatory biological networks of increasing size and complexity. Specifically, we apply GPU-accelerated simulated annealing algorithm to parameter identification in a simplified model of the hypothalamic-pituitary-gonadal (HPG) axis with 102 parameters¹⁰ as well as in a much larger network model of host immune response in Mycobacterium tuberculosis infection with 723 parameters¹⁵. Together these models span an over 7-fold increase in the number of parameters, and an increase of several orders of magnitude in the size of search space. In solving for sets of feasible parameter values, we compare GPU configurations with various thread counts against both a serial and parallel CPU implementations. Our results show that most of the GPU configurations offer considerable performance advantages in the case of both networks, though some thread counts perform more efficiently than others. These performance advantages are observed in both the serial and parallel CPU comparisons, and they become more noticeable as the network size is increased. In an attempt to estimate further increases in performance with scale we apply this same GPU base strategy to the identification of regulatory network parameters in an even larger preliminary model of pathogen-host immune interaction in African Swine Fever Virus infection described by over 2,400 parameters. Based on these results, we estimate that migration to GPU may support convergence in less than 12 h for established models of immune signaling and less than 6 days for networks describing neural transmission in animals, making the use of such model-based at industry-relevant scales a practical reality.

Methodology

For a more detailed presentation of the logic modeling framework, we refer to previous work by our group^6,7,10. In brief, the optimization problem serving as the reference use case in this work consists of identifying combinations of discrete integer parameter values that when applied to the directed network in question support dynamic behaviors that include those observed experimentally. A first set of parameters defines the logical state transition function at each network node. At iteration time t, all upstream mediator nodes that are activated at a level above a perception threshold will exercise a positive or negative regulatory action on their respective target nodes that is weighted according to their relative importance. A logical state transition rule acting on these threshold-filtered and weighted regulatory inputs will dictate the next target state for that node given the regulatory context^6,7. The update of a node state to its context dependent target state is scheduled for a future iteration based on its current update priority as determined by its state transition speed, allowable incremental change and allowable delay time in the queue¹⁰. The corresponding parameters describing the network regulatory function, node state transition, temporal update scheme, and the objective or fitness function used to direct parameter value suitability are described further in the following sections. Importantly, in this approach, missing observations are treated conceptually in much the same way as candidate parameter estimates. Values for these missing observations are proposed and accepted as part of an overall solution if they can co-exist with observed values given the network structure of interdependencies the proposed regulatory logic and update parameter values. They are essentially imputed based on their network relationship with observed state variables.

Model parameters

In the current discrete logic formulation of biological regulatory networks, individual biological entities (proteins, signaling molecules, etc.) are referred to as “nodes” and the source-to-target directed interactions between entities as “edges”.

The activation state of nodes can assume one of three values at any given time step:

• is the state value of entity i, where 0 is associated with low, 1 with moderate, and 2 with high levels of that entity.

Note, state variable values are proposed and where measured must correspond to that measured value as a primary and direct constraint. However, when a state variable value is unobserved its estimate must still satisfy the constraints of being consistent with all measured states given the network structure as well as the choice of network logic and network update kinetics. For this reason, the activation state might be considered loosely as a model parameter.

Each directed regulatory interaction (edge) between a source node is j and target node i is described by three parameters, namely:

• is the relative weight of edge  ji, where 0, 1, and 2 denote non-existent, moderate or strong interactions respectively.

• is the polarity of the edge  ji, where − 1 denotes an inactivating effect, 1 an activating effect, and 0 a non-existent one.

• is the threshold that denotes the minimum level of required for the interaction  ji to be active.

Collectively these parameters contribute to determining each node’s target state, or “image”, at any given time step, wherein the image of node i is the cumulative weighted input of all upstream mediator nodes that share node i as a target:

1

The departure of image from the current value of acts as a “force” on , pushing it toward the value of . The number of steps that is incrementally adjusted in that direction is determined by whether is already at the minimum or maximum allowable state. In addition, the iteration at which any such change is enacted will depend on its update kinetic parameters. This is further outlined in the following section.

Modeling regulatory networks over time

To model the evolution of a regulatory system in time, an update scheme must be selected. Typically, two basic strategies are used: a synchronous update scheme where all nodes are updated, or an asynchronous update scheme where only one randomly selected node is updated⁵. In this study, we use a priority update scheme presented previously by our group which has shown promise in capturing more complex pulsatile dynamics compared to conventional methods¹⁰. In brief, all nodes are assigned a new update status at each time step based on their respective kinetic parameters described below and only that subset of nodes with the highest priority status is updated at time t.

This priority update scheme requires the definition of three additional nodal kinetic parameters:

• or is the speed of action of network node i, the former range assigned to entities known to be slower and the latter to those known to be faster. These parameters increase or decrease a node’s update priority.

• is the memory parameter of i, i.e., how many consecutive time steps the network node i must be consistently eligible for a change in state before this next prescribed state is enacted.

• is the step size, the maximum number of increments that can be applied to a change in the state of network node i at a given time step.

If a node is eligible to update, meaning its image is different from its current state, it has room to increment or decrement in the direction of the image, and its memory criteria has been met, then its update priority is computed as the product of its speed and the number of time steps since the last state change. The subset of nodes with the highest priority values are then updated at the end of the current time step, moving a maximum of steps toward . With the primary focus consisting of replicating a sequence of state transition events, a physical time step or increment that most succinctly captures these transition events is selected and applied to translate physical time into iterations.

Optimization objective function

Given a set of model parameters, the framework defined in the previous subsections can be used to generate theoretical time-course data describing the evolution of each of the biological entities in the network over time. Parameter values are selected such that when applied to the network’s state transition and update framework the network dynamic behaviors supported include those observed experimentally and otherwise expected of the system (e.g. homeostatic stability, etc…). As the state transitions are articulated in a discrete state space, measured levels of any experimentally observed markers are assigned to bins corresponding to ranges of activation, for example, “Low” (0), “Moderate” (1), and “High” (2) as used here. The optimization problem then consists of identifying often competing combinations of parameter values that support minimal departure from experimental observations and expected states measured as a weighted aggregate Manhattan distance. Furthermore, as an extension of previous work¹⁰ we choose here to weight the element-wise departure from sequential target state vectors with the corresponding gradient to favor additional fidelity around short duration events such as pulses or bursts. This is formalized as follows.

We define a reference data matrix R as

2

where each element is the discrete value of measured entity i at time step , and n is the number of entities in the system. A reference gradient matrix G can then be computed:

3

Similarly, the simulated data matrix S can be defined as

4

where each element is the state of node i at time step , and n is the number of nodes in the system. Importantly, unobserved states are treated as free variables and solution values resolved in much the same way as estimates for logic and update parameters. We can then define the solution y to the minimization of the objective function as

5

Where,

.

The aim therefore is to find some vector x of model parameters, and unobserved states, that when applied to the network will support the simulated state matrix S that corresponds to a minimal Manhattan distance in departure from the reference data R, weighted by the gradient matrix G. For examples of these arrays, we refer the reader to the Data and Code Availability section.

Computational Implementation

Parallel Simulated Annealing Algorithm. In this work we solve for a parameter set that minimizes the above-mentioned objective function by reformulating and migrating onto GPU a simulated annealing algorithm. Simulated annealing is a common method used in global combinatorial optimization problems¹⁶ where the optimization landscape is expected to contain many local minima. Its stochastic nature allows the acceptance, with a certain probability, of worse solutions for the purpose of escaping local basins in favor of a global minimum, while still systematically refining promising solutions as in classical gradient-based methods^17,18. In Table I, we outline a parallelized version of simulated annealing, adapted from the synchronous implementation proposed by Ferreiro et al.¹⁹, that we apply to both CPU and GPU computations. All computations use the following parameters: T = 1000, T_min = 0.001, = 0.99, and n_iterations = 5000. The value of n_threads is variable according to the numerical experiment in question.

Software Frameworks. Both CPU and GPU optimization codes are implemented in Python 3.12 with standard Python libraries. The CPMPy constraint solver library²⁰ is used to ensure the initial solution vectors comply with the network constraints outlined at the end of the following subsection. The concurrent.futures Python module is used for concurrent CPU computations. Of specific note, in the CPU implementation pseudo-random number generation is performed using the Python random module with the latter using the popular Mersenne Twister algorithm²¹. In the GPU implementation, we use the Numba Python compiler²² to interface with CUDA¹² and the GPU device. Under this implementation, pseudo-random number generation is conducted by the CUDA Numba random function using the xoroshiro128p_uniform algorithm²³. Each of these pseudo-random number generation algorithms is a popular choice for their respective platforms as a result of how well each generator scales with parallelism and manages memory. Though the traditional Mersenne Twister algorithm provides a longer period, it is not without its limitations^24,25. While somewhat less robust, the increased bias of the xoroshiro128p algorithm is offset by its much small state size delivering an efficiency that is far better suited for large scale parallel applications²⁴. The current code implementation does not utilize shared memory; all solution and problem parameter values are stored globally as an attempt to standardize resolution irrespective of the number of candidates solutions being assessed or the size and complexity of the network.

Hardware Specifications. All computations are conducted on a high-performance cluster compute node. The CPU components utilize Intel Xeon Platinum 8356 H processors, and the GPU computations are carried out on Nvidia A100 GPU.

Construction of biological networks and reference data

Two biological regulatory networks are considered as case studies for the model parameterization method in question, namely (i) the female hypothalamic-pituitary-gonadal (HPG) axis and (ii) a host immune response network implicated in persistent Mycobacterium tuberculosis infection. A schematic of each network’s structure is shown in Fig. 1.

Fig. 1.

Benchmark biological regulatory networks. Unparameterized directed graphs describing (a) the structure of the HPG female sex hormone regulatory, axis consisting of 9 nodes linked by 25 directed edges, and (b) a host immune response network to M. tuberculosis infection consisting of 63 nodes and 178 edges. Edges are show as dotted lines as their corresponding regulatory action may not be retained after parameterization (weight = 0). Similarly, target arrows are represented by half circles because the regulatory action may be positive, negative, or 0.

Hypothalamic-pituitary-gonadal (HPG) Axis. The HPG axis regulates the cyclical expression of reproductive hormones^26,27 and its function is coupled with like systems responsible for the concurrent regulation of stress and metabolic hormones in most animals²⁸. The human female HPG axis is a canonical example of regular pulsatile dynamics under normal healthy conditions, making this biological system an attractive use case for assessing the ability of the proposed update algorithm and weighted fitness function to capture complex regulatory dynamics. The basic HPG axis model network used here is adapted from our previous work¹⁰ and consists of 9 nodes, each representing a specific hormone, interconnected through 25 regulatory interaction edges (~ 35% connection density) (Fig. 1(a)). In addition to reproducing the discrete state trajectory of a full hormone cycle, parameter values must also support adherence to the additional network structure and regulation constraints outlined in the next subsection. Leveraging prior knowledge of the biology, we assign a higher range of allowable update speeds to the hormones follicle-stimulating hormone (FSH), gonadotropin-releasing hormone (GNRH1), and luteinizing hormone (LH), and a lower range to the remaining hormones, consistent with the previous definition of u_i. Given the allowable ranges for the edge weights, mode of action polarity and perception thresholds as well as the maximum incremental change, update speed and delay memory, we obtain 3.06 × 10⁴⁴ possible combinations of model parameter values. We use the hormone expression time course data collected originally by McLachlan et al.²⁷ as the reference trajectories to be supported. Of the 9 hormones in the model, the expression levels of only 5 were recorded, namely estradiol (E2), FSH, inhibin A (InhA), LH, and progesterone (P4). Including the initial state, the reference data consists of 20 time points, resulting in a reference data matrix of size 5 by 20.

M. tuberculosis Host Immune Response Network. To generate a M. tuberculosis host immune response network, relevant signaling molecules were autonomously extracted from existing literature^29,30 using our own named entity recognition (NER)¹⁵. Documented regulatory relationships linking these entities as well as additional mediators were then extracted from the BEL Selventa Large Corpus³¹ and the BioPax/Pathway Commons³² databases using reading systems available under the INDRA environment³³. As there remain gaps in our knowledge of these systems, a generative AI model was used to propose any undocumented candidate regulatory interactions needed to support a fully closed-loop feedback architecture for the network. In an attempt to mitigate the introduction of spurious relationships we used the Trustworthy Language Model (TLM) from Cleanlab (San Francisco, CA) which offered a “model as judge” estimate of the reliability of any hypothesized relationships^34,35. The resulting network has 63 nodes and 178 edges (~ 5% connection density) (Fig. 1(b)). To preserve a similar ratio of fast to slow entities found in the HPG network, we randomly assign one third of the entities to the fast-update speed class. Together, these features define a second use case that is nearly an order of magnitude scale-up in size from the HPG network with a corresponding parameter value search space consisting of 1.46 × 10³¹⁵ possible parameter value combinations or 271 orders of magnitude larger.

Because experimental time course data was not available, we used the Simulate() function of our code (Table I) to generate an artificial time series as reference data for this network. To generate this artificial data, model parameters were assigned random values within their specified ranges. The constraint satisfaction library CPMPy²⁰ was used here to ensure that the assigned weight and polarity values complied with the network constraints outlined in the following subsection. In keeping with the proportion of entities for which reference data was available in the HPG axis use case, we only generated reference data for 35 out of the 63 nodes. In our selection of these 35 observed nodes, we prioritized the selection of nodes that exhibited time courses with features similar to those in the first use case i.e. pulsatile. Data was generated for 20 time points including the initial state, giving a reference matrix of size 35 by 20.

Auxiliary Constraints. There are two structural constraints that must be applied as part of the optimization problem to ensure that only those sets of network parameter values that can logically sustain biological regulation are proposed as solutions.

a) No sources or sinks: The first constraint states that the network must constitute a closed-loop system. In other words, every node must have both incoming and outgoing interactions with other nodes such that there are no sources or sinks. To enforce this, we first apply the following constraint to the downstream edge weights to control for the introduction of sink nodes, or nodes with zero out-degree:

6

where, in this case, is a list of all nodes that are downstream targets of node i, and I is the set of all nodes. Because all nodes are required under condition (b) to be subject to the actions of at least one positive upstream mediator, no additional constraint is required to guard against to the introduction of source nodes or nodes with a zero indegree.

b) Positive and negative regulation: Under the second constraint, the opportunity for both positive and negative regulation must be available to all nodes. We ensure that at least one opportunity for negative regulation exists implicitley by setting the state transition bias term or basal activation level to 0. As a result, if all upstream mediators are inactive, the activation state of any downstream target node intrinsically falls to 0. Conversely the opportunity for positive regulation is enforced by ensuring that for every target node there is at least one upstream mediator acting as an agonist. This constraint on the joint actions or polarities of upstream mediators is articulated as:

7

In this case, J_i is a list of all upstream mediators of node i. We sum up all incoming edge polarities on i that are equal to + 1, provided that the weight of that edge is non-zero, and ensure this is at least 1.

We first consider these constraints when initializing each thread with a solution vector. CPMPy is again employed here to efficiently ensure that each thread is assigned a dynamically stable and structurally valid initial solution vector from which to begin. Within the simulated annealing algorithm, the constraints are enforced under the GetCandidate() function (Table I) by using conditionals when perturbing the weight or polarity components of the parameter vector. For the most part, the function is designed to eliminate the perturbation options that would result in an invalid parameter vector; however, in the case where a positive polarity has been randomly selected for perturbation and changing it would result in the violation of Eq. (7) (i.e., removal of a sole positive regulator), we refrain from perturbing and this cycle is essentially wasted. Profiling has revealed that the average frequency with which wasted cycles occurs is 5.5% of iterations for the HPG network and 6.8% for the MTb network. All parameter value searches were conducted using the same annealing curve or cooling rate, with a maximum number of 5000 iterations allowed at each temperature until the terminal annealing temperature is reached and the search halted. In the cases studied here, job completion time often greatly exceeded the time needed to identify a stable solution. Accordingly, when discussing search results in the following section, we refer to convergence time as the earliest time for which the lowest error solution is found and remains unchanged until job completion.

Results

Two sets of simulated annealing optimization runs were conducted on both CPU and GPU platforms using 1 and 32 threads to provide a direct comparison. To explore the contribution of increasing thread-count on GPU, additional simulated annealing optimization runs are conducted on 256, 1024, 10,240 and 102,400 GPU threads. These numerical experiments were conducted using both the HPG and M. tuberculosis network models. We note that all GPU computations are conducted with a block size of 256, supported by Ferreiro et al. in a similar scaling study¹⁹.

Performance of HPG axis optimization

The performance profiles obtained for the HPG-axis optimization experiments are provided in Fig. 2(a). Even for such a modestly sized problem these results show the immediate gains achievable in migrating from a serial to a multi-threaded approach, though the choice of GPU over CPU is less obvious at this scale. A baseline serial search conducted on a single CPU delivered an optimal weighted Manhattan departure from reference data of y = 22 at 80 min eventually reaching converged in 175 min. In comparison the 32 thread CPU implementation delivered a minimal departure of y = 21 within 87 min, eventually satisfying convergence criteria at almost 200 min of wall time. Most of the GPU implementations surpass this, taking approximately 60 min to reach the same answer (70 min to reach y = 20), and completing convergence in roughly 150 min. This is a moderate performance improvement, representing a roughly 30% increase in speed when moving from CPU to GPU with equivalent thread count. Interestingly while many of the GPU implementations also allow for better solutions there appears to be an optimal range of thread count for this to be true. Employing a very low or very high number of GPU threads can be detrimental to performance. The 32-thread GPU implementation converges quickly, but results in a suboptimal solution (departure y = 24). Conversely, the 102,400-thread GPU implementation requires significantly longer wall time to converge than all other configurations, including those on CPU, namely 377 min to deliver a minimal departure of y = 20 and 939 min to achieve full convergence.

Fig. 2.

Performance profile (a) and acceptance rate (b) of HPG-network optimization. a: A plot of each configuration’s progress in reducing the objective function over time. The best-performing configurations for this problem seem to be a medium-range of GPU threads, as the 256-, 1024-, and 10,240-thread configurations have the fastest convergence rate and lowest final error. The 102,400-thread configuration takes 939.2 min to run; it is left off the graph so as to not distort the other plots, but eventually it converges to the same final error as the other best-performing GPU runs (y = 20). b: Acceptance rate in the optimization over the progress of the algorithm; i.e., the degree to which worse answers are accepted for the sake of finding even better ones. In this problem, the steeper curves are associated with better performance overall, as is the case with the GPU computations.

In Fig. 2(b), the acceptance rates of each optimization are plotted over the course of the simulated annealing algorithm. This time-independent profile is calculated by tracking the number of worse or equivalent solutions that are accepted per n_iterations; i.e. the proportion of iterations in which the acceptance criterion for a worst or equivalent solution is met by the probabilistic event r. This is done for every iteration of temperature reduction implemented as a while-loop (Table 1). The acceptance rates for both CPU implementations start to decay almost immediately as opposed to the those demonstrated by all GPU implementations which remain high then drop sharply beyond an annealing of temperature of 1/T ~ 0.1. In addition, the acceptance rates for the CPU implementations, as well as for the 32 thread GPU implementation, settle at a higher final value slightly above ~ 0.4 beyond 1/T ~ 1.0. In comparison, for all GPU implementations with higher thread counts, the acceptance rates decrease sharply to final values below 0.3, all delivering the better function evaluation of y = 20. For illustration purposes, the best solution proposed by the 1024-thread GPU implementation was used to produce the simulated time-course traces shown with the reference data in Fig. 3. In a testament to the complexity of the system, the general pulsatile nature of the dynamics is captured relatively well while the higher frequency features may require further adjustments to the search hyperparameters.

Table 1.

Parallel Simulated Annealing Algorithm.

Fig. 3.

Simulated time course of HPG-axis regulatory network model. Example solution from the 1024-thread GPU configuration with a best objective function value of 20 (weighted Manhattan distance) and an unweighted Manhattan distance of 14 from the reference data. The nodes with the best fit are LH and P4; however, the other three show a general capturing of the measured activity. Only plots of the 5 nodes for which reference data is available are shown here.

Performance of M. tuberculosis network optimization

The M. tuberculosis network represents nearly an order of magnitude increase in complexity compared to the HPG-axis network, and a much greater increase in the solution space. The results of numerical experiments presented in Fig. 4(a), immediately show how this increase in problem size immediately impacts and amplifies the increased performance achievable with GPU. A baseline serial search conducted on single CPU converged at minimal departure from reference data of y = 38 in 577 min despite solutions corresponding to y = 0 being achievable. In contrast, optimization with the 32-thread GPU implementation converged to a departure from reference of y = 0 in under 400 min, reaching the best solution on 32-thread CPU of y = 6 in 247 min. This represents a reduction in wall time of roughly 57%. Interestingly this 32-thread GPU configuration also performs better than its counterparts with even moderately more threads. GPU implementations with 256 and 1024 threads perform better than CPU implementations for the first 300 min but converge in roughly the same wall time as the latter with an equivalent final departure of y = 6. Similarly, the resource-heavy GPU implementations counting 10,240 and 102,400 threads perform much better than all other configurations for the first 2 h or so but then slow considerably until finally converging to a departure of y = 0 after 840 and 3516 min, respectively. In solving for this problem, the impact of increased thread count appears less straightforward with both very large and relatively small GPU resource allocations performing well and intermediate allocations delivering less impressive compromises in both time and solution quality. Indeed the 32-thread and 102,400 thread GPU instances both deliver the baseline optimal solution of y = 6 at roughly 250 min, much faster than all other instances. While both of these instances also deliver a final solution of y = 0 departure, the additional overhead inherent in the larger GPU implementation results in the latter requiring a much longer wall time to converge. To illustrate the complexity and range of marker dynamic behavior in the current network, observed immune response profiles are presented in Fig. 5 alongside model-predicted profiles derived using parameter values from a 32-thread GPU instance with a final departure of y = 0.

Fig. 4.

Performance profile (a) and acceptance rate (b) of M. tuberculosis network optimization. a: Each configuration’s progress in reducing the objective function over time. The best-performing configurations for this network appear to be a low-medium range of GPU threads; while the higher-threaded configurations perform faster at first, they take much longer than the lower-threaded ones to converge. Again, the 102,400-thread plot is truncated because it takes so long to complete (approximately 7 days), but eventually converges to 0 as well. b: Acceptance rate as the algorithm progresses. Again, steeper curves are generally associated with better performance, seen especially in the difference between the CPU and GPU curves.

Fig. 5.

Simulated time course data compared to reference data, derived from an optimized M. tuberculosis host-pathogen network model. For the M. tuberculosis network, the optimization successfully determined several models that fit the computationally generated reference data perfectly, one of which is used to produce the data here. Only plots of the 35 nodes represented in the reference data are shown.

These differences in search trajectory and performance are further reflected in the acceptance rates of equivalent or worst solutions presented in Fig. 4(b). The GPU instances that achieve a final departure of y = 0, namely those using 32, 10,240 and 102,400 threads, show a markedly different evolution in acceptance rate with decreasing annealing temperature compared to those instances converging to a departure of y = 6 (or higher). Once again, initial acceptance rates at temperatures above 1/T ~ 0.1 are higher for GPU implementations, albeit only slightly, than CPU implementations with the latter again settling at a higher final acceptance rate of approximately 0.75 beyond 1/T ~ 1.0. The GPU implementations with mid-range resource allocations, namely 256 and 1024 threads, decrease sharply in acceptance rate along with other GPU implementations but quickly increase again to settle at a final acceptance rate of roughly 0.5–0.6 above 1/T ~ 1.0. Those GPU instances eventually delivering an optimal departure of y = 0, namely those counting 32, 10,240 and 102,400 threads, show acceptance rates that continue decreasing to below ~ 0.2 up to temperature just above 1/T ~ 10. At those temperatures both the 32-thread and 102,400-thread instances sharply increase back to final acceptance rates of close to 0.5. In contrast, the third instance that also produced a final departure of y = 0, namely the instance with 10,240 threads, continued to hold a low acceptance rate of less than 0.2 as temperatures continue to decrease beyond 1/T ~ 10. With the probability of accepting of worst solutions becoming very small at such low temperatures, these different final stable acceptance rates will reflect the availability of equivalent solutions in that vicinity of the fitness landscape.

Scale up to larger networks

In an effort to predict the time to convergence for larger parameter identification problems, a third network describing pathogen-host regulatory interaction in African Swine Fever Virus (ASFV) infection was also analyzed. This network consists of 133 host and pathogen proteins connected through 676 regulatory interactions (~ 4% connection density). Even at this relatively sparse density, the detection threshold, ternary decisional logic, and event scheduling parameters occupy a space of ~ 10 ¹⁰⁰⁰ combinations. The best solutions obtained supported a departure of y = 0 from the reference data in 37.5 h (2250 min) using 10,240 GPU threads. This optimal wall time, together with best results obtained for the HPG and Mtb networks using 10,240 GPU threads, were used to describe a tentative second order trend of wall time with respect to the log size of the search space (Fig. 6). This trend was then used to estimate the expected time to convergence using a 10,240-thread GPU configuration given search spaces corresponding to ternary logic networks of varying connectivity and number of nodes ranging from 25-node networks with 31 edges to 400-node networks with 48,000 edges (Table 2). This range of network architectures was selected as representative of well-documented biological networks published in the literature³⁶. Results suggest for example that a model describing the discrete state dynamics for the complete transcription factor (TF) network in Drosophila (110 TF nodes, 307 edges) could be identified using this platform in less than 28 h of compute time³⁷. Only slightly more would suffice to compute network models describing metabolic cascades in M. pneumoniae (178 metabolite nodes, 470 edges)³⁸. Yet another example use case would be the protein-protein interaction (PPI) network linking 400 upregulated genes in the Dentate Gyrus (DG) through 990 interactions in mouse brain³⁹, a network for which plausible regulatory kinetics could be identified in roughly 10 days of compute time. Compelling as these use cases may be, they remain relatively small compared to recent estimates of PPI networks in human which can exceed 15,000 nodes and 120,000 interactions⁴⁰. Indeed, by our current estimates, roughly 90 days of computer time on a 10,240-thread GPU instance would be needed to identify regulatory logic and kinetic parameters for a 400-node network with 48,000 regulatory interactions. Of note, given the resources and time required to study each new and more comprehensive use case, these results offer only a very exploratory analysis of scalability beyond the example networks presented. The breadth and complexity of any biological network, summarized here as the size of the corresponding model parameter space, constitute only one component of overall computational requirements with increased complexity in the topology of the solution fitness landscape also being a significant contributing factor. Nonetheless, these use cases may still offer a partial glimpse of the resources that might be required to achieve more comprehensive higher- fidelity network models and a testament to the challenge ahead.

Fig. 6.

Expected performance of GPU-accelerated optimization as search space size increases. The time to convergence of two networks examined in this study as well as an additional larger one (African Swine Fever Virus host-pathogen network), all using the 10,240 thread GPU configuration. These times are plotted against the log of the search space size in order to come up with a trendline to predict the time to convergence of even larger networks.

Table 2.

Expected time to convergence based on a 10,240 thread GPU configuration, given various network sizes and connection densities.

Discussion

In this work, we explore the efficiency gains that might be possible when deploying simulated annealing in a GPU-based versus CPU-based computing environment to identify regulatory logic and kinetic parameters in biological networks of increasingly large combinatorial solution spaces. We chose as a basis for comparison two biologically relevant regulatory networks, with the first being a smaller but more densely connected network and the second being larger but more sparsely connected. While these example biological networks differ in size and connection density by roughly an order of magnitude, the corresponding parameter value search spaces span several hundred orders of magnitude providing a reasonable basis for assessing a migration to GPU architecture. As might be expected, performance gains from GPU were less noticeable in the case of the smaller HPG regulatory network. While equivalent transient solutions for the smaller HPG network were found much more quickly on GPU, final optimal solutions were achieved at similar execution times though GPU implementations typically delivered better solutions i.e. lower departures of predicted trajectories from reference trajectories. Very similar trends were observed for the larger Mtb immune response network with at least a 10-fold or greater reduction in wall time for transient equivalent solutions and at least a 2-fold reduction in wall time to match the best solution produced by a 32-thread CPU implementation. Once again, GPU implementations were capable of delivering better optimal solutions including parameter value sets that supported an exact adherence of predicted trajectories to reference time courses.

These greatly reduced execution times for GPU over CPU implementations using the same number of threads is not surprising and speaks to foundational differences in chip architecture. CPUs are designed for general purpose use, mainly serial computing, whereas GPUs are designed to be optimal for massively parallel operations⁴¹. For the class of problem presented here, both the function evaluation and the simulated annealing optimization itself involved many iterative loops and repetitive tasks for which the GPU architecture is quite well suited. Perhaps of greater interest here is the ability in several instances of GPU-based optimizations to not only execute faster but also deliver better solutions altogether. A closer examination of the rate at which equivalent or worst candidate solutions are accepted during the parameter search might offer some insight in this regard. Given the functional shape of this curve we can distinguish three basic regimes, namely a first initial regime where the high annealing temperature dominates and acceptance probabilities are high, a second transitional regime where decreasing temperatures quickly reduce the probability of acceptance, and a final regime where low temperatures dramatically reduce acceptance of worst solutions, allowing only equivalent solutions. At high initial temperatures and correspondingly high acceptance probabilities, GPU-based instances typically accept equivalent or worst solutions at a noticeably higher rate suggesting a more aggressive exploration of the global landscape. As annealing temperature drops in a transitional phase, acceptance rates decline much more steeply in the case of GPU instances. In comparison, the decline in acceptance rates with respect to annealing temperature is much less steep and much more shallow for CPU instances, even for an equivalent number of threads. In addition, these changes in CPU acceptance rates occur more smoothly and do not show the same degree of stochasticity as similar curves for GPU instances where we observe an increased variability that potentially results in beneficial probabilistic events. During this transition phase GPU instances appear to shift very quickly to more locally driven exploitation of incrementally better solutions while still allowing for comparatively higher variability, albeit across an increasingly proximal region. Finally, as annealing temperatures continue to decline the probabilities of accepting even slightly worst solutions become very small with only the only exception being the acceptance probability of 1 afforded to exactly equivalent solutions. These final stable acceptance rates differ between individual CPU and GPU instances suggesting that stochastic events during the transitory cooling phase direct each search to different regions of the search space with widely different local topologies. Flatter fitness regions will offer a greater number of equivalent solutions and correspondingly higher final acceptance rates, as might be the case for CPU instances. Conversely the typically lower final acceptance rates demonstrated by GPU instances might suggest that the latter discover more irregular regions of the fitness landscape where equivalent solutions are less abundant. Unsurprisingly, these very different behaviors can most likely be traced back to basic differences in architecture between GPU and CPU, particularly the distinct way in which each handles memory allocation. Random number generation, a key component of this algorithm, is particularly challenging in GPU settings, where access to global memory is limited. Therefore, in a GPU framework, each thread requires its own random number generator that can exist in local memory but that still produces sufficiently distinct numbers from the other threads. The need for a GPU-specific random number generator follows from this, and indeed, there have been many methods proposed for generating random numbers on GPU, the qualities of which tend to vary considerably, depending on the application of interest^42,43. In this study, we use Numba’s xoroshiro128 + algorithm random number generator for GPU²³. Despite the shorter period resulting from the reduced memory requirements of the xoroshiro128 + PRNG, we observe steeper, more variable acceptance curves and better performance even when the number of threads is held constant. This would suggest that the latter’s use of a strong scrambler to disrupt linear dependencies⁴⁴ supports the introduction of additional stochasticity into the search when compared to the CPU’s use of the more memory intensive Mersenne Twister algorithm^24,25. Given that simulated annealing global search algorithms depend heavily on such stochasticity, the use of different random number generators might explain a large portion of these observed performance differences with the xoroshiro algorithm’s efficiency and small state size making it inherently better suited for deployment on large sets of parallel threads compared to the largely CPU-bound Mersenne Twister. Admittedly in this study, the specific PRNG used is technically confounded with the choice of hardware platform given that best practices typically favor the targeted design and deployment of algorithms that make optimal use of a specific hardware architecture. Performance improvements should therefore be interpreted as resulting from the combined effects of high parallel throughput capacity and the advantage offered by optimally adapted PRNG algorithms which together contribute to the GPU’s overall better performance in both efficiency and quality of solution.

In addition to speed and thoroughness, we also observe the increased potential for GPU to scale efficiently to larger problems. While we typically observe significant improvements in wall time and quality of solution as we scale up to 10,240 threads for larger problems, such improvements follow a law of diminishing returns for the problems studied here where significant decreases in performance where observed when increasing thread count further to 102,400 threads. Close investigation of this effect suggests that this is primarily due to longer kernel execution times, indicating that these problems eventually become memory bound. The current implementation requires each thread to have access to solution and reference matrices for error checking as well as several intermediate matrices for the update scheme. With an end goal of parameterizing networks of any size, up to the scale of the known human regulatory proteome, future development will involve an examination of the algorithm for memory-saving opportunities in order to optimize scale up capabilities. Were these problems to transcend memory limitations, further improvements to the basic simulated annealing algorithm used here will also be required, such as recent advances drawing on co-evolutionary computing reported to deliver order of magnitude reductions in execution time⁴⁵. Hardware optimizations might also be considered, including higher memory GPU chips, or translating the algorithm to a quantum computing framework, which shows promise for efficiently solving NP-hard problems such as this one⁴⁶. The principal advantage of migration to GPU and other hardware platforms remains the tractability of ever larger, more complex and higher-fidelity biological networks rather than speed outright. Indeed, in practically terms GPU are significantly more expensive and energy-intensive than a standard CPU platform making them a much less attractive alternative if the sole purpose is faster resolution of smaller networks. With a focus on increasingly larger network scales, more problem-specific quantum annealing architectures have shown early promise in solving a variety of large combinatorial optimization problems. Indeed, in a recent benchmark study⁴⁷ the D-Wave Advantage platform (D-Wave Quantum Inc., Palo Alto, CA, USA) with 5640 qubits with 15-way connectivity, delivered an order of magnitude reduction in execution time and higher quality solutions for the subset sum and the vertex cover problems, and an almost 2-fold decrease for the travelling salesman problem, when compared to a platform consisting of four NVIDIA GV100 TITAN-V GPUs. Admittedly the same study reported a reverse performance for the max cut problem among others suggesting that these advantages may be quite problem specific. As these and other platforms continue to evolve, the results of this current work offer a tentative benchmark of our current ability to create digital clones of immune regulatory networks that span multiple pathways using GPU as a promising workhorse. With quantum computing platforms becoming a growing reality, level of fidelity compatible with advanced personalized therapeutics may soon no longer be the stuff of science fiction.

Author contributions

JR and GB developed the numerical methodology, co-designed the experiments and co-authored the manuscript. JR wrote all optimization codes, conducted the experiments, analyzed the data. PS, CC, and ND generated the M. tuberculosis interaction network. BB and SR supported access and use of computing clusters. GB conceived and received funding for the project. All authors reviewed the manuscript.

Funding

This work was supported by the University of Saskatchewan’s Centre for Quantum Topology and Its Applications (quanTA) and by the Vaccine and Infectious Disease Organization (VIDO). VIDO receives operational funding from the Canada Foundation for Innovation (CFI) through the Major Science Initiatives Fund and from the Government of Saskatchewan through Innovation Saskatchewan and the Ministry of Agriculture. The quanTA Centre’s high-performance computational work has been advanced through a CFI John R. Evans Leaders Fund grant while access to quantum computing resources through IBM Quantum and PINQ2 has been made possible by a PrairiesCan Regional Innovation Ecosystems (RIE) contract (both awarded to SR).

Data availability

Data and Python scripts used in this work are available at [https://github.com/MIILUsaskVIDO/scientificReports-codes](https:/github.com/MIILUsaskVIDO/scientificReports-codes).

Declarations

Competing interests

The authors declare no competing interests.