Brownian dynamics simulations of mesoscale chromatin fibers

Abstract

The relationship between chromatin architecture and function defines a central problem in biology and medicine. Many computational chromatin models with atomic, coarse-grained, mesoscale, and polymer resolution have been used to shed light onto the mechanisms that dictate genome folding and regulation of gene expression. The associated simulation techniques range from Monte Carlo to molecular, Brownian, and Langevin dynamics. Here, we present an efficient Compute Unified Device Architecture (CUDA) implementation of Brownian dynamics (BD) to simulate chromatin fibers at the nucleosome resolution with our chromatin mesoscale model. With the CUDA implementation for computer architectures with graphic processing units (GPUs), we significantly accelerate compute-intensive hydrodynamic tensor calculations in the BD simulations by massive parallelization, boosting the performance a hundred-fold compared with central processing unit calculations. We validate our BD simulation approach by reproducing experimental trends on fiber diffusion and structure as a function of salt, linker histone binding, and histone-tail composition, as well as Monte Carlo equilibrium sampling results. Our approach proves to be physically accurate with performance that makes feasible the study of chromatin fibers in the range of kb or hundreds of nucleosomes (small gene). Such simulations are essential to advance the study of biological processes such as gene regulation and aberrant genome-structure related diseases.

Significance

Brownian dynamics simulations of chromatin fibers can provide valuable insight into genome-related processes such as chromatin folding and epigenetic regulation. By using a Compute Unified Device Architecture programming platform on graphics processing units, we accelerate the expensive Brownian dynamics force calculations including hydrodynamic interactions. Such an implementation opens the possibility to study kb- to Mb-range chromatin fibers containing explicit histone tails and linker histones at nucleosome resolution.

Introduction

Chromatin, a complex of DNA and histone proteins, has an essential role in gene expression (1). Yet, its three-dimensional (3D) structure and folding mechanisms to condense meters of eukaryotic DNA into a micrometer nucleus remain obscure (2). The regulation of these mechanisms by posttranslational modifications, binding of linker histone (LH) or other proteins, or the wrapping/unwrapping of nucleosomes is essential to control chromatin architecture and hence DNA accessibility and genome integrity (3).

With state-of-the-art experimental techniques, such as chromosome conformation capture, superresolution microscopy, and high-throughput sequencing, as well as computational modeling, we have changed our view of the 3D genome organization from a regular ordered folding to a heterogeneous ensemble of folds (4). Experiments have shown that chromosomes are hierarchically organized, and the genome is segregated into transcriptionally active and inactive compartments. Inside these compartments, chromatin establishes self-interactions, creating substructures called topological-associated domains, which often arise from the formation of loops mediated by the CTCF-cohesin complex.

Computational efforts have similarly contributed to our understanding of chromatin architecture and its relationship to function (5,6,7,8,9). These studies have provided valuable information on chromatin’s ionic dependence, the role of histone tails and LH, DNA electrostatics and sequence dependence, the effect of LH and nucleosome depletion or posttranslational modifications, gene silencing and folding mechanisms, loop formation, and regulation of topological-associated domains, among others.

The approaches used in chromatin modeling span a huge range, from all-atom simulations of single nucleosomes to coarse-grained and mesoscale simulations of oligonucleosome fibers and gene systems and polymer models of whole chromosomes, with decreasing resolution as system size increases (see recent reviews in (7,10,11,12,13,14,15,16,17)). While all-atom, coarse-grained, and mesoscale models are constructed based on the atomic structures of chromatin components, such as nucleosomes and LHs, some polymer models are data driven and use experimental information, usually from chromosome conformation capture, to describe chromatin architecture (6,14).

These multiscale modeling approaches for chromatin rely on various simulation techniques for conformational sampling like molecular dynamics (MD) (e.g., (18,19,20)), Monte Carlo (MC) (e.g., (21,22)), Brownian dynamics (BD) (e.g., (23) (24)), and Langevin dynamics (e.g., (25) (26)). In particular, BD is suitable for simulating physical systems containing large particles where diffusive forces are significant compared with internal forces. The bottleneck in BD is the frequent calculation of the hydrodynamic interactions between BD particles, a matrix D, which has an $O\left(n^2\right)$ complexity with system size n (n particles). Moreover, to calculate the Brownian random force, whose properties depend on D, a Cholesky factorization of the dense matrix D is required, and this task has an $O\left(n^3\right)$ complexity. This limits BD system size (27).

With the emerging Compute Unified Device Architecture (CUDA) for simulations on graphics processing units (GPUs), the calculation of D can be parallelized on as many threads (basic computing elements) as the number of interactions between BD particles, reducing the time complexity to $O\left(1\right)$. The corresponding Cholesky factorization can be parallelized to reduce its time complexity to $O\left(n\right)$. As has been proven for other molecular modeling applications, such as MD simulations (28) and quantum chemistry (29), the use of GPUs allows significant speed ups and, hence, simulation of larger system sizes over longer simulation times (30,31).

In our early chromatin modeling studies, Beard and Schlick developed and applied BD to investigate the structure and dynamics of small oligonucleosome fibers that did not include LH and flexible histone tails (32). When we incorporated flexible histone tails into the mesoscale model to investigate their role in chromatin folding (33), MC simulations became more efficient for equilibrium sampling of large systems (34,35,36,37,38).

While MC has allowed us to study numerous important structural features of chromatin fibers and gene systems (e.g., (39,40,41,42,43,44)), BD introduces an exciting dynamic component to study the evolution of chromatin structure, transitions, and rearrangements. Here, we present the implementation of BD simulations of our mesoscale chromatin model with CUDA for calculation on GPUs and examine its performance using different metrics. We then validate the strategy by reproducing experimentally determined properties of different oligonucleosome systems, as well as previous MC results.

Overall, our implementation of BD to simulate mesoscale chromatin fibers reproduces experimental trends as MC and demonstrates exceptional computational performance, opening the possibility to study chromatin fibers of several hundred nucleosomes in a reasonable computer time. Such studies, in turn, could help reveal the mechanisms by which chromatin is compacted inside the cell nucleus and the impact of protein regulators on its folding.

Materials and methods

Chromatin mesoscale model

Our mesoscale model of chromatin has evolved for over 20 years, with the current model considering flexible histone tails (45) and their acetylation (46,47), two LH variants (H1E and H1C) with several binding modes (on versus off dyad) (48,49), and nonuniform linker DNA lengths (36), among other features (7).

The chromatin elements are coarse grained at different levels of resolution. The nucleosome cores are treated as charged disks according to the atomistic core particle, whereas linker DNA, histone tails, and LHs are treated as beads (Fig. 1).

Figure 1.

Chromatin mesoscale model. 50-nucleosome chromatin fiber with the enlarged basic unit (chromatosome) showing each element and the enlarged tri-nucleosome showing the connection between nucleosomes by linker DNA. Linker DNA is shown as red beads; histone tails as green (H3), yellow (H2A), red (H2B), and blue (H4) beads; LHs are shown as orange (globular head) and cyan (C-terminal domain) beads; and nucleosome cores are shown with their distributed charge beads. To see this figure in color, go online.

The nucleosome core with 300 point charges distributed on its irregular surface calculated by the DiSCO algorithm is designed to reasonably approximate the electric field of the atomistic core particle by a Debye-Hückel (screened electrostatics) approximation (50). An optimization is performed once the charges are distributed to determine the approximation error. DiSCO assigns charges as follows: 1) a distance vector is defined from each atom to the center of the molecule, and its associated surface area is defined as a surface element; 2) multiple rounds of smoothing are performed; 3) surface points are distributed evenly at a density of 4.0 per Ų; and 4) the overall surface points are divided into 300 groups with representative surface charges assigned. The number 300 was determined to be a good balance between computational cost and electric field approximation. Linker DNAs are treated with a combined worm-like chain and bead model (51) based on the charged colloidal cylinder approach derived by Stigter (52). Histone tails (53) and LHs (48,49) are coarse grained as five residues per bead with the Levitt-Warshel united-atom protein model.

The water solvent surrounding the chromatin fibers is modeled with a continuum of dielectric constant of 80. The screening effect of monovalent salt on electrostatic interactions within the chromatin fiber is treated by the Debye-Hückel electrostatic potential. Thus, electrostatic interactions between all charged beads are screened depending on the salt concentration.

The chromatin model can be considered as a ball/spring system. In particular, linker DNAs connect nucleosome cores, and histone tails and LHs are attached to the nucleosome cores. Each bead can move freely during the simulation, except for the 300 charge beads within each nucleosome core, one fixed tail bead per histone tail, and the six LH globular head beads, which move together with the nucleosome core. Hence, the mesoscale chromatin model has flexible linker DNA, histone tails, LHs, and rigid nucleosome cores.

The total potential energy function of the model includes stretching, bending, and twisting terms for the linker DNA ($E_S$, $E_B$, $E_T$), stretching and bending terms for histone tails ($E_{tS}$, $E_{tB}$) and LHs ($E_{lhS}$, $E_{lhB}$), and excluded volume ($E_V$) and electrostatic ($E_C$) terms for all beads as

$$E\left(\mathbf{r}\right)=E_S+E_B+E_T+E_{tS}+E_{tB}+E_{lhS}+E_{lhB}+E_C+E_V,$$ (1)

where r is the collective position vector.

The forces (F) on the system are defined as the negative gradient of the potential energy as

$${\mathbf{F}}_i=-{\nabla }_{r_i}E.$$ (2)

The torques (τ) due to the forces (F) are calculated by the equation

$$\tau =\mathbf{r}\times \mathbf{F}.$$ (3)

The supporting material provides further details on parameters and connections between elements, energy terms, and force calculations.

Simulation

A commonly used BD algorithm was first proposed in 1978 (54) based on the Langevin equation and then was improved in 1989 (55) with a second-order algorithm based on the Runge-Kutta method. The latter overcomes an issue associated with the first-order approximation: when the time step is small, the procedure can be inefficient and, when large, unstable. In 2001, Beard and Schlick (32) further modified the second-order BD algorithm to include rotational motions and to evaluate the dense matrix D less frequently. The derived second-order translational and rotational equations are given by

$$\text{Translation}:\begin{array}{ll}{\mathbf{r}}^{n+1,\text{∗}}={\mathbf{r}}^n+\frac{\text{Δ}t}{k_{B}T}\mathbf{D}\left({\mathbf{r}}^n\right){\mathbf{F}}^n+{\mathbf{R}}^n,& \text{first–order}\\ {\mathbf{r}}^{n+1}={\mathbf{r}}^n+\frac{\text{Δ}t}{2k_{B}T}\mathbf{D}\left({\mathbf{r}}^n\right)\left({\mathbf{F}}^n+{\mathbf{F}}^{n+1,\text{∗}}\right)+{\mathbf{R}}^n,& \text{second–order }\end{array}$$ (4a) and (4b)

$$\text{Rotation}:\begin{array}{ll}\text{Δ}{\text{Ω}}_i^{n,\ast }=\frac{\text{Δ}t}{{\xi }_i}\left({\tau }_i^n+{\omega }_i^n\right),& \text{first–order}\\ \text{Δ}{\text{Ω}}_i^n=\frac{\text{Δ}t}{{\xi }_i}\left(\frac{{\tau }_i^n+{\tau }_i^{n+1,\text{∗}}}{2}+{\omega }_i^n\right),& \text{second–order }\end{array}$$ (5a) and (5b)

where r is the position vector, $\text{Δ}t$ is the time step, $k_B$ is the Boltzmann’s constant, T is the absolute temperature, D is the diffusion tensor, F is the force, τ is the torque, ${\xi }_i$ is the rotational friction coefficient, and R and ${\omega }_i$ are the stochastic terms (32,54), which are Gaussian-distributed random vectors with means of 0 and variance-covariance of

$$\begin{array}{l}⟨{\mathbf{R}}^n\left(\text{Δ}t\right){\left({\mathbf{R}}^m\left(\text{Δ}t\right)\right)}^{\text{T}}⟩=2{\mathbf{D}}^n\text{Δ}t{\delta }_{nm}\hfill \\ ⟨{\omega }_i^n\left(\text{Δ}t\right){\omega }_i^m\left(\text{Δ}t\right)⟩=2k_{B}T{\xi }_i{\delta }_{nm}\hfill \end{array}.$$ (6a) and (6b)

The Cholesky approach is used to calculate the vector R from the relationship in Eq. 6a. The Cholesky decomposition of the diffusion tensor D is determined by $\mathbf{D}=\mathbf{L}{\mathbf{L}}^T$, where L is a lower triangular matrix and each element in L is given by

$$l_{ij}=\left\{\begin{array}{cc}{\left(D_{ii}-{\sum }_{k=1}^{i-1}{l}_{ik}^2\right)}^{\frac{1}{2}},& \text{if }i=j\\ \left(D_{ij}-{\sum }_{k=1}^{j-1}{l}_{ik}{l}_{jk}\right)/s_{jj},& \text{if }i>j\\ 0,& \text{if }i<j\end{array},\right\}.$$ (7)

The terms D, F, and R are derived from three types of forces. D is derived from the friction force, F is a systematic force that includes the interaction between Brownian particles and any external force, and R is derived from a random force and determined by the Cholesky decomposition of the diffusion tensor D.

Following the first-order estimate of the translation and rotation from Eq. 4a at time $\left(n+1\right)\text{Δ}t$, namely $\text{Δ}{\mathbf{r}}^{n,\ast }=\frac{\text{Δ}t}{k_{B}T}\mathbf{D}\left({\mathbf{r}}^n\right){\mathbf{F}}^n+{\mathbf{R}}^n$ and $\text{Δ}{\text{Ω}}_i^{n,\ast }=\frac{\text{Δ}t}{{\xi }_i}\left({\tau }_i^n+{\omega }_i^n\right)$, we calculate the forces ${\mathbf{F}}^{n+1,\text{∗}}$ and torques ${\tau }_i^{n+1,\text{∗}}$ at the end of the $n+1$ time step and use them to construct an explicit second-order update as shown in Eq. 4b.

The calculation of forces and torques are described in the supporting material. Below, we show the calculation of the diffusion tensor.

Diffusion tensor

There are two types of diffusion tensors introduced in (54), the Oseen and Rotne-Prager tensors. Our approach uses the Rotne-Prager tensor because in the BD simulation, the diffusion tensor needs to be a positive definite matrix to apply the Cholesky decomposition. The Oseen tensor becomes a nonpositive definite when the separation between the particles is small. The Rotne-Prager tensor is given by

$$\left\{{\mathbf{D}}_{ij}=\begin{array}{ll}\frac{k_{B}T}{6\pi \eta a_i}\mathbf{I},\hfill & \text{for }i=j\hfill \\ \frac{k_{B}T}{8\pi \eta r_{i,j}}\left[\left(\mathbf{I}+\frac{{\mathbf{r}}_{i,j}{\mathbf{r}}_{i,j}}{r_{i,j}^2}\right)+\frac{\left(a_i^2+a_j^2\right)}{r_{i,j}^2}\left(\frac{1}{3}\mathbf{I}-\frac{{\mathbf{r}}_{i,j}{\mathbf{r}}_{i,j}}{r_{i,j}^2}\right)\right],\hfill & \text{for }i\ne j\hfill \end{array},\right\},$$ (8)

where η is the solvent viscosity and a is the sphere radius of the particles.

Eq. 8 is for the two nonoverlapping particles, $i,j$. For overlapping particles, we use

$${\mathbf{D}}_{ij}=\frac{k_{B}T}{6\pi \eta a_{eff}}\left[\left(1-\frac{9}{32}\frac{r_{i,j}}{a_{eff}}\right)\mathbf{I}+\frac{3}{32}\frac{{\mathbf{r}}_{i,j}{\mathbf{r}}_{i,j}}{a_{eff}{r}_{i,j}}\right].$$ (9)

Here, $a_{eff}=\sqrt{a_i^2+a_j^2}$, which has been proven for $a_i=a_j$ (56) and has been proposed for $a_i\ne a_j$ (53,57).

CUDA application

To efficiently run BD simulations, we first incorporated the message-passing interface (MPI) procedure, which can compute the forces on 24 processors in parallel. However, the calculation of H requires significant memory as the system size increases. Thus, we calculated it on one processor, and as a result, only small systems could be simulated with proper time consumption.

With the increasing availability of GPU computing, CUDA implementation becomes feasible and preferable in our application. Because the time complexity to calculate forces and H is $O\left(n^2\right)$, and the Cholesky decomposition $O\left(n^3\right)$, with parallelizable algorithms, the use of GPUs has the potential to accelerate computations significantly.

Instead of nested “for” loops through all the interactions in the original MPI code, we assign many threads (basic computing elements) in CUDA and use “if” statements to accomplish the job. Namely, the $n^2$ calculations are performed in parallel instead of one by one. Thus, we assign an individual thread with index $i,j$ to calculate the interaction between beads i and j. This means that, for example, for a 100-nucleosome system, 519 million threads are assigned to calculate the interactions between all pairs of beads. Compared with the 24 processors used in computer processing unit (CPU) parallel computation, this is a huge speed up.

In the GPU architecture, the threads are grouped into blocks, which are grouped onto a 3D grid. Each block might contain up to 1,024 threads, and the 3D grid contains up to 2,147,483,647, 65,535, and 65,535 blocks in the x, y, and z dimensions. Thus, the maximum number of threads in a single GPU is ∼2 billion, which is more than enough to assign one thread for each interaction in a 100-nucleosome system. While the threads inside the same block can communicate with each other, those in different blocks can only communicate through the global memory, which increases noncalculation latencies due to the launching of extra kernel invocations (parallel function executions) and transferring of data to the global memory. Namely, in the CUDA code, each calculation is performed by a separate kernel (i.e., kernel A calculates forces, and kernel B calculates translation), and each kernel uses all needed threads; only after kernel A’s calculations are done are the data saved to the global memory, and kernel B calculations begin.

Implementation

To convert the MPI to a CUDA application, we create $M_{force}$ as an N × N matrix where each element $M_{forc{e}_{ij}}$ is a three-component row vector corresponding to the force between beads i and j as

$$M_{force}=\left[\begin{array}{llll}{M}_{forc{e}_{11}}\hfill & M_{forc{e}_{12}}\hfill & \dots M_{forc{e}_{1N}}\hfill & \hfill \\ M_{forc{e}_{21}}\hfill & M_{forc{e}_{22}}\hfill & \dots M_{forc{e}_{2N}}\hfill & \hfill \\ ⋮\hfill & ⋮\hfill & \hfill & ⋮\hfill \\ M_{forc{e}_{N1}}\hfill & M_{forc{e}_{N2}}\hfill & \dots M_{forc{e}_{NN}}\hfill & \hfill \end{array}\right],M_{forc{e}_{ij}}=\left[\begin{array}{lll}{F}_{i{j}_1}\hfill & F_{i{j}_2}\hfill & F_{i{j}_3}\hfill \end{array}\right].$$

Thus, we calculate all energies and forces as described in the supporting material and store them in $M_{force}$. Then, we reduce the matrix by adding the collection of rows in $M_{force}$ into one row ($F_j=\sum _{i=1}^N{M}_{forc{e}_{ij}}$) to obtain the array of the forces F applied to each bead and feed it into Eqs. 4a and 4b.

Similarly, we create $M_D$ as an N × N matrix where each element $M_{D_{ij}}$ is a 3 × 3 matrix that stores the diffusion tensor coming from the hydrodynamic interactions between beads i and j as

$$M_D=\left[\begin{array}{llll}{M}_{D_{11}}\hfill & M_{D_{12}}\hfill & \dots M_{D_{1N}}\hfill & \hfill \\ M_{D_{21}}\hfill & M_{D_{22}}\hfill & \dots M_{D_{2N}}\hfill & \hfill \\ ⋮\hfill & ⋮\hfill & \hfill & ⋮\hfill \\ M_{D_{N1}}\hfill & M_{D_{N2}}\hfill & \dots M_{D_{NN}}\hfill & \hfill \end{array}\right],M_{D_{ij}}=\left[\begin{array}{lll}{D}_{i{j}_{11}}\hfill & D_{i{j}_{12}}\hfill & D_{i{j}_{13}}\hfill \\ D_{i{j}_{21}}\hfill & D_{i{j}_{22}}\hfill & D_{i{j}_{23}}\hfill \\ D_{i{j}_{31}}\hfill & D_{i{j}_{32}}\hfill & D_{i{j}_{33}}\hfill \end{array}\right].$$

As shown in Eqs. 8 and 9, each component $D_{ij}$ can be calculated individually. Thus, it is straightforward to assign 3N × 3N threads, one for each $D_{ij}$ calculation, to obtain the dense matrix $M_D$ with a time complexity of only $O\left(1\right)$ and use it directly as D in Eqs. 4a and 4b.

For the Cholesky decomposition of the linear system, we implement the library provided by the vendor cuSolver and use the function “cusolverDnDpotrf,” which computes the Cholesky factorization of a real double-precision Hermitian positive-definite matrix, suitable for the diffusion tensor matrix. In theory, positive definiteness is guaranteed by the diffusion tensor construction (56), but in practice, round-off error can make the matrix nonpositive definite. When this happens (infrequently), we use the diffusion tensor from the previous BD step.

The parallelization of the Cholesky decomposition is not straightforward, since the calculation of each element ij depends on information from the previous elements 1 to i − 1 and 1 to j − 1. In practice, this is calculated column by column rather than element by element. By looping through the columns, the results of each column_i can be achieved by calling three parallelized CUDA kernels: square root kernel (calculates the diagonal element ij), normalization kernel (updates all the elements below the current diagonal element ij), and submatrix update kernel (updates the submatrix ranging from i + 1, i + 1 to $N,N$). In this way, the time complexity of the Cholesky decomposition is reduced from $O\left(n^3\right)$ to $O\left(n\right)$.

Since we do not need D and F directly, just their product (Eqs. 4a and 4b), besides the calculation of the large matrices $M_{force}$ and $M_D$ and the Cholesky decomposition, we calculate the translation and rotation inside the CUDA kernel. The efficient way to do so is to assign N threads for each bead i and let each thread i only calculate one translation and rotation for bead i with the corresponding data from D and F based on Eqs. 4a and 4b and Eqs. 5a and 5b.

Because GPUs cannot read and write data, after calculations are complete, we transfer the results of the locations and orientations of all the beads back from GPU to CPU to generate the desired files (i.e., trajectory, energy, checkpoint files).

Results and discussion

Performance

To assess performance, we run on both CPU and GPU simulations of chromatin fibers ranging from 2- to 100-nucleosome cores with nucleosome repeat length (NRL) = 191 bp, histone tails, and one LH per nucleosome. The timestep is 1 ps. The CPU used is an Intel Xeon Platinum 8268 24C 205W 2.9 GHz processor, and the GPU is an Nvidia Tesla v.100 16 GB processor. We recorded times for 1,000 simulation steps using the MPI code run on the CPU and the CUDA code run on the GPU. We display the wall time in Fig. 2, and its breakdown by task in Fig. 3. The wall time measures the actual time taken by the program to complete a task, including calculations, writing/reading, and communication.

Figure 2.

CPU versus GPU architecture for chromatin fibers of 2–100 nucleosomes. Wall time needed to complete 1,000 steps with a time step of 1 ps when running on CPU (triangle) versus GPU (circle) as a function of system size. Blue: plot in seconds. Red: plot in logarithmic scale. To see this figure in color, go online.

Figure 3.

Computation time breakdown on CPU and GPU for chromatin systems of 2–100 nucleosomes. (A–D) Force (A), diffusion tensor (B), Cholesky decomposition (C), and other calculations (D). Note different timescales for each plot. The force includes all forces and energy calculations mentioned in the supporting material, such as stretching, bending, twisting, nonbonded electrostatics, and excluded volume. Other calculations include reading and writing from/to files; translation and rotation of the beads; Euler angle calculations; and math functions such as random, exp., log, sin, cos, tan, cross product, norm, etc. To see this figure in color, go online.

The CPU wall time increases with the system size steeply (Fig. 2, blue triangles). On the other hand, no increase in wall time is observed on GPUs (Fig. 2, blue circles). For example, for a 100-nucleosome system, a calculation of 1,000 steps on CPUs takes 25 h, versus less than 10 min on GPUs. As observed in the plot with log scale (Fig. 2, red curves), the CPU calculation (red triangles) is advantageous compared with the GPU (red circles) for very small systems (1–9 nucleosomes) due to the “noncalculation latencies” associated with data transfer, kernel launch, and synchronization. As mentioned above, we must transfer data, such as locations and orientations of all beads, as well as random numbers between GPUs and CPUs. The kernel launch overhead reflects the time required for the GPU to execute a launched kernel. Because we cannot limit usage to a single GPU block, we must launch additional kernels, and this increases the time needed in each step of the simulation. During the simulation, each thread either belonging to the same block or different blocks needs to be synchronized to ensure that data are ready for the calculations in the next step. While synchronization of threads belonging to the same block is fast due to the use of shared memory, synchronization of threads belonging to different blocks requires the use of the global memory, which is more time consuming.

Fig. 3 shows the profiling breakdown for wall times by the MPI code for the CPU and the CUDA code for the GPU. The wall time includes the calculation time and the noncalculation latencies. When the system size increases, the time consumption of noncalculation latency increases as $O\left(n\right)$. For the MPI code, all calculations show exponential trends, and for CUDA, the calculations show linear and exponential trends (which are constant and linear trends when ignoring the noncalculation latency).

Fig. 3A shows that in the MPI code, the force calculations range from 0 to ∼45,000 s for systems containing 2–100 nucleosomes, which is due to the $O\left(n^2\right)$ complexity with system size. Meanwhile, the time is ∼400 s for all the systems on CUDA because this calculation is performed in parallel on many threads. In addition (see implementation), each thread handles one interaction between one pair of Brownian particles, and since the nucleosome core is treated as one particle, the calculation for 300 charged beads is assigned to one thread. For calculations between two nucleosome cores, 300 × 300 = 90,000 calculations are done on one thread, explaining the force calculation time of 400 s. Although this approach might increase the force calculation time compared with treating each nucleosome charge individually, it reduces the force matrix size and thus the memory usage, allowing us to simulate larger systems.

The calculations for the diffusion tensor and Cholesky decomposition are performed every 20 steps. Fig. 3 B shows that the time used for the diffusion tensor calculation is smaller than the time used for the force calculation due to our treatment of the nucleosome core as one hydrodynamic particle instead of 300 charged beads as in the force calculation. The calculation on the MPI code takes as much as 1,400 s for the 100-nucleosome system, while on the CUDA code, it takes less than 1 s for all the systems.

Fig. 3C shows that the most time-consuming computation is the Cholesky decomposition. Even though the Cholesky decomposition is calculated every 20 steps, it requires 36,000 s in the MPI code due to its $O\left(n^3\right)$ complexity. In comparison, in the CUDA code, the Cholesky decomposition only increases from 0 to ∼50 s ($O\left(n\right)$).

Fig. 3D shows the time for other calculations (such as reading and writing from and to files; translation and rotation of the beads; Euler angles; math functions such as random, exp., log, sin, cos, tan, cross product, norm, etc.) is also reduced from the CPU to the GPU due to the parallel implementations. The GPU time shows a linear trend due to remaining serial computations that increase with system size, such as random number generation at each step. Note that the “zigzag” pattern results from system sizes not being divisible by the kernel’s block size, in which case larger matrices filled with zeros need to be allocated in the GPU memory before transferring the data and launching the kernel, increasing the allocation time. Still, the time consumed for these calculations remains very small.

Validation

For our chromatin BD simulations, we set the temperature to 293.15 K and the time step to $\text{Δ}t=1$ ps. We saved coordinates every 100 steps and simulated at least five independent replicas started from different random seeds for each trajectory. Full details of parameters are provided in the supporting material.

Translational diffusion coefficient of nucleosomes

The velocity of the Brownian motion is defined by the translational diffusion coefficient (D), which depends on the size of the particle, the surface structure, and the ionic concentration and viscosity of the medium.

Here, we calculate D from the Einstein relation for 3D space as

(10)

where t is the simulation time, $t_0$ is the initial time, and is an average over the simulation time.

We calculate D for nucleosome arrays containing 1 to 4 nucleosomes with NRL = 209 bp at salt of 150 mM NaCl. We perform 10 independent trajectories started from different random seeds for 700 ns and compare the average (D) values with experimental data (Fig. 4) by (8,58,59,60) of 1- to 4-nucleosome systems obtained by fluorescence correlation spectroscopy experiments that measure fluorescence intensity fluctuations due to particle diffusion. These in vitro experiments are comparable with our BD simulations because major relevant parameters can be matched. Nonetheless, in vivo nucleosome diffusion experiments in the context of interphase chromatin report values on the order of ${10}^{-9}$ to ${10}^{-10}$ cm²/s (61,62), close to predicted values here.

Figure 4.

Diffusion coefficients for 1- to 4-nucleosome systems obtained from BD simulations (green) compared with experimental results (blue) from (8,58,59,60). To see this figure in color, go online.

As shown in Fig. 4, the trend for diffusivity as a function of system size is reproduced by our BD simulations, and the values are close to experimental data.

Salt-dependent chromatin folding

The effect of salt concentration on chromatin folding is well known. Here, we study the folding of 12-nucleosome chromatin arrays at different monovalent salt concentrations and compare our results with two sets of experimental values obtained by Hansen et al. (63) and Howe et al. (64). In particular, we study 1) arrays with NRLs = 173, 191, and 209 bp, without LHs, and with 20, 50, 100, 150, and 200 mM NaCl, and 2) arrays with an NRL of 209 bp, with and without LH, and with 25, 50, 75, and 100 mM NaCl.

Fibers are simulated for 700 ns with a time step of $\text{Δ}t=1$ ps. For the systems without LHs, we start the simulations from an ideal zigzag extended geometry, whereas for the systems with LHs, which have 336 more particles, we start them from the equilibrated structures obtained with MC simulations to reduce the simulation time needed to reach equilibrium conformations. We run 10 independent trajectories starting with different random seeds. Results are obtained from the last 10 ns of each simulation and averaged over the 10 trajectories.

We calculate the sedimentation coefficient at each salt concentration as follows:

$$S_{20,w}=\left(\left(S_1-S_0\right)\text{∗}\rho +S_0\right)\text{∗}\left(1+\frac{R_1}{N_C}\sum _i\sum _j\frac{1}{R_{ij}}\right),$$ (11)

where $S_0$ and $S_1$ are the sedimentation coefficients of a mononucleosome without LHs ($S_0=11.1$ S) (65) and with LHs ($S_1=12$ S) (66), respectively, ρ is the LH density on the fiber, $R_1$ is the radius of a nucleosome ($R_1=$ 5.5 nm), $N_C$ is the number of nucleosomes in the chromatin fiber, and $R_{ij}$ is the distance between the nucleosomes i and j.

BD simulations accurately reproduce the fiber compaction as a function of salt concentration observed experimentally. The sedimentation coefficient increases with the salt concentration, with a plateau reached at around 100 mM NaCl (Fig. 5). This difference in compaction is also clearly seen in the fiber configurations. Fibers at low salt appear unfolded, whereas fibers in physiological conditions are compact. Similar results are seen for the other two fibers with NRLs = 173 and 191 bp (see Table 1). Moreover, in agreement with the observations of Hansen et al. in (63), the increase of fiber compaction with salt concentration does not depend on the NRL.

Figure 5.

Salt-dependent folding of 12-nucleosome chromatin fibers. Sedimentation coefficients for 12-nucleosome fibers with NRL = 209 bp, without LHs, and with 10, 20, 50, 100, 150, and 200 mM NaCl. Blue dashed line represents experimental data taken from (63). Green squares with error bars show the average and standard deviation obtained from our BD simulations. Chromatin fiber configurations at increasing salt concentrations, from 10 to 200 mM. Linker DNA is shown in orange, and histone tails beads are shown in green (H3), yellow (H2A), red (H2B), and blue (H4). To see this figure in color, go online.

Table 1.

Chromatin compaction assessment by sedimentation coefficients of 12-nucleosome fibers with NRLs = 173, 191, and 209 bp and with 20, 50, 100, 150, and 200 mM NaCl

NRL (bp)	Monovalent salt (mM)	Computed $S_{20,w}$ (S)
173	20	35.4 $\pm$ 0.4
	50	41.5 $\pm$ 0.8
	100	47.5 $\pm$ 0.9
	150	49.8 $\pm$ 1.7
	200	51.9 $\pm$ 2.0
191	20	32.7 $\pm$ 0.5
	50	37.0 $\pm$ 1.1
	100	40.0 $\pm$ 1.4
	150	41.0 $\pm$ 1.1
	200	42.0 $\pm$ 1.3
209	20	32.1 $\pm$ 0.4
	50	34.9 $\pm$ 0.4
	100	36.0 $\pm$ 0.5
	150	36.5 $\pm$ 0.7
	200	36.5 $\pm$ 0.6

See Fig. 5 for plots with NRL = 209 bp.

At a single salt concentration, an increase in NRL produces an increase in fiber diameter, and two distinct chromatin structural classes emerge, as previously shown by Rhodes and coworkers by cryo-electron microscopy (67). As we show in Fig. 6, although absolute values are smaller, BD simulations reproduce the experimental trend for fiber diameters, showing a first structural class, composed of fibers with NRLs = 177 to 207 bp, and a second class, composed of fibers with NRLs = 217 to 237 bp. Additionally, the fiber structures resemble those seen by Rhodes et al. (67). These simulations of 20 ns were started from equilibrated structures obtained from MC simulations. Five independent trajectories were run for each system starting from different random seeds, and the last 10 ns of each replica’s trajectory were used for analysis.

Figure 6.

Fiber diameters for NRL-dependent folding of chromatin fibers. Fiber systems of 72, 52, 52, 61, 47, 55, 66, and 56 nucleosomes, with NRLs = 173, 182, 191, 200, 209, 218, 227, and 237 bp, respectively, without LHs, and with 120 mM NaCl. Blue dashed line represents experimental data taken from (67). Green squares with error bars show the average and standard deviation obtained from our BD simulations. Representative chromatin fiber configurations at NRLs = 173, 191, 209, and 237 bp are shown with linker DNA in orange and histone tails beads in green (H3), yellow (H2A), red (H2B), and blue (H4). To see this figure in color, go online.

As shown in Fig. 7 A, BD simulations can also accurately reproduce the experimental fiber compaction for arrays with NRL = 209 bp with and without LHs. As before, the sedimentation coefficient increases with salt concentration, and LH further increases compaction compared with fibers without LHs. Fiber configurations (Fig. 7 B) clearly show the impact of salt and LHs on compaction.

Figure 7.

Salt- and LH-dependent folding of 12-nucleosome chromatin fibers. (A) Sedimentation coefficients of chromatin fibers with NRL = 209 bp, with and without LHs, and with 25, 50, 75, and 100 mM NaCl. Green (with LHs) and red (without LHs). Dashed lines represent experimental data taken from (64), and squares with error bars show average and standard deviation obtained from BD. (B) Fiber configurations with and without LHs at 25 and 100 mM NaCl. To see this figure in color, go online.

Salt, Mg²⁺, and LH roles in chromatin compaction

In a previous study, we combined our MC simulations of chromatin fibers with electron microscopy nucleosome interaction capture experiments to study the internal organization of chromatin fibers (68).

There, we assessed chromatin compaction at different monovalent salt concentrations and with/without Mg²⁺ and LHs. We showed that physiological salt concentrations and LHs produce a zigzag topology with dominant k ± 2 internucleosome contacts. Addition of Mg²⁺ further compacts the fibers and increases the bending of the linker DNA, promoting k ± 1 contacts.

Here, we reproduce these results by simulating 12-nucleosome chromatin arrays with NRL = 209 bp in the following conditions: 1) 10 and 150 mM NaCl, 2) 10 and 150 mM NaCl plus 1 LH per nucleosome, and 3) 150 mM NaCl plus 1 LH per nucleosome and 1 mM Mg²⁺. BD simulations are run for 100 ns starting from MC-equilibrated configurations in the case of fibers with LHs and from ideal zigzag structures in the case of fibers without LHs. Note that although similar results are expected from MC and BD, MC samples the equilibrium conformational ensemble, while BD samples thermal fluctuations around representative conformations.

The Mg²⁺ effect is modeled using our phenomenological approach developed in (68). That is, we reproduce the effect of 1 mM Mg²⁺ on fiber compaction by reducing the DNA persistence length from 50 to 30 nm, according to published data (69), and by increasing the inverse Debye length in the electrostatic term from 1.48 to 2.52 nm^-1 to reflect altered electrostatic repulsion between linker DNAs.

BD simulations reproduce the experimental trends, as well as the MC results (Fig. 8 A). Fibers without LHs show the same compaction as the experimental and MC values. Such compaction increases with the addition of LHs and further with the addition of Mg²⁺ (Fig. 8 B). For low salt with LHs, BD conformations are slightly more open than experimental measurements suggest, while MC configurations are much more compact. For physiological salt with LHs, both BD and MC fibers are less compact than the experiment suggests. This is likely due to our fixed (rather than dynamic) LH treatment.

Figure 8.

Effect of salt, Mg²⁺, and LHs on chromatin compaction of 12-nucleosome chromatin fibers. (A) Sedimentation coefficient as a function of simulation condition for five fiber conditions (left to right): (a) 0.01 mM NaCl, no LH; (b) 0.15 mM NaCl, no LH; (c) 0.01 mM NaCl, with LH, no Mg²⁺; (d) 0.15 mM NaCl, with LH, no Mg²⁺; and (e) 0.15 mM NaCl, with LH, with Mg²⁺. Blue circles represent experimental data and red circles MC results, both taken from (68). Green squares with error bars show averages and standard deviations obtained from our BD simulations. (B) Fiber configurations from (b), (d), and (e), respectively. To see this figure in color, go online.

Role of LH in chromatin transitions

The role of LH in compacting chromatin is well known (70). Recently, we further studied how LH density, ρ, regulates chromatin architecture in relationship with the development of lymphoma (43). There, we showed that ρ triggers a transition from a flexible and globular to a rigid and straight configuration that could be related to the overexpression of genes that drive lymphoma when LHs are mutated or deficient.

Here, we reproduce this transition by studying the same chromatin fibers for different ρ values. In particular, we study 50-nucleosome arrays with an irregular DNA linker length distribution found in mammalian cells (71), and $\rho =$ 0, 0.25, 0.5, and 0.75 and 1 LH per nucleosome, with LHs randomly distributed along with the fiber. Simulations of 100 ns are started from equilibrated structures obtained from MC simulations. Five independent trajectories are run for each system starting from different random seeds, and the last 10 ns of each replica’s trajectory are used for analysis.

To characterize the transition, we measure the fiber packing ratio as the number of nucleosomes contained in 11 nm of fiber length as

$$P_R=\frac{11\text{∗}{N}_C}{Fiber-length},$$ (12)

where $N_C$ is the total number of nucleosomes, and the fiber length is calculated by defining the fiber axis with a cubic smoothing spline interpolation to the nucleosomes x, y, and z coordinates (see (72)).

We also measure the internucleosome interaction patterns. That is, we determine the tail-mediated interactions between each pair $i,j$ of cores in the chromatin fiber, constructing an interaction intensity matrix $I^{\text{′}}\left(i\text{,}j\right)$ that describes the fraction of configurations that nucleosome pairs interact with one another. An interaction is defined when any nucleosome tail i is less than 2 nm apart from any charged bead or tail of nucleosome j. The internucleosome matrix $I^{\text{′}}$ is further decomposed into a normalized 1D plot that represents the relative intensity of interactions between nucleosomes separated by k neighbors as

$$I\left(k\right)=\frac{{\sum }_{i=1}^{N_C}{I}^{\text{′}}\left(i\text{,}i\pm k\right)}{{\sum }_{j=1}^{N_C}I\left(j\right)}.$$ (13)

As we see from Fig. 9 A, BD simulations accurately reproduce the compaction trends obtained by MC simulations. As expected, fiber compaction increases with ρ. The internucleosome interaction plots (Fig. 9 B) show that lower ρ produces high k ± 1 contacts due to higher linker DNA bending and a more globular and disordered structure, whereas higher ρ increases the k ± 2 interactions, which indicates a strong zigzag topology. Fiber configurations (Fig. 9 C) reveal this structural transition from a globular and loose fiber to a straight and compact fiber as ρ increases.

Figure 9.

The effect of ρ on compaction of a 50-nucleosome fiber. (A) Packing ratio as a function of ρ (LH density) as obtained from BD simulations (green) and MC simulations (red). (B) Internucleosome interactions for each ρ from BD. (C) Representative fiber configurations from BD at each ρ. To see this figure in color, go online.

Role of histone tails in chromatin folding

Many experimental studies have shown how truncation of all tails or specific tails, such as H3 and H4, affect chromatin compaction (73,74,75). As we have shown (45), histone tails mediate chromatin compaction and higher-order folding through internucleosome interactions, which are regulated by salt conditions and the presence of LHs.

To further validate our BD approach, we reproduce sedimentation coefficient values obtained in (73) on the effect of histone tails in chromatin folding. In particular, we study 12-nucleosome fibers with NRL = 207 bp and without LHs in the following conditions: 1) all tails wild-type at 10 and 150 mM NaCl; 2) H2A and H2B tails truncated at 10, 20, 75, and 150 mM NaCl; 3) H3 and H4 tails truncated at 10, 20, 75, and 150 mM NaCl; and 4) all tails truncated at 10 and 150 mM NaCl. To truncate any tails, we set selected tail beads charges to zero.

Each system is run for 700 ns with $\text{Δ}t=1$ ps starting from ideal zigzag structures. Data are collected over the last 10 ns of five independent replicas started from different random seeds.

As is evident by the sedimentation coefficient values (Fig. 10 A) and chromatin fiber configurations (Fig. 10B), truncation of all tails produces a dramatic opening of the chromatin fibers. Similar results are obtained when H3/H4 tails are truncated, but the extent of the effect is smaller. In the case of H2A/H2B truncation, fibers look similar to the wild-type fibers, with almost the same sedimentation coefficients. These results are in agreement with our previous findings demonstrating the importance of histone tails in chromatin folding and, particularly, that H3 and H4 tails have a major role by mediating internucleosome interactions through cross fiber contacts that enhance long-range interactions (45).

Figure 10.

Effect of histone tails on chromatin folding at different monovalent salt concentrations for 12-nucleosome fibers. (A) Sedimentation coefficients as a function of salt concentration and wild-type/truncated tails. Green: all tails wild-type at 10 and 150 mM NaCl. Red: truncated H2A/H2B tails at 10, 20, 75, and 150 mM NaCl. Blue: truncated H3/H4 tails at 10, 20, 75, and 150 mM NaCl. Magenta: all tails truncated at 10 and 150 mM NaCl. (B) Fiber configurations obtained at 150 mM NaCl with all tails wild-type, H3/H4 tails truncated, and all tails truncated. To see this figure in color, go online.

Moreover, as shown in Table 2, our trends on sedimentation coefficients reproduce the experimental trends of (73) reasonably. Our BD computed values generally reflect slightly more compact fibers.

Table 2.

Chromatin compaction assessment by sedimentation coefficients of 12-nucleosome fibers with different tails

Array type	NaCl (mM)	Computed $S_{20,w}$ (S)	Experimental $S_{20,w}$ (S)
All tails wild-type	10	29.3 $\pm$ 0.2	30
	150	36.5 $\pm$ 0.7	36.5
Truncated H2A/H2B	10	29.0 $\pm$ 0.1	29.5
	20	31.3 $\pm$ 0.6	31
	75	34.3 $\pm$ 0.8	32
	150	35.8 $\pm$ 1.7	34
Truncated H3/H4	10	26.6 $\pm$ 0.6	26
	20	28.8 $\pm$ 0.6	27.5
	75	32.4 $\pm$ 1.0	28.5
	150	33.9 $\pm$ 1.0	31
All tails truncated	10	24.6 $\pm$ 0.1	22
	150	32.4 $\pm$ 0.4	28

See Fig. 10.

Summary and outlook

We have developed an efficient BD simulation protocol for GPU computing architectures to simulate chromatin fibers using our mesoscale model at nucleosome resolution. With the CUDA implementation, our BD is faster compared with a CPU implementation by two orders of magnitude, which allows us to study kb-range chromatin fibers with hundreds of nucleosomes in a few days.

Our mesoscale model for chromatin fibers has proven valuable for many applications including metaphase chromatin folding (38), compartmentalization (76), mouse cell differentiation (40), lymphoma-associated genome folding (43), yeast cell genome silencing mechanism (77), and gene folding (39,41). Although we originally started our chromatin modeling using BD (32), we moved to MC as our model increased in complexity with the introduction of flexible histone tails and LH variants with different orientations, as MC was more efficient for large systems. Here, we demonstrate that, with the advance of GPU computing, BD simulations of large nucleosome fibers are feasible with our nucleosome-resolution chromatin mesoscale model. We show the agreement of our BD results with experimental measurements such as diffusion coefficients and chromatin sedimentation on different systems, including nucleosomes at different salt concentrations and with/without LHs, ions, and tails, as well as the effect of different LH densities. With BD, dynamic chromatin problems at the nucleosome level can be studied, including chromatin flexibility, effect of binding proteins on chromatin mobility, and dynamics of gene loops.

Although BD is not likely efficient for folding fibers from ideal zigzag structures into equilibrium structures, it is suitable to study the dynamics of systems around equilibrium. Moreover, BD can be used to generate and simulate efficiently fiber topologies based on Hi-C maps. Our CUDA code ideas could also be applied to MC sampling to quickly and efficiently fold fibers into equilibrated structures in parallel. Together, MC and BD implementations for GPU architectures will allow the study of Mb-range chromatin fibers, a size not reached until now for nucleosome resolution chromatin models with explicit histone tails and LHs.

Author contributions

Z.L. designed the research, developed the BD simulation C++ code with MPI and CUDA implementation, performed data analysis, prepared figures, and wrote the manuscript. S.P.-L. performed results analysis, wrote the manuscript, prepared figures, and performed some MC simulations. T.S. designed the research, reviewed the data and results analysis, wrote the manuscript, and obtained funding.

Editor: Alberto Perez.

Data availability

The working BD code has been deposited in GitHub under the Schlick lab group: https://github.com/Schlicklab/Brownian_Dynamics_CUDA.