Solving the Hubbard model with neural quantum states

Abstract

The rapid development of neural quantum states (NQS) has established it as a promising framework for studying quantum many-body systems. In this work, by leveraging the transformer-based architectures and developing efficient optimization algorithms, we achieve the state-of-the-art results for the doped two-dimensional (2D) Hubbard model, arguably the minimum model for high-Tc superconductivity. Interestingly, we find different attention heads in the NQS ansatz can directly encode correlations at different scales, making it capable of capturing long-range correlations in strongly correlated systems. With these advances, we find evidence for the half-filled stripe in the ground state of 2D Hubbard model with the next nearest neighboring hopping, consistent with experimental observations in cuprates. Our work establishes NQS as a powerful tool for solving challenging many-fermions systems.

Accurately solving the doped two dimensional Hubbard model is challenging due to strong correlations and finite size effects. Here, the authors present a transformer based neural quantum state with an adaptive optimization scheme that achieves state of the art accuracy on large lattices, capturing stripe ground states and reducing boundary artifacts.

Introduction

The Hubbard model^1–3 is the iconic model for studying the correlation effect in many-electron systems. Though its Hamiltonian is simple, the physics it can harbor is extremely rich, including quantum magnetism, Mott insulator, charge and spin density waves, and so on. Following the discovery of cuprate high-Tc superconductors, the two-dimensional (2D) Hubbard model has attracted increasing attention due to its potential connection to the microscopic mechanism of high-Tc superconductivity^4,5. The study of the 2D Hubbard model relies mainly on numerical many-body methods due to the lack of analytic solutions. In the past few decades, a variety of many-body methods were developed, including density matrix renormalization group (DMRG)⁶ and its higher dimension generalization in the language of tensor network states^7,8, such as projected entangled pair states (PEPS)⁹, quantum embedding methods such as dynamical mean-field theory (DMFT)¹⁰, density matrix embedding theory (DMET)¹¹, and different flavors of quantum Monte Carlo approaches^12–14 and so on. In the past, different methods usually gave inconsistent results due to the limitations of each method¹⁵. In recent years, large scale cross-validation has been performed among different methods, and consensus has begun to reach about the ground state of the doped 2D Hubbard model, with¹⁶ and without the next nearest neighboring hopping¹⁷.

On another front, researchers are also striving to develop more accurate and efficient new methods, aiming to provide more reliable results for strongly correlated systems like the Hubbard model to unveil the underlying exotic physics. In recent years, benefiting from the rapid development in the fields of neural networks and machine learning, Neural Quantum States (NQS) have emerged as a promising approach for solving quantum many-body systems. Significant progress has been made in the development of advanced NQS methods in quantum systems since their inception by Carleo et al.¹⁸, which have markedly improved the accuracy of NQS approaches, establishing them as a powerful framework for investigating systems in quantum chemistry^19–21 and interacting spin models^22–24, as shown in Fig. 1A. In light of these significant achievements, a crucial question arises: how effective is this new paradigm for the study of the more challenging strongly correlated fermionic lattice models?

Fig. 1. Application, architecture, optimization, and performance of Neural Quantum States (NQS) on the Hubbard model.

A Guided by the variational principle, NQS are powerful and versatile, applicable to a wide range of quantum systems, including ab initio quantum chemistry, spin models, and correlated fermionic models. B Various neural network architectures can be used to construct NQS, with increasing expressivity from Restricted Boltzmann Machines (RBM)²⁵, Multi-Layer Perceptrons (MLP)^27,28,32,36 to Transformers. These networks are optimized using methods inspired by quantum information, including Min-SR^37,38, SPRING³⁹, and the advanced MARCH method proposed in this work. C A performance comparison of different NQS methods on the 2D Hubbard model, plotting simulation accuracy against system size. We benchmark the accuracy on systems where exact solutions are available (see Supplementary Note 3 for details). This work achieves higher accuracy on larger system sizes than previously reported in other notable NQS studies, demonstrating an advancement in the field.

In this direction, extensive effort has been made to investigate the utilization of NQS methods in the context of the Hubbard model, particularly in the most challenging 2D lightly doped cases. Early attempts to apply NQS approaches to the 2D Hubbard model^25,26 faced challenges in accurately representing fermionic nodal structures. Subsequent work by Luo et al.²⁷ introduced a neural network backflow ansatz that substantially improved the representation power of the wavefunction, demonstrating its effectiveness on systems as large as 12 × 8. More recently, in 2022, Robledo et al.²⁸ achieved highly accurate results for various geometries of the Hubbard model up to 8 × 8 by combining Hidden-Fermion Determinant States (HFDS) with Stochastic Reconfiguration (SR)^29–31. Despite this notable progress, scaling NQS simulations to even larger, more physically relevant parameter regimes is hindered by network architectural limitations and optimization barriers^32,33.

In this work, we make significant progress in expanding the scalability and accuracy of the NQS approach for solving the Hubbard model with two key innovations as shown in Fig. 1B. First, a NQS architecture is proposed, which leverages the self-attention mechanism³⁴, a technique proven to be very effective in scaling up³⁵. The built-in long-range structure makes the NQS ansatz different from tensor network states, which consist of only local tensors. This ensures that the NQS can precisely capture long-range correlations in strongly correlated systems and can accurately handle larger systems, which is infeasible for other neural network architectures like Restricted Boltzmann Machine (RBM)²⁵ and Multi-Layer Perceptron (MLP)^27,28,32,36. Second, we propose an enhanced optimization algorithm, termed Moment-Adaptive ReConfiguration Heuristic (MARCH), designed to achieve substantially faster and more stable convergence than previous optimization algorithms such as SR^29–31, Min-SR^37,38, and SPRING³⁹, especially for large and frustrated systems.

With these architectural and algorithmic advancements, we achieve state-of-the-art variational ground state energy of the 2D Hubbard model at larger sizes up to 16 × 16, for both open and periodic boundary conditions, as shown in Fig. 1C. As a result, we confirm the filled stripe ground state in the pure Hubbard model and identify a half-filled stripe ground state with the next nearest neighboring hopping $t^{\prime }=-0.2$, consistent with experimental observations in cuprate⁴⁰. Moreover, in the $t^{\prime }=-0.2$ case, we find the stripe is horizontal, i.e., stripe arrangements prefer the longer side of the geometry. This is quite different from previous calculations on cylinders¹⁶, indicating the effectiveness of our method in eliminating the boundary effects. We also have a careful analysis of the structure of our NQS ansatz and find that different attention heads can directly encode correlations at different scales after optimization, making it capable to capture long-range correlations in the Hubbard model. Our developments in architectures and optimization strategies establish NQS as a promising paradigm for solving the Hubbard model. This framework extends readily to a broader range of strongly correlated systems, offering a new lens to explore exotic quantum many-body states resulting from strong correlations.

Results

A transformer-based NQS framework

The Hamiltonian of the Hubbard model is:

$$\widehat{H}=-t{\sum }_{⟨ij⟩,\sigma }{\widehat{c}}_{i\sigma }^{†}{\widehat{c}}_{j\sigma }-t^{\prime }{\sum }_{⟨⟨ij⟩⟩,\sigma }{\widehat{c}}_{i\sigma }^{†}{\widehat{c}}_{j\sigma }+U{\sum }_i{\widehat{n}}_{i\uparrow }{\widehat{n}}_{i\downarrow }.$$ 1

Here ${\widehat{c}}_{i\sigma }^{†}\left({\widehat{c}}_{j\sigma }\right)$ is the creation (annihilation) operator for an electron with spin σ at site i( j), ${\widehat{n}}_{i\sigma }={\widehat{c}}_{i\sigma }^{†}{\widehat{c}}_{i\sigma }$ is the particle number operator. The terms 〈ij〉 and 〈〈ij〉〉 denote pairs of nearest neighboring and next nearest neighboring sites, respectively. We study the 2D Hubbard model defined on square lattices with size N = L_x × L_y. Both open boundary conditions (OBC) and periodic boundary conditions (PBC) are adopted in this work. We set the nearest neighboring hopping amplitude t = 1 as the unit of energy. The on-site Coulomb repulsion is fixed at U = 8, a value relevant to cuprates^41,42. We consider both the pure Hubbard case ($t^{\prime }=0$) and the case with the next nearest neighboring hopping ($t^{\prime }=-0.2$). In this work, we focus on the lightly doped regime with hole concentration of δ = 1/8, which is the most intriguing region in both the 2D Hubbard model and in cuprates.

Figure 2 presents an overview of our NQS algorithm. As shown in Fig. 2A, we employ a transformer-based NQS, parameterized by θ, to represent the amplitude ψ_θ(n) associated with each configuration $∣\mathbf{n}⟩$. Our transformer-based NQS assigns a unique high-dimensional embedding vector to each site i. These embeddings are subsequently processed through a series of multi-head attention blocks, the core component of modern large language models⁴³. The attention block is defined as:

$$\mathrm{Attn}\left(Q,K,V\right)=\mathrm{softmax}\left(Q{K}^{\top }/\sqrt{d_H}\right)V,$$ 2

where $Q,K,V\in {\mathbb{R}}^{N\times d_H}$ are feature matrices produced by neural networks. Here, N represents the number of sites in the Hubbard model, d_H is the hidden dimension of the attention block. The i-th row of these matrices corresponds to the i-th site in the Hubbard model, arranged in dictionary order. The normalized attention scores S_attn, given by $\mathrm{softmax}\left(Q{K}^{\top }/\sqrt{d_H}\right)$, allow the block to aggregate information from all sites by assigning weights. Thus, the attention block can capture complex, long-range correlations, which is crucial for modeling strongly correlated systems like the Hubbard model. The multi-head attention block employs distinct feature matrices to capture diverse correlation patterns in the system. After self-attention layers, a linear layer is employed to produce backflow orbitals M(n). The amplitude is then derived as the determinant of orbitals occupied by electrons.

Fig. 2. NQS architecture, optimization scheme, and benchmark performance on the Hubbard model.

A An input lattice configuration is processed through a transformer-based neural network to generate backflow orbitals. These orbitals are then used to construct the NQS wavefunction ψ_θ. The variational parameters θ are optimized to find the ground state. B Optimization trajectories for MARCH and SPRING. MARCH follows a more direct path by dynamically adapting its step size for each parameter. The inset shows how SPRING's reliance on momentum might lead to overshooting, whereas MARCH's adaptive-momentum step prevents oscillations and accelerating convergence to the minimum. C Comparison of ground-state energies with PEPS (bond dimension D≥20)⁴⁸ for the pure Hubbard model on 16 × L_y system with OBC. Our NQS results consistently achieve lower variational energies than PEPS across different lattice geometries. The inset shows we also outperform reference DMRG energies⁴⁸ (with 32000 SU(2) multiplets) when L_y = 8. Statistical uncertainties from Monte Carlo sampling are smaller than the marker size.

In our work, we aim to find the ground state by minimizing the total energy of the wavefunction. In previous NQS studies¹⁸, SR-based optimizers have been used to minimize the energy. These optimizers effectively simulate the imaginary time evolution, enhancing the convergence rate over the standard first-order optimizer by exploiting the quantum geometric tensor⁴⁴. One particular method is the SPRING algorithm³⁹, which leverages the history of gradients to accelerate convergence and smooth out the optimization path. Building on this, we introduce the MARCH algorithm. MARCH enhances SPRING by also incorporating an estimate of the second moment of the gradients, drawing a direct parallel to how the popular Adam optimizer⁴⁵ improves upon Momentum⁴⁶. This addition of second-moment information is crucial for navigating the complex and high-dimensional energy landscapes that typically exist in quantum systems. It provides an estimate of the gradient’s variance, allowing MARCH to adapt the learning rate for each parameter individually. This adaptive capability offers a significant advantage, particularly when encountering challenging regions like saddle points or flat plateaus where the gradient is small. While a momentum-based method like SPRING might overshoot, MARCH can use the second-moment information to maintain progress and march confidently out of these areas. As illustrated in Fig. 2(B), this results in a more direct and efficient path to the energy minimum, leading to faster and more robust convergence.

These combined architectural and algorithmic advancements enable us to accurately tackle 2D Hubbard model with expanded sizes beyond the reach of previously NQS methods. We first perform benchmark calculations for systems under PBC for the pure Hubbard model ($t^{\prime }=0$) at half-filling, where numerical exact ground state energies can be obtained by Auxiliary Field Quantum Monte Carlo (AFQMC)⁴⁷. The NQS energies are consistent with the AFQMC results within twice the statistical errors (see Supplementary Table 1).

We then benchmark the new NQS method on the doped pure Hubbard model under OBC with varying sizes L_x and L_y. Direct comparisons of ground-state energies with state-of-the-art DMRG and PEPS⁴⁸ results are presented in Fig. 2C (with Hartree-Fock energies as reference). In narrow systems such as L_y≤6, the DMRG energy is the lowest. However, when reaching the 2D limit by increasing the width of the system, where DMRG fails to capture the area law of entanglement entropy, the NQS results surpass DMRG. Noticeably, in the entire range of L_y we studied, NQS achieves lower variational energies than those reported by state-of-the-art PEPS with bond dimension D≥20⁴⁸.

Attention mechanism reveals physical correlations

To gain deeper insight into how the transformer architecture captures the intricate physics of the Hubbard model, we examine the attention patterns learned by the network⁴⁹. In our transformer ansatz, the heads after optimization are found to specialize in capturing the correlations at different scales present in the ground-state wave function. This emergent specialization is analogous to observations in fields like natural language processing, where different attention heads in a transformer learn distinct, interpretable patterns such as syntactical dependencies³⁴. In Fig. 3, we visualize the normalized attention scores, S_attn, from the first layer for representative half-filled systems, which quantify the learned pairwise importance between sites. Similar results for the doped cases or deeper layers can be found in Supplementary Figs. 1–10.

Fig. 3. Visualization of the attention mechanism.

Shown is how a lattice site attends to another target site in the 2D pure Hubbard model for lattice sizes of 8 × 8, 10 × 10, and 12 × 12 at half filling. The four attention heads, which are equivalent and have been rearranged by pattern for clarity, each correspond to a specific physical aspect: short-range correlations, nearest-neighbor hopping, antiferromagnetic correlations, and complex and delocalized correlations in the ground state.

The results show: the first head clearly learns to prioritize local interactions, with S_attn decaying with the distance between sites to capture short-range correlations. The second head specializes in nearest-neighbor hopping, focusing its attention almost exclusively on the four adjacent sites–a fundamental process in the Hubbard model. The third head successfully identifies the emergent antiferromagnetic order, assigning high attention scores in a checkerboard pattern characteristic of this magnetic correlation. The final head appears to capture more complex, delocalized phase information, exhibiting a sophisticated pattern that is crucial for accurately representing the full complexity of the wave function. Importantly, varying the total number of attention heads in our architecture reveals that the network consistently converges to the three specific local patterns mentioned above (short-range, nearest-neighbor, and antiferromagnetic), suggesting their physical relevance. Further details regarding these structural variations are provided in Supplementary Fig. 11. Ultimately, this specialization demonstrates the ability of the transformer’s attention mechanism to identify the fundamental physical correlations governing the system.

Filled stripe order in pure Hubbard model

The ground state of the lightly doped pure ($t^{\prime }=0$) 2D Hubbard model was established as the filled stripe before¹⁶. Recently Liu et al.⁴⁸ observed a crossover of the stripe direction with the increase of system widths for systems under OBC, indicating the existence of a large finite size effect in systems with open boundaries. We performed systematic calculations of the pure Hubbard model for different sizes under both OBC and PBC. For systems with OBC, we find stripe states that match well with the results obtained from PEPS but with lower energies (see Fig. 2C). In particular, our results also demonstrate a crossover between horizontal (i.e., stripe stretching along the longer direction of the lattice) and vertical stripes (i.e., stripe stretching along the shorter direction of the lattice) as a function of system width (from L_y = 8 to L_y = 12, see Supplementary Fig. 15–16). But when we switch to PBC, the crossover vanishes (see Supplementary Fig. 18–19), indicating the crossover is a finite size and open boundary effect. The ability to handle systems under PBC with NQS can mitigate the finite size effect, allowing to obtain more reliable results in the thermodynamic limit. By studying systems under PBC, we observe a robust, filled stripe phase with wavelength of λ = 8 as shown in Fig. 4A, a result that is consistent with previous findings¹⁶. Simulations on smaller 8 × 8 and 12 × 12 lattices (see Supplementary Fig. 20-21) show clear finite-size effects due to their incommensurability with the wavelength λ = 8.

Fig. 4. Hole and spin density distributions in the ground state for different lattice sizes with U = 8 and hole doping δ = 1/8 under PBC in the Hubbard model.

The magnitude of the local spin density is represented by the size of the arrows, while the spin orientation expectation value of S_z is denoted by the up/down direction of the arrows. Hole density is depicted using a color scale. Charge and spin stripes are clearly visible in all three systems. A Results for the 16 × 16 pure Hubbard model ($t^{\prime }=0$), where a stripe state with period λ = 8 is observed. B Results for the 8 × 16 system with $t^{\prime }=-0.2$, exhibiting a stripe state with λ = 4 extending along the y-axis. C Results for the 32 × 8 system with $t^{\prime }=-0.2$, exhibiting a stripe state with λ = 4 extending along the x-axis. Notice that in the rectangular lattices for $t^{\prime }=-0.2$ (panels B and C), the stripes consistently orient themselves along the longer geometrical axis of the system (horizontal), even though both spatial orientations are allowed under PBC. Additional results can be found in the Supplementary Material.

Half-filled stripe order in $t^{\prime }$-Hubbard model

It was found that the doped pure Hubbard model may not host superconductivity for parameter relevant to the under-doped regime of cuprate superconductors⁵⁰. To pursue superconducting ground states, the simplest modification is to include the next nearest neighboring hopping ($t^{\prime }$) term. On one hand, a finite $t^{\prime }$ can account for the electron and hole doping asymmetry in the phase diagram of cuprates. On the other hand, electronic structure calculations^41,42 show a finite $t^{\prime }\approx -0.2$ term indeed exists in the one band effective Hamiltonian of cuprates. A recent study by Xu et al.¹⁷ found that the phase diagram of the Hubbard model with $t^{\prime }=-0.2$ agrees well with the cuprates, where fractional filling stripe coexists with the d-wave superconductivity in the hole doping region. The $t^{\prime }$ term frustrates the anti-ferromagnetic order in the parent compound and makes the doped stripe state fragile. Importantly, large finite size and boundary effect⁵¹ was found for the wavelength of the stripe state in the hole doping region, which fluctuates in the vicinity of λ = 4¹⁷.

We perform systematic calculations for systems under PBC with $t^{\prime }=-0.2$. With PBC, we can get rid of the boundary effect. Our calculations on large sizes show the ground state for the $t^{\prime }=-0.2$ Hubbard model is consistent with the half-filled stripe, i.e., with wavelength λ = 4, as shown in Fig. 4B. These results indicate that the previous found fluctuating wavelengths for stripe may result from the finite size and boundary conditions. The λ = 4 stripe agrees well with the experimental observation of cuprates⁴⁰, providing further evidence to support the relevance of the $t^{\prime }$ Hubbard model to cuprates.

Horizontal stripes are energetically favored over vertical stripes

Previous numerical studies on stripe order in the Hubbard and related models have been conducted on cylindrical geometries with DMRG^52–56. However, such boundary conditions and their related finite-size effect can inherently favor the formation of vertical stripe orientations. To alleviate the possible boundary effect, we study systems with PBC in both directions. We systematically study systems with size L_x × 8 where L_x is ranged from 8 to 32, with $t^{\prime }=-0.2$ and δ = 1/8 (see Eq. (1)). We consistently observe that a horizontal stripe configuration is energetically favored in the ground state while the system can accommodate half-filled stripe in both horizontal and vertical directions (see Fig. 4B, C), Fig. 5 and Supplementary Figs. 22–27.

Fig. 5. Energy difference between vertical stripe and horizontal stripe.

Kinetic energy, potential energy, and total energy results are all shown. We systematically study systems L_x × 8 with L_x ranging from 8 to 32, with $t^{\prime }=-0.2$ and δ = 1/8. We can find that the horizontal stripe has lower energy for all the systems (notice that for 8 × 8 system, vertical and horizontal stripes are the same). Moreover, the results show that the gain of energy in the horizontal stripe is mainly from the kinetic energy. Statistical uncertainties from Monte Carlo sampling are smaller than the marker size.

These results suggest the existence of a tendency toward fewer long stripes rather than multiple fragmented ones in the systems, different from previous studies on cylinders¹⁷. From Fig. 5, we can also find that the gain of energy for the horizontal stripe state is mainly from the kinetic energy, while the potential energy in the horizontal stripe is slightly higher. A simple explanation is that the formation of a long river of holes in the horizontal stripe is beneficial to the kinetic energy. A comprehensive understanding of this observation needs further investigation.

For rectangular systems with a width not divisible by 8, a horizontal stripe with wavelength λ = 4 is impossible, resulting in geometric frustration. One such example is the 16 × 10 system, for which we find that the trend for horizontal stripe wins and the ground state has a wavelength of λ = 5 while the vertical stripe state with λ = 4 has a higher energy (see Supplementary Figs. 29-30). In another example for the 16 × 12 system, the trend for wavelength 4 wins in the competition and the ground state is a vertical stripe (see Supplementary Fig. 31).

For the pure Hubbard model, it is also possible that the true ground state is a horizontal stripe. But the resolution of this issue requires the study of systems that can accommodate stripes with a wavelength 8 in both vertical and horizontal directions. The smallest size for such systems under PBC is 32 × 16, which is beyond our present capacity.

Accuracies and scalability

We then evaluate the accuracy and scalability of our approach under PBC. As illustrated in Fig. 6A, the computational cost scales polynomially (approximately O(N³)) with the number of sites. Furthermore, variance per site, a proxy for relative error when exact ground-state energies are unknown, grows slowly as the system expands up to 256 sites (Fig. 6B). Finally, Fig. 6C shows that the error can be systematically reduced by increasing the expressivity of the ansatz.

Fig. 6. Scalability and accuracy benchmarks under PBC.

A Computation time per step as a function of the number of lattice sites, demonstrating polynomial scaling. B Variance (which serves as a proxy for relative error) versus the number of sites, indicating a controlled, slow growth for larger systems when $t^{\prime }=0$. C Error (using the zero variance extrapolated value as reference) systematically decreasing as a function of the training cost on the 32 × 8 ($t^{\prime }=-0.2$) system, parameterized by the number of heads and determinants. Statistical uncertainties from Monte Carlo sampling are smaller than the marker size.

Additionally, we conduct a zero-variance extrapolation on a 16 × 4 pure Hubbard model and a 32 × 8 lattice with $t^{\prime }=-0.2$. The extrapolated ground state energies are—0.7673 and –0.7491, respectively. Therefore, the estimated errors for the two systems are 0.006 and 0.009. Furthermore, we observe that the d-wave correlation is progressively enhanced as the calculation becomes more accurate. This finding highlights the necessity for careful and controlled calculations when drawing conclusions about superconductivity, which will be our pursuit in the future. Detailed results regarding the variance extrapolation and d-wave correlations are provided in Supplementary Note 4.

Discussion

In this work, we propose a NQS framework for studying quantum many-body systems by leveraging the cutting-edge machine learning architectures and developing accurate optimization algorithms. State-of-the-art results for the 2D doped Hubbard model with expanded system sizes are achieved with this new framework. NQS enables the study of 2D Hubbard model under periodic boundary conditions, which is crucial to get rid of the boundary effect and makes the extrapolation to the thermodynamic limit reliable. By studying 2D Hubbard model with large systems under periodic boundary conditions, we confirm the filled stripe phase in the ground state of the doped pure ($t^{\prime }=0$) Hubbard model. When the next nearest neighboring hopping term ($t^{\prime }=-0.2$) is included, the ground state is found to be half-filled stripes, consistent with experimental observation in cuprates. Noticeably, the direction of the stripe for $t^{\prime }=-0.2$ is determined to be horizontal when the width and the length are not equal, different from previous studies¹⁷. These results indicate that previous calculations on systems with open boundary suffer from the boundary effect and underscore the effectiveness of the NQS method in gaining more reliable results.

We also have a careful analysis of the attention patterns learned in the ground state of 2D Hubbard model. Interestingly, we find that after the energy optimizations, different heads within our NQS specialize in encoding the correlations at different scales of the ground-state wave function. This special structure enables our NQS ansatz to effectively capture long-range correlations, which is crucial for the accurate calculation of strongly correlated quantum many-body systems.

There is much room for the future development of NQS. On one hand, the development of more efficient optimization approaches could enable the study of even large system sizes, which is crucial for more reliable finite size scaling in the doped Hubbard model. On the other hand, more sophisticated NQS ansatz for the calculation of dynamic and finite temperature properties⁵⁷ are needed to further unveil the physics of the Hubbard model and establish the microscopic theory for cuprate. We anticipate that NQS framework will continue benefiting from the development in the broad machine learning community. Nevertheless, our results demonstrate that NQS is an effective framework for solving challenging many-fermions systems.

Methods

Variational Monte Carlo

VMC is a quantum Monte Carlo method based on the variational principle of quantum mechanics. Upon choosing a wavefunction ansatz ψ_θ, minimizing the energy of this ansatz with respect to the parameters θ makes ψ_θ an approximation to the ground state. In this work, we adopt a neural network as the wavefunction ansatz. The neural network takes occupation number n as input and outputs the amplitude of the wavefunction, $⟨\mathbf{n}\mid {\psi }_{\mathit{\theta }}⟩={\psi }_{\mathit{\theta }}\left(\mathbf{n}\right)$. The energy of the wavefunction can be written as the expectation value of the local energy, i.e.

$${Ē}_{\mathit{\theta }}=\frac{⟨{\psi }_{\mathit{\theta }}\mid \widehat{H}\mid {\psi }_{\mathit{\theta }}⟩}{⟨{\psi }_{\mathit{\theta }}\mid {\psi }_{\mathit{\theta }}⟩}={\mathbb{E}}_{\mathbf{n}~{\psi }_{\mathit{\theta }}^2/{∥{\psi }_{\mathit{\theta }}∥}^2}\left[E_{\mathrm{loc}}^{\mathit{\theta }}\left(\mathbf{n}\right)\right],$$ 3

where

$$E_{\mathrm{loc}}^{\mathit{\theta }}\left(\mathbf{n}\right)={\sum }_{{\mathbf{n}}^{\prime }}\frac{{\psi }_{\mathit{\theta }}\left({\mathbf{n}}^{\prime }\right)}{{\psi }_{\mathit{\theta }}\left(\mathbf{n}\right)}⟨{\mathbf{n}}^{\prime }∣\widehat{H}∣\mathbf{n}⟩$$ 4

is the local energy. Exact calculation of the energy using Equation (3) is infeasible in an exponentially large Hilbert space. However, we can approximate this summation using the Monte Carlo method. A set of occupation numbers ${\left\{{\mathbf{n}}_i\right\}}_{i=1}^B$ is sampled from the probability distribution defined by the squared amplitude of the wavefunction using Markov chain Monte Carlo (MCMC) techniques. The local energy of occupation numbers, $E_{\mathrm{loc}}^{\mathit{\theta }}\left(\mathbf{n}\right)$, can be efficiently calculated from the Hubbard model Hamiltonian, with the number of summation terms in Equation (4) scaling linearly with the size of the system. Therefore, parameters of the neural network can be optimized to approach the ground state of the Hubbard model:

$$\mathit{\theta }*={\arg \min }_{\mathit{\theta }}{Ē}_{\mathit{\theta }}.$$ 5

Wavefunction ansatz

As illustrated in Fig. 2A, we employ a neural quantum state (NQS) ansatz to represent the many-body quantum state $∣\psi ⟩$ of a fermionic system with N lattice sites. Each individual site corresponds to a 4 dimensional physical Hilbert space spanned by $∣0⟩$, $∣\uparrow ⟩$, $∣\downarrow ⟩$, and $∣\uparrow \downarrow ⟩$. We encode the four possible tokens into a d dimensional vector via an embedding matrix $\mathbf{E}\in {\mathbb{R}}^{4\times d}$. To account for spatial arrangement, crucial due to the permutation equivariance of self-attention, these embeddings are augmented with learnable positional encodings. To this end, we add the matrix $\mathbf{P}\in {\mathbb{R}}^{N\times d}$ to the input embedding. The embedded input can be compactly written into a matrix ${\mathbf{X}}^{\left(0\right)}={\left[\mathbf{E}\left({\mathbf{n}}_1\right)+{\mathbf{P}}_1,...,\mathbf{E}\left({\mathbf{n}}_N\right)+{\mathbf{P}}_N\right]}^{\top }\in {\mathbb{R}}^{N\times d}$. Then, L Transformer blocks follow, each of which transforms the input via the following equation:

$${\mathbf{Y}}^{\left(l\right)}={\mathbf{X}}^{\left(l-1\right)}+{\mathrm{Attn}}^{\left(l\right)}\left({\mathbf{X}}^{\left(l-1\right)}\right),$$ 6

$${\mathbf{X}}^{\left(l\right)}={\mathbf{Y}}^{\left(l\right)}+{\mathrm{FFN}}^{\left(l\right)}\left({\mathbf{Y}}^{\left(l\right)}\right),$$ 7

where l ∈ [L], Attn^(l) and FFN^(l) denote the multi-head self-attention layer and the feed-forward network for the l-th Transformer block, respectively:

$${\mathrm{Attn}}^{\left(l\right)}\left(\mathbf{X}\right)={\sum }_{h=1}^H\mathrm{softmax}\left(\mathbf{X}{\mathbf{W}}_Q^{\left(l,h\right)}{\left(\mathbf{X}{\mathbf{W}}_K^{\left(l,h\right)}\right)}^{\top }/\sqrt{d_H}\right)\mathbf{X}{\mathbf{W}}_V^{\left(l,h\right)}{\left({\mathbf{W}}_O^{\left(l,h\right)}\right)}^{\top },$$ 8

$${\mathrm{FFN}}^{\left(l\right)}\left(\mathbf{X}\right)=\sigma \left(\mathbf{X}{\mathbf{W}}_F^{\left(l\right)}\right).$$ 9

The variational parameters ${\mathbf{W}}_Q^{\left(l,h\right)},{\mathbf{W}}_K^{\left(l,h\right)},{\mathbf{W}}_V^{\left(l,h\right)},{\mathbf{W}}_O^{\left(l,h\right)}\in {\mathbb{R}}^{d\times d_H}$ are the query, key, value, and output matrices of the head h, respectively. d_H = d/H is the hidden dimension of each head. ${\mathbf{W}}_F^{\left(l\right)}\in {\mathbb{R}}^{d\times d}$ is the weight matrix in the FFN^(l). The activation σ is chosen as SiLU⁵⁸.

The final output X^(L) is transformered by a linear layer to produce K sets of backflow orbitals ${\mathbf{M}}^k\in {\mathbb{R}}^{2N\times N_e}$, where k ∈ [K]. The Slater matrix Φ^k is formed by selecting rows of M^k corresponding to the occupied position in the input configuration n. The overall NQS wavefunction is a linear combination of these determinants:

$$⟨\mathbf{n}\mid \psi ⟩={\sum }_{k=1}^K\det \left[{\mathit{\Phi }}^k\right].$$ 10

Optimization

Stochastic reconfiguration (SR) is one of the most popular optimization methods for NQS. In this work, however, we propose an advanced SR algorithms named Moment-Adaptive ReConfiguration Heuristic (MARCH), most directly inspired by the SPRING algorithm³⁹. MARCH combines ideas from the widely used Adam optimizer in traditional machine learning⁴⁵ and the recently introduced efficient Min-SR algorithm^37,38.

The SR parameter update is typically defined as

$$\mathrm{d}\mathit{\theta }={\arg \min }_{\mathrm{d}{\mathit{\theta }}^{\prime }}{\mathbb{E}}_{\mathbf{n}}\left[\mid \mathbf{O}\left(\mathbf{n}\right)\mathrm{d}{\mathit{\theta }}^{\prime }-\mathit{ϵ}\left(\mathbf{n}\right){\mid }^2\right]\approx {\arg \min }_{\mathrm{d}{\mathit{\theta }}^{\prime }}\frac{1}{B}{\sum }_{i=1}^B\mid \mathit{O}\left({\mathbf{n}}_i\right)\mathrm{d}{\mathit{\theta }}^{\prime }-\mathit{ϵ}\left({\mathbf{n}}_i\right){\mid }^2,$$ 11

where $\mathbf{O}\left(\mathbf{n}\right)=\frac{\partial }{\partial \mathit{\theta }}\log ∣{\psi }_{\mathit{\theta }}\left(\mathbf{n}\right)∣-{\mathbb{E}}_{{\mathbf{n}}^{\prime }}\left[\frac{\partial }{\partial \mathit{\theta }}\log ∣{\psi }_{\mathit{\theta }}\left({\mathbf{n}}^{\prime }\right)∣\right]$, $ϵ\left(\mathbf{n}\right)=-\delta \tau \left(E_{\mathrm{loc}}\left(\mathbf{n}\right)-{\mathbb{E}}_{{\mathbf{n}}^{\prime }}\left[E_{\mathrm{loc}}\left({\mathbf{n}}^{\prime }\right)\right]\right)$, B denotes batch size in MCMC sampling.

To express the problem in a standard linear algebra form, we construct a matrix $\tilde{\mathbf{O}}$ and a vector $\widetilde{\mathit{ϵ}}$ by stacking the contributions from each of the B samples in the batch:

$$\tilde{\mathbf{O}}=\left(\begin{array}{c}\hfill \mathbf{O}\left({\mathbf{n}}_1\right)\hfill \\ \hfill \mathbf{O}\left({\mathbf{n}}_2\right)\hfill \\ \hfill ⋮\hfill \\ \hfill \mathbf{O}\left({\mathbf{n}}_B\right)\hfill \end{array}\right)\in {\mathbb{R}}^{B\times N_{\mathrm{params}}}\mathrm{and}\widetilde{\mathit{ϵ}}=\left(\begin{array}{c}\hfill \mathit{ϵ}\left({\mathbf{n}}_1\right)\hfill \\ \hfill \mathit{ϵ}\left({\mathbf{n}}_2\right)\hfill \\ \hfill ⋮\hfill \\ \hfill \mathit{ϵ}\left({\mathbf{n}}_B\right)\hfill \end{array}\right)\in {\mathbb{R}}^B.$$ 12

Since $\tilde{\mathbf{O}}$ might be ill-conditioned, the conventional method is to add some form of regularization. For example, a Tikhonov regularization is usually added to yield the regularized problem as follows:

$$\mathrm{d}\mathit{\theta }={\arg \min }_{\mathrm{d}{\mathit{\theta }}^{\prime }}\frac{1}{\lambda }\mid \mid \tilde{\mathbf{O}}\mathrm{d}{\mathit{\theta }}^{\prime }-\widetilde{\mathit{ϵ}}\mid {\mid }^2+\mid \mid \mathrm{d}{\mathit{\theta }}^{\prime }\mid {\mid }^2.$$ 13

MARCH differs from previous SR algorithms by adaptively translating and scaling dθ individually for each parameter based on the estimation of the first and second moments.

The overall objective function can be written as:

$$\mathrm{d}{\mathit{\theta }}_k={\arg \min }_{\mathrm{d}{\mathit{\theta }}^{\prime }}\frac{1}{\lambda }\mid \mid \tilde{\mathbf{O}}\mathrm{d}{\mathit{\theta }}^{\prime }-\widetilde{\mathit{ϵ}}\mid {\mid }^2+\mid \mid \mathrm{diag}{\left({\mathbf{v}}_{k-1}\right)}^{1/4}\left(\mathrm{d}{\mathit{\theta }}^{\prime }-{\mathit{\varphi }}_{k-1}\right)\mid {\mid }^2,$$ 14

where k denotes the current step, diag(v_k−1) is the diagonal matrix whose main diagonal is composed of v_k−1. The first-order moment estimator, ϕ_k = μdθ_k, acts as a momentum term. Intuitively, ϕ carries the momentum, which accelerates updates in directions of persistent gradients and dampens oscillations. The second moment estimator, ${\mathbf{v}}_k=\beta {\mathbf{v}}_{k-1}+{\left(\mathrm{d}{\mathit{\theta }}_k-\mu \mathrm{d}{\mathit{\theta }}_{k-1}\right)}^2$, tracks the volatility of the gradient. A large v_k indicates that the gradient is changing erratically, so the update of the parameter is suppressed to prevent divergence. Conversely, a small v_k implies a stable gradient, allowing the parameter to be updated more aggressively. The behavior of these estimators is governed by the hyperparameters μ, β, and λ, which are detailed in Supplementary Note 6.

Therefore, the explicit update formula can be derived. We first define $\mathit{\pi }=\mathrm{diag}{\left({\mathbf{v}}_{k-1}\right)}^{1/4}\left(\mathrm{d}{\mathit{\theta }}^{\prime }-{\mathit{\varphi }}_{k-1}\right),\mathbf{U}=\tilde{\mathbf{O}}\mathrm{diag}{\left({\mathbf{v}}_{k-1}\right)}^{-1/4},\mathit{\zeta }=\widetilde{\mathit{ϵ}}-\tilde{\mathbf{O}}{\mathit{\varphi }}_{k-1}$. The Eq. (14) can be recast as:

$$\mathit{\pi }={\arg \min }_{{\mathit{\pi }}^{\prime }}\frac{1}{\lambda }\mid \mid \mathbf{U}{\mathit{\pi }}^{\prime }-\mathit{\zeta }\mid {\mid }^2+\mid \mid {\mathit{\pi }}^{\prime }\mid {\mid }^2.$$ 15

We can reduce the computational budget from $\mathcal{O}\left(N_{\mathrm{params}}^3\right)$ to $\mathcal{O}\left(B^2{N}_{\mathrm{params}}\right)$ by Woodbury matrix identity:

$$\mathit{\pi }={\left({\mathbf{U}}^T\mathbf{U}+\lambda \mathbf{I}\right)}^{-1}{\mathbf{U}}^T\mathit{\zeta }$$ 16

$$=\frac{1}{\lambda }\left[\mathbf{I}-{\mathbf{U}}^T{\left(\lambda \mathbf{I}+\mathbf{U}{\mathbf{U}}^T\right)}^{-1}\mathbf{U}\right]{\mathbf{U}}^T\mathit{\zeta }$$ 17

$$=\frac{1}{\lambda }\left[{\mathbf{U}}^T-{\mathbf{U}}^T{\left(\lambda \mathbf{I}+\mathbf{U}{\mathbf{U}}^T\right)}^{-1}\left(\mathbf{U}{\mathbf{U}}^T+\lambda \mathbf{I}-\lambda \mathbf{I}\right)\right]\mathit{\zeta }$$ 18

$$={\mathbf{U}}^T{\left(\lambda \mathbf{I}+\mathbf{U}{\mathbf{U}}^T\right)}^{-1}\mathit{\zeta }.$$ 19

Finally, we can write down the parameters update formula as:

$$\mathrm{d}\mathit{\theta }=\mathrm{diag}{\left({\mathbf{v}}_{k-1}\right)}^{-1/2}{\tilde{\mathbf{O}}}^T{\left(\tilde{\mathbf{O}}\mathrm{diag}{\left({\mathbf{v}}_{k-1}\right)}^{-1/2}{\tilde{\mathbf{O}}}^T+\lambda \mathbf{I}\right)}^{-1}\left(\widetilde{\mathit{ϵ}}-\tilde{\mathbf{O}}{\mathit{\varphi }}_{k-1}\right)+{\mathit{\varphi }}_{k-1}.$$ 20

Author contributions

L.W. and D.L. conceived this project. Y.G. designed the algorithm, wrote the code and performed the calculation with helps from W.L., H.L., R.L and Y.H, all of whom contributed to preliminary explorations. Y.G. and W.L. visualized the data. B.Z., Y.W., M.Q. and T.X. provided valuable insight in the physics of the Hubbard model. Y.W. wrote the PEPS code for the benchmark with $t^{\prime }$ hopping. D.H. provided insights into neural network model design. All authors contributed to the analysis of the data, the discussion of the results and the writing of the manuscript.

Peer review

Peer review information

Nature Communications thanks the anonymous reviewers for their contribution to the peer review of this work. A peer review file is available.

Data availability

We have not analysed any datasets as our work proceeds within a theoretical and mathematical approach. The figures and tables shown contain all the available data, which was generated using the provided code.

Code availability

The initial version of the code for producing the results in this work is available on Code Ocean: https://codeocean.com/capsule/8740417/tree/v1. The full code is available at https://github.com/bytedance/laqx.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41467-026-74028-6.