Multi-Q spin-valley order in twisted WSe₂

Abstract

We report on a study of the interacting phase diagram of 3.65°-twisted WSe₂ at moiré hole filling ν = 1, in which we find previously-overlooked types of magnetism. Specifically, in part of the phase diagram we obtain a magnetic order parameter which modulates in space with four different non-zero wave vectors, corresponding to the three M-points and one K-point of the moiré Brillouin zone. These multi-Q orders, which can be coplanar or non-coplanar, are continuous deformations of the 120° spin-valley anti-ferromagnet (AFM), where the unit cell has expanded by a factor of four. Interestingly, we find that the multi-Q states are stabilized for experimentally relevant values of interaction strength and displacement field, and are accompanied by a softening of the spin fluctuations near the M-points of the moiré Brillouin zone.

Introduction

Moiré superlattice systems have opened up a new pathway for the study of strongly-correlated electron systems in two dimensions. An interesting subclass of these moiré materials consists of twisted transition metal dichalcogenides (TMD’s). Twisted TMD’s combine strong spin-orbit coupling with non-trivial band topology, and allow for a high tunability of moiré band structures by applying an out-of-plane displacement field, making them an ideal platform for studying strongly-correlated quantum matter.

In this work we numerically study the interacting phase diagram of twisted bilayer WSe₂ (tWSe₂) at twist angle 3.65° and moiré hole filling ν = 1, where recent experiments have found superconductivity at small displacement fields¹. Superconductivity has subsequently also been observed in larger twist-angle devices at hole fillings slightly shifted away from ν = 1, and at higher displacement fields^2,3. The phase diagram of tWSe₂ in the vicinity of the superconducting dome shares many similarities with that of high-temperature cuprate superconductors³, which has spurred a variety of theoretical works^4–22.

Below we uncover a new type of magnetic order in the zero-temperature phase diagram of tWSe₂, characterized by intricate spin textures that modulate in space with four different wave vectors. These multi-Q magnetic orders arise in a region of the phase diagram close to where superconductivity is observed, and form out of the 120° anti-ferromagnet (AFM) via a continuous phase transition. The approach to this phase transition is accompanied by a gradual softening of the spin fluctuations near the M-points of the moiré Brillouin zone, which we expect to play an important role for superconductivity. By varying the interaction strength and displacement field we find two distinct multi-Q orders, with one being coplanar and the other one non-coplanar. The system remains incompressible at the transition between the 120° AFM and the non-coplanar order, implying that the latter has zero Chern number. We check the robustness of our results by using two different tWSe₂ continuum model, which we now introduce.

Results

Continuum models

The generic continuum model for a twisted homo-bilayer TMD in valley + K reads

$$H^{+K}\left(\mathbf{r}\right)=\left(\begin{array}{cc}{T}_1+{\mathrm{\Delta }}_1\left(\mathbf{r}\right)& {\mathrm{\Delta }}_T\left(\mathbf{r}\right)\\ {\mathrm{\Delta }}_T^{†}\left(\mathbf{r}\right)& T_2+{\mathrm{\Delta }}_2\left(\mathbf{r}\right)\end{array}\right),$$ 1

where the matrix structure is in layer space. The Hamiltonian for the − K valley is related by time-reversal symmetry, $H^{-K}\left(\mathbf{r}\right)=\mathcal{T}{H}^{+K}\left(\mathbf{r}\right){\mathcal{T}}^{-1}$. Due to the large Ising spin-orbit coupling there is spin-valley locking²³, and hence we will use spin and valley interchangeably in this work. Single-particle intervalley couplings are neglected due to the large separation between the moiré and atomic length scales²⁴.

The kinetic energy T_l, the potentials Δ_l(r) and the inter-layer tunneling amplitude Δ_T(r) are constrained by the symmetries of the system, which include a three-fold rotation ${\mathcal{C}}_{3z}$, and the ${\mathcal{C}}_{2y}\mathcal{T}$ symmetry. Importantly, the anti-unitary symmetry ${\mathcal{C}}_{2y}\mathcal{T}$ does not preclude a non-zero (valley) Chern number, as the action of the ${\mathcal{C}}_{2y}$ is improper, so that the valley-resolved Chern number is even under ${\mathcal{C}}_{2y}\mathcal{T}$. Indeed, the quantum anomalous Hall responses have already been explored when the model in (1) was first introduced in ref. ²⁴, and have now been observed in different experimental setups^25–31.

To ensure that our results hold with sufficient generality, we use two different continuum models. The first continuum model uses a lowest-order approximation for T_l, Δ_l(r) and Δ_T(r), as in ref. ²⁴. This Lowest-Order (LO) model contains only three parameters, for which we use the values of ref. ³². One caveat in using these parameters, however, is that they were extracted from DFT simulations on a 5° twist-angle structure, whereas here we aim to study a smaller twist angle θ = 3.65°.

The second continuum model we use is based on ref. ³³, where the authors took both higher moiré -harmonics and higher-order gradient terms into account, and parameters were fitted to relaxed DFT band structures obtained at θ = 3.48°. The original Higher-Order (HO) continuum model of ref. ³³ contains 77 parameters. In this work, however, we use an approximation which keeps only 16 parameters, corresponding to the first moiré harmonics (but including higher-order gradient terms). The detailed expressions for T_l, Δ_l(r) and Δ_T(r) in the HO model can be found in the Supplementary Section 1. There we also show that the 16-parameter model is able to reproduce the bands from the original 77-parameter model to very high accuracy.

Interacting phase diagram

We make the model interacting by adding a density-density interaction with a gate-screened Coulomb potential. Concretely, in momentum space the interaction potential is written as $V\left(\mathbf{q}\right)=\frac{e^2}{2{ϵ}_0ϵ}\tanh \left(qD\right)/q$, where D is the sample-gate distance, and ϵ is the relative dielectric constant determined by the dielectric environment of the sample. We fix D = 50a₀ ≈ 16.5nm, with a₀ the WSe₂ lattice constant, and take ϵ to be a tuning parameter. As the WSe₂ continuum model is obtained by fitting to DFT bands, it already incorporates some effects of the interaction. Therefore, to reduce the double counting of interaction effects, we add quadratic ‘subtraction terms’ to the Hamiltonian which ensure that the continuum model bands constitute an exact solution of the Hartree-Fock (HF) self-consistency equations. For our numerical simulations we keep 4 active bands per valley. The remaining bands are assumed to be completely occupied. A detailed derivation of the complete Hamiltonian is given in the Supplementary Section 2. The numerical method used in this work is reviewed in ref. ³⁴, to which we refer the reader for further technical details about the mean-field simulations.

In experiment, the tWSe₂ sample is encapsulated in a dual-gated device, which can be used to apply an electric displacement field E perpendicular to the WSe₂ layers. We incorporate the displacement field in the continuum model by adding a potential energy difference between the top and bottom layers given by $u_D=Ed\frac{{ϵ}_{hBN}}{{ϵ}_{TMD}}$, where d ≈ 0.7 nm is the distance between the layers, ϵ_hBN ≈ 3 and ϵ_TMD ≈ 8¹. Our main focus is the phase diagram at hole doping ν = 1, i.e. one hole per moiré unit cell, as a function of both E and ϵ. In the main text we show the results for the HO model, and comment on the quantitative differences that occur in the LO model. The detailed numerical results for the LO model can be found in the Supplementary Section 4.

From previous works^{16,18,19,22,35}, we know that part of the phase diagram hosts a 120° anti-ferromagnet (AFM). On the moiré scale, the AFM modulates with a wave vector κ₊ or κ₋, with κ_± the K-points of the moiré - or mini-Brillouin zone (mBZ) (see Supplementary Section 1 for definition). Exactly which K-point determines the spin modulation depends on the orientation of the displacement field (at E = 0 the wave vector is determined via a spontaneous breaking of the ${\mathcal{C}}_{2y}$ symmetry). The 120° AFM is invariant under $T_{{\mathbf{a}}_i}^{\prime }=e^{-\frac{i}{2}{\mathit{\kappa }}_{\pm }\cdot {\mathbf{a}}_i{\sigma }^z}{T}_{{\mathbf{a}}_i}$, where a_i are moiré lattice vectors, and $T_{{\mathbf{a}}_i}$ is the translation operator: $T_{{\mathbf{a}}_i}^{-1}{\psi }^{†}\left(\mathbf{r}\right)T_{{\mathbf{a}}_i}={\psi }^{†}(\mathbf{r}+{\mathbf{a}}_i)$. So $T_{{\mathbf{a}}_i}^{\prime }$ is a combination of a translation and a spin rotation along the z-axis. To exploit this symmetry in our numerical simulations we work in the Bloch basis associated with $T_{{\mathbf{a}}_i}^{\prime }$ (i.e. the single-particle states are labeled with $\tilde{\mathbf{k}}$, which refers to the $T_{{\mathbf{a}}_i}^{\prime }$ eigenvalues $e^{i\tilde{\mathbf{k}}\cdot {\mathbf{a}}_i}$). Here we have chosen a convention such that high-symmetry points corresponding to the moiré Brillouin zone are denoted with small symbols (γ, κ, m), while the mono-layer high-symmetry point are denoted with capital symbols (Γ, K, M).

The 120° AFM is also invariant under ${\mathcal{T}}^{\prime }={\sigma }^z\mathcal{T}={\sigma }^x\mathcal{K}$, where $\mathcal{K}:i\to -i$ represents complex conjugation. This time-reversal symmetry is a combination of the standard Kramers time-reversal symmetry $\mathcal{T}=i{\sigma }^y\mathcal{K}$ and a π spin rotation along the z-axis. The ${\mathcal{T}}^{\prime }$ symmetry leaves the in-plane XY components of the spin invariant, but flips the out-of-plane Z component. The AFM is thus compatible with the ${\mathcal{T}}^{\prime }$ symmetry because it is coplanar.

From the numerical solution of the HF self-consistency equations we obtain

$${\left[P(\tilde{\mathbf{k}},{\tilde{\mathbf{k}}}^{\prime })\right]}_{\alpha \sigma ;{\alpha }^{\prime }{\sigma }^{\prime }}:=\langle {\psi }_{{\alpha }^{\prime }{\sigma }^{\prime }}^{†}\left({\tilde{\mathbf{k}}}^{\prime }\right){\psi }_{\alpha \sigma }\left(\tilde{\mathbf{k}}\right)\rangle ,$$ 2

where ${\psi }_{\alpha \sigma }^{†}\left(\tilde{\mathbf{k}}\right)$ creates an electron with spin σ in band α of the WSe₂ continuum model. From P, we define an order parameter for Inter-Valley Coherence (IVC), i.e. U(1) spin or valley symmetry breaking, as $O_{IVC}:=N_{UC}^{-1}{\sum }_{\tilde{\mathbf{k}},{\tilde{\mathbf{k}}}^{\prime }}{\sum }_{\alpha ,{\alpha }^{\prime }}{\mid {\left[P\left(\tilde{\mathbf{k}},{\tilde{\mathbf{k}}}^{\prime }\right)\right]}_{\alpha \uparrow ;{\alpha }^{\prime }\downarrow }\mid }^2$, with N_UC the number of moiré unit cells. Similarly, we define an order parameter $O_{T^{\prime }}:=N_{UC}^{-1}{\sum }_{\tilde{\mathbf{k}},{\tilde{\mathbf{k}}}^{\prime }}(1-{\delta }_{\tilde{\mathbf{k}},{\tilde{\mathbf{k}}}^{\prime }})\mid \mid P(\tilde{\mathbf{k}},{\tilde{\mathbf{k}}}^{\prime })\mid {\mid }^2$ for $T_{{\mathbf{a}}_i}^{\prime }$ translation-symmetry breaking, and $O_{{TR}^{\prime }}:=N_{UC}^{-1}{\sum }_{\tilde{\mathbf{k}},{\tilde{\mathbf{k}}}^{\prime }}\mid \mid P(\tilde{\mathbf{k}},{\tilde{\mathbf{k}}}^{\prime })-{\sigma }^x{P}^{*}(-\tilde{\mathbf{k}},-{\tilde{\mathbf{k}}}^{\prime }){\sigma }^x\mid {\mid }^2$ for ${\mathcal{T}}^{\prime }$ time-reversal symmetry breaking (∣∣ ⋅ ∣∣ is the Frobenius norm).

Figure 1a-d shows the interacting phase diagram in terms of these order parameters, as well as the mean-field bandgap and Goldstone mode dispersion. All results were obtained from numerical solutions of the HF self-consistency equations for a system with 24 × 24 moiré unit cells.

Fig. 1. Numerical results for HO model obtained on a 24 × 24 moiré lattice.

a Spin-valley U(1) order parameter O_{IV C}, b ${\mathcal{T}}^{\prime }$ time-reversal order parameter $O_{{TR}^{\prime }}$, c order parameter $O_{T^{\prime }}$ for the generalized translation symmetry $T_{{\mathbf{a}}_i}^{\prime }$. d Mean-field bandgap. TDHF Goldstone mode dispersion relation ω_q along q = (q_x, q_y = 0), both for fixed E as a function of ϵ (e), and for fixed ϵ as a function of E (f).

For small ϵ (ϵ ≲ 24), we obtain the spin-Valley Polarized (VP) state and the 120° AFM reported in previous works^{16,18,19,22,35}. The VP state is diagnosed by $O_{T{R}^{\prime }}\ne 0$ and $O_{IVC}=O_{T^{\prime }}=0$, whereas the 120° AFM has O_{IV C} ≠ 0 and $O_{{TR}^{\prime }}=O_{T^{\prime }}=0$. The VP and AFM states are separated by a first-order transition at ϵ ~ 9, in agreement with aforementioned previous studies.

For larger values of ϵ we find IVC states, i.e. states with non-zero O_{IV C}, which differ from the 120° AFM because they break the generalized translation symmetry $T_{{\mathbf{a}}_i}^{\prime }$ ($O_{T^{\prime }}\ne 0$). These IVC states are the multi-Q orders which, as we show below, host spin textures that indeed modulate in space with four different wave vectors. In Fig. 1d we see that in most of the phase diagram the multi-Q orders are incompressible, albeit with a smaller charge gap than the 120° AFM. When the system becomes metallic at the largest values of ϵ used in this work (e.g. ϵ = 41 at E = 0), the IVC and $T_{{\mathbf{a}}_i}^{\prime }$ order parameters quickly drop to zero, after which a symmetric metal is formed.

The transition from the 120° AFM to the multi-Q order is continuous, irrespective of whether the transition is driven by changing ϵ or E. A first indication of this can be seen by looking at the mean-field bandgap, which does not close and evolves smoothly across the transition. In Fig. 1e-f we also show the energy ω_q of the 120° AFM Goldstone mode along (q_x, q_y = 0), obtained via the RPA Bethe-Salpeter equation (or equivalently via the Time-Dependent HF (TDHF) equation), as reviewed in ref. ³⁶. Here q is defined as the momentum of the collective mode excitation relative to the ground state, which itself modulates with wave vector κ, such that a gapless spin-wave excitation at momentum q = 0 shows up in the spin-susceptibility as a peak at q = κ. Figure 1e shows ω_q at E = 0 as a function of ϵ. When the transition to the multi-Q order at ϵ ~ 24 is approached, the Goldstone mode at the m-points of the mBZ continuously lowers in energy or ‘softens’, until the gap closes at the transition (the plot shows only one m-point, but ${\mathcal{C}}_{3z}$ symmetry is preserved, and identical behaviour is therefore found at all three m-points of the mBz). In Fig. 1f a similar softening of the Goldstone mode energy near the m-points can be seen to occur as a function of E, working at fixed ϵ = 24. As suggested by this condensation of the Goldstone mode, we indeed find that the multi-Q orders are characterized by four different wave vectors: the κ_± wave vector of the original 120° state, and the three m wave vectors.

From Figs. 1b, c we see that there are two distinct multi-Q magnetic orders: there is a ${\mathcal{T}}^{\prime }$-breaking variant with non-coplanar spin order, and a ${\mathcal{T}}^{\prime }$-symmetric variant with coplanar order. The non-coplanar multi-Q order appears at smaller values of ϵ. The transition from non-coplanar to coplanar order occurs at ϵ = 33 in the absence of a displacement field. For non-zero E, the transition shifts to higher ϵ.

For the HO model used in the main text, the non-coplanar order is only found in a relatively small region of the phase diagram, corresponding to 24 ≲ ϵ ≲ 33 and E ≲ 10 mV/nm. In contrast, for the LO model discussed in the Supplementary Section 4, the non-coplanar order is the dominant multi-Q state, and the coplanar order is only found in a small region of the phase diagram at high ϵ. But otherwise both phase diagrams are very similar: they host the same broken-symmetry orders, and nearly identical multi-Q states are found in both the HO and LO model.

Spin textures for the non-coplanar multi-Q state at two representative values of ϵ are shown in Fig. 2. We plot $\mid M_{xy}^l\left(\mathbf{r}\right)\mid$, θ_l(r) and $M_z^l\left(\mathbf{r}\right)$, which are defined as

$$\begin{array}{cc}\langle {\psi }_{l,\uparrow }^{†}\left(\mathbf{r}\right){\psi }_{l,\downarrow }\left(\mathbf{r}\right)\rangle \hfill & =M_x^l\left(\mathbf{r}\right)+i{M}_y^l\left(\mathbf{r}\right)\hfill \\ \hfill & =:e^{-2i{\mathbf{K}}_l\cdot \mathbf{r}}\mid M_{xy}^l\left(\mathbf{r}\right)\mid e^{i{\theta }_l\left(\mathbf{r}\right)}\hfill \end{array}$$ 3

$$\langle {\psi }_l^{†}\left(\mathbf{r}\right){\sigma }^z{\psi }_l\left(\mathbf{r}\right)\rangle =M_z^l\left(\mathbf{r}\right)$$ 4

Here, l = t/b is the layer index, and K_t/b = R(± θ/2)K are the K-points of the mono-layer Brillouin zones in the top and bottom layers (which are rotated over respectively + θ/2 and − θ/2). The colors in Fig. 2 represent $\mid M_{xy}^l\left(\mathbf{r}\right)\mid$ and $M_z^l\left(\mathbf{r}\right)$. The in-plane direction of the magnetic order, i.e. θ_l(r), is represented by black arrows.

Fig. 2. Spin textures for non-coplanar multi-Q order.

a-d Non-coplanar multi-Q spin texture in the top and bottom layers at ϵ = 25. a In-plane magnetic order in the top layer, described by $\mid M_{xy}^t\left(\mathbf{r}\right)\mid$ (color) and θ_t(r) (arrows). b $M_z^t\left(\mathbf{r}\right)$, the out-of-plane component of magnetic order in the top layer. c In-plane magnetic order in the bottom layer, described by $\mid M_{xy}^b\left(\mathbf{r}\right))\mid$ (color) and θ_b(r) (arrows). d $M_z^b\left(\mathbf{r}\right)$, the out-of-plane component of magnetic order in bottom layer. e-h Same as (a-d), but for ϵ = 32.

For ϵ = 25 (Fig. 2a-d), which is close to the onset of the non-coplanar order, the in-plane spin order for the top layer closely resembles that of the 120° state, while for the bottom layer a nearly constant θ_b(r) is found. For ϵ = 32 (Fig. 2e-h), we see a clear doubling of the periodicity along all moiré lattice directions, leading to a unit cell that is four times larger than the unit cell of the 120° state. Figure 2e shows that the 120° in-plane order on the top layer becomes significantly weaker in 1/4 of the moiré unit cells, but survives on a Kagome lattice made out of the remaining 3/4 of moiré unit cells. Concomitantly, out-of-plane order ($M_z^l\left(\mathbf{r}\right)\ne 0$) develops on the 2 × 2 triangular lattice consisting of the moiré unit cells with negligible in-plane order, as shown in Fig. 2f and h for both top and bottom layer.

The spin texture for the coplanar multi-Q state at ϵ = 40 is shown in Fig. 3. The in-plane spin order again forms a 120° state, but on a triangular lattice with a periodicity of two moiré lattice constants. More surprising is that the spin order on the bottom layer, as shown in Fig. 3b, is no longer uniform, but consists of isolated patches which have a different spin orientation than their environment.

Fig. 3. Spin-texture for coplanar multi-Q state.

In-plane spin texture for the coplanar multi-Q state at ϵ = 40 in (a) top layer and (b) bottom layer. Color represents $\mid M_{xy}^l\left(\mathbf{r}\right)\mid$, and arrows represent θ_l(r).

Next to the magnetic multi-Q modulations, we also check if the multi-Q order has additional charge density modulations. To this end we compute the layer resolved charge density

$$\langle {\psi }_l^{†}\left(\mathbf{r}\right){\psi }_l\left(\mathbf{r}\right)\rangle =n_l\left(\mathbf{r}\right),$$ 5

and define the charge density modulations arising due to the multi-Q order as

$$\delta n_l\left(\mathbf{r}\right)=n_l\left(\mathbf{r}\right)-n_l^{AFM}\left(\mathbf{r}\right)$$ 6

Here, $n_l^{AFM}\left(\mathbf{r}\right)$ is the local charge density of the 120° AFM, which is shown in Fig. 4a and b for respectively the top and bottom layer. In Fig. 4c and d we plot δn_l(r) obtained in the (non-coplanar) multi-Q state. Because the shape and magnitude of the charge density modulations does not change noticeably between the different multi-Q states, we only show them for the non-coplanar phase. Note that δn_l(r) is very small. This can be understood from the fact that the non-coplanar multi-Q order forms out of the 120° state, which is incompressible.

Fig. 4. Charge density modulations induced by multi-Q state.

Charge density of the 120° AFM phase for (a) top and (b) bottom layer at ϵ = 16. Additional charge density modulations induced by the (non-coplanar) multi-Q state for (c) top and (d) bottom layer at ϵ = 32.

Discussion

We have shown that at moderately large values of the relative dielectric constant, and for small displacement fields, the 120° AFM in tWSe₂ at ν = 1 hole doping becomes unstable towards the formation of a multi-Q magnetic order with enlarged spatial unit cell. Reference ¹⁹ has estimated the appropriate value for the dielectric constant to be ϵ ~ 30, by comparing the layer-polarized regime in Hartree-Fock with the layer-polarized regime observed in experiment. For this value of ϵ we find a multi-Q spin order in the HO model. For both the HO and LO model, ϵ = 30 is close to the critical value at which the 120° state becomes unstable to the multi-Q order.

Our TDHF simulations show that the transition from the 120° state to the multi-Q order is continuous, and happens via a gradual softening of the spin fluctuations at the m-points of the mini-Brillouin zone. A recent experiment³⁷ has observed two types of neutral collective modes in tWSe₂: one fast mode, corresponding to the spin or valley Goldstone mode, and one slow mode. The slow mode was interpreted as a gapped amplitude mode³⁷. Our results provide an alternative interpretation, however: the slow mode likely corresponds to the soft spin fluctuations near the m-points of the mBz. In our simulations, the gap of the m-point spin fluctuations is less than 1 meV, and thus an order of magnitude smaller than the gap of the amplitude mode, which is comparable to the mean-field bandgap ~ 5 − 10 meV.

Interestingly, the multi-Q orders appear around zero displacement field, which is where superconductivity is observed in experiment^1,3. So even if no static multi-Q order develops in the tWSe₂ samples, the softening of the spin fluctuations near the m-points could still play an important role for Cooper pairing. One interesting scenario is that the multi-Q order is preempted by the formation of superconductivity, which is stabilized by the soft spin fluctuations near the m-points of the mBz. We will investigate this scenario in future work.

Methods

HF phase diagram

We investigate the interacting phase diagram of twisted bilayer WSe₂ at hole filling ν = 1 and twist angle θ = 3.65°. To this end we consider two generic continuum models for a twisted homo-bilayer TMD of the form presented in (1), which we make interacting by adding a density-density interaction with a gate-screened Coulomb potential. Details about both models can be found in Supplementary Section 1. We map out the phase diagram by varying two parameters: out-of-plane displacement field E and dielectric constant ϵ, which we treat as an effective parameter to incorporate the effects of screening. The density-density interaction gets treated using HF self-consistency equations.

Collective modes within RPA

Collective mode spectra are calculated via the RPA Bethe-Salpeter equation (or equivalently via the TDHF equation), from which we identify a continuous phase transition from the 120° AFM order to two different multi-Q orders, indicated by the continuous softening of the m-point fluctuations. A detailed review about the numerical methods used in this work can be found in ref. ³⁴. Results for the LO model and tests for robustness of the results are presented in the Supplementary Section 4, 5 and 6.

Author contributions

All authors conceived the project. A.B. and N.C.H. performed the numerical simulations. Interpretation of the data was done by all authors. The manuscript was written by N.B. and A.B., with input from N.C.H.

Data availability

The data that supports the findings presented in this manuscript are available via Zenodo through the following URL: 10.5281/zenodo.17647828.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41535-026-00884-6.