Long-range magnetic order with disordered spin orientations in a high-entropy antiferromagnet

Abstract

Disorder in magnetic systems typically suppresses long-range order, promoting short-range states such as spin glasses and magnetic clusters. This is particularly prominent in high-entropy materials, characterized by the random distributions of local magnetic entities and exchange interactions. However, in rare exceptions, long-range magnetic order can persist in high-entropy systems, while the microscopic characters and underlying mechanisms remain elusive, especially the magnetic behaviors of individual elements. Here, combining neutron diffraction and resonant soft x-ray scattering, we have conducted an element-specific investigation into the magnetic order of a high-entropy honeycomb-lattice van der Waals material (Mn_1/4Fe_1/4Co_1/4Ni_1/4)PS₃. Despite significant atomic disorder, long-range zigzag antiferromagnetic order is observed below 72 K, with all four transition-metal elements participating in a unified phase transition. However, the spin orientations of various elements are distinct, attributed to the competition between single-ion anisotropies and exchange interactions. Our findings showcase a novel form of long-range magnetic order with disordered spin orientations, which is synergically stabilized by distinct magnetic elements in a high entropy magnet, offering a new paradigm for understanding complex magnetic systems.

While atomic disorder normally promotes glassy states, in some rare cases long-range magnetic order can persist. However, the mechanism, and microscopic characters, remain elusive. Here, Shen et al combine neutron diffraction and resonant X-ray scattering to shed light on the formation of long-range magnetic order in a high-entropy honeycomb van der Waals magnet.

Introduction

Conventionally, magnetism is characterized by order parameters indicative of symmetry breaking and features a repeating pattern of magnetic moments. Disorder can be introduced through chemical substitution, which is commonly treated as local perturbations when its effects are weak. However, when disorder becomes significant—such as when dopant concentrations approach those of the host material—the periodic spin arrangement can be entirely disrupted, leading to dramatically weakened magnetic correlations. In classical magnetic systems, such disordered states with short-range order are generally understood within the framework of spin glass theory^1,2.

Such is the case for the magnetism in high-entropy (HE) materials—such as alloys^3–7, ceramics⁸, oxides^9,10, and van der Waals compounds^11–13. These materials feature multiple principal elements randomly distributed at comparable concentrations, yielding exceptionally high configurational entropy. They can exhibit highly tunable properties, novel topological electronic structures¹⁴, and outstanding performance in applications^15–17. Regarding magnetic properties, the substantial disorder arising from randomly distributed magnetic moments and exchange interactions is expected to suppress long-range magnetic order. Indeed, glass-like magnetic behaviors are commonly observed in HE magnets^18,19, where spin moments are frozen in random directions, pinned by local lattice distortions. Yet, in some rare cases, long-range magnetic order can persist^20–23, presenting a fundamental paradox given the inherent disorder of the system, and the nature of such ordering remains unresolved. For instance, it is often presumed that once magnetic order is established in HE magnets, it uniformly affects all constituent elements with identical transition temperatures and spin orientations, even though each element, with its distinct orbital filling, could contribute its unique magnetic characteristics such as spin magnitudes and single-ion anisotropies. In fact, the magnetic behaviors of individual magnetic elements have never been thoroughly explored in HE materials. The lack of HE single crystals poses an additional barrier for detailed studies of their magnetic properties.

In this regard, HEPS₃, short for (Mn_1/4Fe_1/4Co_1/4Ni_1/4)PS₃, emerges as a pivotal candidate to address this gap¹¹. As a newly discovered HE van der Waals material, it features a slightly distorted two-dimensional (2D) honeycomb lattice with the monoclinic space group C2/m and lattice constants of a = 5.9321 Å, b = 10.2737 Å, c = 6.7061 Å, α = γ = 90° and β ≈ 107° (Supplementary Note 1), belonging to the TMPX₃ family (TM: transition metal, X: S/Se)^24–30. For the non-HE individual TMPX₃ members, as shown in Fig. 1a–d, MnPS₃ exhibits Néel-type magnetic order with a propagation vector k = (0, 0, 0), while FePS₃, CoPS₃, and NiPS₃ display zigzag antiferromagnetic (AFM) order with a propagation vector k = (0, 1, L) where L varies with the specific TM element (Table 1). The magnetic moments in these materials are confined to the ac^* plane (Fig. 1e), and their precise orientations are also element-dependent (Table 1). Here, a, b, and c represent the real-space vectors, while a^* b^*, and c^* denote reciprocal vectors. Note that c^* is orthogonal to a, and b^* is parallel to b. For HEPS₃, which incorporates four different TM elements in equal concentrations, long-range magnetic order is hinted by magnetic susceptibility measurements at low temperatures¹¹, presenting a unique instance of magnetic order within a highly disordered system. Furthermore, the availability of HEPS₃ single crystals enables detailed exploration of its magnetic properties.

Fig. 1. Magnetic properties of TMPX₃ and HEPS₃.

a–d Schematics of the magnetic structure of MnPS₃, FePS₃, CoPS₃, and NiPS₃, respectively. All spin moments lie within the ac^* plane. e We define Θ as the angle between the spin moment M and lattice a direction. f Atomic-resolved elemental scanning transmission electron microscopy (STEM) mapping of HEPS₃, demonstrating the random distributions of various elements. g Magnetic susceptibility of HEPS₃ and other derivatives of TMPX₃ measured in the zero-field-cooling (ZFC) and field-cooling (FC) modes, respectively. For each material, the susceptibility exhibits an anomaly at T₁, and the ZFC and FC profiles split below T₂. h Schematic of the magnetic structure of HEPS₃ determined from neutron diffraction and RSXS measurements. i, Schematic diagram showing the evolution of spin orientations with different exchange interaction (J) to single-ion anisotropy (Δ) ratio. When J is negligible, the spin moments tend to preserve their anisotropy as in the non-HE materials, e.g., Mn²⁺/Fe²⁺ and Co²⁺/Ni²⁺ being almost perpendicular and parallel to the ab plane, respectively (left panel). In another aspect, dominant J would enforce a universal parallel alignment (right panel). A compromised case is realized in HEPS₃ (middle panel, see also Supplementary Note 6).

Table 1.

Magnetic order properties of the Mn, Fe, Co, and Ni ions in HEPS₃ and non-HE individual TMPX₃ compounds

Element	Mn	Fe	Co	Ni
Θ (deg) in TMPX3	82	90	10	0
M (μB) in TMPX3	4.43	4.52	3.36	1.05
k in TMPX3	(0, 0, 0)	(0, 1, 0.5)	(0, 1, 0)	(0, 1, 0)
TN (K) in TMPX3	78	118	120	155
	Ref. 41	Ref. 31	Ref. 42	Ref. 43
Θ (deg) in HEPS3	35	72	32	50
M (μB) in HEPS3	1.51 (1)
k in HEPS3	(0, 1, 0)
TN (K) in HEPS3	72

Here, Θ is the angle between the magnetic moment and lattice a direction (Fig. 1e), M is the moment magnitude determined by neutron diffraction, and k is the propagation vector.

Here, combining neutron diffraction and resonant soft x-ray scattering (RSXS) techniques, we have comprehensively investigated the magnetic ordering in HEPS₃ single crystals. Neutron diffraction is a powerful tool for detecting various forms of magnetic order, including long-range and short-range types, with greater access to reciprocal space. In contrast, resonant soft x-ray scattering (RSXS), despite its limited reciprocal space coverage, achieves element-specific resolution by tuning the incident photon energy to the resonant edge of the target element. Our neutron diffraction experiments reveal the presence of three-dimensional (3D) long-range magnetic order, with a Néel temperature of T_N  ≈ 72 K, coexisting with 2D magnetic signals. RSXS measurements further confirm that all four TM elements participate in the magnetic ordering through a unified phase transition. The resulting magnetic structure features element-dependent spin orientations, attributed to the competition between single-ion anisotropies and exchange interactions.

Results

Sample characterization

First, we verify the random distribution of TM elements in HEPS₃ through atomic-resolved elemental mapping using scanning transmission electron microscopy (STEM) on an exfoliated HEPS₃ thin flake. The sample, with a thickness of dozens of unit cells, enables the STEM mapping to be interpreted as spatial averages across multiple stacked honeycomb layers. As shown in Fig. 1f, the images reveal a homogeneous intensity distribution for each element, indicating that all elements are randomly distributed throughout the lattice. While a high-entropy scenario is well established, an anomaly indicative of magnetic phase transition is observed in the magnetic susceptibility measurements of HEPS₃ at T_N  = T₁ ≈ 72 K, followed by another transition taking place at T₂ ≈ 40 K, below which the zero-field-cooling (ZFC) and field-cooling (FC) profiles split (Fig. 1g). This second transition likely originates from glassy behaviors or spin reorientations (Supplementary Note 2). Note that T_N observed in HEPS₃ is very close to that of non-HE MnPS₃ (T_N  ≈ 78 K, see Table 1). Above 80 K, the susceptibility follows Curie-Weiss behavior with a Curie-Weiss temperature of -43 K and an effective moment of 3.03 μ_B/site, corresponding to a saturated moment of 2.19 μ_B/site assuming a g-factor of 2 (Supplementary Note 2). Intriguingly, in binary or ternary TMPX₃ compounds, where disorder is expected to be less significant than in HEPS₃, the anomaly related to T₁ is more rounded, suggesting reduced correlation lengths (Fig. 1g). This observation underscores the importance of high configurational entropy in stabilizing the long-range magnetic order in HEPS₃.

Neutron diffraction

To elucidate the origin of the magnetic susceptibility anomaly at T₁, we conducted neutron diffraction measurements on HEPS₃ single crystals. At 6 K, additional Bragg peaks can be observed in the diffraction patterns alongside structural nuclear peaks that are also available at 250 K, demonstrating the emergence of magnetic order (Fig. 2a–d). These magnetic reflections can be described by a single propagation vector k = (0, 1, 0), with three equivalent domains arising from the honeycomb lattice within each van der Waals layer³¹. Detailed examination of the H and K dependence reveals that the in-plane magnetic peak widths are comparable to those of the nuclear reflections, confirming the establishment of long-range order (Fig. 2e, f). In contrast, along the L direction, in addition to the sharp peaks corresponding to 3D magnetic order, diffuse scattering is also observed (Fig. 2d, g), suggesting the simultaneous presence of 2D magnetic order. Note that the in-plane k vector is identical for both 3D and 2D magnetic signals, implying that the 2D signals maintain the same in-plane spin configuration and correlation characteristics as the 3D ones. The primary distinction between the 2D and 3D magnetic order is the reduced inter-layer correlations. For the 3D magnetic order, adjacent layers are strongly coupled with parallel alignment of magnetic moments between neighboring planes. In contrast, the 2D magnetic order exhibits reduced or essentially decoupled layers, where the spin arrangement in one layer weakly affects neighboring layers. While stacking faults may play a role here, they cannot completely explain the observed 2D signals (Supplementary Note 3).

Fig. 2. Neutron diffraction measurements on HEPS₃ single crystals.

a, b Neutron diffraction patterns in the HK0 plane collected at 250 K and 6 K, respectively, with L integrated in the range of [-0.2, 0.2]. c, d Neutron diffraction patterns in the 0KL plane. The black circles highlight the nuclear peaks. e, f One-dimensional Q cuts along H and K directions, respectively, across the nuclear [Q = (0, 2, 0)] and magnetic reflections [Q = (0, 1, 0)] at the indicated temperatures. The solid lines are fits with Gaussian profiles, yielding FWHM (in unit of r.l.u.) of 0.0539(6) (nuclear) and 0.0496(5) (magnetic) along H, and 0.070(1) (nuclear) and 0.068(1) (magnetic) along K. g Q cuts along L direction across the magnetic signals. The shaded area denotes the 3D magnetic signals, which exhibit different peak widths due to different contributions of sample mosaicity. The 250 K data are treated as a background reference. Note that the intensities are presented on a log scale. h Temperature dependence of the integrated intensities of 3D magnetic signals at Q = (0, 1, 1) and 2D magnetic signals at Q = (0.5, 0.5, 0), which corresponds to Q = (0, − 1, 1/6) in another domain³¹. The solid line is a fit to $I-\mathrm{BG}\propto {\left(T_{\mathrm{N}}-T\right)}^{2\beta }$ (BG: background), and the dashed line is a guide for the eye. i Schematic of the spin structure derived from the neutron diffraction refinements with Θ = 49^∘, which can be regarded as an average result over all the TM elements. All the data were collected at CORELLI, including the temperature dependence. r.l.u., reciprocal lattice units.

Regarding the temperature dependence, the intensity of the 3D magnetic signals decreases with increasing temperature, and a phase transition takes place at T_N  ≈ 72 K (Fig. 2h), consistent with the susceptibility measurements. The temperature dependence was fitted using $I-\mathrm{BG}\propto {\left(T_{\mathrm{N}}-T\right)}^{2\beta }$ (BG: background), yielding a critical exponent β = 0.40(5). Meanwhile, the 2D magnetic signals remain detectable above T_N, persisting weakly at 90 K (Fig. 2g) before vanishing around that temperature (Fig. 2h). Through a global magnetic refinement of the 3D signals (Supplementary Note 4), we determined the preliminary spin structure of HEPS₃, which exhibits zigzag magnetic order with Θ = 49°, where Θ denotes the angle between spin moment and lattice a direction (Fig. 2i). The refined moment magnitude is 1.51(1) μ_B/site, much smaller than the average value of the non-HE individual counterparts (Table 1). The reduced moment is partially attributed to the presence of 2D magnetic signals, which account for a substantial portion of the spectral weight. To make a quantitative estimation of the total static moment, we integrated the intensity of Q = (0, 1, 0) over L between -0.5 and 0.5, capturing both the 2D and 3D contributions (Fig. 2g). We find that the 3D signal spectral weight, obtained by fitting with a Gaussian profile, accounts for 67% of the total spectral weight. Since neutron diffraction intensity is proportional to the squared static moment, we estimate the total static moment to be around 1.85 μ_B/site, slightly smaller than that derived from the magnetic susceptibility (2.19 μ_B/site). The discrepancy may arise from fluctuating moments or element-specific spin orientations, which will be discussed subsequently. Overall, these analyses suggest that the long-range order is mostly homogeneous across the sample.

Resonant soft X-ray scattering

While the long-range magnetic order in HEPS₃ is confirmed by neutron diffraction, the microscopic properties, particularly the behaviors of individual TM elements, remain unclear. Note that neutron diffraction averages contributions from disordered elements, masking element-specific magnetic properties. To overcome this limitation, we employed RSXS measurements at the TM L edges, during which process 2p core electrons are resonantly excited to 3d valence orbitals and then de-excite³². By tuning incident photon energy to the resonance edge of the target element, we can selectively probe the magnetic signals of a specific element in the samples. This is evident in the x-ray absorption spectrum (XAS) data, which also reveals that all TM elements in HEPS₃ adopt a divalent state (Supplementary Fig. 11). Moreover, as the HEPS₃ sample is very thin with an ab cleavage surface, 3D magnetic signals at Q = (0, 1, 0) were inaccessible. However, we successfully resolved 2D magnetic signals at Q = (0, 1, L) with finite L (Fig. 3a).

Fig. 3. Resonant soft x-ray scattering (RSXS) measurements on HEPS₃.

a Schematic diagram of the RSXS experiment setup. Note that θ is the sample angle and ψ is the azimuthal angle. Negative K is defined for grazing-incidence conditions. The polarization of the outgoing photons is not distinguished. b–e Background subtracted RSXS scans across the 2D magnetic signals at the indicated temperatures for each TM element. The solid curves are fits using pseudo-Voigt profiles. f Temperature dependence of the fitted peak heights of different elements, indicating a unified magnetic transition. The neutron 2D signals are the same as in Fig. 2h. All error bars represent 1 standard deviation. g–j Background subtracted fix-Q energy dependence of the 2D magnetic signals for each TM element in HEPS₃ with different incident photon polarizations. Data presented in (b–f) were collected at REIXS beamline with π polarization, and data in g–j were collected at 13-3 beamline at 12 K with ψ ≈ 0^∘.

Figure 3b–e depict the RSXS scans across the 2D magnetic signals for each TM element in HEPS₃. These scans reveal well-defined peak-like structures centered around K = 1 at low temperatures, which disappear as the temperature increases, consistent with the magnetic phase transition. The magnetic signals of Ni exhibit sharper peak profiles (Fig. 3e, see also Supplementary Fig. 10). However, it should be noted that Q resolution varies for different elements due to the various incident energy, penetration depth, and beam footprint. A detailed investigation of the temperature-dependent RSXS signals shows a unified magnetic phase transition, whose behaviors are in good agreement with the neutron diffraction measurements on the 2D signals (Fig. 3f). This suggests that all TM elements in HEPS₃ undergo a simultaneous magnetic phase transition at the same temperature. Note that the amplitudes of the magnetic signals are affected by multiple factors, including the azimuthal angle, incident photon energy, number of electrons in the valence band, and self-absorption effects, making direct intensity comparisons between elements unreliable. While XAS data exhibit negligible linear dichroism for all four TM elements (Supplementary Fig. 11), their magnetic signals show distinct polarization-dependent energy profiles (Fig. 3g–j). Given the strong RSXS cross-section dependence on spin orientation, these distinct behaviors imply different spin orientations for each element.

To verify these observations and quantitatively determine the spin orientations of each TM element in HEPS₃, we examined the azimuthal dependence of the 2D magnetic signals by rotating the sample about the Q vector (Fig. 3a). Upon rotation, the cross-sectional profile of the magnetic RSXS signals changes for both polarization channels^33,34. As shown in Fig. 4, each TM element displays unique azimuthal intensity modulations, directly indicating distinct spin orientations even without theoretical modeling. Although the azimuthal scan range is restricted by hardware limitations, we can model the intensity modulation by assuming that the spin moments rotate within the ac^* plane. This assumption is validated by the observed spin structures in the non-HE TMPX₃ counterparts and the aforementioned neutron diffraction measurements on HEPS₃. By simultaneously fitting the azimuthal dependence in both polarization channels (Methods), we derived the best fit for each TM element, the results of which are presented in Fig. 4. The resulting inclined angles, Θ, vary among elements and deviate from those observed in the non-HE TMPX₃ materials (Table 1). Notably, the average Θ across the four TM elements (47°) closely matches the neutron refinement result (49°), highlighting the consistency between neutron and RSXS measurements. Furthermore, the element-specific spin orientation may contribute to the reduced net spin moment derived from neutron diffraction refinement, an analysis that is based on an average spin structure.

Fig. 4. Azimuthal dependence of the 2D magnetic signals in HEPS₃ for each element.

a–d ψ = 0 corresponds to the position with Q = (0, 1, 0) lying in the scattering plane. All the data were collected at 13-3 beamline at 12 K. Note that we use a different data processing protocol here so that the intensities are not directly comparable to Fig. 3g–j (Methods).

Discussion

Combining neutron diffraction and RSXS results, we establish a comprehensive picture of the magnetic order in HEPS₃, as schematically illustrated in Fig. 1h. On a large scale, long-range zigzag AFM order is stabilized at low temperatures, resembling FePS₃, CoPS₃, and NiPS₃, but contrasting with MnPS₃, which exhibits Néel-type magnetic order. Given that the exchange interactions of MnPS₃ are the weakest among the four non-HE individual TMPX₃ compounds^28,35–40, it is very likely that the Néel-type magnetic order is disrupted and the Mn atoms in HEPS₃ are compelled to adopt a zigzag spin arrangement due to stronger exchange interactions with other transition metals.

At the local level, each TM element in HEPS₃ adopts a preferential spin orientation governed by single-ion anisotropy, which is shaped by the ionic environment, electronic hoppings, and potentially spin-orbit coupling. Based on the magnetic structure and spin dynamics of TMPX₃ probed by neutron scattering, it is concluded that Fe²⁺ and Co²⁺ ions exhibit robust easy-axis and easy-plane anisotropy, respectively, while Mn²⁺ and Ni²⁺ ions are mostly isotropic^{28,31,36,37,39,41–43}. This is due to their different d-orbital occupations. For Mn²⁺ and Ni²⁺ ions, which have half-filled 3d and e_g shells, respectively, orbital degrees of freedom are mostly quenched, leading to negligible anisotropy, whereas Fe²⁺ and Co²⁺ ions tend to exhibit robust but distinct anisotropy due to combined spin-orbit coupling and trigonal distortion (Supplementary Note 5).

Exchange interactions, however, promote parallel spin alignment to minimize the total energy, creating a strong competition with single-ion anisotropy. This interplay generates a novel frustration mechanism, distinct from the more commonly discussed geometric frustration⁴⁴. If exchange interactions were negligible, various elements would retain their intrinsic anisotropy, whereas dominant exchange interactions would impose a uniform spin orientation (Fig. 1i). In HEPS₃, an intermediate scenario prevails, giving rise to a synergic magnetic order where each TM element adopts a spin orientation that reflects a compromise between local anisotropy and exchange interactions, deviating from the ones observed in the non-HE TMPX₃ analogs (Table 1). This competition can be illustrated by a toy model, which demonstrates how spin orientations are tuned by the anisotropy-to-interaction ratio (Supplementary Note 6). Note that the single-ion anisotropy is least profound for Mn²⁺ and Ni²⁺, making their spin orientations less resistant to competition with exchange interactions.

The observed spin structure represents an exotic, new type of magnetic ordering that is synergically established by different TM elements in a highly disordered system. It features a long-range coherent magnetic pattern, where spin moments from various TM elements are arranged in a zigzag AFM fashion and collectively participate in a unified magnetic phase transition. While the order parameter might have a form deviated from conventional ones, it remains detectable through neutron diffraction and RSXS measurements. It would also be insightful to see how it responds to local probes such as nuclear magnetic resonance (NMR) or muon spin rotation/relaxation/resonance (μSR). Nevertheless, due to the random distribution of different elements, local properties — such as spin size, exchange interactions, and spin orientations — vary from site to site without a repeating pattern. Consequently, explicit periodicity is absent, precluding the definition of a conventional magnetic unit cell. Notably, RSXS can still resolve the element-specific magnetic signatures. This is supported by a toy model simulation, in which a quarter of the sites are randomly selected and occupied, and the magnetic peak at Q = (0, 1) is preserved, but with significantly reduced amplitude (Supplementary Note 7).

In conclusion, combining neutron diffraction and RSXS measurements, we have identified long-range magnetic order in a HE antiferromagnet HEPS₃, despite significant atomic disorder. The magnetic order is synergically established by four different magnetic elements—each with distinct magnetic characteristics and randomly distributed across the lattice—via a unified phase transition. This observation showcases a novel type of synergic magnetic order, devoid of periodic spin patterns, which challenges conventional understanding of magnetism and opens new avenues for engineering materials with tailored magnetic properties. Our findings highlight the complexity of HE magnets and the importance of element-specific magnetic behavior analysis in future studies. However, several questions remain unresolved. For instance, the microscopic mechanism underlying the long-range order in such a highly disordered system, as well as the reasons behind the suppression of spin glass behaviors by high entropy, remain to be fully understood. Whether this arises from enhanced quantum fluctuations, emergent collective modes, or other fundamental processes, and what role kinetic constraints or entropic barriers play in stabilizing or destabilizing the spin glass phases, are questions that warrant further exploration. These open challenges highlight the need for advanced theoretical and experimental efforts to unravel the physics of this exotic magnetic state.

Methods

Sample synthesis and characterization

Single-crystalline (Mn_1/4Fe_1/4Co_1/4Ni_1/4)PS₃ (HEPS₃) samples, as well as the binary and ternary TMPX₃ compounds, were grown using the chemical vapor transport (CVT) method as detailed in the previous report¹¹. Atomic-resolved EDS mapping was performed using an aberration-corrected STEM (JEM NEOARM), operated at 200 kV. As mentioned in the main text, the STEM mapping is interpreted as an average result of multiple stacked honeycomb layers, leading to a triangular arrangement considering ABC stacking. Consequently, elements that are randomly distributed throughout the lattice will display uniform intensity at each site.

Neutron diffraction experiment

All neutron diffraction measurements were conducted using the CORELLI instrument at Spallation Neutron Source (SNS), Oak Ridge National Laboratory (ORNL)^45,46. One piece of single crystal (27 mg in weight; 6 × 3 mm in size) was aligned first in the H0L scattering plane and subsequently in the 0KL scattering plane, utilizing monoclinic notation. Diffraction patterns were collected at 6 K, 90 K, and 250 K by rotating the sample with a fixed tilting angle. For the temperature dependence measurements, the sample position and orientations were unchanged; data were acquired with a 5 min counting time per temperature point with a step of 2 K.

RSXS experiments

The RSXS experiments were performed at the REIXS (10ID-2) beamline of the Canadian Light Source (CLS)⁴⁷ and 13-3 beamline of the Stanford Synchrotron Radiation Lightsource (SSRL) at the SLAC National Accelerator Laboratory⁴⁸. Both have an in-vacuum 4-circle diffractometer to access the magnetic signals.

For the REIXS experiment, XAS data were collected using a silicon drift detector (SDD) at a fixed position, while RSXS signals were measured with another SDD equipped at the goniometer arm at the peak position of the L₃ edge for each transition-metal element in HEPS₃ (Supplementary Fig. 11). RSXS scans were conducted by rotating the sample at a fixed scattering angle 2θ, considering that the signal is independent of L within a small range. The x-ray energies were chosen to be 639.3 eV, 706.9 eV, 777.7 eV, and 851.9 eV for the Mn, Fe, Co, and Ni L₃ edges, respectively. The intensity was normalized to the incoming x-ray flux. Background subtraction was performed using high-temperature data for all RSXS scans (Supplementary Fig. 12).

In the 13-3 experiment, a 2D CCD detector was employed, covering a finite range of Q. The collected signals were normalized by electron current. In Fig. 4, the data were further normalized by fluorescence to account for variations in measurement efficiency, particularly due to self-absorption effects and changes in the x-ray footprint on the sample, which is not relevant in the energy dependence presented in Fig. 3g–j. Furthermore, for the energy-dependent measurements, all diffraction angles, including the scattering angle 2θ, were fixed. Although Q varies slightly with energy, the large area of the CCD detector ensured that magnetic signals were still captured, enabling equivalent fix-Q energy scans. The azimuthal-dependent measurements shown in Fig. 4 were conducted at the peak position of the L₃ edge (Supplementary Fig. 13). To be more specific, the x-ray energies are 638.4 eV, 707 eV, 779 eV, and 853.9 eV for the Mn, Fe, Co, and Ni L₃ edges, respectively. For each azimuthal angle, a RSXS scan was performed by rotating the sample. The magnetic signals in the HK plane were fitted with a 2D Lorentzian function to determine the peak height. The signals along the L direction were averaged as they exhibited no significant dependence on L within the small L range investigated.

Simulation of the azimuthal dependence

Considering that the linear dichroism is negligible in HEPS₃, we use a simplified model based on spherical symmetry to simulate the azimuthal dependence of the magnetic signals:

$$\mathcal{F}=\kappa \cdot \mathit{ϵ}\times {\mathit{ϵ}}^{\prime }\cdot \mathbf{M}$$ 1

where κ is a scale factor, ϵ and ${\mathit{ϵ}}^{\prime }$ are the incident and emission photon polarization, respectively, and M is the spin moment. The magnetic RSXS cross-section can be obtained as $\mathcal{I}={\mathcal{F}}^2$. M is confined within the ac^* plane and the fitting results are presented in Fig. 4. Note that for each element, a unified scale factor κ is used for different polarizations. The outgoing photon polarization is not distinguished. Thus, the simulated signals are summed over two polarization channels, ${\sigma }^{\prime }$ and ${\pi }^{\prime }$, for the outgoing photons. Note that magnetic signals will contribute to the σ-${\pi }^{\prime }$, π-${\pi }^{\prime }$, and π-${\sigma }^{\prime }$ channels but not σ-${\sigma }^{\prime }$ channel. The full-range calculation results for the azimuthal dependence are presented in Supplementary Fig. 14, which are simulated based on calculated motor positions, while for Fig. 4, experimental motor positions are used. The discrepancy is negligible.

Author contributions

Y.S., J.W., and T.Y. conceived the project. Q.Z., Y.Z., and T.Y. synthesized and characterized the samples. X.G., F.Y., and J.W. performed the neutron measurements. Y.S., G.Z., H.L., C.T.K., J.S.L., and R.S. performed the X-ray measurements. Y.S., Z.P., X.Q., J.W., T.Y., and Y.L. interpreted the data. The paper was written by Y.S., J.W., and T.Y. with input from all co-authors.

Peer review

Peer review information

Nature Communications thanks the anonymous, reviewers for their contribution to the peer review of this work. A peer review file is available.

Data availability

All data that support the findings of this study have been deposited in the Zenodo database with the access code 17636716⁴⁹.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41467-026-70184-x.