Hyperbolic Neural Population Geometry Benefits Computation

Abstract

Neural population geometry shapes downstream computation. Recent empirical findings in neurobiology suggest that a hyperbolic structure underlies population activity in the hippocampus. Here we provide a theoretical framework for this phenomenon. First, we propose a plausible construction of hippocampal tuning curves that statistically induces hyperbolic geometry. Next, we establish a connection between neural decoding and associative memory by demonstrating that the Modern Hopfield Network update rule computes the minimum mean-squared-error (MMSE) estimator. Finally, we introduce a novel associative memory model defined in hyperbolic space that yields significantly larger capacity than leading models. Our results suggest that animals encode spatial information as a latent hyperbolic cognitive map, improving both memory capacity and decoding accuracy.

1. Introduction

A central goal in understanding the brain is to see how neural activity patterns relate to animal behavior (Kriegeskorte & Wei, 2021). In recent years, advances in large-scale neural recordings have shifted the focus from individual neurons to the collective representations formed by large neural populations. Consequently, there is growing interest in characterizing the neural population geometry induced by these activities (De Kamps et al., 2019; Kriegeskorte & Wei, 2021). This geometric perspective may reveal how animals store and process information (Gallego et al., 2017; Kriegeskorte & Diedrichsen, 2019; Chung & Abbott, 2021; Mathis et al., 2024).

Similarly, in machine learning (ML), by understanding the latent representation of artificial neural networks (ANNs), we gain insight into how these models learn and store information (Bengio et al., 2013). Recent studies demonstrate how one can borrow insights from population geometry in neural systems to improve ML models (Chung & Abbott, 2021; Chou et al., 2025). A growing number of recent studies suggest that hyperbolic geometry emerges in a range of biological systems (Zhang et al., 2022; Zhou et al., 2018; Lee et al., 2024; Ghaninia et al., 2022). However, these studies remain largely empirical. There has yet to be a theoretical framework that: (i) explains how hyperbolic geometry is induced by neural populations, (ii) characterizes its impact on downstream decoding, and (iii) provides design principles for machine learning models.

For (i), we develop a plausible population coding model of the hippocampus that induces hyperbolic geometry. Specifically, we show that when the widths of Gaussian place fields follow an exponential distribution, the induced semi-metric space is tree-like and statistically hyperbolic in Gromov’s sense (Gromov, 1987). Interestingly, this exponential distribution of place field widths matches the experimental observations in Zhang et al. (2023); Rich et al. (2014).

For (ii), we establish a formal link between neural decoding and associative memory. We show that the recall dynamics of modern Hopfield networks (MHNs) approximate the optimal Minimum Mean Square Error (MMSE) estimator. Building on this insight, together with the hyperbolic geometry induced by the neural encoder, we propose a new associative memory model that operates directly in hyperbolic space. Theoretically, we show that this model achieves substantial capacity improvements over previous models (Ramsauer et al., 2020; Krotov & Hopfield, 2021).

Regarding (iii), we demonstrate the utility of this framework for both pattern completion and general machine learning tasks. We show that our hyperbolic associative memory model outperforms existing memory models with superior accuracy on pattern completion. Inspired by the connection between MHNs (Ramsauer et al., 2020) and the attention mechanism (Vaswani et al., 2017), we introduce a hyperbolic memory module that integrates seamlessly into ML architectures. Our simulations demonstrate that this model provides substantial performance gains in ML tasks. Notably, these improvements are most pronounced when the hidden dimensionality is constrained, suggesting that hyperbolic geometry offers a more efficient representation space for information storage in low dimensions.

Organization.

The remainder of this paper is organized as follows. Section 2 introduces the encoding model and derives the Bayes-optimal decoder, which motivates our later connection between neural decoding and associative memory. Section 3 collects the geometric tools we need. The heart of the paper is Section 4, where we show that exponentially-distributed place-field widths induce a statistically hyperbolic semi-metric (Theorem 4.2). Then we build a hyperbolic associative memory model with double-exponential capacity (Theorem 4.8). Section 5 validates these theoretical results through simulations on pattern completion and downstream ML tasks. Section 6 summarizes our findings and implications for computational neuroscience and machine learning. Our limitations can be found in Section A. The overall logic is visualized in Figure 1. The table of notations is in Table 2.

Figure 1.

(a) Illustration of the stimulus space $𝒮$ and place cell firing patterns. (b) Illustration of how neurons encode stimulus $s$ through tuning curves $\lambda$ and output $n\left(s\right)$. (c) Our constructed tuning curve model that induces hyperbolic geometry. (d) Illustration of how the MMSE estimator (decoder) is realized by the memory retrieval dynamics in latent hyperbolic space.

2. Neural Computing

This section introduces a neural encoding model based on tuning curves and Poisson spiking, and derives the corresponding Bayes-optimal decoder. We model spatial coding in the hippocampus and denote the space of possible locations by $𝒮$. A single location is denoted by $s\in 𝒮$. Neural population activity is represented by a vector $n\in {\mathbb{R}}^N$. The index $i\in \left[N\right]$ refers to neurons. We use $\mu \in \left[M\right]$ to index latent states or discretized stimulus locations.

2.1. Tuning Curve

Neural population codes are commonly modeled using tuning curves (Dayan & Abbott, 2005). Tuning curves are functions describing the firing rate for each neuron when given a stimulus $s$. Each neuron responds preferentially to a subset of the stimulus space. Let $s\in 𝒮$ denote the physical location of the animal. We model the firing rate of a neuron $i$ in the hippocampus by a tuning curve

$${\lambda }_i\left(s\right)=\sum _{k=1}^K{\lambda }_{ik}\cdot \mathrm{e}\mathrm{x}\mathrm{p}\left(-\frac{{‖s-s_{i,k}‖}_2^2}{2{\sigma }_{ik}^2}\right),$$ (2.1)

where $s_{ik}$ denotes the location of the $k$-th place field of neuron $i,{\sigma }_{ik}$ is its width, and ${\lambda }_{ik}$ is its scale. Neural spiking is modeled as a Poisson process. Given a time window of duration $T$, the spike count $n_i$ for neuron $i$ is modeled as:

$$n_i\mid s\sim \mathrm{P}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{o}\mathrm{n}\left({\lambda }_i\left(s\right)T\right).$$ (2.2)

The population response is denoted by $n=\left(n_1,\cdots ,n_N\right)$.

We will henceforth assume that $K=1$ and ${\lambda }_i={\lambda }_{\text{max}}$, in which case Equation (2.1) corresponds to the case of uniform spatial tiling through single place fields, as is typical when animals explore a simple environment (Fenton et al., 2008). Future work is needed to explore the multi-field case typical of large environments (Rich et al., 2014). Statistically, one can consider this model as an encoder (generative model) that encodes the input stimulus $s$ into a high-dimensional noisy vector $n$, or into $\lambda \left(s\right)=\left({\lambda }_1\left(s\right),\cdots ,{\lambda }_N\left(s\right)\right)$ without noise. In the next subsection, we review standard statistical approaches to decoding under the Poisson tuning-curve model above, including MLE, MAP, and Bayes-optimal estimators such as the MMSE rule. We then relate these estimators to the update rule of modern Hopfield networks.

2.2. From Bayes-Optimal Decoding to Recall

We first explain the standard decoding method for neural tuning curves. Next, we establish an equivalence between the recall dynamics of associative memory models and decoding. Specifically, the intractability of Bayes-optimal decoding naturally motivates a tractable approximation in the form of memory recall. This connection allows us to interpret memory retrieval as a decoding process.

Statistical Decoders.

From a statistical point of view, neurons encode the stimulus $s$ into a noisy high-dimensional code $n$. Decoding can be formulated as inferring a latent stimulus $s$ from neuronal population activity $n$ under a prescribed loss function. In computational neuroscience (Dayan & Abbott, 2005; Pillow et al., 2005), this process is typically formulated as maximum likelihood estimation (MLE) or maximum a posteriori (MAP) estimation,

$$s_{\mathrm{M}\mathrm{L}\mathrm{E}}^{\mathrm{*}}\left(n\right)=\underset{s}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}}\mathrm{l}\mathrm{o}\mathrm{g}p(n\mid s).$$ (2.3)

$$s_{\mathrm{M}\mathrm{A}\mathrm{P}}^{\mathrm{*}}\left(n\right)=\underset{s}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}}\mathrm{l}\mathrm{o}\mathrm{g}p(s\mid n).$$ (2.4)

Another common choice is to minimize the expected squared loss, $\ell \left(\hat{s}\right)={\mathbb{E}}_{p(s\mid n)}\left[\Vert s-\hat{s}\left(n\right){\Vert }_2^2\right]$. Under this loss, the Bayes-optimal estimator is given by the posterior mean (Hastie, 2009),

$$s_{\mathrm{M}\mathrm{M}\mathrm{S}\mathrm{E}}^{\mathrm{*}}\left(n\right)={\int }_{𝒮}p\left(s∣n\right)s\mathrm{d}s,$$ (2.5)

commonly referred to as the Minimum Mean Squared Error (MMSE) estimator.

Although $s_{\text{MAP}}^{\mathrm{*}}\left(n\right)$ and $s_{\text{MMSE}}^{\mathrm{*}}\left(n\right)$ are not equal in general, the Bernstein‒von Mises theorem (Ghosh & Ramamoorthi, 2003) shows that, as the population size $N$ becomes large, the MAP estimate concentrates around the MLE, which is asymptotically normally distributed. Consequently, under regularity conditions, the posterior mean $s_{\text{MMSE}}^{\mathrm{*}}\left(n\right)$ converges to $s_{\text{MAP}}^{\mathrm{*}}\left(n\right)$. This asymptotic behavior motivates us to link the update rule in an associative memory model to the posterior mean.

To compute $s_{\text{MMSE}}^{\mathrm{*}}$, we discretize the stimulus space into $M$ grid points ${\left\{s_{\mu }\right\}}_{\mu =1}^M$ and assume a uniform prior, i.e., $p\left(\mu \right)=\frac{1}{M}$. Assuming conditional independence of spike counts across neurons given $s$, the posterior takes the form

$$p\left(s_{\mu }\mid n\right)=\frac{e^{h_{\mu }\left(n\right)}}{\sum _{\nu }{e}^{h_{\nu }\left(n\right)}}:={\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}}_{\mu }\left(h_1\left(n\right),\cdots ,h_M\left(n\right)\right),$$

where $h_{\mu }\left(n\right)=\sum _{i=1}^N{n}_i\mathrm{l}\mathrm{o}\mathrm{g}{\lambda }_i\left(s_{\mu }\right)$ are log-likelihood scores for each stimulus. The Bayes-optimal decoder (fully derived in Section E.9.2) is therefore

$$s^{\mathrm{*}}\left(n\right)=\sum _{\mu =1}^M{\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}}_{\mu }\left(h\right(n\left)\right)s_{\mu }.$$ (2.6)

Modern Hopfield Networks.

Modern Hopfield networks (Ramsauer et al., 2020; Krotov & Hopfield, 2021) (MHNs) are associative memory models defined over continuous state spaces. Let ${\psi }^E:{\mathbb{R}}^N\to {\mathbb{R}}^d$ be an embedding map. We define the network state as $v={\psi }^E\left(\lambda \right(s\left)\right)$ and stored memory patterns ${\xi }_{\mu }={\psi }^E\left(\lambda \left(s_{\mu }\right)\right)$. Given a state $v$, the MHN update is

$$\mathrm{M}\mathrm{H}\mathrm{N}\left(v\right):=\sum _{\mu =1}^M{\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}}_{\mu }\left(h_1^{\mathrm{M}\mathrm{H}\mathrm{N}}\left(v\right),\dots ,h_M^{\mathrm{M}\mathrm{H}\mathrm{N}}\left(v\right)\right){\xi }_{\mu }.$$ (MHN)

where $h_{\mu }^{\mathrm{M}\mathrm{H}\mathrm{N}}\left(v\right)=v^{\mathrm{\top }}{\xi }_{\mu }=⟨v,{\xi }_{\mu }⟩$. The shared form of (2.6) and (MHN) implies that MHN is a probabilistic inference machine. As this relation plays a central role in our paper, we formalize it in Section 4.

Remark 2.1.

Here we introduce a novel notation ${\psi }^E$. It denotes the non-linear map connecting the tuning curve encoder to MHN, which takes $\lambda \left(s\right)$ as input. This decoupling avoids imposing a strict biological constraint between encoder and decoder, while ensuring compatibility with the Boltzmann form assumed in Proposition 2.2. This mapping is not seen in the typical MHN literature, where existing works do not consider the case where inputs are generated by some encoder (tuning curves).

One can now see that (MHN) and (2.6) are structurally similar. To understand the condition of having MHN to compute the Bayes-optimal estimator, we present the following proposition.

Proposition 2.2.

If the posterior takes the Boltzmann form

$$p(\mu \mid v)=\frac{e^{⟨v,{\xi }_{\mu }⟩}}{\sum _{\nu }{e}^{⟨v,{\xi }_{\nu }⟩}},$$ (2.7)

then the MHN update computes the Bayes-optimal estimator of the following problem

$$\mathrm{M}\mathrm{H}\mathrm{N}\left(v\right)=\underset{z\in 𝒱}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}{\mathbb{E}}_{p(\mu \mid v)}\left[{‖{\xi }_{\mu }-z‖}_2^2\right].$$ (2.8)

The proof is in Section E.1. Proposition 2.2 establishes a connection between Bayes-optimal decoding and MHN dynamics. Specifically, it demonstrates that under the assumption in (2.7) and the MMSE loss, a single MHN update effectively computes the posterior mean estimator. Proposition 2.2 has a second consequence: associative memory can be viewed as an MMSE estimator whose loss respects the underlying geometry of $\lambda \left(s\right)$. This view allows us to derive, in Section 4, a non-Euclidean associative memory model.

3. Preliminary on Hyperbolic Geometry

Theorem 4.2 and Theorem 4.8, our main geometric results, require three notions from hyperbolic geometry: $\delta$-hyperbolic metric spaces, the hyperboloid model, and the Fréchet mean. Readers familiar with these can skim to Section 4; we collect them here for self-containedness. For a deeper treatment we refer readers to Jost (2005); Bridson & Haefliger (2013).

$\delta$-hyperbolic Metric Space.

The following notion measures the extent to which a given metric space resembles a hyperbolic space in terms of thin triangles. See visualization in the appendix Figure 3.

Definition 3.1 ($\mathit{\delta }$-thin triangles (Gromov, 1987)).

Let $(𝒳,d)$ be a geodesic metric space.¹ A geodesic triangle $\mathrm{\Delta }(x,y,z)$ is a triangle with sides

$$\left[x,y\right],\left[y,z\right],\left[z,x\right],$$

where $[x,y]$ is the geodesic between $x$ and $y$. We say $\mathrm{\Delta }(x,y,z)$ is $\delta$-thin if

$$\forall p\in \left[x,y\right],\mathrm{m}\mathrm{i}\mathrm{n}\left\{d\left(p,\left[y,z\right]\right),d\left(p,\left[z,x\right]\right)\right\}\le \delta ,$$

and cyclic permutations.

Definition 3.2 ($\mathit{\delta }$-hyperbolic (Gromov, 1987)).

A geodesic metric space ($𝒳,d$) is $\delta$-hyperbolic if every geodesic triangle in $𝒳$ is $\delta$-thin. Equivalently for all $x,y,z,w\in 𝒳$,

$$\begin{gathered} d\left(x,z\right)+d\left(y,w\right)\le \\ \mathrm{m}\mathrm{a}\mathrm{x}\left\{d\left(x,y\right)+d\left(z,w\right),d\left(y,z\right)+d\left(x,w\right)\right\}+2\delta , \end{gathered}$$ (3.1)

which is called the 4-point condition.

In general, low $\delta$ means the geodesic metric space is more tree-like, as tree triangles (Chiswell, 2001) are 0-hyperbolic. For a geodesic metric space to behave like a negatively curved space in Gromov's sense (Gromov, 1987), it requires $\delta$ to be uniform, meaning every geodesic triangle is $\delta$-thin for the same $\delta$, independent of the size of the triangle. Note that any geodesic metric space with finite domain is $\delta$-hyperbolic for some constant $\delta$. Thus, only an infinite domain metric space can fail to be $\delta$-hyperbolic. An indicator that the geodesic metric space is tree-like is that $\delta$ remains $𝒪\left(1\right)$ as the diameter of the domain $L\to \mathrm{\infty }$. Next, we introduce a relaxed condition to Definition 3.2.

Hyperboloid Model.

A $d$-dimensional hyperboloid (Lorentz) model is a Riemannian manifold (${𝓜}^d,g^d$) equipped with the Riemannian metric tensor $g^d=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}(-\mathrm{1,1},\cdots ,1)$ and defined by constant negative curvature $\kappa <0$, denoted as ${\mathbb{H}}_{\kappa }^d$.² We will drop the superscript indices on $\mathbb{H},g,𝓜$ whenever $d$ is clear from the context or irrelevant.

Each point $\mathit{x}\in {\mathbb{H}}_{\kappa }^d$ has the parameterized form $\left[{\mathit{x}}_t,{\mathit{x}}_s\right]$, where ${\mathit{x}}_t\in \mathbb{R}$ and ${\mathit{x}}_s\in {\mathbb{R}}^d.{\mathbb{H}}_{\kappa }^d$ is equipped with the Lorentz (Minkowski) inner product. For points $\mathit{x},\mathit{y}\in {\mathbb{H}}_{\kappa }^d$, their inner product $⟨\mathit{x},\mathit{y}{⟩}_L$ is given by

$$⟨\mathit{x},\mathit{y}{⟩}_L=-{\mathit{x}}_t{\mathit{y}}_t+{\mathit{x}}_s^{\mathrm{\top }}{\mathit{y}}_s={\mathit{x}}^{\mathrm{\top }}{g}_{\kappa }^d\mathit{y},$$

with $\Vert \mathit{x}{\Vert }_L:=\sqrt{\left|⟨\mathit{x},\mathit{x}{⟩}_L\right|}$ being the Lorentzian norm. Formally, ${𝓜}^d$ is the set

$${𝓜}^d=\left\{\mathit{x}\in {\mathbb{R}}^{d+1}:⟨\mathit{x},\mathit{x}{⟩}_L=\frac{1}{\kappa },{\mathit{x}}_t>0\right\}.$$

The origin $\mathit{o}\in {\mathbb{H}}_{\kappa }^d$ is the point $[\sqrt{-1/\kappa },0,\cdots ,0{]}^{\mathrm{\top }}$.

We denote the distance of any two points $\mathit{x},\mathit{y}\in 𝓜$ as $d_g(\mathit{x},\mathit{y})$. Notably, in the hyperboloid model, hyperbolic distance can be expressed in terms of the Lorentz inner product as $\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}\left(\left|\kappa {|}^{1/2}{d}_g\right(\mathit{x},\mathit{y})\right)=-\kappa ⟨\mathit{x},\mathit{y}{⟩}_L$.

Tangent Space.

The tangent space at a point $\mathit{x}\in {\mathbb{H}}_{\kappa }^d$ is the set of points orthogonal to $\mathit{x}$, defined as

$$T_{\mathit{x}}{\mathbb{H}}_{\kappa }^d=\left\{\mathit{y}\in {\mathbb{R}}^{d+1}:⟨\mathit{x},\mathit{y}{⟩}_L=0\right\},$$ (3.2)

where the tangent space is isometric to the Euclidean space (Lee, 2018). If a manifold is smooth, it admits a tangent space at every point. For any $\mathit{x},\mathit{y}\in {\mathbb{H}}_{\kappa }^d$ and $\mathrm{v}\in T_{\mathit{x}}{\mathbb{H}}_{\kappa }^d$, the exponential map ${\mathrm{E}\mathrm{x}\mathrm{p}}_{\mathit{x}}(\cdot ):T_{\mathit{x}}{\mathbb{H}}_{\kappa }^d\mapsto {\mathbb{H}}_{\kappa }^d$, and the inverse exponential map ${\mathrm{E}\mathrm{x}\mathrm{p}}_{\mathit{x}}^{-1}:{\mathbb{H}}_{\kappa }^d\mapsto T_{\mathit{x}}{\mathbb{H}}_{\kappa }^d$ are given by

$$\left\{\begin{array}{ll}{\mathrm{E}\mathrm{x}\mathrm{p}}_{\mathit{x}}\left(\mathrm{v}\right)& =\cosh \left(\sqrt{-\kappa }\Vert \mathrm{v}{\Vert }_L\right)\mathit{x}+\frac{\sinh \left(\sqrt{-\kappa }\Vert \mathrm{v}{\Vert }_L\right)}{\sqrt{-\kappa }\Vert \mathrm{v}{\Vert }_L}\mathrm{v},\\ {\mathrm{E}\mathrm{x}\mathrm{p}}_{\mathit{x}}^{-1}\left(\mathit{y}\right)& =\frac{{\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}}^{-1}\left(-\kappa ⟨\mathit{x},\mathit{y}{⟩}_L\right)}{\sqrt{{\left(-\kappa ⟨\mathit{x},\mathit{y}{⟩}_L\right)}^2-1}}\left(\mathit{y}+\kappa ⟨\mathit{x},\mathit{y}{⟩}_L\mathit{x}\right).\end{array}\right.$$

For a smooth manifold $𝓜$ and a point $p\in 𝓜$, the tangent space $T_p𝓜$ is the vector space that represents the first-order approximation of $𝓜$ at $p$, which then describes the neighborhood of $p$ to first order. $T_p𝓜$ consists of all possible velocities of smooth curves through $p$. To relate this linear structure back to the manifold, the exponential map sends each tangent vector v to the endpoint of the geodesic starting at $p$ with initial velocity $v$, evaluated at unit time. This map allows operations defined linearly on $T_p𝓜$ to be transported to $𝓜$ itself.

Fréchet mean.

Next, we define the Fréchet mean, which computes the geometric mean under general metrics.

Definition 3.3 (Fréchet mean).

Given a set of points $𝓑={\left\{{\mathit{x}}_{\mu }\right\}}_{\mu =1}^M$ on a Riemannian manifold ($𝓜,g$). The Fréchet mean of them, denoted as $\mathrm{F}\mathrm{M}\left(𝓑\right)$, is defined as the solution and optimal values of the following optimization problem (Bačák, 2014)

$$\mathrm{F}\mathrm{M}\left(𝓑\right):=\underset{\mathit{z}\in 𝓜}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}\frac{1}{M}\sum _{\mu =1}^M{d}_g^2\left({\mathit{x}}_{\mu },\mathit{z}\right).$$

Furthermore, given a set of weights $𝒲=\left\{w_1,\cdots ,w_M\right\}\in {\mathbb{R}}^{+}$, the weighted Fréchet mean $\mathrm{W}\mathrm{F}\mathrm{M}(𝓑,𝒲)$ over $𝓑$ is the solution to the following optimization problem

$$\mathrm{W}\mathrm{F}\mathrm{M}\left(𝓑,𝒲\right):=\underset{\mathit{z}\in 𝓜}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}\sum _{\mu =1}^M{w}_{\mu }\cdot d_g^2\left({\mathit{x}}_{\mu },\mathit{z}\right).$$

4. Hyperbolic Neural Population Geometry

In this section, we introduce a plausible tuning curve model that approximates hyperbolic geometry via neural population codes. Our setup is inspired by the representational similarity analysis (RSA) (Kriegeskorte et al., 2008). We look at the pair-wise distance of neural activity patterns between different stimuli, and then study the semi-metric space³ that underlies it. Next, we discuss the corresponding decoder in hyperbolic geometry and propose a plausible hyperbolic associative memory model with large capacity.

4.1. Geometry of Neural Population Activities

Recent findings in neurobiology (Zhang et al., 2023) show that in the CA1 region of the hippocampus, the place field sizes of the tuning curves ${\sigma }_{i\mu }$ are well approximated by an exponential distribution. Zhang et al. (2023) discover that this distribution is in turn well approximated by uniform sampling in a hyperbolic ball,

$$p\left(\sigma \right)\approx \zeta \cdot e^{-\zeta \sigma }\approx \zeta \frac{\sinh \left(\zeta \left({\sigma }_{\mathrm{m}\mathrm{a}\mathrm{x}}-\sigma \right)\right)}{\cosh \left(\zeta {\sigma }_{\mathrm{m}\mathrm{a}\mathrm{x}}\right)-1},$$ (4.1)

where $\sigma$ equals the distance from the origin $\mathit{o},{\sigma }_{\text{max}}\ge \sigma >0$, and $\zeta =(N-1)\sqrt{\left|\kappa \right|}$, with $\kappa <0$ and $N>1$ as the curvature and dimensionality of the space.

Biologically, this corresponds to: (a) most hippocampal place cells have small, localized place fields; and (b) a smaller number of neurons have progressively larger fields. This implies that population activity is sparse and localized, as most neurons respond only in a small region around their preferred location. In our next analysis, we will show that not only is the representation sparse, but the induced stimulus distance is also tree-like.

Stimulus Distance in Population Codes.

Here we study the geometry induced by the exponential place-field size distribution under (2.1). Define the population response vector given stimulus $s$ (assume noiseless),

$$\lambda \left(s\right)=\left({\lambda }_1\left(s\right),\cdots ,{\lambda }_N\left(s\right)\right)\in {\mathbb{R}}^N.$$

Let the stimulus space be $𝒮=[0,L{]}^D$, equipped with the Euclidean metric. Assuming a single place field with fixed amplitude, the Gaussian tuning curve of neuron $i$ is

$${\lambda }_i\left(s\right)={\lambda }_{\mathrm{m}\mathrm{a}\mathrm{x}}\exp \left(-\frac{{‖s-s_i‖}_2^2}{2{\sigma }_i^2}\right),{\sigma }_i\sim \mathrm{E}\mathrm{x}\mathrm{p}\left(\beta \right),$$

where centers ${\left\{s_i\right\}}_{i=1}^N$ are distributed according to a Poisson point process with density $\rho$ (Rich et al., 2014), and $\beta$ is the mean of the exponentially distributed place field sizes. We measure distances between population responses $\lambda \left(s\right)\in {\mathbb{R}}^N$. Assuming $\underset{i}{\mathrm{m}\mathrm{i}\mathrm{n}}{\sigma }_i>0$, we define a semi-metric between stimuli as

$$d_{ab}=-\varphi \left(⟨\lambda \left(s_a\right),\lambda \left(s_b\right)⟩\right)+C,s_a,s_b\in 𝒮,$$

where $\varphi$ is any increasing and continuous scalar function, and $C$ is an universal constant depending only on ${\lambda }_{\text{max}},N$ and the Lipschitz constant of $\varphi$. We define stimulus distances, $d_{ab}$, to be functions of firing-rate inner products to match the standard RSA assumption (Kriegeskorte et al., 2008). This makes our definition consistent with many other neuroscience studies. We mainly study the case where $\varphi =\mathrm{l}\mathrm{n}$. This is for the simplicity of the proof; one can obtain a similar result using the Lipschitz constant of $\varphi$.

Because the widths ${\sigma }_i$ are random variables, $d_{ab}$ is a semi-metric rather than a metric. Concretely, for any three stimuli $s_a,s_b,s_c$, there exist realizations of ${\left\{{\sigma }_i\right\}}_{i=1}^N$ under which $d_{ab}+d_{bc}<d_{ac}$, violating the triangle inequality. In order to gain insight from the hyperbolic structure of the neural population code, we introduce a definition that allows us to study the hyperbolicity of $d_{ab}$.

Definition 4.1 (Statistically $\mathit{\delta }$-hyperbolic).

Let $𝒮$ be a space equipped with a distance function $d$. We say $(𝒮,d)$ is statistically $\delta$-hyperbolic at confidence $1-\eta$ if there exists a constant $\delta >0$ such that

$$\mathrm{P}\mathrm{r}\left[\mathrm{\Delta }\left(s_x,s_y,s_z,s_w\right)>2\delta \right]<\eta ,$$

where $s_x,s_y,s_z,s_w$ are sampled independently and uniformly from $𝒮$, and $\mathrm{\Delta }$ denotes the empirical Gromov four-point excess in (3.1).

Unlike Definition 3.2, we relax the universal 4-point condition to a probabilistic one. Rather than requiring the 4-point condition to hold for every choice of the quadruples, we only require it to hold for a $(1-\eta )$-fraction of quadruples drawn uniformly at random. This relaxation is motivated by the case where one needs to study neural population geometry under some observed overall statistics of the neurons.

For any quadruple ($s_a,s_b,s_c,s_d$) uniformly i.i.d. sampled from $𝒮$, let $L_{\left(1\right)}\ge L_{\left(2\right)}\ge L_{\left(3\right)}$ be the sorted values of the pair-sum set $𝒫:=\left\{d_{ab}+d_{cd},d_{ac}+d_{bd},d_{ad}+d_{bc}\right\}$. For convenience, we'll write $\underset{\left(k\right)}{\mathrm{m}\mathrm{a}\mathrm{x}}A$ for the $k$-th largest member of $A$:

$$L_{\left(1\right)}=\underset{\left(1\right)}{\mathrm{m}\mathrm{a}\mathrm{x}}𝒫,L_{\left(2\right)}=\underset{\left(2\right)}{\mathrm{m}\mathrm{a}\mathrm{x}}𝒫,L_{\left(3\right)}=\underset{\left(3\right)}{\mathrm{m}\mathrm{a}\mathrm{x}}𝒫.$$

Following Definition 4.1, we define $\mathrm{\Delta }$ as the difference between the two largest sums of opposite distances $\mathrm{\Delta }(a,b,c,d):=L_{\left(1\right)}-L_{\left(2\right)}$. Recall that from Definition 4.1, a semi-metric space is said to be statistically $\delta$-hyperbolic if under uniform sampling in $𝒮,\mathrm{\Delta }\le 2\delta$ with high probability. Now we are ready to present our result on the induced inter-stimulus geometry.

Theorem 4.2.

Let $s_a,s_b,s_c,s_d\in 𝒮$ be any quadruple each independently sampled uniformly from $𝒮$. For any $\eta >0$ and $N=𝒪\left((L/\beta {)}^D\right)$, there exists a constant $\delta (\beta ,\rho )$ such that

$$Pr\left[\mathrm{\Delta }\left(s_a,s_b,s_c,s_d\right)>2\delta \right]<\eta .$$

Furthermore, $\delta$ is non-trivial since $\underset{L\to \mathrm{\infty }}{\mathrm{l}\mathrm{i}\mathrm{m}}\frac{\delta }{L}=0$.

Proof Sketch.

We compute the 4-point excess in Definition 3.2 to show that $d_{ab}$ is statistically hyperbolic. The proof is given in Section E.2, and the discussion is provided in Section E.2.1. Note that one can also obtain a similar bound by replacing ln in $d_{ab}$ with any other increasing and continuous function that is Lipschitz. We choose the specific function ln for simplicity of the proof. □

Remark 4.3.

This theorem implies with a sufficient number of neurons $N$, the distance induced by neural population activity $\lambda \left(s\right)\in {\mathbb{R}}^N$ is tree-like. This structure arises from incorporating an exponential place-field size distribution into Gaussian tuning. This corresponds to the hippocampus encoding stimulus distances using a hyperbolic semi-metric. This is analogous to neurons representing spatial information as latent hyperbolic embedding. Not only is this result inspired by experimental findings, but it also provides a realizable construction of tree-like geometry induced by neural populations. Numerical simulations validating Theorem 4.2 can be found in Section G.2 and its Figure 5.

4.2. Hyperbolic Decoder

Theorem 4.2 establishes that population activity induces a hyperbolic semi-metric on the stimuli. To mirror the construction in Section 2.2, where we moved from the MMSE estimator to an associative memory model, we commit to a Riemannian manifold that is intrinsically $\delta$-hyperbolic: the hyperboloid (Lorentz) model ${\mathbb{H}}_{\kappa }^d$. This model has two properties we rely on. First, it is intrinsically $\delta$-hyperbolic. Second, its Lorentz inner product encodes geodesic distance through a single linear operation, a property we exploit heavily in Section 4.3. Together, these properties let us define the memory model as the MMSE estimator under geodesic rather than Euclidean loss.

Given a set of preferred stimuli ${\left\{s_{\mu }\right\}}_{\mu =1}^M$, assume there exists a function ${\psi }^H:{\mathbb{R}}^N\to {\mathbb{H}}_{\kappa }^d$ such that ${\mathit{\xi }}_{\mu }={\psi }^H\left(\lambda \left(s_{\mu }\right)\right)$⁴. Under this latent model, information about $s$ may be inferred by estimating its latent representation $\mathit{\xi }$. The observed input query is $\mathit{\xi }\left(s\right)={\psi }^H\left(\lambda \right(s\left)\right)$ generated from the firing-rate vector. Decoding is formulated as a squared-loss estimation problem in $\mathbb{H}$. The expected squared geodesic loss and the optimal estimator under it are defined as following (Pennec, 2006):

$${\ell }_{\mathbb{H}}\left(\stackrel{\mathit{ˆ}}{\mathit{\xi }}\right)={\mathbb{E}}_{p\left(\mathit{\xi }∣n\right)}\left[d_g^2\left(\stackrel{\mathit{ˆ}}{\mathit{\xi }}\left(n\right),\mathit{\xi }\right)\right].$$ (4.2)

$${\mathit{\xi }}_{\mathrm{M}\mathrm{M}\mathrm{S}\mathrm{E}}^{\mathrm{*}}\left(n\right)=\underset{\mathit{z}}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}{\int }_{\mathit{\xi }\in {\mathbb{H}}_{\kappa }^d}p\left(\mathit{\xi }∣n\right)\cdot d_g^2\left(\mathit{z},\mathit{\xi }\right)d\mathit{\xi },$$ (4.3)

where (4.3) is considered as the posterior Fréchet mean.

Similar to Section 2.2, If we again discretize the domain $𝒮$ into $M$ points ${\left\{s_{\mu }\right\}}_{\mu =1}^M$, the optimal decoder under the same loss has the form:

$${\mathit{\xi }}^{\mathrm{*}}\left(n\right)=\mathrm{W}\mathrm{F}\mathrm{M}\left(\left\{{\mathit{\xi }}_{\mu }\right\},\left\{p\right(\mu \mid n\left)\right\}\right),\mu \in \left[M\right],$$ (4.4)

where the posterior over memory indices, $p(\mu \mid n)$, is discussed and defined in hyperbolic space in Section 4.3.

From Fréchet mean to Karcher flow.

The above optimal estimator is generally not available in closed form. A common way to compute (4.4) is to apply the Karcher flow algorithm (Karcher, 1977). Given some iterate ${\mathit{\xi }}^t$ at iteration $t$, it approximates (4.4) by iterating

$${\mathit{\xi }}^{t+1}={\mathrm{E}\mathrm{x}\mathrm{p}}_{{\mathit{\xi }}^t}\left(\sum _{\mu =1}^{M}p(\mu \mid n)\cdot {\mathrm{E}\mathrm{x}\mathrm{p}}_{{\mathit{\xi }}^t}^{-1}\left({\mathit{\xi }}_{\mu }\right)\right).$$ (4.5)

Here the weights are the posterior, and the average is performed in the tangent space $T_{{\mathit{\xi }}^t}\mathbb{H}$. Karcher flow not only makes the decoder tractable, but it also approximates ${\mathit{\xi }}^{\mathrm{*}}$ via iterative updates and converges to the weighted frechet mean (Karcher, 1977). As we show in the next subsection, this corresponds to the recurrent update in associative memory.

4.3. Associative Memory in Hyperbolic Geometry

Inspired by the connection between MHN and neural tuning curves, we now define the following update rule for the hyperbolic memory model.

Definition 4.4 (Karcher-flow model).

Given memory patterns ${\left\{{\mathit{\xi }}_{\mu }\right\}}_{\mu =1}^M\subseteq {\mathbb{H}}_{\kappa }^d$, and some input query $\mathit{v}\in {\mathbb{H}}_{\kappa }^d$. The Karcher-flow model update $H:{\mathbb{H}}_{\kappa }^d\to {\mathbb{H}}_{\kappa }^d$ is defined by

$$H\left(\mathit{v}\right):={\mathrm{E}\mathrm{x}\mathrm{p}}_{\mathit{v}}\left(\sum _{\mu =1}^M{w}_{\mu }\left(\mathit{v}\right)\cdot {\mathrm{E}\mathrm{x}\mathrm{p}}_{\mathit{v}}^{-1}\left({\mathit{\xi }}_{\mu }\right)\right),$$ (KFM)

where $w_{\mu }\left(\mathit{v}\right)=\frac{e^{{⟨\mathit{v},{\mathit{\xi }}_{\mu }⟩}_L}}{\sum _{\nu }{e}^{{⟨\mathit{v},{\mathit{\xi }}_{\nu }⟩}_L}}$.

Definition 4.4 has two major distinctions from the MHN. First, (KFM) is defined in ${\mathbb{H}}_{\kappa }^d$. Second, we use the Lorentz inner product instead of Euclidean inner product. This provides an advantage to KFM: the Lorentz inner product naturally encodes hyperbolic distance. In contrast, the Euclidean inner product in general does not represent Euclidean distance well. This allows our model to better distinguish patterns with similar angular directions but different norms with minimal cost, as $⟨\cdot ,\cdot {⟩}_L$ has the same complexity as $⟨\cdot ,\cdot ⟩$. Furthermore, the form of $⟨\cdot ,\cdot {⟩}_L$ computationally has the same structure as the $d_{ab}$ used in Theorem 4.2.

Next, we introduce a similar setup to that of Proposition 2.2. Let the latent index be distributed, $\mu \sim p\left(\mu \right)$, and denote the random query in hyperbolic space as $\mathit{v}$. Now we are ready to present the hyperbolic correspondence to Proposition 2.2.

Proposition 4.5.

If we model the posterior over indices by the Boltzmann distribution

$$p(\mu \mid \mathit{v})=\frac{e^{\beta {⟨\mathit{v},{\mathit{\xi }}_{\mu }⟩}_L}}{\sum _{\nu =1}^M{e}^{\beta {⟨\mathit{v},{\mathit{\xi }}_{\nu }⟩}_L}},$$ (4.6)

and

$${⟨\mathit{v},{\mathit{\xi }}_{\mu }⟩}_L=\log p\left(\mathit{v}∣\mu \right)+\log p\left(\mu \right)+C\left(\mathit{v}\right),$$ (4.7)

Then (KFM) approximates the following estimator via the Karcher flow

$$H\left(\mathit{v}\right)\approx {\xi }^{\mathrm{*}}\left(\mathit{v}\right)=\underset{z\in 𝓜}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}{\mathbb{E}}_{p\left(\mu ∣v\right)}\left[d_{𝓜}^2\left({\mathit{\xi }}_{\mu },z\right)\right].$$ (4.8)

Remark 4.6.

One can consider the above proposition as the non-Euclidean version of Proposition 2.2. However, different from Proposition 2.2, our tractable update rule (KFM) is not the exact optimal estimator. Instead, to converge to the optimal Fréchet mean estimator, one should consider fixed weights throughout the iterative updates. Given some initial state ${\mathit{v}}^{\left(0\right)}$, we set $w_{\mu }\left(\mathit{v}\right)=w_{\mu }\left({\mathit{v}}^{\left(0\right)}\right)$. This reduces the computational cost for $w_{\mu }\left(\mathit{v}\right)$. However, in our later analysis, both update rules have the same order of stable fixed points under pattern separation (Theorem 4.8). In the context of modern Hopfield networks, one can think of (4.8) as approximate minimization of the loss of the system, ${\ell }_{\mathbb{H}}\left(\stackrel{\mathit{ˆ}}{\mathit{\xi }}\right)$, via memory retrieval dynamics given by (KFM).

4.4. Pattern Completion

Here we study the problem of pattern completion under ${\mathbb{H}}_{\kappa }^d$. Informally, we show that the Karcher-flow model has capacity that scales exponentially in $d$, the dimensionality of the hyperbolic embedding, and doubly exponential in $r_{\text{max}}$, the largest norm among the memory patterns. This capacity improves the capacity of the MHN by a rate of double exponential in $r_{\text{max}}$. We identify the Euclidean space ${\mathbb{R}}^d$ with the tangent space at the origin, $T_{\mathit{o}}{\mathbb{H}}_{\kappa }^d$.

Let ${\left\{x_{\mu }\right\}}_{\mu =1}^M\subseteq T_{\mathit{o}}{\mathbb{H}}_{\kappa }^d$ be the stored patterns. We assume that the pattern embeddings satisfy ${\mathit{\xi }}_{\mu }={\mathrm{E}\mathrm{x}\mathrm{p}}_{\mathit{o}}\left(x_{\mu }\right)$, for $\mu \in \left[M\right]$. A query is generated by first sampling $\mu \sim \mathrm{U}\mathrm{n}\mathrm{i}\mathrm{f}\left(\right[M\left]\right)$. Conditioned on $\mu$, we sample the query $\mathit{v}$ via

$$\mathit{v}={\mathrm{E}\mathrm{x}\mathrm{p}}_{{\mathit{\xi }}_{\mu }}\left(\mathrm{v}\right),\mathrm{v}=x_{\mu }+\mathit{\sigma }z,z\sim 𝒩\left(0,{\mathit{I}}_d\right).$$

Given a decoder $H:{\mathbb{H}}_{\kappa }^d\to {\mathbb{H}}_{\kappa }^d$ and a target tolerance $\epsilon >0$, we aim to control the recall success probability

$$P_{\mathrm{r}\mathrm{e}\mathrm{c}}\left(\epsilon \right):=\mathrm{P}\mathrm{r}\left[d_{\mathbb{H}}^2\left(H\left(\mathit{v}\right),{\mathit{\xi }}_{\mu }\right)\le \epsilon \right],$$

where the probability is taken over $\mu$ and the query noise. Note that the above condition matches the calculations used to define the memory capacity in classic work (Hopfield, 1982; 1984).

Assumption 4.7 (Chernoff condition).

Fix distinct $\mu \ne \nu$. Under the conditional law $\mathrm{v}\mid \mu$, define

$${\mathrm{\Delta }}_{\mu \nu }^{\mathrm{E}}\left(\mathrm{v}\right):={‖\mathrm{v}-s_{\mu }‖}_2^2-{‖\mathrm{v}-s_{\nu }‖}_2^2.$$

Assume there exist constants ${\gamma }_{\mathrm{E}}>0,K_{\mathrm{E}}<\mathrm{\infty }$, and $d_0\in \mathbb{N}$ such that for all $d\ge d_0$:

• (A1) $\mathbb{E}\left[{\mathrm{\Delta }}_{\mu \nu }^{\mathrm{E}}\left(\mathrm{v}\right)\mid \mu \right]\ge {\gamma }_{\mathrm{E}}d$.

• (A2) Conditioned on $\mu ,{\mathrm{\Delta }}_{\mu \nu }^{\mathrm{E}}\left(\mathrm{v}\right)-\mathbb{E}\left[{\mathrm{\Delta }}_{\mu \nu }^{\mathrm{E}}\left(\mathrm{v}\right)\mid \mu \right]$ is sub-Gaussian with proxy variance at most $K_{\mathrm{E}}{\mathit{\sigma }}^{2}d$.

Note that both assumptions are naturally satisified when the memory patterns are uniformly distributed and noise is added as above.

Theorem 4.8.

Let $\kappa <0$ and $\alpha =\sqrt{\left|\kappa \right|}$. Under Assumption 4.7, if both

$$\mathit{\sigma }=O\left(r_{\mathrm{m}\mathrm{i}\mathrm{n}}\cdot \mathrm{m}\mathrm{i}\mathrm{n}\left\{\frac{1}{\sqrt{d}},\sqrt{\left|\kappa \right|}{e}^{-\alpha r_{\mathrm{m}\mathrm{i}\mathrm{n}}}\right\}\right)$$

and

$$r_{\text{max}}-r_{\text{min}}=o\left(\frac{d}{\alpha r_{\text{min}}^2}\right),\mathrm{l}\mathrm{o}\mathrm{g}M=\mathrm{\Theta }\left(\frac{d}{\left|\kappa \right|}\frac{e^{2\alpha r_{\text{min}}}}{r_{\text{min}}^2}\right)$$

hold. Then as $d\to \mathrm{\infty }$,

$$\underset{d\to \mathrm{\infty }}{\mathrm{l}\mathrm{i}\mathrm{m}}{P}_{rec}\left(\epsilon \right)=1.$$

The proof can be found in Section E.3. Intuitively, this says that memory retrieval is successful with high probability within a nonzero radius basin of attraction as long as the memory patterns have sufficient separation in their amplitudes. The capacity for different regimes of ${\mathrm{\Delta }}_r$ is summarized in Table 3.

Remark 4.9.

A notable difference between our assumptions and prior work (Ramsauer et al., 2020; Santos et al., 2024) is that we do not require memory patterns to be normalized. This relaxation is possible because the Lorentz inner product encodes $d_g(\cdot ,\cdot )$ in ${\mathbb{H}}^d$, rather than angular similarity. This provides significant benefits for storing memories in hyperbolic space: the Lorentz inner product has the same computational complexity as Euclidean inner products, yet captures the nonlinear quantity $e^{d_g(\mathit{a},\mathit{b})}$. Furthermore, distinguishing patterns via the Lorentz inner product rather than the Euclidean inner product allows models to separate patterns with similar directions but different magnitudes.

5. Simulations

We conduct simulations on pattern completion, image classification, and multiple instance learning. We compare our model with Dense Associative Memory (DAM) (Krotov & Hopfield, 2021) and Modern Hopfield Networks (MHN) (Ramsauer et al., 2020), two well-studied memory models for the continuous domain. For ML tasks, we compare against the Hopfield-based layers proposed in (Ramsauer et al., 2020). Source code is available on GitHub.

5.1. Pattern Completion

The task involves recalling a memory pattern from a stored memory set. We aim to reconstruct memories based on a query that is a corrupted/noisy version of the target memory. We assume that all memory patterns lie in $T_{\mathit{o}}\mathbb{H}$. We also conduct analysis where we fix $d=3$, where the state pattern is represented by only 3 neurons. We observe how different values of $r_{\text{max}}$ affect memory capacity.

Setup and Evaluation Metrics.

For memory patterns, we utilize: (1) synthetic points uniformly sampled within a tangent ball, (2) MNIST (LeCun et al., 2002), and (3) CIFAR10 (Krizhevsky et al., 2009). For the Karcher-flow model, we follow the query generation process described in Section 4 to generate the corresponding queries in $\mathbb{H}$. For each trial, we sweep through different values of $M$ (the memory set size) and perform pattern completion, treating each pattern in the set as a target in turn. A successful recall means the retrieval error is less than a tolerance 0.01 under either hyperbolic or Euclidean metric. The recall success rate is then calculated for each value of $M$.

Results.

The simulation results are presented in Figure 2. In Figure 2(a), we report results when keeping $d\in \{10,20,100\}$ PCA dimensions with $r_{\text{max}}=3$. We repeated the task across 10 different random seeds and report the mean and standard deviation. For Figure 2(b), we vary $r_{\text{max}}$ from 1 to 6. We observe that our Karcher-flow model demonstrates a high recall success rate, whereas the other two baselines fail to store even a small number of patterns in low-dimensional settings. As shown in the left panel of Figure 2(b), our model exhibits significant capacity improvement as $r_{\text{max}}$ increases. In contrast, the results in the right panel of Figure 2(b) show that MHN do not benefit much from this rescaling operation. Additional results are provided in Section G.1.

Figure 2.

(a) Left to right columns: pattern dimension $d\in \{10,20,100\}$. (a) Top to bottom rows: Recall success rate of three models on synthetic, MNIST, CIFAR10 datasets. We observe that the Karcher-flow model outperforms other models on the synthetic, MNIST, and CIFAR10 datasets with superior scaling. (b): Left to right: The recall rate of the Karcher-flow model and the MHN under different values of $r_{\text{max}}$, when $d=3$, respectively. The stored patterns are sampled from a ball of radius $r_{\text{max}}$. MHN shows marginal improvement on recall rate while our model benefits from larger $r_{\text{max}}$.

5.2. Classification

Inspired by Ramsauer et al. (2020), who parameterized MHNs and proposed various machine learning layers, we describe our proposed layers in Section F.3. Interestingly, we can construct these layers without any hyperbolic parameters. This allows our model to benefit from superior capacity scaling while still utilizing Euclidean optimizers. We also include two existing hyperbolic attention modules: (1) hyperbolic attention networks (Gulcehre et al., 2019), and (2) hyperbolic neural networks ++ (Shimizu et al., 2021). A major difference between (Shimizu et al., 2021) and other models is that it requires a Riemannian-based optimizer as the parameters are defined in the hyperbolic space.

Setup and Evaluation Metrics.

We evaluate the layers across both image classification and multiple instance learning (MIL) benchmarks. For image classification, we use MNIST (LeCun et al., 2002). For MIL we use the Elephant, Fox, and Tiger datasets. We construct our network with one embedding layer, followed by the layer of interest, and a readout layer. The pooling layers have a single learnable static query. We use the AdamW optimizer (Loshchilov & Hutter, 2019) for training, and hyperparameters are provided in Table 6. We report mean accuracy with standard deviation over 5 trials for classification, and mean ROC AUC across trials under 10-fold cross-validation for MIL.

Results.

On MNIST, Karcher flow–based models achieve performance comparable to Hopfield models under high hidden dimensions. However, Karcher flow models consistently yield higher accuracy in low-dimensional representation spaces, as shown in Table 1. Similar trends have been reported in other implementations of hyperbolic attention networks when compared against their Euclidean counterparts (Gulcehre et al., 2019; Zhang et al., 2022). On the Elephant, Fox, and Tiger MIL benchmarks, KFPooling or previous hyperbolic attention networks consistently outperformed Euclidean HopfieldPooling models in terms of ROC AUC, but other hyperbolic networks sometimes outperformed KFPooling. See MIL results in Table 1.

Table 1.

Performance comparison across MNIST and MIL benchmarks. MNIST results use KFAttention and Hopfield attention mechanisms evaluated across hidden dimensions $d$, while MIL results use the corresponding pooling variants (KFPooling and HopfieldPooling) on three animal datasets. Values are reported as mean ± standard deviation.

	MNIST (% accuracy)			MIL (AUC)
Model	$d=4$	$d=8$	$d=32$	Tiger	Fox	Elephant
KarcherFlow	85.52 ± 2.50	92.42 ± 0.64	96.89 ± 0.13	87.34 ± 1.40	66.00 ± 2.32	91.20 ± 0.82
Hopfield	83.70 ± 2.12	92.29 ± 0.63	96.71 ± 0.21	83.52 ± 1.50	60.54 ± 2.58	91.65 ± 0.42
(Gulcehre et al., 2019)	84.85 ± 1.85	91.71 ± 1.07	96.80 ± 0.18	89.20 ± 0.94	62.92 ± 2.41	93.04 ± 0.42
(Shimizu et al., 2021)	67.35 ± 5.76	84.17 ± 7.21	84.17 ± 6.12	80.32 ± 5.11	57.76 ± 4.84	85.32 ± 3.43

6. Discussion

Here we discuss three questions: (i) how hyperbolic geometry arises from neural population activity, (ii) what this geometry implies for downstream decoding, and (iii) how to transfer this insight to machine learning. Our theoretical results and the simulations of the previous sections answer these in turn, and we now discuss what each one says.

Hyperbolic Structure in Neural Population Codes.

First, our definition of $d_{ab}=-\varphi \left(⟨\lambda \left(s_a\right),\lambda \left(s_b\right)⟩\right)+C$ follows the standard RSA convention in computational neuroscience (Kriegeskorte et al., 2008). We theoretically show that the semi-metric between stimuli is statistically hyperbolic. In other words, with the number of active neurons scales linearly with the size of the environment, the semi-metric is 0-hyperbolic, which corresponds to a tree metric. This result is informative for two reasons. (1) By Kriegeskorte & Wei (2021), tuning curves define the geometry, which informs how the downstream decoding process can exploit hierarchical relationships. (2) Furthermore, this result provides a theoretical foundation for the findings in (Zhang et al., 2022), suggesting that the cognitive map may have a tree-like organization.

A Hyperbolic Associative Memory Model.

In Proposition 2.2, we link memory recall and decoding by showing that the Modern Hopfield Network (MHN) computes the optimal Minimum Mean Square Error (MMSE) estimator for decoding. Combining this connection with Theorem 4.2 allows us to study the corresponding decoder, viewed as an associative memory model under hyperbolic geometry. Theoretically, Theorem 4.8 shows that our Karcher-flow model exhibits an extra double-exponential scaling term in $r_{\text{max}}$ (the maximum norm among memory patterns). This scaling behavior is not present in existing Euclidean-based models (Ramsauer et al., 2020; Krotov & Hopfield, 2021; Krotov, 2023). Specifically, the double-exponential term arises for two reasons: (1) hyperbolic geometry exhibits exponential volume growth with respect to its radius, and (2) the Lorentz inner product naturally encodes a term approximately proportional to $e^{d_g(\mathit{a},\mathit{b})}$. Interestingly, the Lorentz inner product is a linear operation and requires the same computational complexity as the Euclidean inner product.

Simulation Results.

We validate Theorem 4.8 via pattern completion in Figure 2. For our case study in Figure 2(b), rescaling the memory patterns corresponds to rescaling the synaptic connection strength between neurons. Figure 2(b) shows that both DAM and MHN, do not benefit much from this modification. The traditional approach in AM to increase model capacity is to increase network width or depth. Notably, our model offers an additional architectural perspective, obtaining performance gains without any modification to width or depth. Furthermore, our model offers an additional perspective on how systems with a small number of neurons can store many states.

Future Directions.

This work suggests several avenues for future research. In computational neuroscience, an interesting direction is to understand how synaptic plasticity rules in the hippocampus either promote or constrain the emergence of this geometry. Additionally, our theory could be generalized to the multi-field case to determine how complex tuning curves induce hyperbolic structure. From a machine learning perspective, our proof-of-concept simulations demonstrate that these bio-inspired layers outperform their Euclidean counterparts in classification and Multiple Instance Learning (MIL) tasks, particularly in low-dimensional regimes. A promising future direction involves scaling these hyperbolic architectures to larger scales, and discovering suitable application scenarios.

Impact Statement.

From the perspective of (Bender & Hanna, 2025), the reliance on massive computational resources concentrates power within a few large companies (e.g., Google). Our method introduces an additional dimension—hyperbolic geometry—while keeping the overall model size unchanged. Despite this added complexity, the required computational resources remain comparable, yet the model demonstrates superior memory capacity. This improvement in efficiency could lower the barrier to entry, enabling smaller companies and research labs to develop competitive models, thereby helping to reduce power concentration in AI research.

From the perspective of (Crawford & Calo, 2016), current LLMs largely depend on scaling laws to increase intelligence, which leads to significant carbon emissions. While our model does not fully resolve this issue, it provides a promising, under-explored approach for energy-efficient AI. As shown in Theorem 4.8, our model achieves double-exponential scaling in memory capacity, allowing efficient performance even in strictly low-dimensional settings. This offers a theoretical foundation for designing more energy-conscious architectures. However, we acknowledge potential negative consequences. Efficiency gains in memory and representation may enable deployment of ML models in resource-constrained devices, which could facilitate surveillance or autonomous weapons systems. Furthermore, consistent with Jevons Paradox, improvements in computational efficiency may paradoxically increase overall energy consumption by lowering the barrier to large-scale deployment.