Extrinsic vs Intrinsic Criticality in Systems with Many Components

Abstract

Biological systems with many components often exhibit seemingly critical behaviors, characterized by atypically large correlated fluctuations. Yet the underlying causes remain unclear. Here we define and examine two types of criticality. Intrinsic criticality arises from interactions within the system which are fine-tuned to a critical point. Extrinsic criticality, in contrast, emerges without fine tuning when observable degrees of freedom are coupled to unobserved fluctuating variables. We unify both types of criticality using the language of learning and information theory. We show that critical correlations, intrinsic or extrinsic, lead to diverging mutual information between two halves of the system, and are a feature of learning problems, in which the unobserved fluctuations are inferred from the observable degrees of freedom. We argue that extrinsic criticality is equivalent to standard inference, whereas intrinsic criticality describes fractional learning, in which the amount to be learned depends on the system size. We show further that both types of criticality are on the same continuum, connected by a smooth crossover. In addition, we investigate the observability of Zipf’s law, a power-law rank-frequency distribution often used as an empirical signature of criticality. We find that Zipf’s law is a robust feature of extrinsic criticality but can be nontrivial to observe for some intrinsically critical systems, including critical mean-field models We further demonstrate that models with global dynamics, such as oscillatory models, can produce observable Zipf’s law without relying on either external fluctuations or fine tuning. Our findings suggest that while possible in theory, fine tuning is not the only, nor the most likely, explanation for the apparent ubiquity of criticality in biological systems with many components. Our work offers an alternative interpretation in which criticality, specifically extrinsic criticality, results from the adaptation of collective behavior to external stimuli.

Life emerges from an intricate interplay among a large number of components, yet how it achieves such exquisite organization remains unexplained. Several aspects of this question fall within the domain of statistical physics, which studies the emergence of collective behaviors from the interaction between microscopic degrees of freedom. Perhaps the greatest success of statistical physics is in describing spontaneous transitions between two phases of matter such as liquid and gas. For a class of phase transitions, such as between ferromagnetic and paramagnetic displays unique properties, absent from either phase. Many of, displays unique properties, absent from either phase. Many of these properties are relevant to biological function: scale invariance allows scaling up without the need for redesign [1–3], insensitivity to microscopic details can form a basis for robust behaviors [4, 5], and strong correlations between components appear useful for effective information propagation [6–9].

A tantalizing question arises whether biology operates near a critical point [10, 11]. This idea has a long history, see, e.g., Ref. [12]. However, it is not until recently that high-precision, simultaneous measurements of hundreds to thousands of components in biological systems allow quantitative empirical tests of the criticality hypothesis.

Modern quantitative biology experiments have indeed observed seemingly critical behaviors in many systems across scales, from amino acid sequences [13] to spatiotemporal dynamics of gene expressions [4, 14] to firing patterns of neurons [15–20] to velocity fluctuations in bird flocks [21–23]. In these systems, correlations among the components and susceptibilities to perturbations often appear to diverge with the system size. Yet, this ubiquity is somewhat surprising, not least because equilibrium statistical physics tells us that criticality requires a hard to achieve fine tuning of models to a special point in their parameter space.

While biology may well be capable of fine tuning [24, 25], alternative explanations for the observed criticality exists [26–28]. Some signatures of criticality arise without fine tuning when observable degrees of freedom are coupled to an unobserved fluctuating variable or variables [26, 27, 29, 30]. Provided that the number of fluctuating variables is relatively small and their fluctuations are sufficiently large, this latent fluctuation needs not depend on the specifics of the observable degrees of freedom such as the system size. In this case, criticality results from an extrinsic effect. Although this mechanism seems to differ from the fine-tuning explanation, they are not entirely unrelated; interacting systems at criticality also generate large fluctuations. In fact, the usual definition of criticality describes not its mechanisms, but rather the behavior of the observable degrees of freedom such as diverging correlation length, scale invariance and nonanalytic thermodynamic functions (see, e.g., Refs. [31–33]). Such properties can emerge intrinsically from interactions between components when model parameters are carefully chosen, as is often the case in statistical physics. However, this intrinsic mechanism is by no means the only one, nor is it a defining feature of criticality.

Here we introduce a new definition of criticality that spans both intrinsic and extrinsic mechanisms. Using the languages of learning and information theory, we show that a unifying feature of both types of criticality is a divergence of mutual information between two halves of the system.¹ The rates of divergence depend on how the fluctuations scale with the system size and generally differ between intrinsic and extrinsic criticality. We show further that critical systems are equivalent to the problem of learning parameters from iid samples, with the fluctuating fields playing the role of the parameters and system components of iid samples. This learning problem is characterized by diverging information between the parameters and samples. Through the learning-theoretic lens, we interpret intrinsic criticality as a fractional learning problem, in which only a fraction of parameters is available for learning due to a sharpening of a priori distribution as the system size grows. In contrast, extrinsic criticality has a fixed a priori distribution whose entropy, i.e., information available to be learned, does not shrink with the system size.

In addition, we examine the observability of Zipf’s law—the inverse relationship between the rank and frequency of system states—commonly used as an empirical signature of criticality [10, 17]. While the correspondence between Zipf’s law and critical behaviors is expected in the thermodynamic limit [10], little is known whether it remains robust under finite sample size and especially in finite systems, for which the definition of criticality becomes blurred. Our analyses reveal that the extrinsic mechanism is a likelier explanation of experimentally observed Zipfian behaviors than typical intrinsic critical systems, such as critical mean-field models, which usually generate fluctuations that are too small. We show further that, when endowed with certain dynamics, intrinsically induced fluctuations can become large enough to support empirically observable Zipf-like distributions over a range of model parameters without the need for fine tuning.

We emphasize that the distinction between both types of criticality is not sharp. Varying the system-size scaling of critical fluctuations leads to a smooth crossover between intrinsic and extrinsic criticality with the latter corresponding to the limit, in which the fluctuations are independent of the system size. Importantly, this crossover behavior means that intrinsically critical fluctuations that depend very weakly on the system size are empirically indistinguishable from extrinsic critical ones.

I. INFORMATION AND LEARNING–THEORETIC VIEW OF CRITICALITY

Large fluctuations are a defining feature of criticality.

Criticality is often characterized by nonanalytic behaviors of the derivatives of thermodynamic energy or the divergence of correlations of an order parameter [31, Ch XIV]. Such definitions assume the knowledge of the probabilistic model that governs the system, and rely on quantities that are not readily accessible in biology experiments. As a result, they are hard to extend to living systems. Alternatively, one can attempt to define criticality, directly using properties of the joint probability distribution of the system’s components which can be estimated directly from data. Zipf’s law in the rank-ordered frequencies of the system states is one possibility [10]. This definition of criticality also describes the samples that encode maximum information about the unknown generative process at a fixed level of compression [41–44]. More generally, energy is an extensive quantity and thus the central limit theorem ensures that the variance of energy fluctuations for a typical, weakly correlated statistical physics system of size $N$ scales as $\mathrm{v}\mathrm{a}\mathrm{r}\left(E\right)~N$. A faster growth, $\mathrm{v}\mathrm{a}\mathrm{r}\left(E\right)~N^{\alpha }$ with $\alpha >1$, indicates correlated microscopic fluctuations that violate the iid assumptions of the central limit theorem.² We use the presence of these atypically large fluctuations as a definition of criticality.

Divergent mutual information between subsystems is another signature of criticality.

Large correlated fluctuations allow predictions of the state of one part of the system, ${\mathbf{x}}^A=\left(x_1,x_2,\dots ,x_{N_A}\right)$, from measurements of its other part, ${\mathbf{x}}^B=\left(x_{N_A+1},x_{N_A+2},\dots ,x_{N_A+N_B}\right)$, with $\mathbf{x}=\left(x_1,x_2,\dots ,x_N\right)$ and $N_A+N_B=N$. Specifically the mutual information between these parts reads [45]

$$I\left({\mathbf{x}}^A;{\mathbf{x}}^B\right)=S\left({\mathbf{x}}^A\right)+S\left({\mathbf{x}}^B\right)-S\left(\mathbf{x}\right),$$ (1)

where $S(\cdot )$ denotes the entropy of a random variable. At a continuous phase transition, whether equilibrium or not, this mutual information diverges logarithmically, $I\left({\mathbf{x}}^A;{\mathbf{x}}^B\right)\propto {\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$ [34–40]. We take this divergence as another, equivalent definition of criticality. We will show that this definition provides a unifying way of treating extrinsic and intrinsic criticality (see Fig. 1), and to isolate the effects of critical correlations from that due to geometry, we will focus on non-spatial systems here and in the following.

FIG. 1. Divergent information signifies criticality.

A: Intrinsic criticality in equilibrium generally requires fine tuning. We depict the mutual information between two equal halves of the system, $I_{1/2}\left(\mathit{\sigma }\right)=I\left({\mathit{\sigma }}^A;{\mathit{\sigma }}^B\right)$ with $N_A=N_B=N/2$, for the fully connected Ising model as a function of temperature for a range of system sizes $N$ (see legend). We see that the information diverges with $N$ only at the critical temperature $T=T_c$. This divergence is logarithmic, with the asymptotic information given approximately by $\frac{1}{4}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$ (inset). B: Extrinsic criticality, on the other hand, emerges without fine tuning. For a system of noninteracting spins, coupled to a common Gaussian fluctuating field $\varphi$ (inset), the information $I_{1/2}\left(\mathit{\sigma }\right)$ always diverges with $N$. In contrast to the fully connected Ising model at criticality, the asymptotic information grows faster with $N,I_{1/2}\left(\mathit{\sigma }\right)\approx \frac{1}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$. C: The scaling exponent of the critical fluctuation, $\mathrm{v}\mathrm{a}\mathrm{r}\left(\varphi \right)~N^{-\gamma }$, controls the information divergence rate. We illustrate the information between two halves of the system $I_{1/2}\left(\mathit{\sigma }\right)$ (filled circles), and between the system and the latent field $I(\mathit{\sigma };\varphi )$ (empty circles) for conditionally independent identical spins, Eq. (5), under a Gaussian fluctuating field $\varphi ~𝒩\left(\mu =0,s^2=N^{-\gamma }\right)$ for various values of $\gamma$ (see legend). The asymptotic scaling of the information is in good agreement with the expected logarithmic divergence $\frac{1-\gamma }{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$ (lines), Eq. (19). We also see that $I_{1/2}\left(\mathit{\sigma }\right)\le I(\mathit{\sigma };\varphi )$, as expected from the data processing inequality for the Markov chain ${\mathit{\sigma }}^A-\varphi -{\mathit{\sigma }}^B$.

Diverging information also characterizes statistical inference.

Generally, the information that iid samples encode about the parameters of a distribution grow logarithmically with sample size $N$ in the limit $N\to \infty$ [35, 36]. Take, for example, a process that generates iid samples $\mathbf{x}=\left(x_1,x_2,\dots ,x_N\right)$ from a probability distribution $P\left(x_i\mid \varphi \right)$ where $\varphi$ is an unknown, real-valued, and possibly multi-dimensional parameter, which one wants to estimate from $\mathbf{x}$. Given an a priori distribution $P\left(\varphi \right)$, the joint marginal distribution of the samples is

$$P\left(\mathbf{x}\right)=\int d\varphi P\left(\varphi \right){\prod }_{i=1}^{N}P\left(x_i\mid \varphi \right).$$ (2)

If we divide the samples into two macroscopic groups, i.e., $\mathbf{x}=\left({\mathbf{x}}^A,{\mathbf{x}}^B\right)$, then the mutual information between these groups reads [35] (see also Appendix A)

$$I\left({\mathbf{x}}^A;{\mathbf{x}}^B\right)=\frac{k}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_2\frac{N_A{N}_B}{N}+O\left(N^0\right)\approx \frac{k}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N.$$ (3)

where $k$ denotes the dimensionality of the parameter $\varphi$, or, more precisely, the dimensionality of the subset of the parameters that can be inferred from $\mathbf{x}$.³ Importantly, the information between the two halves exists only to the extent that both depend on $\varphi$. In fact, this information is nearly the same as the information one has about the parameter $\varphi$ having observed $\mathbf{x}$, i.e., $I\left({\mathbf{x}}^A;{\mathbf{x}}^B\right)\approx I(\mathbf{x};\varphi )$ [35] (see Fig. 1C). We emphasize that this logarithmic divergence arises without fine tuning and for a broad range of assumptions about $P\left(\varphi \right)$ and $P\left(x_i\mid \varphi \right)$.

Extrinsic criticality is equivalent to statistical learning.

We can interpret the information divergence in inference problems as a signature of criticality, imposed extrinsically by the unknown, fluctuating variable $\varphi$. We consider a physical manifestation of an extrinsically critical system [26, 27] and write down the joint probability of $N$ noninteracting, identical binary spins in an external magnetic field $\varphi$,

$$P(\mathit{\sigma }\mid \varphi )={\prod }_{i=1}^N\frac{e^{\varphi {\mathit{\sigma }}_i}}{2\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}\varphi }$$ (4)

where $\mathit{\sigma }=\left({\sigma }_1,{\sigma }_2,\dots ,{\sigma }_N\right)$ and ${\mathit{\sigma }}_i\in \{\pm 1\}$. If the field is not constant, but a random variable that fluctuates for different realizations of the system, then marginalizing over this field yields

$$P\left(\mathit{\sigma }\right)=\int d\varphi P\left(\varphi \right){\prod }_{i=1}^N\frac{e^{\varphi {\sigma }_i}}{2\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}\varphi },$$ (5)

where $P\left(\varphi \right)$ is the marginal distribution of the fluctuating field. This equation takes the exact same form as Eq. (2), with the magnetic field playing the role of the model parameter and the spins of the iid samples. As a result, it follows immediately that $I(\mathit{\sigma };\varphi )\approx I\left({\mathit{\sigma }}^A;{\mathit{\sigma }}^B\right)\approx \frac{1}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$ [35], where ${\mathit{\sigma }}^A$ and ${\mathit{\sigma }}^B$ denote two halves of the system, see Fig. 1B. In fact, we can derive this result by noticing that the a posteriori distribution, $(\varphi \mid \mathit{\sigma })~P\left(\varphi \right)\mathrm{e}\mathrm{x}\mathrm{p}\left(\varphi {\sum }_i{\sigma }_i-N\mathrm{l}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}\varphi \right)$, sharpens as $N$ grows, with the asymptotic variance decreasing as $\mathrm{v}\mathrm{a}\mathrm{r}(\varphi \mid \mathit{\sigma })~1/N$. In other words, as our knowledge of the parameter improves with $N$, its a posteriori differential entropy decreases as $S(\varphi \mid \mathit{\sigma })\approx$ $-\frac{1}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$. Given that the a priori entropy $S\left(\varphi \right)$ is finite and does not change with $N$, we obtain

$$I\left({\mathit{\sigma }}^A;{\mathit{\sigma }}^B\right)\approx I(\mathit{\sigma };\varphi )=S\left(\varphi \right)-S(\varphi \mid \mathit{\sigma })\approx \frac{1}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N,$$ (6)

where the approximations hide the terms of order $O\left(N^0\right)$.

We can glean additional insights from a more traditional statistical mechanics argument. First we recast Eq. (5) as

$$P\left(\mathit{\sigma }\right)=2^{-N}\int d\varphi P\left(\varphi \right)e^{N(\varphi m-\mathrm{l}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}\varphi )},$$ (7)

where $m=m\left(\mathit{\sigma }\right)={\sum }_i{\mathit{\sigma }}_i/N$ is the magnetization. For large $N$, we evaluate the above integral, using the saddle point approximation and assuming that $P\left(\varphi \right)$ satisfies certain technical conditions [26],

$$P\left(\mathit{\sigma }\right)\approx 2^{-N}\sqrt{\frac{2\pi }{N\left(1-m^2\right)}}P\left({\varphi }^{\mathrm{*}}\right)e^{N\left({\varphi }^{\mathrm{*}}m-\mathrm{l}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}{\varphi }^{\mathrm{*}}\right)},$$ (8)

where ${\varphi }^{\mathrm{*}}={\varphi }^{\mathrm{*}}\left(m\right)={\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}}^{-1}m$ denotes the saddle point. Defining the energy function as $E\left(\mathit{\sigma }\right)=-\mathrm{l}\mathrm{n}P\left(\mathit{\sigma }\right)$ and recalling that the thermodynamic entropy is the logarithm of the density of states—i.e., $𝒮\left(m\right)=\mathrm{l}\mathrm{n}\left[\frac{N}{2}\left(\begin{array}{c}N\\ N_{+}\end{array}\right)\right]$ with $N_{+}=N(1+m)/2$—we obtain to the leading order in $N$ [26]

$$E\left(m\right)-𝒮\left(m\right)=\mathrm{l}\mathrm{n}\left(1-m^2\right)-\mathrm{l}\mathrm{n}P\left({\varphi }^{\mathrm{*}}\right)+O\left(N^{-1}\right),$$ (9)

which does not grow with $N$. This equivalence between the energy and entropy to all increasing order in $N$ signifies a very strong form of criticality [10].

To compute the mutual information, we write down the entropy of the system

$$S\left(\mathit{\sigma }\right)=-\sum _{\mathit{\sigma }}P\left(\mathit{\sigma }\right)\mathrm{l}\mathrm{n}P\left(\mathit{\sigma }\right)={\int }_{-1}^{1}dm{e}^{𝒮\left(m\right)-E\left(m\right)}E\left(m\right).$$ (10)

Using Eq. (9) and noting that $\frac{d{\varphi }^{\mathrm{*}}}{dm}=\frac{1}{1-m^2}$, we have

$$\begin{gathered} S\left(\mathit{\sigma }\right)\approx {\int }_{-1}^1\frac{dm}{1-m^2}P\left({\varphi }^{\mathrm{*}}\right)E\left(m\right) \\ \approx N⟨\mathrm{l}\mathrm{n}2-\varphi \mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\varphi +\mathrm{l}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}\varphi {⟩}_{\varphi ~P\left(\varphi \right)}+\frac{1}{2}\mathrm{l}\mathrm{n}N, \end{gathered}$$ (11)

where we drop the terms of order $O\left(N^0\right)$ and smaller. As a result, the information between the two halves of the system reads

$$I\left({\mathit{\sigma }}^A;{\mathit{\sigma }}^B\right)\approx \frac{1}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_2{N}_A+\frac{1}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_2{N}_B-\frac{1}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N\approx \frac{1}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N,$$ (12)

which exhibits the same logarithmic divergence as one-parameter learning, Eq. (6).

The specifics of our calculations (conditional independence of spins, binary spins, etc.) are not important for the main result: an extrinsically varying parameter gives the mutual information between two subparts of the system (surface interaction terms excluded) that grows as $\frac{1}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$. This is a very specific form of criticality, mathematically equivalent to a problem of learning the said parameter from observations of the system’s microstate. For larger dimensional parameters, the calculation generalizes: mutual information is $\frac{1}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$ per inferable parameter component [35] (so long as the number of parameters is much smaller than $N)$.

Intrinsic criticality is described by ‘fractional’ parameter learning.

To illustrate the physics of intrinsic criticality, we consider a minimal model of fully connected identical Ising spins, defined by the energy function $E\left(\mathit{\sigma }\right)=-\frac{1}{4N}{\left({\sum }_i{\sigma }_i\right)}^2$. As usual, the joint probability distribution of the spins is $\left(\mathit{\sigma }\right)=$ $e^{-\beta E\left(\mathit{\sigma }\right)}/Z$, where $\beta =1/T$ denotes the inverse temperature. The partition function $Z$ guarantees proper normalization,

$$Z=\sum _{\mathit{\sigma }}{e}^{\frac{\beta }{4N}{\left(\sum _i{\sigma }_i\right)}^2}=\sqrt{\frac{N}{\pi \beta }}\int d\varphi \sum _{\mathit{\sigma }}{e}^{-\frac{N}{\beta }{\varphi }^2+\varphi \sum _i{\sigma }_i},$$ (13)

where the last equality follows from the Hubbard–Stratonovich transformation. We see that the system consists of fluctuating degrees of freedom that include the spins $\mathit{\sigma }$ as well as a scalar field $\varphi$, and the joint distribution of these variables reads

$$P(\mathit{\sigma },\varphi )=\frac{1}{Z}\sqrt{\frac{N}{\pi \beta }}{e}^{-\frac{N}{\beta }{\varphi }^2+\varphi \sum _i{\sigma }_i}.$$ (14)

Marginalizing out the spins yields

$$P_N\left(\varphi \right)=\frac{2^N}{Z}\sqrt{\frac{N}{\pi \beta }}{e}^{-\frac{N}{\beta }{\varphi }^2+N\mathrm{l}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}\varphi },$$ (15)

where the subscript $N$ emphasizes that this distribution is $N$-dependent. Now the distribution over the spin states is

$$P\left(\mathit{\sigma }\right)=\int d\varphi P_N\left(\varphi \right)\frac{P(\mathit{\sigma },\varphi )}{P_N\left(\varphi \right)}=\int d\varphi P_N\left(\varphi \right)\prod _i\frac{e^{\varphi {\mathit{\sigma }}_i}}{2\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}\varphi },$$ (16)

which is the same form as the extrinsically critical model in Eq. (5), albeit with an a priori distribution that depends on $N$.

Intrinsically induced fluctuations behave differently at and away from the critical point. To see this, we recall that $\mathrm{l}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}x\approx \frac{1}{2}{x}^2-\frac{1}{12}{x}^4$ for small $x$. Substituting this approximation into the exponent of Eq. (15) gives, for small $\varphi$,

$$P_N\left(\varphi \right)~e^{-N\left(\frac{1}{\beta }-\frac{1}{2}\right){\varphi }^2-N\frac{1}{12}{\varphi }^4}.$$ (17)

We see that $\mathrm{v}\mathrm{a}\mathrm{r}\left(\varphi \right)~1/N$ for $\beta <2$, but $\mathrm{v}\mathrm{a}\mathrm{r}\left(\varphi \right)~1/\sqrt{N}$ at $\beta =2$. For $\beta >2$, we have again $\mathrm{v}\mathrm{a}\mathrm{r}\left(\varphi \right)~1/N$ with a prefactor that depends on the curvature around the maxima of $P_N\left(\varphi \right)$. This change in scaling behaviors is what defines criticality: the fluctuation becomes atypically large only at the critical temperature ${\beta }_c=2$, with a variance that scales as $1/\sqrt{N}$ instead of $1/N$ for this simple fully connected model.

Divergent mutual information emerges only at the critical point, see Fig. 1A. The scaling behaviors of the variance of intrinsically induced fluctuations result in the asymptotic a priori differential entropy, $S\left(\varphi \right)\approx -\frac{1}{4}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$ at $\beta ={\beta }_c$ and $S\left(\varphi \right)\approx -\frac{1}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$, otherwise. From Eq. (14), we have $\mathrm{v}\mathrm{a}\mathrm{r}(\varphi \mid \mathit{\sigma })~1/N$, hence the a posteriori differential entropy scales as $S(\varphi \mid \mathit{\sigma })\approx -\frac{1}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$. We see that the logarithmic divergences in a priori and a posteriori entropy cancel exactly away from criticality, leading to mutual information that does not grow with $N$. At criticality, on the other hand, we have [38]

$$I\left({\mathit{\sigma }}^A;{\mathit{\sigma }}^B\right)\approx I(\mathit{\sigma };\varphi )\approx \frac{1}{4}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N.$$ (18)

We emphasize that the prefactor here is 1/4 instead of 1/2—that is, we learn only half of a parameter in this intrinsically critical setting.

Fractional learning is typical of intrinsically induced criticality.

In the above examples of intrinsic and extrinsic criticality, we see that the a posteriori differential entropy of the fluctuating field is asymptotically identical, i.e., $S(\varphi \mid \mathit{\sigma })\approx -\frac{1}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$. The a priori entropy, in contrast, differs: $S\left(\varphi \right)$ does not depend on $N$ for extrinsic fluctuations, whereas $S\left(\varphi \right)\approx -\frac{1}{4}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$ for the fully connected Ising model at criticality. As a result, the mutual information diverges at different rates [Eqs. (6) & (18)]. More generally, intrinsically induced fluctuations depend on the system size but need not take the same form as Eq. (15). For a unimodal fluctuating field with $\mathrm{v}\mathrm{a}\mathrm{r}\left(\varphi \right)~N^{-\gamma }$, the differential entropy reads $S\left(\varphi \right)\approx -\frac{\gamma }{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$. Therefore (see Appendix A),

$$I\left({\mathit{\sigma }}^A;{\mathit{\sigma }}^B\right)\approx I(\mathit{\sigma };\varphi )\approx \frac{1-\gamma }{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N.$$ (19)

Away from criticality, $\mathrm{v}\mathrm{a}\mathrm{r}\left(\varphi \right)~N^{-1}$, and the mutual information is finite: we cannot learn much about the field because, in noncritical thermodynamic systems, fluctuations are small, and we already know almost everything a priori. In contrast, critical fluctuations are larger, $\gamma <1$, so that observing the spins provides information about the specific realization of $\varphi$, and hence about the other spins. However, the entropy of intrinsically induced critical fluctuations decreases with $N$ quite generally, $0<\gamma <1$, resulting effectively in only a fraction of a parameter being available for learning, thereby a decrease in the information from its maximum possible of $\frac{1}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$. In Fig. 1C, we see that Eq. (19) agrees well with the asymptotic behavior of mutual information for a range of $\gamma$.

Thus, we have shown that, at least for a simple model, intrinsic criticality can be viewed as a learning problem, where the underlying large fluctuations in the order parameter leave sufficient freedom to learn its specific realization from observations of the system state. The expression of the information in terms of the difference of the a priori and the a posteriori entropy, Eq. (6), shows that these results will generalize to other critical systems: critical exponents will govern the a priori differential entropy of the order parameter, while the a posteriori differential entropy remains approximately $-\frac{1}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$. Similarly, for multi-dimensional order parameters, each dimension will contribute to the mutual information essentially independently. For correlated fluctuating parameters, the logarithmic divergence rate provides a measure of effective dimensions of the parameters [47]. Finally, extrinsic and intrinsic criticality will add up as well so that each extrinsically or intrinsically critical field (i.e., with $N$-independent or $N$-dependent fluctuations) will contribute $1/2$ or a smaller amount to the coefficient in front of ${\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$ in the information.

II. SIGNATURE OF CRITICALITY IN FINITE SYSTEMS

In reality, we can only observe a finite number of components, and the analysis of asymptotic behaviors, while instructive, becomes less precise. To this end, we now turn to the observability of criticality in finite systems. Criticality admits a number of potentially observable signatures. We focus on the properties of the empirical joint distribution of the system components, which can be constructed directly from observational data and thus is readily usable in the context of living systems. A critical system is expected to exhibit Zipf’s law, i.e., an inverse relationship between ranks and frequencies of the system states [10].

First we recap how Zipf’s behavior emerges from large fluctuations [26, 27] in the asymptotic limit. Consider the joint distribution of $N$ conditionally independent spins,

$$P\left(\mathit{\sigma }\right)=\int d\varphi P\left(\varphi \right){\prod }_{i=1}^{N}P\left({\sigma }_i\mid \varphi \right).$$ (20)

We see that ${\sum }_i\mathrm{l}\mathrm{n}P\left({\sigma }_i\mid \varphi \right)~N$, and thus $P(\mathit{\sigma }\mid \varphi )$ becomes a sharper function of $\varphi$ as $N$ increases, with a characteristic width that scales as $1/\sqrt{N}$. This scaling sets a threshold above which fluctuations are critical—that is, critical fluctuations are characterized by $\mathrm{v}\mathrm{a}\mathrm{r}\left(\varphi \right)~N^{-\gamma }$ with $\gamma <1$. As $N\to \mathrm{\infty }$, a critical prior, $P\left(\varphi \right)$, appears flat with respect to $P(\mathit{\sigma }\mid \varphi )$ which becomes infinitely sharp. Therefore, we can make the approximation,

$$P(\mathit{\sigma }\mid \varphi )\approx e^{-𝒮\left(\varphi \right)}\delta \left(\varphi -{\varphi }_{\mathit{\sigma }}^{\mathrm{*}}\right),$$ (21)

where $𝒮\left(\varphi \right)\equiv \mathrm{l}\mathrm{n}{\mathrm{\Sigma }}_{\mathit{\sigma }}\delta \left(\varphi -{\varphi }_{\mathit{\sigma }}^{\mathrm{*}}\right)$ plays the role of the thermodynamic entropy and ${\varphi }_{\mathit{\sigma }}^{\mathrm{*}}$ is the maximum of $P(\mathit{\sigma }\mid \varphi )$, assuming only one exists. Substituting the above approximation into Eq. (20) yields

$$P\left(\mathit{\sigma }\right)\approx P\left({\varphi }_{\mathit{\sigma }}^{\mathrm{*}}\right)e^{-𝒮\left({\varphi }_{\mathit{\sigma }}^{\mathrm{*}}\right)}\equiv e^{-ℰ\left({\varphi }_{\mathit{\sigma }}^{\mathrm{*}}\right)},$$ (22)

which illustrates that the energy function depends on $\mathit{\sigma }$ only through ${\varphi }_{\mathit{\sigma }}^{\mathrm{*}}$, i.e., $E\left(\mathit{\sigma }\right)=-\mathrm{l}\mathrm{n}P\left(\mathit{\sigma }\right)=ℰ\left({\varphi }_{\mathit{\sigma }}^{\mathrm{*}}\right)$. Importantly, the above equation signifies Zipf’s law via the equivalence between the extensive parts of the entropy and energy [10],

$$\underset{N\to \mathrm{\infty }}{\mathrm{l}\mathrm{i}\mathrm{m}}\left(ℰ\right(\varphi )-𝒮(\varphi \left)\right)/N=\underset{N\to \mathrm{\infty }}{\mathrm{l}\mathrm{i}\mathrm{m}}-N^{-1}\mathrm{l}\mathrm{n}P\left(\varphi \right)=0.$$ (23)

For mean-field criticality, $\mathrm{l}\mathrm{n}P\left(\varphi \right)~N{\varphi }^4$ [Eq. (17)] and the above cancellation holds when $\varphi$ is adequately small. This condition is guaranteed for a typical realization of $\varphi$ since $\mathrm{v}\mathrm{a}\mathrm{r}\left(\varphi \right)~1/\sqrt{N}$. For extrinsic criticality, $\mathrm{l}\mathrm{n}P\left(\varphi \right)~O\left(N^0\right)$ and the entropy-energy equivalence needs not rely on $\varphi$ being small.

In Fig. 2, we depict exact (infinite samples) rank-frequency plots for the fully connected Ising model at a range of temperatures. We see a clear deviation from the power-law behavior at all temperatures. The rank-frequency plot at $T_c$ approaches Zipf’s scaling, but only in the tail region and for an adequately large system (Fig. 2B). In a smaller system, Zipf’s law can appear more accurate at $T<T_c$ (Fig. 2A). This disconnect between the Zipf behavior and criticality in mean-field models is likely to be more visible under finite samples, which can only probe parts of the exact rank-frequency plots (see also Fig. 6). We emphasize here that the number of observations required to resolve the tail of this rank-frequency plot would be experimentally impractical, $2^N~{10}^{20}$ for $N=2^6$ and $2^N~{10}^{300}$ for $N=2^{10}$ (Fig. 2A&B).

FIG. 2. Zipf’s law is an inaccurate description of critical meanfield systems.

We depict exact rank-ordered distributions for the fully connected Ising model [Eq. (13)] at various temperatures $T$ (see legend) for the system size $N=2^6$ and $2^{10}$ (A and B, respectively). Note that the axes correspond to log-rank and log-frequency. The slope of the smoothed log-log plot illustrates how close the models are to Zipf’s law (dashed). Here we obtain the approximate slope (bottom row) by fitting a cubic polynomial to the ‘knees’ of the rank-frequency log-log plot. We see that the rank-frequency plots deviate from the power-law behavior at all temperatures, and this deviation does not appear to improve as the system grows. Importantly the critical model ($T=T_c$) exhibits Zipf’s scaling (dashed) only for the least frequent states in the tail region of the rank-ordered distributions and only when the system is large enough (B). In fact, in a smaller system, the rank-frequency plot can appear more Zipf-like at $T<T_C$ (A), see also Fig. 6.

FIG. 6. Empirical Zipf behavior needs not coincide with the critical point of the system.

A: We depict empirical rank-ordered distributions of the rank-one Ising model [Eqs. (26–27)] at various temperatures (see color legend in B). We see that the distribution is closest to Zipf’s law at an intermediate temperature, significantly lower than the critical temperature $T_c$. As an empirical, alternative definition of a critical temperature for finite systems, we consider the maximum of the specific heat or, equivalently the energy variance. This definition results in $T^{\mathrm{*}}<T_C$ (see B), which is still significantly higher than the temperature that exhibits approximate Zipf behavior. B: The energy variance $\mathrm{v}\mathrm{a}\mathrm{r}\left(E\right)$ (solid, left axis) and mutual information $I(\mathit{\sigma };\varphi )$ (dashed, right axis) provide measures of correlations in the system. Both maximize below $T_c$ roughly at the same temperature $\left(T^{\mathrm{*}}\right.$ for $\mathrm{v}\mathrm{a}\mathrm{r}\left(E\right)$). C: The prior $P\left(\varphi \right)$ changes from unimodal to bimodal at $T_c$, which indicates maximum correlations in the thermodynamic limit. However, in finite systems, $P\left(\varphi \right)$ cannot be accurately described by its behavior near maxima. We see that the specific heat peaks at $T^{\mathrm{*}}<T_C$ (see $\mathrm{B}$), where $P\left(\varphi \right)$ is bimodal but with significant density at $\varphi =0$. Here the results are for a system of 60 spins, 10⁸ realizations per model and $w_i~𝒩(\mu =1,s=0.3)$ [see Eq. (24)].

On the other hand, Fig. 3 shows that the rank-ordered distributions of identical spins under extrinsic fluctuations become more Zipf-like over the entire range of frequencies and ranks as the fluctuation variance increases and as the system grows. However, the rank-frequency plots approach Zipf’s law only when the system is sufficiently large (Fig. 3B).

FIG. 3. Zipf’s law emerges robustly for adequately large extrinsic fluctuations.

We show exact rank-ordered distributions for independent spins under a Gaussian fluctuating field of varying variance (see legend). We see that Zipf’s law becomes more accurate as the variance of the fluctuations and the system size increase. However, the rank-ordered distributions approach Zipf’s scaling over the entire range only when the system is adequately large (B). Here we obtain the approximate slope (bottom row) using the same method as in Fig. 2.

Although mean-field criticality results in a Zipf-like rankfrequency plot only in the hard-to-observe tail region, intrinsic criticality with a more general critical exponent—i.e., $\mathrm{v}\mathrm{a}\mathrm{r}\left(\varphi \right)~N^{-\gamma }$ with $\gamma \ne 1/2$—can generate rank-ordered distributions that are much closer to Zipf’s law. In Fig. 4, we compare the system-size dependence of the rank-frequency plots for the fully connected Ising model at $T_c(\gamma =1/2)$ to several models with larger fluctuations, including those potentially induced by intrinsic criticality of non-mean-field models $(0<\gamma <1/2)$ as well as the limiting case of extrinsic criticality $(\gamma =0)$. Figure 4A shows that the critical mean-field model exhibits Zipf scaling only in the tail region of the rank-ordered distribution (see also Fig. 2) and the agreement with Zipf’s law does not improve, nor degrade, as the system grows. On the other hand, we see that if the fluctuation variance decreases more slowly with $N$, i.e., $\gamma <1/2$, Zipf’s law gradually becomes more accurate as $N$ increases, see Fig. 4B–F. This behavior implies that empirically observed Zipf behavior is likely to indicate either extrinsic criticality or intrinsic criticality of non-mean-field type, characterized by critical fluctuations that scale only weakly with the system size.

FIG. 4. Rank-ordered distributions exhibit a smooth crossover between intrinsic and extrinsic criticality.

Normalized exact rank-frequency plots for a range of system sizes $N$ (see legend in F) illustrate a gradual crossover from the fully connected Ising model at criticality (A) to spins under an extrinsic fluctuating field (F). This crossover is induced by varying the scaling exponent of the variance of fluctuations $\mathrm{v}\mathrm{a}\mathrm{r}\left(\varphi \right)~N^{-\gamma }$, from $\gamma =1/2$ for the mean-field criticality in A [see Eq. (15)] to $\gamma <1/2$ for non-mean-field intrinsic criticality in B-E (see label) and $\gamma =0$ for extrinsic criticality in F. We see that the rank-frequency plots approach Zipf’s law (dashed) as the system grows, and at a faster rate for a smaller scaling exponent $\gamma$. For mean-field criticality, the agreement with Zipf scaling neither improves, nor degrades, with increasing $N$ (Panel A). In B-E, the prior over fluctuations takes the form $P\left(\varphi \right)~e^{-c{\varphi }^4}$ where $c\propto N^{2\gamma }$ [cf. Eq. (17)]. In all panels, the fluctuation variance at $N=64$ (the smallest $N$ shown) is the same and equal to that of the critical mean-field case. That is, in B-F, we have $\mathrm{v}\mathrm{a}\mathrm{r}\left(\varphi \right)={\lambda }_0(64/N{)}^{\gamma }$ with ${\lambda }_0$ denoting the variance of critical mean-field fluctuation at $N=64$.

III. ZIPF’S LAW AS A SIGNATURE OF CRITICALITY UNDER FINITE SAMPLES

Experimental measurements are finite in not only the number of observable degrees of freedom but also the number of observations. We now turn to examine the behavior of rank-frequency plots constructed from finite samples.

In the following, we generalize our conditionally independent model, Eq. (5), to describe nonidentical spins,

$$P(\mathit{\sigma }\mid \varphi )=\prod _i\frac{e^{{\mathit{\sigma }}_i{w}_i\varphi }}{2\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}{w}_i\varphi }.$$ (24)

Here $w_i~O\left(N^0\right)$ is the coupling strength between the spin ${\sigma }_i$ and the field $\varphi$, which can differ from one spin to another. For convenience, we also define

$$\mathrm{\Delta }\equiv 1/\sqrt{{\sum }_i{w}_i^2},$$ (25)

which is the characteristic width of $P(\mathit{\sigma }\mid \varphi )$ at large $N$, i.e., $P(\mathit{\sigma }\mid \varphi )~e^{-{\left(\varphi -{\varphi }_{\mathit{\sigma }}^{\mathrm{*}}\right)}^2/2{\mathrm{\Delta }}^2}$ with ${\varphi }_{\mathit{\sigma }}^{\mathrm{*}}={\sum }_i{w}_i{\mathit{\sigma }}_i/{\sum }_i{w}_i^2$. Following the argument in the preceding section, we expect Zipf behavior when $\mathrm{v}\mathrm{a}\mathrm{r}\left(\varphi \right)\gg {\mathrm{\Delta }}^2~O\left(N^{-1}\right)$.

Similarly, for mean-field criticality, we consider the rank-one Ising model, which generalizes the fully connected Ising model to nonidentical spins and is defined by the energy function,

$$E\left(\mathit{\sigma }\right)=-\frac{1}{4N}{\sum }_{ij}{w}_i{w}_j{\sigma }_i{\sigma }_j=-\frac{1}{4N}{\left({\sum }_i{w}_i{\sigma }_i\right)}^2,$$ (26)

where $w_i~O\left(N^0\right)$ and $w_i{w}_j$ describes the pairwise interaction between spins $i$ and $j$. This model can be recast as a conditionally independent model, with the conditional distribution of the spins given by Eq. (24) and an a priori distribution that depends on both $N$ and $\left\{w_i\right\}$ (see Appendix B),

$$P_N\left(\varphi \right)=\frac{2^N}{Z}\sqrt{\frac{N}{\pi \beta }}{e}^{-\frac{N}{\beta }{\varphi }^2+\sum _i\mathrm{l}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}{w}_i\varphi },$$ (27)

where $Z={\mathrm{\Sigma }}_{\mathit{\sigma }}{e}^{-\beta E\left(\mathit{\sigma }\right)}$. The thermodynamic critical temperature, ${\beta }_c=2N/{\sum }_i{w}_i^2$, marks the point at which this distribution changes from unimodal to bimodal.

Figure 5 illustrates that critical mean-field fluctuations are too small to generate experimentally observable Zipf’s law. We consider a system of 60 conditionally independent spins under a number of a priori distributions, $P\left(\varphi \right)$ (see inset), including that induced by a rank-one Ising model at $T_c$ [Eq. (27)]. For each a priori distribution, we draw 10⁸ iid realizations of the system and construct an empirical rank-frequency plot. In Fig. 5A, we see that the rank-one Ising model at $T_c$ does not produce Zipf’s law. Yet, if we make the fluctuation larger while fixing the shape (standardized moments) of the fluctuation prior, the resulting rank-frequency plot edges closer to Zipf scaling. However, the fluctuation variance is not the only factor that controls the behavior of the rank-frequency plot. In Fig. 5B, we see that at a fixed variance, an a priori distribution with thicker tails (larger standardized moments) produces a more Zipf-like rank-order plot. We emphasize that the resolution of these plots, especially in the tails, is limited by the number of samples. While we do not rule out the possibility that mean-field criticality may exhibit Zipf behavior in the tail region (see also Fig. 2), observing such behavior would require orders of magnitude more samples than 10⁸ and would therefore be experimentally impractical.

FIG. 5. Critical mean-field fluctuations are too small to support empirically observable Zipf behavior.

Panels A and B illustrate the effects of the width and the structure of the tails of the a priori distribution $P\left(\varphi \right)$ (inset) on finite-sample rank-frequency plots, respectively. The red dotted curves correspond to the rank-one Ising model at criticality [Eqs. (26–27)]. In A, we consider linear scaling of critical mean-field fluctuations such that $\mathrm{v}\mathrm{a}\mathrm{r}\left(\varphi \right)={\delta }^2{\lambda }_c$, where ${\lambda }_c$ is the fluctuation variance of the rank-one Ising model at $T_c$, for various scaling coefficient $\delta$ (see legend). We see that critical mean-field fluctuations $(\delta =1)$ do not result in Zipf behavior. Increasing the fluctuation variance while maintaining the overall shape of the fluctuation prior produces rank-frequency plots that progressively appear closer to Zipf scaling (dashed). In B, we consider the fluctuating field of the form $P\left(\varphi \right)~e^{-c|\varphi {|}^q}$. We vary the probability in the tails of $P\left(\varphi \right)$ with the shape parameter $q$ (see legend), and choose the scale parameter $c$ such that the fluctuation variance is fixed and equal to that of critical mean-field fluctuations (red dotted lines). Decreasing $q$ increases the probability of large $\varphi$, i.e., puts more mass in the tails of $P\left(\varphi \right)$, and improves the agreement between rank-frequency plots and Zipf’s law. Overall, while instructive in understanding critical behaviors, mean-field models are an unlikely candidate for explaining experimentally observed Zipf behavior. Here the results are for a system of 60 spins, 10⁸ realizations per model and $w_i~𝒩(\mu =1,s=0.3)$ [see Eq. (24)].

Extrapolating the asymptotic critical temperature to finite systems is, of course, somewhat dubious. In Fig. 6, we consider another frequently used empirical definition of criticality which identifies the critical point with the maximum in the specific heat or equivalently the energy variance. In the asymptotic limit $N\to \mathrm{\infty }$, this definition is identical to the thermodynamic critical temperature $T_c$. For finite systems, however, the specific heat maximum occurs at a lower temperature $T^{\mathrm{*}}<T_c$ (Fig. 6B). This temperature also coincides roughly with another possible empirical definition of criticality, namely the maximum of the mutual information $I(\mathit{\sigma };\varphi )$ which indicates maximum correlations and learnability (Fig. 6B). In Fig. 6A, we see that lowering the temperature of the rank-one Ising model from $T_c$ to $T^{\mathrm{*}}$ makes the system closer to, but still visibly different from, Zipf’s law. In fact, the closest agreement to Zipf’s law occurs at an even lower temperature. Two factors contribute to this intriguing temperature dependence. First, for mean-field criticality, Zipf scaling is expected only in the tail of the rank-frequency plot (see Fig. 2B), which requires a very large number of samples to resolve. Second, the correspondence between criticality and Zipf behavior is blurred in finite systems with the tendency for Zipf’s law to be more accurate at subcritical temperatures (see Fig. 2A). In sum, we demonstrate that for mean-field models, empirically observable Zipf behavior can be completely uncoupled from the usual notion of criticality.

Figure 6C illustrates the interaction-induced a priori distribution $P\left(\varphi \right)$ at various temperatures. We see that $P\left(\varphi \right)$ is flat around its maximum at $T_c$. In the thermodynamic limit, this condition leads to non-Gaussian fluctuations which break the central limit theorem and generate critical correlations. However, in finite systems, the field $\varphi$ is not well described by fluctuations in the immediate vicinity of its most likely values. We see that at the specific heat maximum $T^{\mathrm{*}}$, the a priori distribution has a non-negligible density at $\varphi =0$ even though it is bimodal with maxima at $\varphi \ne 0$. This distribution results in a larger fluctuation than at $T_c$, resulting in higher energy variance as well as a rank-ordered plot closer to Zipf’s law. As the temperature drops below $T^{\mathrm{*}}$, the most likely field values move further away from zero. Larger values of $\varphi$ suppress the variability of the system: each spin ${\mathit{\sigma }}_i$ aligns with $\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\left(w_i\varphi \right)$ with increasing probability, thereby the decrease in the energy variance. At high temperatures $T>T_c$, the a priori distribution becomes sharply peaked at $\varphi =0$, resulting in more random systems and thus a decrease in correlations. Rank-ordered plots also reflect this competition; reduced variability leads to a rank-ordered plot that decays faster than Zipf’s law at low $T$, whereas increased randomness yields a plot that appears flatter than Zipf’s law at high $T$.

Intrinsic fluctuations can lead to Zipf’s law without fine tuning.

So far we see that, in the absence of external fluctuations, the empirical signatures of large correlated fluctuations, such as Zipf-like distributions, are hard to observe. While this statement is true for equilibrium systems, critically large intrinsic fluctuations can emerge generically when the system is endowed with certain dynamics. To illustrate this point, we consider a discrete-time, dynamical generalization of the conditionally independent spin model [Eq. (24)],

$$P\left({\mathit{\sigma }}_{t+1}\mid {\varphi }_t\right)=\prod _i\frac{e^{{\mathit{\sigma }}_{i,t}{w}_i{\varphi }_t}}{2\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}{w}_i{\varphi }_t},$$ (28)

with ${\varphi }_t=\beta \left(m_t+\alpha m_{t-1}\right)$ and $m_t=\frac{1}{N}{\sum }_i{w}_i{\mathit{\sigma }}_{i,t}$. Here the index $i$ labels each spin and $t$ the time step. The parameter $\beta$ controls the stochasticity of the spins much like the inverse temperature and $\alpha$ couples states separated by two time steps. We illustrate the dynamics of the fluctuating field ${\varphi }_t$ for various $\beta$ and the corresponding a priori distributions in Fig. 7B&C, respectively. In Fig. 7A, we see that this model can result in Zipf behavior over a range of model parameters, demonstrating that fine tuning is not a requirement for empirically observable Zipf behavior in the absence of external fields. Although the fluctuations are generated entirely internally, we can interpret this emergence of Zipf’s law as extrinsic criticality. To see this, we note that the model parameters $\alpha$ and $\beta$ control the amplitude of the oscillations and thus the fluctuation variance (Fig. 7B&C). The system size $N$ only determines the stochasticity of the dynamics. As a result, the a priori distribution depends only weakly on $N$ and becomes completely independent of $N$ in the asymptotic limit. This diminishing system-size dependence makes the resulting fluctuations indistinguishable from extrinsic ones.

FIG. 7. Zipf’s law can emerge from intrinsically induced fluctuations without fine tuning.

A: Rank-ordered distributions for a dynamical spin model, Eq. (28), display Zipf’s law for a range of effective inverse temperatures $\beta$ (see legend). B&C : We illustrate typical empirical distributions of the fluctuating field $\varphi$ and its typical dynamics (same legend as A). The results shown are for a system of 60 spins, ${10}^8$ realizations per model and $w_i~𝒩(\mu =1,s=0.6)$, and we set $\alpha =-0.8$ [see Eq. (28)].

IV. DISCUSSION

Here we introduced general definitions of criticality encompassing extrinsically and intrinsically critical systems, and we showed that information-theoretic and learning-theoretic considerations allow us to view all non-spatial, critical systems on a similar footing. Namely, criticality leads to the logarithmic divergence in the information between two subsystems or between the system’s observable degrees of freedom and its fluctuating latent field. The coefficient in front of the divergence is semi-integer for systems with extrinsic criticality, and other fractions for intrinsic criticality. Both situations can be viewed as learning the parameter from measurements of the state of system components, and the a priori variance of this parameter is independent of the system size (extrinsic criticality) or decreases as the system size grows (intrinsic criticality).

We focused on the scenario where the most critical a system can be is when it has $\frac{1}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$ bits of mutual information per dimension of the order parameter, which is equivalent to the iid learning problems. Additional intrinsic couplings would then reduce the a priori variance of the order parameter and hence reduce the mutual information. However, some biological systems may have the a priori parameter variance that increases with $N$ [48], so that more than $\frac{1}{2}{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$ bits are contributed to the mutual information per latent field.

One can imagine this happening in an optimally designed sensory system, where spins are coupled to the field in a way to reduce the redundancy of the information they obtain about it. Investigating the properties of such systems from information-theoretic, learning, and statistical physics angles is clearly needed. Similarly, it is worth investigating systems in which the mutual information between macroscopic parts scales as a sublinear power of $N$, (rather than a logarithm), which correspond to an infinite number of latent fields with hierarchically smaller a priori variances [35].⁴ Finally, since all of our learning and information-theoretic arguments are asymptotic, and $O\left(1\right)$ corrections may not be negligibly small compared to ${\mathrm{l}\mathrm{o}\mathrm{g}}_{2}N$, subleading corrections are also worth investigating.

In addition, there is a striking similarity between the critical behavior of mutual information in classical systems and entanglement entropy, a quantum information-theoretic measure of correlations. At quantum critical points, long-ranged correlations lead to diverging entanglement entropy, violating the area law [51]. For infinite quantum critical spin chains, this divergence is logarithmic in the subsystem size with a universal prefactor that is related to the central charge of the corresponding conformal field theory [52–56]. It would be interesting to develop a learning-theoretic picture of quantum criticality and explore whether and how the central charge relates to the effective number of latent parameters.

We also investigated the observability of an empirical signature of criticality, namely Zipf’s law. While the correspondence between Zipf behavior and criticality is precise in the thermodynamic limit, whether it holds for a finite system depends on how critical the system is, i.e., how large the a priori variance $\mathrm{v}\mathrm{a}\mathrm{r}\left(\varphi \right)$ is, compared to the width of the conditional distribution $P(\mathit{\sigma }\mid \varphi )$. For extrinsic criticality, Zipf’s law emerges robustly under an adequately broad a priori fluctuation distribution. Intrinsic criticality, on the other hand, does not always induce large enough fluctuations to support Zipf behavior. In particular, mean-field critical fluctuations are too small to generate Zipf’s law even in the infinite-sample limit. Indeed, under finite samples, the closest agreement to Zipf’s law can occur at a temperature significantly lower than the thermodynamic critical temperature as well as the specific heat maximum, an alternative, empirical definition of a critical point. This approximate Zipf behavior at intermediate temperature results from the competition between order-promoting interactions and thermal noise, and is unlikely to be a signature of equilibrium intrinsic criticality in the usual sense.

Perhaps the disconnect between intrinsic criticality and Zipf behavior in finite systems is unsurprising, not least because of the blurred notion of criticality away from the thermodynamic limit. While using the specific heat maximum to indicate criticality is tempting, it requires assumptions on the probabilistic model that describes the data. Zipf behavior offers an alternative, model-free definition, but as we showed, it can be nontrivial to observe in intrinsically critical systems. We emphasize that no finite-system definition of criticality captures all of its thermodynamic signatures; for instance, finite-sample Zipf behavior does not correspond to maximum energy fluctuations or maximum correlations (see Fig. 6).

While our asymptotic analysis suggests that the correspondence between intrinsic criticality and Zipf behavior is more precise for larger systems, we focus on a relatively small system of 60 spins (Figs. 5–7) since it is more relevant to real measurements. In particular, a well-sampled rank-ordered plot becomes exceedingly difficult to achieve as the system size grows. For example, a rank-ordered distribution for the critical rank-one Ising model with 80 spins shows almost no structure even at 10⁸ samples (see Fig. 8). This loss of structure due to finite samples is less severe for extrinsic fluctuations but the agreement with Zipf’s law degrades with increasing $N$ (Fig. 9), in contrast to the infinite-sample case, in which Zipf’s law becomes more accurate as the system grows (Fig. 4F).

FIG. 8. Subsampling does not make intrinsically critical systems appear more Zipf-like.

We depict the rank-ordered frequency for the rank-one Ising model of $N$ spins, Eq. (26), at the critical temperature, for $N=\mathrm{40,80}$ (a & b), and a range of observed subsystem size $K$ (see legend). We see that limiting observation to a fraction of the full system, i.e., $K<N$, does not result in more Zipf-like behavior (dashed). It leads however to more structured rank-ordered distributions, especially when the system is large (b), since a system of fewer spins requires a smaller sample size to be well-sampled. The results shown are for 10⁸ realizations per model and $w_i~𝒩(\mu =1,s=0.3)$.

FIG. 9. Zipf’s law emerges robustly from large extrinsic fluctuations.

We display rank-ordered distributions for the conditionally independent model with $N$ spins for $N=\mathrm{40,60,80}$ (left to right) under a Gaussian fluctuating field of different widths (see legend). When the fluctuation $\varphi$ is large compared to the width of the conditional distribution $P(\mathit{\sigma }\mid \varphi )$—i.e., when $\mathrm{s}\mathrm{t}\mathrm{d}\left(\varphi \right)\gg \mathrm{\Delta }$, see Eq. (25)—the rank-ordered plots exhibit Zipf behavior (dashed). The empirical rank-ordered distribution displays meaningful structures only when constructed from adequate samples. Larger systems require more samples; for $N=80$, the distribution is completely flat for $\mathrm{s}\mathrm{t}\mathrm{d}\left(\varphi \right)/\mathrm{\Delta }=2$ even with 10⁸ samples (c). The results shown are for 10⁸ realizations per model and $w_i~𝒩(\mu =1,s=0.3)$.

We show further that some oscillatory systems can generate observable Zipf’s law without fine tuning or external fluctuations. We argue that the mechanism behind this behavior is mathematically equivalent to the extrinsic mechanism since the scale of the collective dynamics—hence the variance of the fluctuations once the time variable is integrated out—is often independent of the system size. Indeed, collective oscillations are common both in mathematical models (e.g., Refs. [57, 58]) and in biological systems (e.g., Refs. [59–61]). Our results suggest that models with a global dynamical variable that stays within a certain range could offer another plausible explanation for empirically observed Zipf’s law, without the need for extrinsic fluctuations.

Finally, we discuss how subsampling may affect the observability of Zipf behavior. Many experiments do not measure the system in its entirety and what we can vary is the number of the observed components rather than the system size. Perhaps the unobserved degrees of freedom could play the role of an extrinsic source of fluctuations for the observed ones, resulting in extrinsically induced criticality and thus making observation of Zipf’s law more probable. However, our simple model of intrinsic criticality does not support this thinking. Suppose we observe $K$ out of the total of $N$ spins. Intrinsic fluctuations quite generally become smaller with $N$, i.e., $\mathrm{v}\mathrm{a}\mathrm{r}\left(\varphi \right)~N^{-\gamma }$ with $0<\gamma$, whereas the width of the conditional probability $P(\mathit{\sigma }\mid \varphi )$ decreases with $K$, i.e., $\mathrm{\Delta }~K^{-1/2}$ [cf. Eq. (25)]. As a result, the relative fluctuation variance is $\mathrm{v}\mathrm{a}\mathrm{r}\left(\varphi \right)/{\mathrm{\Delta }}^2~(K/N)\times N^{1-\gamma }$ with $\gamma <1$ for critical systems. In other words, decreasing the observable fraction makes the fluctuation appear smaller and Zipf’s law less likely (even though the smaller number of degrees of freedom makes it easier to obtain better-sampled rank-frequency plots, see Fig. 8). This effect is even more acute when the system size far outnumbers the observed components, e.g., a recording of neural spikes in the brain.

Real systems can of course be more complicated than our simple model. For example, in spatially extended systems, the order parameter could be a field in space and the number of inferable parameters, e.g., the Fourier components of the order parameter, can depend on how many spins we observe. Thus, the bits available to be learned can depend on the number of observed spins, even though the a priori variances of the parameters do not (as they are set by the size of the entire system). Investigations of criticality and its empirical signatures in this setting are in order.

We end by pointing out that many biological critical systems become more Zipf-like as they grow [10], which begs the question of why this happens. As pointed out in Ref. [26], consider a sensory system that is learning the state of the outside world (that is, responds to its different values differently); one would expect this system to be constructed in a way not to decrease the variability of the world when the system size grows. Such systems would always be critical, and specifically extrinsic critical, maybe explaining their ubiquity.

Appendix A: Mutual information in conditionally independent models

In this appendix, we consider conditionally independent spin models and derive the leading contribution to the mutual information between the spins and the fluctuating field and between two halves of the system in the many-spin limit.

First, we write down the probability distribution of the spins,

$$P\left(\mathit{\sigma }\right)=\int d\varphi P\left(\varphi \right){\prod }_{i}P\left({\sigma }_i\mid \varphi \right),$$ (A1)

where $P\left(\varphi \right)$ denotes the distribution of the fluctuating field $\varphi$. The conditional probability of each spin reads

$$P\left({\sigma }_i\mid \varphi \right)=\frac{e^{{\sigma }_i{w}_i\varphi }}{2\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}{w}_i\varphi },$$ (A2)

where $w_i$ parametrizes the influence of the fluctuating field $\varphi$ on spin $i$. For convenience, we introduce

$$g(\varphi ;\mathit{\sigma })=-\frac{1}{N}{\sum }_i\left({\sigma }_i{w}_i\varphi -\mathrm{l}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}{w}_i\varphi \right),$$ (A3)

such that

$$P(\mathit{\sigma }\mid \varphi )={\prod }_{i}P\left({\mathit{\sigma }}_i\mid \varphi \right)=e^{-Ng(\varphi ;\mathit{\sigma })}/2^N.$$ (A4)

Therefore the full joint distribution can be written as

$$P(\mathit{\sigma },\varphi )=P(\mathit{\sigma }\mid \varphi )P\left(\varphi \right)=P\left(\varphi \right)\times e^{-Ng(\varphi ;\mathit{\sigma })}/2^N.$$ (A5)

We now consider the limit $N\to \mathrm{\infty }$. We assume that the weights $\left\{w_i\right\}$ are independent of the system size $N$ (e.g., they are drawn from a fixed distribution) such that $g(\varphi ;\mathit{\sigma })$ is intensive. For a smooth prior—i.e., ${\mathrm{l}\mathrm{i}\mathrm{m}}_{N\to \mathrm{\infty }}\frac{1}{N}\mathrm{l}\mathrm{n}P\left(\varphi \right)=0$—the joint distribution [Eq. (A5)] is dominated by fluctuations around the minimum of $g(\varphi ;\mathit{\sigma })$,

$$P(\mathit{\sigma },\varphi )\approx P\left({\varphi }_{\mathit{\sigma }}^{\mathrm{*}}\right)P\left(\mathit{\sigma }\mid {\varphi }_{\mathit{\sigma }}^{\mathrm{*}}\right)\times e^{-\frac{N}{2}{g}^{\mathrm{\text{'}}\mathrm{\text{'}}}\left({\varphi }_{\mathit{\sigma }}^{\mathrm{*}}\right){\left(\varphi -{\varphi }_{\mathit{\sigma }}^{\mathrm{*}}\right)}^2},$$ (A6)

where ${\varphi }_{\mathit{\sigma }}^{\mathrm{*}}$ is the root of $g^{\mathrm{\text{'}}}(\varphi ;\mathit{\sigma })=0$ and we drop the superfluous dependence on $\mathit{\sigma }$ from $g^{\mathrm{\text{'}}\mathrm{\text{'}}}(\varphi ;\mathit{\sigma })=\frac{1}{N}{\sum }_i{w}_i^2{\mathrm{s}\mathrm{e}\mathrm{c}\mathrm{h}}^2{w}_i\varphi$. We see that when conditioned on the spins, the fluctuation is Gaussian,

$$P(\varphi \mid \mathit{\sigma })\approx 𝒩\left(\varphi \mid \mu ={\varphi }_{\mathit{\sigma }}^{\mathrm{*}},s^2=\frac{1}{N{g}^{\mathrm{\text{'}}\mathrm{\text{'}}}\left({\varphi }_{\mathit{\sigma }}^{\mathrm{*}}\right)}\right).$$ (A7)

As a result, we obtain the conditional differential entropy,

$$S(\varphi \mid \mathit{\sigma })\approx -\frac{1}{2}\mathrm{l}\mathrm{n}N+\frac{1}{2}\mathrm{l}\mathrm{n}2\pi e-\frac{1}{2}\sum _{\mathit{\sigma }}P\left(\mathit{\sigma }\right)\mathrm{l}\mathrm{n}{g}^{\mathrm{\text{'}}\mathrm{\text{'}}}\left({\varphi }_{\mathit{\sigma }}^{\mathrm{*}}\right).$$ (A8)

We see that the logarithmic divergence is the leading contribution since the last two terms do not grow with $N$. For extrinsic fluctuations, $P\left(\varphi \right)$, and thus $S\left(\varphi \right)$, is independent of $N$; therefore, we obtain

$$I(\varphi ;\mathit{\sigma })=S\left(\varphi \right)-S(\varphi \mid \mathit{\sigma })\approx \frac{1}{2}\mathrm{l}\mathrm{n}N+O\left(N^0\right).$$ (A9)

On the other hand, if $P\left(\varphi \right)~e^{-N^{\gamma }\times c{\left(\varphi -{\varphi }_0\right)}^2}$ for some $\gamma \in \left[\mathrm{0,1}\right)$ and a constant $c>0$, its entropy diverges logarithmically $S\left(\varphi \right)\approx -\frac{\gamma }{2}\mathrm{l}\mathrm{n}N+O\left(N^0\right)$ and the mutual information reads

$$I(\varphi ;\mathit{\sigma })\approx \frac{1-\gamma }{2}\mathrm{l}\mathrm{n}N+O\left(N^0\right).$$ (A10)

We see that the decrease in information results from the fluctuation entropy that decreases logarithmically with $N$.

The same logarithmic divergence also emerges in the mutual information between two macroscopic halves of the system. To see this, we note that the entropy of the spins is given by

$$S\left(\mathit{\sigma }\right)=S\left(\varphi \right)-S(\varphi \mid \mathit{\sigma })+S(\mathit{\sigma }\mid \varphi )$$ (A11)

Similarly, the entropy of each half of the system reads

$$S\left({\mathit{\sigma }}^{\nu }\right)=S\left(\varphi \right)-S\left(\varphi \mid {\mathit{\sigma }}^{\nu }\right)+S\left({\mathit{\sigma }}^{\nu }\mid \varphi \right)$$ (A12)

where $v\in \{A,B\}$ and $\mathit{\sigma }=\left({\mathit{\sigma }}^A,{\mathit{\sigma }}^B\right)$. When the subsystems are large and the spins in each half are randomly chosen, we have [see Eq. (A8)]

$$S\left(\varphi \mid {\mathit{\sigma }}^{\nu }\right)\approx S(\varphi \mid \mathit{\sigma })-\frac{1}{2}\mathrm{l}\mathrm{n}\frac{N_v}{N}.$$ (A13)

We now write the mutual information between the two halves in terms of the above entropy,

$$I\left({\mathit{\sigma }}^A;{\mathit{\sigma }}^B\right)=S\left({\mathit{\sigma }}^A\right)+S\left({\mathit{\sigma }}^B\right)-S\left(\mathit{\sigma }\right)$$ (A14)

$$\approx I\left(\varphi ;\mathit{\sigma }\right)+\frac{1}{2}\text{ln}\frac{N_A{N}_B}{N^2}.$$ (A15)

where we use the fact that $I(\varphi ;\mathit{\sigma })=S\left(\varphi \right)-S(\varphi \mid \mathit{\sigma })$ and the property of conditional independence, $S(\mathit{\sigma }\mid \varphi )=S\left({\mathit{\sigma }}^A\mid \varphi \right)+$ $S\left({\mathit{\sigma }}^B\mid \varphi \right)$. For $N_A=N_B=N/2$, we see that $I\left({\mathit{\sigma }}^A;{\mathit{\sigma }}^B\right)$ is smaller than $I(\varphi ;\mathit{\sigma })$ by one bit.

Appendix B: Rank-one Ising models

Here we provide an analysis of rank-one Ising models—those with pairwise interaction matrices of rank one—defined by the energy function,

$$E\left(\mathit{\sigma }\right)=-\frac{1}{4N}\sum _{ij}{w}_i{w}_j{\sigma }_i{\sigma }_j=-\frac{1}{4N}{\left(\sum _i{w}_i{\sigma }_i\right)}^2,$$ (B1)

where $\mathit{\sigma }=\left({\sigma }_1,{\sigma }_2,\dots ,{\sigma }_N\right)$ denotes the state of the system, ${\sigma }_i\in \{\pm 1\}$ the spin at site $i\in \{\mathrm{1,2},\dots ,N\}$, and the product $w_i{w}_j$ describes the interaction between spins $i$ and $j$. We note that the terms with $i=j$ only add an irrelevant constant. This model generalizes the fully-connected Ising model which corresponds to setting $w_i=w_j$ for all $i$ and $j$. As usual, the probability distribution of the system configuration is given by

$$P\left(\mathit{\sigma }\right)=e^{-\beta E\left(\mathit{\sigma }\right)}/Z,$$ (B2)

where we introduce the inverse temperature $\beta =1/T$ and the partition function $Z={\sum }_{\mathit{\sigma }}{e}^{-\beta E\left(\mathit{\sigma }\right)}$.

Computing the partition function by directly summing over all possible spin states is generally analytically intractable. Instead, we trade this summation for an integral using the Hubbard–Stratonovich transformation,

$$Z=\sum _{\mathit{\sigma }}{e}^{\frac{\beta }{4N}{\left(\sum _i{w}_i{\sigma }_i\right)}^2}=\sqrt{\frac{N}{\pi \beta }}\int d\varphi \sum _{\mathit{\sigma }}{e}^{-\frac{N}{\beta }{\varphi }^2+\varphi \sum _i{w}_i{\sigma }_i}.$$ (B3)

We see that the spins become noninteracting at the cost of introducing a new fluctuating field $\varphi$ which correlates with the spins via the joint distribution,

$$P(\mathit{\sigma },\varphi )=\frac{1}{Z}\sqrt{\frac{N}{\pi \beta }}{e}^{-\frac{N}{\beta }{\varphi }^2+\varphi \sum _i{w}_i{\mathit{\sigma }}_i}.$$ (B4)

Summing out each spin variable from Eq. (B3) yields

$$Z=2^N\sqrt{\frac{N}{\pi \beta }}\int d\varphi e^{-\frac{N}{\beta }{\varphi }^2+{\mathrm{\Sigma }}_i\mathrm{l}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}{w}_i\varphi }.$$ (B5)

As a result, we can express various thermodynamic variables of the spins as integrals over a continuous field which are usually more convenient than summations over discrete spin states. In particular, the internal energy, entropy and heat capacity—U, S and C, respectively—read

$$U=-\frac{\partial }{\partial \beta }\text{ln}Z=\frac{1}{2\beta }\left(1-\frac{2N}{\beta }\langle {\varphi }^2\rangle \right)$$ (B6)

$$S=\beta U+\mathrm{l}\mathrm{n}Z$$ (B7)

$$C={\beta }^2\frac{{\partial }^2}{\partial {\beta }^2}\mathrm{l}\mathrm{n}Z=2\beta U-\frac{1}{2}+\frac{N^2}{{\beta }^2}\mathrm{v}\mathrm{a}\mathrm{r}\left({\varphi }^2\right),$$ (B8)

where $⟨{\varphi }^2⟩$ and $\mathrm{v}\mathrm{a}\mathrm{r}\left({\varphi }^2\right)$ denote the mean and variance of ${\varphi }^2$.

In addition, we see that a rank-one Ising model is equivalent to a conditionally independent model with a fluctuation field, induced by the intrinsic interactions between spins. The marginal distribution of this field is given by

$$P_N\left(\varphi \right)=\sum _{\mathit{\sigma }}P(\mathit{\sigma },\varphi )=\frac{2^N}{Z}\sqrt{\frac{N}{\pi \beta }}{e}^{-\frac{N}{\beta }{\varphi }^2+\sum _i\mathrm{l}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}{w}_i\varphi },$$ (B9)

and thus we have

$$P(\mathit{\sigma }\mid \varphi )=\frac{P(\mathit{\sigma },\varphi )}{P_N\left(\varphi \right)}=\prod _i\frac{e^{{\mathit{\sigma }}_i{w}_i\varphi }}{2\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}{w}_i\varphi }.$$ (B10)

We see again that conditioning on the fluctuating field removes the interactions between spins.

Recalling that $\mathrm{l}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{h}x\approx \frac{x^2}{2}-\frac{x^4}{12}$ for small $x$, we see that the fluctuation distribution $P_N\left(\varphi \right)$ exhibits a structural transition at

$${\beta }_c=\frac{2}{\frac{1}{N}\sum _i{w}_i^2}$$ (B11)

where it changes from unimodal at $\beta <{\beta }_c$ to bimodal at $\beta >{\beta }_c$. In the limit $N\to \mathrm{\infty }$, this point corresponds to the critical temperature of an order-disorder phase transition. In the disordered phase at high temperatures $\beta <{\beta }_c$, the spins are mostly random. In the ordered phase at low temperatures $\beta >{\beta }_c$, on the other hand, they mimic the pattern set by the signs of $\left\{w_i\right\}$.