Visualizing probabilistic models and data with Intensive Principal Component Analysis

Significance

We introduce Intensive Principal Component Analysis (InPCA), a widely applicable manifold-learning method to visualize general probabilistic models and data. Using replicas to tune dimensionality in high-dimensional data, we use the zero-replica limit to discover a distance metric, which preserves distinguishability in high dimensions, and an embedding with superior visualization performance. We apply InPCA to the model of cosmology which predicts the angular power spectrum of the cosmic microwave background, allowing visualization of the space of model predictions (i.e., different universes).

Abstract

Unsupervised learning makes manifest the underlying structure of data without curated training and specific problem definitions. However, the inference of relationships between data points is frustrated by the “curse of dimensionality” in high dimensions. Inspired by replica theory from statistical mechanics, we consider replicas of the system to tune the dimensionality and take the limit as the number of replicas goes to zero. The result is intensive embedding, which not only is isometric (preserving local distances) but also allows global structure to be more transparently visualized. We develop the Intensive Principal Component Analysis (InPCA) and demonstrate clear improvements in visualizations of the Ising model of magnetic spins, a neural network, and the dark energy cold dark matter ($\mathrm{\Lambda }\text{CDM}$) model as applied to the cosmic microwave background.

Visualizing high-dimensional data is a cornerstone of machine learning, modeling, big data, and data mining. These fields require learning faithful and interpretable low-dimensional representations of high-dimensional data and, almost as critically, producing visualizations which allow interpretation and evaluation of what was learned (1–4). Unsupervised learning, which infers features from data without manually curated data or specific problem definitions (5), is especially important for high-dimensional, big data applications in which specific models are unknown or impractical. For high dimensions, the relative distances between features become small and most points are orthogonal to one another (6). A trade-off between preserving local and global structure must often be made when inferring a low-dimensional representation. Classic manifold learning techniques include linear methods such as principal component analysis (PCA) (7) and multidimensional scaling (MDS) (8), which preserve global structure but at the cost of obscuring local features. Existing nonlinear manifold learning techniques, such as t-distributed stochastic network embedding (t-SNE) (9) and diffusion maps (10), preserve the local structure while maintaining only some qualitative global patterns such as large clusters. The uniform manifold approximation (UMAP) (11) better preserves topological structures in data, a global property.

In this article, we develop a nonlinear manifold learning technique which achieves a compromise between preserving local and global structure. We accomplish this by developing an isometric embedding for general probabilistic models based on the replica trick (12). Taking the number of replicas to zero, we reveal an intensive property—an information density characterizing the distinguishability of distributions—ameliorating the canonical orthogonality problem and “curse of dimensionality.” We then describe a simple, deterministic algorithm that can be used for any such model, which we call Intensive Principal Component Analysis (InPCA). Our method quantitatively captures global structure while preserving local distances. We first apply InPCA to the canonical Ising model of magnetism, which inspired the zero-replica limit. Next, we show how InPCA can capture and summarize the learning trajectory of a neural network. Finally, we visualize the dark energy cold dark matter ($\mathrm{\Lambda }\text{CDM}$) model as applied to the cosmic microwave background (CMB), using InPCA, t-SNE, and diffusion maps.

Model Manifolds of Probability Distributions

Any measurement obtained from an experiment with uncertainty can generally be understood as a probability distribution. For example, when some data $x$ are observed with normally distributed noise $\xi$ of variance ${\sigma }^2$, under experimental conditions ${\mathit{\theta }}_j$, a model is expressed as

$$x=f\left({\mathit{\theta }}_j\right)+\xi \mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}\mathcal{L}\left(\xi \right)\backsim \hspace{0.17em}\mathcal{N}\left(0,{\sigma }^2\right),$$ [1]

and $f\left({\mathit{\theta }}_i\right)$ is a prediction given the experimental conditions. This relationship is equivalent to saying that the probability of measuring data $x$ given some conditions $\mathit{\theta }$ is

$$\mathcal{L}\left(x\mid \mathit{\theta }\right)\backsim \hspace{0.17em}\mathcal{N}\left(f\left(\mathit{\theta }\right),{\sigma }^2\right).$$ [2]

More complicated noise profiles with asymmetry or correlations can be accommodated with this picture. Measurements without an underlying model can also be seen as distributions, where a measurement $x_i$ with uncertainty $\sigma$ can induce a probability $\mathcal{L}\left(x\mid x_i,\sigma \right)$ of observing new data $x$.

We define a probabilistic model $\mathcal{L}\left(\mathit{x}\mid \mathit{\theta }\right)$, the likelihood of observing data $\mathit{x}$ given parameters $\mathit{\theta }$. The model manifold is defined as the set of all possible predictions, $\left\{\mathcal{L}\left(\mathit{x}\mid {\mathit{\theta }}_i\right)\right\}$, which is a surface parameterized by the model parameters $\left\{{\mathit{\theta }}_i\right\}$. The parameter directions related to the longest distances along the model manifold have been shown to predict emergent behavior (how microscopic parameters lead to macroscopic behavior) (13). We will see that InPCA orders its principal components by the length of the model manifold along their direction, highlighting global structure. The boundaries of the model manifold represent simplified models which retain predictive power (14), and the constraint of data lying near the model manifold has been used to optimize experimental design (15). In this article, we study the Ising model, which defines probabilities of spin configurations given interaction strengths; a neural network, which predicts the probability of an image representing a single handwritten digit given weights and biases; and $\mathrm{\Lambda }\text{CDM}$, which predicts the distribution of CMB radiation given fundamental constants of nature.

Hypersphere Embedding

We promised an embedding which both is isometric and preserves global structures. We satisfy the first promise by considering the hypersphere embedding

$$\left\{z_{\mathit{x}}\left({\mathit{\theta }}_{\mathit{i}}\right)\right\}=\left\{2\sqrt{\mathcal{L}\left(\mathit{x}\mid {\mathit{\theta }}_{\mathit{i}}\right)}\right\},$$ [3]

where the normalization constraint of $\mathcal{L}\left(\mathit{x}\mid \mathit{\theta }\right)$ forces $z_{\mathit{x}}$ to lie on the positive orthant of a sphere. A natural measure of distance on the hypersphere is the Euclidean distance, in this case also known as the Hellinger divergence (16)

$$\begin{array}{ccc}\hfill d^2\left({\mathit{\theta }}_{\mathbf{1}},{\mathit{\theta }}_{\mathbf{2}}\right)& =\hfill & {∥\mathit{z}\left({\mathit{\theta }}_{\mathbf{1}}\right)-\mathit{z}\left({\mathit{\theta }}_{\mathbf{2}}\right)∥}^2\hfill \\ \hfill & =\hfill & 8{\left(1-\sqrt{\mathcal{L}\left(\mathit{x}\mid {\mathit{\theta }}_1\right)}\cdot \sqrt{\mathcal{L}\left(\mathit{x}\mid {\mathit{\theta }}_2\right)}\right)}^2,\hfill \end{array}$$ [4]

where $\cdot$ represents the inner product over $x$. Now we can see that the hypersphere embedding is isometric: The Euclidean metric of this embedding is equal to the Fisher information metric $\mathcal{I}$ of the model manifold (17),

$$d^2\left(z_i,z_i+d{z}_i\right)=\sum _{i}d{z}_{i}d{z}_i=\sum _{kl}{\mathcal{I}}_{kl}d{\theta }_{k}d{\theta }_l.$$ [5]

The Fisher information metric (FIM) is the natural metric of the model manifold (18), so the hypersphere embedding preserves the local structure of the manifold defined by $\mathcal{L}\left(\mathit{x}\mid \mathit{\theta }\right)$.

As the dimension of the data increases, almost all features become orthogonal to each other, and most measures of distance lose their ability to discriminate between the smallest and largest distances (19). For the hypersphere embedding, we see that as the dimension of $x$ increases, the inner product in the Hellinger distance of Eq. 4 becomes smaller as the probability is distributed over more dimensions. In the limit of large dimension, all nonidentical pairs of points become orthogonal and equidistant around the hypersphere (a constant distance $\sqrt{8}$ apart), frustrating effective dimensional reductions and visualization.

To illustrate this problem with the hypersphere embedding, consider the Ising model, which predicts the likelihood of observing a particular configuration of binary random variables (spins) on a lattice. The probability of a spin configuration is determined by the Boltzmann distribution and is a function of a local pairwise coupling and a global applied field. The dimension is determined by the number of spin configurations, $2^N$, where $N$ is the number of spins. Holding temperature fixed at one, we vary $h$ and $J$: external magnetic field ($h\in \left(-1.3,1.3\right)$) and nearest-neighbor coupling ($J\in \left(-0.4,0.6\right)$), using a Monte Carlo method weighted by Jeffrey’s prior to sample 12,000 distinct points. From the resulting set of parameters, we compute $X_{ij}=\left\{{\mathit{z}}_i\left({\mathit{\theta }}_{\mathit{j}}\right)\right\}$ using the Boltzmann distribution and visualize the model manifold in the $N$-sphere embedding of Eq. 3 by projecting the predictions onto the first three principal components of $X$. Fig. 1A shows this projection of the model manifold of a $2\times 2$ Ising model which is embedded in $2^4$ dimensions. Fig. 1B shows a larger, $4\times 4$ Ising model, of dimension $2^{16}$. As the dimension is increased from $2^4$ to $2^{16}$, we see the points starting to wrap around the hypersphere, becoming increasingly equidistant and less distinguishable.

Fig. 1.

(A–C) Hypersphere embedding, illustrating an embedding of the 2D Ising model. Points were generated through a Monte Carlo sampling and visualized by projecting the probability distributions onto the first three principal components (28). The points are colored by magnetic field strength. As the system size increases from $2\times 2$ to $4\times 4$, the orthogonality problem is demonstrated by an increase in “wrapping” around the hypersphere. This effect can also be produced by instead considering four replicas of the original system, motivating the replica trick which takes the embedding dimension or number of replicas to zero.

A natural way to increase the dimensionality of a probabilistic model is by drawing multiple samples from the distribution. If $D$ is the dimension of $x$, then $N$ identical draws from the distribution will have dimension $D^N$. The more samples drawn, the easier it is to distinguish between distributions, mimicking the curse of dimensionality for large systems. We see this demonstrated for our Ising model in Fig. 1C, where we drew four replica samples from the same model. Note that compared with the original 2 $\times$ 2 model, the model manifold of the four-replica 2 $\times$ 2 model “wraps” more around the hypersphere, just like the larger, $4\times 4$ Ising model. High-dimensional systems have “too much information,” in the same way that large numbers of samples have too much information. In the next section, we consider the contraposition of the insight that a large number of replicas lead to the curse of dimensionality and discover an embedding which not only is isometric but also ameliorates the high-dimensional wrapping around the $n$-sphere.

Replica Theory and the Intensive Embedding

We saw in Fig. 1 that increasing the dimension of the data led to a saturation of the distance function Eq. 4. This problem is referred to as the loss of relative contrast or the concentration of distances (19), and to overcome it requires a non-Euclidean distance function, discussed below. In the previous section we saw the same saturation of distance could be achieved by adding replicas, increasing the embedding dimension. Fig. 2A shows this process taken to an extreme: the model manifold of the $2\times 2$ Ising model with the number of replicas taken to infinity. All of the points cluster together, obscuring the fact that the underlying manifold is 2D. To cure the abundance of information which makes all points on the hypersphere equidistant, we seek an intensive distance, such as the distance per number of replicas observed. Next, because the limit of many replicas artificially leads to the same symptoms of the curse of dimensionality, we consider the limit of zero replicas, a procedure which is often used in the study of spin glasses and disordered systems (20). Fig. 2B shows the result of this analysis, the intensive embedding, where the distance concentration has been cured, and the inherent 2D structure of the Ising model has been recovered.

Fig. 2.

Replicated Ising model illustrating the derivation of our intensive embedding. All points are colored by magnetic field strength. (A) Large dimensions are characterized by large system sizes; here we mimic a $128\times 128$ Ising model which is of dimension $2^{128^2}$. The orthogonality problem becomes manifest as all points are effectively orthogonal, producing a useless visualization with all points clustered in the cusp. (B) Using replica theory, we tune the dimensionality of the system and consider the limit as the number of replicas goes to zero. In this way, we derive our intensive embedding. Note that the z axis reflects a negative-squared distance, a property which allows violations of the triangle inequality and is discussed in the text.

To find the intensive embedding, we must first find the distance between replicated models. The likelihood for $N$ replicas of a system is given by their product

$${\mathcal{L}\left(\left.\left\{{\mathit{x}}_{\mathbf{1}},\dots ,{\mathit{x}}_{\mathit{N}}\right\}\mid \mathit{\theta }\right)\right)}^{\left(N\right)}=\mathcal{L}\left({\mathit{x}}_{\mathbf{1}}\mid \mathit{\theta }\right)\cdots \mathcal{L}\left({\mathit{x}}_{\mathit{N}}\mid \mathit{\theta }\right),$$ [6]

where the set $\left\{{\mathit{x}}_{\mathbf{1}},\dots ,{\mathit{x}}_{\mathit{N}}\right\}$ represents the observed data in the replicated systems. Writing the inner product or cosine angle between two distributions as

$$⟨{\mathit{\theta }}_{\mathbf{1}};{\mathit{\theta }}_{\mathbf{2}}⟩=\sqrt{\mathcal{L}\left(\mathit{x}\mid {\mathit{\theta }}_{\mathbf{1}}\right)}\cdot \sqrt{\mathcal{L}\left(\mathit{x}\mid {\mathit{\theta }}_{\mathbf{2}}\right)},$$ [7]

and using Eq. 4, the distance per replica $d_N^2$ between two points on the model manifold is

$$d_N^2\left({\mathit{\theta }}_{\mathbf{1}},{\mathit{\theta }}_{\mathbf{2}}\right)=\frac{d^2\left({\mathit{\theta }}_{\mathbf{1}},{\mathit{\theta }}_{\mathbf{2}}\right)}{N}=-8\frac{{⟨{\mathit{\theta }}_{\mathbf{1}};{\mathit{\theta }}_{\mathbf{2}}⟩}^N-1}{N}.$$ [8]

We are now poised to define the intensive distance by taking the number of replicas to zero:

$$d_I^2\left({\mathit{\theta }}_{\mathbf{1}},{\mathit{\theta }}_{\mathbf{2}}\right)=\underset{N\to 0}{lim}{d}_N^2\left({\mathit{\theta }}_{\mathbf{1}},{\mathit{\theta }}_{\mathbf{2}}\right)=-8\log ⟨{\mathit{\theta }}_{\mathbf{1}};{\mathit{\theta }}_{\mathbf{2}}⟩.$$ [9]

The last equality is achieved using the standard trick in replica theory, $\left(x^N-1\right)/N\to \log x$ as $N\to \infty$, which is a basis trick used to solve challenging problems in statistical physics (20). The trick is most evident using the identity $x^N=\exp \left(\log Nx\right)\approx 1+N\log x$. One can check that the intensive distance is isometric,

$$d_I^2\left(\mathit{\theta },\mathit{\theta }+\delta \mathit{\theta }\right)=\delta {\theta }^{\alpha }\delta {\theta }^{\beta }{g}_{\alpha \beta }=\delta {\theta }^{\alpha }\delta {\theta }^{\beta }{\mathcal{I}}_{\alpha \beta },$$ [10]

where again $\mathcal{I}$ is the Fisher information metric in Eq. 5, so that we can be confident the intensive embedding distance preserves local structures.

Importantly, the intensive distance does not satisfy the triangle inequality (and is thus non-Euclidean): The distance between points on the hypersphere can go to infinity, rather than lie constrained to the finite radius of the hypersphere embedding. Because of this, the intensive embedding can overcome the loss of relative contrast (19) discussed at the beginning of this section. Distances in the intensive embedding maintain distinguishability in high dimensions, as illustrated in Fig. 2B, wherein the 2D nature of the Ising model has been recovered. We hypothesize that this process, which cures the curse of dimensionality for models with too many samples, will also cure it for models with intrinsically high dimensionality. The intensive distance obtained here is proportional to the Bhattacharyya distance (21). Considering the zero-replica limit of the Hellinger divergence, we discovered a way to derive the Bhattacharyya distance. The importance of this is discussed further in the following section.

Connection to Least Squares.

Consider the concrete and canonical paradigm of models $f_i\left(\theta \right)$ with data points $x_i$ and additive white Gaussian noise, usually called a nonlinear least-squares model. The likelihood $\mathcal{L}\left(\mathit{x}\mid \mathit{\theta }\right)$ is defined by

$$-\log \mathcal{L}\left(\mathit{x}\mid \mathit{\theta }\right)=\sum _i\frac{{\left(f_i\left(\mathit{\theta }\right)-x_i\right)}^2}{2{\sigma }_i^2}+\log \mathcal{Z}\left(\mathit{\theta }\right),$$ [11]

where $\mathcal{Z}$ sets the normalization. A straightforward evaluation of the intensive distance given by Eq. 9 finds for the case of nonlinear least squares that

$$d_I^2\left({\mathit{\theta }}_{\mathbf{1}},{\mathit{\theta }}_{\mathbf{2}}\right)=\sum _i\frac{{\left(f_i\left({\mathit{\theta }}_{\mathbf{1}}\right)-f_i\left({\mathit{\theta }}_{\mathbf{2}}\right)\right)}^2}{{\sigma }_i^2},$$ [12]

so that the intensive distance is simply the variance-scaled Euclidean distance between model predictions.

Intensive Principal Component Analysis

Classical PCA takes a set of data examples and infers features which are linearly uncorrelated. (7). The features to be analyzed with PCA are compared via their Euclidean distance. Can we generalize this comparison to use our intensive embedding distance? Given a matrix of data examples $X\in {\mathbb{R}}^{m\times p}$ (with features along the rows), PCA first requires the mean-shifted matrix $M_{ij}=X_{ij}-{\overline{X}}_i=PX$, where $P_{ij}={\delta }_{ij}-1/p$ is the mean-shift projection matrix and $p$ is the number of sampled points. The covariance and its eigenvalue decomposition are then

$$\mathrm{c}\mathrm{o}\mathrm{v}\left(X,X\right)=\frac{1}{p}{M}^{T}M=X^{T}PPX=V\mathrm{\Sigma }{V}^T,$$ [13]

where the orthogonal columns of the matrix $V$ are the natural basis onto which the rows of $M$ are projected,

$$MV=\left(UD{V}^T\right)V=UD=U\sqrt{\mathrm{\Sigma }},$$ [14]

where the columns of $U\sqrt{\mathrm{\Sigma }}$ are called the principal components of the data $X$.

The principal components can also be obtained from the cross-covariance matrix, $M{M}^T$, since

$$M{M}^T=PX{X}^{T}P=\left(UD{V}^T\right){\left(UD{V}^T\right)}^T=U\mathrm{\Sigma }{U}^T.$$ [15]

The eigenbasis $U$ of the cross-covariance is the natural basis for the components of the data, and the eigenbasis $V$ of the covariance is the natural basis of the data points. For us this flexibility is invaluable, as the cross-covariance is more natural for expressing the distances between distributions of different parameters.

Writing our data matrix as $X_{ij}=z_i\left({\theta }_j\right)$ using Eq. 3 for replicated systems, the cross-covariance is

$$\begin{array}{ccc}\hfill {\left(M{M}^T\right)}_{ij}^{\left(N\right)}& =\hfill & {\left(PX{X}^{T}P\right)}_{ij}\hfill \\ \hfill & =\hfill & \left(\mathit{z}\left({\mathit{\theta }}_{\mathit{i}}\right)-\overline{\mathit{z}}\right)\cdot \left(\mathit{z}\left({\mathit{\theta }}_{\mathit{j}}\right)-\overline{\mathit{z}}\right)\hfill \\ \hfill & =\hfill & 4{⟨{\mathit{\theta }}_i;{\mathit{\theta }}_j⟩}^N+\frac{4}{p^2}\sum _{k,k\prime =1}^p{⟨{\mathit{\theta }}_k;{\mathit{\theta }}_{k\prime }⟩}^N\hfill \\ \hfill & \hspace{0.17em}\hfill & -\frac{4}{p}\sum _{k=1}^p\left({⟨{\mathit{\theta }}_i;{\mathit{\theta }}_k⟩}^N+{⟨{\mathit{\theta }}_j;{\mathit{\theta }}_k⟩}^N\right),\hfill \end{array}$$ [16]

where $\overline{z}$ is the average over all sampled parameters, and we used the definition of $z$ in Eq. 6. As with the intensive embedding, we can take the limit as the number of replicas goes to zero to find

$$W_{ij}=\underset{N\to 0}{lim}\frac{1}{N}{\left(M{M}^T\right)}_{ij}^{\left(N\right)}.$$ [17]

Explicitly, the intensive cross-covariance matrix is

$$W_{ij}=4\log ⟨{\mathit{\theta }}_{\mathit{i}};{\mathit{\theta }}_{\mathit{j}}⟩+\frac{4}{p^2}\sum _{k,k\prime =1}^p\log ⟨{\mathit{\theta }}_{\mathit{k}\prime };{\mathit{\theta }}_{\mathit{k}}⟩-\frac{4}{p}\sum _{k=1}^p\left(\log ⟨{\mathit{\theta }}_{\mathit{i}};{\mathit{\theta }}_{\mathit{k}}⟩+\log ⟨{\mathit{\theta }}_{\mathit{j}};{\mathit{\theta }}_{\mathit{k}}⟩\right)$$ [18]

$$={\left(PLP\right)}_{ij},$$ [19]

where $L_{i,j}=4\log ⟨{\mathit{\theta }}_{\mathit{i}};{\mathit{\theta }}_{\mathit{j}}⟩$ and $P$ is the same projection matrix as defined above. In taking the limit of zero replicas, the structure of the cross-covariance has transformed

$$PX{X}^{T}P\underset{N\to 0}{\to }PLP,$$ [20]

and thus the symmetric Wishart structure is lost. It is therefore possible to obtain negative eigenvalues in this decomposition, which give rise to imaginary components in the projections. Note the similarity between the form of this cross-covariance and the double-centered distance matrix used in PCA and multidimensional scalding (MDS). This arises because both InPCA and PCA/MDS rely on mean shifing the input data before finding an eigenbasis. Thus, we view InPCA as a natural generalization of PCA to probability distributions and MDS to non-Euclidean embeddings.

In summary, InPCA is achieved by the following procedure: (i) Compute the cross-covariance matrix from a set of probability samples: Compute $W_{ij}$ as derived in Eq. 18. (ii) Compute the eigenvalue decomposition $W=U\mathrm{\Sigma }{U}^T$. (iii) Compute the coordinate projections, $T=U\sqrt{\mathrm{\Sigma }}$. (iv) Plot the projections using the columns of $T$.

Neural Network MNIST Digit Classifier.

To demonstrate the utility of InPCA, we use it to visualize the training of a two-layer convolution neural network (CNN), constructed using TensorFlow (22), trained on the MNIST dataset of hand-written digits (23). A set of 55,000 images was used to train the network, which was then used to predict the likelihood that an additional set of 10,000 images is classified each as a specific digit between 0 and 9. We use softmax (24) to calculate the probabilities from the category estimates supplied by the network. The CNN defines the likelihood $\mathcal{L}\left(x\mid \theta \right)$ that some input image $\theta$ contains the image of a particular handwritten digit $x$. The InPCA projections of the CNN output in Fig. 3 visualize the clustering learned by the CNN as a function of the number of learning epochs. The initialized network’s model manifold shows no knowledge of the digits (colored dots), but as training commences, the network separates digits into separate regions of its manifold (Movie S1). InPCA can be used as a fast, interpretable, and deterministic method for qualitatively evaluating what a neural network has learned.

Fig. 3.

Stages of training a CNN. Each point in the 3D projections represents one of 10,000 test images supplied to the CNN (29). At the first epoch, the neural network is untrained and so is unable to reliably classify images, with about a $90\%$ error rate—an effect reflected in the cloud of points. As training progresses and error rate decreases, the cloud begins to cluster as shown by InPCA at the 20th epoch. Finally, when completely trained, the clustered regions are manifest at the 2000th epoch with 10 clusters representing the 10 digits.

Properties of the Intensive Embedding and InPCA

The space characterized by our intensive embedding has two weird properties: First, it is formally 1D, yet there are multiple orthogonal directions upon which it can be projected; and second, it is Minkowski-like, in that it has negative squared distances, violating the triangle inequality. We posit that, fundamentally, this second property is what allows InPCA to cure the orthogonality problem.

We begin with a discussion of the the 1D nature of the embedding space. The embedding dimension is given by $D^N$, where $D$ is the original dimension of data $x$ and $N$ is the number of replicas. In the case of noninteger replicas the space becomes “fractional” in dimension and in the limit of zero replicas ultimately goes to one. However, it is still possible to obtain projections themselves along the dominant components of this space, by leveraging the cross-covariance instead of the covariance, summarized in step ii of our algorithm. Visualizations produced by InPCA are cross-sections of a space of the dimension equal to the number of sampled points of the model manifold $p$, instead of the dimension $D$ or $D^N$.

In the limit of zero replicas in Eq. 18, the positive-definite, Wishart structure of the cross-covariance matrix is lost. It is therefore possible to have negative squared distances. The non-Euclidean nature of the embedding (flat, but Minkowski-like) does not suffer from the concentration of distances which plagues Euclidean measures in high dimensions, thus allowing the model manifold to be “unwound” from the $N$-sphere and for InPCA to produce useful, low-dimensional representations.

Finally, the eigenvalues of InPCA correspond to the cross-sectional widths of the model manifold. We see this quite explicitly with the following example of a biased coin (specifically, in Fig. 4B) where the eigenvalues extracted from InPCA map directly to the manifold widths measured along the direction of the corresponding InPCA eigenvector. Therefore, we see that InPCA produces a hierarchy of directions, ordered by the global widths of the model manifold. Note that, as with classical PCA, this correspondence depends on how faithfully the model manifold was originally sampled; that is, InPCA can tell you about the structure of the manifold only from observed points.

Fig. 4.

InPCA visualization of biased coins (30). (A) The first two InPCA components correspond to the coin bias and variance, yet the first one is real and the second one is imaginary (the aspect ratio between axes is one). The contour lines represent constant distances from a fair coin and are hyperbolas: Points can be a finite distance from a fair coin yet an infinite distance from each other. (B) The ordered eigenvalues correspond to the manifold lengths, illustrating the hierarchical nature of the components extracted from InPCA.

Biased Coins.

To illustrate the properties of InPCA, we use it to visualize a simple probabilistic model, that of a simple biased coin. A biased coin has one parameter, the odds ratio of heads to tails, and so forms a 1D manifold. Fig. 4A shows the first two InPCA components for the manifold of biased coins, for 2,000 sampled points with probabilities uniformly spread between 0 and 1 (excluding the endpoints, since they are orthogonal and thus are infinitely far apart). The two extracted InPCA components correspond to the bias and variance of the coin, respectively. The hierarchy of components extracted from InPCA therefore corresponds to known features of the model (i.e., they are meaningful).

The importance of the negative-squared distances is illustrated in Fig. 4. The contour lines representing constant distances from a fair coin and are hyperbolas: Points can be a finite distance from a fair coin yet an infinite distance from each other. As two oppositely biased coins become increasingly biased, their distance from each other can go to infinity (because an outcome of a coin which always lands on heads will never be the same as an outcome of a coin which always land on tails) yet all points remain a finite distance from a fair coin. Note that all points are in the left and right portions of Fig. 4A, representing net positive distances (the intensive pairwise distances are all positive).

Comparing with t-SNE and Diffusion Maps.

We compare our manifold learning technique to two standard methods, t-SNE and the diffusion maps, by applying each one to the six-parameter $\mathrm{\Lambda }\text{CDM}$ cosmological model predictions of the CMB. The $\mathrm{\Lambda }\text{CDM}$ predicts $\mathcal{L}\left(x\mid \theta \right)$, where $x$ represents fluctuations in the CMB, and $\theta$ are the different cosmological parameters (i.e., it predicts the angular power spectrum of temperature and polarization anisotropies in sky maps of the CMB). Observations of the CMB from telescopes on satellites, balloons, and the ground provide thousands of independent measurements from large angular scales to a few arcminutes, which are used to fit model parameters. Here we consider only CMB observations from the 2015 Planck data release (25). The $\mathrm{\Lambda }\text{CDM}$ model we consider has six parameters, the Hubble constant ($H_0$) which we sampled in a range of 20–100 $\text{km}\cdot {\text{s}}^{-1}\cdot {\text{Mpc}}^{-1}$, the physical baryon density (${\mathrm{\Omega }}_b{h}^2$) and the physical cold dark matter density (${\mathrm{\Omega }}_c{h}^2$) both sampled from 0.0009 to 0.8, the primordial fluctuation amplitude ($A_s$) sampled from $10^{-11}$ to $10^{-8}$, the scalar spectral index ($\eta$) sampled from 0 to 0.98, and the optical depth at reionization ($\tau$) sampled from 0.001 to 0.9.

To determine the likelihood functions, we use the Code for Anisotropies in the Microwave Background software package to generate power spectra (26). We perform a Monte Carlo sampling of 50,000 points around the best-fit parameters provided by the 2015 Planck data release (25), with sample weights based on the intensive distance to the best fit.

In Fig. 5 we show the first two components of the manifold embedding for InPCA, t-SNE, and diffusion maps. To apply t-SNE and the diffusion map to probabilistic data we must provide a distance. We therefore use our intensive distance, from Eq. 9, for consistency and ease of comparison. In all three cases, the first component from each method is directly related to the primordial fluctuation amplitude $A_s$, which reflects the amplitude of density fluctuations in the early universe and is the dominant feature in real data (25). The second InPCA component predicts the Hubble constant, whereas the diffusion map predicts the scalar spectral index (a reflection of the size variance of primordial density fluctuations). In all cases, the projected components were plotted against the corresponding parameters to determine correlations, such as how one can see that $A_s$ corresponds with the first component in all three cases. Detailed plots and correlation coefficients for all three methods are provided in SI Appendix.

Fig. 5.

Model manifold of the six-parameter $\mathrm{\Lambda }\text{CDM}$ cosmological model predictions of temperature and polarization power spectra in the CMB using InPCA, t-SNE, and the diffusion map. Axes reflect the true aspect ratio from extracted components in all cases. Here the model manifold is colored by the primordial fluctuation amplitude, the most prominent feature in CMB data. (A) InPCA extracts, as the first and second component, this amplitude term as well as the Hubble constant. These parameters control the two most dominant features in the Planck data and so reflect a physically meaningful hierarchy of importance. In contrast, (B) t-SNE extracts only the amplitude term and (C) the diffusion map extracts the amplitude term and a different parameter, the scalar spectral index $\eta$, which reflects the scale variance of the density fluctuations in the early universe. In all plots, the orange point represents our universe, as represented by Planck 2015 data.

Such stark differences between manifold learning methods are surprising, as all techniques aim to extract important features in the data distribution, i.e., important geometric features in the manifolds. Given the ranges of sampled parameters, one would expect the variation in the Hubble constant to relate in some way to one of the dominant components, as it does for InPCA. Figures illustrating the effect of different parameters are provided in SI Appendix, following results from ref. 27.

There are two important differences between InPCA and other methods. First, InPCA has no tunable parameters and yields a geometric object defined entirely by the model distribution. For example, t-SNE embeddings rely on parameters such as the perplexity, a learning rate, and a random seed (yielding nondeterministic results), and the diffusion maps rely on a diffusion parameter and choice of diffusion operator, all of which must be manually optimized to obtain good results. Second, t-SNE and diffusion maps embed manifolds in Euclidean spaces in a way which aims to preserve local features. However, InPCA seeks to preserve both global and local features, by embedding manifolds in a non-Euclidean space.

Summary

In this article, we introduce an unsupervised manifold learning technique, InPCA, which captures low-dimensional features of general, probabilistic models with wide-ranging applicability. We consider replicas of a probabilistic system to tune its dimensionality and consider the limit of zero replicas, deriving an intensive embedding that ameliorates the canonical orthogonality problem. Our intensive embedding provides a natural, meaningful way to characterize a symmetric distance between probabilistic data and produces a simple, deterministic algorithm to visualize the resulting manifold.