Machine learning of the prime distribution

Abstract

In the present work we use maximum entropy methods to derive several theorems in probabilistic number theory, including a version of the Hardy–Ramanujan Theorem. We also provide a theoretical argument explaining the experimental observations of Y.–H. He about the learnability of primes, and posit that the Erdős–Kac law would very unlikely be discovered by current machine learning techniques. Numerical experiments that we perform corroborate our theoretical findings.

Introduction

Below we briefly recall some known results from Kolmogorov’s complexity theory and algorithmic randomness. The reader may find a detailed exposition of this theory in the monographs [1, 2]. Here, however, we assume that most fundamental notions in computability theory are known to the reader. We also provide interpretations for some of these results in the wider epistemological context.

Kolmogorov’s Invariance Theorem

Let U be a Turing–complete language that is used to simulate a universal Turing machine. Let p be an input of U that produce a given binary string x ∈ {0, 1}*. Then the Kolmogorov Complexity (or Minimal Description Length) of x is defined as

$$\begin{array}{c}\hfill K_U\left(x\right)=\underset{p}{\text{min}}\left\{\right|p|:U\circ p=x\}\end{array}$$ (1)

where U ∘ p denotes the output of U on input p.

Kolmogorov’s Invariance Theorem states that the above definition is (on the large scale) invariant of the choice of U. Namely, any other Turing–complete language (or, equivalently, another universal Turing machine) U′ satisfies

$$\begin{array}{c}\hfill \forall x\in {\{0,1\}}^{*}:|K_U\left(x\right)-K_{U^{\prime }}\left(x\right)|\le O\left(1\right)\end{array}$$ (2)

In other terms,

$$\begin{array}{c}\hfill \forall x\in {\{0,1\}}^{*}:-c(U,U^{\prime })\le K_U\left(x\right)-K_{U^{\prime }}\left(x\right)\le c(U,U^{\prime })\end{array}$$ (3)

for some positive constant c(U, U′) that depends only on U and U′.

The minimal description p such that U ∘ p = x serves as a natural representation of the string x relative to the Turing–complete language U. However, it turns out that p, and thus K_U(x), is not computable.

Levin’s Universal Distribution

The Algorithmic Probability of a binary string x can be defined as the probability of x being generated by U on random input p, where we consider p to be a binary string generated by fair coin flips:

$$\begin{array}{c}\hfill P\left(x\right)=\sum _{p:U\circ p=x}{2}^{-\left|p\right|}\end{array}$$ (4)

However, this quantity is not well–defined: we can choose one such input p and use it as a prefix for some p′ that is about log₂ k bits longer than p and such that U produces the same binary string: U ∘ p′ = x. Then 2^−|p′| ∼ 2^−|p|/k, and we have that

$$\begin{array}{c}\hfill P\left(x\right)\ge 2^{-\left|p\right|}\sum _k\frac{1}{k}\end{array}$$ (5)

for k from any subset of integers. Thus, neither P(x) ≤ 1, nor even has it to be finite.

Levin’s idea effectively formalizes Occam’s razor: we need to consider prefix–free Turing–complete languages only. Such languages are easy to imagine: if we agree that all documents end with the instruction \end{document} that cannot appear anywhere else, then we have a prefix–free language.

Given that any prefix–free code satisfies the Kraft-McMillan inequality, we obtain the Universal Distribution:

$$\begin{array}{c}\hfill 2^{-K_U\left(x\right)}\le P\left(x\right)=\sum _{p:U\circ p=x}{2}^{-\left|p\right|}\le 1\end{array}$$ (6)

where from now on we consider U to be prefix–free, and K_U is now corresponds to prefix–free Kolmogorov Complexity.

Interpretation

The above facts may also be given a physical interpretation: in a computable Universe, given a phenomenon with encoding x ∈ {0, 1}* generated by a physical process, the probability of that phenomenon is well–defined and equal to the sum over the probabilities of distinct and independent causes. The prefix–free condition is precisely what guarantees causal independence.

Furthermore, Levin’s Universal Distribution formalizes Occam’s razor as the most likely cause for that process is provided by the minimal description of x and more lengthy explanations are less probable causes.

Levin’s Coding Theorem

In the setting of prefix–free Kolmogorov complexity, Levin’s Coding theorem states that

$$\begin{array}{c}\hfill K_U\left(x\right)=-{\text{log}}_{2}P\left(x\right)+O\left(1\right)\end{array}$$ (7)

or, in other terms,

$$\begin{array}{c}\hfill P\left(x\right)=\Theta \left(2^{-K_U\left(x\right)}\right)\end{array}$$ (8)

The algorithmic probability of x was defined by Solomonoff simply as

$$\begin{array}{c}\hfill R\left(x\right)=2^{-K_U\left(x\right)},\end{array}$$ (9)

that is proportional to the leading term in the Universal Distribution probability.

Interpretation

Relative to a prefix–free Turing–complete language U (or, equivalently, a universal prefix–free Turing machine), the number of fair coin flips required to generate the shortest program that outputs x is on the order of ∼K_U(x).

Maximum entropy via Occam’s razor

Given a discrete random variable X with computable probability semi–measure P, it holds that

$$\begin{array}{c}\hfill \mathbb{E}\left[K_U\left(X\right)\right]=\sum _{x\in X}P\left(x\right)·K_U\left(x\right)=H\left(X\right)+O\left(1\right),\end{array}$$ (10)

where H(X) is the Shannon Entropy of X in base 2.

Moreover, we have that

$$\begin{array}{c}\hfill \mathbb{E}\left[R\left(X\right)\right]=2^{-\mathbb{E}\left[K_U\left(X\right)\right]}=2^{-H\left(X\right)+O\left(1\right)},\end{array}$$ (11)

which means that the expected Solomonoff probability is inverse proportional to the effective alphabet size.

Interpretation

The Shannon Entropy of a random variable in base 2 equals its Expected Kolmogorov Complexity up to a constant that becomes negligible in asymptotic analysis. This provides us with a precise answer to von Neumann’s original question.

As follows from (10), the average Kolmogorov Complexity and Entropy have the same order of magnitude. Hence machine learning systems that minimise the KL–Divergence are implicitly applying Occam’s razor.

However, what exactly do we mean by random variable? In a computable Universe the sample space of a random variable X represents the state–space of a Turing Machine with unknown dynamics whose output sequence is computable. As the generated sequence is computable, it is finite–state incompressible in the worst-case i.e. a normal number. Hence, a random variable corresponds to a stochastic source that is finite–state random.

This definition comes from the well–known correspondence between finite–state machines and normal numbers that establishes that a sequence is normal if and only if there is no finite–state machine that accepts it.

Algorithmic randomness

Let x be a binary sequence, and x|_N be the initial segment of N bits. Then x is algorithmically random if

$$\begin{array}{c}\hfill K_U\left(x{|}_N\right)\ge N-c,\end{array}$$ (12)

for some constant c ≥ 0. Such a string x is also called incompressible.

Thus, an integer $n\in \mathbb{N}$ is algorithmically random (or incompressible) if

$$\begin{array}{c}\hfill K_U\left(n\right)={\text{log}}_{2}n+O\left(1\right),\end{array}$$ (13)

and a discrete random variable X taking N values with computable distribution is algorithmically random if

$$\begin{array}{c}\hfill \mathbb{E}\left[K_U\left(X\right)\right]={\text{log}}_{2}N+O\left(1\right),\end{array}$$ (14)

which corresponds to the maximal entropy case of the uniform distribution.

Interpretation

A random binary string cannot be compressed because it does not contain any information about itself. Thus, a random number n needs log₂ n bits to be written in its binary form, and a random variable is “truly random” in a computable Universe if its expected Kolmogorov complexity is maximal.

Maximum entropy methods for probabilistic number theory

Below we provide an illustration of information–theoretical approach to classical theorems in number theory. Some proofs using the notions of entropy and Kolmogorov’s complexity have already been known [3], and other proofs below are definitely new. We prefer to keep both kinds of them to retain the whole picture.

The Erdős–Euclid theorem

This information–theoretic adaptation of Erdős’ proof of Euclid’s theorem is originally due to Ioannis Kontoyiannis [3]. In essence, this proof demonstrates that the information content of finitely many primes is insufficient to generate all the integers.

Let π(N) be the number of primes that are less or equal to a given natural number N. Let us suppose that the set of primes $\mathbb{P}$ is finite so we have $\mathbb{P}={\left\{p_i\right\}}_{i=1}^{\pi \left(N\right)}$ where π(N) is constant for N big enough.

Then we can define a uniform integer–valued random variable Z chosen uniformly from [1, N], such that

$$\begin{array}{c}\hfill Z=\left(\prod _{i=1}^{\pi \left(N\right)}{p}_i^{X_i}\right)·Y^2\end{array}$$ (15)

for some integer–valued random variables 1 ≤ Y ≤ N and X_i ∈ {0, 1}, such that Z/Y² is square–free. In particular, we have that $Y\le \sqrt{N}$, thus the upper bound for Shannon’s Entropy from Jensen’s inequality implies:

$$\begin{array}{c}\hfill H\left(Y\right)\le {\text{log}}_2\sqrt{N}=\frac{1}{2}{\text{log}}_{2}N.\end{array}$$ (16)

Also, since X_i is a binary variable, we have H(X_i) ≤ 1.

Then, we compute

$$\begin{array}{c}\hfill H\left(Z\right)={\text{log}}_{2}N.\end{array}$$ (17)

Moreover, we readily obtain the following inequality:

$$\begin{array}{c}\hfill \begin{array}{c}H\left(Z\right)=H(Y,X_1,\dots ,X_{\pi \left(N\right)})\hfill \\ \le H\left(Y\right)+\sum _{i=1}^{\pi \left(N\right)}H\left(X_i\right)\hfill \\ \le \frac{1}{2}{\text{log}}_{2}N+\pi \left(N\right)\hfill \end{array}\end{array}$$ (18)

which implies

$$\begin{array}{c}\hfill \pi \left(N\right)\ge \frac{1}{2}{\text{log}}_{2}N\end{array}$$ (19)

This clearly contradicts the assumption that π(N) is a constant for any natural N, and provides us with a simple but far from reasonable lower bound.

Chebyshev’s theorem via algorithmic probability

An information–theoretic derivation of Chebyshev’s Theorem [4], an important precursor of the Prime Number Theorem, from the Maximum Entropy Principle. Another proof was given by Ioannis Kontoyiannis in [3].

Chebyshev’s Theorem is a classical result that states that

$$\begin{array}{c}\hfill \sum _{p\le N}\frac{1}{p}·\text{ln}p\sim \text{ln}N,\end{array}$$ (20)

where we sum over the primes p ≤ N.

Here, two functions f(n)∼g(n) are asymptotically equivalent once ${\text{lim}}_{n\to \infty }\frac{f\left(n\right)}{g\left(n\right)}=1$.

In information–theoretical terms, the expected information gained from observing a prime number in the interval [1, N] is on the order of ∼log₂ N.

For an integer Z sampled uniformly from the interval [1, N] we may define its random prime factorization in terms of the random variables X_p:

$$\begin{array}{c}\hfill Z=\prod _{p\le N}{p}^{X_p}.\end{array}$$ (21)

As we have no prior information about Z, it has the maximum entropy distribution among all possible distributions on [1, N].

Since the set of finite strings is countable so there is a 1–to–1 map from {0, 1}* to $\mathbb{N}$ and we may define the Kolmogorov Complexity as a map from integers to integers, $K_U:\mathbb{N}\to \mathbb{N}$. As almost all finite strings are incompressible [2, Theorem 2.2.1], it follows that almost all integers are algorithmically random. Thus, for large N and for n ∈ [1, N], we have K_U(n) = log₂ n + O(1) almost surely.

Thus,

$$\begin{array}{c}\hfill \begin{array}{c}\mathbb{E}\left[K_U\left(Z\right)\right]\sim \mathbb{E}\left[{\text{log}}_{2}Z\right]=\hfill \\ \frac{1}{N}\sum _{k=1}^N{\text{log}}_{2}k=\frac{{\text{log}}_2(N!)}{N}\sim {\text{log}}_{2}N,\hfill \end{array}\end{array}$$ (22)

by using Stirling’s approximation

$$\begin{array}{c}\hfill {\text{log}}_2(N!)=N{\text{log}}_{2}N-N{\text{log}}_{2}e+O\left({\text{log}}_{2}N\right).\end{array}$$ (23)

On the other hand,

$$\begin{array}{c}\hfill \mathbb{E}\left[K_U\left(Z\right)\right]\sim \mathbb{E}\left[{\text{log}}_{2}Z\right]=\sum _{p\le N}\mathbb{E}\left[X_p\right]·{\text{log}}_{2}p.\end{array}$$ (24)

All of the above implies

$$\begin{array}{c}\hfill \mathbb{E}\left[K_U\left(Z\right)\right]\sim \sum _{p\le N}\mathbb{E}\left[X_p\right]·{\text{log}}_{2}p\sim {\text{log}}_{2}N.\end{array}$$ (25)

Here, we may apply the Maximum Entropy Principle to X_p, provided that $\mathbb{E}\left[X_p\right]$ is fixed: then X_p belongs to the geometric distribution [5, Theorem 12.1.1]. This can also be verified directly as shown in [3].

Thus, we compute:

$$\begin{array}{c}\hfill \mathbb{E}\left[X_p\right]=\sum _{k\ge 1}P(X_p\ge k)=\sum _{k\ge 1}\frac{1}{N}\lfloor \frac{N}{p^k}\rfloor \sim \frac{1}{p}.\end{array}$$ (26)

Thus, we rediscover Chebyshev’s theorem:

$$\begin{array}{c}\hfill \begin{array}{cc}& H(X_{p_1},\dots ,X_{p_{\pi \left(N\right)}})=H\left(Z\right)\sim \mathbb{E}\left[K_U\left(Z\right)\right]\hfill \\ & \sim \sum _{p\le N}\frac{1}{p}·{\text{log}}_{2}p\sim {\text{log}}_{2}N.\hfill \end{array}\end{array}$$ (27)

It should be noted that the asymptotic equivalence above does not depended on the logarithm’s base (as long as the same base is used on both sides).

The Hardy–Ramanujan theorem from information theory

Based on the ideas explained in the previous paragraphs, we can deduce a version of the classical Hardy–Ramanujan theorem.

The probability of a prime factor

Given an integer Z chosen uniformly from [1, N] with prime factorisation

$$\begin{array}{c}\hfill Z=\prod _{p\le N}{p}^{X_p},\end{array}$$ (28)

we observe that

$$\begin{array}{c}\hfill \begin{array}{cc}& P(X_1\ge k_1,\dots ,X_n\ge k_n)=\frac{1}{N}\lfloor \frac{N}{p_1^{k_1}\dots p_n^{k_n}}\rfloor \sim \hfill \\ & {\prod }_{i\le n}\frac{1}{N}\lfloor \frac{N}{p_i^{k_i}}\rfloor =\prod _{i\le n}P(X_i\ge k_i),\hfill \end{array}\end{array}$$ (29)

which means that X_i’s are asymptotically independent (i.e. independent in the large N limit).

In particular, the probability of having $p\in \mathbb{P}$ as a prime factor satisfies

$$\begin{array}{c}\hfill P(X_p\ge 1)\sim \frac{1}{p}.\end{array}$$ (30)

Likewise, the probability that two distinct primes $p,q\in \mathbb{P}$ are happen as factors simultaneously satisfies

$$\begin{array}{c}\hfill P(X_p\ge 1\wedge X_q\ge 1)\sim \frac{1}{p}·\frac{1}{q}.\end{array}$$ (31)

Thus, any two sufficiently large prime numbers $p,q\in \mathbb{P}$ are statistically independent.

The expected number of unique prime factors

For any integer Z sampled uniformly from [1, N], we may define its number of unique prime factors w(Z) = ∑_p≤N I(X_p ≥ 1). Thus, we calculate the expected value

$$\begin{array}{c}\hfill \mathbb{E}\left[w\left(Z\right)\right]=\sum _{p\le N}P(X_p\ge 1)\sim \sum _{p\le N}\frac{1}{p}=\text{ln}\text{ln}N+O\left(1\right),\end{array}$$ (32)

where we use Mertens’ 2nd Theorem for the last equality.

The variance of w(Z)

Given our previous analysis, the random variables I_p = I(X_p ≥ 1) are asymptotically independent, so that the variance of w(Z) satisfies

$$\begin{array}{c}\hfill \begin{array}{cc}& \text{Var}\left[w\left(Z\right)\right]\sim \sum _{p\le N}\mathbb{E}\left[I_p^2\right]-\mathbb{E}{\left[I_p\right]}^2\sim \hfill \\ & \sum _{p\le N}(\frac{1}{p}-\frac{1}{p^2})=\text{ln}\text{ln}N+O\left(1\right),\hfill \end{array}\end{array}$$ (33)

since ${\sum }_{p\le N}\frac{1}{p^2}\le \frac{{\pi }^2}{6}$.

The Hardy–Ramanujan theorem

For any positive k > 0 and a random variable Y with $\mathbb{E}\left[Y\right]=\mu$ and Var[Y] = σ², Chebyshev’s inequality informs us that

$$\begin{array}{c}\hfill P\left(\right|Y-\mu |>k\sigma )\le \frac{1}{k^2}.\end{array}$$ (34)

With Y = w(Z) and k = (ln ln N)^ε we obtain a probabilistic version of the Hardy–Ramanujan theorem. Namely, for a random integer $Z\sim \mathcal{U}\left(\right[1,N\left]\right)$, and any function ψ(N)∼(ln ln N)^1/2+ε, in the large N limit we have

$$\begin{array}{c}\hfill \left|w\right(Z)-\text{ln}\text{ln}N|\le \psi \left(N\right)\end{array}$$ (35)

with probability

$$\begin{array}{c}\hfill 1-O\left(\frac{1}{{\left(\text{ln}\text{ln}N\right)}^{2\epsilon }}\right).\end{array}$$ (36)

Discussion and conclusions

The Erdős–Kac theorem states that

$$\begin{array}{c}\hfill \frac{w\left(Z\right)-\text{ln}\text{ln}N}{\sqrt{\text{ln}\text{ln}N}}\end{array}$$ (37)

converges to the standard normal distribution $\mathcal{N}(0,1)$ as N → ∞.

This theorem is of great interest to the broader mathematical community as it is impossible to guess from empirical observations. In fact, it is far from certain that Erdős and Kac would have proved the Erdős–Kac theorem if its precursor, the Hardy–Ramanujan theorem, was not first discovered.

More generally, in the times of Big Data this theorem forces the issue of determining how some scientists were able to formulate correct theories based on virtually zero empirical evidence.

In our computational experiments Z was chosen to run through all possible N–bit integers with N = 24, and no normal distribution emerged according to the D’Agostino–Pearson test. The p–value associated to this test equals the probability of sampling normally distributed data that produces at least as extreme value of the D’Agostino–Pearson statistics as the actual observations. Thus, if the p–value is small it may be taken as evidence against the normality hypothesis. In our case we obtained a p–value of 0.0.

For comparison, the Central Limit Theorem clearly manifests itself for 2¹⁶, not even 2²⁴, binomial distribution samples. In this case, the p–value associated to the D’Agostino–Pearson test equals ∼0.4657.

The code used in our numerical experiments is available on GitHub [6], and all computations can be reproduced on a laptop computer such as MacBook Pro M1 with 8 GB RAM, or comparable.

In order to observe the normal order in the Erdős–Kac law, one would need Z to reach about 2²⁴⁰, not 2²⁴, instead [7].

Thus, non–trivial scientific discoveries of this kind that are provably beyond the scope of computational induction (and hence machine learning) do not yet have an adequate explanation.

Learning the prime distribution

Below we provide a theoretical study of the Kolmogorov complexity of the prime distribution.

Entropy of primes

In what follows $p\in \mathbb{P}$ will always be a fixed prime number, and X ∈ [1, N] will be chosen at random among the first N natural numbers. Considering prime numbers as random variables, or as elements of random walks, goes back to Billingsley’s heuristics in [8].

From the information–theoretic perspective, Chebyshev’s theorem states that the average code length of X expressed as the sequence of prime exponents $X_{p_1},\dots ,X_{p_{\pi \left(N\right)}}$ satisfies

$$\begin{array}{c}\hfill H\left(X\right)=H(X_{p_1},\dots ,X_{p_{\pi \left(N\right)}})\sim \sum _{p\le N}\frac{1}{p}·\text{ln}p\sim \text{ln}N,\end{array}$$ (38)

where we use the natural logarithm entropy instead of binary entropy. As previously noted, this is only a matter of computational convenience.

It turns out that the entropy of X, which is almost surely composite in the large N limit, essentially depends on the available primes p ≤ N.

A given non–negative integer n has binary code length

$$\begin{array}{c}\hfill \ell \left(n\right)=\lfloor {\text{log}}_{2}n\rfloor +1\sim {\text{log}}_{2}n+O\left(1\right).\end{array}$$ (39)

Given an integer n ∈ [1, N], we need at most ℓ(n) = bits to encode it, and thus it can be produced by a program of length ≤ℓ(n). Note that ℓ(N) is so far irrelevant here as we need a prefix–free encoding and do not consider adding zero bits for padding all binary strings to same length.

By Levin’s coding theorem,

$$\begin{array}{c}\hfill P(X=n)=\Theta \left(2^{-K_U\left(n\right)}\right),\end{array}$$ (40)

and thus we have

$$\begin{array}{c}\hfill 2^{-\ell \left(n\right)}\le P(X=n)\le \alpha ·2^{-\ell \left(n\right)},\end{array}$$ (41)

with α > 1, for most numbers n ∈ [1, N], as most binary strings are incompressible in the large N limit [2, Theorem 2.2.1].

Thus, we may as well resort to the following Ansatz:

$$\begin{array}{c}\hfill P(X=n)=\frac{1}{n},\end{array}$$ (42)

as it provides a computable measure on [1, N] that is roughly equivalent to the initial Levin’s distribution. The same discrete measure arises in the heuristic derivation of Benford’s law [9].

Let us consider a random variable Y that generates primes within [1, N] with probability

$$\begin{array}{c}\hfill P(Y=p)=\frac{1}{p},\end{array}$$

that appears to represent the algorithmic probability of a prime rather than its frequentist probability.

Then we can write

$$\begin{array}{c}\hfill H\left(Y\right)=-\sum _{p\le N}P(Y=p)·\text{ln}P(Y=p)=\sum _{p\le N}\frac{\text{ln}p}{p}\sim \text{ln}N.\end{array}$$

There are π(N) primes p satisfying p ≤ N, and thus we need exactly π(N) random variables Y_i to generate them all. Each Y_i has the same distribution as Y, and we assume that Y_i’s are independent.

Let ${\tilde{Y}}_N=(Y_1,\dots ,Y_{\pi \left(N\right)})$ be the ordered sequence generating all primes in [1, N]. Then Shannon’s source coding theorem informs us that ${\tilde{Y}}_N$ needs no less than π(N) ⋅ H(Y) bits to be encoded without information loss. Any smaller encoding will almost surely lead to information loss. This means that

$$\begin{array}{c}\hfill H\left({\tilde{Y}}_N\right)=\pi \left(N\right)·H\left(Y\right)+O\left(1\right).\end{array}$$ (43)

Thus,

$$\begin{array}{c}\hfill \mathbb{E}\left[K_U\left({\tilde{Y}}_N\right)\right]=H\left({\tilde{Y}}_N\right)+O\left(1\right)=\pi \left(N\right)·H\left(Y\right)+O\left(1\right)\sim \pi \left(N\right)·\text{ln}\left(N\right)\sim N,\end{array}$$ (44)

as the Prime Number Theorem provides the asymptotic equivalence

$$\begin{array}{c}\hfill \pi \left(N\right)\sim \frac{N}{\text{ln}N}.\end{array}$$ (45)

On the other hand, the most obvious encoding of primes is the N–bit string where we put 1 in position k if k is prime, and 0 if k is composite. The above equality for expected Kolmogorov’s complexity of ${\tilde{Y}}_N$ implies that for large values of N this string is almost surely incompressible.

Discussion and conclusions

The above discussion provides a theoretical corroboration of the experimental fact observed by Yang–Hui He in [10]. Namely, the complexity of “machine learning” the prime distribution on the interval [1, N] is equivalent to learning an algorithmically random sequence. Thus, the true positive rate of any model predicting primes should be very low.

There are, however, several theoretical issues with the above argument. One is that the proposed Benford’s probability is not a finite measure, as we have

$$\begin{array}{c}\hfill \sum _{k\le N}P(X=k)=\sum _{k\le N}\frac{1}{k}=\text{ln}N+O\left(1\right).\end{array}$$ (46)

The 1st Mertens’ theorem gives that summing over the primes only gives

$$\begin{array}{c}\hfill \sum _{p\le N}P(Y=p)=\sum _{p\le N}\frac{1}{p}=\text{ln}\text{ln}N+O\left(1\right),\end{array}$$ (47)

which is a much smaller quantity asymptotically, yet unbounded.

We can apply some regularization, such as setting

$$\begin{array}{c}\hfill P(X=n)=\frac{1}{n^s},\end{array}$$ (48)

for some s > 1, and thus obtaining a finite measure and trying to study its s → 1 limit. This measure is not normalized to be a probability measure, however this is not very crucial (e.g. Levin’s universal probability is not normalized either).

Indeed, we shall have

$$\begin{array}{c}\hfill \sum _{k=1}^{\infty }P(X=k)=\zeta \left(s\right)<\infty ,\end{array}$$ (49)

for any s > 1.

Moreover,

$$\begin{array}{c}\hfill H_s\left(Y\right)=s·\sum _{p\le N}\frac{\text{ln}p}{p^s}\underset{s\to 1}{\to }H\left(Y\right).\end{array}$$ (50)

In contrast, P will not converge to a finite measure as ζ(s) has a pole at s = 1.

Given the fact that the Prime Encoding produces a bit–string that is algorithmically random (at least, asymptotically), the expected True Positive Rate (TPR) for inferring the first N numbers as primes or composites is on the order of

$$\begin{array}{c}\hfill \mathbb{E}\left[TPR\right]\sim \frac{1}{N}\sum _{k=1}^{N}P(X=k)=\frac{1}{N}\sum _{k=1}^N\frac{1}{n}=\frac{\text{ln}N}{N},\end{array}$$ (51)

and thus tends to 0 as N becomes large. Our numerical experiments corroborate the fact that the TPR of a machine learning model is indeed small and, moreover, that it diminishes as N grows.

Another point worthy discussion is the independence of primes Y₁, …, Y_π(N). This assumption allows Shannon’s source coding theorem into play, however it does not seem to fully hold (even theoretically). Indeed, a Turing machine that enumerates any finite initial segment of primes exists. Arguably, since K(n)∼log₂ n for most numbers, we may write the following upper bound for the complexity of primes up to 2^N

$$\begin{array}{c}\hfill K\left(\pi \left(2^N\right)\right)+O\left(1\right)\sim {\text{log}}_2\left(\frac{2^N}{N\text{ln}2}\right)+O\left(1\right)\sim N-{\text{log}}_{2}N+O\left(1\right).\end{array}$$ (52)

Thus, the primes are somewhat compressible, as their complexity is not outright N − O(1). However, our result about the expected Kolmogorov complexity of primes holds in the sense of asymptotic equivalence, and N − log₂ N ∼ N in the large N limit.

Numerical experiments

We posit that a machine learning model may not be reliably used to predict the locations of prime numbers. Indeed, X being algorithmically random means that no prediction can be reasonably inferred for it by using inductive learning.

Previous experiments on prime number inference using deep learning were done in [10], and showed a very low true positive rate (∼ 10⁻³). The neural network had a three–layer architecture, and no specific training was performed. Modern tree–based classifiers are approximating the Kolmogorov complexity very efficiently by using Huffman encoding or a more advanced variant of thereof.

For example, XGBoost often outperforms other models in Kaggle competitions, especially on tabular data and in classification tasks. Thus, we may take XGBoost as a more practical experimental model.

We performed XGBoost experiments on prime learning for N–bit integers with N = 18 and N = 24. Each integer was represented as a binary string of length N with leading zeroes where appropriate. The code used for our experiments is available on GitHub [6], and the computations may be reproduced on a GPU–equipped laptop, e.g. MacBook Pro M1 with 8 GB memory.

For N = 18, there are 23′000 primes and 239′144 composites out of 262′144 numbers in total. We have the following probability confusion matrix:

$$\begin{array}{cc}\begin{array}{c}\end{array}& \begin{array}{cc}Composite& Prime\end{array}\\ \begin{array}{c}Composite\\ Prime\end{array}& \begin{array}{cc}0.638886& 0.273278\\ 0.058209& 0.029628\end{array}\end{array}$$

For N = 24, there are 1′077′871 primes and 15′699′345 composites out of 16′777′216 numbers in total. The probability confusion matrix turns out

$$\begin{array}{cc}\begin{array}{c}\end{array}& \begin{array}{cc}Composite& Prime\end{array}\\ \begin{array}{c}Composite\\ Prime\end{array}& \begin{array}{cc}0.635707& 0.300054\\ 0.041912& 0.022326\end{array}\end{array}$$

In both cases we used Bayesian hyper–parameter optimization with HyperOpt, and 80% / 20% train/test split. This achieves a better (by an order of magnitude) true positive rate than in [10], which is still insignificant. In fact, as follows from the above confusion matrices, the true positive rate declines as N grows. All this is expected given our theoretical analysis, and our experiments corroborate the theoretical conclusion that “machine learning” primes turns out no better than guessing a random number.

As shown in [11], the recent experiments with Neural Network classifiers for semi–primes largely fail to infer the semi–prime distribution. More precisely, for N = 426 detecting semi–primes with N–bit factors has the number of false negatives on par with the number of true positives (both on the order of ∼0.25). It is worth mentioning that the number of false positives (∼0.05) is relatively small. However, the problem considered in [11] is substantially different from ours.

Data Availability

Data are available at: https://github.com/sashakolpakov/xgb-primes.

Funding Statement

This study was supported by the Swiss National Science Foundation project PP00P2--202667 awarded to A.K. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.