Identifiability for Gaussian Processes with Holomorphic Kernels

Abstract

Gaussian processes (GPs) are widely recognized for their robustness and flexibility across various domains, including machine learning, analysis of time series, spatial statistics, and biomedicine. In addition to their common usage in regression tasks, GP kernel parameters are frequently interpreted in various applications. For example, in spatial transcriptomics, estimated kernel parameters are used to identify spatial variable genes, which exhibit significant expression patterns across different tissue locations. However, before these parameters can be meaningfully interpreted, it is essential to establish their identifiability. Existing studies of GP parameter identifiability have focused primarily on Matérn-type kernels, as their spectral densities allow for more established mathematical tools. In many real-world applications, particuarly in time series analysis, other kernels such as the squared exponential, periodic, and rational quadratic kernels, as well as their combinations, are also widely used. These kernels share the property of being holomorphic around zero, and their parameter identifiability remains underexplored. In this paper, we bridge this gap by developing a novel theoretical framework for determining kernel parameter identifiability for kernels holomorphic near zero. Our findings enable practitioners to determine which parameters are identifiable in both existing and newly constructed kernels, supporting application-specific interpretation of the identifiable parameters, and highlighting non-identifiable parameters that require careful interpretation.

1. Introduction

Gaussian Processes (GPs) are powerful and flexible tools extensively used across multiple fields, such as machine learning (ML), geospatial and spatiotemporal analysis, biomedicine, finance, and environmental modeling (Rasmussen and Williams, 2006; Banerjee et al., 2014; Cressie and Wikle, 2015). They serve various purposes: as regression or classification methods through GP regression or GP classification; as priors over functions in Bayesian inference (Ghosal and van der Vaart, 2017); for modeling latent distributions via Gaussian Process Latent Variable Models (GPLVM, Lawrence (2003)); and in demonstrating equivalencies to deep neural networks with infinite width (Lee et al., 2018). The flexibility of GPs as universal approximators, their inherent interpretability – especially regarding kernel parameters – and their capability to quantify uncertainty, are among their key advantages.

The kernel function, also known as the covariance function or covariogram, which defines the covariance structure within a GP, is pivotal to application and effectiveness. Over recent decades, there has been a proliferation of research into developing specialized kernels tailored for specific data types including time-series, spatial, imaging, and spatiotemporal datasets. Popular choices such as the squared exponential (SE, also known as RBF or Gaussian), rational quadratic (RQ), periodic (Per), and Matérn kernels are frequently employed, often in innovative combinations that enhance model performance (Wang et al., 2018). These combinations involve operations such as summation, multiplication, and spectral mixtures (Duvenaud et al., 2011; 2013; Kronberger and Kommenda, 2013; Wilson and Adams, 2013; Samo and Roberts, 2015; Remes et al., 2017; Cheng et al., 2019; Verma and Engelhardt, 2020), which enable the leveraging of individual kernel strengths to better capture complex data patterns.

Despite the extensive literature on GP theory and its application to regression or prediction tasks, less attention has been paid to the parameter inference, particularly the identifiability and interpretability of kernel parameters. Parameter inference is critical in applications that use estimated parameters in downstream tasks such as model comparison and problem-specific parameter interpretation.

One such application is in the study of spatial transcriptomics, which measures gene expression across different tissue locations to understand cellular and tissue-level biological processes (Marx, 2021). One important task within this field is identifying spatially variable genes (SVGs), which are genes that show significant changes in expression patterns across spatial locations, among tens of thousands of genes. Svensson et al. (2018) models gene expressions as a GP across spatial coordinates using a SE kernel with nuggets: $K\left(x,x^{\prime }\right)={\sigma }_s^2\exp \left(-\frac{{‖x-x^{\prime }‖}^2}{2{\ell }^2}\right)+{\sigma }_e^2{1}_{\left\{x=x^{\prime }\right\}}$. The kernel parameter ${\sigma }_s^2$ was then interpreted as the magnitude of the spatial effects to identify SVGs, estimated by the Maximum Likelihood Estimator (MLE). Other applications of GP parameter inference to spatial transcriptomics include Weber et al. (2023) and Sun et al. (2020).

Another example where kernel parameter estimates are interpreted is the decomposition of the Mauna Loa CO₂ time series data (Tans and Keeling, 2023) into four kernel components in the impactful book Rasmussen and Williams (2006):

$$\begin{gathered} K\left(x,x^{\prime }\right)=K_1\left(x,x^{\prime }\right)+K_2\left(x,x^{\prime }\right)+K_3\left(x,x^{\prime }\right)+K_4\left(x,x^{\prime }\right) \\ ={\theta }_1^2\exp \left(-\frac{{\left(x-x^{\prime }\right)}^2}{2{\theta }_2^2}\right)+{\theta }_3^2\exp \left(-\frac{{\left(x-x^{\prime }\right)}^2}{2{\theta }_4^2}-\frac{2{\sin }^2\left(\pi \left(x-x^{\prime }\right)\right)}{{\theta }_5^2}\right) \\ +{\theta }_6^2{\left(1+\frac{{\left(x-x^{\prime }\right)}^2}{2{\theta }_8{\theta }_7^2}\right)}^{-{\theta }_8}+{\theta }_9^2\exp \left(-\frac{{\left(x-x^{\prime }\right)}^2}{2{\theta }_{10}^2}\right)+{\theta }_{11}^2{1}_{\left\{x=x^{\prime }\right\}} \end{gathered}$$ (1)

where $K_1\left(x,x^{\prime }\right)$ is a SE kernel that captures the long-term smooth rising trend, $K_2\left(x,x^{\prime }\right)$, called the damped periodic kernel, is a multiplication of a SE kernel and a periodic kernel that accounts for seasonal variations, $K_3\left(x,x^{\prime }\right)$ is a rational quadratic kernel that models medium-term irregularities, and $K_4\left(x,x^{\prime }\right)$ is a sum of a SE and a white noise kernel that measure correlated and independent noise respectively. This kernel is also used as an example in the tutorial of the widely used Python package “sklearn.gaussian_process”, with detailed interpretation of all 11 parameters proposed by the authors in Section 3. Although the interpretation seems reasonable, a theoretical understanding with a rigorous proof is missing.

Although parameter identifiability in a GP model might seem straightforward at first glance, it is a challenging and nuanced problem. In fact, not all parameters in widely used GP kernels are identifiable: if a parameter is not identifiable, consistent estimation and subsequent interpretation are impossible. For example, for the Matérn kernel in dimension $p\le 3$ with spatial variance ${\sigma }^2$, lengthscale $\ell$, and known smoothness parameter $\nu$, Zhang (2004) proved that neither ${\sigma }^2$ nor $\ell$ is identifiable or consistently estimable, no matter how sophisticated the estimator is. In fact, the only identifiable parameter in the Matérn kernel, termed the microergodic parameter, is ${\sigma }^2{\ell }^{-2\nu }$. Follow-up studies for a single Matérn kernel include Anderes (2010); Kaufman and Shaby (2013); Li (2022); Li et al. (2023), and Chen et al. (2024) for a linear combination of Matérns with different smoothness. Such negative results raise a natural question: are all parameters used in practice, including those in Equation (1), identifiable so that their interpretations are justified?

As far as we are aware, identifiability of the parameters in Equation (1) has not been proven before. More importantly, there is still a gap in the literature for more complicated kernel combinations like those popularized in ML, especially when the combinations involve periodic kernels. The lack of theoretical examination is partly due to the failure of traditional methods used to study GP parameter identifiability, such as the integral test (Stein, 1999), which requires conditions on the spectral density not met by common kernels like SE, Per, and RQ, and even more so when these kernels are combined. This necessitates the development of new analytic tools to better understand kernel parameter identifiability and interpretability.

Motivated by these observations and challenges, this paper proves a general theorem (Theorem 3.4) that determines all the identifiable functions of the parameters in any family of stationary kernels holomorphic around 0. The result applies to complex combinations of kernels, particularly those common in the ML community, such as the one used in Equation (1). We demonstrate that all parameters in this kernel are identifiable under mild constraints, supporting the interpretation of kernel parameters in Rasmussen and Williams (2006) and the “sklearn.gaussian_process” Python package tutorial. Additionally, we establish a general result that is used to determine the identifiable functions of the parameters for a kernel that is a sum of products of other kernels.

The paper is organized as follows. Section 2 provides a comprehensive background, introduces the necessary notation and concepts, and reviews the relevant literature. Section 3 presents our main theoretical contributions. Section 4 contains simulation studies to support our theories, followed by Section 5 with a discussion of limitations and future work. A brief discussion of the connection between parameter identifiability and prediction is given in Appendix C.

2. Background

This section defines key concepts and notations and summarizes existing literature on GP kernel identifiability and interpretability. We begin with the definition of GPs.

2.1. Gaussian process

Definition 1 (GP).

A stochastic process $Z$ is said to follow a GP in domain $\text{Ω}$ with a mean function $\mu :\text{Ω}\to ℝ$ and a positive definite covariance/kernel/covariogram function $K:\text{Ω}\times \text{Ω}\to ℝ$ if for all $x_1,\cdots ,x_n\in \text{Ω}$,

$$\left[Z\left(x_1,\right),\cdots ,Z\left(x_n\right)\right]\sim N(v,\text{Σ}),v=\left[\mu \left(x_1\right),\cdots ,\mu \left(x_n\right)\right],{\text{Σ}}_{ij}=K\left(x_i,x_j\right).$$

For our study, as well as presentation simplicity, we assume $\mu =0$, without loss of generality (Stein, 1999). In this situation, since the distribution of $Z$ is completely determined by $K$, we sometimes call $K$ a GP, which refers to a GP with covariance kernel $K$.

Throughout this paper, we focus on the infill domain (also known as fixed domain or interpolation), i.e., the domain $\text{Ω}={[0,T]}^p$ does not grow with sample size, a situation commonly considered in the literature (Stein, 1999). Next, we introduce the commonly accepted stationarity assumption:

Definition 2 (Stationarity).

$K$ is called stationary if $K\left(x,x^{\prime }\right)=K\left(x+h,x^{\prime }+h\right)$, $\forall h\in {ℝ}^p$.

For a stationary kernel, we can reformulate the kernel to a function on ${ℝ}^p$ instead of ${ℝ}^p\times {ℝ}^p$ by $K_0(x):=K(x,0)$. Stationarity is a common assumption in GP literature due to its satisfactory practical performance and simplicity in both implementation and theoretical analysis. Throughout this paper, we focus only on stationary kernels, and still denote the simplified kernel $K$ without causing any confusion.

2.2. Kernels

We first note that all kernel functions considered in this paper are continuous functions unless noted otherwise. Then we introduce the following commonly used kernels in Table 1.

Table 1:

Example kernels, parameters, and domain dimension

Name	$K(x)$	Parameters	Dimension
SE	${\sigma }^2\exp \left(-\frac{\Vert x{\Vert }^2}{2{\ell }^2}\right)$	${\sigma }^2$, $\ell >0$	$p\ge 1$
Per	${\sigma }^2\exp \left(-\frac{2{\sin }^2\left(\frac{\pi x}{\gamma }\right)}{{\ell }^2}\right)$	${\sigma }^2$, $\ell$, $\gamma >0$	$p=1$
RQ	${\sigma }^2{\left(1+\frac{\Vert x{\Vert }^2}{2\alpha {\ell }^2}\right)}^{-\alpha }$	${\sigma }^2$, $\ell$, $\alpha >0$	$p\ge 1$
Matérn	${\sigma }^2\frac{2^{1-\nu }}{\text{Γ}(\nu )}{\left(\sqrt{2\nu }\frac{\Vert x\Vert }{\ell }\right)}^{\nu }{K}_{\nu }\left(\sqrt{2\nu }\frac{\Vert x\Vert }{\ell }\right)$	${\sigma }^2$, $\ell$, $\nu >0$	$p\ge 1$

In this table, ${\sigma }^2$ is called the spatial variance, or partial sill, which measures point-wise variance; $\ell$ is called the length scale that measures the spatial dependency; $\gamma >0$ is the period parameter; $\alpha$ is called the scale mixture parameter; $\nu$ is called the smoothness parameter. Among them, Per is well-defined only when $p=1$, i.e., $\text{Ω}=[0,T]$ is a closed interval, while others are well-defined on ${ℝ}^p$ for any $p$.

Each individual kernel in the above table captures some unique behavior in the process $Z$. However, when the process has complicated structure, a common approach is to combine some of these kernels to create a new one. Such a combination can be simply a sum of products of these kernels, which is guaranteed to be a positive definite function.

The following example extends Equation (1) used in Rasmussen and Williams (2006) to study the Mauna Loa CO₂ time series data:

$$\begin{array}{l}{K}_{\theta }(x)={\theta }_1^2\exp \left(-\frac{x^2}{2{\theta }_2^2}\right)+{\theta }_3^2\exp \left(-\frac{x^2}{2{\theta }_4^2}-\frac{2{\sin }^2\left(\frac{\pi x}{\gamma }\right)}{{\theta }_5^2}\right)\\ +{\theta }_6^2{\left(1+\frac{x^2}{2{\theta }_8{\theta }_7^2}\right)}^{-{\theta }_8}+{\theta }_9^2\exp \left(-\frac{x^2}{2{\theta }_{10}^2}\right)+{\theta }_{11}{1}_{\{x=0\}},\end{array}$$ (2)

where $\theta =\left({\theta }_1,\cdots ,{\theta }_{11},\gamma \right)$ be the vector of all parameters in the above kernel.

Note that this kernel is more flexible than the one in Equation (1), which assumes the period $\gamma =1$. We adopt this more challenging modification since the period is sometimes unknown in practice so practitioners have to estimate it from the data.

2.3. Identifiability

The study of identifiability of GP kernel parameters relies on the notation of equivalence of measures defined below:

Definition 3 (Equivalence of measures).

Two measures $P_1$ and $P_2$ are said to be equivalent if they are absolutely continuous with respect to each other, denoted by $P_1\equiv P_2$. That is, $P_1(A)=0\iff P_2(A)=0$. Two measures are said to be orthogonal, denoted by $P_1\perp P_2$ if there exists a measurable set $A$ such that $P_1(A)=0$ but $P_2(A)=1$.

Two GP laws $P_1$ and $P_2$ are either equivalent, or are orthogonal (Feldman, 1958), which means they assign probability 1 to disjoint sets: $P_1\left(A^c\right)=1$ and $P_2(A)=1$. We define the identifiability of GP parameters as follows:

Definition 4 (Microergodicity).

Let ${\left(K_{\theta }\right)}_{\theta \in \text{Θ}}$ be a family of covariance kernels of a GP. Then a function $h=h(\theta )$ of $\theta$ is said to be microergodic if $K_{{\theta }_1}\equiv K_{{\theta }_2}\iff h\left({\theta }_1\right)=h\left({\theta }_2\right)$.

If $h$ and $\tilde{h}$ are both microergodic, then $h\left({\theta }_1\right)=h\left({\theta }_2\right)\iff \tilde{h}\left({\theta }_1\right)=\tilde{h}\left({\theta }_2\right)$, so $h$ and $\tilde{h}$ are related by a bijection. Thus the migroergodic function is unique up to a bijective transformation, and it makes sense to speak of ‘the’ microergodic function $h$.

Definition 5 (Identifiability).

Let ${\left(K_{\theta }\right)}_{\theta \in \text{Θ}}$ be a family of covariance kernels of a GP. A function $g=g(\theta )$ of $\theta$ is said to be identifiable if $K_{{\theta }_1}\equiv K_{{\theta }_2}\Rightarrow g\left({\theta }_1\right)=g\left({\theta }_2\right)$, or equivalently, $g$ is a function of the microergodic function $h$. We say that the family ${\left(K_{\theta }\right)}_{\theta \in \text{Θ}}$ is identifiable if $\theta$ is identifiable.

Note that a consistent estimator of $g(\theta )$ can exist only when $g(\theta )$ is identifiable – when $g(\theta )$ is not identifiable, say $K_{{\theta }_1}\equiv K_{{\theta }_2}$ with $g\left({\theta }_1\right)\ne g\left({\theta }_2\right)$, it is not possible to find a consistent estimator of $g(\theta )$, since there is no way to distinguish between data generated from $K_{{\theta }_1}$ and those from $K_{{\theta }_2}$ almost surely (see Stein (1999); Zhang (2004) for more detailed discussion). Thus anything that can be consistently estimated is identifiable. The microergodic function $h(\theta )$ is the maximal identifiable function, so knowing the microergodic function completely solves the identifiability problem for the family of kernels. However, in some cases, it is difficult to fully determine the microergodic function $h(\theta )$, whereas it is easier to determine that some specific function $g(\theta )$ is identifiable.

To study the identifiability of GP kernel parameters, it suffices to determine when two GPs in the same parametric family are equivalent. However, to determine whether two GPs are equivalent is not an easy task, and the methods for doing so highly depend on the form of the kernels. There is a rich literature focusing on identifiability for Matérn kernels, where it has been shown that

$$K_{{\sigma }_1^2,{\ell }_1,{\nu }_1}\equiv K_{{\sigma }_2^2,{\ell }_2,{\nu }_2}\iff \left\{\begin{array}{ll}\left({\sigma }_1^2{\ell }_1^{-2{\nu }_1},{\nu }_1\right)=\left({\sigma }_2^2{\ell }_2^{-2{\nu }_2},{\nu }_2\right)\hfill & 1\le p\le 3,\hfill \\ \left({\sigma }_1^2,{\ell }_1,{\nu }_1\right)=\left({\sigma }_2,{\ell }_2,{\nu }_2\right)\hfill & p\ge 5.\hfill \end{array}\right.$$

That is, when the domain dimension is greater than or equal to 5, then $h\left({\sigma }^2,\ell ,\nu \right)=\left({\sigma }^2,\ell ,\nu \right)$, so all three parameters are identifiable (Anderes, 2010; Bolin and Kirchner, 2023); when the domain dimension is less than or equal to 3, then $h\left({\sigma }^2,\ell ,\nu \right)=\left({\sigma }^2{\ell }^{-2\nu },\nu \right):\nu$ is identifiable (Loh et al., 2021), but not ${\sigma }^2$ or $\ell$ (Zhang, 2004). As a result, there is no consistent estimator of ${\sigma }^2$ or $\ell$, but instead, a consistent estimator of ${\sigma }^2{\ell }^{-2\nu }$, called the microergodic parameter, does exist (Kaufman and Shaby, 2013; Loh et al., 2021), namely the MLE. The microergodic function for $p=4$ is an open problem.

Although the identifiability of Matérn has been understood, the study of other kernels including Per and RQ is much sparser. The key reason is that the tool to study equivalence between Matérn kernels, known as the integral test (Stein, 1999), requires strong conditions on the spectral densities of the kernel, which are not often satisfied by other kernels. The spectral density is defined below:

Definition 6 (Spectral measure).

For a stationary kernel $K$, its spectral measure, denoted by $F$, is defined through

$$K(x)=\int e^{i{\omega }^{\top }x}F(d\omega ).$$

Bochner’s theorem guarantees the existence and uniqueness of $F$. The density of $F$ w.r.t. the Lebesgue measure $d\omega$, denoted by $f$, if it exists, is called the spectral density.

The condition to use the integral test is that $f(\omega )\Vert \omega {\Vert }^{\alpha }\asymp 1$ as $\omega \to \infty$ for some $\alpha >0$. That is, the spectral density is required to behave like $\Vert \omega {\Vert }^{-\alpha }$ for some positive $\alpha >0$. The spectral density of Matérn is $f(\omega )={\sigma }^2\frac{2^p{\pi }^{-p/2}\text{Γ}\left(\nu +\frac{p}{2}\right){(2\nu )}^{\nu }}{\text{Γ}(\nu ){\ell }^{2\nu }}{\left(\frac{2\nu }{{\ell }^2}+\Vert \omega {\Vert }^2\right)}^{-\left(\nu +\frac{p}{2}\right)}\asymp \Vert \omega {\Vert }^{-2\nu -p}$ (Rasmussen and Williams, 2006, p. 84) (note that we use a different Fourier transform convention than (Rasmussen and Williams, 2006).) However, this condition is not met by RBF, Per, or RQ, as their spectral densities decay very rapidly due to the infinite differentiability of the kernels.

Due to the popularity of these kernels in ML, we aim to address these challenges and study the equivalence of GPs with these kernels and their combinations. The next section provides theoretical support for the success of these kernels in terms of identifiability and interpretability.

3. Theory

In this section, we present our main theory regarding equivalence of GPs, as outlined in the previous sections. We first determine the identifiable parameters of the individual kernels used in Equation (2), i.e., SE, Per, and RQ, with some extensions.

Theorem 3.1.

The microergodic functions of 5 individual kernels in Table 1, including all four components $K_1$, $K_2$, $K_3$, $K_4$ in Equation (2) and an additional kernel, Cosine, are summarized in Table 2.

Table 2:

Microergodic functions of five kernels

Name	$K(x)$	Parameters	Microergodicity	$p$
SE	${\sigma }^2\exp \left(-\frac{\Vert x{\Vert }^2}{2{\ell }^2}\right)$	${\sigma }^2$, $\ell >0$	$\left({\sigma }^2,\ell \right)$	≥ 1
Per	${\sigma }^2\exp \left(-\frac{2{\sin }^2\left(\frac{\pi x}{\gamma }\right)}{{\ell }^2}\right)$	${\sigma }^2$, $\ell$, $\gamma >0$	$\left({\sigma }^2,\ell ,\gamma \right)$	1
Damped Per	${\sigma }^2\exp \left(-\frac{x^2}{2{\ell }_1^2}-\frac{2{\sin }^2\left(\frac{\pi x}{\gamma }\right)}{{\ell }_2^2}\right)$	${\sigma }^2$, ${\ell }_1$, ${\ell }_2$, $\gamma >0$	$\left({\sigma }^2,{\ell }_1,{\ell }_2,\gamma \right)$	1
RQ	${\sigma }^2{\left(1+\frac{\Vert x{\Vert }^2}{2\alpha {\ell }^2}\right)}^{-\alpha }$	${\sigma }^2$, $\alpha$, $\ell >0$	$\left({\sigma }^2,\alpha ,\ell \right)$	≥ 1
Cosine	${\sigma }^2\cos \left(s^{\top }x\right)$	${\sigma }^2$, $s_1>0$, $s\in {ℝ}^p$	$s$	≥1

Theorem 3.1 supports the identifiability and interpretability of each kernel parameter in SE, Per and RQ, as discussed in Section 2.2. In addition, we include the cosine kernel, which will be revisited later in this section.

Then we consider the combination of SE, PER and RQ in Equation (2), an extension of the kernel used by the impactful book Rasmussen and Williams (2006) and the tutorial of the widely used Python package “sklearn.gaussian_process”.

Theorem 3.2.

All parameters in Equation (2) are identifiable provided ${\theta }_{10}$, the length-scale of the SE component to model the correlated noise, is less than ${\theta }_2$, the length-scale of the SE component to model the long-term trend.

Such a constraint is necessary, and not surprising, since otherwise, say, if ${\theta }_2={\theta }_{10}$, then we can merge the two SE components into a single SE: $\left({\theta }_1^2+{\theta }_9^2\right)\exp \left(-\frac{x^2}{2{\theta }_2^2}\right)$, making ${\theta }_1^2+{\theta }_9^2$ identifiable instead of ${\theta }_1^2$ and ${\theta }_9^2$. This distinction of two SE components is also discussed in Section 5.4.3 in Rasmussen and Williams (2006). Excluding this trivial case, all parameters are identifiable. As a consequence, these parameters are interpretable, as discussed in the same section in Rasmussen and Williams (2006). For example, ${\theta }_1$ measures the amplitude and ${\theta }_2$ measures the characteristic length-scale of the long-term smooth rising trend; within the seasonal trend, ${\theta }_3$ gives the magnitude, ${\theta }_4$ gives the decay time for the periodic component, $\gamma$ gives the period, while ${\theta }_5$ is the smoothness of the periodic component; for the (small) medium term irregularities, ${\theta }_6$ is the magnitude, ${\theta }_7$ is the typical length-scale and ${\theta }_8$ is the shape parameter determining diffuseness of the length-scales; ${\theta }_9$ is the magnitude of the correlated noise component, ${\theta }_{10}$ is its lengthscale and ${\theta }_{11}$ is the magnitude of the independent noise component.

Now we would like to answer the following more challenging question with a broader implication: Given a new kernel, how do we determine the microergodic function? Specifically, if we combine a finite number of kernels, such as those in Table 2, by finite multiplication and addition like Equation (2), what is the microergodic function of the resulting kernel? To answer these questions, we need to introduce the following notions first.

Lemma 3.3 (Kernel decomposition).

For any stationary kernel $K$, $K$ can be uniquely decomposed as $K=K^c+K^d$, where $K^c$ is a kernel with continuous spectral measure and $K^d$ is another kernel with discrete spectral measure.

A direct consequence is that if $K$ admits a spectral density, then $K=K^c$; while if $K$ is periodic, then $K=K^d$. We call $K^c$ the continuous component and $K^d$ the discrete component. Note that this notion is different from continuous and discrete functions, and we do assume all kernels are continuous functions themselves. Here the continuous and discrete notion is at the spectrum level. For example, the Per kernel, is continuous as a function, but has a purely discrete spectrum. Moreover, we denote the spectral measure of $K^c$ as $F^c$ and the spectral measure of $K^c$ as $F^d$, where $F=F^c+F^d$ is the spectral measure of $K$.

Such a decomposition offers deeper insights to understand different types of kernels. Moreover, to understand the equivalence of GPs, it suffices to understand the equivalence of its continuous component and discrete component separately, given by the following key theorem, which is the main result of the paper:

Theorem 3.4.

Given two kernels $K_1$ and $K_2$ with $K_1$ holomorphic on some ball around 0 in ${ℂ}^p$, the $p$-dimensional complex space, then $K_1\equiv K_2$ if and only if the following two conditions hold:

1. $K_1^c(x)=K_2^c(x)$ for every $x\in {ℝ}^p$.

2. There are $c$, $C>0$ such that $c{F}_1^d(\{\omega \})\le F_2^d(\{\omega \})\le C{F}_1^d(\{\omega \})$ for all $\omega \in {ℝ}^p$ and $\sum _{\omega :F_1^d(\{\omega \})>0}{\left(1-\frac{F_2^d(\{\omega \})}{F_1^d(\{\omega \})}\right)}^2<\infty$.

Note that $K:{ℝ}^p\to ℝ$ is said to be holomorphic on a ball around 0 in ${ℂ}^p$, if it has a holomorphic extension $\tilde{K}$ to some ball $B\subset {ℂ}^p$ around 0, such that $\tilde{K}=K$ on $B\cap {ℝ}^p$. While being holomorphic on a ball around 0 is a stronger condition than being infinitely differentiable, most infinitely differentiable kernels used in practice, including all those in Table 2, are holomorphic on a ball around 0. Condition 1 means the continuous components of $F_1$ and $F_2$ are the same, while Condition 2 means the discrete components of $F_1$ and $F_2$ have the same support, and their relative difference, although allowed to be nonzero, should decay fast enough.

Notably, Theorem 3.4 provides a general pipeline to study the identifiability of kernel parameters, summarized in the following theorem:

Theorem 3.5.

Let ${\left(K_{\theta }\right)}_{\theta \in \Theta }$ be a family of stationary kernels on ${ℝ}^p$, each of which is holomorphic on some ball around 0 in ${ℂ}^p$. We have the following assertions regarding the microergodic function:

1. If $h(\theta )$ is microergodic for the continuous component ${\left(K_{\theta }^c\right)}_{\theta \in \Theta }$ and $g(\theta )$ is microergodic for the discrete component ${\left(K_{\theta }^d\right)}_{\theta \in \Theta }$, then $(h(\theta ),g(\theta ))$ is microergodic for ${\left(K_{\theta }\right)}_{\theta \in \Theta }$.

2. Moreover,

   1. $h(\theta )$ is microergodic for ${\left(K_{\theta }^c\right)}_{\theta \in \Theta }$ if and only if

      $$K_{{\theta }_1}^c(x)=K_{{\theta }_2}^c(x),\forall x\in {ℝ}^p\iff h\left({\theta }_1\right)=h\left({\theta }_2\right).$$
   2. $g(\theta )$ is microergodic for ${\left(K_{\theta }^d\right)}_{\theta \in \Theta }$ if and only if

      $$\left.\begin{array}{l}\exists c,C>0,s.t.c{F}_1^d(\{\omega \})\le F_2^d(\{\omega \})\le C{F}_1^d(\{\omega \}),\forall \omega \in {ℝ}^p\\ \sum _{\omega :F_{{\theta }_1}^d(\{\omega \})>0}{\left(1-\frac{F_{{\theta }_2}^d(\{\omega \})}{F_{{\theta }_1}^d(\{\omega \})}\right)}^2<\infty \end{array}\right\}\iff g\left({\theta }_1\right)=g\left({\theta }_2\right)$$

That is, in order to find the microergodic function of a parametric family of kernels $K_{\theta }$, it suffices to find the microergodic function $h\left(\theta \right)$ of the continuous component, and $g\left(\theta \right)$ of the discrete component separately. Moreover, to find $h\left(\theta \right)$, it suffices to understand when two continuous components are equal everywhere; to find $g\left(\theta \right)$, we need to investigate the conditions 2b about the discrete measure $F^d$.

Having established the foundational aspects of kernel identifiability, we now apply our results to determine the microergodic function of various combinations of kernels. Our general strategy is to use Fourier transform identities to compute the spectral measure of the combined kernel (see, for example, Theorem B.6) and then apply Theorem 3.4. These combinations not only illustrate the practical applications of our theoretical findings in Theorem 3.4, but also provide insights into designing new kernels with desired properties. We start with the squared exponential kernel with automatic relevance determination (ARD).

Theorem 3.6.

For the family

$$K_{\sigma ,M}(x)={\sigma }^2\exp \left(-\frac{1}{2}{x}^{T}Mx\right),$$

where ${\sigma }^2>0$ and $M$ is a positive-definite matrix, the microergodic function is $\left({\sigma }^2,M\right)$.

Next, we study the sum of cosine kernels:

Theorem 3.7.

For the family

$$K_{{\sigma }_1,\dots ,{\sigma }_m,s_1,\dots ,s_m}(x)={\sigma }_1^2\cos \left(s_{1}x\right)+{\sigma }_2^2\cos \left(s_{2}x\right)+\cdots +{\sigma }_m^2\cos \left(s_{m}x\right),$$

where ${\sigma }_1^2,\dots ,{\sigma }_m^2>0$ and $s_1,\dots ,s_m\in ℝ$, under the natural constraint $0\le s_1<s_2<\cdots <s_m$, the microergodic function is $\left(s_1,\dots ,s_m\right)$.

Theorem 3.7 shows that when cosine kernels are combined linearly, their individual frequencies (or periods) remain identifiable, provided they are distinct. This scenario often arises in signal processing where different periodic components need to be isolated and identified. Notably, the last kernel in Table 1 is a special case of the kernel in Theorem 3.7 with $m=1$.

Next, we study the product of Cosine kernels:

Theorem 3.8.

For the family

$$K_{\sigma ,s_1,\dots ,s_m}(x)={\sigma }^2\cos \left(s_{1}x\right)\cos \left(s_{2}x\right)\cdots \cos \left(s_{m}x\right),$$

where ${\sigma }^2>0$ and $s_1,\dots ,s_m\in ℝ$, under the natural constraint $0<s_1\le s_2\le \cdots \le s_m$, the microergodic function is $\left\{\pm s_1\pm s_2\pm \cdots \pm s_m\right\}:=\left\{a_1{s}_1+a_2{s}_2+\cdots +a_m{s}_m:a_1,a_2,\dots ,a_m\in \right.\{-1,1\}\}$. If $m=1,2,3$, then the mircoergodic function simplifies to $\left(s_1,\dots ,s_m\right)$.

In Theorem 3.8, when $m\ge 4$, we do not have identifiability of $\left(s_1,\dots ,s_m\right)$. For example, for $m=4$, when $\left(s_1,s_2,s_3,s_4\right)=(1,2,2,3)$ and $\left({\tilde{s}}_1,{\tilde{s}}_2,{\tilde{s}}_3,{\tilde{s}}_4\right)=(1,1,3,3)$, the values of the microergodic function coincide, that value being $\{0,\pm 2,\pm 4,\pm 6,\pm 8\}$. Theorem 3.8 shows that for a product of discrete spectrum kernels that are all a function of the same variable $x$, the parameters of each individual kernel may not be identifiable.

Finally, we explore the sum of periodic kernels as previously discussed.

Theorem 3.9.

Let $K_{\ell ,\gamma }$ denote the periodic kernel with variance parameter 1, length-scale $\ell$, and period $\gamma$. For the family

$$K_{{\sigma }_1,{\sigma }_2,{\gamma }_1,{\gamma }_2}(x)={\sigma }_1^2{K}_{{\ell }_1,{\gamma }_1}(x)+{\sigma }_2^2{K}_{{\ell }_2,{\gamma }_2}(x),$$

where ${\sigma }_1^2$, ${\sigma }_2^2$, ${\ell }_1$, ${\ell }_2>0$, ${\gamma }_1>{\gamma }_2>0$, the microergodic function is $\left({\sigma }_1^2,{\ell }_1,{\gamma }_1,{\sigma }_2^2,{\ell }_2,{\gamma }_2\right)$, that is, all parameters are identifiable.

This result is crucial for scenarios where multiple periodic processes operate at different scales or periods, as often encountered in geospatial, financial, and environmental data analysis.

4. Simulation

In this section, we provide empirical support to our theoretical results on kernel parameter identifiability, presented in Section 3, by investigating the behavior of the maximum likelihood estimators (MLEs) as the sample size $n$ increases.

Before moving to the simulation details, we would like to clarify the broader picture of parameter inference for GPs, which involves three steps: first, determining which parameters are identifiable; second, finding a consistent estimator of identifiable parameters, such as the MLEs or others estimators; and third, developing numerical methods to compute these estimators. While the second and the third steps are crucial, they fall beyond the scope of this paper, which focuses solely on the first step–a theoretical framework to find all the identifiable parameters. In fact, even for simple kernels like the SE and Matérn kernels, whether the MLE is consistent remains open (Loh and Sun, 2023).

Despite these complexities, we use standard optimization packages commonly applied in the GP literature to find the MLEs. Our simulations are not intended to solve the open problem of MLE consistency or introduce new numerical techniques; rather, they serve to illustrate the theoretical results on identifiability through practical examples.

We start from individual kernels, followed by the combination in Equation (2).

4.1. Individual kernels

We consider the individual kernels: SE, Damped Per (DPer), Per, RQ, and Cosine. For the cosine kernel, we parameterize in terms of the period $\gamma$ so that $s=\frac{2\pi }{\gamma }$. Input samples are generated by adding a unif $\left(-\frac{1}{4n},\frac{1}{4n}\right)$ random shift to $n$ evenly spaced points in $\left[\frac{1}{4n},1-\frac{1}{4n}\right]$, where $n\in \{500,1000,2000,5000\}$. After generating the outcomes by sampling a GP with the given kernel at the inputs, we added independent Gaussian noise from $N(0$, $\epsilon ),\epsilon =0.01$, to model measurement errors (see Section D of the appendix for the experiments repeated with $\epsilon =0.1$). All kernel parameters were estimated by MLEs, with 100 replicates for each kernel configuration to assess the convergence of the MLEs. The results are summarized in Figure 1. These boxplots demonstrate that the MLEs of all parameters except ${\sigma }^2$ in the cosine kernel appear consistent, as conjectured by their identifiability, proved in Theorem 3.1.

Figure 1:

Simulation results for various kernel types. Each subfigure shows the boxplots of MLEs for the corresponding kernel, with ground truth in horizontal dashed line.

Some of the MLE standard deviations appear to plateau for large $n$. One explanation for this is numerical limitations – for our squared exponential simulation, where ${\sigma }^2=1$ and $\ell =1/500$, the condition numbers of the covariance matrix of the observations are $6.82\cdot {10}^1$, $1.80\cdot {10}^8$, $1.89\cdot {10}^{19}$, and $3.37\cdot {10}^{24}$ for sample sizes 500, 1000, 2000, and 5000, respectively.

The failure of the MLE of ${\sigma }^2$ in the cosine kernel to converge is in agreement with the microergodicity of $\gamma$. In fact, if we treat $\gamma$ as known and let the noise variance $\epsilon$ decrease to 0, then since the covariance matrix has rank 2 for all $n\ge 2$, it can be shown that the MLE $\widehat{\sigma }(\epsilon )$ converges to $\widehat{\sigma }(0)~{\sigma }^2\frac{{\chi }_2^2}{2}$.

4.2. The combined kernel in Equation (2)

Then, we study the combined kernel, one motivating kernel of this paper, defined in Equation (2). Since the kernel was proposed for forecasting CO₂level on the Mona Loa dataset, we set the time interval to be [0, 45], presenting the time span of 45 years. Input samples are generated by adding a unif $\left(-\frac{45}{4n},\frac{45}{4n}\right)$ random shift to $n$ evenly spaced points in $\left[\frac{45}{4n},45-\frac{45}{4n}\right]$, where $n\in \{50,100,200,500\}$. All kernel parameters were estimated by MLEs, with 100 replicates to assess the convergence of the MLEs. Moreover, to further mimic this dataset, the ground truth parameters and noise variance ${\theta }_{11}^2$ are set to be the MLEs learned from running the “Gaussian process regression” package from the scikit-learn Python package. All truth parameters to be estimated are given by Table 3.

In Figure 2, we again observe that the MLEs generally are unbiased, but for some parameters, their variance does not strictly decrease with sample size. This is likely due to the relatively large number of parameters (10) compared to the small sample size of 500.

Figure 2:

MLEs of parameters in Equation (2), with ground truth in horizontal dashed line.

5. Discussion

This paper has introduced a novel analytical framework that advances the theory of identifiability of kernel parameters in GPs for a large class kernels, those holomorphic around 0. We have demonstrated that all the parameters in certain combinations of kernels, such as the example employed on the Mauna Loa CO₂ time series data, are indeed identifiable. This establishes a robust theoretical foundation for selecting or constructing GP kernels and determining the identifiable functions of the parameters in practical applications.

Looking ahead, several avenues of future research present themselves as particularly promising and interesting. First, while establishing the identifiability of kernel parameters is a critical step, it does not necessarily guarantee the consistency of the MLE. The analysis of MLEs is complicated due to the complex nature of the likelihood function involved, which is often multi-modal and difficult to handle. Second, extending our theoretical framework to encompass non-stationary kernels could enhance the flexibility of GPs in modeling data with evolving trends and dynamics. This area is notably challenging due to the current limitations in mathematical tools available, presenting a largely open problem in the field. Third, another intriguing direction for research involves extending our findings to infinitely differentiable kernels that are not holomorphic near 0, though most infinitely differentiable kernels used in applications are holomorphic near 0.