Spatial Multivariate Trees for Big Data Bayesian Regression

Abstract

High resolution geospatial data are challenging because standard geostatistical models based on Gaussian processes are known to not scale to large data sizes. While progress has been made towards methods that can be computed more efficiently, considerably less attention has been devoted to methods for large scale data that allow the description of complex relationships between several outcomes recorded at high resolutions by different sensors. Our Bayesian multivariate regression models based on spatial multivariate trees (SpamTrees) achieve scalability via conditional independence assumptions on latent random effects following a treed directed acyclic graph. Information-theoretic arguments and considerations on computational efficiency guide the construction of the tree and the related efficient sampling algorithms in imbalanced multivariate settings. In addition to simulated data examples, we illustrate SpamTrees using a large climate data set which combines satellite data with land-based station data. Software and source code are available on CRAN at https://CRAN.R-project.org/package=spamtree.

1. Introduction

It is increasingly common in the natural and social sciences to amass large quantities of geo-referenced data. Researchers seek to use these data to understand phenomena and make predictions via interpretable models that quantify uncertainty taking into account the spatial and temporal dimensions. Gaussian processes (GP) are flexible tools that can be used to characterize spatial and temporal variability and quantify uncertainty, and considerable attention has been devoted to developing GP-based methods that overcome their notoriously poor scalability to large data. The literature on scaling GPs to large scale is now extensive. We mention low-rank methods (Quiñonero-Candela and Rasmussen, 2005; Snelson and Ghahramani, 2007; Banerjee et al., 2008; Cressie and Johannesson, 2008); their extensions (Low et al., 2015; Ambikasaran et al., 2016; Huang and Sun, 2018; Geoga et al., 2020); methods that exploit special structure or simplify the representation of multidimensional inputs—for instance, a Toeplitz structure of the covariance matrix scales GPs to big time series data, and tensor products of scalable univariate kernels can be used for multidimensional inputs (Gilboa et al., 2015; Moran and Wheeler, 2020; Loper et al., 2020; Wu et al., 2021). These methods may be unavailable or perform poorly in geostatistical settings, which focus on small-dimensional inputs, i.e. the spatial coordinates plus time. In these scenarios, low-rank methods oversmooth the spatial surface (Banerjee et al., 2010), Toeplitz-like structures are typically absent, and so-called separable covariance functions obtained via tensor products poorly characterize spatial and temporal dependence. To overcome these hurdles, one can use covariance tapering and domain partitioning (Furrer et al., 2006; Kaufman et al., 2008; Sang and Huang, 2012; Stein, 2014; Katzfuss, 2017) or composite likelihood methods and sparse precison matrix approximations (Vecchia, 1988; Rue and Held, 2005; Eidsvik et al., 2014); refer to Sun et al. (2011), Banerjee (2017), Heaton et al. (2019) for reviews of scalable geostatistical methods.

Additional difficulties arise in multivariate (or multi-output) regression settings. Multivariate geostatistical data are commonly misaligned, i.e. observed at non-overlapping spatial locations (Gelfand et al., 2010). Figure 1 shows several variables measured at non-overlapping locations, with one measurement grid considerably sparser than the others. In these settings, replacing a multi-output regression with separate single-output models is a valid option for predicting outcomes at new locations. While single-output models may some-times perform equally well or even outperform multi-output models, they fail to characterize and estimate cross-dependences across outputs; testing the existence of such dependences may be scientifically more impactful than making predictions. This issue can be solved by modeling the outputs via latent spatial random effects thought of as a realization of an underlying multivariate GP and embedded in a larger hierarchical model.

Figure 1:

Observed data of Section 4.2. Missing outcomes are in magenta. GHCN data are much more sparsely observed compared to satellite imaging from MODIS.

Unfortunately, GP approximations that do not correspond to a valid stochastic process may inaccurately characterize uncertainty, as the models used for estimation and interpolation may not coincide. Rather than seeking approximations to the full GP, one can develop valid standalone spatial processes by introducing conditional independence across spatial locations as prescribed by a sparse directed acyclic graph (DAG). These models are advantageous because they lead to scalability by construction; in other words, posterior computing algorithms for these methods can be interpreted not only as approximate algorithms for the full GP, but also as exact algorithms for the standalone process.

This family of methods includes nearest-neighbor Gaussian processes, which limit dependence to a small number of neighboring locations (NNGP; Datta et al. 2016a,b), and block-NNGPs (Quiroz et al., 2019). There is a close relation between DAG structure and computational performance of NNGPs: some orderings may be associated to improved approximations (Guinness, 2018), and graph coloring algorithms (Molloy and Reed, 2002; Lewis, 2016) can be used for parallel Gibbs sampling. Inferring ordering or coloring can be problematic when data are in the millions, but these issues can be circumvented by forcing DAGs with known properties onto the data; in meshed GPs (MGPs; Peruzzi et al., 2020), patterned DAGs associated to domain tiling are associated to more efficient sampling of the latent effects. Alternative so-called multiscale or multiresolution methods correspond to DAGs with hierarchical node structures (trees), which are typically coupled with recursive domain partitioning; in this case, too, efficiencies follow from the properties of the chosen DAG. There is a rich literature on Gaussian processes and recursive partitioning, see e.g Ferreira and Lee (2007); Gramacy and Lee (2008); Fox and Dunson (2012); in geospatial contexts, in addition to the GMRF-based method of Nychka et al. (2015), multi-resolution approximations (MRA; Katzfuss, 2017) replace an orthogonal basis decomposition with approximations based on tapering or domain partitioning and also have a DAG interpretation (Katzfuss and Guinness, 2021).

Considerably less attention has been devoted to process-based methods that ensure scalability in multivariate contexts, with the goal of modeling the spatial and/or temporal variability of several variables jointly via flexible cross-covariance functions (Genton and Kleiber, 2015). When scalability of GP methods is achieved via reductions in the conditioning sets, including more distant locations is thought to aid in the estimation of unknown covariance parameters (Stein et al., 2004). However, the size of such sets may need to be reduced excessively when outcomes are not of very small dimension. One could restrict spatial coverage of the conditioning sets, but this works best when data are not misaligned, in which case all conditioning sets will include outcomes from all margins; this cannot be achieved for misaligned data, leading to pathological behavior. Alternatively, one can model the multivariate outcomes themselves as a DAG; however this may only work on a case-by-case basis. Similarly, recursive domain partitioning strategies work best for data that are measured uniformly in space as this guarantees similarly sized conditioning sets; on the contrary, recursive partitioning struggles in predicting the outcomes at large unobserved areas as they tend to be associated to the small conditioning sets making up the coarser scales or resolutions.

In this article, we solve these issues by introducing a Bayesian regression model that encodes spatial dependence as a latent spatial multivariate tree (SpamTree); conditional independence relations at the reference locations are governed by the branches in a treed DAG, whereas a map is used to assign all non-reference locations to leaf nodes of the same DAG. This assignment map controls the nature and the size of the conditioning sets at all locations; when severe restrictions on the reference set of locations become necessary due to data size, this map is used to improve estimation and predictions and overcome common issues in standard nearest-neighbor and recursive partition methods while maintaining the desirable recursive properties of treed DAGs. Unlike methods based on defining conditioning sets based solely on spatial proximity, SpamTrees scale to large data sets without excessive reduction of the conditioning sets. Furthermore, SpamTrees are less restrictive than methods based on recursive partitioning and can be built to guarantee similarly-sized conditioning sets at all locations.

The present work adds to the growing literature on spatial processes defined on DAGs by developing a method that targets efficient computations of Bayesian multivariate spatial regression models. SpamTrees share similarities with MRAs (Katzfuss, 2017); however, while MRAs are defined as a basis function expansion, they can be represented by a treed graph of a SpamTree with full “depth” as defined later (the DAG on the right of Figure 2), in univariate settings, and “response” models. All these restrictions are relaxed in this article. In considering spatial proximity to add “leaves” to our treed graph, our methodology also borrows from nearest-neighbor methods (Datta et al., 2016a). However, while we use spatial neighbors to populate the conditioning sets for non-reference locations, the same cannot be said about reference locations for which the treed graph is used instead. Our construction of the SpamTree process also borrows from MGPs on tessellated domains (Peruzzi et al., 2020); however, the treed DAG we consider here induces markedly different properties on the resulting spatial process owing to its recursive nature. Finally, a contribution of this article is in developing self-contained sampling algorithms which, based on the graphical model representation of the model, will not require any external libraries.

Figure 2:

Three SpamTrees on M = 4 levels with depths δ = 1 (left), δ = 3 (center), and δ = 4 (right). Nodes are represented by circles, with branches colored in brown and leaves in green.

The article builds SpamTrees as a standalone process based on a DAG representation in Section 2. A Gaussian base process is considered in Section 3 and the resulting properties outlined, along with sampling algorithms. Simulated data and real-world applications are in Section 4; we conclude with a discussion in Section 5. The Appendix provides more in-depth treatment of several topics and additional algorithms.

2. Spatial Multivariate Trees

Consider a spatial or spatiotemporal domain $\mathcal{D}$. With the temporal dimension, we have $\mathcal{D}\subset {\Re }^d\times \left[0,\infty \right)$, otherwise $\mathcal{D}\subset {\Re }^d$. A q-variate spatial process is defined as an uncountable set of random variables $\left\{\mathit{w}\left(\ell \right):\ell \in \mathcal{D}\right\}$, where $\mathit{w}\left(\ell \right)$ is a q × 1 random vector with elements $w_i\left(\ell \right)$ for i = 1, 2,…, q, paired with a probability law P defining the joint distribution of any finite sample from that set. Let $\left\{{\ell }_1,{\ell }_2,\dots ,{\ell }_{n_{\mathcal{L}}}\right\}=\mathcal{L}\subset \mathcal{D}$ be of size $n_{\mathcal{L}}$. The $n_{\mathcal{L}}q\times 1$ random vector ${\mathit{w}}_{\mathcal{L}}={\left(\mathit{w}{\left({\ell }_1\right)}^{\top },\mathit{w}{\left({\ell }_2\right)}^{\top },\dots \mathit{w}{\left({\ell }_{n_{\mathcal{L}}}\right)}^{\top }\right)}^{\top }$ has joint density $p\left({\mathit{w}}_{\mathcal{L}}\right)$. After choosing an arbitrary order of the locations, $p\left({\mathit{w}}_{\mathcal{L}}\right)={\prod }_{i=1}^{n_{\mathcal{L}}}p\left(\mathit{w}\left({\ell }_i\right)|\mathit{w}\left({\ell }_1\right),\dots ,\mathit{w}\left({\ell }_{i-1}\right)\right)$, where the conditioning set for each $\mathit{w}\left({\ell }_i\right)$ can be interpreted as the set of nodes that have a directed edge towards $\mathit{w}\left({\ell }_i\right)$ in a DAG. Some scalable spatial processes result from reductions in size of the conditioning sets, following one of several proposed strategies (Vecchia, 1988; Stein et al., 2004; Gramacy and Apley, 2015; Datta et al., 2016a; Katzfuss and Guinness, 2021; Peruzzi et al., 2020). Accordingly,

$$p\left({\mathit{w}}_{\mathcal{L}}\right)=\prod _{i=1}^{n_{\ell }}p\left(\mathit{w}\left({\ell }_i\right)|\mathit{w}\left(\text{Pa}\left[{\ell }_i\right]\right)\right),$$ (1)

where Pa[ℓ_i] is the set of spatial locations that correspond to directed edges pointing to ℓ_i in the DAG. If Pa[ℓ_i] is of size J or less for all $i=1,\dots ,n_{\mathcal{L}}$, then $\mathit{w}\left(\text{Pa}\left[{\ell }_i\right]\right)$ is of size J_q. Methods that rely on reducing the size of parent sets are thus negatively impacted by the dimension q of the multivariate outcome; if q is not very small, reducing the number of parent locations J may be insufficient for scalable computations. As an example, an NNGP model has Pa[ℓ_i] = N (ℓ_i), where N (·) maps a location in the spatial domain to its neighbor set. It is customary in practice to consider J_q = m ≤ 20 for accurate and scalable estimation and predictions in univariate settings, but this may be restrictive in some multivariate settings as one must reduce J to maintain similar computing times, possibly harming estimation and prediction accuracy.

We represent the ith component of the q × 1 vector $\mathit{w}\left(\ell \right)$ as $w\left(\ell ,{\xi }_i\right)$, where ${\xi }_i={\left({\xi }_{i1},\dots ,{\xi }_{ik}\right)}^{\top }\in \Xi$ for some k and Ξ serves as the k-dimensional latent spatial domain of variables. The q-variate process $\mathit{w}\left(\ell \right)$ is thus recast as $\left\{w\left(\ell ,\xi \right):\left(\ell ,\xi \right)\in \mathcal{D}\times \Xi \right\}$, with $\xi$ representing the latent location in the domain of variables. We can then write (1) as

$$p\left({\mathit{w}}_{{\mathcal{L}}^{\ast }}\right)=\prod _{i=1}^{n_{{\mathcal{L}}^{\ast }}}p\left(w\left({\ell }_i^{\ast }\right)|w\left(\text{Pa}\left[{\ell }_i^{\ast }\right]\right)\right),$$ (2)

where ${\mathcal{L}}^{\ast }={\left\{{\ell }_i^{\ast }\right\}}_{i=1}^{n_{{\ell }^{\ast }}}$, ${\ell }_i^{\ast }\in \mathcal{D}\times \Xi ={\mathcal{D}}^{\ast }$, and w(·) is a univariate process on the expanded domain ${\mathcal{D}}^{\ast }$. This representation is useful as it provides a clearer accounting of the assumed conditional independence structure of the process in a multivariate context.

2.1. Constructing Spatial Multivariate DAGs

We now introduce the necessary terminology and notation, which are the basis for later detailing of estimation and prediction algorithms involving SpamTrees. The specifics for building treed DAGs with user-specified depth are in Section 2.1.1, whereas Section 2.1.2 gives details on cherry picking and its use when outcomes are imbalanced and misaligned.

The three key components to build a SpamTree are (i) a treed DAG $\mathcal{G}$ with branches and leaves on M levels and with depth δ ≤ M; (ii) a reference set of locations $\mathcal{S}$; (iii) a cherry picking map. The graph is $\mathcal{G}=\left\{\mathit{V},\mathit{E}\right\}$ where the nodes are $\mathit{V}=\left\{{\mathit{v}}_1,\dots ,{\mathit{v}}_{m_V}\right\}=\mathit{A}\cup \mathit{B}$, $\mathit{A}\cap \mathit{B}=\cancel{0}$. We separate the nodes into reference A and non-reference B nodes, as this will aid in showing that SpamTrees lead to standalone spatial processes in Section 2.2. The reference or branch nodes are $\mathit{A}=\left\{{\mathit{a}}_1,\dots ,{\mathit{a}}_{m_A}\right\}={\mathit{A}}_0\cup {\mathit{A}}_1\cup \cdots \cup {\mathit{A}}_{M-1}$, where ${\mathit{A}}_i=\left\{{\mathit{a}}_{i,1},\dots ,{\mathit{a}}_{i,m_i}\right\}$ for all i = 0,…, M − 1 and with ${\mathit{A}}_i\cap {\mathit{A}}_j=\cancel{0}$ if $i\ne j$. The non-reference or leaf nodes are $\mathit{B}=\left\{{\mathit{b}}_1,\dots ,{\mathit{b}}_{m_B}\right\}$, $\mathit{A}\cap \mathit{B}=\cancel{0}$. We also denote V_r = A_r for r = 0,…, M – 1 and V_M = B. The edges are $\mathit{E}=\left\{\text{Pa}\left[\mathit{v}\right]\subset \mathit{V}:\mathit{v}\in \mathit{V}\right\}$ and similarly $\text{Ch}\left[\mathit{v}\right]=\left\{{\mathit{v}}^{\prime }\in \mathit{V}:\mathit{v}\in \text{Pa}\left[{\mathit{v}}^{\prime }\right]\right\}$. The reference set $\mathcal{S}$ is partitioned in M levels starting from zero, and each level is itself partitioned into reference subsets:$\mathcal{S}={\cup }_{r=0}^{M-1}{\mathcal{S}}_r={\cup }_{r=0}^{M-1}{\cup }_{i=1}^{m_i}{S}_{ri}$, where $S_{ri}\cap S_{r^{\prime }{i}^{\prime }}=\cancel{0}$ if $r\ne r^{\prime }$ or $i\ne i^{\prime }$ and its complement set of non-reference or other locations $\mathcal{U}={\mathcal{D}}^{\ast }\backslash \mathcal{S}$. The cherry picking map is $\eta :{\mathcal{D}}^{\ast }\to \mathit{V}$ and assigns a node (and therefore all the edges directed to it in $\mathcal{G}$) to any location in the domain, following a user-specified criterion.

2.1.1. Branches and Leaves

For a given M and a depth δ ≤ M , we impose a treed structure on $\mathcal{G}$ by assuming that if $\mathit{v}\in {\mathit{A}}_i$ and $i>M-\delta =M_{\delta }$ then there exists a sequence of nodes $\left\{{\mathit{v}}_{r_{M_{\delta }}},\dots ,{\mathit{v}}_{r_{i-1}}\right\}$ such that ${\mathit{v}}_{r_j}\in {\mathit{A}}_j$ for $j=M_{\delta },\dots ,i-1$ and $\text{Pa}\left[\mathit{v}\right]=\left\{{\mathit{v}}_{r_{M_{\delta }}},{\mathit{v}}_{r_1},\dots ,{\mathit{v}}_{r_{j-1}}\right\}$. If $i\le M-\delta =M_{\delta }$ then $\text{Pa}\left[\mathit{v}\right]=\left\{{\mathit{v}}_{i-1}\right\}$ with ${\mathit{v}}_{i-1}\in {\mathit{A}}_{i-1}$. A₀ is the tree root and is such that $\text{Pa}\left[{\mathit{v}}_0\right]=\cancel{0}$ for all ${\mathit{v}}_0\in {\mathit{A}}_0$. The depth δ determines the number of levels of $\mathcal{G}$ (from the top) across which the parent sets are nested. Choosing δ = 1 implies that all nodes have a single parent; choosing δ = M implies fully nested parent sets (i.e. if ${\mathit{v}}_i\in \text{Pa}\left[{\mathit{v}}_j\right]$ then $\text{Pa}\left[{\mathit{v}}_i\right]\subset \text{Pa}\left[{\mathit{v}}_j\right]$ for all ${\mathit{v}}_i,\hspace{0.33em}{\mathit{v}}_i\in \mathit{V}$. The m_i elements of ${\mathit{A}}_i$ are the branches at level i of $\mathcal{G}$ and they have i – M_δ parents if the current level i is above the depth level M_δ and 1 parent otherwise. We refer to terminal branches as nodes $\mathit{v}\in \mathit{A}$ such that $\text{Ch}\left[\mathit{v}\right]\subset \mathit{B}$. For all choices of δ, $\mathit{v}\in {\mathit{A}}_i$, ${\mathit{v}}^{\prime }\in {\mathit{A}}_j$ and $\mathit{v}\in \text{Pa}\left[{\mathit{v}}^{\prime }\right]$ implies i < j; this guarantees acyclicity.

As for the leaves, for all $\mathit{v}\in \mathit{B}$ we assume $\text{Pa}\left[\mathit{v}\right]=\left\{{\mathit{v}}_{r_{M_{\delta }}},\dots ,{\mathit{v}}_{r_k}\right\}$ for some integer sequence $\left\{r_{M_{\delta }},\dots ,r_k\right\}$ and ${\mathit{v}}_{r_i}\in {\mathit{A}}_i$ with $i\ge M_{\delta }$. We allow the existence of multiple leaves with the same parent set, i.e. there can be k and ${\mathit{b}}_{i_1},\dots ,{\mathit{b}}_{i_k}$ such that for all i₂,…, i_k,$\text{Pa}\left[{\mathit{b}}_{i_h}\right]=\text{Pa}\left[{\mathit{b}}_{i_1}\right]$. Acyclicity of $\mathcal{G}$ is maintained as leaves are assumed to have no children. Figure 2 represents the graph associated to SpamTrees with different depths.

2.1.2. Cherry Picking Via η(·)

The link between $\mathcal{G}$, $\mathcal{S}$ and $\mathcal{U}$ is established via the map $\eta :{\mathcal{D}}^{\ast }\to \mathit{V}$ which associates a node in $\mathcal{G}$ to any location ${\ell }^{\ast }$ in the expanded domain ${\mathcal{D}}^{\ast }$:

$$\eta \left({\ell }^{\ast }\right)=\left\{\begin{array}{l}{\eta }_A\left({\ell }^{\ast }\right)={\mathit{a}}_{ri}\in {\mathit{A}}_r\hspace{0.33em}\text{if}\hspace{0.33em}{\ell }^{\ast }\in S_{ri},\hfill \\ {\eta }_B\left({\ell }^{\ast }\right)=\mathit{b}\in \mathit{B}\hspace{0.33em}\text{if}\hspace{0.33em}{\ell }^{\ast }\in \mathcal{U}.\hfill \end{array}\right.$$ (3)

This is a many-to-one map; note however that all locations in S_ij are mapped to a_ij: by calling $\eta \left(X\right)=\left\{\eta \left({\ell }^{\ast }\right):{\ell }^{\ast }\in X\right\}$ then for any i = 0,…, M – 1 and any j = 1,…, m_i we have $\eta \left(S_{ij}\right)={\eta }_A\left(S_{ij}\right)={\mathit{a}}_{ij}$. SpamTrees introduce flexibility by cherry picking the leaves, i.e. using ${\eta }_B:\mathcal{U}\to \mathit{B}$, the restriction of η to $\mathcal{U}$. Since each leaf node b_j determines a unique path in $\mathcal{G}$ ending in b_j, we use η_B to assign a convenient parent set to w(u), $\mathit{u}\in \mathcal{U}$, following some criterion.

For example, suppose that $\mathit{u}=\left(\ell ,{\xi }_{\mathit{s}}\right)$ meaning that $w\left(\mathit{u}\right)=w\left(\ell ,{\xi }_{\mathit{s}}\right)$ is the realization of the s-th variable at the spatial location ℓ, and we wish to ensure that Pa[w(u)] includes realizations of the same variable. Denote $\mathit{T}=\left\{\mathit{v}\in \mathit{A}:\text{Ch}\left[\mathit{v}\right]\subset \mathit{B}\right\}$ as the set of terminal branches of $\mathcal{G}$. Then we find ${\left(\ell ,{\xi }_{\mathit{s}}\right)}_{\text{opt}}=\arg {\min }_{\left({\ell }^{\prime },{\xi }^{\prime }={\xi }_{\mathit{s}}\right)\in {\eta }_A^{-1}\left(\mathit{T}\right)}d\left({\ell }^{\prime },\ell \right)$ where d(·,·) is the Euclidean distance. Since ${\left(\ell ,{\xi }_{\mathit{s}}\right)}_{\text{opt}}\in S_{ij}$ for some i, j we have ${\eta }_A\left({\left(\ell ,{\xi }_{\mathit{s}}\right)}_{\text{opt}}\right)={\mathit{a}}_{ij}$. We then set η_B(u) = b_k where Pa[b_k] = {a_ij}. In a sense a_ij is the terminal node nearest to u; having defined η_B in such a way forces the parent set of any location to include at least one realization of the process from the same variable. There is no penalty in using ${\mathcal{D}}^{\ast }=\mathcal{D}\times \Xi$ as we can write $p\left(\mathit{w}\left(\mathit{u}\right)|\text{Pa}\left[\mathit{w}\left(\mathit{u}\right)\right]\right)=\left.p\left(\mathit{w}\left(\left(\ell ,{\xi }_1\right),\dots ,\left(\ell ,{\xi }_q\right)\right)|\text{Pa}\left[\mathit{w}\left(\mathit{u}\right)\right]\right)={\prod }_{s=1}^{q}p\left(w\left(\ell ,{\xi }_s\right)|w\left(\ell ,{\xi }_1\right),\dots ,w\left(\ell ,{\xi }_{s-1}\right),\text{Pa}\left[\mathit{w}\left(\ell \right)\right]\right)\right)$, which also implies that the size of the parent set may depend on the variable index. Assumptions of conditional independence across variables can be encoded similarly. Also note that any specific choice of η_B induces a partition on $\mathcal{U}$; let $U_j=\left\{\mathit{u}\in \mathcal{U}:{\eta }_B\left(\mathit{u}\right)={\mathit{b}}_j\right\}$, then clearly $\mathcal{U}={\cup }_{j=1}^{m_U}{U}_j$ with $U_i\cap U_j=\cancel{0}$ if $i\ne j$. This partition does not necessarily correspond to the partitioning scheme used on $\mathcal{S}$. ${\eta }_B$ may by designed to ignore part of the tree and result in $m_U<m_B$. However, we can just drop the unused leaves from $\mathcal{G}$ and set $\text{Ch}\left[\mathit{a}\right]=\cancel{0}$ for terminal nodes whose leaf is inactive, resulting in $m_U=m_B$. We will thus henceforth assume that $m_U=m_B$ without loss of generality.

2.2. SpamTrees as a Standalone Spatial Process

We define a valid joint density for any finite set of locations in ${\mathcal{D}}^{\ast }$ satisfying the Kolmogorov consistency conditions in order to define a valid process. We approach this problem analogously to Datta et al. (2016a) and Peruzzi et al. (2020). Enumerate each of the $m_S$ reference subsets as $S_i=\left\{{\mathit{s}}_{i_1},\dots ,{\mathit{s}}_{i_{n_i}}\right\}$ where $\left\{i_1,\dots ,i_{n_i}\right\}\subset \left\{1,\dots ,n_{\mathcal{S}}\right\}$, and each of the $m_U$ non-reference subsets as $U_i=\left\{{\mathit{u}}_{i_1},\dots ,{\mathit{u}}_{i_{n_i}}\right\}$ where $\left\{i_1,\dots ,i_{n_i}\right\}\subset \left\{1,\dots ,n_{\mathcal{U}}\right\}$. Then introduce $\mathcal{V}=\left\{V_1,\dots ,V_{m_V}\right\}$ where $m_V=m_S+m_U$ and $V_i=S_i$ for $i=1,\dots ,m_S$, $V_{m_S+i}=U_i$ for $i=1,\dots ,m_U$. Then take ${\mathit{w}}_i={\left(w\left({\ell }_{i_1}\right),\dots ,w\left({\ell }_{i_{n_i}}\right)\right)}^{\top }$ as the $n_i\times 1$ random vector with elements of $w\left(\ell \right)$ for each $\ell \in V_i$. Denote ${\mathit{w}}_{\left[i\right]}=\mathit{w}\left({\eta }^{-1}\left(\text{Pa}\left[{\mathit{v}}_i\right]\right)\right)$. Then

$$\begin{gathered} \tilde{p}\left({\mathit{w}}_{\mathcal{S}}\right)=\tilde{p}\left({\mathit{w}}_1,\dots ,{\mathit{w}}_{m_S}\right)=\prod _{r=0}^{M-1}\prod _{i:\left\{{\mathit{v}}_i\in {\mathit{A}}_r\right\}}p\left({\mathit{w}}_i|{\mathit{w}}_{\left[i\right]}\right)\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}\tilde{p}\left({\mathit{w}}_{\mathcal{U}}|{\mathit{w}}_{\mathcal{S}}\right)=\prod _{i:\left\{{\mathit{v}}_i\in \mathit{B}\right\}}p\left({\mathit{w}}_i|{\mathit{w}}_{\left[i\right]}\right) \\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\tilde{p}\left({\mathit{w}}_{\mathcal{S}}\right)\tilde{p}\left({\mathit{w}}_{\mathcal{U}}|{\mathit{w}}_{\mathcal{S}}\right)=\prod _{r=0}^{M-1}\prod _{i:\left\{{\mathit{v}}_i\in {\mathit{A}}_r\right\}}p\left({\mathit{w}}_i|{\mathit{w}}_{\left[i\right]}\right)\prod _{i:\left\{{\mathit{v}}_i\in \mathit{B}\right\}}p\left({\mathit{w}}_i|{\mathit{w}}_{\left[i\right]}\right) \end{gathered}$$ (4)

which is a proper multivariate joint density since $\mathcal{G}$ is acyclic (Lauritzen, 1996). All locations inside U_j always share the same parent set, but a parent set is not necessarily unique to a single U_j. This includes as a special case a scenario in which one can assume

$$\tilde{p}\left({\mathit{w}}_{\mathcal{U}}|{\mathit{w}}_{\mathcal{S}}\right)=\prod _{j=1}^{m_U}\prod _{i=1}^{\left|U_j\right|}p\left(w\left({\mathit{u}}_i\right)|\mathit{w}\left({\eta }^{-1}\left(\text{Pa}\left[{\mathit{b}}_j\right]\right)\right)\right);$$ (5)

in this case each location corresponds to a leaf node. To conclude the construction, for any finite subset of spatial locations $\mathcal{L}\subset \mathcal{D}$ we can let $\mathcal{U}=\mathcal{L}\backslash \mathcal{S}$ and obtain

$$\tilde{p}\left({\mathit{w}}_{\mathcal{L}}\right)=\int \tilde{p}\left({\mathit{w}}_{\mathcal{U}}|{\mathit{w}}_{\mathcal{S}}\right)\tilde{p}\left({\mathit{w}}_{\mathcal{S}}\right)\prod _{{\mathit{s}}_i\in \mathcal{S}\backslash \ell }d\left(\mathit{w}\left({\mathit{s}}_i\right)\right),$$

leading to a well-defined process satisfying the Kolmogorov conditions (see Appendix A).

2.2.1. Positioning of Spatial Locations in Conditioning Sets

In spatial models based on sparse DAGs, larger conditioning sets yield processes that are closer to the base process p in terms of Kullback-Leibler divergence (Banerjee, 2020; Peruzzi et al., 2020), denoted as $KL\left(p||\cdot \right)$. The same results cannot be applied directly to SpamTrees given the treed structure of the DAG. For a given $\mathcal{S}$, we consider the distinct but related issues of placing individual locations into reference subsets (1) at different levels of the treed hierarchy; (2) within the same level of the hierarchy.

Proposition 1 Suppose $\mathcal{S}={\mathcal{S}}_0\cup {\mathcal{S}}_1$ where $S_0\cap S_1=\cancel{0}$ and ${\mathcal{S}}_1=S_{11}\cup S_{12}$,$S_{11}\cap S_{12}=\cancel{0}$. Take ${\mathit{s}}^{\ast }\notin \mathcal{S}$. Consider the graph $\mathcal{G}=\left\{\mathit{V}=\left\{{\mathit{v}}_0,{\mathit{v}}_1,{\mathit{v}}_2\right\},\mathit{E}=\left\{{\mathit{v}}_0\to {\mathit{v}}_1,{\mathit{v}}_0\to {\mathit{v}}_2\right\}\right\}$; denote as p₀ the density of a SpamTree using $\eta \left({\mathcal{S}}_0\cup \left\{{\mathit{s}}^{\ast }\right\}\right)={\mathit{v}}_0$, $\eta \left(S_{11}\right)={\mathit{v}}_1$ and $\eta \left(S_{12}\right)={\mathit{v}}_2$, whereas let p₁ be the density of a SpamTree with $\eta \left({\mathcal{S}}_0\right)={\mathit{v}}_0$, $\eta \left(S_{11}\cup \left\{{\mathit{s}}^{\ast }\right\}\right)={\mathit{v}}_1$ and $\eta \left(S_{12}\right)={\mathit{v}}_2$. Then $KL\left(p||p_1\right)-KL\left(p||p_0\right)>0$.

The proof proceeds by an “information never hurts” argument (Cover and Thomas, 1991). Denote ${\mathcal{S}}^{\ast }=\mathcal{S}\cup \left\{{\mathit{s}}^{\ast }\right\}$, ${\mathit{w}}^{\ast }={\mathit{w}}_{{\mathcal{S}}^{\ast }}$, $w^{\ast }=w\left({\mathit{s}}^{\ast }\right)$ and ${\mathit{w}}_j^{\ast }={\left({\mathit{w}}_j^{\top },w^{\ast }\right)}^{\top }$. Then

$$p_0\left({\mathit{w}}^{\ast }\right)=p\left({\mathit{w}}_0^{\ast }\right)p\left({\mathit{w}}_1|{\mathit{w}}_0^{\ast }\right)p\left({\mathit{w}}_2|{\mathit{w}}_0^{\ast }\right)=p\left({\mathit{w}}_0\right)p\left(w^{\ast }|{\mathit{w}}_0\right)p\left({\mathit{w}}_1|{\mathit{w}}_0,w^{\ast }\right)p\left({\mathit{w}}_2|{\mathit{w}}_0^{\ast }\right)$$

$$p_1\left({\mathit{w}}^{\ast }\right)=p\left({\mathit{w}}_0\right)p\left({\mathit{w}}_1^{\ast }|{\mathit{w}}_0\right)p\left({\mathit{w}}_2|{\mathit{w}}_0\right)=p\left({\mathit{w}}_0\right)p\left(w^{\ast }|{\mathit{w}}_0\right)p\left({\mathit{w}}_1|{\mathit{w}}_0,w^{\ast }\right)p\left({\mathit{w}}_2|{\mathit{w}}_0\right),$$

therefore $p_0\left({\mathit{w}}^{\ast }\right)/p_1\left({\mathit{w}}^{\ast }\right)=p\left({\mathit{w}}_2|{\mathit{w}}_0^{\ast }\right)/p\left({\mathit{w}}_2|{\mathit{w}}_0\right)$; then by Jensen’s inequality

$$\begin{array}{l}KL\left(p\left|\right|p_1\right)-KL\left(p\left|\right|p_0\right)=\int \left\{\log \left(\frac{p\left({\mathit{w}}^{\ast }\right)}{p_1\left({\mathit{w}}^{\ast }\right)}\right)-\log \left(\frac{p\left({\mathit{w}}^{\ast }\right)}{p_0\left({\mathit{w}}^{\ast }\right)}\right)\right\}p\left({\mathit{w}}^{\ast }\right)d{\mathit{w}}^{*}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\int \log \left(\frac{p_0\left({\mathit{w}}^{\ast }\right)}{p_1\left({\mathit{w}}^{\ast }\right)}\right)p\left({\mathit{w}}^{\ast }\right)d{\mathit{w}}^{\ast }=\int \log \left(\frac{p\left({\mathit{w}}_2|{\mathit{w}}_0^{\ast }\right)}{p\left({\mathit{w}}_2|{\mathit{w}}_0\right)}\right)p\left({\mathit{w}}^{\ast }\right)d{\mathit{w}}^{\ast }\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\int \log \left(\frac{p\left({\mathit{w}}_2|{\mathit{w}}_0^{\ast }\right)}{p\left({\mathit{w}}_2|{\mathit{w}}_0\right)}\right)p\left({\mathit{w}}_1,{\mathit{w}}_2,{\mathit{w}}_0^{\ast }\right)d{\mathit{w}}_{1}d{\mathit{w}}_{2}d{\mathit{w}}_0^{\ast }\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\int \left\{\int \log \left(\frac{p\left({\mathit{w}}_2|{\mathit{w}}_0^{\ast }\right)}{p\left({\mathit{w}}_2|{\mathit{w}}_0\right)}\right)p\left({\mathit{w}}_1,{\mathit{w}}_2|{\mathit{w}}_0^{\ast }\right)d{\mathit{w}}_{1}d{\mathit{w}}_2\right\}p\left({\mathit{w}}_0^{\ast }\right)d{\mathit{w}}_0^{\ast }\ge 0.\end{array}$$ (6)

Intuitively, this shows that there is a penalty associated to positioning reference locations at higher levels of the treed hierarchy. Increasing the size of the reference set at the root augments the conditioning sets at all its children; since this is not true when the increase is at a branch level, the KL divergence of p₀ from p is smaller than the divergence of p₁ from the same density. In other words there is a cost of branching in $\mathcal{G}$ which must be justified by arguments related to computational efficiency. The above proposition also suggests populating near-root branches with locations of sparsely-observed outcomes. Not doing so in highly imbalanced settings may result in possibly too restrictive spatial conditional independence assumptions.

Proposition 2 Consider the same setup as Proposition 1 and let p₂ be the density of a SpamTree such that $\eta \left(S_{12}\cup \left\{{\mathit{s}}^{\ast }\right\}\right)={\mathit{v}}_2$. Let H_p be the conditional entropy of base process p. Then $H_p\left(w^{\ast }|{\mathit{w}}_0,{\mathit{w}}_2\right)<H_p\left(w^{\ast }|{\mathit{w}}_0,{\mathit{w}}_1\right)$ implies $KL\left(p||p_2\right)<KL\left(p||p_1\right)$.

The density of the new model is

$$p_2\left({\mathit{w}}^{\ast }\right)=p\left({\mathit{w}}_0\right)p\left({\mathit{w}}_1|{\mathit{w}}_0\right)p\left({\mathit{w}}_2^{\ast }|{\mathit{w}}_0\right)=p\left({\mathit{w}}_0\right)p\left({\mathit{w}}_1|{\mathit{w}}_0\right)p\left({\mathit{w}}_2|{\mathit{w}}_0\right)p\left(w^{\ast }|{\mathit{w}}_0,{\mathit{w}}_2\right).$$

Then, noting that $p\left({\mathit{w}}_1^{\ast }|{\mathit{w}}_0\right)=p\left({\mathit{w}}_1|{\mathit{w}}_0\right)p\left(w^{\ast }|{\mathit{w}}_0,{\mathit{w}}_1\right)$, we get $\frac{p_1\left({\mathit{w}}^{\ast }\right)}{p_2\left({\mathit{w}}^{\ast }\right)}=\frac{p\left(w^{\ast }|{\mathit{w}}_0,{\mathit{w}}_1\right)}{p\left(w^{\ast }|{\mathit{w}}_0,{\mathit{w}}_2\right)}$ and

$$\begin{array}{l}KL\left(p\left|\right|p_2\right)-KL\left(p\left|\right|p_1\right)=\int \log p\left(w^{\ast }|{\mathit{w}}_0,{\mathit{w}}_1\right)p\left({\mathit{w}}^{\ast }\right)d{\mathit{w}}^{\ast }-\int \log p\left(w^{\ast }|{\mathit{w}}_0,{\mathit{w}}_2\right)p\left({\mathit{w}}^{\ast }\right)d{\mathit{w}}^{\ast }\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=H_p\left(w^{\ast }|{\mathit{w}}_0,{\mathit{w}}_2\right)-H_p\left(w^{\ast }|{\mathit{w}}_0,{\mathit{w}}_1\right).\end{array}$$

While we do not target the estimation of these quantities, this result is helpful in designing SpamTrees as it suggests placing a new reference location s^∗ in the reference subset least uncertain about the realization of the process at s^∗. We interpret this as justifying recursive domain partitioning on $\mathcal{S}$ in spatial contexts in which local spatial clusters of locations are likely less uncertain about process realization in the same spatial region. In the remainder of this article, we will consider a given reference set $\mathcal{S}$ which typically will be based on a subset of observed locations; the combinatorial problem of selecting an optimal $\mathcal{S}$ (in some sense) is beyond the scope of this article. If $\mathcal{S}$ is not partitioned, it can be considered as a set of knots or “sensors” and one can refer to a large literature on experimental design and optimal sensor placement (see e.g. Krause et al., 2008, and references therein). It might be possible to extend previous work on adaptive knot placement (Guhaniyogi et al., 2011), but this will come at a steep cost in terms of computational performance.

3. Bayesian Spatial Regressions Using SpamTrees

Suppose we observe an l-variate outcome at spatial locations $\ell \in \mathcal{D}\subset {\Re }^d$ which we wish to model using a spatially-varying regression model:

$$y_j\left(\ell \right)={\mathit{x}}_j{\left(\ell \right)}^{\top }{\mathit{\beta }}_j+\sum _k{z}_{jk}\left(\ell \right)w\left(\ell ,{\mathit{\xi }}_k\right)+{\epsilon }_j\left(\ell \right),\hspace{1em}j=1,\dots ,l,$$ (7)

where $y_j\left(\ell \right)$ is the j-th point-referenced outcome at ℓ, ${\mathit{x}}_j\left(\ell \right)$ is a p_j × 1 vector of spatially referenced predictors linked to constant coefficients β_j, ${\epsilon }_j\left(\ell \right)\stackrel{iid}{~}N\left(0,{\tau }_j^2\right)$ is the measurement error for outcome j, and $z_{jk}\left(\ell \right)$ is the k-th (of q) covariates for the j-th outcome modeled with spatially-varying coefficient $w\left(\ell ,{\mathit{\xi }}_k\right)$, $\ell \in \mathcal{D}$,${\mathit{\xi }}_k\in \Xi$. This coefficient $w\left(\ell ,{\mathit{\xi }}_k\right)$ corresponds to the k-th margin of a q-variate Gaussian process $\left\{\mathit{w}\left(\ell \right):\ell \in \mathcal{D}\right\}$ denoted as $\mathit{w}\left(\ell \right)~GP\left(\text{0},{\mathit{C}}_{\mathit{\theta }}\left(\cdot ,\cdot \right)\right)$ with cross-covariance C_θ indexed by unknown parameters θ which we omit in notation for simplicity. A valid cross-covariance function is defined as ${\mathit{C}}_{\mathit{\theta }}:\mathcal{D}\times \mathcal{D}\to {\mathcal{M}}_{q\times q}$, where ${\mathcal{M}}_{q\times q}$ is a subset of the space of all q × q real matrices ${\Re }^{q\times q}$. It must satisfy $\mathit{C}\left(\ell ,{\ell }^{\prime }\right)=\mathit{C}{\left({\ell }^{\prime },\ell \right)}^{\top }$ for any two locations ℓ, ${\ell }^{\prime }\in \mathcal{D}$ and ${\sum }_{i=1}^n{\sum }_{j=1}^n{\mathit{z}}_i^{\top }\mathit{C}\left({\ell }_i,{\ell }_j\right){\mathit{z}}_j>0$ for any integer n and finite collection of points $\left\{{\ell }_1,{\ell }_2,\dots ,{\ell }_n\right\}$ and for all ${\mathit{z}}_i\in {\Re }^q\backslash \left\{\text{0}\right\}$.

We replace the full GP with a Gaussian SpamTree for scalable computation considering the q-variate multivariate Gaussian process w(·) as the base process. Since the (i, j)-th entry of $\mathit{C}\left(\ell ,{\ell }^{\prime }\right)$ is $\mathit{C}{\left(\ell ,{\ell }^{\prime }\right)}_{i,j}=\text{Cov}\left(w_i\left(\ell \right),w_j\left({\ell }^{\prime }\right)\right)$, i.e. the covariance between the i-th and j-th elements of $\mathit{w}\left(\ell \right)$ at $\ell$ and ${\ell }^{\prime }$, we can obtain a covariance function on the augmented domain ${\mathit{C}}^{\ast }:{\mathcal{D}}^{*}\times {\mathcal{D}}^{*}\to \Re$ as ${\mathit{C}}^{\ast }\left(\left(\ell ,\mathit{\xi }\right),\left({\ell }^{\prime },{\mathit{\xi }}^{\prime }\right)\right)=\mathit{C}{\left(\left(\ell ,{\ell }^{\prime }\right)\right)}_{i,i^{\prime }}$ where $\mathit{\xi }$ and ${\mathit{\xi }}^{\prime }$ are the locations in Ξ of variables i and j, respectively. Apanasovich and Genton (2010) use a similar representation to build valid cross-covariances based on existing univariate covariance functions; their approach amounts to considering $\mathit{\xi }$ or $‖\mathit{\xi }-{\mathit{\xi }}^{\prime }‖$ as a parameter to be estimated. Our approach can be based on any valid cross-covariance as we may just set Ξ = 1,…, q . Refer to e.g. Genton and Kleiber (2015) for an extensive review of cross-covariance functions for multivariate processes. Moving forward, we will not distinguish between C^∗ and C. The linear multivariate spatially-varying regression model (7) allows the l outcomes to be observed at different locations; we later consider the case l = q and $\mathit{Z}\left(\ell \right)=I_q$ resulting in a multivariate space-varying intercept model.

3.1. Gaussian SpamTrees

Enumerate the set of nodes as $\mathit{V}=\left\{{\mathit{v}}_1,\dots ,{\mathit{v}}_{m_V}\right\}$, $m_V=m_S+m_U$ and denote ${\mathit{w}}_i=w\left({\eta }^{-1}\left({\mathit{v}}_i\right)\right)$, C_ij as the n_i × n_j covariance matrix between w_i and w_j, C_i,_[i] the n_i × J_i covariance matrix between w_i and w_[i], C_i the n_i × n_i covariance matrix between w_i and itself, and C_[i] the J_i × J_i covariance matrix between w_[i] and itself. A base Gaussian process induces $\tilde{p}\left({\mathit{w}}_{\mathcal{S}}\right)={\prod }_{j:\left\{{\mathit{v}}_j\in \mathit{A}\right\}}N\left({\mathit{w}}_j|{\mathit{H}}_j{\mathit{w}}_{\left[j\right]},{\mathit{R}}_j\right)$, where

$${\mathit{H}}_j={\mathit{C}}_{j,\left[j\right]}{\mathit{C}}_{\left[j\right]}^{-1}\hspace{1em}\text{and}\hspace{1em}{\mathit{R}}_j={\mathit{C}}_j-{\mathit{C}}_{j,\left[j\right]}{\mathit{C}}_{\left[j\right]}^{-1}{\mathit{C}}_{\left[j\right],j},$$ (8)

implying that the joint density $\tilde{p}\left({\mathit{w}}_{\mathcal{S}}\right)$ is multivariate normal with covariance ${\tilde{\mathit{C}}}_{\mathcal{S}}$ and precision matrix ${\tilde{\mathit{C}}}_{\mathcal{S}}^{-1}$. At $\mathcal{U}$ we have $\tilde{p}\left({\mathit{w}}_{\mathcal{U}}|{\mathit{w}}_{\mathcal{S}}\right)={\prod }_{j:\left\{{\mathit{v}}_j\in \mathit{B}\right\}}N\left({\mathit{w}}_j|{\mathit{H}}_j{\mathit{w}}_{\left[j\right]},{\mathit{R}}_j\right)$, where H_j and R_j are as in (8). All quantities can be computed using the base cross-covariance function. Given that the $\tilde{p}$ densities are Gaussian, so will be the finite dimensional distributions.

The treed graph $\mathcal{G}$ leads to properties which we analyze in more detail in Appendix B and summarize here. For two nodes v_i, ${\mathit{v}}_j\in \mathit{V}$ denote the common descendants as $\text{cd}\left({\mathit{v}}_i,{\mathit{v}}_j\right)=\left(\left\{{\mathit{v}}_i\right\}\cup \text{Ch}\left[{\mathit{v}}_i\right]\right)\cap \left(\left\{{\mathit{v}}_j\right\}\cup \text{Ch}\left[{\mathit{v}}_j\right]\right)$. If ${\mathit{v}}_i\in \text{Pa}\left[{\mathit{v}}_j\right]$ denote H_i→j and ${\mathit{H}}_{\backslash i\to j}$ as the matrix obtained by subsetting H_j to columns corresponding to v_i, or to $\text{Pa}\left[{\mathit{v}}_j\right]\backslash \left\{{\mathit{v}}_j\right\}$, respectively. Similarly define ${\mathit{w}}_{\left[i\to j\right]}={\mathit{w}}_i$ and ${\mathit{w}}_{\left[\backslash i\to j\right]}$. As a special case, if the tree depth is δ = 1 and {v_j} = Pa[v_i] then cd(v_i, v_j) = {v_i}, H_i→j = H_j, and w[i→j] = w[j]. Define $\mathcal{H}$ as the matrix whose (i, j) block is ${\mathcal{H}}_{ij}={\mathit{O}}_{n_i\times n_j}$ if ${\mathit{v}}_j\notin \text{Pa}\left({\mathit{v}}_i\right)$, and otherwise ${\mathcal{H}}_{ij}={\mathit{H}}_{j\to i}$.

3.1.1. Precision Matrix

The (i, j) block of the precision matrix at both reference and non-reference locations ${\tilde{\mathit{C}}}^{-1}$ is denoted by ${\tilde{\mathit{C}}}^{-1}\left(i,j\right)$, with i, j = 1,…, m_V corresponding to nodes v_i, ${\mathit{v}}_j\in \mathit{V}$ for some i, j; it is nonzero if $\text{cd}\left({\mathit{v}}_i,{\mathit{v}}_j\right)=\cancel{0}$, otherwise:

$$\begin{array}{l}{\tilde{\mathit{C}}}^{-1}\left(i,j\right)=\sum _{{\mathit{v}}_k\in \text{cd}\left({\mathit{v}}_i,{\mathit{v}}_j\right)}{\left({\mathit{I}}_{ki}-{\mathit{H}}_{i\to k}\right)}^{\top }{\mathit{R}}_k^{-1}\left({\mathit{I}}_{kj}-{\mathit{H}}_{j\to k}\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\sum _{{\mathit{v}}_k\in \text{cd}\left({\mathit{v}}_i,{\mathit{v}}_j\right)}{\left({\mathit{I}}_{ki}-{\mathcal{H}}_{ki}\right)}^{\top }{\mathit{R}}_k^{-1}\left({\mathit{I}}_{kj}-{\mathcal{H}}_{kj}\right),\end{array}$$ (9)

where I_ij is the (i, j) block of an identity matrix with n_S + n_U rows and is nonzero if and only if i = j. We thus obtain that the number of nonzero elements of ${\tilde{\mathit{C}}}^{-1}$ is

$$\text{nnz}\left({\tilde{\mathit{C}}}^{-1}\right)=\sum _{i=1}^{m_V}\left(2n_i{J}_i+n_i^2\text{1}\left\{{\mathit{v}}_i\in \mathit{V}\right\}\right),$$ (10)

where $n_i=\left|{\eta }^{-1}\left({\mathit{v}}_i\right)\right|$, $J_i=\left|{\eta }^{-1}\left(\text{Pa}\left[{\mathit{v}}_i\right]\right)\right|$, and by symmetry ${\left({\tilde{\mathit{C}}}^{-1}\left(i,j\right)\right)}^{\top }={\tilde{\mathit{C}}}^{-1}\left(j,i\right)$.

If δ > 1, the size of C_[i] is larger for nodes v_i at levels of the treed hierarchy farther from ${\mathit{A}}_{M_{\delta }}$. However suppose v_i, v_j are such that $\text{Pa}\left[{\mathit{v}}_j\right]=\left\{{\mathit{v}}_j\right\}\cup \text{Pa}\left[{\mathit{v}}_i\right]$. Then computing ${\mathit{C}}_{\left[j\right]}^{-1}$ proceeds more cheaply by recursively applying the following:

$${\mathit{C}}_{\left[j\right]}^{-1}=\left[\begin{array}{cc}{\mathit{C}}_{\left[i\right]}^{-1}+{\mathit{H}}_i^{\top }{\mathit{R}}_i^{-1}{\mathit{H}}_i& -{\mathit{H}}_i^{\top }{\mathit{R}}_i^{-1}\\ -{\mathit{R}}_i^{-1}{\mathit{H}}_i& {\mathit{R}}_i^{-1}\end{array}\right].$$ (11)

3.1.2. Induced Covariance

Define a path from v_k to v_j as ${\mathcal{P}}_{k\to j}=\left\{{\mathit{v}}_{i_1},\dots ,{\mathit{v}}_{i_r}\right\}$ where ${\mathit{v}}_{i_1}={\mathit{v}}_k$, ${\mathit{v}}_{i_r}={\mathit{v}}_j$, and ${\mathit{v}}_{i_h}\in \text{Pa}\left[{\mathit{v}}_{i_{h+1}}\right]$. The longest path ${\tilde{\mathcal{P}}}_{k\to j}$ is such that if ${\mathit{v}}_k\in {\mathit{A}}_{r_k}$ and ${\mathit{v}}_j\in {\mathit{A}}_{r_j}$ then $\left|{\tilde{\mathcal{P}}}_{k\to j}\right|=r_j-r_k+1$. The shortest path ${\overline{\mathcal{P}}}_{k\to j}$ is the path from v_k to v_j with minimum number of steps. We denote the longest path from the root to v_j as ${\tilde{\mathcal{P}}}_{0\to j}$; this corresponds to the full set of ancestors of v_j, and $\text{Pa}\left[{\mathit{v}}_j\right]\subset {\tilde{\mathcal{P}}}_{0\to j}$. For two nodes v_i and v_j we have $\left(\text{Pa}\left[{\mathit{v}}_i\right]\cap \text{Pa}\left[{\mathit{v}}_j\right]\right)\subset \left({\tilde{\mathcal{P}}}_{0\to i}\cap {\tilde{\mathcal{P}}}_{0\to j}\right)$. We define the concestor between v_i and v_j as $\text{con}\left({\mathit{v}}_i,{\mathit{v}}_j\right)=\arg {\max }_{{\mathit{v}}_k\in \mathit{V}}\left\{k:{\mathcal{P}}_{k\to i}\cap {\mathcal{P}}_{k\to j}\ne \cancel{0}\right\}$ i.e. the last common ancestor of the two nodes.

Take the path ${\tilde{\mathcal{P}}}_{M_{\delta }\to j}$ in $\mathcal{G}$ from a node at ${\mathit{A}}_{M_{\delta }}$ leading to v_j. After defining the cross-covariance function ${\mathit{K}}_i\left(\ell ,{\ell }^{\prime }\right)={\mathit{C}}_{\ell ,{\ell }^{\prime }}-{\mathit{C}}_{\ell ,\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}{\mathit{C}}_{\left[i\right],{\ell }^{\prime }}$ and denoting ${\mathit{K}}_i\left(\ell ,s\right)={\mathit{K}}_i\left(\ell ,{\eta }^{-1}\left({\mathit{v}}_s\right)\right)$ we can write

$${\mathit{w}}_j=\sum _{s=i_{M_{\delta }}}^{i_{r-1}}{\mathit{K}}_s\left(j,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{e}}_s+{\mathit{e}}_j,$$ (12)

where for $s>i_{M_{\delta }}$ the e_s are independent zero-mean GPs with covariance $K_s\left(\ell ,{\ell }^{\prime }\right)$ and we set $K_{i_{M_{\delta }}}\left(\ell ,{\ell }^{\prime }\right)=\mathit{C}\left(\ell ,{\ell }^{\prime }\right)$ and ${\mathit{e}}_{i_{M_{\delta }}}={\mathit{w}}_{i_{M_{\delta }}}~N\left(0,{\mathit{C}}_{i_{M_{\delta }}}\right)$. Take two locations ℓ, ℓ^ꞌ such that ${\mathit{v}}_i=\eta \left(\ell \right)$, ${\mathit{v}}_j=\eta \left({\ell }^{\prime }\right)$ and let ${\mathit{v}}_z=\text{con}\left({\mathit{v}}_i,{\mathit{v}}_j\right)$; if $\text{Pa}\left[{\mathit{v}}_i\right]\cap \text{Pa}\left[{\mathit{v}}_j\right]\ne \cancel{0}$ then the above leads to

$$Co{v}_{\tilde{p}}\left(\mathit{w}\left(\ell \right),\mathit{w}\left({\ell }^{\prime }\right)\right)=\sum _{s\in \text{Pa}\left[{\mathit{v}}_i\right]\cap \text{Pa}\left[{\mathit{v}}_j\right]}{\mathit{K}}_s\left(\ell ,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{K}}_s\left(s,{\ell }^{\prime }\right)+\text{1}\left\{\ell ={\ell }^{\prime }\right\}{\mathit{K}}_j\left(\ell ,{\ell }^{\prime }\right),$$ (13)

where ${\mathit{K}}_z\left(\ell ,{\ell }^{\prime }\right)=\mathit{C}\left(\ell ,{\ell }^{\prime }\right)$. If $\text{Pa}\left[{\mathit{v}}_i\right]\cap \text{Pa}\left[{\mathit{v}}_j\right]=\cancel{0}$ take the shortest paths ${\overline{\mathcal{P}}}_{z\to i}=\left\{i_1,\dots ,i_{r_i}\right\}$ and ${\overline{\mathcal{P}}}_{z\to j}=\left\{j_1,\dots ,j_{r_j}\right\}$; setting ${\mathit{F}}_{i_h}={\mathit{C}}_{i_h,i_{h-1}}{\mathit{C}}_{i_{h-1}}^{-1}$ we get

$$Co{v}_{\tilde{p}}\left(\mathit{w}\left(\ell \right),\mathit{w}\left({\ell }^{\prime }\right)\right)={\mathit{F}}_{i_{r_i}}\cdots {\mathit{F}}_{i_1}{\mathit{C}}_z{\mathit{F}}_{j_1}^{\top }\cdots {\mathit{F}}_{j_{r_j}}^{\top }.$$ (14)

In particular if δ = M then $\text{Pa}\left[{\mathit{v}}_i\right]\cap \text{Pa}\left[{\mathit{v}}_j\right]\ne \cancel{0}$ for all i, j and only (13) is used, whereas if δ = 1 then the only scenario in which (13) holds is $\left\{{\mathit{v}}_z\right\}=\text{Pa}\left[{\mathit{v}}_i\right]\cap \text{Pa}\left[{\mathit{v}}_j\right]$ in which case the two are equivalent. In univariate settings, the special case in which δ = M , and hence M_δ = 0, leads to an interpretation of (12) as a basis function decomposition; considering all leaf paths ${\mathcal{P}}_j$ for ${\mathit{v}}_j\in \mathit{B}$, this leads to an MRA (Katzfuss, 2017; Katzfuss and Gong, 2019). On the other hand, keeping other parameters constant, δ < M and in particular δ = 1 may be associated to savings in computing cost, leading to a trade-off between graph complexity and size of reference subsets; see Appendix B.5.

3.1.3. Block-Sparse Cholesky Decompositions

In recent work Jurek and Katzfuss (2020) consider sparse Cholesky decompositions of co-variance and precision matrices for treed graphs corresponding to the case δ = M above in the context of space-time filtering; their methods involve sparse Cholesky routines on reverse orderings of ${\tilde{\mathit{C}}}^{-1}$ at the level of individual locations. In doing so, the relationship between Cholesky decompositions and $\mathcal{G}$, ${\tilde{\mathit{C}}}^{-1}$ and the block structure in $\mathcal{S}$ remains somewhat hidden, and sparse Cholesky libraries are typically associated to bottlenecks in MCMC algorithms. However we note that a consequence of (9) is that it leads to a direct algorithm, for any δ, for the block-decomposition of any symmetric positive-definite matrix $\text{Λ}$ conforming to $\mathcal{G}$, i.e. with the same block-sparse structure as ${\tilde{\mathit{C}}}^{-1}$. This allows us to write $\text{Λ}={\left(\mathit{I}-\mathit{L}\right)}^{\top }\mathit{D}\left(\mathit{I}-\mathit{L}\right)$ where I is the identity matrix, L is block lower triangular with the same block-sparsity pattern as $\mathcal{H}$ above, and D is block diagonal symmetric positive-definite. In Appendix B.2.3 we outline Algorithm 4 which (i) makes direct use of the structure of $\mathcal{G}$, (ii) computes the decomposition at blocks of reference and non-reference locations, and (iii) requires no external sparse matrix library, in particular no sparse Cholesky solvers. Along with Algorithm 5 for the block-computation of (I – L)⁻¹, it can be used to compute ${\text{Λ}}^{-1}={\left({\tilde{\mathit{C}}}^{-1}+\text{Σ}\right)}^{-1}$ where $\text{Σ}$ is a block-diagonal matrix; it is thus useful in computing the Gaussian integrated likelihood.

3.2. Estimation and Prediction

We introduce notation to aid in obtaining the full conditional distributions. Write (7) as

$$\mathit{y}\left(\ell \right)=\mathit{X}\left(\ell \right)\mathit{\beta }+\mathit{Z}\left(\ell \right)\mathit{w}\left(\ell \right)+\mathit{\epsilon }\left(\ell \right),$$ (15)

where $\mathit{y}\left(\ell \right)={\left({\left\{y_j\left(\ell \right)\right\}}_{j=1}^l\right)}^{\top }$, $\mathit{\epsilon }\left(\ell \right)={\left({\left\{{\epsilon }_j\left(\ell \right)\right\}}_{j=1}^l\right)}^{\top }~N\left(\text{0},{\mathit{D}}_{\tau }\right)$, ${\mathit{D}}_{\tau }=\text{diag}\left({\tau }_1^2,\dots ,{\tau }_l^2\right)$, $\mathit{X}\left(\ell \right)=\text{b}.\text{diag}\left\{{\mathit{x}}_j{\left(\ell \right)}^{\top },j=1,\dots ,l\right\}$, $\mathit{\beta }={\left({\mathit{\beta }}_{p_1}^{\top },\dots ,{\mathit{\beta }}_{p_j}^{\top }\right)}^{\top }$. The l × q matrix $\mathit{Z}\left(\ell \right)=\left({\mathit{z}}_j{\left(\ell \right)}^{\top },j=1,\dots ,l\right)$ with ${\mathit{z}}_j{\left(\ell \right)}^{\top }=\left(z_{jk}\left(\ell \right),k=1,\dots ,q\right)$ acts a design matrix for spatial location ℓ. Collecting all locations along the j-th margin, we build ${\mathcal{T}}_j=\left\{{\ell }_1^{\left(j\right)},\dots ,{\ell }_{N_j}^{\left(j\right)}\right\}$ and $\mathcal{T}={\cup }_j{\mathcal{T}}_j$. We then call ${\mathit{y}}^{\left(j\right)}={\left(y_j\left({\ell }_1^{\left(j\right)}\right),\dots ,y_j\left({\ell }_{N_j}^{\left(j\right)}\right)\right)}^{\top }$ and ${\mathit{\epsilon }}^{\left(j\right)}$ similarly, ${\mathit{X}}^{\left(j\right)}={\left({\mathit{x}}_j\left({\ell }_1^{\left(j\right)}\right),\dots ,{\mathit{x}}_j\left({\ell }_{N_j}^{\left(j\right)}\right)\right)}^{\top }$, ${\mathit{w}}^{\left(j\right)}={\left(\mathit{w}{\left({\ell }_1^{\left(j\right)},\mathit{\xi }\right)}^{\top },\dots ,\mathit{w}{\left({\ell }_{N_j}^{\left(j\right)},\mathit{\xi }\right)}^{\top }\right)}^{\top }$ and ${\mathit{Z}}^{\left(j\right)}=\text{b}.\text{diag}{\left\{{\mathit{z}}_j{\left({\ell }_s^{\left(j\right)}\right)}^{\top }\right\}}_{s=1}^{N_j}$. The full observed data are y, X, Z. Denoting the number of observations as $n={\sum }_{j=1}^l{N}_j$, Z is thus a n × qn block-diagonal matrix, and similarly w is a qn × 1 vector. We introduce the diagonal matrix D_n such that $\text{diag}\left({\mathit{D}}_n\right)={\left({\tau }_1^2{\text{1}}_{N_1}^{\top },\dots ,{\tau }_l^2{\text{1}}_{N_l}^{\top }\right)}^{\top }$.

By construction we may have $\eta \left(S_i\right)={\mathit{v}}_i$ and $\eta \left(S_j\right)={\mathit{v}}_j$ such that $\left(\ell ,\mathit{\xi }\right)\in S_i$ and $\left({\ell }^{\prime },{\mathit{\xi }}^{\prime }\right)\in S_j$ where ${\ell }^{\prime }=\ell$, $\mathit{\xi }\ne {\mathit{\xi }}^{\prime }$ and similarly for non-reference subsets. Suppose $\mathcal{A}\subset \mathcal{D}\times \Xi$ is a generic reference or non-reference subset. We denote $\overline{\mathcal{A}}\subset \mathcal{D}\times \Xi$ as the set of all combinations of spatial locations of $\mathcal{A}$ and variables i.e. $\overline{\mathcal{A}}=\mathcal{A}{\left|{}_{\mathcal{D}}\times \mathcal{A}\right|}_{\Xi }$ where ${\left.\mathcal{A}\right|}_{\mathcal{D}}\subset \mathcal{D}$ is the set of unique spatial locations in $\mathcal{A}$ and ${\left.\mathcal{A}\right|}_{\Xi }$ are the unique latent variable coordinates. By subtraction we find ${\mathcal{A}}_{-}=\overline{\mathcal{A}}\backslash \mathcal{A}$ as the set of locations whose spatial location is in $\mathcal{A}$ but whose variable is not. Let $\mathit{y}\left(\overline{\mathcal{A}}\right)=\mathit{y}\left(\mathcal{A}\right)={\left(\left\{\mathit{y}\left(\ell \right),\ell \in {\left.\mathcal{A}\right|}_{\mathcal{D}}\right\}\right)}^{\top }$,$\mathit{X}\left(\overline{\mathcal{A}}\right)=\mathit{X}\left(\mathcal{A}\right)=\text{b}.\text{diag}\left\{\mathit{X}{\left(\ell \right)}^{\top },\ell \in {\left.\mathcal{A}\right|}_{\mathcal{D}}\right\}$; values corresponding to unobserved locations will be dealt with by defining ${\tilde{\mathit{D}}}_n\left(\mathcal{A}\right)$ as the diagonal matrix obtained from D_n by replacing unobserved outcomes with zeros. Denote $\mathit{Z}\left(\overline{\mathcal{A}}\right)=\text{b}.\text{diag}\left\{\mathit{Z}\left(\ell \right),\ell \in {\left.\mathcal{A}\right|}_{\mathcal{D}}\right\}$ and $\mathit{w}\left(\overline{\mathcal{A}}\right)$ similarly. If $\mathcal{A}$ includes L unique spatial locations then $\mathit{y}\left(\overline{\mathcal{A}}\right)$ is a L l × l vector and $\mathit{X}\left(\mathcal{A}\right)$ is a $L\hspace{0.17em}l\times pl$ matrix. In particular, $\mathit{Z}\left(\overline{\mathcal{A}}\right)$ is a L l × Lql matrix; the subset of its columns with locations in $\mathcal{A}$ is denoted as $\mathit{Z}\left(\mathcal{A}\right)$ whereas at other locations we get $\mathit{Z}\left({\mathcal{A}}_{-}\right)$. We can then separate the contribution of $\mathit{w}\left(\mathcal{A}\right)$ to $\mathit{y}\left(\mathcal{A}\right)$ from the contribution of $\mathit{w}\left({\mathcal{A}}_{-}\right)$ by writing $\mathit{y}\left(\mathcal{A}\right)=\mathit{X}\left(\mathcal{A}\right)\mathit{\beta }+\mathit{Z}\left({\mathcal{A}}_{-}\right)\mathit{w}\left({\mathcal{A}}_{-}\right)+\mathit{Z}\left(\mathcal{A}\right)\mathit{w}\left(\mathcal{A}\right)+\mathit{\epsilon }\left(\mathcal{A}\right)$, using which we let $\tilde{\mathit{y}}\left(\mathcal{A}\right)=\mathit{y}\left(\mathcal{A}\right)-\mathit{X}\left(\mathcal{A}\right)\mathit{\beta }-\mathit{Z}\left({\mathcal{A}}_{-}\right)\mathit{w}\left({\mathcal{A}}_{-}\right)$.

With customary prior distributions $\mathit{\beta }~N\left(\text{0},{\mathit{V}}_{\beta }\right)$ and ${\tau }_j^2~Inv.Gamma\left(a_{\tau },b_{\tau }\right)$ along with a Gaussian SpamTree prior on w, we obtain the posterior distribution as

$$p\left(\mathit{w},\mathit{\beta },{\left\{{\tau }_j^2\right\}}_{j=1}^l,\mathit{\theta }|\mathit{y}\right)\propto p\left(\mathit{y}|\mathit{w},\mathit{\beta },{\left\{{\tau }_j^2\right\}}_{j=1}^l\right)p\left(\mathit{w}|\mathit{\theta }\right)p\left(\mathit{\theta }\right)p\left(\mathit{\beta }\right)\prod _{j=1}^{l}p\left({\tau }_j^2\right).$$ (16)

We compute the full conditional distributions of unknowns in the model, save for θ; iterating sampling from each of these distributions corresponds to a Gibbs sampler which ultimately leads to samples from the posterior distribution above.

3.2.1. Full Conditional Distributions

The full conditional distribution for β is Gaussian with covariance ${\Sigma }_{\mathit{\beta }}^{\ast }={\left({\mathit{V}}_{\mathit{\beta }}^{-1}+{\mathit{X}}^{\top }{\mathit{D}}_n^{-1}\mathit{X}\right)}^{-1}$ and mean ${\mu }_{\mathit{\beta }}^{\ast }={\Sigma }_{\mathit{\beta }}{\mathit{X}}^{\top }{\mathit{D}}_n^{-1}\left(\mathit{y}-\mathit{Z}w\right)$. For $j=1,\dots ,l$, $p\left({\tau }_j^2|\mathit{\beta },\mathit{y},\mathit{w}\right)=Inv.Gamma\left(a_{\tau ,j}^{\ast },b_{\tau ,j}^{\ast }\right)$ where $a_{\tau ,j}^{\ast }=a_{\tau }+N_j/2$ and $b_{\tau ,j}^{\ast }=b_{\tau }+\frac{1}{2}{\mathit{E}}^{\left(j\right)\top }{\mathit{E}}^{\left(j\right)}$ with ${\mathit{E}}^{\left(j\right)}={\mathit{y}}^{\left(j\right)}-{\mathit{X}}^{\left(j\right)}{\mathit{\beta }}_j-{\mathit{Z}}^{\left(j\right)}{\mathit{w}}^{\left(j\right)}$.

Take a node ${\mathit{v}}_i\in \mathit{V}$. If ${\mathit{v}}_i\in \mathit{A}$ then η⁻¹(v_i) = S_i and for ${\mathit{v}}_j\in \text{Ch}\left[{\mathit{v}}_i\right]$ denote ${\tilde{\mathit{w}}}_j={\mathit{w}}_j-{\mathit{H}}_{\backslash i\to j}{\mathit{w}}_{\left[\backslash i\to j\right]}$. The full conditional distribution of w_i is $N\left({\mathit{\mu }}_i,{\text{Σ}}_i\right)$, where

$$\begin{gathered} \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\text{Σ}}_i^{-1}=\mathit{Z}{\left(S_i\right)}^{\top }{\mathit{D}}_n{\left(S_i\right)}^{-1}\mathit{Z}\left(S_i\right)+{\mathit{R}}_i^{-1}+{\mathit{F}}_i^{\left(c\right)} \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\text{Σ}}_i^{-1}{\mathit{\mu }}_i=\mathit{Z}{\left(S_i\right)}^{\top }{\mathit{D}}_n{\left(S_n\right)}^{-1}\tilde{\mathit{y}}\left(S_i\right)+{\mathit{R}}_i^{-1}{\mathit{H}}_i{\mathit{w}}_{\left[i\right]}+{\mathit{m}}_i^{\left(c\right)} \\ {\mathit{F}}_i^{\left(c\right)}=\sum _{j:\left\{{\mathit{v}}_j\in \text{Ch}\left[{\mathit{v}}_i\right]\right\}}{\mathit{H}}_{i\to j}^{\top }{\mathit{R}}_j^{-1}{\mathit{H}}_{i\to j}\hspace{1em}\hspace{1em}\hspace{1em}{\mathit{m}}_i^{\left(c\right)}=\sum _{j:\left\{{\mathit{v}}_j\in \text{Ch}\left[{\mathit{v}}_i\right]\right\}}{\mathit{H}}_{i\to j}^{\top }{\mathit{R}}_j^{-1}{\tilde{\mathit{w}}}_j \end{gathered}$$ (17)

If ${\mathit{v}}_i\in \mathit{B}$ instead ${\Sigma }_i={\left(\mathit{Z}{\left(U_i\right)}^{\top }{\mathit{D}}_n{\left(U_i\right)}^{-1}\mathit{Z}\left(U_i\right)+{\mathit{R}}_i\right)}^{-1}$ and ${\mathit{\mu }}_i={\text{Σ}}_i\left(\mathit{Z}{\left(U_i\right)}^{\top }{\mathit{D}}_n{\left(U_i\right)}^{-1}\tilde{\mathit{y}}\left(U_i\right)+{\mathit{R}}_i^{-1}{\mathit{H}}_i{\mathit{w}}_{\left[i\right]}\right)$. Sampling of w at nodes at the same level r proceeds in parallel given the assumed conditional independence structure in $\mathcal{G}$. It is thus essential to minimize the computational burden at levels with a small number of nodes to avoid bottlenecks. In particular computing ${\mathit{F}}_i^{\left(c\right)}$ and ${\mathit{m}}_i^{\left(c\right)}$ can become expensive at the root when the number of children is very large. In Algorithm 3 we show that one can efficiently sample at a near-root node v_i by updating ${\mathit{F}}_i^{\left(c\right)}$ and ${\mathit{m}}_i^{\left(c\right)}$ via message-passing from the children of v_i.

3.2.2. Update of θ

The full conditional distribution of θ—which may include ${\mathit{\xi }}_j$ for j = 1,…, q or equivalently ${\delta }_{ij}=‖{\mathit{\xi }}_i-{\mathit{\xi }}_j‖$ if the chosen cross-covariance function is defined on a latent domain of variables—is not available in closed form and sampling a posteriori can proceed

Algorithm 1: Computing p(w|θ).

Algorithm 2: Sampling from the full conditional distribution of w_i when δ = 1.

Algorithm 3: Sampling from the full conditional distribution of w_j when δ = M.

via Metropolis-Hastings steps which involve accept/reject steps with acceptance probability $\alpha =\min \left\{1,\frac{p\left(\mathit{w}|{\mathit{\theta }}^{\prime }\right)p\left({\mathit{\theta }}^{\prime }\right)q\left(\mathit{\theta }|{\mathit{\theta }}^{\prime }\right)}{p\left(\mathit{w}|\mathit{\theta }\right)p\left(\mathit{\theta }\right)q\left({\mathit{\theta }}^{\prime }|\mathit{\theta }\right)}\right\}$. In our implementation, we adaptively tune the standard deviation of the proposal distribution via the robust adaptive Metropolis algorithm (RAM; Vihola, 2012). In these settings, unlike similar models based on DAG representations such as NNGPs and MGPs, direct computation via $p\left(\mathit{w}|\mathit{\theta }\right)={\prod }_{i}N\left({\mathit{w}}_i|{\mathit{H}}_i{\mathit{w}}_{\left[i\right]},{\mathit{R}}_i\right)$ is inefficient as it requires computing ${\mathit{C}}_{\left[i\right]}^{-1}$ whose size grows along the hierarchy in $\mathcal{G}$. We thus outline Algorithm 1 for computing $p\left(\mathit{w}|\mathit{\theta }\right)$ via (11). As an alternative we can perform the update using ratios of $p\left(\mathit{y}|\mathit{\beta },\mathit{\theta },\tau \right)=\int p\left(\mathit{y}|\mathit{w},\mathit{\beta },\tau \right)p\left(\mathit{w}|\mathit{\theta }\right)d\mathit{w}=N\left(\mathit{y}|\mathit{X}\beta ,\mathit{Z}\tilde{C}{\mathit{Z}}^{\top }+{\mathit{D}}_n\right)$ using Algorithms 4 and 5 outlined in Appendix B.2.3 which require no sparse matrix library.

3.2.3. Graph Coloring for Parallel Sampling

An advantage of the treed structure of $\mathcal{G}$ is that it leads to fixed graph coloring associated to parallel Gibbs sampling; no graph coloring algorithms are necessary (see e.g. Molloy and Reed, 2002; Lewis, 2016). Specifically, if δ = M (full depth) then there is a one to one correspondence between the M + 1 levels of $\mathcal{G}$ and graph colors, as evidenced by the parallel blocks in Algorithms 1 and 3. In the case δ = 1, $\mathcal{G}$ is associated to only two colors alternating the odd levels with the even ones. This is possible because the Markov blanket of each node at level r, with r even, only includes nodes at odd levels, and vice-versa.

3.2.4. Prediction of the Outcome at New Locations

The Gibbs sampling algorithm will iterate across the above steps and, upon convergence, will produce samples from $p\left(\mathit{\beta },{\left\{{\tau }_j^2\right\}}_{j=1}^q,\mathit{w}|\mathit{y}\right)$. We obtain posterior predictive inference at arbitrary $\ell \in \mathcal{D}$ by evaluating $p\left(\mathit{y}\left(\ell \right)|\mathit{y}\right)$. If $\ell \in \mathcal{S}\cup \mathcal{U}$, then we draw one sample of $\mathit{y}\left(\ell \right)~N\left(\mathit{X}{\left(\ell \right)}^{\top }\mathit{\beta }+\mathit{Z}{\left(\ell \right)}^{\top }\mathit{w}\left(\ell \right),{\mathit{D}}_n\left(\ell \right)\right)$ for each draw of the parameters from $p\left(\mathit{\beta },{\left\{{\tau }_j^2\right\}}_{j=1}^l,\mathit{w}|\mathit{y}\right)$. Otherwise, considering that $\eta \left(\ell \right)={\mathit{v}}_j\in \mathit{B}$ for some j, with parent nodes $\text{Pa}\left[{\mathit{v}}_j\right]$, we sample $\mathit{w}\left(\ell \right)$ from the full conditional $N\left({\mathit{\mu }}_{\ell }^{\ast },{\text{Σ}}_{\ell }^{\ast }\right)$, where ${\text{Σ}}_{\ell }^{\ast }={\left(\mathit{Z}\left(\ell \right){\mathit{D}}_n{\left(\ell \right)}^{-1}\mathit{Z}{\left(\ell \right)}^{\top }+{\mathit{R}}_{\ell }^{-1}\right)}^{-1}$ and ${\mathit{\mu }}_{\ell }^{\ast }={\text{Σ}}_{\ell }^{\ast }\left(\mathit{Z}\left(\ell \right){\mathit{D}}^{-1}\left(\mathit{y}\left(\ell \right)-\mathit{X}{\left(\ell \right)}^{\top }\mathit{\beta }\right)+{\mathit{R}}_{\ell }^{-1}{\mathit{H}}_{\ell }{\mathit{w}}_{\left[j\right]}\right)$, then draw $\mathit{y}\left(\ell \right)~N\left(\mathit{X}{\left(\ell \right)}^{\top }\mathit{\beta }+\mathit{Z}{\left(\ell \right)}^{\top }\mathit{w}\left(\ell \right),{\mathit{D}}_n\right)$.

3.2.5. Computing and Storage Cost

The update of ${\tau }_j^2$ and β can be performed at a minimal cost as typically $p={\sum }_{j=1}^l{p}_j$ is small; almost all the computation budget must be dedicated to computing $p\left(\mathit{w}|\mathit{\theta }\right)$ and sampling $p\left(\mathit{w}|\mathit{y},\mathit{\beta },{\tau }^2\right)$. Assume that reference locations are all observed $\mathcal{S}\subset \mathcal{T}$ and that all reference subsets have the same size i.e. $\left|S_i\right|=N_s$ for all i. We show in Appendix B.5 that the cost of computing SpamTrees is $O\left(n{N}_s^2\right)$. As a result, SpamTrees compare favorably to other models specifically in not scaling with the cube of the number of samples. δ does not impact the computational order, however, compared to δ = M , choosing δ = 1 lowers the cost by a factor of M or more. For a fixed reference set partition and corresponding nodes, choosing larger δ will result in stronger dependence between leaf nodes and nodes closer to the root—this typically corresponds to leaf nodes being assigned conditioning sets that span larger distances in space. The computational speedup corresponding to choosing δ = 1 can effectively be traded for a coarser partitioning of $\mathcal{S}$, resulting in large conditioning sets that are more local to the leaves.

4. Applications

We consider Gaussian SpamTrees for the multivariate regression model (15). Consider the spatial locations ℓ, ${\ell }^{\prime }\in \mathcal{D}$ and the locations of variables i and j in the latent domain of variables ${\mathit{\xi }}_i$, ${\mathit{\xi }}_j\in \Xi$, then denote $\mathit{h}=‖\ell -{\ell }^{\prime }‖$, $\Delta ={\delta }_{ij}=‖{\mathit{\xi }}_i-{\mathit{\xi }}_j‖$, and

$$C\left(\mathit{h},\mathrm{\Delta }\right)=\frac{\exp \left\{-\varphi ‖\mathit{h}‖/\exp \left\{\frac{1}{2}\beta \log \left(1+\alpha \mathrm{\Delta }\right)\right\}\right\}}{\exp \left\{\beta \log \left(1+\alpha \mathrm{\Delta }\right)\right\}}.$$

For j = 1,…, q we also introduce $C_j\left(\mathit{h}\right)=\exp \left\{-{\varphi }_j‖\mathit{h}‖\right\}$. A non-separable cross-covariance function for a multivariate process can be defined as

$$\text{Cov}\left(w\left(\ell ,{\mathit{\xi }}_i\right),w\left({\ell }^{\prime },{\mathit{\xi }}_j\right)\right)={\mathit{C}}_{ij}\left(\mathit{h}\right)=\left\{\begin{array}{ll}{\sigma }_{i1}^{2}C\left(\mathit{h},{\delta }_{ij}\right)+{\sigma }_{i2}^2{C}_i\left(\mathit{h}\right)\hfill & \text{if}\hspace{0.33em}i=j\hfill \\ {\sigma }_{i1}{\sigma }_{j1}C\left(\mathit{h},{\delta }_{ij}\right)\hfill & \text{if}\hspace{0.33em}i\ne j,\hfill \end{array}\right.$$ (18)

which is derived from eq. (7) of Apanasovich and Genton (2010); locations of variables in the latent domain are unknown, therefore $\mathit{\theta }={\left\{{\sigma }_{i1},{\sigma }_{i2},{\varphi }_i\right\}}_{i=1,\dots ,q}\cup {\left\{{\delta }_{ij}\right\}}_{i=1,\dots ,q}^{j<i}\cup \left\{\alpha ,\beta ,\varphi \right\}$ for a total of 3q + q(q − 1)/2 + 3 unknown parameters.

4.1. Synthetic Data

In this section we focus on bivariate outcomes (q = 2). We simulate data from model (15), setting β = 0, Z = I_q and take the measurement locations on a regular grid of size 70 × 70 for a total of 4,900 spatial locations. We simulate the bivariate spatial field by sampling from the full GP using (18) as cross-covariance function; the nuggets for the two outcomes are set to ${\tau }_1^2=0.01$ and ${\tau }_2^2=0.1$. For j = 1, 2 we fix σ_j2 = 1, α = 1, β = 1 and independently sample ${\sigma }_{j1}~U\left(-3,3\right)$, ϕ_j1 ~ U(−0.1, 3), ${\varphi }_j~U\left(0.1,3\right)$, $\varphi ~U\left(0.1,30\right)$, ${\delta }_{12}~Exp\left(1\right)$, generating a total of 500 bivariate data sets. This setup leads to empirical spatial correlations between the two outcomes smaller than 0.25, between 0.25 and 0.75, and larger than 0.75 in absolute value in 107, 330, and 63 of the 500 data sets, respectively. We introduce misalignment and make the outcomes imbalanced by replacing the first outcome with missing values at ≈50% of the spatial locations chosen uniformly at random, and then repeating this procedure for the second outcome keeping only ≈ 10% of the total locations. We also introduce almost-empty regions of the spatial domain, independently for each outcome, by replacing observations with missing values at ≈ 99% of spatial locations inside small circular areas whose center is chosen uniformly at random in [0, 1]². As a result of these setup choices, each simulated data set reproduces some features of the real-world unbalanced misaligned data we consider in Section 4.2 at a smaller scale and in a controlled experiment. Figure 3 shows one of the resulting 500 data sets.

Figure 3:

Left half: Full data set – a bivariate outcome is generated on 4,900 spatial locations. Right half: Observed data set – the training sample is built via independent subsampling of each outcome.

We consider SpamTrees with δ = 1 and implement multiple variants of SpamTrees with δ = M in order to assess their sensitivity to design parameters. Table 1 reports implementation setups and the corresponding results in all cases; if the design variable “All outcomes at ℓ” is set to “No” then a SpamTree is built on the $\mathcal{D}\times \Xi$ domain. If it is set to “Yes” the DAG will be built using $\mathcal{D}$ only – in other words, the q margins of the latent process are never separated by the DAG if they are measured at the same spatial location. “Cherry pick same outcome” indicates whether the map η(·) should search for neighbors by first filtering for matching outcomes – refer to our discussion at Section 2.1.2. We mention here if the DAG is built using $\mathcal{D}$ only, then the nearest neighbor found via cherry picking will always include a realization of the same margin of $\mathit{w}\left(\cdot \right)$. Finally, if SpamTree is implemented with “Root bias” then the reference set and the DAG are built with locations of the more sparsely observed outcome closer to root nodes of the tree as suggested by Proposition 1.

Table 1:

Prediction and estimation performance on multivariate synthetic data. The four columns on the right refer to root mean square error (RMDE) and mean absolute error (MAE) in out-of-sample predictions, average coverage of empirical 95% prediction intervals, and RMDE in the estimation of θ.

	All outcomes at ℓ	Cherry pick same outcome	Root bias	RMDE(y)	MAE(y)	COVG(y)	RMSE(θ)
SpamTrees δ = M	No	No	No	1.078	0.795	0.955	4.168
	No	No	Yes	1.065	0.786	0.955	4.138
	No	Yes	No	1.083	0.799	0.954	4.168
	No	Yes	Yes	1.085	0.799	0.954	4.138
	Yes	Yes	No	1.081	0.797	0.954	4.080
	Yes	Yes	Yes	1.087	0.801	0.954	4.188
SpamTrees δ = 1	Yes	Yes	No	1.198	0.880	0.956	4.221
Q-MGP	Yes	–	–	1.125	0.819	0.951	4.389
IND-PART	Yes	–	–	1.624	1.229	0.948	8.064
LOWRANK	Yes	–	–	1.552	1.173	0.952	5.647
SPDE-INLA	Yes	–	–	1.152	0.862	0.913
SpamTrees Univariate	–	–	–	1.147	0.846	0.953
NNGP Univariate	–	–	–	1.129	0.832	0.952
BART	–	–	–	1.375	1.036	0.488

SpamTrees are compared with multivariate cubic meshed GPs (Q-MGPs; Peruzzi et al., 2020), a method based on stochastic partial differential equations (Lindgren et al., 2011) estimated via integrated nested Laplace approximations (Rue et al., 2009) implemented via R-INLA using a 15 × 15 grid and labeled SPDE-INLA, a low-rank multivariate GP method (labeled LOWRANK) on 25 knots obtained via SpamTrees by setting M = 1 with no domain partitioning, and an independent partitioning GP method (labeled IND-PART) implemented by setting M = 1 and partitioning the domain into 25 regions. Refer e.g. to Heaton et al. (2019) for an overview of low-rank and independent partitioning methods. All multivariate SpamTree variants, Q-MGPs, LOWRANK and IND-PART use (18) as the cross-covariance function in order to evaluate their relative performance in estimating θ in terms of root mean square error (RMDE) as reported in Table 1. We also include results from a non-spatial regression using Bayesian additive regression trees (BART; Chipman et al., 2010) which uses the domain coordinates as covariates in addition to a binary fixed effect corresponding to the outcome index. All methods were setup to target a compute time of approximately 15 seconds for each data set. We focused on comparing the different methods under computational constraints because (a) without constraints it would not be feasible to implement the methods for many large simulated spatial datasets; and (b) the different methods are mostly focused on providing a faster approximation to full GPs; if constraints were removed one would just be comparing the same full GP method.

Table 1 reports average performance across all data sets. All Bayesian methods based on latent GPs exhibit very good coverage; in these simulated scenarios, SpamTrees exhibit comparatively lower out-of-sample prediction errors. All SpamTrees perform similarly, with the best out-of-sample predictive performance achieved by the SpamTrees cherry picking based solely on spatial distance (i.e. disregarding whether or not the nearest-neighbor belongs to the same margin). Additional implementation details can be found in Appendix C.2.1. Finally, we show in Figure 4 that the relative gains of SpamTrees compared to independent univariate NNGP model of the outcomes are increasing with the magnitude of the correlations between the two outcomes, which are only available due to the simulated nature of the data sets.

Figure 4:

Predictive RMSE of the best-performing SpamTree of Table 1 relative to independent univariate NNGP models of the two outcomes, for different empirical correlations between the two outcomes in the full data. Lower values indicate smaller errors of SpamTrees in predictions.

4.2. Climate Data: MODIS-TERRA and GHCN

Climate data are collected from multiple sources in large quantities; when originating from satellites and remote sensing, they are typically collected at high spatial and relatively low temporal resolution. Atmospheric and land-surface products are obtained via post-processing of satellite imaging, and their quality is negatively impacted by cloud cover and other atmospheric disturbances. On the other hand, data from a relatively small number of land-based stations is of low spatial but high temporal resolution. An advantage of land-based stations is that they measure phenomena related to atmospheric conditions which cannot be easily measured from satellites (e.g. precipitation data, depth of snow cover).

We consider the joint analysis of five spatial outcomes collected from two sources. First, we consider Moderate Resolution Imaging Spectroradiometer (MODIS) data from the Terra satellite which is part of the NASA’s Earth Observing System. Specifically, data product MOD11C3 v. 6 provides monthly Land Surface Temperature (LST) values in a 0.05 degree latitude/longitude grid (the Climate Modeling Grid or CMG). The monthly data sets cover the whole globe from 2000–02-01 and consist of daytime and nighttime LSTs, quality control assessments, in addition to emissivities and clear-sky observations. The second source of data is the Global Historical Climatology Network (GHCN) database which includes climate summaries from land surface stations across the globe subjected to common quality assurance reviews. Data are published by the National Centers of Environmental Information (NCEI) of the National Oceanic and Atmospheric Administration (NOAA) at several different temporal resolutions; daily products report five core elements (precipitation, snowfall, snow depth, maximum and minimum temperature) in addition to several other measurements.

We build our data set for analysis by focusing on the continental United States in October, 2018. The MODIS data correspond to 359,822 spatial locations. Of these, 250,874 are collected at the maximum reported quality; we consider all remaining 108,948 spatial locations as missing, and extract (1) daytime LST (LST_Day_CMG), (2) nighttime LST (LST_Night_CMG), (3) number of days with clear skies (Clear_sky_days), (4) number of nights with clear skies (Clear_sky_nights). From the GHCN database we use daily data to obtain monthly averages for precipitation (PRCP), which is available at 24,066 spatial locations corresponding to U.S. weather stations; we log-transform PRCP. The two data sources do not share measurement locations as there is no overlap between measurement locations in MODIS and GHCN, with the latter data being collected more sparsely—this is a scenario of complete spatial misalignment. From the resulting data set of size n =1,027,562 we remove all observations in a large 3 × 3 degree area in the central U.S. (from -100W to -97W and from 35N to 38N, i.e. the red area of Figure 5) to build a test set on which we calculate coverage and RMDE of the predictions.

Figure 5:

Prediction area

We implement SpamTrees using the cross-covariance function (18). Considering that PRCP is more sparsely measured and following Proposition 1, we build SpamTrees favoring placement of GHCN locations at root nodes. We compare SpamTrees with a Q-MGP model built on the same cross-covariance function, and two univariate models that make predictions independently for each outcome. Comparisons with other multivariate methods are difficult due to the lack of scalable software for this data size which also deals with misalignment and imbalances across outcomes. Compute times per MCMC iteration ranged from 2.4s/iteration of the multivariate Q-MGP model, to 1.5s/iteration of the univariate NNGP model. The length of the MCMC chains (30,000 for SpamTrees and 20,000 for Q-MGP) was such that the total compute time was about the same for both models at less than 16 hours. Univariate models cannot estimate cross-covariances of multivariate outcomes and are thus associated to faster compute times; we set the length of their MCMC chains to 15,000 for a total compute time of less than 7 hours for both models. We provide additional details about the models we implemented at Appendix C.

Table 2 reports predictive performance of all models, and Figure 6 maps the predictions at all locations from SpamTrees and the corresponding posterior uncertainties. Multivariate models appear advantageous in predicting some, but not all outcomes in this real world illustration; nevertheless, SpamTrees outperformed a Q-MGP model using the same cross-covariance function. Univariate models perform well and remain valid for predictions, but cannot estimate multivariate relationships. We report posterior summaries of θ in Appendix C.2.2. Opposite signs of σ_i1 and σ_j1 for pairs of variables $i,j\in \left\{1,\dots ,q\right\}$ imply a negative relationship; however, the degree of spatial decay of these correlations is different for each pair as prescribed by the latent distances in the domain of variables δ_ij. Figure 7 depicts the resulting cross-covariance function for three pairs of variables.

Table 2:

Prediction results over the 3 × 3 degree area shown in Figure 5

MODIS/GHCN variables		Multivariate		Univariate
		SpamTree	Q-MGP	SpamTree	NNGP
Clear_sky_days	RMDE	1.611	1.928	1.466	1.825
	COVG	0.980	0.866	0.984	0.986
Clear_sky_nights	RMDE	1.621	1.766	2.002	2.216
	COVG	0.989	0.943	0.992	0.992
LST_Day_CMG	RMDE	1.255	1.699	1.645	1.666
	COVG	1.000	1.000	1.000	1.000
LST_Night_CMG	RMDE	1.076	1.402	0.795	1.352
	COVG	0.999	0.999	1.000	1.000
PRCP	RMDE	0.517	0.632	0.490	0.497
	COVG	0.972	1.000	0.969	0.958

Figure 6:

Predicted values of the outcomes at all locations (top row) and associated 95% uncertainty (bottom row), with darker spots corresponding to wider credible intervals.

Figure 7:

Given the latent dimensions δ_ij, the color-coded lines represent $C\left(\mathit{h},{\delta }_{ij}\right)$ whereas ${\mathit{C}}_{ij}\left(\mathit{h}\right)={\sigma }_{1i}{\sigma }_{1j}C\left(\mathit{h},{\delta }_{ij}\right)$ is shown as a dashed grey line.

5. Discussion

In this article, we introduced SpamTrees for Bayesian spatial multivariate regression modeling and provided algorithms for scalable estimation and prediction. SpamTrees add significantly to the class of methods for regression in spatially-dependent data settings. We have demonstrated that SpamTrees maintain accurate characterization of spatial dependence and scalability even in challenging settings involving multivariate data that are spatially misaligned. Such complexities create problems for competing approaches, including recent DAG-based approaches ranging from NNGPs to MGPs.

One potential concern is the need for users to choose a tree, and in particular specify the number of locations associated to each node and the multivariate composition of locations in each node. Although one can potentially estimate the tree structure based on the data, this would eliminate much of the computational speedup. We have provided theoretical guidance based on KL divergence from the full GP and computational cost associated to different tree structures. This and our computational experiments lead to practical guidelines that can be used routinely in tree building. Choosing a tree provides a useful degree of user-input to refine and improve upon an approach.

We have focused on sampling algorithms for the latent effects because they provide a general blueprint which may be used for posterior computations in non-Gaussian outcome models; efficient algorithms for non-Gaussian big geostatistical data sets are currently lacking and are the focus of ongoing research. SpamTrees can be built on larger dimensional inputs for general applications in regression and/or classifications; such a case requires special considerations regarding domain partitioning and the construction of the tree. In particular, when time is available as a third dimension it may be challenging to build a sparse DAG with reasonable assumptions on temporal dependence. For these reasons, future research may be devoted to building sparse DAG methods combining the advantages of treed structures with e.g. Markov-type assumptions of conditional independence, and applying SpamTrees to data with larger dimensional inputs.

Figure 8:

Posterior means and 95% credible intervals for components of θ for SpamTrees.

Appendix A. Kolmogorov consistency conditions for SpamTrees

We adapt results from Datta et al. (2016a) and Peruzzi et al. (2020). Let $w\left(\mathit{s}\right),\mathit{s}\in {\mathcal{D}}^{\ast }$ be the univariate representation in the augmented domain of the multivariate base process $\left\{\mathit{w}\left(\ell \right),\ell \in \mathcal{D}\subset |{\Re }^d\right\}$. Fix the reference set $\mathcal{S}\subset {\mathcal{D}}^{*}$ and let $\mathcal{L}=\left\{{\ell }_1,\dots ,{\ell }_n\right\}\subset {\mathcal{D}}^{*}$ and $\mathcal{U}=\mathcal{L}\backslash \mathcal{S}$. Then

$$\begin{array}{l}\int \tilde{p}\left({\mathit{w}}_{\mathcal{L}}\right)\prod _{{\mathit{l}}_i\in \mathcal{L}}dw\left({\ell }_i\right)=\int \int \tilde{p}\left({\mathit{w}}_{\mathcal{U}}|{\mathit{w}}_{\mathcal{S}}\right)\tilde{p}\left({\mathit{w}}_{\mathcal{S}}\right)\prod _{{\mathit{s}}_i\in \mathcal{S}\backslash \mathcal{L}}dw\left({\mathit{s}}_i\right)\prod _{{\mathit{l}}_i\in \mathcal{L}}dw\left({\ell }_i\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\int \tilde{p}\left({\mathit{w}}_{\mathcal{S}}\right)\left(\int \tilde{p}\left({\mathit{w}}_{\mathcal{U}}|{\mathit{w}}_{\mathcal{S}}\right)\prod _{{\mathit{l}}_i\in \mathcal{U}}dw\left({\ell }_i\right)\right)\prod _{{\mathit{l}}_i\in \mathcal{S}}dw\left({\ell }_i\right)=1,\end{array}$$

hence $\tilde{p}\left({\mathit{w}}_{\mathcal{L}}\right)$ is a proper joint density. To verify the Kolmogorov consistency conditions, take the permutation ${\mathcal{L}}_{\pi }=\left\{{\ell }_{\pi \left(1\right)},\dots ,{\ell }_{\pi \left(n\right)}\right\}$ and call ${\mathcal{U}}_{\pi }={\mathcal{L}}_{\pi }\backslash \mathcal{S}$. Clearly ${\mathcal{U}}_{\pi }={\mathcal{L}}_{\pi }\backslash \mathcal{S}=\mathcal{L}\backslash \mathcal{S}=\mathcal{U}$ and similarly $\mathcal{S}\backslash {\mathcal{L}}_{\pi }=\mathcal{S}\backslash \mathcal{L}$ so that

$$\begin{gathered} \tilde{p}\left({\mathit{w}}_{\mathcal{L}\pi }\right)=\int \tilde{p}\left({\mathit{w}}_{{\mathcal{U}}_{\pi }}|{\mathit{w}}_{\mathcal{S}}\right)\tilde{p}\left({\mathit{w}}_{\mathcal{S}}\right)\prod _{{\mathit{s}}_i\in \mathcal{S}\backslash {\mathcal{L}}_{\pi }}dw\left({\mathit{s}}_i\right) \\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\int \tilde{p}\left({\mathit{w}}_{\mathcal{U}}|{\mathit{w}}_{\mathcal{S}}\right)\tilde{p}\left({\mathit{w}}_{\mathcal{S}}\right)\prod _{{\mathit{s}}_i\in \mathcal{S}\backslash \mathcal{L}}dw\left({\mathit{s}}_i\right)=\tilde{p}\left({\mathit{w}}_{\mathcal{L}}\right) \end{gathered}$$

Implying

$$\tilde{p}\left(\mathit{w}\left({\ell }_1\right),\dots ,\mathit{w}\left({\ell }_n\right)\right)=\tilde{p}\left(\mathit{w}\left({\ell }_{\pi \left(1\right)}\right)\right),\dots \mathit{w}\left({\ell }_{\pi \left(n\right)}\right).$$

Next, take a new location location ${\ell }_0\in {\mathcal{D}}^{*}$. Call ${\mathcal{L}}_1=\mathcal{L}\cup \left\{{\mathit{l}}_0\right\}$. We want to show that $\int \tilde{p}\left({\mathit{w}}_{{\mathcal{L}}_1}\right)dw\left({\ell }_0\right)=\tilde{p}\left({\mathit{w}}_{\mathcal{L}}\right)$.If ${\ell }_0\in \mathcal{S}$ then ${\mathcal{L}}_1\backslash \mathcal{S}=\mathcal{L}\backslash \mathcal{S}=\mathcal{U}$ and hence

$$\begin{array}{l}\int \tilde{p}\left({\mathit{w}}_{{\ell }_1}\right)dw\left({\ell }_0\right)=\int \left(\tilde{p}\left({\mathit{w}}_{{\ell }_1\backslash \mathcal{S}}|{\mathit{w}}_{\mathcal{S}}\right)\tilde{p}\left({\mathit{w}}_{\mathcal{S}}\right)\prod _{{\mathit{s}}_i\in \mathcal{S}\backslash {\ell }_1}dw\left({\mathit{s}}_i\right)\right)dw\left({\ell }_0\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\int \tilde{p}\left({\mathit{w}}_{\mathcal{U}}|{\mathit{w}}_{\mathcal{S}}\right)\tilde{p}\left({\mathit{w}}_{\mathcal{S}}\right)\prod _{{\mathit{s}}_i\in \mathcal{S}\backslash \mathcal{L}}dw\left({\mathit{s}}_i\right)=\tilde{p}\left({\mathit{w}}_{\mathcal{L}}\right).\end{array}$$

If ${\ell }_0\notin \mathcal{S}$ we have

$$\begin{array}{l}\int \tilde{p}\left({\mathit{w}}_{{\mathcal{L}}_1}\right)dw\left({\mathcal{L}}_0\right)=\int \left(\int \tilde{p}\left({\mathit{w}}_{{\mathcal{L}}_1}{{}_{\backslash }}_{\mathcal{S}}|{\mathit{w}}_{\mathcal{S}}\right)\tilde{p}\left({\mathit{w}}_{\mathcal{S}}\right)\prod _{{\mathit{s}}_i\in \mathcal{S}\backslash {\ell }_1}dw\left({\mathit{s}}_i\right)\right)dw\left({\mathcal{L}}_0\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\int \left(\int \tilde{p}\left({\mathit{w}}_{\mathcal{L}\backslash \mathcal{S}}{}_{\cup \left\{{\mathcal{L}}_0\right\}}|{\mathit{w}}_{\mathcal{S}}\right)\tilde{p}\left({\mathit{w}}_{\mathcal{S}}\right)\prod _{{\mathit{s}}_i\in \mathcal{S}\backslash \mathcal{L}}dw\left({\mathit{s}}_i\right)\right)dw\left({\mathcal{L}}_0\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\int \left(\int \tilde{p}\left({\mathit{w}}_{\left\{{\mathcal{L}}_0\right\}}|{\mathit{w}}_{\ell \backslash \mathcal{S}},{\mathit{w}}_{\mathcal{S}}\right)\tilde{p}\left({\mathit{w}}_{\mathcal{L}\backslash \mathcal{S}}|{\mathit{w}}_{\mathcal{S}}\right)\tilde{p}\left({\mathit{w}}_{\mathcal{S}}\right)\prod _{{\mathit{s}}_i\in \mathcal{S}\backslash \mathcal{L}}dw\left({\mathit{s}}_i\right)\right)dw\left({\mathcal{L}}_0\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\int \tilde{p}\left({\mathit{w}}_{\mathcal{L}\backslash \mathcal{S}}|{\mathit{w}}_{\mathcal{S}}\right)\tilde{p}\left({\mathit{w}}_{\mathcal{S}}\right)\prod _{{\mathit{s}}_i\in \mathcal{S}\backslash \mathcal{L}}dw\left({\mathit{s}}_i\right)\int \tilde{p}\left({\mathit{w}}_{\left\{{\mathit{l}}_0\right\}}|{\mathit{w}}_{\mathcal{S}}\right)dw\left({\mathcal{L}}_0\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\int \tilde{p}\left({\mathit{w}}_{\mathcal{L}\backslash \mathcal{S}}|{\mathit{w}}_{\mathcal{S}}\right)\tilde{p}\left({\mathit{w}}_{\mathcal{S}}\right)\prod _{{\mathit{s}}_i\in \mathcal{S}\backslash \mathcal{L}}dw\left({\mathit{s}}_i\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\tilde{p}\left({\mathit{w}}_{\mathcal{L}}\right).\end{array}$$

Appendix B. Properties of Gaussian SpamTrees

Consider the treed graph $\mathcal{G}$ of a SpamTree. In this section, we make no distinction between reference and non-reference nodes, and instead label V_i = A_i for i = 0,…, M −1 and V _M = B so that V = {A, B} = {V ₀,…, V _M−1, V _M } and the V _M are the leaf nodes. Each w_i is n_i × 1 and corresponds to v_i ∈ V _r for some r = 0,…, M so that $\text{Pa}\left[{\mathit{v}}_i\right]=\left\{{\mathit{v}}_{j_1},\mathrm{...},{\mathit{v}}_{j_r}\right\}$ for some sequence {j₁,…, j_r}, and ${\eta }^{-1}\left(\text{pa}\left[{\mathit{v}}_i\right]\right)=\left\{{\mathit{s}}_{i_1},\mathrm{...},{\mathit{s}}_{j_r}\right\}$. Denote the h-th parent of v_i as Pa[v_i](h).

B.1. Building the precision matrix

We can represent each conditional density $N\left({\mathit{w}}_i|{\mathit{H}}_i{\mathit{w}}_{\left[i\right]},{\mathit{R}}_i\right)$ as a linear regression on w_i:

$${\mathit{w}}_0={\mathit{\omega }}_0~N\left(\text{0},{\mathit{R}}_0\right)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\mathit{w}}_i=\sum _{\left\{j:{\mathit{v}}_j\in \text{Pa}\left[{\mathit{v}}_i\right]\right\}}{h}_{ij}{\mathit{w}}_i+{\omega }_i,\hspace{0.17em}\hspace{0.17em}i=1,2,\mathrm{...},M,$$ (19)

where each h_ij is an n_i × n_j coefficient matrix representing the regression of w_i given ${\mathit{w}}_i,{\omega }_i\stackrel{ind}{~}N\left(0,{\mathit{R}}_i\right)$ for i = 0, 1,…, M , and each R_i is an n_i × n_i residual covariance matrix. We set h_ii = O and h_ij = O, where O is the matrix of zeros, whenever $j\notin \left\{j_1,\mathrm{...},j_r\right\}$. Using this representation, we have $H_i=\left[h_{i,}{}_{j_1},h_i{,}_{j_2},\mathrm{...},h_i{,}_{j_r}\right],$ which is an n_i × J_i block matrix formed by stacking h_i,jk side by side for k = 1,…, r. Since $\text{E}\left[{\mathit{w}}_i|{\mathit{w}}_{\left[i\right]}\right]=H_i{\mathit{w}}_{\left[i\right]}={\mathit{C}}_{i,}{}_{\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}{\mathit{w}}_{\left[i\right]},$ we obtain ${\mathit{H}}_i={\mathit{C}}_{i,}{}_{\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}$. We also obtain ${\mathit{R}}_i=\mathrm{var}\left\{{\mathit{w}}_i|{\mathit{w}}_{\left[i\right]}\right\}={\mathit{C}}_{i,i}-{\mathit{C}}_{i,\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}{\mathit{C}}_{\left[i\right],i}$, hence all H_i’s, h_ij’s, and R_i’s can be computed from the base covariance function.

In order to continue building the precision matrix, define the block matrix $\mathcal{H}=\left\{h_{ij}\right\}$.

We can write

$${\mathit{h}}_{ij}=\left\{\begin{array}{ll}\mathit{O}\hfill & \text{if}\hspace{0.17em}{\mathit{v}}_j\notin \hspace{0.17em}\text{Pa}\left[{\mathit{v}}_i\right]\hfill \\ \left({\mathit{C}}_{i,\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}\right)\left(\cdot ,\hspace{0.17em}h\right)={\mathit{H}}_i\left(\cdot ,h\right)\hfill & \text{if}\hspace{0.17em}{\mathit{v}}_j={\mathit{v}}_{j_h}\in \text{Pa}\left[{\mathit{v}}_i\right]\hfill \end{array}\right.$$ (20)

where $\left(\cdot ,h\right)$ refers to the h-th block column. More compactly using the indicator function $1\left\{\cdot \right\}$ we have ${\mathit{h}}_{ij}=1\left\{\exists h:{\mathit{v}}_j=\text{Pa}\left[{\mathit{v}}_i\right]\left(h\right)\right\}\left({\mathit{C}}_{i,\left[i\right]}{\mathit{C}}_i^{-1}\right)\left(\cdot ,h\right)$. If we stack all the h_ik horizontally for k = 0,…, M_S − 1, we obtain the n_i × n matrix $\mathcal{H}(i,\cdot )$, which is i-th block row of $\mathcal{H}$. Intuitively, $\mathcal{H}\left(i,\cdot \right)$ is a sparse matrix with the coefficients linking the full w to w_i, with zero blocks at locations whose corresponding node is v_j ∈/ Pa[v_i]. The ith block-row of $\mathcal{H}$ is of size n_i × n but only has r non-empty sub-blocks, with sizes n_i × n_j for j ∈ {j₁,…, j_r}, respectively. Instead, H_i is a dense matrix obtained by dropping all the zero-blocks from $\mathcal{H}\left(i,\cdot \right)$, and stores the coefficients linking w_[i] to w_i. The two are linked as ${\mathit{H}}_i{\mathit{w}}_{\left[i\right]}=\mathcal{H}\left(i,\cdot \right)\mathit{w}$.

Since $\mathit{w}=\mathcal{H}\mathit{w}+\mathit{\omega },\tilde{\mathit{C}}=\mathrm{var}\left(\mathit{w}\right)={\left(\mathit{I}-\mathcal{H}\right)}^{-1}\mathit{R}{\left(\mathit{I}-\mathcal{H}\right)}^{-\top }$, where R = b.diag{R_i} and $\mathit{I}-\mathcal{H}$ is block lower-triangular with unit diagonal, hence non-singular. We find the precision matrix as ${\tilde{\mathit{C}}}^{-1}={\left(\mathit{I}-\mathcal{H}\right)}^{\top }{\mathit{R}}^{-1}\left(\mathit{I}-\mathcal{H}\right)$.

B.2. Properties of ${\tilde{\mathit{C}}}^{-1}$

When not ambiguous, we use the notation X_ij to denote the (i, j) block of X. An exception to this is the (i, j) block of ${\tilde{\mathit{C}}}^{-1}$ which we denote as ${\tilde{\mathit{C}}}^{-1}\left(i,j\right)$. In SpamTrees, ${\tilde{\mathit{C}}}^{-1}\left(i,j\right)$ is nonzero if i = j or if the corresponding nodes v_i and v_j are connected in the moral graph ${\mathcal{G}}^m$, which is an undirected graph based on $\mathcal{G}$ in which an edge connects all nodes that share a child. This means that either (1) v_i ∈ Pa[v_j] or vice-versa, or (2) there exists a^∗ such that {v_i, v_j} ⊂ Pa[a^∗]. In SpamTrees,${\mathcal{G}}^m=\mathcal{G}$. In fact, suppose there is a node a^∗ ∈ v_r∗ such that a^∗ ∈ Ch[v_j] ∩ Ch[v_k], where ${\mathit{v}}_j\in {\mathit{v}}_{r_k}and\hspace{0.17em}{\mathit{v}}_k\in {\mathit{v}}_{r_k}$. By definition of $\mathcal{G}$ there exists a sequence {i₁,…, i_r∗ } such that $\text{Pa}\left[a^{*}\right]=\left\{{\mathit{v}}_{i_1},\mathrm{...},{\mathit{v}}_{i_{r^{*}}}\right\}\supset \left\{{\mathit{v}}_j,{\mathit{v}}_k\right\}$, and furthermore $\text{Pa}\left[{\mathit{v}}_{i_h}\right]=\left\{{\mathit{v}}_{i_1},\mathrm{...},{\mathit{v}}_{i_{h--1}}\right\}$ for h ≤ r^∗. This implies that if j = k then v_j = v_k, whereas if j > k then v_k ∈ Pa[v_j], meaning that no additional edge is necessary to build ${\mathcal{G}}^m$.

B.2.1. Explicit Derivation of ${\tilde{\mathit{C}}}^{-1}\left(i,j\right)$

Denote ${\mathit{R}}^{-1}={\mathit{R}}^{-\frac{1}{2}}{\mathit{R}}^{-\frac{\top }{2}},\mathit{U}={\left(\mathit{I}-\mathcal{H}\right)}^{\top }{\mathit{R}}^{-\frac{1}{2}}$, and define the “common descendants” as cd(v_i, v_j) = ({v_i} ∪ Ch[v_i]) ∩ ({v_j} ∪ Ch[v_j]). Then consider a_i ∈ A, v_j ∈ V such that a_i ∈ Pa[v_j] and denote as H_i→j the matrix obtained by subsetting H_j to columns corresponding to a_i and note that ${\mathit{H}}_{i\to j}={\mathcal{H}}_{ji}$. The (i, j) block of U is then

$${\mathit{U}}_{ij}=\left\{\begin{array}{ll}{\mathit{O}}_{n_i\times n_j}\hfill & \text{if}\hspace{0.17em}{\mathit{v}}_j\notin \hspace{0.17em}\text{Ch}\left[{\mathit{v}}_i\right]\hfill \\ {\mathit{I}}_{n_i\times n_i}\hfill & \text{if}\hspace{0.17em}i=j\hfill \\ \begin{array}{l}{\left({\mathit{I}}_{ji}-{\mathit{H}}_{i\to j}\right)}^{\top }{\mathit{R}}_j^{-\frac{1}{2}}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.33em}\hspace{0.17em}={\left({\mathit{I}}_{ji}-{\mathcal{H}}_{ji}\right)}^{\top }{\mathit{R}}_j^{-\frac{1}{2}}\end{array}\hfill & \text{if}\hspace{0.17em}{\mathit{v}}_j\in \hspace{0.17em}\text{Ch}\left[{\mathit{v}}_i\right]\hfill \end{array}\right.$$

Then ${\tilde{\mathit{C}}}^{-1}\left(i,j\right)={\sum }_k{\mathit{U}}_{ik}{\mathit{U}}_{jk}^{\top }$ and, as in (9), each block of the precision matrix is:

$$\begin{gathered} {\tilde{\mathit{C}}}^{-1}\left(i,j\right)=\sum _{{\mathit{v}}_k\in cd\left({\mathit{v}}_{i,}{\mathit{v}}_j\right)}\hspace{0.17em}{\left({\mathit{I}}_{ki}-{\mathit{H}}_{i\to k}\right)}^{\top }{\mathit{R}}_k^{-1}\left({\mathit{I}}_{kj}-{\mathit{H}}_{j\to k}\right) \\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\sum _{{\mathit{v}}_k\in cd\left({\mathit{v}}_{i,}{\mathit{v}}_j\right)}\hspace{0.17em}{\left({\mathit{I}}_{ki}-{\mathit{H}}_{ki}\right)}^{\top }{\mathit{R}}_k^{-1}\left({\mathit{I}}_{kj}-{\mathit{H}}_{kj}\right) \end{gathered}$$ (21)

where $\text{cd}\left({\mathit{v}}_i,{\mathit{v}}_j\right)=\cancel{0}$ implies ${\tilde{\mathit{C}}}^{-1}\left(i,j\right)=\mathit{O}$ and I_ij a zero matrix unless i = j as it is the (i, j) block of an identity matrix of dimension n × n.

B.2.2. Computation of Large Matrix Inverses

One important aspect in building ${\tilde{\mathit{C}}}^{-1}$ is that it requires the computation of the inverse ${\tilde{\mathit{C}}}_{\left[i\right]}^{-1}$ of dimension J_i × J_i for all nodes with parents, i.e. at r > 0. Unlike models which achieve scalable computations by limiting the size of the parent set (e.g. NNGPs and their blocked variant, or tessellated MGPs), this inverse is increasingly costlier when δ > 1 for nodes at a higher-level of the tree as those nodes have more parents and hence larger sets of parent locations (the same conclusion holds for non-reference nodes). However, the treed structure in $\mathcal{G}$ allows one to avoid computing the inverse in $O\left(J_i^3\right)$. In fact, suppose we have a symmetric, positive-definite block-matrix A and we wish to compute its inverse. We write

$$\mathit{A}=\left[\begin{array}{cc}C& B\\ B^{\top }& D\end{array}\right]\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\mathit{A}}^{-1}=\left[\begin{array}{cc}{C}^{-1}+C^{-1}B{S}^{-1}{B}^{\top }{C}^{-1}& -C^{-1}B{S}^{-1}\\ -S^{-1}{B}^{\top }{C}^{-1}& S^{-1}\end{array}\right],$$

where $S=C-B{D}^{-1}{B}^{\top }$ is the Schur complement of D in A. If C⁻¹ was available, the only necessary inversion is that of S. In SpamTrees with δ > 1, suppose v_i, v_j are two nodes such that $\text{Pa}\left[{\mathit{v}}_j\right]=\left\{{\mathit{v}}_i\right\}\cup \text{Pa}\left[{\mathit{v}}_i\right]$ – this arises for nodes ${\mathit{v}}_j\in {\mathit{V}}_{r,}r\ge M_{\delta }$. Regardless of whether v_j is a reference node or not, η⁻¹(Pa[v_j]) = {S_i, S_[i]} and

$${\mathit{C}}_{\left[j\right]}=\left[\begin{array}{cc}{\mathit{C}}_{\left[i\right]}& {\mathit{C}}_{\left[i\right],i}\\ {\mathit{C}}_{i,\left[i\right]}& {\mathit{C}}_i\end{array}\right],\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\mathit{C}}_{\left[j\right]}^{-1}=\left[\begin{array}{cc}{\mathit{C}}_i^{-1}+{\mathit{C}}_{\left[i\right]}^{-1}{\mathit{C}}_{\left[i\right],i}{S}^{-1}{\mathit{C}}_{i,\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}& -{\mathit{C}}_{\left[i\right]}^{-1}{\mathit{C}}_{\left[i\right],i}{S}^{-1}\\ -S^{-1}{\mathit{C}}_{i,\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}& S^{-1}\end{array}\right],$$

where the Schur complement of $C_i$ is $S={\mathit{C}}_i-{\mathit{C}}_{i,\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}{\mathit{C}}_{\left[i\right],i}={\mathit{R}}_i$. Noting that ${\mathit{H}}_i={\mathit{C}}_{i,\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}$ we write

$${\mathit{C}}_{\left[j\right]}^{-1}=\left[\begin{array}{cc}{\mathit{C}}_{\left[j\right]}^{-1}+{\mathit{H}}_i^{\top }{\mathit{R}}_i^{-1}{\mathit{H}}_i& -{\mathit{H}}_i^{\top }{\mathit{R}}_i^{-1}\\ -{\mathit{R}}_i^{-1}{\mathit{H}}_i& {\mathit{R}}_i^{-1}\end{array}\right].$$ (22)

B.2.3. Computing ${\left({\tilde{C}}^{-1}+\sum \right)}^{-1}$ and its determinant without sparse Cholesky

Bayesian estimation of regression models requiring the computation of ${\left(\mathit{Z}\tilde{C}{\mathit{Z}}^{\top }+\mathit{D}\right)}^{-1}$ and its determinant use the Sherman-Morrison-Woodbury matrix identity to find ${\left(\mathit{Z}\tilde{C}{\mathit{Z}}^{\top }+\mathit{D}\right)}^{-1}={\mathit{D}}^{-1}-{\mathit{D}}^{-1}\mathit{Z}{\left({\tilde{\mathit{C}}}^{-1}+\sum \right)}^{-1}{\mathit{Z}}^{\top }{\mathit{D}}^{-1}$, where $\sum ={\mathit{Z}}^{\top }{\mathit{D}}^{-1}\mathit{Z}$. A sparse Cholesky factorization of ${\tilde{\mathit{C}}}^{-1}+\sum$ can be used as typically Σ is diagonal or block-diagonal, thus maintaining the sparsity structure of ${\tilde{\mathit{C}}}^{-1}$. Sparse Cholesky libraries (e.g. Cholmod, Chen et al., 2008), which are embedded in software or high-level languages such as MATLAB™ or the Matrix package for R, scale to large sparse matrices but are either too flexible or too restrictive in our use cases: (1) we know $\mathcal{G}$ and its properties in advance; (2) SpamTrees take advantage of block structures and grouped data. In fact, sparse matrix libraries typically are agnostic of $\mathcal{G}$ and heuristically attempt to infer a sparse $\mathcal{G}$ given its moralized counterpart. While this operation is typically performed once, a priori knowledge of $\mathcal{G}$ implies that reliance on such libraries is in principle unnecessary.

We thus take advantage of the known structure in $\mathcal{G}$ to derive direct algorithms for computing ${\left({\tilde{\mathit{C}}}^{-1}+\sum \right)}^{-1}$ and its determinant. In the discussion below we consider δ = M , noting here that choosing δ = 1 simplifies the treatment as cd(v_i, v_j) = v_i if v_i = v_j, and it is empty otherwise. We now show how (21) leads to Algorithm 4 for the decomposition of any precision matrix Λ which conforms to $\mathcal{G}$ – i.e. it has the same block-sparse structure as a precision matrix built as in Section B.1. Suppose from Λ we seek a block lower-triangular matrix L and a block diagonal D such that

$${\text{Λ}}_{ij}=\sum _{{\mathit{v}}_k\in \text{cd}\left({\mathit{v}}_i,{\mathit{v}}_j\right)}{\left({\mathit{I}}_{ki}-{\mathit{L}}_{ki}\right)}^{\top }{\mathit{D}}_k\left({\mathit{I}}_{kj}-{\mathit{L}}_{kj}\right)$$

Start with v_i, v_j taken from the leaf nodes, i.e. ${\mathit{v}}_i,{\mathit{v}}_j\in {\mathit{V}}_M$. Then $\text{cd}\left(v_i,v_j\right)=\cancel{0}$ and we set ${\mathit{L}}_{ij}=\mathit{L}{\left(j,i\right)}^{\top }=\mathit{O}={\text{Λ}}_{ij}$. If i = j then cd(v_i, v_i) = {v_i} and

$$\begin{array}{l}\sum _{{\mathit{v}}_k\in \text{cd}\left({\mathit{v}}_{i,}{\mathit{v}}_j\right)}{\left({\mathit{I}}_{ki}-{\mathit{L}}_{ki}\right)}^{\top }{\mathit{D}}_k\left({\mathit{I}}_{ki}-{\mathit{L}}_{ki}\right)={\left({\mathit{I}}_{ii}-{\mathit{L}}_{ii}\right)}^{\top }{\mathit{D}}_i\left({\mathit{I}}_{ii}-\mathit{L}\left(i,i\right)\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={\mathit{D}}_i-{\mathit{D}}_i{\mathit{L}}_{ii}-{\mathit{L}}_{ii}^{{}^{\top }}{\mathit{D}}_i+{\mathit{L}}_{ii}^{{}^{\top }}{\mathit{D}}_i{\mathit{L}}_{ii};\end{array}$$

we then set L_ii = O and get the i-th block of D_i simply setting D_i = Λ_ii. Proceeding downwards along $\mathcal{G}$, if ${\mathit{v}}_j\in {\mathit{V}}_{M-1}\cap \text{pa}\left[{\mathit{v}}_i\right]$ we have ${\text{Λ}}_{ij}={\mathit{D}}_i\left({\mathit{I}}_{ij}-{\mathit{L}}_{ij}\right)=-{\mathit{D}}_i{\mathit{L}}_{ij}$ and thus set ${\mathit{L}}_{ij}=-{\mathit{D}}_i^{-1}{\text{Λ}}_{ij}$. We then note that cd(v_j, v_j) = {v_j, v_i} and obtain ${\text{Λ}}_{jj}={\mathit{D}}_j+{\mathit{L}}_{ij}^{\top }{\mathit{D}}_i{\mathit{L}}_{ij}$ where L_ij and D_i have been fixed at the previous step; this results in ${\mathit{D}}_j={\text{Λ}}_{jj}-{\mathit{L}}_{ij}^{\top }{\mathit{D}}_i{\mathit{L}}_{ij}$.

Then, the s-th (of M ) step takes ${\mathit{v}}_j\in {\text{V}}_{M-s}\cap \text{Pa}\left[{\mathit{v}}_i\right]$ and ${\mathit{v}}_i\in {\text{V}}_{M-s+1,}$, implying $\text{cd}\left({\mathit{v}}_i,{\mathit{v}}_j\right)=\left\{{\mathit{v}}_i\right\}\cup \text{ch}\left[{\mathit{v}}_i\right]$. Nothing that ${\mathit{F}}^{*}={\sum }_{{\mathit{v}}_k\in \text{ch}\left[{\mathit{v}}_i\right]}{\left({\mathit{I}}_{ki}-{\mathit{L}}_{ki}\right)}^{\top }{\mathit{D}}_k\left({\mathit{I}}_{kj}-{\mathit{L}}_{kj}\right)$ has been fixed at previous steps since each v_k is at level M−s + 2, we split the sum in (21) and get

$${\text{Λ}}_{ij}-{\mathit{F}}^{\ast }={\mathit{D}}_i\left({\mathit{I}}_{ij}-{\mathit{L}}_{ij}\right)=-{\mathit{D}}_i{\mathit{L}}_{ij},$$

where D_i has been fixed at step s−1, obtaining ${\mathit{L}}_{ij}=-{\mathit{D}}_i^{-1}\left({\text{Λ}}_{ij}-{\mathit{F}}^{\ast }\right)$; D_j can be found using the same logic. Proceeding until M − s = 0 from the leaves of $\mathcal{G}$ to the root, we ultimately fill each non-empty block in L and D resulting in $\text{Λ}={\left(\mathit{I}-\mathit{L}\right)}^{\top }\mathit{D}\left(\mathit{I}-\mathit{L}\right)$. Algorithm 4 unifies these steps to obtain the block decomposition of any sparse precision matrix Λ conforming to $\mathcal{G}$ resulting in $\text{Λ}={\left(\mathit{I}-\mathit{L}\right)}^{\top }\mathit{D}\left(\mathit{I}-\mathit{L}\right)$, where L is block lower triangular and D is block diagonal. This is akin to a block-LDL decomposition of Λ indexed on nodes of $\mathcal{G}$. Algorithm 5 complements this decomposition by providing a $\mathcal{G}$-specific block version of forward substitution for computing (I−L)⁻¹ with L as above.

In practice, a block matrix with K² blocks can be represented as a K² array with rows and columns indexed by nodes in $\mathcal{G}$ and matrix elements which may be zero-dimensional whenever corresponding to blocks of zeros. The specification of all algorithms in block notation allows us to never deal with large (sparse) matrices in practice but only with small block matrices indexed by nodes in $\mathcal{G}$, bypassing the need for external sparse matrix libraries. Specifically we use the above algorithms to compute ${\text{Λ}}^{-1}={\left({\tilde{\mathit{C}}}^{-1}+\text{Σ}\right)}^{-1}$ and its determinant: ${\text{Λ}}^{-1}={\left(\mathit{I}-\mathit{L}\right)}^{-1}{\mathit{D}}^{-1}{\left(\mathit{I}-\mathit{L}\right)}^{-\top }$ and $\left|{\text{Λ}}^{-1}\right|={\prod }_{i=1}^{M_S}1/\left|{\tilde{\mathit{D}}}_i\right|$. We have not distinguished non-reference and reference nodes in this discussion. In cases in which the non-reference set is large, we note that the conditional independence of all non-reference locations, given their parents, results in ${\tilde{\mathit{C}}}^{-1}\left(i,i\right)$ being diagonal for all $\ell \in \mathcal{U}$ (i.e. η(ℓ) = v_i ∈ B). This portion of the precision matrix can just be stored as a column vector.

B.2.4. Sparsity of ${\tilde{\mathit{C}}}^{-1}$

We calculate the sparsity in the precision matrix; considering an enumeration of nodes by level in $\mathcal{G}$, denote $n_{ij}=\left|{\eta }^{-1}\left({\mathit{v}}_{ij}\right)\right|$, $m_j=\left|{\mathit{V}}_j\right|$, and $J_{ij}=\left|{\eta }^{-1}\left(\text{Pa}\left[{\mathit{v}}_{ij}\right]\right)\right|$, and noting that by symmetry ${\left({\tilde{\mathit{C}}}^{-1}\left(i,j\right)\right)}^{\top }={\tilde{\mathit{C}}}^{-1}\left(j,i\right)$, the number of nonzero elements of ${\tilde{\mathit{C}}}^{-1}$ is

$$\text{nnz}\left({\tilde{\mathit{C}}}^{-1}\right)=\sum _{j=0}^M\sum _{i=1}^{m_j}\left(2n_{ij}{J}_{ij}+n_{ij}^2\text{1}\left\{j<M\right\}+n_{ij}\text{1}\left\{j=M\right\}\right),$$

Algorithm 4: Precision matrix decomposition given treed graph $\mathcal{G}$ with M levels.

Algorithm 5: Calculating the inverse of I − L with L output from Algorithm 4.

where n_ij1 {j = M} refers to the diagonal elements of the precision matrix at non-reference locations.

B.3. Properties of SpamTrees with δ = M

We outline recursive properties of $\tilde{C}$ induced by $\mathcal{G}$ when δ = M . In the case 1 < δ < M , these properties hold for nodes at or above level M_δ, using $A_{M_{\delta }}$ as root. We focus on paths in $\mathcal{G}$. These can be represented as sequences of nodes $\left\{v_{i_1},\dots ,v_{i_r}\right\}$ such that $\left\{v_{i_j},\dots ,v_{i_k}\right\}\subset Pa\left[v_{i_{k+1}}\right]$ for 1 < j < k < r. Take two successive elements of such a sequence, i.e. v_i, v_j such that v_i → v_j in $\mathcal{G}$. Consider $E\left[w_j|w_{[j]}\right]={\mathit{H}}_j{\mathit{w}}_{\left[j\right]}={\mathit{C}}_{j,\left[j\right]}{C}_{\left[j\right]}^{-1}{\mathit{w}}_{\left[j\right]}$$ET$ and $R_j=\mathrm{var}\left\{{\mathit{w}}_j|{\mathit{w}}_{\left[j\right]}\right\}={\mathit{C}}_{j,j}-{\mathit{C}}_{j,\left[j\right]}{\mathit{C}}_{\left[j\right]}^{-1}{\mathit{C}}_{\left[j\right]}{}_{,j}$. By (22) we can write

$$\begin{array}{l}{\mathit{H}}_j{\mathit{w}}_{\left[j\right]}=\left[{\mathit{C}}_{j,\left[i\right]}{\mathit{C}}_{j,i}\right]\left[\begin{array}{cc}{\mathit{C}}_{\left[i\right]}^{-1}+{\mathit{H}}_i^T{\mathit{R}}_i^{-1}{\mathit{H}}_i& -{\mathit{H}}_i^T{\mathit{R}}_i^{-1}\\ -{\mathit{R}}_i^{-1}{\mathit{H}}_i& {\mathit{R}}_i^{-1}\end{array}\right]\left[\begin{array}{c}{\mathit{w}}_{\left[i\right]}\\ {\mathit{w}}_i\end{array}\right]\\ =\left[{\mathit{C}}_{j,\left[i\right]}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\mathit{C}}_{j,i}-{\mathit{C}}_{j,\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}{\mathit{C}}_{\left[i\right],i}\right]\left[\begin{array}{cc}{\mathit{C}}_{\left[i\right]}^{-1}& \mathit{O}\\ \mathit{O}& {\left({\mathit{C}}_{i,i}-{\mathit{C}}_{i,\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}{\mathit{C}}_{\left[i\right],i}\right)}^{-1}\end{array}\right]\left[\begin{array}{c}{\mathit{w}}_{\left[i\right]}\\ {\mathit{w}}_i-{\mathit{C}}_{i,\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}{\mathit{w}}_{\left[i\right]}\end{array}\right]\\ =\left[{\mathit{C}}_{j,\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}\left({\mathit{C}}_{j,i}-{\mathit{C}}_{j,\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}{\mathit{C}}_{\left[i\right],i}\right){\left({\mathit{C}}_{i,i}-{\mathit{C}}_{i,\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}{\mathit{C}}_{\left[i\right],i}\right)}^{-1}\right]\left[\begin{array}{c}{\mathit{w}}_{\left[i\right]}\\ {\mathit{w}}_i-{\mathit{C}}_{i,\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}{\mathit{w}}_{\left[i\right]}\end{array}\right].\end{array}$$

Now define the covariance function ${\mathit{K}}_i\left(\ell ,{\ell }^{\prime }\right)={\mathit{C}}_{\ell ,{\ell }^{\prime }}-{\mathit{C}}_{\ell ,\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}{\mathit{C}}_{\left[i\right],{\ell }^{\prime }}$; recalling that the reference set is $\mathcal{S}={\cup }_{i=0}^{M-1}{\cup }_{j=1}^{m_j}{S}_j$ we use a shorthand notation for these subsets: K_i(S_h, S_k) = K_i(h, k). Also denote ${\mathit{e}}_i={\mathit{w}}_i-{\mathit{C}}_{i,\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}{\mathit{w}}_{\left[i\right]}$ for all i. The above expression becomes

$$\begin{array}{l}{\mathit{H}}_j{\mathit{w}}_{\left[j\right]}=\left[{\mathit{C}}_{j,\left[i\right]}{\mathit{C}}_{\left[i\right]}^{-1}{\mathit{K}}_i\left(j,i\right){\mathit{K}}_i^{-1}\left(i,i\right)\right]\left[\begin{array}{c}{\mathit{w}}_{\left[i\right]}\\ {\mathit{e}}_i\end{array}\right]\\ ={\mathit{H}}_i\mathit{w}\left[i\right]+{\mathit{K}}_i\left(j,i\right){\mathit{K}}_i^{-1}\left(i,i\right){\mathit{e}}_i;\end{array}$$

we can use this recursively on $\left\{{\mathit{v}}_{i_0},{\mathit{v}}_{i_1},\dots ,{\mathit{v}}_{i_r}\right\}$ where ${\mathit{v}}_{i_0}\in {\mathit{A}}_0$ and ${\mathit{v}}_{i_r}={\mathit{v}}_j$ and get

$$\begin{array}{l}{\mathit{H}}_j{\mathit{w}}_{\left[j\right]}=\sum _{s=i_1}^{i_{r-1}}{\mathit{K}}_s\left(j,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{e}}_s\\ E\left[{\mathit{w}}_j|{\mathit{w}}_{\left[j\right]}\right]=\sum _{s=i_1}^{i_{r-1}}{E}_{e_s}\left[{\mathit{w}}_j|{\mathit{e}}_s\right],\end{array}$$

where the expectations on the r.h.s. are taken with respect to the distributions of e_s which are Gaussian with mean zero and var{e_h} = K_h(h, h) – this is a compact expression of the conditionals governing the process as prescribed by $\mathcal{G}$. We can also write the above as $E\left({\mathit{w}}_j|{\mathit{w}}_{\left[j\right]}\right)={\sum }_{s=i_0}^{i_r}{\mathit{K}}_s\left(j,s\right){\mathit{K}}_s^{-1}\left(s,s\right)\left({\mathit{w}}_s-E\left[{\mathit{w}}_s|{\mathit{w}}_{\left[s\right]}\right]\right)$;using $E\left[{\mathit{e}}_h|{\mathit{w}}_h,{\mathit{w}}_{\left[h\right]}\right]=0$, for h < k we find

$$\begin{array}{l}\mathrm{cov}\left\{{\mathit{e}}_h,{\mathit{e}}_k\right\}=\mathrm{E}\left[\mathrm{cov}\left\{{\mathit{e}}_h,{\mathit{e}}_k|{\mathit{w}}_h,{\mathit{w}}_{\left[h\right]}\right\}\right]+\mathrm{cov}\left\{\mathrm{E}\left[{\mathit{e}}_h|{\mathit{w}}_h,{\mathit{w}}_{\left[h\right]}\right]\right\}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\mathrm{cov}\left\{\mathrm{E}\left[{\mathit{e}}_h|{\mathit{w}}_h,{\mathit{w}}_{\left[h\right]}\right],\mathrm{E}\left[{\mathit{e}}_k|{\mathit{w}}_h,{\mathit{w}}_{\left[h\right]}\right]\right\}=0.\end{array}$$

The above results also imply ${\mathit{C}}_{j,\left[j\right]}{\mathit{C}}_{\left[j\right]}^{-1}{\mathit{C}}_{\left[j\right],j}=\sum _{s=i_0}^{i_r}{\mathit{K}}_s\left(j,s\right){\mathit{K}}_s^{-1}\left(s,s\right)\mathit{K}s\left(s,j\right)$ and suggest an additive representation via orthogonal basis functions:

$${\mathit{w}}_j=\sum _{s=i_0}^{i_{r-1}}{\mathit{K}}_s\left(j,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{e}}_s+{\mathit{e}}_j$$ (23)

Finally, considering the same sequence of nodes, recursively introduce the covariance functions F ₀(r, s) = C_r,s and for $j>1,{\mathit{F}}_j\left(r,s\right)={\mathit{F}}_{j-1}\left(r,s\right)-{\mathit{F}}_{j-1}\left(r,j-1\right){\mathit{F}}_{j-1}^{-1}\left(j-1,j-1\right){\mathit{F}}_{j-1}\left(j-1,s\right)$.

We get

$$\begin{gathered} {\mathit{F}}_{j+1}\left(r,s\right)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={\mathit{F}}_j\left(r,s\right)-{\mathit{F}}_j\left(r,j\right){\mathit{F}}_j^{-1}\left(j,j\right){\mathit{F}}_j\left(j,s\right) \\ using\left(22\right)\hspace{0.17em}\hspace{0.17em}={\mathit{F}}_{j-1}\left(r,s\right)-{\mathit{F}}_{j-1}\left(r,\left[j-1:j\right]\right){\mathit{F}}_{j-1}^{-1}\left(\left[j-1:j\right],\left[j-1:j\right]\right){\mathit{F}}_{j-1}^{-1}\left(\left[j-1:j\right],s\right) \\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\mathit{C}\left(r,s\right)-\mathit{C}\left(r,\left[0:j\right]\right){\mathit{C}}^{-1}\left(\left[0:j\right],\left[0:j\right],s\right) \\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\mathit{C}\left(r,s\right)-{\mathit{C}}_{r,\left[j+1\right]}{\mathit{C}}_{\left[j+1\right]}^{-1}{\mathit{C}}_{\left[j+1\right],s} \end{gathered}$$

which can be iterated forward and results in an additional recursive way to compute co-variances in SpamTrees. Notice that while K_j is formulated using the inverse of J_j × J_j matrix C_[j], the F _j’s require inversion of smaller n_j × n_j matrices F _j−1(j-1, j-1).

B.4. Properties of $\tilde{\mathit{C}}$

B.4.1. δ = 1

Choosing depth δ = 1 results in each node having exactly 1 parent. In this case the path ${\mathcal{P}}_{k\to }{}_j=\left\{{\mathit{v}}_{i_1},\dots ,{\mathit{v}}_{i_r}\right\}$ from v_k to v_j, where ${\mathit{v}}_{i_1}={\mathit{v}}_k,{\mathit{v}}_{ir}={\mathit{v}}_j$ and $\left\{{\mathit{v}}_{i_h}\right\}=Pa\left[{\mathit{v}}_{i_{h+1}}\right]$, is unique, and there is thus no distinction between shortest and longest paths:${\mathcal{P}}_{k\to j}={\overline{\mathcal{P}}}_{k\to j}={\tilde{\mathcal{P}}}_{k\to j}$. Then denote ${\stackrel{⃛}{\mathit{H}}}_{k\to j}={\mathit{H}}_{i_r}\cdot {\mathit{H}}_{i_{r-1}}\cdots {\mathit{H}}_{i_1}$. Let v_z be the concestor between v_i and v_j i.e. ${\mathit{v}}_z=con\left({\mathit{v}}_i,{\mathit{v}}_j\right)=\arg {\max }_{v_k\in \mathit{V}}\left\{k:{\mathcal{P}}_{k\to i}\cap {\mathcal{P}}_{k\to i}\ne \cancel{0}\right\}$ and the associated paths ${\mathcal{P}}_{z\to i}=\left\{{\mathit{v}}_{i_1},\dots ,{\mathit{v}}_{i_{r_i}}\right\}$ and ${\mathcal{P}}_{z\to j}=\left\{{\mathit{v}}_{j_1},\dots ,{\mathit{v}}_{j_{r_j}}\right\}$ where ${\mathit{v}}_{i_1}={\mathit{v}}_{j_1}={\mathit{v}}_z,{\mathit{v}}_{i_{r_i}}={\mathit{v}}_i$ and ${\mathit{v}}_{j_{r_j}}={\mathit{v}}_j$. Then we can write ${\mathit{w}}_i={\mathit{w}}_{i_{r_i}}={\mathit{H}}_{i_{r_i}}{\mathit{w}}_{i_{r_i-1}}+{\mathit{v}}_{i_{r_i}}$ where ${\mathit{v}}_{i_{r_i}}\sim N\left(\text{0},{\mathit{R}}_{i_{r_i}}\right)$ and proceed expanding ${\mathit{w}}_{i_{r_i-1}}$ to get ${\mathit{w}}_{i_{r_i}}={\mathit{H}}_{i_{r_i}}\left({\mathit{H}}_{i_{r_i-1}}{\mathit{w}}_{i_{r_i-2}}+{\mathit{v}}_{{}_{i_{r_i-1}}}\right)+{\mathit{v}}_{i_{r_i}}={\mathit{H}}_{i_{r_i}}{\mathit{H}}_{i_{r_i}-1}{\mathit{w}}_{i_{r_i-2}}+\left({\mathit{H}}_{i_{r_i}}{\mathit{v}}_{i_{r_i-1}}+{\mathit{v}}_{i_{r_i}}\right)$; continuing downwardly along the tree we eventually find ${\mathit{w}}_i={\mathit{H}}_{i_{r_i}}\cdots {\mathit{H}}_{i_1}{\mathit{w}}_{i_1}+{\tilde{\mathit{v}}}_i={\stackrel{⃛}{\mathit{H}}}_{z\to i}{\mathit{w}}_z+{\tilde{\mathit{v}}}_i$ where ${\tilde{\mathit{v}}}_i$ is independent of w_z. After proceeding analogously with w_j , take ℓ_i, ℓ_j such that η(ℓ_i) = v_i and η(ℓ_j) = v_j. Then

$${\text{Cov}}_{\tilde{p}}\left(w\left({\ell }_i\right),w\left({\ell }_j\right)\right)={\stackrel{⃛}{\mathit{H}}}_{z\to i}\left({\ell }_i\right){\mathit{C}}_z{\stackrel{⃛}{\mathit{H}}}_{z\to j}{\left({\ell }_j\right)}^{\top },$$ (24)

where ${\stackrel{⃛}{\mathit{H}}}_{z\to i}\left({\ell }_i\right)=\mathit{C}\left({\ell }_i,S_i\right)C_i^{-1}{\stackrel{⃛}{\mathit{H}}}_{z\to \left[i\right]}$ and similarly for ${\stackrel{⃛}{\mathit{H}}}_{z\to j}\left({\ell }_j\right)$.

B.4.2. 1 < δ < M

Take two nodes v_i,${\mathit{v}}_j\in \mathit{V}$. If $\text{Pa}\left[{\mathit{v}}_i\right]\cap \text{Pa}\left[{\mathit{v}}_j\right]\ne \cancel{0}$ then we apply the same logic as in B.4.3 using v_z = con(v_i, v_j) as root. If Pa[v_i] ∩ Pa[v_j] = ∅ and both nodes are at levels below M_δ then we use B.4.1. The remaining scenario is thus one in which v_i ∈ A_r, r > M_δ and $\text{Pa}\left[{\mathit{v}}_i\right]\cap \text{Pa}\left[{\mathit{v}}_j\right]=\cancel{0}$. We take ${\mathit{v}}_j\in {\mathit{A}}_s$, $s<M_{\delta }$ for simplicity in exposition and without loss of generality. By (23)

$$\begin{array}{l}{\mathit{w}}_i=\sum _{s=i_{M_{\delta }}}^{i_{r-1}}{\mathit{K}}_s\left(i,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{e}}_s+{\mathit{e}}_i,\\ \hspace{1em}\hspace{0.17em}=\sum _{s=i_{M_{\delta }}+1}^{i_{r-1}}{\mathit{K}}_s\left(i,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{e}}_s+{\mathit{C}}_{ix}{\mathit{C}}_x^{-1}{\mathit{w}}_x+{\mathit{e}}_i,\end{array}$$ (25)

where ${\mathit{v}}_x\in {\mathit{A}}_{M_{\delta }}$ is the parent node of v_i at level M_δ. The final result of (14) is then achieved by noting that the relevant subgraph linking v_x and v_j has depth δ_x = 1 and thus Cov(w_x, w_j) can be found via B.4.1, then $\text{Cov}\left({\mathit{w}}_i,{\mathit{w}}_j\right)={\mathit{C}}_{ix}{\mathit{C}}_x^{-1}\text{Cov}\left({\mathit{w}}_x,{\mathit{w}}_y\right)={\mathit{F}}_i\text{Cov}\left({\mathit{w}}_x,{\mathit{w}}_y\right)$. Notice that F_i directly uses the directed edge ${\mathit{v}}_x\to {\mathit{v}}_i$ in $\mathcal{G}$; for this reason the path between w_i and ${\mathit{w}}_i=\text{con}\left({\mathit{w}}_x,{\mathit{w}}_j\right)$ is the actually the shortest path and we have ${\mathit{v}}_z\to \cdots \to {\mathit{v}}_x\to {\mathit{v}}_i$.

B.4.3. δ = M

Take v_i, ${\mathit{v}}_j\in \mathit{V}$ and the full paths from the root ${\tilde{\mathcal{P}}}_{0\to i}=\left\{i_0,\dots ,i_{r_i}\right\}$ and ${\tilde{\mathcal{P}}}_{0\to j}=\left\{j_0,\dots ,j_{r_j}\right\}$, respectively. Then using (23) we have

$$\begin{array}{l}{\mathit{w}}_i=\sum _{s\in {\tilde{\mathcal{P}}}_{0\to i}}{\mathit{K}}_s\left(i,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{e}}_s+{\mathit{e}}_i\\ \hspace{1em}\hspace{0.17em}=\sum _{s\in {\tilde{\mathcal{P}}}_{0\to i}\cap {\tilde{\mathcal{P}}}_{0\to j}}{\mathit{K}}_s\left(i,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{e}}_s+\sum _{s\in {\tilde{\mathcal{P}}}_{0\to i}\backslash {\tilde{\mathcal{P}}}_{0\to j}}{\mathit{K}}_s\left(i,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{e}}_s+{\mathit{e}}_i\\ \hspace{1em}\hspace{0.17em}=\sum _{s\in {\tilde{\mathcal{P}}}_{0\to i}\cap {\tilde{\mathcal{P}}}_{0\to j}}{\mathit{K}}_s\left(i,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{e}}_s+{\tilde{\mathit{e}}}_i\\ {\mathit{w}}_j=\sum _{s\in {\tilde{\mathcal{P}}}_{0\to j}}{\mathit{K}}_s\left(j,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{e}}_s+{\mathit{e}}_j\\ \hspace{1em}\hspace{0.17em}=\sum _{s\in {\tilde{\mathcal{P}}}_{0\to i}\cap {\tilde{\mathcal{P}}}_{0\to j}}{\mathit{K}}_s\left(j,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{e}}_s+\sum _{s\in {\tilde{\mathcal{P}}}_{0\to j}\backslash {\tilde{\mathcal{P}}}_{0\to i}}{\mathit{K}}_s\left(j,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{e}}_s+{\mathit{e}}_j\\ \hspace{1em}\hspace{0.17em}=\sum _{s\in {\tilde{\mathcal{P}}}_{0\to i}\cap {\tilde{\mathcal{P}}}_{0\to j}}{\mathit{K}}_s\left(j,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{e}}_s+{\tilde{\mathit{e}}}_j,\end{array}$$ (26)

where $\text{Cov}\left({\tilde{\mathit{e}}}_i,{\tilde{\mathit{e}}}_j\right)=0$. Then since e_s are independent and ${\mathit{e}}_s~N\left(\text{0},{\mathit{K}}_s\left(s,s\right)\right)$ we find

$$\begin{array}{l}{\text{Cov}}_{\tilde{p}}\left({\mathit{w}}_i,{\mathit{w}}_j\right)=\text{Cov}\left(\sum _{s\in {\tilde{\mathcal{P}}}_{0\to i}\cap {\tilde{\mathcal{P}}}_{0\to j}}{\mathit{K}}_s\left(i,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{e}}_s+{\mathit{e}}_i,\right.\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\left.\sum _{s\in {\tilde{\mathcal{P}}}_{0\to i}\cap {\tilde{\mathcal{P}}}_{0\to j}}{\mathit{K}}_s\left(j,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{e}}_s+{\mathit{e}}_j\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\sum _{s\in {\tilde{\mathcal{P}}}_{0\to i}\cap {\tilde{\mathcal{P}}}_{0\to j}}{\mathit{K}}_s\left(i,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{K}}_s\left(s,j\right)+{\text{1}}_{i=j}\left\{{\mathit{K}}_i\left(i,i\right)\right\}.\end{array}$$ (27)

We conclude by noting that δ = M implies ${\tilde{\mathcal{P}}}_{0\to i}\cap {\tilde{\mathcal{P}}}_{0\to j}=\text{Pa}\left[{\mathit{v}}_i\right]\cap \text{Pa}\left[{\mathit{v}}_j\right]$; considering two locations ℓ_i, ${\ell }_j\in {\mathcal{D}}^{\ast }$ such that $\eta \left({\ell }_i\right)={\mathit{v}}_i$ and $\eta \left({\ell }_j\right)={\mathit{v}}_j$ we obtain

$${\text{Cov}}_{\tilde{p}}\left(w\left({\ell }_i\right),w\left({\ell }_j\right)\right)=\sum _{s:\left\{{\mathit{v}}_s\in \text{Pa}\left[{\mathit{v}}_i\right]\cap \text{Pa}\left[{\mathit{v}}_j\right]\right\}}{\mathit{K}}_s\left({\ell }_i,s\right){\mathit{K}}_s^{-1}\left(s,s\right){\mathit{K}}_s\left(s,{\ell }_j\right)+{\text{1}}_{{\ell }_i={\ell }_j}\left\{{\mathit{K}}_i\left({\ell }_i,{\ell }_j\right)\right\}.$$

B.5. Computational cost

We make some assumptions here to simplify the calculation of overall cost: first, we assume that reference locations are all observed $\mathcal{S}\subset \mathcal{T}$, and consequently $\mathcal{U}=\mathcal{T}\backslash \mathcal{S}$. Second, we assume that all reference subsets have the same size i.e. $\left|S_i\right|=N_s$ for all i. Third, we assume all nodes have the same number of children at the next level in $\mathcal{G}$, i.e. if ${\mathit{v}}_i\in {\mathit{A}}_r$ with $r<M-1$, then $\left|\text{Ch}\left[{\mathit{v}}_i\right]\cap {\mathit{A}}_{r+1}\right|=C$, whereas if r = M − 1 then $\left|\text{Ch}\left[{\mathit{v}}_i\right]\right|=N_u$. Fourth, we assume that all non-reference subsets are singletons i.e. if ${\mathit{v}}_i\in \mathit{B}$ then |U_i|= 1. The latter two assumptions imply (5). We also fix $C{N}_s=N_u$. As a result, the number of nodes at level $r=0,\dots ,M-1$ is C^r, therefore $\left|\mathit{A}\right|+\left|\mathit{B}\right|={\sum }_{r=0}^{M-1}{C}^r+N_u{C}^{M-1}=\frac{C^M-1}{C-1}+N_s{C}^M$. Then the sample size is $n=\left|\mathcal{T}\right|=\left|\mathcal{S}\right|+\left|\mathcal{U}\right|=N_s\frac{C^{M+1}-1}{C-1}$ hence $M\approx {\log }_C\left(n/N_s\right)$. Starting with $\delta =M$, the parent set sizes J_i for a node ${\mathit{v}}_i\in {\mathit{A}}_r$ grow with r as $J_i=r{N}_s$ and if ${\mathit{v}}_i\in \mathit{B}$ then $J_i=M{N}_s$. The cost of computing p(w θ) is driven by the calculation of ${\mathit{H}}_j$, which is $O\left(r^2{N}_s^3\right)$ for reference nodes at level r, for a total of $O\left(N_s^3{\sum }_{r=0}^{M-1}{C}^r{r}^2\right)$. Since for common choices of C and M we have ${\sum }_{r=0}^{M-1}{C}^r{r}^2{N}_s^3\le {\sum }_{r=0}^{M-1}{C}^{2r}{N}_s^3=\frac{C^{2M}-1}{C^2-1}{N}_s^3\approx C^M{N}_s^3\approx \frac{n}{N_s}{N}_s^3=n{N}_s^2$ then the cost for reference sets is $O\left(n{N}_s^2\right)$. Analogously for non reference nodes we get $O\left(C^M{M}^2{N}_s^3\right)$ which leads to a cost of $O\left(n{N}_s^2\right)$. The cost of sampling w is mainly driven by the computation of the Cholesky factor of a N_s × N_s matrix at each of $\frac{C^M-1}{C-1}$ reference nodes, which amounts to $O\left(n{N}_s^2\right)$. For the $N_s{C}^M$ non-reference nodes the main cost is in computing ${\mathit{H}}_i{\mathit{w}}_{\left[i\right]}$ which is MN_s for overall cost $O\left(C^{M}M{N}_s^2\right)$ which is smaller than $O\left(n{N}_s^2\right)$. Obtaining ${\mathit{F}}_i^{\left(c\right)}$ at the root of $\mathcal{G}$ is associated to a cost $O\left(N_s^2\frac{C^M-C}{C-1}\right)$ which is $O\left(n{N}_s\right)$ but constitutes a bottleneck if such operation is performed simultaneously to sampling; however this bottleneck is eliminated in Algorithm 3.

If δ = 1 then the parent set sizes J_i for all nodes ${\mathit{v}}_i\in \mathit{V}$ are constant $J_i=N_s$; since the nodes at levels 0 to $M-1$ have C children, the asymptotic cost of computing $p\left(\mathit{w}|\mathit{\theta }\right)$ is $O\left(N_s^3{\sum }_{r=0}^{M-1}{C}^r\right)=O\left(N_s^3\frac{C^M-1}{C-1}\right)=O\left(n{N}_s^2\right)$. However there are savings of approximately a factor of M associated to δ = 1 in fixed samples since ${\sum }_{r=1}^{M-1}{C}^r{r}^2>{\sum }_{r=1}^{M-1}{C}^{r}r>\frac{M{C}^M-1}{C-1}-\frac{C^{M+1}}{{\left(C-1\right)}^2}>\frac{M{C}^M-1}{C-1}>M{\sum }_{r=0}^{M-1}{C}^r$. Fixing C and M one can thus choose larger N_s and smaller δ, or vice-versa.

The storage requirements are driven by the covariance at parent locations ${\mathit{C}}_{\left[j\right]}$ for nodes ${\mathit{v}}_j$ with $\text{Pa}\left[{\mathit{v}}_j\right]\ne \cancel{0}$ i.e. all reference nodes at level $r=1,\dots ,M-1$ and non-reference nodes. Taking δ = M , suppose ${\mathit{v}}_i$ is the last parent of ${\mathit{v}}_j$, meaning ${\mathit{v}}_i\cup \text{Pa}\left[{\mathit{v}}_i\right]=\text{Pa}\left[{\mathit{v}}_j\right]$. Then ${\mathit{C}}_{\left[j\right]}=\mathit{C}\left(\left\{S_i,S_{\left[i\right]}\right\},\left\{S_i,S_{\left[i\right]}\right\}\right)$. If ${\mathit{v}}_i\in {\mathit{A}}_r$ then these matrices are of size $\left(r+1\right)N_s\times \left(r+1\right)N_s$; each of these is thus $O\left(r^2{N}_s^2\right)$ in terms of storage. Considering all such matrices brings the overall storage requirement to $O\left({\sum }_{r=0}^{M-1}{C}^r{r}^2{N}_s^2\right)$ which is O(nN_s using analogous arguments as above. For δ = 1 we apply similar calculations as above. The same number of H_j and R_j must be stored but these are smaller in size and therefore do not affect the overall storage requirements. The design matrix Z is stored in blocks and never as a large (sparse) matrix implying a storage requirement of O(nq).

Appendix C. Implementation details

Building a SpamTree DAG proceeds by first constructing a base-tree ${\mathcal{G}}_1$ at depth δ = 1 and then adding edges to achieve the desired depth level. The base tree ${\mathcal{G}}_1$ is built from the root by branching each node v into $\left|\text{Ch}\left[\mathit{v}\right]\right|=c^d$ children where d is the dimension of the spatial domain and c is a small integer. The spatial domain $\mathcal{D}$ is partitioned recursively; after setting $\mathcal{D}$, each recursive step proceeds by partitioning each coordinate axis of $D_i\subset \mathcal{D}$ into c intervals. As a consequence $D_i={\cup }_j{D}_{ij}$ and $D_{ij}\cap D_{i{j}^{\prime }}=\cancel{0}$ if $j\ne j^{\prime }$. This recursive partitioning scheme is used to partition the reference set $\mathcal{S}$ which we consider as a subset of the observed locations. Suppose we wish to associate node v to approximately n_S locations where n_S = k^d for some k. Start from the root i.e.$\mathit{v}\in {\mathit{A}}_0$. Then take ${\mathcal{S}}_0=\mathcal{S}$ and partition it via parallel partitioning of each coordinate axis into k intervals. Collect 1 location from each subregion to build S₀. Then set $\eta \left(S_0\right)={\mathit{v}}_0$ and ${\mathcal{S}}_1=\mathcal{S}\backslash {\mathcal{S}}_0$. Then, take ${\left\{D_{1j}\right\}}_j$ such that ${\cup }_j{D}_{1j}=D_0=\mathcal{D}$. We find $S_{1j}$ via axis-parallel partitioning of ${\mathcal{S}}_1\cap D_{1j}$ into k^d regions and selecting one location from each partition, as above, and setting ${\mathcal{S}}_2=\mathcal{S}\backslash \left\{{\mathcal{S}}_0\cup {\mathcal{S}}_1\right\}$. All other reference subsets are found by sequentially removing locations from the reference set, and proceeding analogously as above. This stepwise procedure is stopped when either the tree reaches a predetermined height M , or when there is an insufficient number of remaining locations to build reference subsets of size n_S. The remaining locations are assigned to the leaf nodes via η_B as defined in Section 2.1 in order to include at least one neighboring realization of the process from the same variable.

One specific issue arises when multivariate data are imbalanced, i.e. one of the margins is observed at a much sparser grid, e.g. in Section 4.2 PRCP is collected at a ratio of 1:10 locations compared to other variables. In these cases, if locations were chosen uniformly at random to build the reference subsets then the root nodes would be associated via η to reference subsets which likely do not contain such sparsely observed variables. This scenario goes against the intuition of 1 suggesting that a naïve approach would result in poor performance at the sparsely-observed margins. To avoid such a scenario, we bias the sampling of locations to favor those at which the sparsely-observed variables are recorded. As a result, in Section 4.2 near-root nodes are associated to reference subsets in which all variables are balanced; the imbalances of the data are reflected by imbalanced leaf nodes instead.

The source code for SpamTrees is available at https://CRAN.R-project.org/package=SpamTree and can be installed as an R package. The SpamTree package is written in C++ using the Armadillo library for linear algebra (Sanderson and Curtin, 2016) interfaced to R via RcppArmadillo (Eddelbuettel and Sanderson, 2014). All matrix operations are performed efficiently by linkage to the LAPACK and BLAS libraries (Blackford et al., 2002; Anderson et al., 1999) as implemented in OpenBLAS 0.3.10 (Zhang, 2020) or the Intel Math Kernel Library. Multithreaded operations proceed via OpenMP (Dagum and Menon, 1998).

C.1. On the dependencies on sparse Cholesky libraries

SpamTrees do not require the use of external Cholesky libraries because Cholesky-like algorithms can be written explicitly by referring to the treed DAG and its edges. This is unusual for DAG-based models commonly used in geostatistical settings. For example, an NNGP model uses neighbors to build a DAG. The DAG can be used to fill the L and D matrices leading to $\ell D{\ell }^{\top }=C^{-1}$, where C⁻¹ is the sparse precision matrix of the latent process. When one then adds measurement error in a regression setting and marginalizes out the latent process, the goal is to find the Cholesky decomposition of C⁻¹ + τ² I_n . The original NNGP DAG used for L and D is not useful for this purpose – hence the need for NNGP to use sparse Cholesky libraries in collapsed samplers (Finley et al., 2019). On the other hand, with SpamTrees we can still look at the original DAG to “update” Land D by using Algorithm 4. We included it in the Appendix as our software package implements the Gibbs sampler in the main article, which does not involve Cholesky decompositions of large sparse precision matrices.

Furthermore, one of the initial steps in Cholesky algorithms for sparse symmetric positive-definite matrices involves finding “good” reordering rows and columns. These reorderings simplify the (undirected) graphical model that corresponds to the sparsity structure in the matrix. Once a simple-enough graphical model is found heuristically, it is used for the decomposition. On the other hand, the sparse Cholesky algorithm for SpamTrees can be written explicitly using the underlying DAG, without any intermediate step, because it is fixed and with a convenient treed structure. It might be possible to write software that out-performs excellent libraries such as CHOLMOD (Chen et al., 2008) at decomposing the matrices needed in collapsed sampling algorithms for SpamTrees.

Regarding a more general perspective about dependencies on well established software libraries, we should clarify that the software for SpamTree does take advantage of highly efficient libraries such as BLAS/LAPACK provided in Intel MKL 2019.5, OpenMP (Dagum and Menon, 1998) for parallelizing the algorithms as described in the main article. These libraries are optional and our code does not strictly depend on them. For example, noting that it is considerably more difficult to compile OpenMP code on Macs, one can just disable OpenMP and let the Accelerate BLAS (rather than OpenBLAS or Intel MKL) deal with all matrix algebra for Apple computers, at the cost of some performance in big data settings. Our code also has R package dependencies for data pre-processing, but these are peripheral to the proposed methods and algorithms. Any improvement in the upstream libraries we used for coding SpamTree will positively impact the performance of our software.

C.2. Applications

C.2.1. Simulated Datasets

SpamTrees with full depth are implemented by targeting reference subsets of size n_S = 25 and tress with c = 4 additional children for each branch. The tree is built starting from a 2 × 2 partition of the domain, hence there are 4 root nodes with no parents in the DAG. The cherry-pickying function η is set as in Section 2.1; with these settings the tree height is M = 3. For SpamTrees with depth δ = 1 we build the tree with reference subsets of size n_S = 80 and c = 4. Multivariate Q-MGPs are implemented via axis-parallel partitioning using 57 intervals along each axis. The multivariate SPDE-INLA method was implemented following the examples in Krainski et al. (2019), Chapter 3, setting the grid size to 15 × 15 to limit the compute time to 15 seconds when using 10 CPU threads. BART was implemented on each dataset via the wbart function in the R package BART; the set of covariates for BART was built using the spatial coordinates in addition to a binary variable representing the output variable index (i.e. taking value 1 whenever y_i is of the first outcome variable, 0 otherwise).

C.2.2. MODIS-TERRA and GHCN

The implemented SpamTrees are built with 36 root nodes and c = 6 additional children for each level of the tree, 25 reference locations for each tree node, and for up to M = 5 levels of the tree and δ = 5 (i.e. full depth). The non-reference observed locations are linked to leaves via cherry-pickying as in Section 2.1. Multivariate models were run on an AMD Epyc 7452-based virtual machine with 256GB of memory in the Microsoft Azure cloud; the SpamTree R package was set to run on 20 CPU threads, on R version 4.0.3 linked to the Intel Math Kernel Library (MKL) version 2019.5–075. The univariate models were run on an AMD Ryzen 5950X-based dedicated server with 128GB of memory, on 16 threads, R version 4.1.1 linked to Intel MKL 2019.5–075. The univariate NNGP model was implemented using R package spNNGP (Finley et al., 2020) using 20 neighbors for all outcomes and a “latent” algorithm. The univariate SpamTree was implemented on each outcome with full depth, c = 16 additional chidren for each level of the tree, and 25 reference locations for each node.

The MGP model was implemented via the development package at github.com/mkln/meshgp targeting a block size with 4 spatial locations, resulting in an effective average block dimension of 20. Caching was unavailable due to the irregularly spaced PRCP values. Fewer MCMC iterations were run compared to SpamTrees to limit total runtime to less than 16h.