Testing for dependence on tree structures

Significance

Tree-like structures are abundant in the empirical sciences as they can summarize high-dimensional data and show latent structure among many samples in a single framework. Prominent examples include phylogenetic trees or hierarchical clustering derived from genetic data. Currently, users employ ad hoc methods to test for association between a given tree and a response variable, which reduces reproducibility and robustness. In this paper, we introduce treeSeg, a simple to use and widely applicable methodology with high power for testing between all levels of hierarchy for a given tree and the response while accounting for the overall false positive rate. Our method allows for precise uncertainty quantification and therefore, increases interpretability and reproducibility of such studies across many fields of science.

Abstract

Tree structures, showing hierarchical relationships and the latent structures between samples, are ubiquitous in genomic and biomedical sciences. A common question in many studies is whether there is an association between a response variable measured on each sample and the latent group structure represented by some given tree. Currently, this is addressed on an ad hoc basis, usually requiring the user to decide on an appropriate number of clusters to prune out of the tree to be tested against the response variable. Here, we present a statistical method with statistical guarantees that tests for association between the response variable and a fixed tree structure across all levels of the tree hierarchy with high power while accounting for the overall false positive error rate. This enhances the robustness and reproducibility of such findings.

In the era of big data where quantifying the relationship between samples is difficult, tree structures are commonly used to summarize and visualize the relationship between samples and to capture latent structure. The hierarchical nature of trees allows the relationships between all samples to be viewed in a single framework, and this has led to their widespread usage in genomics and biomedical science. Examples are phylogenetic trees built from genetic data, hierarchical clustering based on distance measures of features of interest (for example, gene expression data with thousands of markers measured in each sample), evolution of human languages, and more broadly, in machine learning where clustering and unsupervised learning are fundamental tasks (1–7).

Often, samples have additional response measurements $y_i$ (e.g., phenotypes), and a common question is whether there is a relation between the sample’s latent group structure captured by the tree $T$ and the outcome of interest $y_i$ (i.e., whether the distribution of $y_i$ depends on its relative location among the leaves of the tree $T$). Testing for all possible combinations of groupings on the tree is practically impossible as it grows exponentially with sample size. Currently, users typically decide on the number of clusters on an ad hoc basis (e.g., after plotting the response measurement on the leaves of the tree and deciding visually which clusters to choose), which are then tested for association with the outcome of interest. This lack of rigorous statistical methodology has limited the translational application and reproducibility of these methods.

Here, we present a statistical method and accompanying R package, treeSeg, that, given a significance level $\alpha$, test for dependence of the response measurement distribution on all levels of hierarchy in a given tree while accounting for multiple testing. It returns the most likely segmentation of the tree such that each segment has a distinct response distribution while controlling the overall false positive error rate. This is achieved by embedding the tree segmentation problem into a change-point detection setting (8–13).

treeSeg does not require any assumptions on the generation process of the tree $T$. It treats $T$ as given and fixed, testing the response of interest against the given tree structure. Every tree $T$, independent of how it was generated, induces some latent ordering of the samples. treeSeg tests whether, for this particular ordering, the distribution of the independent observations $y_i$ depends on their locations on the tree.

treeSeg is applicable to a wide range of problems across many scientific disciplines, such as phylogenetic studies, molecular epidemiology, metagenomics, gene expression studies, etc. (3, 5, 14–16), where the association between a tree structure and a response variable is under investigation. The only inputs needed are the tree structure $T$ and the outcome of interest $y_i$ for the leaves of the tree. We demonstrate the sensitivity and specificity of treeSeg using simulated data and its application to a cancer gene expression study (1).

Results

For ease of presentation, we restrict to discrete binary response measurements $y_i\in \left\{\mathrm{0,1}\right\}$. However, the procedure is equally applicable to continuous and other observation types (Methods). If there is no association between the tree $T$ and the response measurement $y_i$, then the observed responses $y_i$ would be randomly distributed on the leaves of the tree, independent of the tree structure $T$. However, if the distribution of responses is associated with the tree structure, we may observe clades in the tree with distinct response distributions. The power to detect segments with distinct distributions depends on the size of the clade and the change in response probability, $p_i=\mathbf{\text{P}}\left(Y_i=1\right)=1-\mathbf{\text{P}}\left(Y_i=0\right)$, which means that one can only make statistical statements on the minimum number of clades with distinct distributions on the tree and not the maximum.

Fig. 1 illustrates our method and its output for a simulated dataset. The responses $y_i$ are displayed on the leaves of the tree as black and gray lines in Fig. 1. The tree $T$ is made of three segments with distinct distributions over the responses indicated by dark gray, light gray, and white backgrounds in Fig. 1. Given a confidence level $1-\alpha$ (e.g., $1-\alpha =0.9,0.95$), the treeSeg procedure estimates the most likely segmentation of the tree into regions of common response distributions such that the true number of segments is at least as high as the estimated number of segments with probability of $1-\alpha$.

Fig. 1.

Illustration of the treeSeg method. Binary tree with 200 leaves and three segments with distinct response distributions indicated by dark gray, light gray, and white backgrounds. Outcomes for each sample are shown on the leaves of the tree as gray or black vertical lines. Leaf responses were simulated such that the black line has probabilities of $0.95,0.47$, and 0.05 for each of the dark gray, light gray, and white background sections, respectively. Using $\alpha =0.1$, treeSeg has estimated three segments on the tree with distinct response distributions indicated by the red diamonds on the nodes of the tree. Blue squares constitute a 90% confidence set for the nodes of the tree associated with the change in response distribution. Lower shows the simulation response probabilities (black dotted line), the treeSeg estimate (red line), and its 90% confidence bands (orange).

Our method employs many likelihood ratio (LR) statistics simultaneously to test for changes in the response distribution on all levels of tree hierarchy and estimates at what level, if any, there is a change. The multiple testing procedure of treeSeg is based on a multiscale change-point methodology (11) tailored to the tree structure. The significance levels of the individual tests are chosen in such a way that the overall significance level is the prespecified $\alpha$. As well as the maximum likelihood estimate, our method also provides confidence sets (at the $1-\alpha$ level) for the nodes of the tree associated with the change in response distribution and a confidence band for the response probabilities $p_i$ over the segments (Methods and SI Appendix have theoretical proofs). In the example of Fig. 1, using $\alpha =0.1$, treeSeg estimates three segments in the tree $T$, indicated with Fig. 1, red diamonds on the nodes of the tree, recovering the true simulated changes in response distributions. In Fig. 1, blue squares on the tree indicate the $1-\alpha$ confidence set for the nodes on the tree associated with the change in responses $y_i$. The red line in Fig. 1, Lower shows the maximum likelihood estimate of the response probabilities $p_i$ for each segment, which accurately recovers the true simulated probabilities shown as the black dotted line. The orange band in Fig. 1, Lower shows the $1-\alpha$ confidence band of the response probabilities.

The treeSeg method can handle missing data and make response predictions on new samples. Computationally, treeSeg scales well with sample size: for example, a test simulation for a tree with 100,000 samples (number of leaves in the tree) and no response association took around 110 min to run on a standard laptop. Details on treeSeg’s implementation are in SI Appendix.

Simulation Study.

We confirmed the statistical calibration and robustness of treeSeg using simulation studies. We found that, for reasonable minimal clade sizes and changes in response distribution, treeSeg is able to detect association between response and tree structure reliably (SI Appendix, Figs. S11–S16). More importantly, treeSeg almost never detects segments that are not present (it can be mathematically proven that the inclusion of a false positive segment only happens with probability $\le \alpha$) (Methods), and the nominal guarantee of $1-\alpha$ is exceeded in most cases. For instance, in a simulation study of 1,000 randomly generated trees (200 leaves) with no changes in response distribution, treeSeg (using $\alpha =0.1$) correctly detected no association in $98.6\%$ of the runs.

The treeSeg algorithm uses a fixed ordering of the leaf nodes according to the tree structure $T$. In principle, any such ordering is equally valid as long as this is made independent of the response variables. In our implementation in SI Appendix, we apply a standardized ordering of the nodes so that treeSeg’s output is independent of any user-specified ordering. Furthermore, we provide simulation results showing that treeSeg is robust to random changes in node ordering and consistently infers the correct number of segments on the tree (SI Appendix, section 2.B and Figs. S17–S25). In SI Appendix, section 2.B.1, we provide some discussion on signal-dependent branch orderings that may yield a higher detection power compared with others. We also present a procedure for aggregating results across random orderings to ensure that the output is independent of any specific tip ordering (SI Appendix, section 1.E). SI Appendix has full details on simulation studies.

The treeSeg procedure is conditioned on the input tree and thus, independent of the particular way that the tree is generated. In real applications, the input tree structure $T$ is usually just a noisy version of some true neighborhood structure $\stackrel{\sim }{T}$ of interest. Therefore, whenever the noise in the input tree $T$ is reasonably small such that $T$ and $\stackrel{\sim }{T}$ essentially describe the same neighborhood structure, treeSeg’s output is robust to this noise (SI Appendix, section 1.F has further illustration).

Application to Cancer Data Example.

We illustrate the application of the method on a breast cancer gene expression study (1) where data are publicly available. Following the original study, we used correlation of gene expression data as a distance measure between samples to build a hierarchical clustering tree. In the original study, based on visual inspection, the authors divided the samples into two clusters, observing differences in the distributions of various clinical responses between the two clusters.

In contrast, treeSeg only requires a significance level $\alpha$ as input and searches for associations between responses and the tree on all levels of hierarchy while accounting for multiple testing. Our results are shown in Fig. 2. Using an $\alpha =0.05$, for one of the responses treeSeg delineated the tree into two clusters with distinct response distribution as in the original study. However, treeSeg reports different patterns of association between the tree and the other five responses, including one that has no association with tree structure.

Fig. 2.

Application of treeSeg to a cancer gene expression study (1). Gene expressions for 98 breast cancer samples were clustered based on correlation between samples. Six clinical responses were collected for the samples (A) BRCA mutation, (B) estrogen receptor (ER) expression, (C) histological grade, (D) lymphocytic infiltration, (E) angioinvasion, and (F) development of distant metastasis within 5 y. In each panel, the treeSeg estimation (at $\alpha =0.05$) for clades with distinct response distribution and their probabilities are indicated by the red diamonds on the tree and the red lines below the tree, respectively. The orange band shows the 95% confidence band for the response probabilities $p_i$ for the estimated segments. In E, there is no association between the tree and the response (angioinvasion), and in F, some of the samples have missing observations for the response (distant metastasis within 5 y).

The treeSeg algorithm can be applied to any tree structure and is not restricted to trees generated using hierarchical clustering. An application to a maximum likelihood phylogenetic tree generated from pathogen sequence data is in SI Appendix, section 3.

Discussion

The only tuning parameter for the treeSeg method is a significance level $\alpha$. Depending on the application, the user can decide which value of $\alpha$ is appropriate or screen through several values of $\alpha$ (e.g., $\alpha =0.01,0.05,0.1,0.5$). A small $\alpha$ gives a higher confidence that all detected associations are, indeed, present in the data (with probability of at least $1-\alpha$). A larger $\alpha$ allows us to detect more clusters but increases the risk of including false positive clusters.

The confidence statement for detected clades and response probabilities that accompany treeSeg’s segmentation account for multiple testing at the level of $1-\alpha$. This allows for precise uncertainty quantification when detecting associations between tree structure and the responses. We highlighted treeSeg’s potential with an example from a gene expression study but note its ubiquitous applicability in various settings and its potential to be used across many fields of science. Our method treeSeg is implemented as an R package available on GitHub (https://github.com/merlebehr/treeSeg) and accompanied by a detailed Jupyter notebook with reproductions of all figures in this text.

Methods

Model Assumptions.

For illustration purposes, we focus on binary traits $Y_i\in \left\{\mathrm{0,1}\right\}$. SI Appendix, section 1.B shows how treeSeg generalizes for arbitrary continuous and discrete data. We assume a fixed given rooted tree $T$ with $n$ leaves that captures some neighborhood structure of interest. For the $n$ samples (the leaves of the tree), independent binary traits $Y_i$, $i=1,\dots ,n$, with success probability $p_i$, are observed: that is,

$$Y_i\sim \text{Bern}\left(p_i\right)\hspace{1em}\iff \hspace{1em}\mathbf{\text{P}}\left(Y_i=1\right)=1-\mathbf{\text{P}}\left(Y_i=0\right)=p_i,$$ [1]

independently for $i=1,\dots ,n$, where $\text{Bern}\left(p\right)$ denotes a Bernoulli distribution with success probability $p$. The aim is to estimate the underlying success probabilities $p_1,\dots ,p_n$ from the observations $Y_i$. Without any additional structural information on the success probabilities, we cannot do better than estimating $p_i=Y_i$. However, taking into account the tree structure, we can assume that the success probabilities are associated with the tree such that samples on the same clade of the tree may have the same success probabilities. Our methodology is based on a testing problem, where the null model assumes that all isolates have the same success probability, say $p_0$, and the alternative model assumes that some of the clades on the tree have different success probabilities ($p_0+c\in \left[\mathrm{0,1}\right]$). In the following, we denote an internal node (which demarcates a clade on the tree) with a distinct success probability as an active node.

For simplicity, we will assume in the following that the tree $T$ is binary. Extensions to arbitrary trees are straight forward. We use the following notation. For a binary, rooted tree $T=\left(V,E\right)$, we assume vertices $V=\left\{1,\dots ,n,n+1,\dots ,2n-1\right\}$ and edges $E=\left\{\left(i,j\right)\hspace{0.17em}:\hspace{0.17em}i,j\in V\text{with}i,j\text{connected}\right\}$. The leaves are labeled $V_L=\left\{1,\dots ,n\right\}$, the inner nodes are labeled $V_I=\left\{n+1,\dots ,2n-1\right\}$, and the root is labeled $2n-1$. For a node $i\in V$, its set of offspring leaves in $V_L$ is denoted as $\text{Off}\left(i\right)$. For a node $i\in V$, the subtree of $T$ with root $i$ is denoted as $T\left(i\right)$. An illustrative example for this notation is shown in SI Appendix, Fig. S28. Moreover, for an inner node $i\in V_I$ with offspring leaves $\text{Off}\left(i\right)=\left\{i_1,\dots ,i_m\right\}\subset \left\{1,\dots ,n\right\}$ and for some $ϵ\in \left(\mathrm{0,1}\right)$, we denote the left $ϵ$-leaf neighborhood of $i$ as ${\text{N}}_{\text{L}}\left(i,ϵ\right)=\left\{i_1-\lfloor nϵ\rfloor ,i_1-\lfloor nϵ\rfloor +1,\dots ,i_1+\lfloor nϵ\rfloor -1,i_1+\lfloor nϵ\rfloor \right\}$ and analog, the right $ϵ$-leaf neighborhood of $i$ as ${\text{N}}_{\text{R}}\left(i,ϵ\right)=\left\{i_m-\lfloor nϵ\rfloor ,i_m-\lfloor nϵ\rfloor +1,\dots ,i_m+\lfloor nϵ\rfloor -1,i_m+\lfloor nϵ\rfloor \right\}$.

We consider the following statistical model.

Model 1.

For a given binary, rooted tree $T=\left(V,E\right)$ as above, assume that one observes for each of the leaves $i\in V_L$ independent Bernoulli random variables $Y_i\sim B\left(p_i\right)$, $1\le i\le n$, where the vector of success probabilities $p=\left(p_1,\dots ,p_n\right)$ is an element of $\mathcal{S}=\mathcal{S}\left(T\right)≔\left\{{\left(p_0+{\sum }_{j=1}^k{c}_j{\mathbb{1}}_{i\in \text{Off}\left(v_j\right)}\right)}_{1\le i\le n}\in {\left[\mathrm{0,1}\right]}^n:\hspace{0.17em}{v}_j\in V,\hspace{0.17em}{p}_0,c_j\in \mathbb{R},\hspace{0.17em}0\le k\le 2n-1\right\}$.

For an element $p\in \mathcal{S}$, we denote the set of nodes $V\left(p\right)≔\left\{v_1,\dots ,v_k\right\}$ as a set of active nodes and $k\left(p\right)=k$ as the number of active nodes. To ensure identifiability of active nodes, we further assume that, for each active node $v_j$, $j=1,\dots ,k$, there exists at least one offspring leaf $i\in 1,\dots ,n$ that has the same success probability as $v_j$. This just excludes the trivial case where the influence of one active node (or the root) is completely masked by other active nodes. Equivalently, this means that we assume that $\#\text{supp}\left(p\right)=\#\left\{p_i\hspace{0.17em}:\hspace{0.17em}i=1,\dots ,n\right\}=k+1$. We provide a simple example in SI Appendix, Fig. S30.

We stress that the set $V\left(p\right)$ is not necessarily unique (SI Appendix, Fig. S29 has an example). That is, for a given vector $p\in \mathcal{S}$, there may exist two (or more) sets of active nodes $\left\{v_1,\dots ,v_k\right\}$ and $\left\{v_1^{\prime },\dots ,v_k^{\prime }\right\}$ such that $p_i=p_0+{\sum }_{j=1}^k{c}_j{\mathbb{1}}_{i\in \text{Off}\left(v_j\right)}=p_0^{\prime }+{\sum }_{j=1}^k{c}_j^{\prime }{\mathbb{1}}_{i\in \text{Off}\left(n_j^{\prime }\right)}$. To overcome this ambiguity, we will implicitly associate with each $p\in \mathcal{S}\left(T\right)$ a set of active nodes $V\left(p\right)$ of size $k\left(p\right)$. When a specific vector $p\in \mathcal{S}\left(T\right)$ has more than one possible sets of active nodes $V^{\prime },V^{\prime \prime },\dots$, we assume several copies of $p$ in $\mathcal{S}\left(T\right)$, one associated with each of the sets $V^{\prime },V^{\prime \prime },\dots$. In the following, we explore the tree structure to estimate the underlying success probabilities $p_i$ and hence, their segmentation into groups of leaves where observations within a group have the same success probability and observations between different groups have different success probabilities.

Multiscale Segmentation.

The procedure that we propose extends (11) from totally ordered structures to trees and is a hybrid method of estimating and testing. A fundamental observation is that one can never rule out an additional active node. This is because a node could be active but change the success probability of its offspring nodes only by an arbitrarily small amount. On the other hand, if in a subtree, successes are much more common than in the remaining tree, it is possible to significantly reject the hypothesis that all leaves in the tree have the same success probability.

For a given candidate vector $\stackrel{\sim }{p}\in \mathcal{S}$, our procedure employs on each subtree $T\left(i\right)$ where $\stackrel{\sim }{p}$ is constant, with $\stackrel{\sim }{p}\equiv \stackrel{\sim }{p}\left(\text{Off}\left(i\right)\right)$, an LR test for the hypothesis that the corresponding observations all have the same success probability $\stackrel{\sim }{p}\left(\text{Off}\left(i\right)\right)$. The levels of the individual tests are chosen in such a way that the overall level of the multiple test is $\alpha$ for a given prespecified $\alpha \in \left(\mathrm{0,1}\right)$. A statistical hypothesis test can always be inverted into a confidence statement and vice versa. Therefore, we can derive from the above procedure a confidence set for the vector of success probabilities $p=\left(p_1,\dots ,p_n\right)$. We require our final estimate $\widehat{p}={\widehat{p}}_{1-\alpha }$ to lie in this confidence set. That is, we require that, whenever $\widehat{p}$ has constant success probability on a subtree, the respective LR test accepts. Within all vectors $p\in \mathcal{S}$ that lie in this confidence set, we choose one which comes from a minimum number of active nodes, and within this set, we choose the maximum likelihood solution. Thereby, our procedure not only provides an estimate but also, provides a confidence statement for all quantities (11). More precisely, the following asymptotic confidence statements hold true.

• 1)With probability at least $1-\alpha$, the true underlying signal $p\in \mathcal{S}$ originates from at least $\widehat{k}$ active nodes, where $\widehat{k}$ is the number of active nodes of $\widehat{p}$ (Theorem 1).
• 2)treeSeg yields a set of nodes ${\mathcal{C}}_{1-\alpha }$, such that the active nodes of $p$, $V\left(p\right)$, are contained with probability at least $1-\alpha$ in ${\mathcal{C}}_{1-\alpha }$ (Corollary).
• 3)treeSeg yields a confidence band for the underlying signal $p$, denoted as ${\underset{\_}{p}}_{1-\alpha }$ and ${\overline{p}}_{1-\alpha }$, such that with probability at least $1-\alpha$ it holds that ${\underset{\_}{p}}_{1-\alpha }\le p\le {\overline{p}}_{1-\alpha }$ simultaneously for all $i=1,\dots ,n$ (Theorem 3).

Moreover, the coverage probability of the confidence sets allows us to derive (up to log factors) optimal convergence rates of the treeSeg estimator as the sample size $n$ increases (11). In particular, we show the following.

• 4)For fixed overestimation bound $\alpha \in \left(\mathrm{0,1}\right)$, the probability that treeSeg underestimates the number of active nodes $k$ vanishes exponentially as $n$ increases (Theorem 2).
• 5)The localization of the estimated active nodes is optimal up to a leaf node set of order $\log \left(n\right)$ (Theorem 4).

In the following, we will give details of the method and of the statements 1 to 5. The proofs of Theorems 1 to 4 and Corollary are similar to the totally structured setting (11). Necessary modifications are outlined in SI Appendix.

For an arbitrary given test vector $\stackrel{\sim }{p}$ (which may depend on $Y$), we define the multiscale statistic (11, 17–19)

$$T_n\left(\stackrel{\sim }{p},Y\right)=\underset{\begin{array}{c}1\hspace{0.17em}\le \hspace{0.17em}i\hspace{0.17em}\le \hspace{0.17em}j\hspace{0.17em}\le \hspace{0.17em}n\\ \stackrel{\sim }{p}{|}_{\left[i,j\right]}\text{const.}\end{array}}{max}\sqrt{2T_i^j\left(Y_i^j,{\stackrel{\sim }{p}}_i\right)}-pen\left(\hspace{0.17em}j-i+1\right),$$ [2]

where $Y_i^j=\left(Y_i,\dots ,Y_j\right)$ and $pen\left(x\right)≔\sqrt{2\log \left(e/x\right)}$. Here, $T_i^j$ is the local log LR test statistic (20) for the testing problem

$$H:\hspace{0.17em}{p}_i=\dots =p_j=\stackrel{\sim }{p}{|}_{\left[i,\hspace{0.17em}j\right]}\hspace{1em}\text{vs.}\hspace{1em}K:\hspace{0.17em}{p}_i=\dots =p_j\ne \stackrel{\sim }{p}{|}_{\left[i,\hspace{0.17em}j\right]}.$$

The calibration term $pen\left(\cdot \right)$ serves as a balancing of the different scales in a way that the maximum in Eq. 2 is equally likely attained on all scales (11, 17) and guarantees certain optimality properties of the statistic [2] (17). Assuming a minimal segment scale $\lambda \in \left(\mathrm{0,1}\right)$ of the underlying success probability vector $p$: that is,

$${\mathcal{S}}_{\lambda }=\left\{p\in \mathcal{S}\text{with const. segments’ length at least}\hspace{0.17em}n\lambda \right\},$$ [3]

it can be shown that $T_n\left(p,Y\right)$ converges in distribution to a functional of the Brownian motion (11), which is stochastically bounded by

$$M≔\underset{0\le s<t\le 1}{sup}\left(\frac{B\left(t\right)-B\left(s\right)}{\sqrt{t-s}}-\sqrt{2\log \frac{e}{t-s}}\right).$$ [4]

Thereby, the minimal scale $\lambda$ may depend on $n$ such that $n\lambda /log{\left(n\right)}^3\to \infty$ as $n\to \infty$ (11). As the distribution of $M$ does not depend on the true underlying signal $p$, its quantiles can be obtained by Monte Carlo simulations and are in the following denoted as $q_{1-\alpha }$: that is,

$$\underset{n\to \infty }{lim}\underset{p\in {\mathcal{S}}_{\lambda }}{sup}\mathbf{\text{P}}\left(T_n\left(p,Y\right)>q_{1-\alpha }\right)\le \mathbf{\text{P}}\left(M>q_{1-\alpha }\right)=\alpha .$$ [5]

For a given confidence level $\alpha \in \left(\mathrm{0,1}\right)$ or equivalently, a threshold value $q=q_{1-\alpha }$ in Eq. 5, we first define an estimator for the number of active nodes $k$ in Model 1 via

$$\widehat{k}\left(q\right)≔\underset{p\in \mathcal{S}}{min}k\left(p\right)\hspace{1em}\text{s.t.}{T}_n\left(p,Y\right)\le q.$$ [6]

After the number of active nodes $k$ is estimated, we estimate $p$ as the constrained maximum likelihood estimator

$$\widehat{p}\left(q\right)≔{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}}_{p\in \mathcal{H}\left(q\right)}\sum _{i=1}^n{l}_{p_i}\left(Y_i\right),$$ [7]

where $l_p\left(y\right)$ is the log likelihood function of the binomial distribution and

$$\mathcal{H}\left(q\right)≔\left\{p\in \mathcal{S}:k\left(p\right)=\widehat{k}\left(q\right)\text{and}{T}_n\left(p,Y\right)\le q\right\}.$$ [8]

Note that the maximum likelihood solution in Eq. 7 is not necessarily unique. On the one hand, this is due to the nonuniqueness of the active nodes. On the other hand, this might happen with positive probability by the discreteness of the Bernoulli observations $Y$. In that case, treeSeg just reports the first available solution, with all other equivalent solutions listed in the confidence set. Clearly, if we choose $q$ as in Eq. 5 for some given confidence level $\alpha \in \left(\mathrm{0,1}\right)$, the estimator $\widehat{k}\left(q_{1-\alpha }\right)$ asymptotically controls the probability to overestimate the number of active nodes as summarized in the following theorem.

Theorem 1.

For fixed minimal scale $\lambda >0$ and significance level $1-\alpha \in \left(\mathrm{0,1}\right)$, let ${\mathcal{S}}_{\lambda }$ be as in Eq. 3, $q_{1-\alpha }$ be as in Eq. 5, and $\widehat{k}\left(q\right)$ be treeSeg’s estimated number of active nodes in Eq. 6. Then, it holds that

$$\underset{n\to \infty }{lim}\underset{p\in {\mathcal{S}}_{\lambda }}{sup}\mathbf{\text{P}}\left(\widehat{k}\left(q_{1-\alpha }\right)>k\left(p\right)\right)\le \alpha .$$ [9]

We stress that, in Theorem 1, it is possible to let $\lambda$ go to zero as $n$ increases (11); recall the paragraph after Eq. 4. In particular, from the construction of $\widehat{k}$ in Eq. 6, it follows that $T_n\left(p,Y\right)\le q$ implies $\widehat{k}\left(q\right)\le k$, and thus, for the set $\mathcal{H}\left(q_{1-\alpha }\right)$ in Eq. 8, one obtains that

$$\mathbf{\text{P}}\left(p\in \mathcal{H}\hspace{0.17em}\left(q_{1-\alpha }\right)\right)\ge \mathbf{\text{P}}\left(T_n\left(p,Y\right)\le q_{1-\alpha }\right)-\mathbf{\text{P}}\left(\widehat{k}\hspace{0.17em}\left(q_{1-\alpha }\right)<k\right).$$ [10]

By Eq. 5, it follows that the first term on the right-hand side, $\mathbf{\text{P}}\left(T_n\left(p,Y\right)\le q_{1-\alpha }\right)$ , is asymptotically lower bounded by $1-\alpha$. Moreover, as we show in Theorem 2, the underestimation error $\mathbf{\text{P}}\left(\widehat{k}\hspace{0.17em}\left(q_{1-\alpha }\right)<k\right)$ vanishes exponentially fast as sample size $n$ increases. From this, it follows that the set $\mathcal{H}\left(q_{1-\alpha }\right)$ constituents an asymptotically honest confidence set (11) for the whole vector $p$ from which confidence sets for the active nodes and confidence bands for $p$ as in statements 2 and 3 follow (Theorem 3 and Corollary).

Any bound on the underestimation error necessarily must depend on the minimal segment scale $\lambda$ in Eq. 3 as well as a minimal pairwise difference $\delta \in \left(\mathrm{0,1}\right)$ of success probabilities in different active segments. That is, for $p\in {\mathcal{S}}_{\delta }$, we assume $\delta <\underset{p_i\ne p_j}{min}\left|p_i-p_j\right|$ and let

$${\mathcal{S}}_{\delta ,\lambda }={\mathcal{S}}_{\lambda }\cap {\mathcal{S}}_{\delta }.$$ [11]

With this, one obtains that the underestimation probability decreases exponentially (11) in $n$ [for fixed $\delta ,\lambda$ and significance level $1-\alpha \in \left(\mathrm{0,1}\right)$] as the following theorem shows.

Theorem 2.

For fixed minimal scale $\lambda >0$, minimal probability difference $\delta >0$, and significance level $1-\alpha \in \left(\mathrm{0,1}\right)$, let ${\mathcal{S}}_{\lambda ,\delta }$ be as in Eq. 11, $q_{1-\alpha }$ be as in Eq. 5, and $\widehat{k}\left(q\right)$ be treeSeg’s estimated number of active nodes in Eq. 6. Then, it holds that

$$\underset{p\in {\mathcal{S}}_{\lambda ,\delta }}{sup}\mathbf{\text{P}}\left(\widehat{k}\left(q_{1-\alpha }\right)<k\left(p\right)\right)\le C_1\hspace{0.17em}{e}^{-C_2\hspace{0.17em}n},$$ [12]

where $C_1$ and $C_2$ are positive constants, which only depend on $\alpha ,\lambda ,\delta$.

Again, it is possible to let $\alpha ,\lambda ,$ and $\delta$ go to zero as the sample size $n$ increases (11). The proof of Theorem 2 is similar as for totally ordered structures (11). We outline necessary modifications in SI Appendix. From Theorem 2 and [10], we directly obtain that $\mathcal{H}\left(q_{1-\alpha }\right)$, indeed, constitutes a $1-\alpha$ asymptotic confidence set for the segmentation $p$.

Theorem 3.

For fixed minimal scale $\lambda >0$, minimal probability difference $\delta >0$, and significance level $1-\alpha \in \left(\mathrm{0,1}\right)$, let ${\mathcal{S}}_{\lambda ,\delta }$ be as in Eq. 11, $q_{1-\alpha }$ be as in Eq. 5, and $\mathcal{H}\left(q_{1-\alpha }\right)$ be as in Eq. 8; then,

$$\underset{n\to \infty }{lim}\underset{p\in {\mathcal{S}}_{\lambda ,\delta }}{sup}\mathbf{\text{P}}\left(p\in \mathcal{H}\left(q_{1-\alpha }\right)\right)\ge 1-\alpha .$$

As a corollary, we also obtain a confidence set of the active nodes.

Corollary.

For fixed minimal scale $\lambda >0$, minimal probability difference $\delta >0$, and significance level $1-\alpha \in \left(\mathrm{0,1}\right)$, let ${\mathcal{S}}_{\lambda ,\delta }$ be as in Eq. 11, $q_{1-\alpha }$ be as in Eq. 5, and $\mathcal{H}\left(q_{1-\alpha }\right)$ be as in Eq. 8; then,

$$\underset{n\to \infty }{lim}\underset{p\in {\mathcal{S}}_{\lambda ,\delta }}{sup}\mathbf{\text{P}}\left(V\left(p\right)\subset \left\{v\in V\left(\stackrel{\sim }{p}\right)\hspace{0.17em}:\hspace{0.17em}\stackrel{\sim }{p}\in \mathcal{H}\left(q_{1-\alpha }\right)\right\}\right)\ge 1-\alpha .$$

Theorems 1 and 2 reveal treeSeg’s ability to accurately estimate the number of active nodes in Model 1. For any (arbitrarily small) $\alpha \in \left(\mathrm{0,1}\right)$, we can control the overestimation probability by $1-\alpha$ (Theorem 1). Simultaneously, as the sample size $n$ increases, the underestimation error probability vanishes exponentially fast (Theorem 2). The next theorem shows that treeSeg does not just estimate the number of active nodes correctly with high probability but that it also estimates the location of those active nodes with high accuracy. To this end, note that, for any active node $v\in V\left(p\right)$ and any $ϵ\ge 1/n$, the leaf nodes of its left $ϵ$-leaf neighborhood ${\text{N}}_{\text{L}}\left(v,ϵ\right)$ have nonconstant success probability. The same is true for the right $ϵ$-leaf neighborhood ${\text{N}}_{\text{R}}\left(v,ϵ\right)$. Now assume that treeSeg estimates the number of active nodes correctly $\widehat{k}=k$, which is the case with high probability by Theorems 1 and 2. Then, $\widehat{p}$ being nonconstant on both ${\text{N}}_{\text{R}}\left(v,1/n\right)$ and ${\text{N}}_{\text{L}}\left(v,1/n\right)$ for any true active nodes $v\in V\left(p\right)$, implies a perfect segmentation. Thus, the following theorem shows that treeSeg, indeed, yields such a perfect segmentation up to a leaf node set of size $\mathcal{O}\left(\log \left(n\right)\right)$. That is, conditioned on the correct model dimension $\widehat{k}=k$, treeSeg’s segmentation is perfect up to at most an order of $\log \left(n\right)$ misclassified leaf nodes.

Theorem 4.

For fixed minimal scale $\lambda >0$, minimal probability difference $\delta >0$, and significance level $1-\alpha \in \left(\mathrm{0,1}\right)$, for any $p\in {\mathcal{S}}_{\lambda ,\delta }$ it holds true that

$$\widehat{p}{|}_{{\text{N}}_{\text{L}}\left(v,\frac{C_3\log \left(n\right)}{n}\right)}\text{and}\widehat{p}{|}_{{\text{N}}_{\text{R}}\left(v,\frac{C_3\log \left(n\right)}{n}\right)}$$

are not constant, for all $v\in V\left(p\right)$, with probability at least $1-C_1\hspace{0.17em}{e}^{-C_2\hspace{0.17em}n}$, where $C_1,C_2,C_3$ are positive constants that only depend on $\alpha ,\lambda ,\delta$.

Theorem 4 follows directly by translating change-point location estimation accuracy results for the totally ordered case (11) to the tree setting.

A natural question is whether the localization rate in Theorem 4 is optimal. In particular, one can compare this result with the totally ordered setting, where the minimax optimal change-point estimation rate is known to be of the same order [possibly up to $\log \left(n\right)$ factors]. One would expect that the additional tree structure leads to a strictly better segmentation rate. It turns out, however, that without making further assumptions on the tree the rate in Theorem 4 cannot be improved in general (SI Appendix, Theorem 6). More precisely, for an arbitrary number of observations $n$, there always exist trees that do not contain any additional information other than the ordering of the tips. In that case, the tree structured setting is essentially equivalent to a regular change-point setting, and thus, treeSeg cannot yield any better performance.

On the other hand, when one imposes additional structural assumptions on the tree, it can be shown that treeSeg yields a perfect segmentation with high probability SI Appendix, Theorem 7. Essentially, when a tree structure is such that the segmentation from different sets of actives nodes is either the same or differs by some nonvanishing fraction $\gamma \in \left(\mathrm{0,1}\right)$ (more precisely, this is captured by the $\gamma$-spreading property in SI Appendix, Definition 1), then treeSeg will recover those active nodes exactly with high probability. A simple example of trees that provide such additional structure are perfect trees, where all tip nodes have the same depth. In summary, treeSeg efficiently leverages the tree structure to overcome the minimax lower bound from a simple change-point estimation problem whenever the tree allows this. We provide more details in SI Appendix, section 1.D.