Nonparametric prediction distribution from resolution-wise regression with heterogeneous data

Abstract

Modeling and inference for heterogeneous data have gained great interest recently due to rapid developments in personalized marketing. Most existing regression approaches are based on the conditional mean and may require additional cluster information to accommodate data heterogeneity. In this paper, we propose a novel nonparametric resolution-wise regression procedure to provide an estimated distribution of the response instead of one single value. We achieve this by decomposing the information of the response and the predictors into resolutions and patterns respectively based on marginal binary expansions. The relationships between resolutions and patterns are modeled by penalized logistic regressions. Combining the resolution-wise prediction, we deliver a histogram of the conditional response to approximate the distribution. Moreover, we show a sure independence screening property and the consistency of the proposed method for growing dimensions. Simulations and a real estate valuation dataset further illustrate the effectiveness of the proposed method.

1. Introduction

A common nonparametric regression model establishes the effects of the explanatory variables on the response variable in the form of

$$Y=f(X)+\epsilon ,$$ (1)

where Y is the response variable, X = (X₁, …, X^q)^T is the q-dimensional explanatory variable vector, and ε is the random error, which is often assumed with mean 0 and variance σ^2, and is independent of X.

In recent years there has been a growing demand for exploring regression methods for heterogeneous populations, which has broad applications in personalized marketing and other fields. One characteristic of data heterogeneity is the existence of subpopulations in the data. In practice, the heterogeneity can be regarded as the result of some latent variables. This happens frequently since it is difficult to collect all the explanatory variables for the response. For example, in the real estate data in Section 6, a river and a highway through the city create subpopulations and heterogeneous distributions of housing prices. However, the information of this river and this highway is not available in the data.

Denote the unobserved categorical variable by Z taking values 1, … T, where T is unknown. Suppose the potential true relationship of the response and all the explanatory variables can be expressed by

$$Y=\sum _{t=1}^T{f}_t(X)I(Z=t)+\epsilon ,$$ (2)

with unknown functions ft’s, t = 1, …, T. In this paper, our goal is to relate Y with X without knowing Z. However, this differs from fitting model (1), since the true relationship between Y and X may not even be a function. As an illustration, the housing prices on the two sides of Tamsui river with respect to longitude and latitude are shown in Figure 1. The plot shows a mixture of two subgroups: the housing prices on the west and the east of the river behave differently. The latent variable Z, that is, the indicator for which side the position on, determines the two subgroups. Without knowing Z, the relationship between Y and X cannot be captured by a single function. Hence new methods to model the effects of X on Y with such a challenging heterogeneous population are in great needs.

Fig. 1.

Left panel: An illustration of heterogeneous data in the housing prices on two sides of Tamsui River (light blue curve). Right panel: The housing prices follow different distributions: The housing prices on the west monotonically increase with latitude, while those on the east are concave and parabolic.

One possible idea is to use smoothing splines (Green and Silverman, 1994) which can capture local behaviors. A more general setting is the smoothing spline analysis of variance (SSANOVA) (Wahba, 1990; Gu, 2002), which fits an additive model for main effects and interactions. These approaches use regression to estimate the overall conditional mean function, which tends to fit a compromised effect of the subgroups. Hence they may fail to identify the subgroups of the population, and the results might not be informative for either of the subgroups. Moreover, the estimated distribution might not really reflect the pattern of the true one.

Another possible strategy is to cluster the data first, then fit a regression model within each subgroup. Existing model-based clustering approaches include Jacobs et al. (1991), Pan and Shen (2006), Raftery and Dean (2006), and Guo et al. (2010). As alternative approaches, Lindsten et al. (2011), Hocking et al. (2011), and Pan et al. (2013) formulated clustering as a penalized regression problem with fusion-type penalties. However, these methods focus on finding the groups based on the similarity of the explanatory variables, instead of identifying groups with different effects on the response.

In the literature, some individualized methods were proposed to handle heterogeneity. Ma and Huang (2017) employed subject-specific intercepts to model the unobserved factors which leads to the heterogeneity. They used a concave pairwise fusion penalty to shrink some intercepts to be the same, which can produce a partition of subgroups. Chen, Tran-Dinh, Kosorok and Liu (2021) considered a more general fusing method than that of Ma and Huang (2017) to identify subgroups. Tang and Qu (2017) proposed a multi-directional penalty to shrink individuals to different groups. The performance of these methods depends on how well the subgroups are separated. If the subgroups are close to each other, the performance can be less accurate.

In this paper, we tackle the heterogeneity from a new perspective. Instead of estimating ft’s in (2) through nonparametric regressions, we propose to estimate the conditional distribution of the response variable given observed explanatory variables. The estimated distribution provides an overall picture of the response variable, and can indicate the heterogeneity by the modes of the probability density function (PDF). To achieve this goal, one single regression is not enough, because the pattern of two or more possible values of the response corresponding to one observation of predictor variables cannot be expressed by an explicit function. Our idea is to consider binary expansion statistics proposed in Zhang (2019) and to decompose the response variable into several resolutions which can capture the local information. By establishing a set of logistic regressions, we relate the resolution information to the predictors. The set of regressions can model the heterogeneity since various estimations can be obtained from different local logistic regression models. To achieve the localization, we decompose the response variable by marginal binary expansions, which provides a balanced design and orthogonal resolutions. Our method eventually estimates the distribution of the response variable by a histogram, which shows the possible heterogeneity and even more complicated distributions, without any assumption of subgroup patterns. We show that the method has a sure independence screening property (Fan and Lv, 2008; Fan and Song, 2010) and provides consistent estimates for cell probabilities of the histogram for growing dimensions.

The rest of this paper is organized as follows. In Section 2, we introduce resolution-wise regression, including the decomposition of the response variable and the establishment of the logistic regressions. Section 3 extends the proposed method to high-dimensional settings. In Section 4, we show the consistency of the estimated histogram. In Sections 5 and 6, we demonstrate the performance of our method by the simulated data and the real estate valuation dataset. Section 7 concludes this paper. Some technical proofs and additional simulation results are presented in the Appendix and Supplement.

2. Methodology

A distribution estimation provides more information than a point estimation for heterogeneous data, as the estimated distribution can identify the subgroups by the shape of the PDF. For the case that the subpopulations are not obviously distinguishable from each other, the estimated distribution can still reflect the dispersion of the data.

A direct idea is splitting the range of Y by a partition min(Y) = a₀ ≤ a₁ ⩽ ⋯ ⩽ a_B = max(Y) and modeling the probability of Y falling into each interval with X. This idea handles the heterogeneity by decomposing the information of Y into several non-overlapping intervals. These intervals capture the local information of Y and work together to show the whole histogram. However, a drawback of this approach is the possible loss of information from negligence of joint information in two intervals and insufficient samples in each interval. In this paper, we propose to construct overlapped resolutions based on binary expansions, where each resolution groups the distribution information of the union of several intervals, and includes all corresponding samples. In essence, the proposed construction leads to a balanced design and has a non-redundant orthogonality property. A histogram can be obtained by a transformation from resolution probabilities to cell probabilities. Here we refer to a cell as a bin of the histogram. In this section, we consider the one-dimensional X, and then extend to the high-dimensional case in Section 3. The rest of this section is organized as follows. In Section 2.1, we introduce the construction of the resolutions. In Section 2.2, a set of resolution-wise penalized logistic regressions are established. In Section 2.3, we introduce the binary interaction design (BID) equation to accomplish the transformation from the frequency domain to the probability domain.

2.1. Frequency domain from binary expansions

To overcome the imbalance of non-overlapping intervals, we use a balanced design based on binary expansions. A classical result on the binary expansion of a uniform random variable (Kac 1959) is given as follows:

Lemma 1. For [U ~ Uniform[0, 1], we have $U=\sum _{k=1}^{\infty }\frac{V_k}{2^k}$, where V₁, V₂, …, V_k, … i.i.d. follow Bernoulli(1 / 2).

Denote the cumulative distribution function (CDF) transformation of Y by U_Y. By Lemma 1, we have

$$U_Y=\sum _{k=1}^{\infty }\frac{B_k}{2^k},B_1,B_2,\dots ,B_k,\dots \stackrel{i\text{.}i\text{.}d.}{~}\mathit{\text{Bernoulli}}(1/2).$$ (3)

Through the expansion, the information of Y is decomposed into the information of Bk’s. In (3), binary variables Bk’s can be regarded as indicator functions:

$$B_k=I\left(U_Y\in \left[\frac{1}{2^k},\frac{2}{2^k}\right)\cup \left[\frac{3}{2^k},\frac{4}{2^k}\right)\cup \cdots \cup \left[\frac{2^k-1}{2^k},1\right]\right);k=1,2,\dots$$

Figure 2 shows the binary variables B₁, B₂, B₃ respectively with respect to U_Y. As a finite approximation of the infinite binary expansion, we can truncate the binary expansion of U_Y up to the d_Y-th order

$$U_Y=\sum _{k=1}^{d_Y}\frac{B_k}{2^k}.$$ (4)

Fig. 2.

Binary variables B₁, B₂, B₃ from binary expansions of U_Y. Regions with B_k = 1, k = 1, 2, 3 are in white and regions with B_k = 0, k = 1, 2, 3 are in blue.

Now we introduce the notations of resolutions. Using the binary variables taking values {− 1, 1} instead of {0, 1} by the transformation ${\dot{B}}_k=2B_k-1$, the interactions of Bk’s can be written as products. For example, the event {B₁ = 1, B₂ = 1} ∪ {B₁ = 0, B₂ = 0} is equivalent to $\left\{{\dot{B}}_1{\dot{B}}_2=1\right\}$. In the remainder of this paper, we shall use ${\dot{B}}_k\in \{-1,1\}$.

To approximate the information given by Y, say the σ-field σ(Y), we can use the σ-field generated by ${\dot{B}}_k$’s, denoted by $\sigma \left({\dot{B}}_1,\dots ,{\dot{B}}_{d_Y}\right)$. For the truncation up to the d_Y-th order, we can find a basis with $2^{d_Y}-1$ variables

$$W_{\dot{B}}=\left\{{\dot{B}}_1,\dots ,{\dot{B}}_{d_Y},{\dot{B}}_1{\dot{B}}_2,\dots ,{\dot{B}}_{d_Y-1}{\dot{B}}_{d_Y},\dots ,\prod _{i=1}^{d_Y}{\dot{B}}_i\right\}.$$ (5)

We shall refer to the binary variables in $W_{\dot{B}}$ as resolutions of Y, and the set of all possible values of these resolutions as the frequency domain. Figure 3 shows the variables in $\sigma \left({\dot{B}}_1,{\dot{B}}_2\right)$ with U_Y expanded up to the second order. Through this resolution decomposition, each variable takes value one on half of [0, 1], and value negative one on the other half.

Fig. 3.

Basis binary variables in $\sigma \left({\dot{B}}_1,{\dot{B}}_2\right)$, where U_Y is expanded up to the second order.

2.2. Logistic regression in the frequency domain

With the resolutions decomposed from the binary expansion, we aim to model the relationship between each resolution and the predictors. The resolutions constructed by the binary expansion are independent with each other, thus they can be modeled marginally. Since the resolutions are binary, it is essentially a classification problem. Note that for every resolution, Y is divided into two classes with groups of intervals according to the sign of the binary interaction. Hence the decision boundary can be nonlinear. Therefore, we propose to use binary expansions of predictors as a nonparametric basis to fit a logistic regression on each resolution. Similar to the construction of U_Y, we have the binary expansion of U_X, up to the d_X-th order to be $U_X=\sum _{k=1}^{d_X}\frac{A_k}{2^k}$. Denote ${\dot{A}}_k=2A_k-1$. The σ-field generated by ${\dot{A}}_k$’s, denoted by $\sigma \left({\dot{A}}_1,\dots ,{\dot{A}}_{d_X}\right)$, has a basis with $2^{d_x}-1$ variables $W_{\dot{A}}=\left\{{\dot{A}}_1,\dots ,{\dot{A}}_{d_X},{\dot{A}}_1{\dot{A}}_2,\dots ,{\dot{A}}_{d_X-1}{\dot{A}}_{d_x},\dots ,\prod _{i=1}^{d_x}{\dot{A}}_i\right\}$ We shall refer to elements in $W_{\dot{A}}$ as patterns of X. After the construction of the patterns, the complicated effect of X on Y can be captured by logistic regression, which enjoys efficiency from the orthogonality of the patterns. We establish a set of $2^{d_Y}-1$ penalized logistic regressions with the ℓ₁ penalty (Tibshirani, 1996) on each resolution in $W_{\dot{B}}$, with all patterns in $W_{\dot{A}}$ as predictors. Denote the pattern vector corresponding to $W_{\dot{A}}$ by ${\dot{A}}^F={\left({\dot{A}}_{(1)}^F,\dots ,{\dot{A}}_{\left(2^{d_X}-1\right)}^F\right)}^T\triangleq {\left({\dot{A}}_1,\dots ,{\dot{A}}_{d_X},{\dot{A}}_1{\dot{A}}_2,\dots ,{\dot{A}}_{d_X-1}{\dot{A}}_{d_X},\dots ,\prod _{i=1}^{d_X}{\dot{A}}_i\right)}^T$. Similarly, denote the resolution vector corresponding to $W_{\dot{B}}$ by $\dot{B}={\left({\dot{B}}_{(1)},\dots ,{\dot{B}}_{\left(2^{d_Y}-1\right)}\right)}^T\triangleq {\left({\dot{B}}_1,\dots ,{\dot{B}}_{d_{\text{Y}}},{\dot{B}}_1{\dot{B}}_2,\dots ,{\dot{B}}_{d_{\text{Y}}-1}{\dot{B}}_{d_{\text{Y}}},\dots ,\prod _{i=1}^{d_{\text{Y}}}{\dot{B}}_i\right)}^T$. Let ${\left\{\left(x_i,y_i\right)\right\}}_{i=1}^n$. be n independent observations of (X, Y). Denote ${\dot{A}}_i^F={\left({\dot{A}}_{(1),i}^F,\dots ,{\dot{A}}_{\left(2^{d_X}\right),i}^F\right)}^T$ and ${\dot{B}}_i={\left({\dot{B}}_{(1),i},\dots ,{\dot{B}}_{\left(2^{d_Y}\right),i}\right)}^T$ as the i-th pattern vector and the i-th resolution vector obtained by binary expansions of the empirical CDF transformation of x_i and y_i respectively. The m-th logistic regression, $m=1,\dots ,2^{d_Y}-1$, which models the effect of ${\dot{A}}^F$ on ${\dot{B}}_{\text{(}m\text{)}}$, is established as

$$\text{log}\frac{P\left({\dot{B}}_{(m),i}=1\mid {\stackrel{.}{A}}_i^F\right)}{P\left({\dot{B}}_{(m),i}=-1\mid {\stackrel{.}{A}}_i^F\right)}={\stackrel{.}{A}}_i^{F^T}{\beta }_m,m=1,\dots ,2^{d_Y}-1,$$ (6)

where β_m is the coefficient vector. We employ a ℓ₁ regularization to give the estimator

$${\widehat{\beta }}_m=\underset{\beta }{\text{arg}min}\sum _{\text{i}=1}^{\text{n}}\text{log}\left(1+{\text{e}}^{-{\dot{B}}_{(\text{m}),\text{i}}{\dot{A}}_{\text{i}}^{{\text{F}}^{\text{T}}}\beta }\right)+\lambda \Vert \beta {\Vert }_1,m=1,\dots ,2^{d_Y}-1.$$ (7)

The conditional expectation of ${\dot{B}}_{\text{(}m\text{)}}$ given ${\dot{A}}^F$, denoted by $e_m\text{(}{\dot{A}}^F\text{)}$, can be estimated by ${\widehat{e}}_m\left({\dot{A}}^F\right)=\frac{\text{exp}\left({\stackrel{.}{A}}^{F^T}{\widehat{\beta }}_m\right)}{1+\text{exp}\left({\stackrel{.}{A}}^{F^T}{\widehat{\beta }}_m\right)}$.

2.3. Binary interaction design: from frequency domain back to probability domain

As a final step of estimating the distribution of Y, we aim to transform the conditional expectations of resolutions into the conditional cell probabilities of the corresponding histogram. First, we simplify the notation of the conditional expectation by using a d_Y-dimensional binary index. Namely, denote the conditional expectation $\mathbb{E}({\dot{B}}_{k_1}\cdots {\dot{B}}_{k_p}\mid {\dot{A}}^F)$, p ∈ {1, …, d_y}, {k₁, …, k_p} ⊂ {1, …, d_y}, by E_b, where $\mathbb{E}(\cdot )$ stands for the expectation, $b=\left(b_1,\dots ,b_{d_Y}\right)$, is a vector of length d_Y with value one at k₁, …, k_p and zero otherwise. Let E be the d_Y-dimensional conditional expectation vector whose entries are sorted in an ascending order according to the binary system. Hence in some sense, b identifies the resolutions. Note that we set $E_0=\mathbb{E}\left(1\right)=1$ as the first entry. The $\left(\sum _{i=1}^{d_Y}{b}_i{2}^{d_Y-i}+1\right)$ -th entry is E_b. For example, the expectation with d_Y = 3 is

$$\begin{gathered} E={\left(E_{000},E_{001},E_{010},E_{011},E_{100},E_{101},E_{110},E_{111}\right)}^T, \\ =(\mathbb{E}(1),\mathbb{E}\left({\dot{B}}_3\mid {\dot{\mathit{\text{A}}}}^F\right),\mathbb{E}\left({\dot{B}}_2\mid {\stackrel{.}{A}}^F\right),\mathbb{E}\left({\dot{B}}_2{\dot{B}}_3\mid {\stackrel{.}{A}}^F\right),\mathbb{E}\left({\dot{B}}_1\mid {\stackrel{.}{A}}^F\right),{\mathbb{E}\left({\dot{B}}_1{\dot{B}}_3\mid {\stackrel{.}{A}}^F\right),\mathbb{E}\left({\dot{B}}_1{\dot{B}}_2\mid {\stackrel{.}{A}}^F\right),\mathbb{E}\left({\dot{B}}_1{\dot{B}}_2{\dot{B}}_3\mid {\stackrel{.}{A}}^F\right))}^T. \end{gathered}$$

We denote the conditional probabilities of the $2^{d_Y}$ cells in terms of the index $b=\left(b_1,\dots ,b_{d_Y}\right)$ as above. Define the conditional cell probability p_b as the conditional probability of ${\dot{B}}_k$’s taking values clarified by b given ${\dot{A}}^F$, i.e., $p_b=p_{\left(b_1,\dots ,b_{d_Y}\right)}=P\left({\dot{B}}_1=2b_1-1,\dots ,{\dot{B}}_{d_Y}=2b_{d_Y}-1\mid {\stackrel{.}{A}}^F\right)$. As an example, for d_Y = 3, $p_{101}=P\left({\dot{B}}_1=1,{\dot{B}}_2=-1,{\dot{B}}_3=1\mid {\dot{A}}^F\right)$. Let p be the d_Y-dimensional conditional probability vector of the cells whose entries are sorted by a descending order according to the binary system, i.e., the $\left(2^{d_Y}-\sum _{i=1}^{d_Y}{b}_i{2}^{d_Y-i}\right)$-th entry is p_b. For example, the conditional probability vector of the cells with d_Y = 3 is

$$p={\left(p_{111},p_{110},p_{101},p_{100},p_{011},p_{010},p_{001},p_{000}\right)}^T.$$

With the above notations, we establish the binary interaction design (BID) equation (Zhang, 2019) to transform the expectations of resolutions into cell probabilities. The equation is established by the Sylvester’s construction of Hadamard matrix $H=H_{2^{d_Y}}$ (Sylvester, 1867).

Lemma 2 (BID equation). Let E be the conditional expectation vector of the resolutions from the binary expansion, and p be the conditional probability vector of the cells. Then

$$\mathit{\text{E}}=H\mathit{\text{p}},$$ (8)

where H is the Hadamard matrix (Sylvester, 1867).

From the BID equation, the conditional probabilities of Y given X falling into each cell can be obtained by estimating the conditional expectation of resolutions. Denoting the estimator of E by E. In a common sense, p can be estimated by p = H⁻¹ E, since the Hadamard matrix H is invertible and $H^{-1}=\frac{1}{2^{d_Y}}H$. However, this p may not be a probability measure. From the structure of the Hadamard matrix, the following lemma shows the summation of the cell probabilities is one.

Lemma 3. For any E, the estimation of p by p = H⁻¹ E has a sum of entries of one.

Proof of Lemma 3. Denote $p={\left({\widehat{p}}_1,\dots ,{\widehat{p}}_{2^{d_Y}}\right)}^T$. Since $E=\left(1,{\widehat{e}}_1({\dot{\mathit{\text{A}}}}^F),\dots ,{\widehat{e}}_{2^{d_Y}-1}\left({\dot{\mathit{\text{A}}}}^F\right)\right)$, the summation of p is

$$\sum _{i=1}^{2^{d_Y}}{\widehat{p}}_i=1_{2^{d_Y}}^{T}p=\frac{1}{2^{d_Y}}{1}_{2^{d_Y}}^T{H}^{-1}E=\frac{1}{2^{d_Y}}\left(2^{d_Y},0,\cdots ,0\right)E=1.$$

□

Note that we cannot guarantee ${\widehat{p}}_i,i=1,\dots ,2^{d_Y}$ to be positive. Instead of p = H⁻¹ E, we consider the following optimization problem to solve p:

$$\begin{gathered} \text{min}\Vert Hp-E{\Vert }_1, \\ s\text{.}t.p_i⩾0,i=1,\dots ,2^{d_Y}, \\ \sum _i{p}_i=1. \end{gathered}$$ (9)

Since the cells can be viewed as the bins of the histogram of Y, we essentially estimate the distribution of Y as the resolutions are decomposed in an arbitrary delicate fashion. In practice, a finite d_Y is used, and we can smooth the histogram to approximate the distribution.

3. Multivariate extensions

Now we extend our framework to the multivariate case. For a q-dimensional X = (X₁, …, X_q)^T, we perform a binary expansion to every marginal CDF-transformation variable, denoted by $U_{X_j},j=1,\dots ,q$, up to the d_X-th order, and we have

$$U_{X_j}=\sum _{k=1}^{d_x}\frac{A_{jk}}{2^k},j=1,\dots ,q.$$ (10)

Denote ${\dot{A}}_{jk}=2A_{jk}-1$, j = 1, …, q, k = 1, … d_x. The σ-field

$$\sigma \left(U_{X_1},\dots ,U_{X_q}\right)=\sigma \left({\dot{A}}_{11},\dots ,{\dot{A}}_{1d_X},\dots ,{\dot{A}}_{q1},\dots ,{\dot{A}}_{q{d}_X}\right)$$

is generated by the binary filtration of all q covariates. We can find a basis of this σ-field with totally $2^{q{d}_x}-1$ variables. This basis set includes all possible patterns with respect to X_j, j = 1, …, q. These patterns can be divided into two groups. One group has terms involving binary variables from only one dimension of X, which capture the marginal patterns. For example, both the terms ${\dot{A}}_{11}$ and ${\dot{A}}_{11}{\dot{A}}_{12}$ are marginal patterns with respect to X₁. The second group has terms with binary variables from at least two covariates, which reflect interactions of the corresponding covariates. For example, ${\dot{A}}_{11}{\dot{A}}_{21}$ corresponds to the interaction of (X₁, X₂), and ${\dot{A}}_{11}{\dot{A}}_{21}{\dot{A}}_{31},{\dot{A}}_{11}{\dot{A}}_{21}{\dot{A}}_{31}{\dot{A}}_{32}$ are terms with respect to the three-way interaction (X₁, X₂, X₃). Therefore we refer to interaction terms as the patterns reflecting interaction of X, instead of product of some ${\dot{A}}_{jk}$ ’s. Note that the basis set considers until q-way interaction terms. However, three-way and higher-order interactions often contribute little to the model, and they are quite complex and difficult to interpret. Thus we only consider the main effects and two-way interaction terms in the basis set. Including $2^{d_x}-1$ main effect terms for each of the q explanatory variables, and ${\left(2^{d_x}-1\right)}^2$ interaction terms of each pair of the explanatory variables, there are $L=q\left(2^{d_x}-1\right)+C_q^2{\left(2^{d_x}-1\right)}^2$ patterns in total.

To cope with high dimensional data, some pre-screening procedures can be performed to reduce the dimension of the patterns. We do not rashly reduce the maximum number of variables in a pattern term, since each of them includes information of some specific pattern, due to the orthogonal property of the binary expansion. Instead, an approach to pairwisely test the effect of each pattern on the response variable is reasonable. To this end, we modify the binary expansion testing (BET) method (Zhang, 2019), which was originally developed to test independence of two variables, as a pre-screening method for patterns. We also extend BET to a general version, which tests the independence of multiple variables and is used to pre-screen interactions.

In the following, we first revisit the BET method, and extend it as a method of pattern pre-screening in Section 3.1. In Section 3.2, we generalize BET to pre-screen interactions by testing the independence of the response and the interaction patterns.

3.1. BET as a pre-screening approach

BET is a nonparametric method of testing dependence between two continuous variables in a distribution-free setting. Hence BET can be used on Y and X. With the binary expansion on U_Y and U_X, the interactions of the basis of the σ-field $\sigma \left({\dot{B}}_1,\dots ,{\dot{B}}_{d_Y}\right)$ and $\sigma \left({\dot{A}}_1,\dots ,{\dot{A}}_{d_X}\right)$ show all possible dependence patterns. Similar as the definition of b in Section 2.3, we use $a={\left(a_1,\dots ,a_{d_X}\right)}^T$ to identify the patterns of X. We denote the interaction pattern of a and b by $ab≔{\left(a_1,\dots ,a_{d_X},b_1,\dots ,b_{d_Y}\right)}^T$. The interaction pattern ab can partition the unit square [0, 1]² with half positive regions and half negative regions. The difference of the counts in the two regions reflects whether Y and X are independent in terms of the particular interaction pattern. When U_Y and U_X are independent, the counts of the observations in the positive and negative regions should be similar. When they are not independent, there will be significant difference of counts. Denote S_ab as the difference of the counts with respect to the interaction pattern ab. Zhang (2019) gave the result of the distribution of S_ab in the following lemma.

Lemma 4.

1. When marginal distributions are known, U_Y and U_X are independent if and only if

   $$\frac{S_{ab}+n}{2}~\mathit{\text{Binomial}}\left(n,\frac{1}{2}\right),a\ne 0,b\ne 0.$$
2. When marginal distributions are unknown, U_Y and U_X are estimated by the empirical CDF transformations ${\widehat{U}}_Y$ and ${\widehat{U}}_X$ respectively, then ${\widehat{U}}_Y$ and ${\widehat{U}}_X$ are independent if and only if

   $$\frac{{\widehat{S}}_{ab}+n}{4}~\mathit{\text{Hypergeometric}}\left(n,\frac{n}{2},\frac{n}{2}\right),a\ne 0,b\ne 0,$$

   where ${\widehat{S}}_{ab}$ denotes the difference of the counts with respect to the interaction pattern ab according to ${\widehat{U}}_Y$ and ${\widehat{U}}_X$.

In this way, BET decomposes the information of the relationship between Y and X into interaction patterns. Figure 4 shows all the 9 interaction patterns with depth d_X = 2 and d_Y = 2. An obvious dependence pattern is ${\dot{A}}_1{\dot{A}}_2{\dot{B}}_1$, which includes most points in the white region.

Fig. 4.

The dependence of Y on X is from the model Y = X² + ε, where X ~ Uniform (−2, 2) and ε ~ N (0, 0.25). BET detects the dependence through the nine patterns. The pattern ${\dot{A}}_1{\dot{A}}_2{\dot{B}}_1$ shows the most obvious difference of counts of blue and white regions.

With d_X and d_Y large enough, BET can detect arbitrarily complicated dependence. BET also helps indicate the pattern of dependence, since the significant patterns from BET imply how Y depends on X. This inspires us to focus on the detection of the significant patterns and regard BET as a pattern-screening approach. Performing BET pairwisely on {(Y, X_j⁾, j = … 1, q}, one can reduce all the patterns on X_j to only those dependent ones. We regard the patterns of X that are detected to be dependent with at least one resolution of Y as relevant variables in the penalized logistic regressions. Namely, denoting the pattern vector of X_j by ${\dot{\mathit{\text{A}}}}^j={\left({\dot{A}}_{(1)}^j,\dots ,{\dot{A}}_{\left(2^{d_X}-1\right)}^j\right)}^T\triangleq {\left({\dot{A}}_1^j,\dots ,{\dot{A}}_{d_X}^j,{\dot{A}}_1^j{\dot{A}}_2^j,\dots ,{\dot{A}}_{d_X-1}^j{\dot{A}}_{d_X}^j,\dots ,\prod _{i=1}^{d_X}{\dot{A}}_i^j\right)}^T$, we obtain the sets of relevant patterns $R_j^{\mathit{\text{main}}}≔\left\{{\dot{A}}_{(l)}^j:\exists {\dot{B}}_{(m)}s\text{.}t.\left({\dot{B}}_{(m)},{\dot{A}}_{(l)}^j\right)is\mathit{\text{dependent}},l=1,\dots ,2^{d_x}-1\right\}$, j = 1, …, q. Note that we consider the patterns in ${\cup }_{j=1}^q{R}_j^{\mathit{\text{main}}}$ as predictors in regressions for all resolutions of Y, rather than only in the regression for the particular ${\dot{B}}_{\text{(}m\text{)}}$ such that $\left({\dot{B}}_{\text{(}m\text{)}},{\dot{A}}_{(l)}^j\right)$ is dependent. This can help to avoid false negatives. False positives can be controlled by the lasso shrinkage.

3.2. A generalized BET as an interaction pre-screening method

The pairwise BET procedure selects dependent marginal patterns. Furthermore, aiming to select interaction patterns, we first generalize the original BET to test independence of Y and the joint distribution of X_i and X_j, i, j = 1, …, q. With marginal binary expansions on $U_Y,U_{X_i},U_{X_j}$ respectively, the σ-field generated by U_Y, $U_{X_i}$ and $U_{X_j}$ are $\sigma \left(U_Y\right)=\sigma \left({\dot{B}}_1,\dots ,{\dot{B}}_{d_Y}\right)$ and $\sigma \left(U_{X_i},U_{X_j}\right)=\sigma \left({\dot{A}}_{i1},\dots ,{\dot{A}}_{i{d}_X},{\dot{A}}_{j1},\dots ,{\dot{A}}_{j{d}_X}\right)$. We aim to test all possible dependence patterns from each pair of the two σ-fields. Denote the pattern of X_j, j = 1, …, q, by $a_j={\left(a_{j1},\dots ,a_{j{d}_X}\right)}^T$, and the three-way interaction pattern of a_i, a_j and b by $a_i{a}_{j}b={\left(a_{i1},\dots ,a_{i{d}_x},a_{j1},\dots ,a_{j{d}_x},b_1,\dots ,b_{d_Y}\right)}^T$. Similar to the idea of the original BET, a_i, a_j b can be viewed as a partition of the cube [0, 1]³ with half positive and half negative regions. One can test the dependence of Y and the joint X_i, X_j by the different counts of the two regions $S_{a_i{a}_{j}b}$. Figure 5 shows three aspects of a significant interaction pattern. Hence we use the generalized BET to pre-screen the interaction predictors in the regressions. We perform the generalized BET pairwisely on {(Y, X_i, X_j⁾, j = 1, …, q, i = 1, …, j − 1} and obtain the sets of significant interactions

$$\begin{gathered} R_{ij}^{\mathit{\text{interaction}}}=\left\{\left({\dot{A}}_{\left(l_1\right)}^i,{\dot{A}}_{\left(l_2\right)}^j\right):\exists {\dot{B}}_{(q)}s\text{.}t.\left({\dot{B}}_{(m)},{\dot{A}}_{\left(l_1\right)}^i,{\dot{A}}_{\left(l_2\right)}^j\right)is\mathit{\text{dependent}},l_1,l_2=1,\dots ,2^{d_X}-1\right\}, \\ j=1,\dots ,q,i=1,\dots ,j. \end{gathered}$$

Fig. 5.

The dependence of Y and X₁, X₂ is from the model Y = X₁X₂ + ε, where $X_1,X_2\stackrel{i\text{.}i\text{.}d\text{.}}{~}\mathit{\text{Uniform}}(-2,2)$ and ε ~ N (0, 0.25). The generalized BET detects the pattern ${\dot{A}}_{11}{\dot{A}}_{21}{\dot{B}}_1$ with the most obvious difference in counts of the positive region (white region with red points) and the negative region (blue region with yellow points).

With the two pre-screening procedures, eventually, the predictor set is

$$R=\left({\cup }_{i=1}^q{R}_i^{\mathit{\text{main}}}\right)\cup \left({\cup }_{i<j}{R}_{ij}^{\mathit{\text{interaction}}}\right).$$ (11)

We refer to the pre-screening based on BET and the generalized BET as the BET screening. Algorithm 1 gives the procedure of resolution-wise regression, including the pre-screening and the framework of the estimation.

4. Theoretical Studies

In this section, we first show that the BET screening is a sure independence screening approach (Fan and Song, 2010), which reduces the number of patterns L from exponential growth to O(n). The consistency result of the estimated cell probabilities is also established with the random design and a fixed d_Y. We allow the dimension q and d_X to grow with n.

For the m-th logistic regression, assume that the binary data ${\left\{Z_{m,i}\right\}}_{i=1}^n={\left\{\left({\dot{A}}_i,{\dot{B}}_{(m),i}\right)\right\}}_{i=1}^n$ from the marginal empirical CDF transformation observations ${\left\{\left(x_i,y_i\right)\right\}}_{i=1}^n$ are i.i.d. copies of $\left(\dot{A},{\dot{B}}_{(m)}\right)$, where ${\dot{A}}_i={\left({\dot{A}}_{(1),i},\dots ,{\dot{A}}_{(L),i}\right)}^T$ is the i-th sample of the L-dimensional binary random vector $\dot{A}={\left({\dot{A}}_{(1)},\dots ,{\dot{A}}_{(L)}\right)}^T$, and ${\dot{B}}_{(m),i}$ is the i-th sample of the binary response ${\dot{B}}_{(m)}$. Denote $A_i≔{\left({\tilde{A}}_{(1),i},\dots ,{\tilde{A}}_{(L),i}\right)}^T$, i = 1, …, n, as the samples of the covariates $A≔\left({\tilde{A}}_{(1)},\dots ,{\tilde{A}}_{(L)}\right)$ that are standardized to have mean zero and standard deviation one for each covariate. We have $A_i={\dot{\mathit{\text{A}}}}_i/\sqrt{n}$. Denote ${\tilde{B}}_{(m),i}$ as the i-th sample of ${\tilde{B}}_{(m)}$ taking values from {0, 1}. We have ${\tilde{B}}_{(m)}=\left({\dot{B}}_{(m)}+1\right)/2$. The maximum marginal likelihood estimator (MMLE) ${\tilde{\beta }}_{m,j}$ for the logistic regression (6), which is a special case of the models in Fan and Song (2010), is defined as the minimizer of the negative log-likelihood of the component-wise regression,

$${\tilde{\beta }}_{m,j}=\underset{{\beta }_{m,j}}{\text{arg}\text{min}}\frac{1}{n}\sum _{i=1}^n-{\tilde{B}}_{(m),i}{\tilde{A}}_{(j),i}{\beta }_{m,j}+\text{log}\left(1+e^{{\tilde{A}}_{(j),i}{\beta }_{m,j}}\right),j=1,\dots ,L.$$ (12)

We correspondingly define the population version of the MMLE by

$${\beta }_{m,j}^M=\underset{{\beta }_{m,j}}{\text{arg}\text{min}}\frac{1}{n}\sum _{i=1}^n\mathbb{E}\left[-{\tilde{B}}_{(m),i}{\tilde{A}}_{(j),i}{\beta }_{m,j}+\text{log}\left(1+e^{{\tilde{A}}_{(j),i}{\beta }_{m,j}}\right)\right],j=1,\dots ,L.$$

Denote the true regression coefficient vector by ${\beta }_m^0≔{\left({\beta }_{m,1}^0,\dots ,{\beta }_{m,L}^0\right)}^T$. Let ${ℳ}_m^{*}≔\left\{1⩽j⩽L:{\beta }_{m,j}^0\ne 0\right\}$ be the true index set of non-zero coefficients. We remark here that our overall goal of the analysis is prediction of the response rather than inference of slopes. Therefore, although when d_Y and d_X are large, and the overall paramerization might become unidentifiable, it will not harm the prediction results, as studied in Greenshtein and Ritov (2004).

We now provide the theoretical justifications of our method.

Assumption 1. $\left|cov\left(\frac{e^{{\mathit{\text{A}}}^T{\beta }_m^0}}{1+e^{{\mathit{\text{A}}}^T{\beta }_m^0}},{\widetilde{\mathit{\text{A}}}}_{(j)}\right)\right|⩾c_{1,m}{n}^{-{\kappa }_m}$ for $j\in {ℳ}_m^{*}$ with constants c_1,m > 0 and 0 < κ_m < 1 / 2.

Assumption 1 is analogous to Condition E of Fan and Song (2010). It ensures that the marginal signals are stronger than the stochastic noise. Within the selected set R, denote a_(j) b_(m) as the pattern corresponding to ${\dot{A}}_{(j)}$ and ${\dot{B}}_{\text{(}m\text{)}}$, and let the index set of selected variables using BET screening be ${ℳ}_{m,{\delta }_{n,m}}≔\left\{1⩽j⩽L:S_{a_{(j)}{b}_{(m)}}⩾{\delta }_{n,m}\right\}$, for some threshold δ_n,m. The following theorem shows that BET screening possesses the sure independence screening property.

Theorem 1. For any c_2,m > 0, there exists a positive constant c_3,m such that

$$\begin{gathered} P\left(\underset{1⩽j⩽L}{\text{max}}\left|{\tilde{\beta }}_{m,j}-{\beta }_{m,j}^M\right|⩾c_{2,m}{n}^{-{\kappa }_m}\right) \\ ⩽L\left\{\text{exp}\left(-c_{3,m}{n}^{1-2{\kappa }_m}/{\left(k_{n,m}{K}_{n,m}\right)}^2\right)+n{h}_{1,m}\text{exp}\left(-h_{0,m}{K}_{n,m}^{{\alpha }_m}\right)\right\}, \end{gathered}$$

where k_n,m, K_n,m, h_0,m, h_1,m, α_m are some positive constants. If, in addition, Assumption 1 holds, the BET screening possesses a sure independence screening property. By taking ${\delta }_{n,m}=O\left(n^{\frac{1}{2}-{\kappa }_m}\right)$, we have

$$P\left({ℳ}_m^{*}\subset {ℳ}_{m,{\delta }_{n,m}}\right)⩾1-s_m\left\{\text{exp}\left(-c_{3,m}{n}^{1-2{\kappa }_m}/{\left(k_{n,m}{K}_{n,m}\right)}^2\right)+n{h}_{1,m}\text{exp}\left(-h_{0,m}{K}_{n,m}^{{\alpha }_m}\right)\right\},$$

where $s_m≔\left|{ℳ}_m^{*}\right|$, the number of nonsparse elements.

Assumption 2. The variance $var\left(A^T{\beta }_m^0\right)$ is bounded from above and below.

Assumption 2 is analogous to Condition F of Fan and Song (2010). The following theorem shows that the BET screening can reduce the dimension from $O\left(q^2{4}^{d_x}\right)$ to $O\left(n^{2{\kappa }_m}\right)$.

Theorem 2. Under Assumption 2, we have for any ${\delta }_{n,m}=O\left(n^{\frac{1}{2}-{\kappa }_m}\right)$, and the same constants c_3,m, k_n,m, K_n,m, h_0,m, h_1,m, α_m as in Theorem 1 such that

$$P\left(\left|{ℳ}_{m,{\delta }_{n,m}}\right|⩽O\left(n^{2{\kappa }_m}\right)\right)⩾1-L\left\{\text{exp}\left(-c_{3,m}{n}^{1-2k_m}/{\left(k_{n,m}{K}_{n,m}\right)}^2\right)+n{h}_{1,m}\text{exp}\left(-h_{0,m}{K}_{n,m}^{{\alpha }_m}\right)\right\}.$$

Here, we briefly describe the results, whose details are given in the Appendix. Let $r≔{\text{max}}_{1⩽m⩽2^{d_Y}-1}\left|{ℳ}_{m,{\delta }_{n,m}}\right|$ and ${\dot{A}}^s≔{\left({\dot{A}}_{(1)}^s,\dots ,{\dot{A}}_{(r)}^s\right)}^T$ is the predictor vector including the selected r patterns. Denote the true coefficient vector of β_m by ${\beta }_m^0$. The estimate of β_m is

$${\widehat{\beta }}_m=\underset{\beta }{\text{arg}\text{min}}\sum _{i=1}^n\text{log}\left(1+e^{-{\dot{B}}_{(m),i}{f}_m\left({\dot{A}}_i^s\right)}\right)+{\lambda }_m\Vert \beta {\Vert }_1\},m=1,\dots ,2^{d_Y}-1,$$

where ||·||₁ is the ℓ₁-norm, λ_m is a tuning parameter ${\dot{A}}_i^S$, is the binary expansion corresponding to ${\dot{A}}^S$ for the ith observation, and f_m is the mth logistic regression function $m=1,\dots ,2^{d_Y}-1$. Let $f_m^0$ be the true function between ${\dot{B}}_{\text{(}m\text{)}}$ and ${\dot{A}}^S$. Denote the index set of non-zero coefficients by $S_m^0≔\left\{j:{\beta }_{m,j}^0\ne 0\right\}$, and the cardinality of $S_m^0$ by $s_m≔\left|S_m^0\right|$.

According to the BID equation, we estimate p by solving the optimization (9). From the optimization H_m+1 p, is an approximation of ${\widehat{e}}_m$, where H_{m +1} is the (m + 1) -th row of H, since ${\widehat{e}}_m$ is the (m + 1) -th entry of E. Hence g(H_m+1 p) is the estimated m-th regression function corresponding to p. The following theorem gives the consistency of cell probability vector p in terms of excess risk of g(H_m+1 p).

Theorem 3. Assume Assumptions (1) and (2), and (3) to (5) given in the Appendix hold, where Assumption (3) in the Appendix holds with the set $S_m^0$. For the logistic regression with covariates corresponding to the BET screening set ${ℳ}_{m,{\delta }_{n,m}}$, suppose that λ_m satisfies ${\lambda }_m⩾8{\lambda }_m^0$. Then on the set 𝒯_m, we have,

$$ℰ\left(g\left(H_{m+1}p\right)\right)+{\lambda }_m{\Vert {\widehat{\beta }}_m-{\beta }_m^0\Vert }_1⩽6^{ℰ}\left(f_m^0\right)+\frac{16{\lambda }_m^2{s}_m}{c_m{\varphi }_m^2}+\frac{32\lambda Ks{2}^{d_Y}}{c{\varphi }^2},$$

where K > 0 is a constant, $\lambda ={\text{max}}_{1⩽m⩽2^{d_Y}-1}{\lambda }_m$, $s={\text{max}}_{1⩽m⩽2^{d_Y}-1}{s}_m$, $c={\text{min}}_{1⩽m⩽2^{d_Y}-1}{c}_m$, ${\varphi }_m^2$ is a compatibility constant, and $\varphi ={\text{min}}_{1⩽m⩽2^{d_Y}-1}{\varphi }_m$.

5. Simulation Studies

In this section, we perform simulations to show the performance of resolution-wise regression approach. We compare our method with the following four methods:

1. Naive method, which first finds a small neighborhood of each test sample in the training set, where ∥X − X_new∥₂ is bounded by a constant, and predicts the distribution of Y | X_new by the kernel density estimation of the responses in this neighborhood.

2. SSANOVA, which fits a cubic spline with all main effects and interaction effects. Its prediction distribution is $Y\mid X_{new}~N\left({\widehat{Y}}_{\mathit{\text{ssanova}}}\mid X_{new},{\widehat{\sigma }}^2+var\left({\widehat{Y}}_{\mathit{\text{ssanova}}}\mid X_{new}\right)\right)$, where ${\widehat{Y}}_{\mathit{\text{sanova}}}\mid X_{new}$ is the SSANOVA estimation of Y given a new X_new, and ${\widehat{\sigma }}^2$ is the estimated variance of the random error.

3. Random Forest, which fits a multitude of regression trees and then averages the predictions. Its prediction distribution is $Y\mid X_{new}~N\left({\widehat{Y}}_{rf}\mid X_{new},{\sigma }_s^2\right)$, where Y_rf | X_new is the estimation of Y from random forest given a new X_new, and ${\sigma }_s^2$ is the standard error.

4. Regression mixture model (only for Example 2), which identifies the subgroups of dataset and fits multiple linear regression models. Its prediction distribution is $Y\mid X_{new}~N\left({\widehat{Y}}_{\mathit{\text{mixreg}}}\mid X_{new},{\sigma }_s^2\right)$, where ${\widehat{Y}}_{\mathit{\text{mixreg}}}\mid X_{new}$ is the estimation of Y from the regression mixture model given a new X_new which is randomly assigned into subgroups with the weights derived from training data, and ${\widehat{\sigma }}_s^2$ is the standard error.

We study the following four examples with 1024 samples for both training and testing sets.

Example 1. (Crossing lines) The predictor $x_i\stackrel{i.i.d.}{~}U(-10,10)$, i = 1, …, n. For the example with one cross on the plane, the response y_i is generated by y_i = x_iI(g_i = 0) − x_iI(g_i = 1) + ε_i, where the error ${\epsilon }_i\stackrel{i.i.d.}{~}N(0,0.5)$, i = 1, …, n, and I(·) is the indicator function with $g_i\stackrel{i.i.d.}{~}\mathit{\text{Bernoulli}}(1/2)$, i = 1, …, n. For the example with multiple crosses, the response y_i = (x_iI(g_1i = 1) − x_iI(g_1i = 2) + (x_i − 10)I(g_1i = 3) + (−x_i + 10)I(g_1i = 4)) I(x_i ≥ 0) + (x_iI(g_2i = 1) − x_iI(g_2i = 2) + (−x_i − 10)I(g_2i = 3) + (x_i = 10)I(g_2i = 4))I(x_i < 0) + ε_i, where the error ${\epsilon }_i\stackrel{i.i.d.}{~}N(0,0.5)$, i = 1, …, n, and I(·) is the indicator function with $g_{ki}\stackrel{i.i.d.}{~}\mathit{\text{Multi}}–\mathit{\text{Bern}}\left(\{1,2,3,4\},(1/4,1/4,1/4,1/4)\right)$, i = 1, …, n, k = 1, 2.

Example 2. (A mixture of linear and quadratic effects) The predictor vector (x_i1, …, x_iq)^T is generated by $x_{ij}\stackrel{i.i.d.}{~}U(-2,2)$, i = 1, …, n, j = 1, …, q. with q = 1, 5, 10. The response y_i is generated by $y_i=x_{i1}I\left(g_i=0\right)+x_{i1}^{2}I\left(g_i=1\right)+{\epsilon }_i$, which depends on only the first variable x_i1 and other variables are regarded as noise. The error ${\epsilon }_i\stackrel{i.i.d.}{~}N(0,0.05)$, i = 1, …, n, and I(·) is the indicator function with $g_i\stackrel{i.i.d.}{~}\mathit{\text{Bernoulli}}(1/2)$, i = 1, …, n.

Example 3. (Circular and spherical implicit functional relationship) The predictor vector (x_i1, …, x_iq^)T has q = 5. For the circle example, the predictors and the responses are generated from the polar coordinates x_i1 = sin(θ_i), yi = cos(θ_i) + ε_i, where the latent variable ${\theta }_i\stackrel{i.i.d.}{~}U(0,2\pi )$, i = 1, …, n, and the error ${\epsilon }_i\stackrel{i.i.d.}{~}N(0,0.05)$, i = 1, …, n. The noise variables (x_i2,…, x_iq)^T are generated by $x_{ij}\stackrel{i.i.d.}{~}U(-1,1)$ i = 1, …, n, j = 2, …, q. For the sphere example, where the latent variables ${\theta }_i\stackrel{i.i.d.}{~}U(0,\pi )$, the predictors and the responses are generated from x_i1 = sin(θ_i)cos(ϕ_i), x_i2 = sin(θ_i)sin(ϕ_i), y_i = cos(θ_i) + ε_i, where ${\varphi }_i\stackrel{i.i.d.}{~}U(0,2\pi )$, i = 1, …, n, and the error ${\epsilon }_i\stackrel{i.i.d.}{~}N(0,0.05)$, i = 1, …, n. The noise variables ${\left(x_{i3},\dots ,x_{iq}\right)}^T$ are generated by $x_{ij}\stackrel{i.i.d.}{~}U(-1,1)$, i = 1, …, n, j = 3, …, q.

Example 4. (Heterogeneous mean vesus heteroscedastic error) The predictor $x_i\stackrel{i.i.d.}{~}U(-2,2)$, i = 1, …, n. The response y_i is generated by $y_i=\left(x_i+2+x_i{\epsilon }_i\right)I\left(g_i=0\right)+\left(x_{i1}^2+{\epsilon }_i\right)I\left(g_i=1\right)$, where the error ${\epsilon }_i\stackrel{i.i.d.}{~}N\left(0,{\sigma }^2\right)$, i = 1, …, n, σ² = 0.05, 0.25, 0.5.

In each example, BET with depth 5 and a threshold of $\sqrt{\frac{2\text{log}p}{n}}$ for symmetry statistics, where p is the total number of interactions and n is the sample size, is performed for main effects screening, while generalized BET with depth 4 and the same threshold is performed for interaction effects screening. For the selection of depth, with a small depth 3, BET reaches a high power (Zhang, Zhao and Zhou, 2021). We perform simulation with different depths and the results are reported in the Appendix. We pick a depth 5 and 4, which is high enough. Two types of smoothing approaches are considered: fixed smoothing parameter (“Fixed smoothness”), and tuning the smoothing parameter by cross-validation (“CV”).

We repeat the simulation 100 times for each example. To measure the test error, we calculate the differences of prediction distributions from different methods and the underlying true distributions, the following distance measures are used: (1) Kolmogorov-Smirnov statistic $D_{KS}(P,Q)=\underset{x}{\text{sup}}|P(x)-Q(x)|$, (2) Kullback-Leibler divergence $D_{KL}={\int }_{-\infty }^{\infty }p(x)\text{log}\left(\frac{p(x)}{q(x)}\right)dx$, (3) L₁ distance $D_{L_1}={\int }_{-\infty }^{\infty }|p(x)-q(x)|dx$, where P and Q are two distributions with corresponding PDFs p (·) and q (·) respectively.

Here we display the results of Example 1–4, which are heterogeneous data, and the result from a case of a nonlinear functional relationship is in supplementary materials. Tables 1 – 4 list the results of testing errors for the three simulation examples. Figures 6–9 show the heatmaps of the prediction distributions of all test data, where the x axis is the involved variable of the predictors. We discard the heatmap of the spherical case, since it has two involved variables and cannot be shown explicitly in a heatmap.

Table 1.

Comparison of average test errors (and corresponding standard errors in parentheses) for Example 1 with respect to one or multiple crosses and three distance measures. The results for naive method, mixture of regression, SSANOVA, resolution-wise regression with fixed smoothness, and resolution-wise regression with CV are listed in the columns from left to right respectively.

Example	Measure	Naive	Mixreg	SSANOVA	Random Forest	Fixed smoothness	CV
one cross	KS	0.151	0.351	0.294	0.377	0.165	0.129
		(0.015)	(0.010)	(0.016)	(0.017)	(0.009)	(0.005)
	KL	0.362	1.396	1.031	1.835	0.386	0.265
		(0.032)	(0.067)	(0.062)	(0.156)	(0.025)	(0.014)
	L 1	0.639	1.086	1.181	1.120	0.717	0.364
		(0.034)	(0.063)	(0.073)	(0.069)	(0.035)	(0.019)
multiple	KS	0.188	0.405	0.229	0.394	0.165	0.145
		(0.007)	(0.026)	(0.016)	(0.020)	(0.010)	(0.008)
	KL	0.319	2.511	0.548	1.615	0.338	0.257
		(0.016)	(0.128)	(0.036)	(0.084)	(0.024)	(0.016)
	L 1	0.653	1.212	0.850	1.054	0.658	0.397
		(0.033)	(0.052)	(0.042)	(0.057)	(0.034)	(0.019)

Table 4.

Comparison of average test errors (and corresponding standard errors in parentheses) for Example 4 with respect to different variances of random errors and three distance measure. The results for naive method, SSANOVA, Random Forest, resolution-wise regression with fixed smoothness, and resolution-wise regression with CV are listed in the columns from left to right respectively.

Example	Measure	Naive	SSANOVA	Random Forest	Fixed smoothness	CV
σ2 = 0.05	KS	0.192	0.189	0.243	0.153	0.154
		(0.008)	(0.009)	(0.016)	(0.005)	(0.005)
	KL	0.435	0.417	0.512	0.229	0.743
		(0.022)	(0.026)	(0.030)	(0.012)	(0.034)
	L 1	0.656	0.622	0.652	0.424	0.569
		(0.034)	(0.035)	(0.033)	(0.022)	(0.024)
σ2 = 0.25	KS	0.156	0.131	0.239	0.123	0.123
		(0.006)	(0.006)	(0.014)	(0.009)	(0.010)
	KL	0.258	0.245	0.430	0.166	0.330
		(0.016)	(0.014)	(0.026)	(0.009)	(0.020)
	L 1	0.491	0.452	0.571	0.346	0.400
		(0.025)	(0.030)	(0.033)	(0.017)	(0.023)
σ2 = 0.5	KS	0.139	0.121	0.236	0.098	0.110
		(0.008)	(0.007)	(0.018)	(0.007)	(0.009)
	KL	0.201	0.208	0.421	0.142	0.167
		(0.013)	(0.017)	(0.035)	(0.009)	(0.012)
	L 1	0.422	0.408	0.554	0.296	0.308
		(0.035)	(0.024)	(0.038)	(0.019)	(0.020)

Fig. 6.

Heatmaps of prediction distributions for Example 1 with respect to one or multiple crosses and six methods: naive method, regression mixture, SSANOVA, Random Forest, resolution-wise regression with fixed smoothness, and resolution-wise regression with CV, from left to right respectively. A darker color indicates a larger PDF value at the corresponding predicted response.

Fig. 9.

Heatmaps of prediction distributions for Example 4 with respect to different variances of random errors and five methods: naive method, SSANOVA, random forest, resolution-wise regression with fixed smoothness, and resolution-wise regression with CV, from left to right respectively. A darker color indicates a larger PDF value at the corresponding predicted response.

The results indicate the best performance of resolution-wise regression. For Example 1, resolution-wise regression especially with cross-validation smoothness can identify the subgroups, while the regression mixture model does not perform well as the number of subgroups increases. The naive method has a good performance since the dependence is linear. For Example 2, resolution-wise regression can predict the probabilities around the two subgroups, thus has the best performance. SSANOVA does not perform well since it cannot recognize the subgroups. The naive method performs well only in the low-dimensional case, because in the high-dimensional case, it is difficult to find a small neighborhood with substantial train data. Random forest does not perform well because it averages the predictions from multiple regression trees and mixes the two subgroups up. Resolution-wise regression performs well in the high-dimensional case, and the distance to the true distribution only increases slightly with the effect of noise variables. For Example 3, resolution-wise regression performs the best, while the naive method has poor performance due to the dimension issue; SSANOVA fails to capture the relationship, since it cannot be expressed in an explicit regression function form; Random forest gives an averaged prediction and fails to recognize the multiple patterns at one position. For Example 4, only the resolution-wise regression successfully distinguishes the two subgroups of the heterogeneous mean with homogeneous error model and the homogeneous mean with a heteroscedastic error model.

Based on the simulation results, we can see that resolution-wise regression has a better distribution prediction when the variance σ² of the error is larger, which seems paradoxical with the point prediction of some common regression methods. In fact, for point prediction, the loss comes from the random error term. A large variance leads to a large loss. However, for distribution prediction, the loss comes from the accumulated probability of possible response values.

The variance affects the shape of the distribution of the response. For a smaller variance, the data are concentrated, and more precise resolutions are needed, which bring the difficulties for the estimation of the corresponding logistic regressions. Hence for the same expansion order d_Y, our method gives less accurate results on smaller variances. This also shows some insights for a general prediction problem that distribution prediction is a good alternative to capture the whole picture if the random error is relatively large.

6. Real Data Analysis

We analyze the real estate valuation dataset (Yeh and Hsu, 2018), obtained from UCI machine learning Repository (https://archive.ics.uci.edu). The dataset contains the unit-area price and the corresponding six explanatory variables of 414 houses collected from Sindian District, New Taipei City. The six explanatory variables include the transaction date, the house age, the distance to the nearest MRT station, the number of convenience stores in the living circle on foot, the latitude, and the longitude. We are interested in how the house price can be explained by these variables. We consider three methods: (1) Naive method with the small neighborhood of the nearest ten samples, where the distance is measured by the Mahalanobis distance (Rosenbaum, 1995) of the rank vector within each predictor; (2) SSANOVA with cubic splines on each main effects and interactions; (3) Resolution-wise regression with d_X = 5 and d_Y = 5.

There are three interesting results we find from the application of resolution-wise regression.

House prices rely on some features through a nonlinear relationship. Such a nonlinear pattern can be detected by the BET screening, as shown in Figures 10 and 11. The latitude and the longitude show skewed quadratic effects which are captured by the relevant patterns in depth 1 and depth 2 respectively. For the screening of interaction patterns, all the $C_6^2$ interactions are significant. As an example shown in Figure 11, the interaction of the latitude and the longitude shows a concentration of high-price houses, which can be pinpointed around the downtown area. There is also some linear pattern found in the data. As shown in Figure 10, the distance to the nearest MRT station and the number of convenience stores in the living circle on foot has a linear effect on the house price which is captured by the relevant pattern in depth 1.

Fig. 10.

Relevant variables and the corresponding most significant patterns. For the distance to the nearest MRT station and the latitude, there exists a linear relationship to the housing prices. For the longitude, the most asymmetric interaction is A₁A₂B₁, which implies a nonlinear dependence.

Fig. 11.

Interaction of latitude and longitude. The high-price houses concentrate around the downtown area.

Potential heterogeneity can be detected by resolution-wise regression. The proposed method clearly predicts the house price concentrating on two groups around 30 and 60, which is new information that is not provided by existing methods. For examples, the SSANOVA completely misses such heterogeneity. The naive method provides a distribution which vaguely suggests probability mass towards the right tail. However, the subgroup information identified by the naive method is not very clear.

The detected heterogeneity can also be demonstrated through the additional information from the map of the city. As shown in Figure 13, the nearest ten samples measured by the Mahalanobis distance form two groups classified by the river and the highway. The two groups differ in the house price, and both contribute to the distribution estimation. The effect of the river and the highway play the role of unobserved variable, which leads to the bimodal shape of the prediction distribution. The prediction from our method suggests that the price of the particular house is more likely to be close to the three houses on the lower-left side of the river, which has an average price of 25.75. This is verified as correct prediction through the actual map. In particular, the location of the house is indeed on the lower-left side of the river. This example thus illustrates the advantage of the proposed method. Specifically, it can detect heterogeneity in the data and provide accurate probability statements about subgroup information.

Fig. 13.

The testing sample (red pin) and the nearest 10 samples measured by the Mahalanobis distance of the rank vector within every predictor (blue pins, where two samples on the power left side have the same location, and two samples on the lower left side have the same location) on the map. In these 10 houses, four are on the lower left side of the river with an average price of 25.75, and six are on the upper right side of river with an average price of 43.87.

In summary, this real data analysis indicates that resolution-wise regression model can capture the heterogeneous pattern, thus can deliver more detailed prediction information than traditional methods. Since no distribution assumption is required, our method is rather general and robust.

7. Conclusion

In this paper, we propose resolution-wise regression model to predict the distribution of the response with heterogeneous data. The complicated relationship between the response and the explanatory variables can be decomposed into the relationship of resolutions of the response and patterns of the predictors based on binary expansions. A set of penalized logistic regressions establish the effect of patterns having on the resolutions. By BID transformation, our method can estimate the cell probability of the histogram of the response, which is an approximation of the distribution of the response. We also show the consistency of the cell probabilities. Numerical studies demonstrate the effectiveness of the proposed method.

Fig. 7.

Heatmaps of prediction distributions for Example 2 with respect to dimension q = 1, 5, 10 and five methods: naive method, SSANOVA, Random Forest, resolution-wise regression with fixed smoothness, and resolution-wise regression with CV, from left to right respectively. A darker color indicates a larger PDF value at the corresponding predicted response.

Fig. 8.

Heatmaps of prediction distributions for Example 3 with respect to circular implicit functional relationship and five methods: naive method, SSANOVA, Random Forest, resolution-wise regression with fixed smoothness, and resolution-wise regression with CV, from left to right respectively. A darker color indicates a larger PDF value at the corresponding predicted response.

Fig. 12.

Predicted distributions by naive method by the nearest ten samples, SSANOVA, and resolution-wise regression.

Table 2.

Comparison of average test errors (and corresponding standard errors in parentheses) for Example 2 with respect to dimension q = 1, 5, 10 and three distance measures. The results for naive method, SSANOVA, Random Forest, resolution-wise regression with fixed smoothness, and resolution-wise regression with CV are listed in the columns from left to right respectively.

Example	Measure	Naive	SSANOVA	Random Forest	Fixed smoothness	CV
q = 1	KS	0.103	0.215	0.256	0.169	0.167
		(0.006)	(0.005)	(0.008)	(0.013)	(0.014)
	KL	0.261	0.605	0.787	0.089	0.693
		(0.010)	(0.021)	(0.049)	(0.008)	(0.037)
	L 1	0.383	0.765	0.716	0.283	0.517
		(0.018)	(0.033)	(0.035)	(0.015)	(0.027)
q = 5	KS	0.345	0.219	0.222	0.180	0.179
		(0.019)	(0.016)	(0.012)	(0.015)	(0.014)
	KL	0.701	0.589	0.730	0.146	0.532
		(0.034)	(0.025)	(0.037)	(0.011)	(0.028)
	L 1	0.937	0.774	0.693	0.348	0.506
		(0.039)	(0.035)	(0.028)	(0.015)	(0.022)
q = 10	KS	0.344	0.210	0.213	0.169	0.183
		(0.006)	(0.006)	(0.015)	(0.018)	(0.018)
	KL	0.767	0.589	0.666	0.185	0.572
		(0.033)	(0.024)	(0.050)	(0.022)	(0.035)
	L 1	0.952	0.735	0.693	0.346	0.498
		(0.038)	(0.032)	(0.035)	(0.015)	(0.026)

Table 3.

Comparison of average test errors (and corresponding standard errors in parentheses) for Example 3 with respect to circular and spherical implicit functional relationship and three distance measures. The results for naive method, SSANOVA, Random Forest, resolution-wise regression with fixed smoothness, and resolution-wise regression with CV are listed in the columns from left to right respectively.

Example	Measure	Naive	SSANOVA	Random Forest	Fixed smoothness	CV
Circle	KS	0.175	0.201	0.398	0.172	0.156
		(0.010)	(0.014)	(0.030)	(0.010)	(0.008)
	KL	0.378	0.436	4.180	0.264	0.404
		(0.019)	(0.032)	(0.322)	(0.008)	(0.027)
	L 1	0.677	0.761	1.384	0.515	0.535
		(0.039)	(0.034)	(0.108)	(0.022)	(0.025)
Sphere	KS	0.170	0.184	0.413	0.183	0.161
		(0.012)	(0.010)	(0.037)	(0.009)	(0.006)
	KL	0.369	0.411	4.553	0.310	0.339
		(0.015)	(0.021)	(0.355)	(0.009)	(0.016)
	L 1	0.664	0.735	1.439	0.580	0.570
		(0.026)	(0.038)	(0.080)	(0.024)	(0.025)

Appendix

Proof of Theorem 1:

We split the whole proof into two steps: (1) the selected variables from the BET screening are equivalent to those from the sure independence screening based on the MMLE, (2) the proposed logistic regression satisfies the conditions in Fan and Song (2010) to achieve the sure independence screening property.

1. The BET test statistic

   $$\begin{gathered} S_{a_{(j)}{b}_{(m)}}=\left|\sum _{i=1}^{n}I\left({\dot{A}}_{(j),i}{\dot{B}}_{(m),i}=1\right)-\sum _{i=1}^{n}I\left({\dot{A}}_{(j),i}{\dot{B}}_{(m),i}=-1\right)\right|=\left|\sum _{i=1}^n{\dot{A}}_{(j),i}{\dot{B}}_{(m),i}\right| \\ =\sqrt{n}\left|\sum _{i=1}^n{\tilde{A}}_{(j),i}{\dot{B}}_{(m),i}\right|. \end{gathered}$$
   By the definition in (12), and ${\tilde{B}}_{(m)}=\left({\dot{B}}_{(m)}+1\right)/2$, the MMLE can be obtained by the optimization with respect to ${\dot{B}}_{(m),i}$ ’s, i.e.,

   $${\tilde{\beta }}_{m,j}=\underset{{\beta }_{m,j}}{\text{arg}\text{min}}\frac{1}{n}\sum _{i=1}^n-\frac{{\dot{B}}_{(m),i}+1}{2}{\tilde{A}}_{(j),i}{\beta }_{m,j}+\text{log}\left(1+e^{{\tilde{A}}_{(j),i}{\beta }_{m,j}}\right),j=1,\dots ,L.$$
   Setting the derivative of the above objective function with respect to β_m,j to be zero, we have that ${\tilde{\beta }}_{m,j}$ satisfies

   $$\frac{1}{n}\sum _{i=1}^n\frac{{\dot{B}}_{(m),i}{\tilde{A}}_{(j),i}}{2}=\frac{1}{n}\sum _{i=1}^n\left(\frac{{\tilde{A}}_{(j),i}{e}^{{\tilde{A}}_{(j),i}{\tilde{\beta }}_{m,j}}}{1+e^{{\tilde{A}}_{(j),i}{\tilde{\beta }}_{m,j}}}-\frac{{\tilde{A}}_{(j),i}}{2}\right)=\frac{1}{n}\sum _{i=1}^n\frac{{\tilde{A}}_{(j),i}{e}^{{\tilde{A}}_{(j),i}{\tilde{\beta }}_{m,j}}}{1+e^{{\tilde{A}}_{(j),i}{\tilde{\beta }}_{m,j}}}.$$
   The second equation holds since there are n / 2 samples with ${\tilde{A}}_{(j),i}=1/\sqrt{n}$ and the other n / 2 samples with ${\tilde{A}}_{(j),i}=-1/\sqrt{n}$, due to the binary expansion from the empirical CDF transformation. Denote $t\left({\tilde{\beta }}_{m,j}\right)≔\sum _{i=1}^n\frac{{\tilde{A}}_{(j),i}{e}^{{\tilde{A}}_{(j),i}{\tilde{\beta }}_{m,j}}}{1+e^{{\tilde{A}}_{(j),i}{\tilde{\beta }}_{m,j}}}$. Differentiating with respect to ${\tilde{\beta }}_{m,j}$, we obtain $\frac{dt\left({\tilde{\beta }}_{m,j}\right)}{d{\beta }_{m,j}}=\sum _{i=1}^n\frac{{\tilde{A}}_{(j),i}^2{e}^{{\tilde{A}}_{(j),i}{\tilde{\beta }}_{m,j}}}{{\left(1+e^{{\tilde{A}}_{(j),i}{\tilde{\beta }}_{m,j}}\right)}^2}>0$. Hence $t\left({\tilde{\beta }}_{m,j}\right)$ is strictly increasing with respect to ${\tilde{\beta }}_{m,j}$. For γ_m,n > 0, if ${\tilde{\beta }}_{m,j}⩾{\gamma }_{n,m}$, $t\left({\tilde{\beta }}_{m,j}\right)⩾t\left({\gamma }_{n,m}\right)$. If ${\tilde{\beta }}_{m,j}⩽-{\gamma }_{n,m}$,

   $$\begin{gathered} t\left({\tilde{\beta }}_{m,j}\right)⩽t\left(-{\gamma }_{n,m}\right)=\sum _{i=1}^n\frac{{\tilde{A}}_{(j),i}{e}^{-{\tilde{A}}_{(j),i}{\gamma }_{n,m}}}{1+e^{-{\tilde{A}}_{(j),i}{\gamma }_{n,m}}}=\sum _{i=1}^n\frac{{\tilde{A}}_{(j),i}}{1+e^{{\tilde{A}}_{(j),i}{\gamma }_{n,m}}}=\sum _{i=1}^n\left({\tilde{A}}_{(j),i}-\frac{{\tilde{A}}_{(j),i}{e}^{{\tilde{A}}_{(j),i}{\gamma }_{n,m}}}{1+e^{{\tilde{A}}_{(j),i}{\gamma }_{n,m}}}\right) \\ =-t\left({\gamma }_{n,m}\right). \end{gathered}$$
   Since $S_{a_{(j)}{b}_{(m)}}=2\sqrt{n}\left|t\left({\tilde{\beta }}_{m,j}\right)\right|$, we have $S_{a_{(j)}{b}_{(m)}}⩾2\sqrt{n}{\gamma }_{n,m}$. Hence we have the variable selection index set $\left\{1⩽j⩽L:S_{a_{(j)}{b}_{(m)}}⩾2\sqrt{n}t\left({\gamma }_{n,m}\right)\right\}\supseteq \left\{1⩽j⩽L:\left|{\tilde{\beta }}_{m,j}\right|⩾{\gamma }_{n,m}\right\}$. Similarly, we have $\left\{1⩽j⩽L:S_{a_{(j)}{b}_{(m)}}⩾2\sqrt{n}t\left({\gamma }_{n,m}\right)\right\}\subseteq \left\{1⩽j⩽L:\left|{\tilde{\beta }}_{m,j}\right|⩾{\gamma }_{n,m}\right\}$. Denote $I_j^{+}≔\left\{1⩽i⩽n:{\tilde{A}}_{(j),i}=1/\sqrt{n}\right\},I_j^{-}≔\left\{1⩽i⩽n:{\tilde{A}}_{(j),i}=-1/\sqrt{n}\right\}$. Taking ${\gamma }_{n,m}=c_{4,m}{n}^{-{\kappa }_m}$ for some c_4,m > 0, we have

   $$\begin{gathered} 2\sqrt{nt}\left({\gamma }_{n,m}\right)=2\sqrt{n}\sum _{i=1}^n\frac{{\tilde{A}}_{(j),i}{e}^{{\tilde{A}}_{(j),i}{c}_{4,m}{n}^{-{\kappa }_m}}}{1+e^{{\tilde{A}}_{(j),i}{c}_{4,m}{n}^{-{\kappa }_m}}} \\ =2\sqrt{n}\sum _{i\in I_j^{+}}\frac{\frac{1}{\sqrt{n}}{e}^{c_{4,m}{n}^{-\frac{1}{2}-{\kappa }_m}}}{1+e^{c_{4,m}{n}^{-\frac{1}{2}-{\kappa }_m}}}-2\sqrt{n}\sum _{i\in I_j^{-}}\frac{\frac{1}{\sqrt{n}}{e}^{-c_{4,m}{n}^{-\frac{1}{2}-{\kappa }_m}}}{1+e^{-c_{4,m}{n}^{-\frac{1}{2}-{\kappa }_m}}} \\ =\frac{n{e}^{c_{4,m}{n}^{-\frac{1}{2}-{\kappa }_m}}}{1+e^{c_{4,m}{n}^{-\frac{1}{2}-{\kappa }_m}}}-\frac{n}{1+e^{c_{4,m}{n}^{-\frac{1}{2}-{\kappa }_m}}} \\ =\frac{n\left(e^{c_{4,m}{n}^{-\frac{1}{2}-{\kappa }_m}}-1\right)}{1+e^{c_{4,m}{n}^{-\frac{1}{2}-{\kappa }_m}}}=\frac{nO\left(n^{-\frac{1}{2}-{\kappa }_m}\right)}{1+e^{c_{4,m}{n}^{-\frac{1}{2}-{\kappa }_m}}} \\ =O\left(n^{\frac{1}{2}-{\kappa }_m}\right). \end{gathered}$$

   Hence the BET screening is equivalent to the sure independence screening based on MMLE. This ensures that the estimation methods are the same as those in Fan and Song (2010).

2. Under the binary expansion, the variables are bounded in [0,1] after empirical CDF transformation, so that the conditions A – C in Fan and Song (2010) are naturally satisfied. Assumption 1 is analogous to Condition E. So we only need to check Condition D. For the proposed logistic regression, let w₀ = 1, and we have

   $$\begin{gathered} {\mathbb{E}}_{\text{exp}\left(\text{log}\left(1+e^{A^T{\beta }_m^0+w_0}\right)-\text{log}\left(1+e^{A^T{\beta }_m^0}\right)\right)}+{\mathbb{E}}_{\text{exp}\left(\text{log}\left(1+e^{A^T{\beta }_m^0-w_0}\right)-\text{log}\left(1+e^{A^T{\beta }_m^0}\right)\right)} \\ =\mathbb{E}\frac{1+e^{A^T{\beta }_m^0+w_0}}{1+e^{A^T{\beta }_m^0}}+\mathbb{E}\frac{1+e^{A^T{\beta }_m^0-w_0}}{1+e^{A^T{\beta }_m^0}}=2+\mathbb{E}\frac{{\left(e^{w_0/2}-e^{-w_0/2}\right)}^2}{1+e^{-A^T{\beta }_m^0}} \\ ⩽2+{\left(e^{w_0/2}-e^{-w_0/2}\right)}^2=2+{\left(e^{1/2}-e^{-1/2}\right)}^2. \end{gathered}$$

   Taking w₁ = 2, h_1,m = 3, h_0,m = 1, α = 1 satisfies Condition D.

   Hence by Theorem 4 in Fan and Song (2010), for ${\delta }_{n,m}=O\left(n^{\frac{1}{2}-{\kappa }_m}\right)$, the sure independence screening property is achieved.

Proof of Theorem 2:

For logistic regression, the function b (·) of Condition G in Fan and Song (2010) is b(x) = log(1 + e^x). Since $0<\frac{d^{2}b(x)}{d{x}^2}=\frac{e^{-x}}{{\left(1+e^{-x}\right)}^2}<1$. Condition G in Fan and Song (2010) is met. Together with Assumption 2, by Theorem 5 in Fan and Song (2010), the proof is completed.

Let ℱ be a normed real vector space. For the m-th logistic regression function $f_m\in ℱ:{\{-1,1\}}^r\to ℝ$, where $r≔{\text{max}}_{1⩽m⩽2^{d_Y}-1}\left|{ℳ}_{m,{\delta }_{n,m}}\right|$, the negative log-likelihood loss ${\rho }_{f_m}:{\{-1,1\}}^{r+1}\to ℝ$ is ${\rho }_{f_m}\left(Z_m\right)={\rho }_{f_m}\left({\dot{\mathit{A}}}^S,{\dot{B}}_{(m)}\right)=\text{log}\left(1+e^{-{\dot{B}}_{(m)}f\left({\dot{A}}^s\right)}\right)$, $m=1,\dots ,2^{d_Y}-1$ where ${\stackrel{.}{A}}^S≔{\left({\dot{A}}_{(1)}^S,\dots ,{\dot{A}}_{(r)}^S\right)}^T$ is the predictor vector including the selected r patterns. For a loss function ${\rho }_{f_m}$, define the empirical risk for ${\rho }_{f_m}$ by $P_n{\rho }_{f_m}≔\frac{1}{n}\sum _{i=1}^n{\rho }_{f_m}\left(Z_{m,i}\right)$, and the theoretical risk by $P{\rho }_{f_m}≔\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\rho }_{f_m}\left(Z_{m,i}\right)$. Consider the collection ℱ to be a linear-model class, i.e., $ℱ≔\left\{f^{\beta }:\beta \in {ℝ}^p\right\}$, where β ↦ f_β is linear. For the m-th regression, note that the true coefficient vector ${\beta }_m^0$ is the minimizer of the theoretical risk

$${\beta }_m^0≔\underset{\beta }{\text{arg}\text{min}}P{\rho }_{f_m^{\beta }},$$ (13)

and $f_m^0≔f_m^{{\beta }_m^0}$. We assume for simplicity that the minimum exists and unique. For $f_m^{\beta }\in ℱ$, the excess risk is defined by $ℰ\left(f_m^{\beta }\right)≔P\left({\rho }_{f_m^{\beta }}-{\rho }_{f_m^0}\right)$. The lasso estimator is ${\widehat{\beta }}_m=\text{arg}\text{mi}{\text{n}}_{\beta }\left\{P_n{\rho }_{f_m^{\beta }}+{\lambda }_m\Vert \beta {\Vert }_1\right\}$, $m=1,\dots ,2^{d_Y}-1$, where ∥·∥₁ is the ℓ₁-norm and λ_m is a tuning parameter. The estimation of the regression function is ${\widehat{f}}_m=f_m^{{\widehat{\beta }}_m}$.

Denote ${\pi }_m\left({\dot{A}}^S\right)=P\left({\dot{B}}_{(m)}=1\mid {\dot{A}}^S\right)$ and $e_m\left({\dot{A}}^S\right)=\mathbb{E}\left({\dot{B}}_{(m)}\mid {\dot{A}}^S\right)$. We have $e_m\left({\dot{A}}^S\right)=2{\pi }_m\left({\dot{A}}^S\right)-1$. Hence based on the link function of logistic regression, we can define a functional g mapping e_m (·) to the regression function $f_m^{\beta }(\cdot )$:

$$g\left(e_m\right)(\cdot )≔f_m^{\beta }(\cdot )=\text{log}\left(\frac{{\pi }_m(\cdot )}{1-{\pi }_m(\cdot )}\right)=\text{log}\left(\frac{\frac{e_m(\cdot )+1}{2}}{1-\frac{e_m(\cdot )+1}{2}}\right).$$ (14)

Denoting $e_m^0({\dot{A}}^S)$ as the true expectation corresponding to ${\beta }_m^0$, by (14), we have $f_m^0=g\left(e_m^0\right)$. Similarly, recall that ${\widehat{e}}_m({\dot{A}}^S)$ is the estimated expectation, and thus we have ${\widehat{f}}_m=g\left({\widehat{e}}_m\right)$.

For a given index set S_m ⊂ {1, …, r}, define ${\beta }_{m,j,S_m}≔{\beta }_{m,j}1\left\{j\in S_m\right\}$, $m=1,\dots ,2^{d_Y}-1$, j = 1, …, r. Denote the estimator restricted to ${\beta }_{m,S_m}={\left({\beta }_{m,1,S_m},\dots ,{\beta }_{m,r,S_m}\right)}^T$ by ${\widehat{\beta }}_{m,S_m}≔\text{arg}\text{mi}{\text{n}}_{\beta \text{=}{\beta }_{m,S_m}}{P}_n{\rho }_{f_m^{\beta }}$. Write ${\widehat{f}}_{m,S}≔f_m^{{\widehat{\beta }}_{m,S_m}}$. Restricted to ${\beta }_{m,S_m}$’s, the best approximation of $f_m^0$ is $f_{m,S}^0≔f_m^{{\beta }_{m,S_m}^0}$, where ${\beta }_{m,S_m}^0≔\text{arg}{\text{min}}_{\beta ={\beta }_{m,S_m}}P{\rho }_{f_m^{\beta }}$.

The following assumption requires a certain compatibility of ℓ₁-norm with the norm on ℱ, which is a regular assumption for the theoretical framework for lasso.

Assumption 3. (Compatibility condition) We say that the compatibility condition is met for the set S_m with constant ϕ_m > 0, if for all β_m satisfying ${\Vert {\beta }_{m,S_m^c}\Vert }_1⩽3{\Vert {\beta }_{m,S_m}\Vert }_1$, it holds that ${\Vert {\beta }_{m,S_m}\Vert }_1^2⩽{\Vert f_m\Vert }^2{s}_m/{\varphi }_m^2$.

Next, we show the definition of the margin condition (Bülmann and van de Geer, 2011) and demonstrate that the penalized logistic regression satisfies the condition with a quadratic margin.

Definition 1 (Margin condition). Denote a “neighborhood” of $f_m^0\in ℱ$ by ${ℱ}_{{\eta }_m}≔\left\{f\in ℱ:{\Vert f-f_m^0\Vert }_{\infty }⩽{\eta }_m\right\}$ with constant η_m > 0. We say that the margin condition holds with a strictly convex function G, if for all $f\in {ℱ}_{{\eta }_m}$, we have $ℰ(f)⩾G\left(\Vert f-f_m^0\Vert \right)$, where ∥·∥ is the norm defined on ℱ.

Assumption 4. For any fixed ${\dot{A}}^S$, there exists some constant $0⩽{\epsilon }_m^0⩽1$ such that ${\epsilon }_m^0⩽{\pi }_m\left({\dot{A}}^S\right)⩽1-{\epsilon }_m^0$, $m=1,\dots ,2^{d_Y}-1$.

Lemma 5. Under Assumption 4, the margin condition holds for all $2^{d_Y}-1$ penalized logistic regressions with a quadratic margin, i.e., G_m(u) = c_mu² for the m-th regression.

The technical proof of Lemma 5 can be found in supplement. For the m-th regression, the oracle ${\beta }_m^{*}$ (Bülmann and van de Geer, 2011) is defined by

$${\beta }_m^{*}≔\underset{\beta :S_{\beta }\in \Psi }{\text{arg}min}\left\{3ℰ\left(f_m^{\beta }\right)+\frac{8{\lambda }_m^2{s}_{\beta }}{c_m{\varphi }_m^2}\right\},$$ (15)

where S_β ≔ {j: β_j ≠ 0}, s_β ≔ |S_β| denotes the cardinality of S_β, ${\varphi }_m^2$ is a compatibility constant, and Ψ is a suitable large collection of index sets. Denote the index set of non-zero coefficients by $S_m^0≔\left\{j:{\beta }_{m,j}^0\ne 0\right\}$, and the cardinality of $S_m^0$ by $s_m≔\left|S_m^0\right|$. Assuming $f_m^0$ is linear, we can take $\Psi =\left\{S_m^0\right\}$. Hence the definition of ${\beta }_m^{*}$ is consistent with the definition of ${\beta }_m^0$ in (13), since the second term of (15) does not rely on β. In this context, we only use the notation ${\beta }_m^0$. Denote the minimum of (15) by $2{ϵ}_m^{*}≔3ℰ\left(f_m^0\right)+\frac{8{\lambda }_m^2{s}_m}{c_m{\varphi }_m^2}$. Define $Z_{M_m}≔\underset{\Vert \beta -{\beta }_m^0\Vert ⩽M_m}{\text{sup}}\left|v_n(\beta )-v_n\left({\beta }_m^0\right)\right|$, where $v_n\left({\beta }_m\right)≔\left(P_n-P\right){\rho }_{f_m^{{\beta }_m}}$ is the empirical process. Set $M_m^{*}≔{ϵ}_m^{*}/{\lambda }_m^0$, and ${𝒯}_m≔\left\{Z_{M_m^{*}}⩽{\lambda }_m^0{M}_m^{*}\right\}=\left\{Z_{M_m^{*}}⩽{ϵ}_m^{*}\right\}$. Bülmann and van de Geer (2011) showed that one can choose ${\lambda }_m^0\asymp \sqrt{\text{log}\left(n^{2{\kappa }_m}\right)/n}$ such that the set 𝒯_m holds with large probability.

Assumption 5. For some constant η_m > 0, $f_m^{{\beta }_m}\in {ℱ}_{{\eta }_m}≔\left\{\Vert f_m^{{\beta }_m}-f_m^0{\Vert }_{\infty }⩽{\eta }_m\right\}$ for all ${\Vert {\beta }_m-{\beta }_m^0\Vert }_1⩽M^{*}$, as well as $f_m^0\in {ℱ}_{{\eta }_m}$.

According to the BID equation, we estimate p by solving the optimization (9). From the optimization, H_m+1 p is an approximation of ${\widehat{e}}_m$, where H_m+1 is the (m + 1)-th row of H, since ${\widehat{e}}_m$ is the (m + 1)-th entry of E. Hence g(H_m+1 p) is the estimated m-th regression function corresponding to p. The following theorem gives the consistency of cell probability vector p in terms of excess risk of g (H_m+1 p).

We now turn to the proof of Theorem 3. Below is its statement again for convenience.

Theorem 4. Assume Assumptions 1–5 hold, where Assumption 3 holds with the set $S_m^0$. For the logistic regression with covariates corresponding to the BET screening set ${ℳ}_{m,{\delta }_{n,m}}$, suppose that λ_m satisfie ${\lambda }_m⩾8{\lambda }_m^0$. Then on the set 𝒯_m, we have,

$$ℰ\left(g\left(H_{m+1}p\right)\right)+{\lambda }_m{\Vert {\widehat{\beta }}_m-{\beta }_m^0\Vert }_1⩽6ℰ\left(f_m^0\right)+\frac{16{\lambda }_m^2{s}_m}{c_m{\varphi }_m^2}+\frac{32\lambda Ks{2}^{d_Y}}{c{\varphi }^2},$$

where K > 0 is a constant,

$$\lambda ={\text{max}}_{1⩽m⩽2^{d_Y}-1}{\lambda }_m,s={\text{max}}_{1⩽m⩽2^{d_Y}-1}{s}_m,c={\text{min}}_{1⩽m⩽2^{d_Y}-1}{c}_m,\varphi ={\text{min}}_{1⩽m⩽2^{d_Y}-1}{\varphi }_m.$$

Before the proof of Theorem 3, we first state the oracle inequality for penalized logistic regression by Bülmann and van de Geer (2011) as follows.

Lemma 6. Assume Assumptions 3–5 hold, where 3 holds with the set $S_m^0$. Suppose that λ_m satisfies the inequality ${\lambda }_m⩾8{\lambda }_m^0$. Then on the set 𝒯_m, we have

$$ℰ\left({\widehat{f}}_m\right)+{\lambda }_m{\Vert {\widehat{\beta }}_m-{\beta }_m^0\Vert }_1⩽6ℰ\left(f_m^0\right)+\frac{16{\lambda }_m^2{s}_m}{c_m{\varphi }_m^2},$$

where $c_m={\left(\frac{e^{{\eta }_m}}{{\epsilon }_m^0}+1\right)}^{-2}$.

Then we have the proof for Theorem 3 as follows.

Proof of Theorem 3:

For the excess risk of g (H_m+1 p), we have

$$\begin{gathered} ℰ\left(g\left(H_{m+1}p\right)\right)=P{\rho }_{g\left(H_{m+1}p\right)}-P{\rho }_{f_m^0}=P{\rho }_{g\left(H_{m+1}p\right)}-P{\rho }_{g\left(e_m^0\right)} \\ =\left(P{\rho }_{g\left(H_{m+1}p\right)}-P{\rho }_{g\left({\widehat{e}}_m\right)}\right)+\left(P{\rho }_{g\left({\widehat{e}}_m\right)}-P{\rho }_{g\left(e_m^0\right)}\right) \\ =\left(P{\rho }_{g\left(H_{m+1}p\right)}-P{\rho }_{g\left({\widehat{e}}_m\right)}\right)+ℰ\left({\widehat{f}}_m\right)\triangleq I+II. \end{gathered}$$

It can be shown that the function $P{\rho }_{g\left(e_m\right)}$ is Lipschitz continuous, with the Lipschitz constant K_m obtained from the first derivative

$$\left|\frac{dP{\rho }_{g\left(e_m\right)}}{d{e}_m}\right|=\left|\left(\frac{e^{f_m}}{1+e^{f_m}}-{\pi }_m\right)\left(\frac{1}{1+e_m}+\frac{1}{1-e_m}\right)\right|⩽\frac{2\left(1-{\epsilon }_m^0\right)}{1-1{\left(1-2{\epsilon }_m^0\right)}^2}\triangleq K_m.$$

For part I, denoting the true expectation vector by E⁰ and the corresponding true cell probability vector by p⁰ = H⁻¹ E⁰, we have

$$\begin{gathered} \left|I\right|⩽K_m\left|H_{m+1}p-{\widehat{e}}_m\right|⩽K_m\Vert Hp-E{\Vert }_1⩽K_m{\Vert H{p}^0-E\Vert }_1 \\ ⩽K_m\left({\Vert H{p}^0-E^0\Vert }_1+{\Vert E^0-E\Vert }_1\right). \end{gathered}$$

By Lemma 6, ${\Vert {\widehat{\beta }}_m-{\beta }_m^0\Vert }_1⩽\frac{16{\lambda }_m{s}_m}{c_m{\varphi }_m^2}$. Since $\left|{\widehat{f}}_m-f_m^0\right|⩽{\Vert {\widehat{\beta }}_m-{\beta }_m^0\Vert }_1$ with predictors taking values from {− 1, 1}, we have $\left|{\widehat{f}}_m-f_m^0\right|⩽\frac{16{\lambda }_m{s}_m}{c_m{\varphi }_m^2}$. One can similarly show that the function g⁻¹ (·) is Lipschitz continuous with Lipschitz constant two. Hence we have $\left|{\widehat{e}}_m-e_m^0\right|⩽2\left|{\widehat{f}}_m-f_m^0\right|⩽\frac{32{\lambda }_m{s}_m}{c_m{\varphi }_m^2}$, and thus ${\Vert E^0-E\Vert }_1⩽\frac{32\lambda s{2}^{d_Y}}{c{\varphi }^2}$, where $\lambda ={\text{max}}_{1⩽m⩽2^{d_Y}-1}{\lambda }_m$, $s={\text{max}}_{1⩽m⩽2^{d_Y}-1}{s}_m$, $c={\text{min}}_{1⩽m⩽2^{d_Y}-1}{c}_m$, and $\varphi ={\text{min}}_{1⩽m⩽2^{d_Y}-1}{\varphi }_m$.

For the true expectation vector and the true cell probability vector, we have ∥H p⁰ − E⁰∥₁ = 0. Denote $K=m{\text{ax}}_{1⩽m⩽2^{d_Y}-1}{K}_m$. Together with the inequality in Lemma 6 to handle the part II, the proof is completed.