Inference for Deep Neural Network Estimators in Generalized Nonparametric Models

Abstract

While deep neural networks (DNNs) are used for prediction, inference on DNN-estimated subject-specific means for categorical or exponential family outcomes remains underexplored. We address this by proposing a DNN estimator under generalized nonparametric regression models (GNRMs) and developing a rigorous inference framework. Unlike existing approaches that assume independence between estimation errors and inputs to establish the error bound, a condition often violated in GNRMs, we allow for dependence and our theoretical analysis demonstrates the feasibility of drawing inference under GNRMs. To implement inference, we consider an Ensemble Subsampling Method (ESM) that leverages U-statistics and the Hoeffding decomposition to construct reliable confidence intervals for DNN estimates. We show that, under GNRM settings, ESM enables model-free variance estimation and accounts for heterogeneity among individuals in the population. Through simulations under nonparametric logistic, Poisson, and binomial regression models, we demonstrate the effectiveness and efficiency of our method. We further apply the method to the electronic Intensive Care Unit (eICU) dataset, a large scale collection of anonymized health records from ICU patients, to predict ICU readmission risk and offer patient-centric insights for clinical decision making.

1. Introduction

Given an outcome variable $y$ and covariates $\mathbf{x}$, estimating the conditional expectation $\mathbb{E}(y\mid \mathbf{x})$ is central to supervised learning. Under squared loss, $\mathbb{E}(y\mid \mathbf{x})$ is the optimal estimator that minimizes the expected estimation error among all measurable functions, making it a natural and widely adopted target in statistical modeling and machine learning. Even when optimality does not hold under more general losses, $\mathbb{E}(y\mid \mathbf{x})$ remains a valuable target, offering insight into the relationship between covariates and outcomes. It serves as a foundational component for constructing more complex quantities across domains such as conditional variances, counterfactual means, and risk-adjusted estimates (Hastie et al. 2009). Methods that estimate $\mathbb{E}(y\mid \mathbf{x})$ include linear, parametric and nonparametric regression, kernel regression (Nadaraya 1964), support vector regression (Drucker et al. 1996), random forests (Breiman 2001) and deep neural networks (DNNs) (LeCun et al. 2015).

With the rapid advancement of artificial intelligence, DNN–based estimators have gained widespread popularity. However, the statistical inference surrounding the DNN estimator of $\mathbb{E}(y\mid \mathbf{x})$, especially when $y$ is categorical or belongs to the exponential family, remains underdeveloped. We address this gap by establishing a rigorous framework for inference on $\mathbb{E}(y\mid \mathbf{x})$ estimated by DNNs under a generalized nonparametric regression model (GNRM) setting [see Eq. (2.3)].

Modern inference approaches include distribution-free conformal estimation inference (Lei et al. 2018, Angelopoulos & Bates 2021, Huang et al. 2024), extensions of jackknife inference within conformal estimation (Alaa & Van Der Schaar 2020, Kim et al. 2020, Barber et al. 2021), debiasing methods (Athey et al. 2018, Guo et al. 2021), residual-based bootstrap techniques (Zhang & Politis 2023), and the use of $U$ statistics in regressions, random forests and neural networks (Mentch & Hooker 2016, Wager & Athey 2018, Schupbach et al. 2020, Wang & Wei 2022, Fei & Li 2024). Several distinctions and limitations remain. For example, conformal prediction constructs distribution-free prediction sets for $y$, targeting outcome coverage rather than inference on $\mathbb{E}(y\mid \mathbf{x})$. Its intervals thus absorb both outcome noise and model uncertainty, making them conservative and inefficient for mean inference. Clinically, inference on risk classifications derived from patients’ risk profiles, which are based on the estimated conditional mean (i.e., risk score), is more actionable, as medical decisions (e.g., triage or treatment intensity) rely on risk stratifications rather than prediction sets (Lundberg et al. 2018, Van Calster et al. 2019). Jackknife and related leave-one-out methods assume stable, nearly independent residuals, conditions violated by DNNs whose estimates depend heavily on individual samples. Debiasing approaches (Athey et al. 2018, Guo et al. 2021) and residual bootstraps (Zhang & Politis 2023) remain limited to linear models, while resampling-based inference for high-dimensional GLMs (Fei & Li 2021) is fully parametric. Recent work on uncertainty quantification for random forests (Mentch & Hooker 2016, Wager & Athey 2018) and neural networks (Schupbach et al. 2020, Fei & Li 2024) offers valuable insights but remains largely confined to continuous outcomes, leaving inference for GNRMs such as logistic and Poisson regressions unresolved.

Theoretically, for valid inference on $\mathbb{E}(y\mid \mathbf{x})$, we need to establish the convergence rate of neural networks in a GNRM framework. Since the universal approximation theorem (Hornik et al. 1989), there has been much progress in understanding the convergence rates of neural networks (McCaffrey & Gallant 1994, Kohler & Krzyżak 2005, Yarotsky 2017, Shen et al. 2019, Kohler & Langer 2021, Lu et al. 2021, Shen et al. 2021, Jiao et al. 2023, Fan et al. 2024, Wang et al. 2024, Bhattacharya et al. 2024, Yan et al. 2025). For instance, Shen et al. (2021) established the convergence rate under nonparametric regression assuming the $p$-th moment of response is bounded for some $p>1$; Fan et al. (2024) explored nonasymptotic error bounds of the estimators under several loss functions, such as the least-absolute deviation loss and Huber loss; Wang et al. (2024) proposed an efficient and robust nonparametric regression estimator based on an estimated maximum likelihood estimator; Fan & Gu (2024), Bhattacharya et al. (2024) investigated deep neural network estimators for nonparametric interaction models with diverging input dimensions, focusing on theoretical properties such as minimax optimality, approximation error control, and sparsity-driven regularization. However, quantifying uncertainty of DNN-estimated $\mathbb{E}(y\mid \mathbf{x})$ within the framework of GNRM introduces additional challenges. The existing approaches typically assume a nonparametric regression framework, where the distribution of $y-\mathbb{E}(y\mid \mathbf{x})$ does not depend on $\mathbf{x}$. This assumption simplifies the theoretical analysis and the derivation of convergence results (Takezawa 2005, Györfi et al. 2006, Fan 2018). In generalized nonparametric regression settings, however, the assumption often fails due to heteroskedasticity, as the distribution of $y-\mathbb{E}(y\mid \mathbf{x})$ may still depend on $\mathbf{x}$. Consequently, the standard techniques for bounding error terms and controlling variance no longer apply, and the established results on uniform convergence or concentration inequalities (Bartlett et al. 2020) are invalid. As a result, it is unclear whether these methods are applicable for drawing inference on GNRMs estimated by DNNs.

Our methodological and theoretical contributions are as follows. First, we propose a DNN estimator for subject specific means under GNRMs and develop an inference framework that explicitly accounts for covariate-dependent error distributions induced by heteroskedasticity. We provide new proof techniques, including new criterion loss function, covering-number based concentration and a truncation strategy that handles exponential-tailed, heteroskedastic noise in Section D in the Supplementary Material, improving the results of Schmidt-Hieber (2020). For example, with the general loss induced by GNRMs, the quadratic-based arguments of Schmidt-Hieber (2020) no longer apply. This necessitates a complete redevelopment of the error bounds, derived instead from the intrinsic curvature and Lipschitz properties in the new GNRM setting. As a result, all estimation error analyses are novel and move beyond Gaussian approximations. Additionally, the presence of exponential tailed, heteroskedastic noise complicates uniform convergence, which we address through a truncation strategy. The truncation scheme partitions the analysis into bounded and unbounded noise regimes, where standard convergence arguments apply in the former and concentration inequalities control the latter, yielding sharper error bounds. These theoretical tools enable us to extend the continuous outcome setting of Schmidt-Hieber (2020), accommodating both continuous and discrete outcomes, and thereby generalizing their framework to the broader GNRM class. Second, to quantify the uncertainty of the estimates, we propose an ensemble subsampling method (ESM) under GNRMs, developed from U-statistics (Frees 1989, Hoeffding 1992, Lee 2019, Boroskikh 2020) and Hoeffding decomposition (Hoeffding 1992). Our model-free variance estimator leverages the intrinsic properties of Hoeffding decomposition and accommodates varying population levels as well as individual variance structures. Moreover, we employ incomplete U-statistics to reduce computational cost, enabling the resample size to grow proportionally with the sample size while limiting the total number of resamples. To mitigate the resulting Monte Carlo bias, we refine the variance estimator of Wager & Athey (2018) for bias correction.

In Section 2, we introduce deep neural networks (DNNs), generalized nonparametric regression models (GNRMs) and the ensemble subsampling method (ESM). Section 3 presents theoretical results for the convergence rate of DNNs under the GNRM framework, as well as results for statistical inference using ESM. In Section 4, we perform simulations to evaluate the finite sample performance of our method. Section 5 demonstrates the proposed methodology using the electronic Intensive Care Unit (eICU) dataset to estimate both the probability and expected number of ICU readmissions based on patient characteristics, with accompanying statistical inference. The eICU dataset is a multi-center resource comprising anonymized clinical data from ICU patients, developed through a collaboration between the Massachusetts Institute of Technology and multiple healthcare partners. The findings may support the identification of high-risk individuals and guide the delivery of personalized, risk-informed care. Section 6 concludes the paper with discussions. Additional experiments and proofs are provided in the Supplementary Material.

Notation.

We employ lower letters (e.g. $a$) for scalars and boldface letters for vectors and matrices (e.g. $\mathbf{a}$ and $\mathbf{A}$). For any matrix $\mathbf{A}$, we use $\Vert \mathbf{A}{\Vert }_0$ and $\Vert \mathbf{A}{\Vert }_{\infty }$ to denote its zero norm and infinity norm, respectively. The set of natural and real numbers are denoted by $\mathbb{N}$ and $\mathbb{R}$ respectively. For any function $f$ mapping a vector to $\mathbb{R}$, we use $\Vert f{\Vert }_{\infty }$ to denote its infinity norm. For an integer $n,\left[n\right]=\{1,\dots ,n\}$. We write $X_1\left(n\right)=O\left(X_2\left(n\right)\right)$ or $X_1\left(n\right)\precsim X_2\left(n\right)$ or $X_2\left(n\right)\succsim X_1\left(n\right)$ if there exist $C>0$ and $N_0>0$ such that $\left|X_1\left(n\right)\right|\le C\left|X_2\left(n\right)\right|$ when $n>N_0$. We denote $X_2\left(n\right)\asymp X_1\left(n\right)$ if $X_1\left(n\right)=O\left(X_2\left(n\right)\right)$ and $X_2\left(n\right)=O\left(X_1\left(n\right)\right)$. We denote $X_1\left(n\right)=o\left(X_2\left(n\right)\right)$ if $X_1\left(n\right)/X_2\left(n\right)\to 0$, and $X_1\left(n\right)~X_2\left(n\right)$ if $\left\{X_1\left(n\right)/X_2\left(n\right)\right\}$ converges to 1. When clear from context, we add the index $p$ (for probability) such as $O_p$ for a random variable and its realized value to avoid redundancy.

2. The Preamble

2.1. Deep Neural Networks

Let $L$ be a positive integer denoting the depth of a neural network, and $\mathbf{p}={\left(p_0,\dots ,p_L,p_{L+1}\right)}^{\top }$ be a vector of positive integers specifying the width of each layer, with $p_0$ corresponding to the input dimension, $p_1,\dots ,p_L$ the dimensions of the $L$ hidden layers, and $p_{L+1}$ the dimension of the output layer. An $(L+1)$-layer DNN with layer-width $\mathbf{p}$ is defined as

$$f\left(\mathbf{x}\right)={\mathbf{W}}_L{f}_L\left(\mathbf{x}\right)+{\mathbf{v}}_L,$$ (2.1)

with the recursive expression $f_l\left(\mathbf{x}\right)=\sigma \left({\mathbf{W}}_{l-1}{f}_{l-1}\left(\mathbf{x}\right)+{\mathbf{v}}_{l-1}\right),f_1\left(\mathbf{x}\right)=\sigma \left({\mathbf{W}}_0\mathbf{x}+{\mathbf{v}}_0\right)$. Here, ${\mathbf{W}}_l\in {\mathbb{R}}^{p_{l+1}\times p_l}$ and vectors ${\mathbf{v}}_l\in {\mathbb{R}}^{p_{l+1}}(l\in [L\left]\right)$ are the parameters of this DNN, and $\sigma :\mathbb{R}\to \mathbb{R}$ is the activation function, applied element-wise to vectors. A commonly used activation function is the rectified linear unit (ReLU) function (Nair & Hinton 2010), or $\sigma \left(x\right)=\text{max}(x,0)$. We focus on ReLU due to its piecewise linearity and projection properties, with potential extensions to other activation functions as discussed in Section 6.

Overparameterized DNNs can memorize noise and overfit (Cao et al. 2022), so sparsity is used to control complexity. We focus on the following class of $s$-sparse and $F$-bounded networks for our theoretical analysis:

$$ℱ\left(L,\mathbf{p},s,F\right)=\left\{f\text{of form}(2.1):{‖{\mathbf{W}}_l‖}_{\infty },{‖{\mathbf{v}}_l‖}_{\infty }\le 1,\sum _{l=1}^L{‖{\mathbf{W}}_l‖}_0+{‖{\mathbf{v}}_l‖}_0\le s,\Vert f{\Vert }_{\infty }\le F\right\}.$$ (2.2)

Here, $s\in {\mathbb{N}}_{+}$, and $F>0$ is a constant. $\Vert \cdot {\Vert }_0$ is the number of nonzero entries of matrix or vector, and $\Vert f{\Vert }_{\infty }$ is the sup-norm of function $f$.

2.2. Generalized Nonparametric Regression

Consider outcomes from the exponential family, including commonly used distributions such as normal, Bernoulli, Poisson, and binomial. The density (or mass) function for a random variable $y$ in the (single-parameter) exponential family is $p(y\mid \theta )=h\left(y\right)\text{exp}\{\theta y-\psi (\theta \left)\right\}$, where $\theta$ is the canonical parameter, $0<h\left(y\right)<\infty$ ensures normalization, and $\psi \left(\theta \right)$ is convex and smooth, satisfying $\mathbb{E}(y\mid \theta )={\psi }^{\prime }\left(\theta \right)$, where ${\psi }^{\prime }(\cdot )$ denotes the derivative of $\psi (\cdot )$ and the inverse function of ${\psi }^{\prime }(\cdot )$ is known as the link function (Brown 1986, Fei & Li 2021).

Suppose we observe $n$ independently and identically distributed (iid) sample ${\left\{\left({\mathbf{x}}_i,y_i\right)\right\}}_{i=1}^n$ from $(\mathbf{x},y)$, where $\mathbf{x}\in {\mathbb{R}}^p$ is a covariate vector and $y\in \mathbb{R}$ is the response variable. Given $\mathbf{x}$, the distribution of $y$ is modeled as

$$p\left(y∣\mathbf{x}\right)=h\left(y\right)\text{exp}\left\{y{f}_0\left(\mathbf{x}\right)-\psi \left(f_0\left(\mathbf{x}\right)\right)\right\},$$ (2.3)

where $f_0:{\mathbb{R}}^p\to \mathbb{R}$ is a nonparametric function with $f_0\left(\mathbf{x}\right)$ representing the mean parameter of the response variable $y$ given $\mathbf{x}$. Model (2.3) is referred to as a Generalized Nonparametric Regression Model (GNRM), where the outcomes follow a distribution from the exponential family. With an independent test point $({\mathbf{x}}_{\text{new}},y_{\text{new}})$ and that $y_{\text{new}}$, given ${\mathbf{x}}_{\text{new}}$, follows (2.3), we aim to estimate $\mathbb{E}\left(y_{\text{new}}\mid {\mathbf{x}}_{\text{new}}\right)$ which is ${\psi }^{\prime }\left(f_0\left({\mathbf{x}}_{\text{new}}\right)\right)$, Consequently, it is of interest to estimate $f_0$. By the universal approximation theorem (Hornik et al. 1989), DNNs are one of suitable candidates for estimating $f_0$. We consider an optimal DNN estimator, ${\widehat{f}}_n^{\text{opt}}$, which minimizes the negative log likelihood function (2.3) over the class of DNNs defined in (2.2), that is,

$${\widehat{f}}_n^{\text{opt}}={\text{argmin}}_{f\in ℱ(L,\mathbf{p},s,F)}\frac{1}{n}\sum _{i=1}^n\left\{-y_{i}f\left({\mathbf{x}}_i\right)+\psi \left(f\left({\mathbf{x}}_i\right)\right)\right\}.$$ (2.4)

The estimator (2.4) is applicable to commonly used models, including these listed below.

Nonparametric Gaussian Regression:

Given $\mathbf{x}$, we assume that the response variable follows a Gaussian distribution with mean $f_0\left(\mathbf{x}\right)$ and variance ${\sigma }^2$. In this setting, ${\sigma }^2$ does not affect the estimation of the mean function $f_0\left(\mathbf{x}\right)$, which can be seen by examining the likelihood and optimization. The likelihood function for a nonparametric Gaussian regression model is given by $p(y\mid \mathbf{x})=\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{{\left(y-f_0\left(\mathbf{x}\right)\right)}^2}{2{\sigma }^2}}$. An application of (2.4) to this setting yields:

$$\begin{gathered} {\widehat{f}}_n^{\text{opt}}={\text{argmin}}_{f\in ℱ(L,\mathbf{p},s,F)}\frac{1}{n}\sum _{i=1}^n\left\{-y_{i}f\left({\mathbf{x}}_i\right)+\frac{1}{2}{f}^2\left({\mathbf{x}}_i\right)\right\} \\ ={\text{argmin}}_{f\in ℱ(L,\mathbf{p},s,F)}\frac{1}{n}\sum _{i=1}^n{\left\{y_i-f\left({\mathbf{x}}_i\right)\right\}}^2, \end{gathered}$$

corresponding to mean squared error minimization as in Schmidt-Hieber (2020).

Nonparametric Logistic Regression:

To model binary outcomes $y\in \{0,1\}$, we set $h\left(y\right)=1,\psi \left(\theta \right)=\text{log}\left(1+e^{\theta }\right)$ in (2.3). The estimator (2.4) in this setting can be written as:

$${\widehat{f}}_n^{\text{opt}}={\text{argmin}}_{f\in ℱ(L,\mathbf{p},s,F)}\frac{1}{n}\sum _{i=1}^n\left(-y_{i}f\left({\mathbf{x}}_i\right)+\text{log}\left[1+\text{exp}\left\{f\left({\mathbf{x}}_i\right)\right\}\right]\right).$$

Nonparametric Poisson Regression:

Considering $y\in \mathbb{N}$ (non-negative integers), we define $h(y)=\frac{1}{y!}$ and $\psi \left(\theta \right)=e^{\theta }$ in (2.3). The estimator (2.4) has the following form:

$${\widehat{f}}_n^{\text{opt}}={\text{argmin}}_{f\in ℱ(L,\mathbf{p},s,F)}\frac{1}{n}\sum _{i=1}^n\left[\text{exp}\left\{f\left({\mathbf{x}}_i\right)\right\}-y_{i}f\left({\mathbf{x}}_i\right)\right].$$

Nonparametric Binomial Regression:

Given covariates $\mathbf{x}$, the response variable $y$ follows a Binomial distribution with a fixed number of trials $n_{\text{trial}}$ and the success probability $p(\mathbf{x})=\frac{1}{1+e^{-f_0\left(\mathbf{x}\right)}}$. This corresponds to $h(y)=\left(\genfrac{}{}{0pt}{}{n_{\text{trial}}}{y}\right)$ and $\psi \left(\theta \right)=n_{\text{trial}}\text{log}\left(1+e^{\theta }\right)$. In this setting, the estimator (2.4) is specified as

$${\widehat{f}}_n^{\text{opt}}={\text{argmin}}_{f\in ℱ(L,\mathbf{p},s,F)}\frac{1}{n}\sum _{i=1}^n\left[-y_{i}f\left({\mathbf{x}}_i\right)+n_{\text{trial}}\text{log}\left\{1+e^{f\left({\mathbf{x}}_i\right)}\right\}\right].$$

2.3. Empirical Risk Minimization and Function Smoothness

Recall that the pointwise Kullback–Leibler divergence between two distributions of the form (2.3), one specified by $f$ and another by the true $f_0$, conditional on $\mathbf{x}$, i.e.,

$$\mathbb{E}\left[-yf\left(\mathbf{x}\right)+\psi \left(f\right(\mathbf{x}\left)\right)+y{f}_0\left(\mathbf{x}\right)-\psi \left(f_0\left(\mathbf{x}\right)\right)\mid \mathbf{x}\right]=-{\psi }^{\prime }\left(f_0\left(\mathbf{x}\right)\right)\left(f\left(\mathbf{x}\right)-f_0\left(\mathbf{x}\right)\right)+\psi \left(f\right(\mathbf{x}\left)\right)-\psi \left(f_0\left(\mathbf{x}\right)\right),$$

where we use $\mathbb{E}[y\mid \mathbf{x}]={\psi }^{\prime }\left(f_0\left(\mathbf{x}\right)\right)$. This equals the Bregman divergence between $f\left(\mathbf{x}\right)$ and $f_0\left(\mathbf{x}\right)$, which is nonnegative and vanishes if and only if $f\left(\mathbf{x}\right)=f_0\left(\mathbf{x}\right)$. We then quantify the estimation error of $\widehat{f}$, any estimator of $f_0$, by evaluating the pointwise estimation error at an independent test point $\mathit{X}:\ell \left(\mathit{X};\widehat{f},f_0\right)=-{\psi }^{\prime }\left(f_0\left(\mathit{X}\right)\right)\left(\widehat{f}\left(\mathit{X}\right)-f_0(\mathit{X})\right)+\psi (\widehat{f}(\mathit{X}))-\psi \left(f_0\left(\mathit{X}\right)\right)$. We further define the average estimation error, our main theoretical target, as

$$R_n\left(\widehat{f},f_0\right)≔\mathbb{E}\left[\ell \left(\mathit{X};\widehat{f},f_0\right)\right],$$ (2.5)

where the expectation is taken with respect to all involved random variables.

Computing the exact minimizer ${\widehat{f}}_n^{\text{opt}}$ in (2.4) is challenging due to the nonconvex loss (Fan & Gu 2024). Instead, we obtain an approximate minimizer ${\widehat{f}}_n$ using optimization algorithms such as gradient descent. To quantify the optimization error, we introduce

$${\Delta }_n\left({\widehat{f}}_n,{\widehat{f}}_n^{\text{opt}}\right)=\mathbb{E}\left\{\frac{1}{n}\sum _{i=1}^n(-y_i{\widehat{f}}_n\left(x_i\right)+\psi \left({\widehat{f}}_n\left(x_i\right)\right)-\frac{1}{n}\sum _{i=1}^n\text{(}-y_i{\widehat{f}}_n^{\text{opt}}\left(x_i\right)+\psi \left({\widehat{f}}_n^{\text{opt}}\left(x_i\right)\right)\right\}$$ (2.6)

that measures the difference between the expected empirical risk of ${\widehat{f}}_n$ and ${\widehat{f}}_n^{\text{opt}}$. For notational ease, we write ${\Delta }_n\left({\widehat{f}}_n\right)≔{\Delta }_n\left({\widehat{f}}_n,{\widehat{f}}_n^{\text{opt}}\right)$ when the reference to ${\widehat{f}}_n^{\text{opt}}$ is clear. We will show that a small empirical deviation ${\Delta }_n\left({\widehat{f}}_n\right)$, combined with controlled model complexity of the neural network class, implies a bound on the average excess risk $R_n\left({\widehat{f}}_n,f_0\right)$, forming the foundation for the inference theory developed in subsequent sections.

For desirable properties of ${\widehat{f}}_n$, we assume $f_0$ belongs to a Hölder class (Schmidt-Hieber 2020, Fan & Gu 2024) with parameters $\gamma ,K>0$, and domain $\mathbb{D}\subset {\mathbb{R}}^r$:

$${𝒢}_r^{\gamma }\left(\mathbb{D},K\right)=\left\{g:\mathbb{D}\to \mathbb{R}:\sum _{\mathit{\beta }:\Vert \mathit{\beta }{\Vert }_1<\gamma }{‖{\partial }^{\mathit{\beta }}g‖}_{\infty }+\sum _{\mathit{\beta }:\Vert \mathit{\beta }{\Vert }_1=\lfloor \gamma \rfloor }\underset{\mathbf{x},\mathbf{y}\in \mathbb{D},\mathbf{x}\ne \mathbf{y}}{\text{sup}}\frac{\left|{\partial }^{\mathit{\beta }}g\left(\mathbf{x}\right)-{\partial }^{\mathit{\beta }}g\left(\mathbf{y}\right)\right|}{\Vert \mathbf{x}-\mathbf{y}{\Vert }_{\infty }^{\gamma -⌊\gamma ⌋}}\le K\right\},$$

where $\lfloor \gamma \rfloor$ is the largest integer strictly smaller than $\gamma$, and ${\partial }^{\mathit{\beta }}={\partial }^{{\beta }_1}\dots {\partial }^{{\beta }_r}$ with $\mathit{\beta }={\left({\beta }_1,\dots ,{\beta }_r\right)}^{\top }\in {\mathbb{N}}^r$. We further assume that $f_0$ is a $(q+1)$-composition $(q\in \mathbb{N})$ of several Hölder functions. That is, for some vectors $\mathbf{d}=\left(d_0,\dots ,d_{q+1}\right)\in {\mathbb{N}}_{+}^{q+2},\mathbf{t}=\left(t_0,\dots ,t_q\right)\in {\mathbb{N}}_{+}^{q+1}$ and $\mathbf{m}=\left(m_0,\dots ,m_q\right)\in {\mathbb{R}}_{+}^{q+1},f_0\in 𝒢(q,\mathbf{d},\mathbf{t},\mathbf{m},K)$, where

$$𝒢\left(q,\mathbf{d},\mathbf{t},\mathbf{m},K\right)≔\left\{f=g_q○\cdots ○g_0:g_i={\left(g_{ij}\right)}_j:{\left[a_i,b_i\right]}^{d_i}\to {\left[a_{i+1},b_{i+1}\right]}^{d_{i+1}},g_{ij}\in {𝒢}_{t_i}^{m_i}\left({\left[a_i,b_i\right]}^{t_i}\right)\text{with}\left|a_i\right|,\left|b_i\right|\le K\right\}.$$

The motivation for the assumptions on $f_0$ is that the Hölder class provides a flexible framework for quantifying smoothness, capturing both integer and fractional orders, and serves as a natural benchmark for assessing neural network approximation (Yarotsky 2017), while the compositional structure mirrors the architectural design of deep neural networks with each layer applying a transformation to the output of the previous one (Yarotsky 2017). We also note that $\mathbf{d}$ and $\mathbf{t}$ characterize the dimensions of the function, while $\mathbf{m}$ represents the measure of smoothness. For instance, if

$$f_0(\mathbf{x})=g_{21}\left(g_{11}\left(g_{01}\left(x_1,x_2,x_3,x_4\right),g_{02}\left(\left(x_5,x_6,x_7,x_8\right)\right)\right),\right.g_{12}\left(g_{03}\left(x_9,x_{10},x_{11},x_{12}\right),g_{04}\left(\left(x_{13},x_{14},x_{15},x_{16}\right)\right)\right),\left.g_{13}\left(g_{05}\left(x_{17},x_{18},x_{19},x_{20}\right),g_{06}\left(\left(x_{21},x_{22},x_{23},x_{24}\right)\right)\right)\right),\mathbf{x}\in [\mathrm{0,1}{]}^{24},$$

and $g_{ij}$ are twice continuously differentiable, then smoothness $\mathbf{m}=(\mathrm{2,2},2)$, dimensions $\mathbf{d}=(\mathrm{24,6},\mathrm{3,1}{)}^{\top }$ and $\mathbf{t}=(\mathrm{4,2},3{)}^{\top }$. We further define $m_j^{*}=m_j{\Pi }_{l=i+1}^q\left(m_l\wedge 1\right)$ and ${\varphi }_n={\text{max}}_{j=0,\dots ,q}{n}^{-2m_j^{*}/\left(2m_j^{*}+t_j\right)}$.

2.4. Ensemble Subsampling Method (ESM)

Consider a scenario where $\mathbf{x}$ is drawn from an unknown probability measure ${\mathbb{P}}_{\mathbf{x}}$ supported on $𝒳$, and for an input ${\mathbf{x}}_{\text{new}}={\mathbf{x}}_{*}\in 𝒳$, the unobserved outcome $y_{\text{new}}$ follows (2.3). Our goal is to estimate $\mathbb{E}\left(y_{\text{new}}\mid {\mathbf{x}}_{\text{new}}={\mathbf{x}}_{*}\right)={\psi }^{\prime }\left(f_0\left({\mathbf{x}}_{*}\right)\right)$ and its uncertainty. For this, we introduce the Ensemble Subsampling Method (ESM), based on subsampling techniques (Wager & Athey 2018, Fei & Li 2024), to construct ensemble estimators and confidence intervals. ESM consists of the following components, which are summarized in Figure 1 as well.

Figure 1:

Overview of the Ensemble Subsampling Method (ESM).

• (Subsampling) Let $ℐ=\{1,\dots ,n\}$ denote the index set of the training dataset ${𝒟}_n$. We construct subsets of size $r(<n)$, yielding $B^{*}=\left(\genfrac{}{}{0pt}{}{n}{r}\right)$ unique combinations. Denoting the $b$-th subset as ${ℐ}^b=\left\{i_1,\dots ,i_r\right\}$, where $i_1<\cdots <i_r$ and $b\in \left[1:B^{*}\right]$.

• (Estimator Construction) For each $1\le b\le B^{*}$, we aim to minimize the objective function in (2.4) within $ℱ(L,\mathbf{p},s,F)$ by using a gradient based optimization algorithm such as Stochastic Gradient Descent (SGD) on the observations indexed by ${ℐ}^b$. Let ${\widehat{f}}^b$ denote the resulting approximate minimizer satisfying

  $${\Delta }_r\left({\widehat{f}}^b,{\widehat{f}}^{b,\text{opt}}\right)\le {\Delta }_b^{\text{opt}},$$ (2.7)

  where ${\Delta }_b^{\text{opt}}$ quantifies the optimization error (Fan & Gu 2024), and ${\widehat{f}}^{b,\text{opt}}$ and ${\Delta }_r\left({\widehat{f}}^b,{\widehat{f}}^{b,\text{opt}}\right)$ are defined respectively in (2.4) and (2.6) on the subsample indexed by ${ℐ}^b$.
• (Ensemble Estimation) Using the estimators $\left\{{\widehat{f}}^b:b=1,\dots ,B^{*}\right\}$, we compute the ensemble estimation for any ${\mathbf{x}}_{*}\in 𝒳$ as:

  $${\widehat{f}}^{B^{*}}\left({\mathbf{x}}_{*}\right)=\frac{1}{B^{*}}\sum _{b=1}^{B^{*}}{\widehat{f}}^b\left({\mathbf{x}}_{*}\right),$$

  and estimate $\mathbb{E}\left(y_{\text{new}}\mid {\mathbf{x}}_{\text{new}}={\mathbf{x}}_{*}\right)$ as $\left.{\psi }^{\prime }\left({\widehat{f}}^{B^{*}}\left({\mathbf{x}}_{*}\right)\right)\right)$.
• (Confidence Interval Construction) To quantify uncertainty, we estimate the variance, denoted ${\widehat{\sigma }}_{*}$, using the infinitesimal jackknife method. The resulting confidence interval (CI) is defined as:

  $$\text{CI}\left({\mathbf{x}}_{*}\right)=\left[{\psi }^{\prime }\left({\widehat{f}}^{B^{*}}\left({\mathbf{x}}_{*}\right)-c_{\alpha }{\widehat{\sigma }}_{*}\right),{\psi }^{\prime }\left({\widehat{f}}^{B^{*}}\left({\mathbf{x}}_{*}\right)+c_{\alpha }{\widehat{\sigma }}_{*}\right)\right].$$ (2.8)

  where $c_{\alpha }$ is a constant controlling the $1-\alpha$ confidence level.

The distribution of ${\widehat{f}}^b\left({\mathbf{x}}_{*}\right)$ for a fixed ${\mathbf{x}}_{*}$ is intractable. However, since ${\widehat{f}}^b\left({\mathbf{x}}_{*}\right)$ is permutation symmetric with respect to ${ℐ}^b$, the ensemble estimator ${\widehat{f}}^{B^{*}}\left({\mathbf{x}}_{*}\right)$ forms a generalized U-statistic, which we later show to be asymptotically normal by leveraging U-statistics theory. In practice, we allow $r$ to grow with $n$, but evaluating all $B^{*}=\left(\genfrac{}{}{0pt}{}{n}{r}\right)$ models becomes computationally infeasible. To overcome this, we use a stochastic approximation by randomly drawing $B$ size-$r$ subsamples from ${𝒟}_n$. This involves independently sampling indices $b_1,\dots ,b_B$ from 1 to $\left(\genfrac{}{}{0pt}{}{n}{r}\right)$, where each subsample is indexed by ${ℐ}^{b_j}$, and computing

$${\widehat{f}}^B\left({\mathbf{x}}_{*}\right)=\frac{1}{B}\sum _{j=1}^B{\widehat{f}}^{b_j}\left({\mathbf{x}}_{*}\right).$$ (2.9)

As (2.9) provides the estimator for $f_0$, the variance ${\widehat{\sigma }}_{*}$ in (2.8) in the Confidence Interval Construction procedure is given as

$${\widehat{\sigma }}_{*}^2=\underset{\text{original}{\widehat{\sigma }}^2}{\underbrace{\frac{n(n-1)}{(n-r{)}^2}\sum _{i=1}^n{\widehat{V}}_i^2}}-\underset{\text{bias correction}}{\underbrace{\frac{n(n-1)}{(n-r{)}^2}\cdot \frac{1}{B(B-1)}\sum _{i=1}^n\sum _{j=1}^B{\left(Z_{b_{j}i}-{\widehat{V}}_i\right)}^2}}.$$ (2.10)

Here, $Z_{b_{j}i}$ and ${\widehat{V}}_i$ are defined as

$$Z_{b_{j}i}=\left(J_{b_{j}i}-J_{\cdot i}\right)\left({\widehat{f}}^{b_j}\left({\mathbf{x}}_{*}\right)-{\widehat{f}}^B\left({\mathbf{x}}_{*}\right)\right),{\widehat{V}}_i=\frac{\sum _{j=1}^B{Z}_{b_{j}i}}{B},$$

where $J_{b_{j}i}=I\left(i\in {ℐ}^{b_j}\right)$ and $J_{.i}=\frac{1}{B}\sum _{j=1}^B{J}_{b_{j}i}$. Extending Peng et al. (2024), (2.10) jointly corrects Monte Carlo bias and finite-sample effects using the factor $n(n-1)/(n-r{)}^2$, enabling valid inference with $B\succsim n$. Its construction relies on the dominance of the first order term in the Hoeffding decomposition, and the bias correction term adequately captures its resampling variability, regardless of the specific scaling between $r$ and $n$.

3. Theoretical Results

Let ${\mathbf{z}}_i=\left(y_i,{\mathbf{x}}_i\right)$ represent an independent observation of sample points. For studying the asymptotic properties of ${\widehat{f}}^B$ in (2.9), we introduce

$${\xi }_{1,r}(\mathbf{x})=\text{Cov}\left(\widehat{f}\left(\mathbf{x};{\mathbf{z}}_1,{\mathbf{z}}_2,\dots ,{\mathbf{z}}_r\right),\widehat{f}\left(\mathbf{x};{\mathbf{z}}_1,{\mathbf{z}}_2^{\prime },\dots ,{\mathbf{z}}_r^{\prime }\right)\right).$$ (3.1)

Here $\mathbf{x}\in 𝒳$ is in the support of probability measure ${\mathbb{P}}_{\mathbf{x}},\widehat{f}\left(\mathbf{x};{\mathbf{z}}_1,{\mathbf{z}}_2,\dots ,{\mathbf{z}}_r\right)$ means that we obtain $\widehat{f}$ from the subsample ${\mathbf{z}}_1,{\mathbf{z}}_2,\dots ,{\mathbf{z}}_r$ and then apply it to the point $\mathbf{x}$, and ${\xi }_{1,r}$ quantifies the covariance between estimates based on two overlapping subsets of size $r$ that share a common point ${\mathbf{z}}_1$ but differ in the remaining elements. ${\xi }_{1,r}$ represents a second-order component in the Hoeffding decomposition of a U-statistic (Hoeffding 1992), capturing the pairwise dependence structure within the subsamples used for constructing the estimator. The terms ${\mathbf{z}}_i^{\prime }$ and ${\mathbf{z}}_i$ are independently generated from the same data generation process. We impose some regularity assumptions for our theoretical guarantees.

Assumption 3.1. Suppose that (2.3) holds and the true function $f_0\in 𝒢(q,\mathbf{d},\mathbf{t},\mathbf{m},K)$. The support of $y$, denoted by $\text{supp}\left(h\right(y\left)\right)$, is fixed and does not depend on $\mathbf{x}$. Moreover, the sufficient statistic $T\left(y\right)=y$ is not degenerate and can take at least two distinct values in $\text{supp}\left(h\right)$. For any $\mathbf{x}$, the conditional distribution $p(y\mid \mathbf{x})$ is sub-exponential. Specifically, with bounded $f_0$ there exists a universal constant $\kappa >0$ such that for all $t>0$,

$$\mathbb{P}\left(\left|y-\mathbb{E}\left(y∣\mathbf{x}\right)\right|\ge t∣\mathbf{x}\right)\le 2\text{exp}\left(-\frac{t}{\kappa }\right).$$

Assumption 3.2. The network class $ℱ(L,\mathbf{p},s,F)$ with $L,\mathbf{p}$ and $s$ satisfies i) $F\ge \text{max}(K,1)$, ii) $\sum _{i=0}^q{\text{log}}_2\left(4t_i\vee 4m_i\right){\text{log}}_{2}n\le L\precsim {\text{log}}^{\alpha }n$ for some $\alpha >1$, iii) $n{\varphi }_n\precsim {\text{min}}_{i=1,\dots ,L}{p}_i\le {\text{max}}_{i=1,\dots ,L}{p}_i\precsim n$, iv) $s\precsim n{\varphi }_n\text{log}n$. Here, ${\varphi }_n={\text{max}}_{j=0,\dots ,q}{n}^{-2m_j^{*}/\left(2m_j^{*}+t_j\right)}$, where $m_j^{*}=m_j{\Pi }_{l=i+1}^q\left(m_l\wedge 1\right)$. The values $t_i,m_i$ and $p_i$ are the $(i+1)$-th element in the vectors $\mathbf{t},\mathbf{m}$ and $\mathbf{p}$, respectively.

Assumption 3.3. For each subsample with index ${ℐ}^b$ and size $r$, there exists a constant $C>0$ such that the optimization error ${\Delta }_b^{\text{opt}}$ defined in (2.7) satisfies ${\Delta }_b^{\text{opt}}\le C{\varphi }_{r}L{\text{log}}^{2}r$.

Assumption 3.4. The covariance ${\xi }_{1,r}\left(\mathbf{x}\right)$ defined in (3.1) satisfies ${\text{inf}}_{\mathbf{x}\in 𝒳}{\xi }_{1,r}\left(\mathbf{x}\right)\ge C{r}^{-(1+\epsilon )}$ for some $\epsilon >0$. The subsample size $r=n^{\gamma }$ satisfies $1/\left(1+{\text{min}}_{j=0,\dots ,q}\left(2m_j^{*}\right)/\left(2m_j^{*}+t_j\right)\right)<\gamma <1$, and the number of subsampling times $B$ is large such that $B\succsim n$.

Assumption 3.1 ensures that $p(y\mid \mathbf{x})$ not degenerate, while the concentration of $y$ does not deviate too widely. Additionally, that $T\left(y\right)=y$ takes at least two distinct values in $\text{supp}\left(h\right)$ guarantees a non-zero variance of $y$, ensuring the log-partition function $\psi \left(f_0\left(\mathbf{x}\right)\right)$ exhibits local convexity. The boundedness of $f_0$ guarantees the existence of $\kappa$. This condition, weaker than the sub-Gaussian tail bound assumed in Fan et al. (2024), is satisfied by most common exponential family distributions, including Beta, Bernoulli, Poisson, and Gaussian. Assumption 3.2 is a technical assumption similar to Schmidt-Hieber (2020), which specifies the requirements on the network parameters relative to the sample size $n$ and smoothness parameters ${\varphi }_n,t_i$ and $m_i$, including the number of layers $L$ and the sparsity $s$. Assumption 3.3, related to the training dynamics of DNNs, stipulates that DNNs return an estimator sufficiently close to the global minimum. We note that the rate in Assumption 3.3 pertains to the prediction error in Theorem 3.5. Our intent is not to assert an explicit convergence rate for SGD in practice, but to impose a sufficient condition that separates optimization error from the inferential analysis. This perspective follows standard practice in the statistical literature (Chernozhukov et al. 2018, Schmidt-Hieber 2020, Fan & Gu 2024), where such assumptions serve as technical devices to ensure valid inference rather than as primary objects of study. Assumption 3.4 on ${\xi }_{1,r}\left(\mathbf{x}\right)$ is not meant to serve as a precise characterization of neural networks. Rather, it ensures that the first-order Hoeffding projection remains non-degenerate and dominates the higher-order components, as in the random forest setting (Wager & Athey 2018). We choose the subsample size $r$ much smaller than $n$ to ensure first-order dominance but large enough to limit bias, and let the number of subsamples $B$ scale with $n$ to control computation.

Theorem 3.5. Suppose that $f_0\in 𝒢(q,\mathbf{d},\mathbf{t},\mathbf{m},K)$ and ${\widehat{f}}_n\in ℱ(L,\mathbf{p},s,F)$. If Assumptions 3.1–3.2 hold, then it holds that

$$\frac{1}{2}{\Delta }_n\left({\widehat{f}}_n\right)-c^{\prime }{\varphi }_{n}L{\text{log}}^{2}n\le R_n\left({\widehat{f}}_n,f_0\right)\le 2{\Delta }_n\left({\widehat{f}}_n\right)+C^{\prime }{\varphi }_{n}L{\text{log}}^{2}n.$$

for some constants $c^{\prime },C^{\prime }>0$. Here, ${\Delta }_n\left({\widehat{f}}_n\right)$ is defined in (2.6).

Theorem 3.5 establishes a bound on the out-of-sample estimation error. Unlike previous studies (Schmidt-Hieber 2020, Fan & Gu 2024), which primarily focus on bounding the in-sample mean squared error, i.e., $\frac{1}{n}\sum _{i=1}^n{\left(y_i-f\left({\mathbf{x}}_i\right)\right)}^2$, to derive bounds for the estimation error, our approach applies to a wide class of training losses (2.6) under GNRMs. The heteroskedasticity inherent in GNRMs necessitates a more refined analysis, and a truncation technique is introduced to address this challenge by separating the noise into different regimes. Specifically, this result enables the analysis of non-Gaussian outcomes with heteroskedasticity depending on covariates, overcoming key limitations of prior methods (Schmidt-Hieber 2020, Fan & Gu 2024) that rely on the independence of $y_i-\mathbb{E}\left(y_i\mid {\mathbf{x}}_i\right)$ and $f_0\left({\mathbf{x}}_i\right)$ to get estimation error bounds.

Proposition 3.6. We suppose that the conditions of Theorem 3.5 hold, the neural network fitted on subsamples belongs to $ℱ(L,\mathbf{p},s,F)$, Assumptions 3.2–3.4 hold for each subsample with size $r$, and sparsity $s\asymp r^{1+\delta }{\varphi }_r\text{log}r$ for some constant $\delta >0$. Then for a fixed ${\mathbf{x}}_{*}$, if the pointwise bias is dominated by the best approximation bias, i.e., there exists constant $C\left({\mathbf{x}}_{*}\right)$ only depends on ${\mathbf{x}}_{*}$ such that $\left|\mathbb{E}{\widehat{f}}^b\left({\mathbf{x}}_{*}\right)-f_0\left({\mathbf{x}}_{*}\right)\right|\precsim C\left({\mathbf{x}}_{*}\right){\text{inf}}_{f\in ℱ}\left|f\left({\mathbf{x}}_{*}\right)-f_0\left({\mathbf{x}}_{*}\right)\right|$, it holds that $\sqrt{\frac{n}{r^2{\text{inf}}_{\mathbf{x}\in 𝒳}{\xi }_{1,r}\left(\mathbf{x}\right)}}\cdot \left|\mathbb{E}{\widehat{f}}^B\left({\mathbf{x}}_{*}\right)-f_0\left({\mathbf{x}}_{*}\right)\right|\to 0$.

Proposition 3.6 implies asymptotic unbiasedness, i.e., $\mathbb{E}\left\{{\widehat{f}}^B\left({\mathbf{x}}_{*}\right)\right\}-f_0\left({\mathbf{x}}_{*}\right)=o\left(\sqrt{\frac{r^2{\xi }_{1,r}\left({\mathbf{x}}_{*}\right)}{n}}\right)$ for a fixed ${\mathbf{x}}_{*}$. Thus, the bias of ${\widehat{f}}^B$ is asymptotically negligible, and enables us to focus on ${\widehat{f}}^B\left({\mathbf{x}}_{*}\right)-\mathbb{E}\left\{{\widehat{f}}^B\left({\mathbf{x}}_{*}\right)\right\}$ and establish the following results.

Theorem 3.7. Suppose the data ${𝒟}_n$ are generated from Model (2.3) with an unknown $f_0\in 𝒢(q,\mathbf{d},\mathbf{t},\mathbf{m},K)$, the used neural networks belong to $ℱ(L,\mathbf{p},s,F)$, and Assumptions 3.1–3.4 hold (Assumption 3.2 holds for each subsample with size $r$). Then with the results of Proposition 3.6, there exists a positive sequence, ${\delta }_1,{\delta }_2,\dots$, with ${\delta }_n\to 0$ and a corresponding set ${𝒜}_{{\delta }_n}\subseteq 𝒳$ with ${\mathbb{P}}_{\mathbf{x}}\left({𝒜}_{{\delta }_n}\right)\ge 1-{\delta }_n$, such that for any fixed ${\mathbf{x}}_{*}\in {𝒜}_{{\delta }_n}$, it holds that

$$\sqrt{n}\cdot \frac{{\widehat{f}}^B\left({\mathbf{x}}_{*}\right)-f_0\left({\mathbf{x}}_{*}\right)}{\sqrt{r^2{\xi }_{1,r}\left({\mathbf{x}}_{*}\right)}}\stackrel{d}{\to }𝒩\left(\mathrm{0,1}\right).$$

Theorem 3.7 establishes the asymptotic normality of the ensembled estimator ${\widehat{f}}^B$ for any fixed ${\mathbf{x}}_{*}\in {𝒜}_{{\delta }_n}$, i.e., the inference result holds with high probability for almost all $\mathbf{x}\in 𝒳$. Here, Theorem 3.7 holds by the fact that the first-order projection of ${\widehat{f}}^B\left({\mathbf{x}}_{*}\right)-\mathbb{E}\left\{{\widehat{f}}^B\left({\mathbf{x}}_{*}\right)\right\}$ dominates the other terms. Moreover, as the analytic form of ${\xi }_{1,r}\left({\mathbf{x}}_{*}\right)$ is unavailable, the following theorem provides a consistent estimator of it.

Theorem 3.8. Under the same setting as in Theorem 3.7 and with ${\widehat{\sigma }}_{*}^2$ defined in (2.10), it holds that for any fixed ${\mathbf{x}}_{*}\in 𝒳$,

$$\sqrt{\frac{r^2{\xi }_{1,r}\left({\mathbf{x}}_{*}\right)}{n{\widehat{\sigma }}_{*}^2}}\stackrel{p}{\to }1.$$

Compared to Wager & Athey (2018) and Fei & Li (2024), since we allow the number of subsamples $B$ to grow at the same rate as the sample size $n$, the Monte Carlo bias in the variance estimation becomes non-negligible. As pointed out by Wager et al. (2014), this bias is typically of order $n/B$, and does not vanish when $B\asymp n$. Different from the correction proposed in Wager et al. (2014), which assumes near asymptotic independence between the inclusion indicators $J_{bi}$ and the estimates ${\widehat{f}}^b$, we propose a bias-correction term that directly accounts for the dependency between subsample structure and estimates. Specifically, we utilize the covariance form of the variance estimator and subtract a U-statistic-based empirical correction that estimates the within-subject variability due to repeated subsampling. This yields a debiased estimator that is asymptotically unbiased even when $B\asymp n$. For the confidence interval in (2.8), Theorem 3.8 and the Slutsky theorem ensure that, as $n\to \infty$, it holds that

$$\mathbb{P}\left(\mathbb{E}\left(y_{\text{new}}\mid {\mathbf{x}}_{*}\right)\in \left[\widehat{L}\left({\mathbf{x}}_{*}\right),\widehat{U}\left({\mathbf{x}}_{*}\right)\right]\right)\to 1-\alpha ,$$

where $\widehat{L}\left({\mathbf{x}}_{*}\right)={\psi }^{\prime }\left({\widehat{f}}^B\left({\mathbf{x}}_{*}\right)-z_{1-\alpha /2}{\widehat{\sigma }}_{*}\right),\widehat{U}\left({\mathbf{x}}_{*}\right)={\psi }^{\prime }\left({\widehat{f}}^B\left({\mathbf{x}}_{*}\right)+z_{1-\alpha /2}{\widehat{\sigma }}_{*}\right)$, and $z_{1-\alpha /2}$ is the $(1-\alpha /2)$-th quantile of the standard normal distribution. In practice, we recommend using this back-transformed interval, as it preserves the parameter’s natural range.

Remark.

Our inference framework relies on a well-trained network; when training is insufficient or overly regularized, inference may become unreliable. For analytical tractability, we adopt an idealized $s$-sparse, $F$-bounded network class (Schmidt-Hieber 2020), while in practice, regularization techniques such as weight decay and dropout promote approximate sparsity, keeping the trained model close to this class. The optimization gap ${\Delta }_n\left({\widehat{f}}_n\right)$ in Theorem 3.5 quantifies the discrepancy between empirical and population risks, filters out degenerate training cases, and accounts for residual optimization and approximation errors. Our theory demonstrates that as long as ${\Delta }_n\left({\widehat{f}}_n\right)$ remains small, inference validity is preserved, consistent with Chernozhukov et al. (2018), Fan & Gu (2024).

4. Numerical Experiments

We conduct simulations under logistic and Poisson generalized nonparametric regression models to evaluate the performance of the ensemble subsampling method (ESM) in point estimation, variance estimation $\left({\widehat{\sigma }}_{*}\right)$, coverage, and interval length. For comparison, we include several alternative methods capable of producing confidence intervals, including HulC (Kuchibhotla et al. 2024), Bayesian Neural Networks (Sun et al. 2021), Ensemble Methods (Lakshminarayanan et al. 2017), and the naive bootstrap. Additional experiments are presented in the Supplementary Material, namely, Section A.1 (binomial regression), Section A.2 (comparison with random forests (Breiman 2001)), Section A.3 (kernel regression), Section A.4 (varying covariate dimensions and nonlinearities), and Sections A.5–A.9 (sensitivity analyses for optimization parameters, subsample size $r$, number of subsamples $B$, network depth $L$, and dropout rate).

We present our basic settings. Define $g\left(\mathbf{x}\right)=x_1+0.25x_2^2+0.1\text{arctan}\left(0.5x_3-0.3\right)$. Here, $x_j$ is the $j$-th element in the vector $\mathbf{x}$. We generate ${\mathbf{x}}_i$ from $𝒩\left(\mathbf{0},{\mathbf{I}}_p\right)$ with $p=10$, i.e., the first 3 covariates are signals and the rest are noise variables. For each $i=1,\dots ,n$ and given ${\mathbf{x}}_i$, we independently simulate $y_i$ under the following models.

(logistic model) $\mathbb{P}\left(y_i=1\mid {\mathbf{x}}_i\right)=\frac{1}{1+\text{exp}\left\{-g\left({\mathbf{x}}_i\right)\right\}}$, with $f_0\left(\mathbf{x}\right)=g\left(\mathbf{x}\right)$ and ${\psi }^{\prime }\left(x\right)=(1+\text{exp}(-x){)}^{-1}$. (Poisson model) With $k\in \{\mathrm{0,1},2,\dots \},\mathbb{P}\left(y_i=k\mid {\mathbf{x}}_i\right)=\frac{e^{-\lambda \left({\mathbf{x}}_i\right){\left(\lambda \left({\mathbf{x}}_i\right)\right)}^k}}{k!}$, where we set $\lambda \left({\mathbf{x}}_i\right)=\text{log}\left(1+e^{g\left({\mathbf{x}}_i\right)}\right)$ to ensure that $\lambda \left({\mathbf{x}}_i\right)>0$. In this model, we have $f_0\left({\mathbf{x}}_i\right)=\text{log}\left\{\lambda \left({\mathbf{x}}_i\right)\right\}$ and ${\psi }^{\prime }\left(x\right)=\text{exp}\left(x\right)$. For evaluation, we generate a total of 80 independent test points ${\mathbf{x}}_{\text{test},i}~𝒩\left(0,{\mathbf{I}}_{10}\right),i=1,\dots ,80$, which remain fixed throughout the following experiment. We then apply (2.9) and (2.10) to obtain the point estimate ${\widehat{f}}^B\left({\mathbf{x}}_{\text{test},i}\right)$ and its associated variance estimate ${\widehat{\sigma }}_{*}\left({\mathbf{x}}_{\text{test},i}\right)$. To assess the accuracy of the variance estimates and the empirical coverage of the confidence intervals (2.8), we repeat the entire procedure 300 times, each time independently regenerating the training data. This yields replicated estimates ${\widehat{f}}_s^B\left({\mathbf{x}}_{\text{test},i}\right)$ and ${\widehat{\sigma }}_{*,s}\left({\mathbf{x}}_{\text{test},i}\right)$ for $s=1,\dots ,300$. Final evaluation is based on the average performance across the fixed test set, using the metrics summarized in Table 1.

Table 1:

Simulation results for different regressions under varying sample sizes $n$ and subsampling ratios $r$ using the following metrics: absolute ${\text{Bias}}_f$ and ${\text{MAE}}_f$ are, respectively, the average and mean absolute bias between ${\widehat{f}}_s^B$ and $f_0;{\text{Bias}}_{{\psi }^{\prime }}$ and ${\text{MAE}}_{{\psi }^{\prime }}$ are, respectively, the average and mean absolute bias in the transformed form ${\psi }^{\prime }\left({\widehat{f}}_s^B\right)$ and ${\psi }^{\prime }\left(f_0\right)$; EmpSD denotes the empirical standard deviation of ${\widehat{f}}_s^B\left({\mathbf{x}}_{\text{test},i}\right)$ computed across the 300 repetitions and then averaged over the 80 test points; SE and ${\text{SE}}_c$ denote the average standard errors without and with bias correction, respectively, averaged over the 80 test points; CP is the average, across test points, of the coverage probability that the 95% confidence interval (2.9) contains the truth over 300 repetitions, and AIL is the mean length of these intervals. Numbers in brackets indicate the standard deviation of each metric across test points.

	${\text{Bias}}_f$	${\text{MAE}}_f$	${\text{Bias}}_{{\psi }^{\prime }}$	${\text{MAE}}_{{\psi }^{\prime }}$	EmpSD	SE	${\mathbf{S}\mathbf{E}}_{\mathit{c}}$	CP	AIL
Logistic Model
$n=400,r=n^{0.8}$	0.11(0.72)	0.57(0.44)	0.02(0.14)	0.11(0.08)	0.65	0.78	0.66	91.7%	0.46(0.13)
$r=n^{0.85}$	0.06(0.59)	0.47(0.36)	0.01(0.12)	0.10(0.07)	0.55	0.68	0.58	94.0%	0.43(0.11)
$r=n^{0.9}$	0.07(0.50)	0.40(0.31)	0.01(0.10)	0.08(0.06)	0.48	0.59	0.50	94.6%	0.39(0.09)
$n=700,r=n^{0.8}$	0.08(0.41)	0.33(0.26)	0.02(0.09)	0.07(0.05)	0.39	0.51	0.39	93.6%	0.31(0.08)
$r=n^{0.85}$	0.05(0.36)	0.28(0.22)	0.01(0.08)	0.06(0.05)	0.34	0.44	0.34	93.9%	0.28(0.06)
$r=n^{0.9}$	0.06(0.33)	0.26(0.21)	0.01(0.07)	0.06(0.05)	0.31	0.40	0.31	92.9%	0.26(0.06)
Poisson Model
$n=400,r=n^{0.8}$	−0.32(0.38)	0.38(0.31)	−0.16(0.23)	0.22(0.18)	0.35	0.43	0.36	88.6%	0.86(0.45)
$r=n^{0.85}$	−0.23(0.36)	0.33(0.27)	−0.12(0.25)	0.21(0.18)	0.34	0.43	0.36	91.9%	0.95(0.52)
$r=n^{0.9}$	−0.14(0.34)	0.29(0.24)	−0.06(0.25)	0.19(0.18)	0.33	0.42	0.35	94.3%	1.02(0.59)
$n=700,r=n^{0.8}$	−0.17(0.26)	0.25(0.20)	−0.10(0.19)	0.16(0.14)	0.25	0.34	0.26	90.7%	0.67(0.34)
$r=n^{0.85}$	−0.10(0.26)	0.21(0.17)	−0.06(0.19)	0.15(0.14)	0.24	0.33	0.25	93.2%	0.71(0.38)
$r=n^{0.9}$	−0.04(0.24)	0.19(0.15)	−0.02(0.19)	0.14(0.13)	0.23	0.32	0.24	94.5%	0.72(0.38)

In the experiment, we randomly select sub-datasets of size $r$ to train neural networks, varying $r$ from $n^{0.8}$ to $n^{0.9}$, with $n=400\text{and}700$. These choices of $r$ are based on our experiments as they provide enough data for neural network convergence while remaining asymptotically smaller than the total sample size $n$. To ensure computational feasibility, we set $B=1400$ and use a three-layer DNN with architecture $\mathbf{p}=(p,\mathrm{128,64,1})$, a learning rate of 0.1, and ReLU activation. We apply early stopping (500 epochs), a 10% dropout rate, ${\ell }_2$ weight decay with rate 0.02, and output truncation $(F=3)$ to effectively control the sparsity $s$ and boundedness $F$ in the function class defined in (2.2), aligning the implemented network with theoretical assumptions.

Table 1 shows that increasing the sample size $n$ reduces biases with decreased mean absolute errors (MAE) for both $f$ and its transformed counterpart ${\psi }^{\prime }$. Figure 2a displays the average estimated $\mathbb{E}(y\mid \mathbf{x})$ and its variation across 300 repetitions for various test points, in the Bernoulli case with $n=700$ and $r=n^{0.9}$. The estimated means closely align with the true values. To further assess the quality of uncertainty quantification, we also examine the empirical standard deviation across repetitions. The corrected standard errors ${\text{SE}}_c$ align much more closely with the empirical standard deviations (EmpSD) than the uncorrected standard errors (SE), confirming the effectiveness of the correction term introduced in (2.10). This is further illustrated in Figure 2b, where the variance estimator with bias correction closely matches the empirical standard deviation, while the uncorrected version overestimates the uncertainty. As a result, our method achieves coverage probabilities (CP) close to the nominal level.

Figure 2:

Estimation and inference in simulation samples: Logistic Model with $n=700,r=n^{0.9}$ and $B=1400$. Figure 2a shows the average estimated $\mathbb{E}(y\mid \mathbf{x})$ with variability across 300 repetitions (gray band) over test points. Figure 2b compares corrected and uncorrected standard errors and their variability (gray and blue bands, respectively) to the empirical standard deviations of estimates across all test samples.

We present the comparison results in Table 2. For ESM, we select our results with $\gamma =0.9$. For HulC method proposed by Kuchibhotla et al. (2024), although the coverage probability is close to the nominal level of 95%, its average interval length is very wide. For Bayesian neural networks (Sun et al. 2021), we observe that the performance is highly sensitive to the tuning of hyperparameters. Due to the difficulty of selecting appropriate priors and optimization settings, the coverage probabilities and average interval lengths in our experiments are worse than those of the proposed ESM, with noticeably wider intervals and less stable coverage. For the ensemble methods (Lakshminarayanan et al. 2017) and the naive bootstrap approach, the intervals are wider than those of ESM. As the sample size $n$ increases, and the coverage probability tends to exceed the nominal 95%. Overall, ESM achieves competitive coverage performance with narrowest intervals, although it may not always provide coverage closest to the nominal level among the compared methods. So, in some cases, using estimators without bias correction or using bootstrap procedures may also have merit, as the presence of bias-correction terms in finite samples can lead to slight undercoverage.

Table 2:

Results of comparison methods.

	${\text{Bias}}_f$	${\text{MAE}}_f$	${\text{Bias}}_{{\psi }^{\prime }}$	${\text{MAE}}_{{\psi }^{\prime }}$	EmpSD	SE	${\mathbf{S}\mathbf{E}}_{\mathit{c}}$	CP	AIL
Logistic Model, $n=400$
HulC	-	-	-	-	-	-	-	94.8%	0.71(0.21)
BNN	0.06(0.61)	0.50(0.35)	0.02(0.14)	0.12(0.08)	0.27	-	-	97.8%	0.68(0.13)
Ensemble	0.06(0.84)	0.67(0.52)	0.01(0.16)	0.13(0.09)	0.80	0.90	-	95.5%	0.59(0.15)
Naive bp	0.05(0.81)	0.64(0.50)	0.01(0.16)	0.13(0.09)	0.77	0.85	-	95.4%	0.57(0.15)
ESM	0.07(0.50)	0.40(0.31)	0.01(0.10)	0.08(0.06)	0.48	0.59	0.50	94.6%	0.39(0.09)
Logistic Model, $n=700$
HulC	-	-	-	-		-	-	94.9%	0.67(0.21)
BNN	0.01(0.64)	0.55(0.34)	0.00(0.15)	0.13(0.07)	0.11	-	-	65.7%	0.31(0.10)
Ensemble	0.04(0.48)	0.38(0.30)	0.01(0.10)	0.08(0.06)	0.47	0.53	-	96.8%	0.41(0.10)
Naive bp	0.03(0.48)	0.37(0.29)	0.01(0.10)	0.08(0.06)	0.46	0.52	-	96.7%	0.40(0.09)
ESM	0.06(0.33)	0.26(0.21)	0.01(0.07)	0.06(0.05)	0.31	0.40	0.31	92.9%	0.26(0.06)
Poisson Model, $n=400$
HulC	-	-	-	-	-	-	-	90.0%	1.40(0.89)
BNN	0.15(0.49)	0.41(0.30)	0.03(0.36)	0.30(0.21)	0.20	-	-	86.3%	1.55(1.18)
Ensemble	−0.22(0.44)	0.37(0.31)	−0.09(0.30)	0.24(0.21)	0.42	0.50	-	96.0%	1.33(0.71)
Naive bp	−0.21(0.44)	0.37(0.31)	−0.08(0.30)	0.24(0.21)	0.42	0.48	-	95.3%	1.28(0.69)
ESM	−0.14(0.34)	0.29(0.24)	−0.06(0.25)	0.19(0.18)	0.33	0.42	0.35	94.3%	1.02(0.59)
Poisson Model, $n=700$
HulC	-	-	-	-	-	-	-	92.8%	1.37(0.84)
BNN	0.26(0.43)	0.40(0.31)	0.12(0.31)	0.28(0.18)	0.09	-	-	73.2%	0.92(0.34)
Ensemble	−0.10(0.30)	0.25(0.20)	−0.04(0.23)	0.17(0.16)	0.29	0.37	-	97.9%	1.06(0.56)
Naive bp	−0.10(0.31)	0.25(0.20)	−0.04(0.24)	0.18(0.16)	0.30	0.35	-	97.1%	1.03(0.54)
ESM	−0.04(0.24)	0.19(0.15)	−0.02(0.19)	0.14(0.13)	0.23	0.32	0.24	94.5%	0.72(0.38)

We also investigate the performance of narrower yet deeper architectures under our method. We design a deeper network with $L=5$ and $\mathbf{p}=(p,\mathrm{10,15,20,30,1})$. As deep networks may cause gradient explosion or vanishing during backpropagation (Glorot & Bengio 2010), we incorporate batch normalization layers after the second and second-tolast layers, aiming to mitigate gradient instability while maintaining the network’s ability to learn complex representations. Table 3 shows that key metrics, including empSD, SE, and ${\text{SE}}_c$, remain largely consistent with those observed for shallower network architectures. Nevertheless, coverage probabilities drop to around 91%. Figure 3 presents representative results for $n=400$ and $r=n^{0.8}$. As shown in Figure 3a, the fitted values exhibit a slight systematic deviation from the diagonal, indicating bias relative to the true function $f_0$. Nevertheless, Figure 3b confirms that the corrected variance estimates remain accurate. This bias shifts the confidence intervals, leading to reduced coverage despite precise variance estimation. Similar bias-related undercoverage has been reported for random forests in Wager & Athey (2018). In our case, greater network depth may induce complex interactions that, without sufficient data, amplify estimation bias.

Table 3:

Simulation results for different models under varying sample sizes $n$ and subsampling ratios $r$ with different DNNs.

	${\text{Bias}}_f$	${\text{MAE}}_f$	${\text{Bias}}_{{\psi }^{\prime }}$	${\text{MAE}}_{{\psi }^{\prime }}$	EmpSD	SE	${\mathbf{S}\mathbf{E}}_{\mathit{c}}$	CP	AIL
Logistic Model
$n=400,r=n^{0.8}$	0.20(0.51)	0.44(0.33)	0.04(0.11)	0.10(0.07)	0.49	0.80	0.51	90.2%	0.40(0.11)
$r=n^{0.85}$	0.20(0.52)	0.44(0.34)	0.04(0.11)	0.09(0.07)	0.50	0.86	0.52	90.2%	0.41(0.11)
$r=n^{0.9}$	0.19(0.53)	0.45(0.34)	0.04(0.11)	0.10(0.07)	0.51	0.95	0.53	91.0%	0.41(0.11)
$n=700,r=n^{0.8}$	0.16(0.39)	0.34(0.26)	0.04(0.09)	0.08(0.06)	0.37	0.84	0.38	90.5%	0.31(0.08)
$r=n^{0.85}$	0.20(0.39)	0.35(0.27)	0.04(0.09)	0.08(0.06)	0.37	0.78	0.38	91.1%	0.31(0.08)
$r=n^{0.9}$	0.16(0.39)	0.34(0.26)	0.04(0.09)	0.07(0.06)	0.37	0.84	0.38	91.4%	0.31(0.08)
Poisson Model
$n=400,r=n^{0.8}$	−0.35(0.38)	0.38(0.28)	−0.19(0.21)	0.23(0.18)	0.30	0.50	0.31	86.1%	0.70(0.35)
$r=n^{0.85}$	−0.26(0.33)	0.33(0.26)	−0.14(0.21)	0.20(0.16)	0.31	0.55	0.32	90.0%	0.78(0.39)
$r=n^{0.9}$	−0.17(0.32)	0.28(0.23)	−0.08(0.23)	0.18(0.16)	0.31	0.56	0.32	91.8%	0.86(0.45)
$n=700,r=n^{0.8}$	−0.22(0.24)	0.26(0.20)	−0.13(0.17)	0.16(0.14)	0.23	0.47	0.23	88.0%	0.57(0.27)
$r=n^{0.85}$	−0.13(0.25)	0.21(0.18)	−0.07(0.18)	0.14(0.13)	0.23	0.51	0.23	92.3%	0.63(0.29)
$r=n^{0.9}$	−0.05(0.25)	0.20(0.16)	−0.02(0.19)	0.14(0.13)	0.23	0.56	0.24	93.4%	0.68(0.34)

Figure 3:

Estimation and inference in simulation samples: Logistic Model with $n=400$ and $r=n^{0.8}$ under a deeper network. Figure 3a displays the average estimated $\mathbb{E}(y\mid \mathbf{x})$ and its deviation (gray band) across 300 runs at various test points. Figure 3b compares corrected and uncorrected SEs and their deviations (gray and blue bands) across 300 runs with empirical SDs across test samples.

5. Analysis of eICU data

Preventing ICU readmissions is vital as readmitted patients face higher mortality risks and increased expenses. Identifying high-risk individuals enables more personalized care and optimized resource use. The eICU Collaborative Research Database (eICU-CRD) (Pollard et al. 2018) is a large-scale, multi-center collection of anonymized ICU patient data, created through collaboration between the MIT Laboratory for Computational Physiology and healthcare organizations. It includes detailed information on patients’ physiological measurements, clinical data, lab results, medication usage, and outcomes during their ICU stays. Our analysis focuses on Hospital #188, a major hospital with 2,632 patients admitted to ICU during 2014–2015. We include ten covariates in our analysis, selected for their clinical relevance and frequent use in prior predictive modeling studies. The features include six lab tests (Hct, chloride, WBC, Hgb, RBC, glucose), patient age, Emergency Department admission status, and two APACHE-based severity indicators: anticipated hospital stay and mortality likelihood. Among the 2,632 patients admitted to the ICU, 1,476 were not readmitted, 1,046 were readmitted once, 77 were readmitted twice, 26 were readmitted three times, and 7 were readmitted five or more times. We apply nonparametric logistic and Poisson models within our ESM framework to estimate, respectively, the probability of ICU readmission and total number of ICU readmissions using the laboratory and demographic information. We validate our results using 5-fold cross-validation. The dataset is split into five equal-sized folds; the model is trained on four folds $(n=2105)$, and the ESM method is applied to estimate the subject-specific means and confidence intervals for each subject in the held-out fold, using only covariates while withholding true outcomes. Repeating this process across all folds yields point estimates and inference intervals for all observations.

To estimate the probability of ICU readmission, we define a binary outcome variable: $y=1$ if any readmission occurs during follow-up, i.e., 2014–2015, and $y=0$ otherwise. When implementing our method, we set the subsampling size to $r=n^{0.9}=979$ and the number of subsampling iterations to $B=3000$. We use a (10, 128, 64, 1) architecture with ReLU activation, selected for its balance of accuracy and efficiency via simulations. Training is performed over 300 epochs with a tuned learning rate of $\eta =0.1$.

As shown in Figure 4, both random forests and neural networks achieve similar predictive performance (AUC = 0.84), but their uncertainty estimates differ much. Neural networks yield shorter average confidence intervals (0.123 vs. 0.225 for RFs). Figures 4b and 4d highlight these differences by displaying predicted readmission probabilities with 95% intervals across patients sorted by risk. The variability in interval widths suggests heteroskedasticity, likely driven by differences in data quality or patient characteristics. Notably, Figure 4b reveals a subset of patients (indices between 800–1600) with unusually high uncertainty under DNNs; further details are provided in Section B.1.

Figure 4:

Evaluation of the nonparametric logistic model estimator. Figure 4a and 4c show the ROC curve with both AUC of 0.84. Figure 4b and 4d display estimated subject-level probabilities of ICU readmission and confidence intervals, illustrating heteroskedasticity across individuals.

To estimate subject-level ICU readmissions during 2014–2015, we fit a nonparametric Poisson regression using a DNN to model $\mathbb{E}\left(y\mid {\mathbf{x}}_{*}\right)$, offering a complementary view of patient well-being. Figure 5 shows a lift curve around 0.72, indicating comparable predictive accuracy for DNNs and RFs. However, uncertainty estimates differ: DNNs yield average interval lengths of 0.124, compared to 0.274 for RFs. Figures 5b and 5d illustrate individual predictions with 95% intervals, again showing heteroskedasticity. Additional analyses (Section B.2) reveal that in some settings, RFs produce narrower intervals than DNNs, highlighting model sensitivity to data structure and signal-to-noise characteristics. We also assess the transferability of ESM by applying models trained on hospital #188 directly to data from hospital #458; see Section B.3 in the Supplementary Material. The results indicate that the model remains transferable in this setting.

Figure 5:

Evaluation of the nonparametric logistic model estimator. Figure 5a and 5c show the ROC curve with both AUC of 0.72. Figure 5b and 5d display estimated subject-level probabilities of ICU readmission and confidence intervals, illustrating heteroskedasticity across individuals.

6. Discussion

This paper presents a general framework for inference on subject-level means estimated by deep neural networks for categorical and exponential family outcomes. We construct a DNN estimator by minimizing the loss function induced from the generalized nonparametric regression model (GNRM). To address a key gap in leveraging this general loss function, we analyze the convergence rates of DNNs under the GNRM framework and establish connections with U-statistics and Hoeffding decomposition theory. Building on these results, we introduce an ensemble subsampling method (ESM) to enable valid inference. Numerical studies and an application to the eICU dataset demonstrate ESM’s utility in quantifying predictive uncertainty and supporting personalized care.

Our methodology may have broad applicability as it enables valid inference for any continuously-differentiable functional of $f_0\left(x\right)$, such as $\text{Var}(y\mid \mathbf{x})={\psi }^{″}\left(f_0\left(x\right)\right)≕g\left(f_0\left(x\right)\right)$, where $g\left(u\right)={\psi }^{″}\left(u\right)$, the second derivative of $\psi \left(u\right)$. We can estimate $\text{Var}(y\mid \mathbf{x})$ via the plug-in estimator $g\left({\widehat{f}}^B(\mathbf{x})\right)$ and apply the delta method to give $\sqrt{\frac{n}{r^2{\xi }_{1,r}(\mathbf{x})}}\cdot (g({\widehat{f}}^B(\mathbf{x}))-g\left(f_0(\mathbf{x})\right))=\sqrt{\frac{n}{r^2{\xi }_{1,r}(\mathbf{x})}}\cdot g^{\prime }\left(f_0(\mathbf{x})\right)\left({\widehat{f}}^B(\mathbf{x})-f_0(\mathbf{x})\right)+o_p(1)$, which will lead to estimated confidence intervals. Our method may also naturally extend to semiparametric causal inference settings where the target parameter can be expressed as a functional of the conditional means. Specifically, to estimate the Conditional Average Treatment Effect (CATE), $\tau (\mathbf{x})=\mathbb{E}\left[y\right(1)\mid \mathbf{x}]-\mathbb{E}\left[y\right(0)\mid \mathbf{x}]$, we apply ESM separately to treated and control groups under standard unconfoundedness. Multiple subsamples are drawn for each group, predictive models are trained to estimate $\mathbb{E}\left[y\right(g)\mid \mathbf{x}]$, and ensemble means ${\widehat{\mu }}_1(\mathbf{x})$ and ${\widehat{\mu }}_0(\mathbf{x})$ are obtained. The CATE estimator is $\widehat{\tau }(\mathbf{x})={\widehat{\mu }}_1(\mathbf{x})-{\widehat{\mu }}_0(\mathbf{x})$. ESM’s variance framework provides inference by obtaining ${\widehat{\sigma }}_g^2(\mathbf{x})=\widehat{\text{Var}}\left\{{\widehat{\mu }}_g(\mathbf{x})\right\}$ for $g=\mathrm{0,1}$, yielding pointwise confidence intervals $\widehat{\tau }(\mathbf{x})\pm z_{1-\alpha /2}\sqrt{{\widehat{\sigma }}_1^2(\mathbf{x})+{\widehat{\sigma }}_0^2(\mathbf{x})}$. Detailed justifications warrant investigation.

Our theoretical analysis focuses on ReLU activations, where monomial-based approximations are tractable. We envision that ESM can be extended to networks with smooth activation functions such as tanh or GELU, although this requires careful consideration. Unlike ReLU, smooth activations are less flexible in local adaptations, which may result in slower approximation rates and influence convergence behavior. To preserve inference validity in this setting, it may be necessary to employ alternative analytical tools or apply stronger regularization to control model complexity and prevent saturation. Moreover, Our theoretical analysis relies on several assumptions that are broadly consistent with recent work on implicit bias and convergence in overparameterized models (Chizat & Bach 2020, Chatterji et al. 2021, Cao et al. 2022). We acknowledge, however, that some of these assumptions may appear strong. In particular, the conditions imposed on ${\xi }_{1,r}\left(\mathbf{x}\right)$, which ensure the validity of the first-order Hoeffding projection, can be further improved. An important direction for future work is to relax these conditions by linking them more directly to of the neural network class, such as its sparsity or other structural characteristics.

Data Availability Statement.

The data used in this study are publicly available from the eICU Collaborative Research Database (https://eicu-crd.mit.edu/) upon completion of a data use agreement and approval through the PhysioNet credentialed access process.