Estimation and Accuracy after Model Selection

Abstract

Classical statistical theory ignores model selection in assessing estimation accuracy. Here we consider bootstrap methods for computing standard errors and confidence intervals that take model selection into account. The methodology involves bagging, also known as bootstrap smoothing, to tame the erratic discontinuities of selection-based estimators. A useful new formula for the accuracy of bagging then provides standard errors for the smoothed estimators. Two examples, nonparametric and parametric, are carried through in detail: a regression model where the choice of degree (linear, quadratic, cubic, …) is determined by the C_p criterion, and a Lasso-based estimation problem.

1 Introduction

Accuracy assessments of statistical estimators customarily are made ignoring model selection. A preliminary look at the data might, for example, suggest a cubic regression model, after which the fitted curve’s accuracy is computed as if “cubic” were pre-chosen. Here we will discuss bootstrap standard errors and approximate confidence intervals that take into account the model-selection procedure.

Figure 1 concerns the Cholesterol data, an example investigated in more detail in Section 2: n = 164 men took cholestyramine, a proposed cholesterol-lowering drug, for an average of seven years each; the response variable was the decrease in blood-level cholesterol measured from the beginning to the end of the trial,

Figure 1. Cholesterol data.

Cholesterol decrease plotted versus adjusted compliance for 164 men in Treatment arm of the cholostyramine study (Efron and Feldman, 1991). Solid curve is OLS cubic regression, as selected by the C_p criterion. How accurate is the curve, taking account of model selection as well as least squares fitting? (Solid arrowed point is Subject 1, featured in subsequent calculations. Bottom numbers indicate compliance for the 11 subjects in the simulation trial of Figure 5.)

$$d=\text{cholesterol decrease};$$ (1.1)

also measured (by pill counts) was compliance, the proportion of the intended dose taken,

$$c=\mathrm{compliance},$$ (1.2)

ranging from zero to full compliance for the 164 men. A transformation of the observed proportions has been made here so that the 164 c values approximate a standard normal distribution,

$$c\dot{\sim }\mathcal{N}(0,1).$$ (1.3)

The solid curve is a regression estimate of decrease d as a cubic function of compliance c, fit by ordinary least squares (OLS) to the 164 points. “Cubic” was selected by the C_p criterion, Mallows (1973), as described in Section 2. The question of interest for us is how accurate is the fitted curve, taking account of the C_p model-selection procedure as well as OLS estimation?

More specifically, let μ_j be the expectation of cholesterol decrease for subject j given his compliance c_j,

$${\mu }_j=E\{d_j\mid c_j\}.$$ (1.4)

We wish to assign standard errors to estimates of μ_j read from the regression curve in Figure 1 . A nonparametric bootstrap estimate ${\widetilde{\mathrm{sd}}}_j$ of standard deviation, taking account of model selection, is developed in Sections 2 and 3. Figure 2 shows that this is usually, but not always, greater than the naive estimate ${\overline{\mathrm{sd}}}_j$ obtained from standard OLS calculations, assuming that the cubic model was pre-selected. The ratio ${\stackrel{\sim }{\mathrm{sd}}}_j/{\overline{\mathrm{sd}}}_j$ has median value 1.52; so at least in this case, ignoring model selection can be deceptively optimistic.

Figure 2.

Solid points: ratio of standard deviations, taking account of model selection or not, for the 164 values ${\widehat{\mu }}_j$ from the regression curve in Figure 1. Median ratio equals 1.52. Standard deviations including model selection are the smoothed bootstrap estimates ${\stackrel{\sim }{\mathrm{sd}}}_B$ of Section 3. Dashed line: ratio of ${\stackrel{\sim }{\mathrm{sd}}}_B$ to ${\widehat{\mathrm{sd}}}_B$, the unsmoothed bootstrap sd estimates as in (2.4), median 0.91.

Data-based model selection can produce “jumpy” estimates that change values discontinuously at the boundaries between model regimes. Bagging (Breiman, 1996), or bootstrap smoothing, is a model-averaging device that both reduces variability and eliminates discontinuities. This is described in Section 2, and illustrated on the Cholesterol data.

Our key result is a new formula for the delta-method standard deviation of a bagged estimator. The result, which applies to general bagging situations and not just regression problems, is described in Section 3. Stated in projection terms (see Figure 4), it provides the statistician a direct assessment of the cost in reduced accuracy due to model selection.

Figure 4.

Illustration of Corollary 1. The ratio ${\stackrel{\sim }{\mathrm{sd}}}_B/{\widehat{\mathrm{sd}}}_B$ is the cosine of the angle between t^* – s₀1 (3.9) and the linear space $ℒ({\mathit{Y}}^{\ast })$ spanned by the centered bootstrap counts (3.2). Model-selection estimators tend to be more nonlinear, yielding smaller ratios, i.e., greater gains from smoothing.

A parametric bootstrap version of the smoothing theory is described in Sections 4 and 5. Parametric modeling allows more refined results, permiting second order-accurate confidence calculations of the BCa or ABC type, as in DiCiccio and Efron (1992), Section 6. Section 7 concludes with notes, details, and deferred proofs.

Bagging (Breiman, 1996) has become a major technology in the prediction literature, an excellent recent reference being Buja and Stuetzle (2006). The point of view here agrees with that in Bühlmann and Yu (2002), though their emphasis is more theoretical and less data-analytic. They employ bagging to “change hard thresholding estimators to soft thresholding,” in the same spirit as our Section 2.

Berk, Brown, Buja, Zhang and Zhao (2012) develop conservative normal-theory confidence intervals that are guaranteed to cover the true parameter value regardless of the preceding model-selection procedure. Very often it may be difficult to say just what selection procedure was used, in which case the conservative intervals are appropriate. The methods of this paper assume that the model-selection procedure is known, yielding smaller standard error estimates and shorter confidence intervals.

Hjort and Claeskens (2003) construct an ambitious large-sample theory of frequentist model-selection estimation and model averaging, while making comparisons with Bayesian methods. In theory, the Bayesian approach offers an ideal solution to model-selection problems, but, as Hjort and Claeskens point out, it requires an intimidating amount of prior knowledge from the statistician. The present article is frequentist in its methodology.

Hurvich and Tsai (1990) provide a nice discussion of what “frequentist” might mean in a model-selection framework. (Here I am following their “overall” interpretation.) The nonparametric bootstrap approach in Buckland, Burnham and Augustin (1997) has a similar flavor to the computations in Section 2.

Classical estimation theory ignored model selection out of necessity. Armed with modern computational equipment, statisticians can now deal with model-selection problems more realistically. The limited, but useful, goal of this paper is to provide a general tool for the assessment of standard errors in such situations. Simple parameters like (1.4) are featured in our examples, but the methods apply just as well to more complicated functionals, for instance the maximum value of a regression surface, or a tree-based estimate.

2 Nonparametric bootstrap smoothing

For the sake of simple notation, let y represent all the observed data, and $\widehat{\mathit{\mu }}=t(\mathit{y})$ an estimate of a parameter of interest μ. The Cholesterol data has

$$\mathit{y}=\{(c_j,d_j),j=1,2,\dots ,n=164\}.$$ (2.1)

If μ = μ_j (1.4) we might take ${\widehat{\mu }}_j$ to be the height of the C_p-OLS regression curve measured at compliance c = c_j.

In a nonparametric setting we have data

$$\mathit{y}=(y_1,y_2,\dots ,y_n)$$ (2.2)

where the y_j are independent and identically distributed (iid) observations from an unknown distribution F, a two-dimensional distribution in situation (2.1). The parameter is some functional μ = T(F), but the plug-in estimator $\widehat{\mu }=T(\widehat{F})$, where $\widehat{F}$ is the empirical distribution of the y_j values, is usually what we hope to improve upon in model-selection situations.

A nonparametric bootstrap sample

$${\mathit{y}}^{\ast }=(y_1^{\ast },y_2^{\ast },\dots ,y_n^{\ast })$$ (2.3)

consists of n draws with replacement from the set {y₁, y₂, … ,y_n}, yielding bootstrap replication ${\widehat{\mu }}^{\ast }=t\left({\mathit{y}}^{\ast }\right)$. The empirical standard deviation of B such draws,

$${\widehat{\mathrm{sd}}}_B={\left[{{\displaystyle \sum _{i-1}^B({\widehat{\mu }}_i^{\ast }-{\widehat{\mu }}_{.}^{\ast })}}^2/(B-1)\right]}^{1/2},\left({\widehat{\mu }}_{.}^{\ast }={\displaystyle \sum {\widehat{\mu }}_i^{\ast }/B}\right),$$ (2.4)

is the familiar nonparametric bootstrap estimate of standard error for $\widehat{\mu }$ (Efron, 1979); ${\widehat{\mathrm{sd}}}_B$ is a dependable accuracy estimator in most standard situations but, as we will see, it is less dependable for setting approximate confidence limits in model-selection contexts.

The cubic regression curve in Figure 1 was selected using the C_p criterion. Suppose that under “Model m” we have

$$\mathit{y}=X_m{\beta }_m+\mathit{\epsilon }[\mathit{\epsilon }\sim (0,{\sigma }^{2}I)]$$ (2.5)

where X_m is a given n by m structure matrix of rank m, and ε has mean 0 and covariance σ² times the Identity (σ assumed known in what follows). The C_p measure of fit for Model m is

$$C_p(m)={\Vert \mathit{y}-X_m{\widehat{\beta }}_m\Vert }^2+2{\sigma }^{2}m$$ (2.6)

with ${\widehat{\beta }}_m$ the OLS estimate of β_m; given a collection of possible choices for the structure matrix, the C_p criterion selects the one minimizing C_p.

Table 1 shows C_p results for the Cholesterol data. Six polynomial regression models were compared, ranging from linear (m = 2) to sixth degree (m = 7); the value σ = 22.0 was used, corresponding to the standard estimate $\widehat{\sigma }$ obtained from the sixth degree model. The cubic model (m = 4) minimized C_p(m), leading to its selection in Figure 1.

Table 1.

C_p model selection for the Cholesterol data; measure of fit C_p(m) (2.6) for polynomial regression models of increasing degree. The cubic model minimizes C_p(m). (Value σ = 22.0 was used here and in all bootstrap replications.) Last column shows percentage each model was selected as the C_p minimizer, among B = 4000 bootstrap replications.

Regression model	m	Cp(m) − 80, 000	(Bootstrap %)
Linear	2	1132	(19%)
Quadratic	3	1412	(12%)
Cubic	4	667	(34%)
Quartic	5	1591	(8%)
Quintic	6	1811	(21%)
Sextic	7	2758	(6%)

B = 4000 nonparametric bootstrap replications of the C_p-OLS regression curve — several times more than necessary, see Section 3 — were generated: starting with a bootstrap sample y ^* (2.3), the equivalent of Table 1 was calculated (still using σ = 22.0) and the C_p minimizing degree m^* selected, yielding the bootstrap regression curve

$${\widehat{\mathit{\mu }}}^{\ast }=X_{m\ast }{\widehat{\beta }}_{m\ast }$$ (2.7)

Where ${\widehat{\beta }}_{m\ast }^{\ast }$ was the OLS coefficient vector for the selected model. The last column of Table 1 shows the various bootstrap model-selection percentages: cubic was selected most often, but still only about one-third of the time.

Suppose we focus attention on Subject 1, the arrowed point in Figure 1, so that the parameter of interest μ₁ can be estimated by the C_p-OLS value $t(\mathit{y})={\widehat{\mu }}_1$, evaluated to be 2.71. Figure 3 shows the histogram of the 4000 bootstrap replications $t({\mathit{y}}^{\ast })={\widehat{\mu }}_1^{\ast }$. The point estimate ${\widehat{\mu }}_1=2.71$ is located to the right, exceeding a surprising 76% of the ${\widehat{\mu }}_1^{\ast }$ values.

Figure 3.

B = 4000 bootstrap replications ${\widehat{\mu }}_1^{\ast }$ of the C_p-OLS regression estimate for Subject 1. The original estimate $t(\mathit{y})={\widehat{\mu }}_1$ is 2.71, exceeding 76% of the replications. Bootstrap standard deviation (2.4) equals 8.02. Triangles indicate 2.5th and 97.5th percentiles of the histogram.

Table 2 shows why. The cases where “Cubic” was selected yielded the largest bootstrap estimates ${\widehat{\mu }}_1^{\ast }$. The actual dataset y fell into the cubic region, giving a correspondingly large estimate ${\widehat{\mu }}_1$. Things might very well have turned out otherwise, as the bootstrap replications suggest: model selection can make an estimate “jumpy” and erratic.

Table 2.

Mean and standard deviation of ${\widehat{\mu }}_1^{\ast }$ as a function of the selected model, 4000 nonparametric bootstrap replications; Cubic, Model 3, gave the largest estimates.

Model	1	2	3	4	5	6
Mean	−13.69	−3.69	4.71	−1.25	−3.80	−3.56
Stdev	3.64	3.48	5.43	5.28	4.46	4.95

We can smooth $\widehat{\mu }=t(\mathit{y})$ by averaging over the bootstrap replications, defining

$$\stackrel{\sim }{\mu }=s(\mathit{y})=\frac{1}{B}{\displaystyle \sum _{i=1}^{B}t({\mathit{y}}^{\ast })}.$$ (2.8)

Bootstrap smoothing (Efron and Tibshirani, 1996), a form of model averaging, is better known as “bagging” in the prediction literature; see Breiman (1996) and Buja and Stuetzle (2006). There its variance reduction properties are emphasized. Our example will also show variance reductions, but the main interest here lies in smoothing; s(y), unlike t(y), does not jump as y crosses region boundaries, making it a more dependable vehicle for setting standard errors and confidence intervals. Suppose, for definiteness, that we are interested in setting approximate 95% bootstrap confidence limits for parameter μ. The usual “standard interval”

$$\widehat{\mu }\pm 1.96{\widehat{\mathrm{sd}}}_B$$ (2.9)

(= 2.71 ± 1.96 . 8.02 in Figure 3) inherits the dangerous jumpiness of $\widehat{\mu }=t(\mathit{y})$. The percentile interval, Section 13.3 of Efron and Tibshirani (1993),

$$\left[{\widehat{\mu }}^{\ast (.025)},{\widehat{\mu }}^{\ast (.975)}\right],$$ (2.10)

the 2.5th and 97.5th percentiles of the B bootstrap replications, yields more stable results. (Notice that it does not require a central point estimate such as $\widehat{\mu }$ in (2.9).)

A third choice, of particular interest here, is the smoothed interval

$$\stackrel{\sim }{\mu }\pm 1.96{\stackrel{\sim }{\mathrm{sd}}}_B$$ (2.11)

where $\stackrel{\sim }{\mu }=s(\mathit{y})$ is the bootstrap smoothed estimate (2.8), while ${\stackrel{\sim }{\mathrm{sd}}}_B$ is given by the projection formula discussed in Section 3. Interval (2.11) combines stability with reduced length.

Table 3 compares the three approximate 95% intervals for μ₁. The reduction in length is dramatic here, though less so for the other 163 subjects; see Section 3.

Table 3.

Three approximate 95% bootstrap confidence intervals for μ₁, the response value for Subject 1, Cholesterol data.

	Interval	Length	Center point
Standard interval (2.9)	(−13.0, 18.4)	31.4	2.71
Percentile interval (2.10)	(−17.8, 13.5)	31.3	−2.15
Smoothed standard (2.11)	(−13.3, 8.0)	21.3	−2.65

The BCa-ABC system goes beyond (2.9)–(2.11) to produce bootstrap confidence intervals having second-order accuracy, as in DiCiccio and Efron (1992). Section 6 carries out the ABC calculations in a parametric bootstrap context.

3 Accuracy of the smoothed bootstrap estimates

The smoothed standard interval $\stackrel{\sim }{\mu }\pm 1.96{\stackrel{\sim }{\mathrm{sd}}}_B$ requires a standard deviation assessment ${\stackrel{\sim }{\mathrm{sd}}}_B$ for the smoothed bootstrap estimate (2.8). A brute force approach employs a second level of bootstrapping: resampling from ${\mathit{y}}_i^{\ast }$ (2.3) yields a collection of B second-level replications ${\mathit{y}}_{\mathit{ij}}^{\ast \ast }$, from which we calculate $s_i^{\ast }={\displaystyle \sum t({\mathit{y}}_{\mathit{ij}}^{\ast \ast })/B}$; repeating this whole process for many replications of ${\mathit{y}}_i^{\ast }$ provides bootstrap values $s_i^{\ast }$ from which we calculate its bootstrap standard deviation.

The trouble with brute force is that it requires an enormous number of recomputations of the original statistic t(⋅). This section describes an estimate ${\stackrel{\sim }{\mathrm{sd}}}_B$ that uses only the original B bootstrap replications $\{t({\mathit{y}}_i^{\ast }),i=1,2,\dots ,B\}$.

The theorem that follows will be stated in terms of the “ideal bootstrap,” where B equals all nⁿ possible choices of ${\mathit{y}}^{\ast }=(y_1^{\ast },y_2^{\ast },\dots ,y_n^{\ast })$ from {y₁, y₂, …, y_n}, each having probability 1/B. It will be straightforward then to adapt our results to the non-ideal bootstrap, with B = 4000 for instance.

Define

$$t_i^{\ast }=t\left({\mathit{y}}_i^{\ast }\right)[{\mathit{y}}_i^{\ast }=(y_{i1}^{\ast },y_{i2}^{\ast },\dots ,y_{ik}^{\ast },\dots ,y_{in}^{\ast }\left)\right],$$ (3.1)

the ith bootstrap replication of the statistic of interest, and let

$$Y_{ij}^{\ast }=\#\{y_{ik}^{\ast }=y_j\},$$ (3.2)

the number of elements of ${\mathit{y}}_i^{\ast }$ equaling the original data point y_j. The vector ${\mathit{Y}}_i^{\ast }=(Y_{i1}^{\ast },Y_{i2}^{\ast },\dots ,Y_{in}^{\ast })$ follows a multinomial distribution with n draws on n categories each of probability 1/n, and has mean vector and covariance matrix

$${\mathit{Y}}_i^{\ast }\sim (1_n,\mathit{I}-1_n{1^{\prime }}_n/n),$$ (3.3)

1_n the vector of n 1’s and I the n × n identity matrix.

Theorem 1. The nonparametric delta-method estimate of standard deviation for the ideal smoothed bootstrap statistic $s(\mathit{y})={\displaystyle {\sum }_{i=1}^{B}t({\mathit{y}}_i^{\ast })/B}$ is

$$\stackrel{\sim }{\mathrm{sd}}={\left[{\displaystyle \sum _{j=1}^n{\mathrm{cov}}_j^2}\right]}^{1/2}$$ (3.4)

where

$${\mathrm{cov}}_j={\mathrm{cov}}_{\ast }(Y_{\mathit{ij}}^{\ast },t_i^{\ast }),$$ (3.5)

the bootstrap covariance between $Y_{\mathit{ij}}^{\ast }$ and $t_i^{\ast }$.

(The proof appears later in this section.)

The estimate of standard deviation for s(y) in the non-ideal case is the analogue of (3.4),

$${\stackrel{\sim }{\mathrm{sd}}}_B={\left[{\displaystyle \sum _{j=1}^n{\widehat{\mathrm{cov}}}_j^2}\right]}^{1/2}$$ (3.6)

where

$${\widehat{\mathrm{cov}}}_j={\displaystyle \sum _{i=1}^n(Y_{ij}^{\ast }-Y_{.j}^{\ast })(t_i^{\ast }-t_{.}^{\ast })}/B$$ (3.7)

with $Y_{.j}^{\ast }={\displaystyle {\sum }_{i=1}^B{Y}_{ij}^{\ast }}/B$ and $t_{.}^{\ast }={\displaystyle {\sum }_{i=1}^B{t}_i^{\ast }/B=s(\mathit{y})}$. Remark J concerns a bias correction for (3.6) that can be important in the non-ideal case (it wasn’t in the Cholesterol example). All of these results apply generally to bagging estimators, and are not restricted to regression situations.

Figure 2 shows that ${\stackrel{\sim }{\mathrm{sd}}}_B$ is less than ${\widehat{\mathrm{sd}}}_B$, the bootstrap estimate of standard deviation for the unsmoothed statistic,

$${\widehat{\mathrm{sd}}}_B={\left[{\displaystyle \sum {(t_i^{\ast }-t_{.}^{\ast })}^2/B}\right]}^{1/2},$$ (3.8)

for all 164 estimators $t(\mathit{y})={\widehat{\mu }}_j$. This is no accident. Returning to the ideal bootstrap situation, let $ℒ({\mathit{Y}}^{\ast })$ be the (n − 1)-dimensional subspace of ${ℛ}^B$ spanned by the columns of the B × n matrix having elements $Y_{ij}^{\ast }-1$. (Notice that ${\displaystyle {\sum }_{i=1}^B{Y}_{ij}^{\ast }/B=1}$ according to (3.3).) Also define $s_0={\displaystyle {\sum }_{i=1}^B{t}_i^{\ast }/B}$, the ideal bootstrap smoothed estimate, so

$${\mathit{U}}^{\ast }\equiv {\mathit{t}}^{\ast }-s_{0}1$$ (3.9)

is the B-vector of mean-centered replications $t_i^{\ast }-s_0$. Note: Formula (3.6) is a close cousin of the “jackknife-after-bootstrap” method of Efron (1992), the difference being the use of jackknife rather than our infinitesimal jackknife calculations.

Corollary 1. The ratio ${\stackrel{\sim }{\mathrm{sd}}}_B/{\widehat{\mathrm{sd}}}_B$ is given by

$$\frac{{\stackrel{\sim }{\mathrm{sd}}}_B}{{\widehat{\mathrm{sd}}}_B}=\frac{\parallel {\widehat{\mathit{U}}}^{\ast }\parallel }{\parallel {\mathit{U}}^{\ast }\parallel }$$ (3.10)

where ${\widehat{\mathit{U}}}^{\ast }$ is the projection of U^* into $ℒ({\mathit{Y}}^{\ast })$.

(See Remark A in Section 7 for the proof. Remark B concerns the relation of Theorem 1 to the Hájek projection.)

The illustration in Figure 4 shows ${\stackrel{\sim }{\mathrm{sd}}}_B/{\widehat{\mathrm{sd}}}_B$ as the cosine of the angle between t^* – s₀1 and $ℒ({\mathit{Y}}^{\ast })$. The ratio is a measure of the nonlinearity of $t_i^{\ast }$ as a function of the bootstrap counts $Y_{\mathit{ij}}^{\ast }$. Model selection induces discontinuities in t(⋅), increasing the nonlinearity and decreasing ${\stackrel{\sim }{\mathrm{sd}}}_B/{\widehat{\mathrm{sd}}}_B$. The 164 ratios shown as the dashed line in Figure 2 had median 0.91, mean 0.89.

How many bootstrap replications B are necessary to ensure the accuracy of ${\stackrel{\sim }{\mathrm{sd}}}_B$? The jackknife provides a quick answer: divide the B replications into J groups of size B/J each, and let ${\stackrel{\sim }{\mathrm{sd}}}_{Bj}$ be the estimate (3.6) computed with the jth group removed. Then

$${\stackrel{\sim }{\mathrm{cv}}}_B={\left[\frac{J}{J-1}{\displaystyle \sum _{j=1}^J{({\stackrel{\sim }{\mathrm{sd}}}_{Bj}-{\stackrel{\sim }{\mathrm{sd}}}_B.)}^2}\right]}^{1/2}/{\stackrel{\sim }{\mathrm{sd}}}_B,$$ (3.11)

${\widetilde{\mathrm{sd}}}_B.={\displaystyle \sum {\stackrel{\sim }{\mathrm{sd}}}_{Bj}/J}$, is the jackknife estimated coefficient of variation for ${\stackrel{\sim }{\mathrm{sd}}}_B$. Applying (3.11) with J = 20 to the first B = 1000 replications (of the 4000 used in Figure 2) yielded ${\stackrel{\sim }{\mathrm{cv}}}_B$ values of about 0.05 for each of the 164 subjects. Going on to B = 4000 reduced the ${\stackrel{\sim }{\mathrm{cv}}}_B$’s to about 0.02. Stopping at B = 1000 would have been quite sufficient. Note: ${\stackrel{\sim }{\mathrm{cv}}}_B$ applies to the bootstrap accuracy of ${\stackrel{\sim }{\mathrm{sd}}}_B$ as an estimate of the ideal value $\stackrel{\sim }{\mathrm{sd}}$ (3.4), not to sampling variability due to randomness in the original data y, while ${\stackrel{\sim }{\mathrm{sd}}}_B$ itself does refer to sampling variability.

Proof of Theorem 1. The “nonparametric delta method” is the same as the influence function and infinitesimal jackknife methods described in Chapter 6 of Efron (1982). It is appropriate here because s(y), unlike t(y), is a smooth function of y. With the original data vector y (2.2) fixed, we can write bootstrap replication $t_i^{\ast }=t({\mathit{y}}_i^{\ast })$ as a function $T({\mathit{Y}}_i^{\ast })$ of the count vector (3.2). The ideal smoothed bootstrap estimate s₀ is the multinomial expectation of T(Y*),

$$s_0=E\left\{T\left({\mathit{Y}}^{\ast }\right)\right\},{\mathit{Y}}^{\ast }\sim {\mathrm{Mult}}_n\left(n,{\mathit{p}}_0\right),$$ (3.12)

p₀ = (1/n, 1/n, … , 1/n), the notation indicating a multinomial distribution with n draws on n equally likely categories.

Now let S(p) denote the multinomial expectation of T(Y*) if the probability vector is changed from p₀ to p = (p₁, p₂, … , p_n),

$$S\left(\mathit{p}\right)=E\{T({\mathit{Y}}^{\ast })\},{\mathit{Y}}^{\ast }\sim {\mathrm{Mult}}_n\left(n,\mathit{p}\right),$$ (3.13)

so S(p₀) = s₀. Define the directional derivative

$${\dot{S}}_j=\underset{\epsilon \to 0}{{\displaystyle \lim }}\frac{S({\mathit{p}}_0+\epsilon ({\mathit{\delta }}_j-{\mathit{p}}_0))-S({\mathit{p}}_0)}{\epsilon },$$ (3.14)

δ_j the jth coordinate vector (0, 0, … , 0, 1, 0, … , 0), with 1 in the jth place. Formula (6.18) of Efron (1982) gives

$${\left({\displaystyle \sum _{j=1}^n{\dot{S}}_j^2}\right)}^{1/2}/n$$ (3.15)

as the delta method estimate of standard deviation for s₀. It remains to show that (3.15) equals (3.4).

Define w_i(p) to be the ratio of the probabilities of ${\mathit{Y}}_i^{\ast }$ under (3.13) compared to (3.12),

$$w_i(\mathit{p})={\displaystyle \prod _{k=1}^n}{(n{p}_k)}^{Y_{ik}^{\ast }},$$ (3.16)

so that

$$S(\mathit{p})={\displaystyle \sum _{i=1}^B{w}_i(\mathit{p})t_i^{\ast }/B}$$ (3.17)

(the factor 1/B reflecting that under p₀, all the ${\mathit{Y}}_i^{\ast }’s$ have probability 1/B = 1/nⁿ).

For p(ε) = p₀ + ε (δ_j − p₀) as in (3.14), we calculate

$$w_i(\mathit{p})={(1+(n-1)\epsilon )}^{Y_{ij}^{\ast }}{(1-\epsilon )}^{{\displaystyle {\sum }_{k\ne j}{Y}_{ik}^{\ast }}}.$$ (3.18)

Letting ε → 0 yields

$$w_i(\mathit{p})\doteq 1+n\epsilon (Y_{ij}^{\ast }-1)$$ (3.19)

where we have used ${\displaystyle {\sum }_k{Y}_{ik}^{\ast }/n=1}$. Substitution into (3.17) gives

$$\begin{array}{ccc}\hfill S(\mathit{p}(\epsilon ))& \doteq & {\displaystyle \sum _{i=1}^B[1+n\epsilon (Y_{ij}^{\ast }-1)]}{t}_i^{\ast }/B\hfill \\ & =& s_0+n\epsilon {\mathrm{cov}}_j\hfill \end{array}$$ (3.20)

as in (3.5). Finally, definition (3.14) yields

$${\dot{S}}_j=n{\mathrm{cov}}_j$$ (3.21)

and (3.15) verifies Theorem 1 (3.4).

The validity of an approximate 95% interval $\widehat{\theta }\pm 1.96\widehat{\sigma }$ is compromised if the standard error σ is itself changing rapidly as a function of θ. Acceleration $\widehat{a}$ (Efron, 1987) is a measure of such change. Roughly speaking,

$$\widehat{a}={\frac{d\sigma }{d\theta }\mid }_{\widehat{\theta }}.$$ (3.22)

If $\widehat{a}=0.10$ for instance, then at the upper endpoint ${\widehat{\theta }}_{\mathrm{up}}=\widehat{\theta }+1.96\widehat{\sigma }$ the standard error will have increased to about $1.196\widehat{\sigma }$, leaving ${\widehat{\theta }}_{\mathrm{up}}$ only 1.64, not 1.96, σ-units above $\widehat{\theta }$. (The 1987 paper divides definition (3.22) by 3, as being appropriate after a normalizing transformation.)

Acceleration has a simple expression in terms of the covariances ${\widehat{\mathrm{cov}}}_j$ used to calculate ${\stackrel{\sim }{\mathrm{sd}}}_B$ in (3.6),

$$\widehat{a}=\frac{1}{6}\left[{\displaystyle \sum _{j=1}^n{\widehat{\mathrm{cov}}}_j^3}/{\left({\displaystyle \sum {\widehat{\mathrm{cov}}}_j^2}\right)}^{3/2}\right],$$ (3.23)

equation (7.3) of Efron (1987). The $\widehat{a}$’s were small for the 164 ${\stackrel{\sim }{\mathrm{sd}}}_B$ estimates for the Cholesterol data, most of them falling between −0.02 and 0.02, strengthening belief in the smoothed standard intervals ${\stackrel{\sim }{\mu }}_i\pm 1.96{\stackrel{\sim }{\mathrm{sd}}}_{Bi}$ (2.11).

Bias is more difficult to estimate than variance, particularly in a nonparametric context. Remark C of Section 7 verifies the following promising-looking result: the nonparametric estimate of bias for the smoothed estimate $\stackrel{\sim }{\mu }=s(\mathit{y})$ (2.8) is

$$\stackrel{\sim }{\mathrm{bias}}=\frac{1}{2}{\mathrm{cov}}_{\ast }(Q_i^{\ast },t_i^{\ast })\mathrm{where}{Q}_i^{\ast }={{\displaystyle \sum _{k=1}^n(Y_{nk}^{\ast }-1)}}^2,$$ (3.24)

with cov_* indicating bootstrap covariance as in (3.5). Unfortunately, $\stackrel{\sim }{\mathrm{bias}}$ proved to be too noisy to use in the Cholesterol example. Section 6 describes a more practical approach to bias estimation in a parametric bootstrap context.

4 Parametric bootstrap smoothing

We switch now from nonparametric to parametric estimation problems, but ones still involving data-based model selection. More specifically, we assume that a p-parameter exponential family of densities applies,

$$f_{\alpha }(\widehat{\beta })=e^{{\alpha }^{\prime }\widehat{\beta }-\psi (\alpha )}{f}_0(\widehat{\beta }),$$ (4.1)

where α is the p-dimensional natural or canonical parameter vector, $\widehat{\beta }$ the p-dimensional sufficient statistic vector (playing the role of y in (2.2)), $\psi (\alpha )$ the cumulant generating function, and $f_0(\widehat{\beta })$ the “carrying density” defined with respect to some carrying measure (which may include discrete atoms as with the Poisson family). Form (4.1) covers a wide variety of familiar applications, including generalized linear models; $\widehat{\beta }$ is usually obtained by sufficiency from the original data, as seen in the next section.

The expectation parameter vector $\beta =E_{\alpha }\left\{\widehat{\beta }\right\}$ is a one-to-one function of α, say β = λ(α), having p × p derivative matrix

$$\frac{d\beta }{d\alpha }=V(\alpha )$$ (4.2)

where V = V(α) is the covariance matrix ${\mathrm{cov}}_{\alpha }(\widehat{\beta })$. The value of α corresponding to the sufficient statistic $\widehat{\beta },\widehat{\alpha }={\lambda }^{-1}(\widehat{\beta })$, is the maximum likelihood estimate (MLE) of α.

A parametric bootstrap sample is obtained by drawing i.i.d. realizations ${\widehat{\beta }}^{\ast }$ from the MLE density $f_{\widehat{\alpha }}(\cdot )$,

$$f_{\widehat{\alpha }}(\cdot )\underrightarrow{\mathrm{iid}}{\widehat{\beta }}_1^{\ast },{\widehat{\beta }}_2^{\ast },\dots ,{\widehat{\beta }}_B^{\ast }.$$ (4.3)

If $\widehat{\mu }=t(\widehat{\beta })$ is an estimate of a parameter of interest μ, the bootstrap samples (4.3) provide B parametric bootstrap replications of $\widehat{\mu }$,

$${\widehat{\mu }}_i^{\ast }=t({\widehat{\beta }}_i^{\ast }),i=1,2,\dots ,B.$$ (4.4)

As in the nonparametric situation, these can be averaged to provide a smoothed estimate,

$$\tilde{\mu }=s(\widehat{\beta })={\displaystyle \sum _{i=1}^{B}t({\widehat{\beta }}_i^{\ast })/B.}$$ (4.5)

When t(⋅) involves model selection, $\widehat{\mu }$ is liable to an erratic jumpiness, smoothed out by the averaging process.

The bootstrap replications ${\widehat{\beta }}^{\ast }~f_{\widehat{a}}(·)$ have mean vector and covariance matrix

$${\widehat{\beta }}^{\ast }~(\widehat{\beta },\widehat{V})\left[\widehat{V}=V(a)\right].$$ (4.6)

Let B be the B × p matrix with ith row ${\widehat{\beta }}_i^{\ast }-\widehat{\beta }.$ As before, we will assume an ideal bootstrap resampling situation where B → ∞, making the empirical mean and variance of the ${\widehat{\beta }}^{\ast }$ values exactly match (4.6):

$${\mathit{B}}^{\prime }{1}_B/B=O\mathrm{and}{\mathit{B}}^{\prime }\mathit{B}/B=\widehat{V},$$ (4.7)

1_B the vector of B 1’s.

Parametric versions of Theorem 1 and Corollary 1 depend on the p-dimensional bootstrap covariance vector between ${\widehat{\beta }}^{\ast }$ and t* = t(y*),

$${\mathrm{cov}}_{\ast }={\mathit{B}}^{\prime }({\mathit{t}}^{\ast }-s_0{1}_B)/B$$ (4.8)

where t* is the B-vector of bootstrap replications $t_i^{\ast }=t\left({\mathit{y}}^{\ast }\right)$, and s₀ the ideal smoothed estimate (4.5).

Theorem 2. The parametric delta-method estimate of standard deviation for the ideal smoothed estimate (4.5) is

$$\widetilde{\mathrm{sd}}={\left[{\mathrm{cov}}_{\ast }^{\prime }{\widehat{V}}^{-1}{\mathrm{cov}}_{\ast }\right]}^{1/2}.$$ (4.9)

(Proof given at the end of this section.)

Corollary 2. $\widetilde{\mathrm{sd}}$ is always less than or equal to $\widehat{\mathrm{sd}}$, the bootstrap estimate of standard deviation for the unsmoothed estimate,

$$\widehat{\mathrm{sd}}={\left[{\parallel {\mathit{t}}^{\ast }-s_0{1}_B\parallel }^2/B\right]}^{1/2},$$ (4.10)

the ratio being

$$\widetilde{\mathrm{sd}}/\widetilde{\mathrm{sd}}={[{({\mathit{t}}^{\ast }-s_0{1}_B)}^{\prime }\mathit{B}{({\mathit{B}}^{\prime }\mathit{B})}^{-1}{\mathit{B}}^{\prime }({\mathit{t}}^{\ast }-s_0{1}_B)]}^{1/2}/\widehat{\mathrm{sd}}.$$ (4.11)

In the ideal bootstrap case, (4.7) and (4.9) show that $\widetilde{\mathrm{sd}}$ equals B^−1/2 times the numerator on the right-hand side of (4.11). This is recognizable as the length of projection of t − s₀1_B into the p-dimensional linear subspace of ℛ^B spanned by the columns of B. Figure 4 still applies, with ℒ(B) replacing ℒ(Y*).

If t(y) = $\widehat{\mu }$ is multivariate, say of dimension K, then cov_* as defined in (4.8) is a p × K matrix. In this case

$${\mathrm{cov}}_{\ast }^{\prime }{\widehat{V}}^{-1}{\mathrm{cov}}_{\ast }$$ (4.12)

(or ${\widehat{\mathrm{cov}}}^{\prime }{\tilde{V}}^{-1}\widehat{\mathrm{cov}}$ in what follows) is the delta-method assessment of covariance for the smoothed vector estimate $s(\mathit{y})={\displaystyle \sum t({\mathit{y}}_i^{\ast }})/B$, also called t^*. below.

Only minor changes are necessary for realistic bootstrap computations, i.e., for B < ∞. Now we define B as the B × p matrix having ith row ${\widehat{\beta }}_i^{\ast }-{\widehat{\beta }}_{.}^{\ast },$ with ${\widehat{\beta }}_{.}^{\ast }=\sum {\widehat{\beta }}_i^{\ast }/B,$ and compute the empirical covariance vector

$$\widehat{\mathrm{cov}}={\mathit{B}}^{\prime }({\mathit{t}}^{\ast }-t_{.}^{\ast }{1}_B)/B$$ (4.13)

and the empirical bootstrap variance matrix

$$\stackrel{\_}{V}={\mathit{B}}^{\prime }\mathit{B}/B.$$ (4.14)

Then the estimate of standard deviation for the smoothed estimate $\stackrel{\sim }{\mu }=s(\widehat{\beta })$. (4.15) is

$${\stackrel{\sim }{\mathrm{sd}}}_B={\left[{\widehat{\mathrm{cov}}}^{\prime }{\stackrel{\_}{V}}^{-1}\widehat{\mathrm{cov}}\right]}^{1/2}.$$ (4.15)

As $B\to \infty ,\widehat{\mathrm{cov}}\to {\mathrm{cov}}_{\ast }$, $\stackrel{\_}{V}\to \widehat{V},$ so ${\stackrel{\sim }{\mathrm{sd}}}_B\to \stackrel{\sim }{\mathrm{sd}}$ (4.9). Corollary 2, with s₀ replaced by $\stackrel{\sim }{\mu }$ (4.5), remains valid.

Figure 5 reports on a simulation test of Theorem 2. This was based on a parametric model for the Cholesterol data of Figure 1,

Figure 5.

Simulation test of Theorem 2, parametric model (4.16)–(4.18), Cholesterol data; 100 simulations, 1000 parametric bootstraps each, for the 11 subjects indicated at the bottom of Figure 1. Heavy line connects observed empirical standard deviations (4.22); dashes show the 100 estimates $\stackrel{\sim }{\mathrm{sd}}$ from Theorem 2 (4.15). Light dashed line connects averages of the $\stackrel{\sim }{\mathrm{sd}}$ values, as discussed in Remark K.

$$\mathit{y}\sim {\mathcal{N}}_{164}(\mathit{\mu },{\mathit{\sigma }}^2),$$ (4.16)

where ${\mathit{\sigma }}^2$ was diagonal, with diagonal elements a cubic function of compliance c (obtained from a regression precentile fit),

$${\sigma }_i=23.7+5.49c-2.25c^2-1.03c^3,$$ (4.17)

making σ_i about twice as large to the right as to the left. The expectation vector μ was taken to be

$$\mathit{\mu }=X\widehat{\beta }(6)=\widehat{\mathit{\mu }}(6),$$ (4.18)

the sixth degree OLS fit for cholesterol decrease as a function of compliance in (4.16), with X the corresponding 164 × 7 structure matrix.

Model (4.16)–(4.18) is a 7-parameter exponential family (4.1), with sufficient statistic

$$\widehat{\beta }=G^{-1}{X}^{\prime }{({\mathit{\sigma }}^2)}^{-1}\mathit{y}\left[G=X^{\prime }{({\mathit{\sigma }}^2)}^{-1}X\right]$$ (4.19)

and covariance matrix (4.2)

$$V=G^{-1},$$ (4.20)

which is all that is necessary to apply Theorem 2.

The simulation began with 100 draws $y_i^{\ast },i=1,2,\dots ,100,$ from (4.16), each of which gave OLS estimate ${\widehat{\mathit{\mu }}}_i^{\ast }=X{\widehat{\beta }}_i^{\ast }(6).$ Then B = 1000 parametric bootstrap draws were generated from ${\widehat{\beta }}_i^{\ast }$,

$${\mathit{y}}_{\mathit{ij}}^{\ast \ast }\sim N({\widehat{\mathit{\mu }}}_i^{\ast },{\mathit{\sigma }}^2),j=1,2,\dots ,1000,$$ (4.21)

from which smoothed estimate ${\stackrel{\sim }{\mu }}_i$ (4.5) and estimated standard deviation $\stackrel{\sim }{s{d}_i}$ were calculated according to (4.15). All of this was done for 11 of the 164 subjects, as indicated in Figure 1.

The dashes in Figure 5 indicate the 100 $\stackrel{\sim }{s{d}_i}$ values for each of the 11 subjects. This is compared with the observed empirical standard deviations of the smoothed estimates,

$$\widetilde{\mathrm{Sd}}={\left[{\displaystyle \sum _1^{100}{({\stackrel{\sim }{\mu }}_i-\stackrel{\sim }{\mu }.)}^2/99}\right]}^{1/2}\left[\stackrel{\sim }{\mu }.={\displaystyle \sum _1^{100}{\stackrel{\sim }{\mu }}_i/100}\right],$$ (4.22)

connected by the heavy solid curve. The $\stackrel{\sim }{\mathrm{sd}}$ values from Theorem 2 are seen to provide reasonable estimates of $\stackrel{\sim }{\mathrm{Sd}}$, though with some bias and variability.

There is more to the story. The empirical standard deviations $\stackrel{\sim }{\mathrm{sd}}$ are themselves affected by model selection problems. Averaging the 100 $\stackrel{\sim }{\mathrm{sd}}$_i values (connected by the dashed line in Figure 5) gives more dependable results, as discussed in Remark K.

Proof of Theorem 2. Suppose that instead of $f_{\widehat{\alpha }}(\cdot )$ in (4.3) we wished to consider parametric bootstrap samples drawn from some other member of family (4.1), $f_{\alpha }(\cdot )$ (α not necessarily the “true value”). The ratio $w_i=f\alpha ({\widehat{\beta }}_i^{\ast })/f_{\widehat{a}}({\widehat{\beta }}_i^{\ast })$ equals

$$w_i=c_{\alpha ,\widehat{\alpha }}{e}^{Q_i}\mathrm{where}{Q}_i={(\alpha -\widehat{\alpha })}^{\prime }\left({\widehat{\beta }}_i^{\ast }-\widehat{\beta }\right),$$ (4.23)

with the factor $c_{\alpha ,\widehat{\alpha }}$ not depending on ${\widehat{\beta }}_i^{\ast }$. Importance sampling can now be employed to estimate $E_{\alpha }\{t(\widehat{\beta })\}$, the expectation under f_α of statistic $t\left(\widehat{\beta }\right)$, using only the original bootstrap replications $\left({\widehat{\beta }}_i^{\ast },t_i^{\ast }\right)$ from (4.3),

$${\widehat{E}}_{\alpha }={\displaystyle \sum _{i=1}^B{w}_i{t}_i^{\ast }}/{\displaystyle \sum _{i=1}^B{w}_i=}{\displaystyle \sum _{i=1}^B{e}^{Q_i}{t}_i^{\ast }}/{\displaystyle \sum _{i=1}^B{e}^{Q_{\ast }}.}$$ (4.24)

Notice that $\widehat{E}\alpha$ is the value of the smoothed estimate (4.5) at parameter α say s_α. The delta-method standard deviation for our estimate $s_{\widehat{\alpha }}$ depends on the derivative vector ds_α/dα evaluated at = $\alpha =\widehat{\alpha }$. Letting $\alpha \to \widehat{\alpha }$ in (4.23)–(4.24) gives,

$$s_{\alpha }\doteq \frac{{\displaystyle \sum (1+Q_i)t_i^{\ast }/B}}{{\displaystyle \sum (1+Q_i)/B}}=s_{\widehat{\alpha }}+{(\alpha -\widehat{\alpha })}^{\prime }{\mathrm{cov}}_{\ast }$$ (4.24)

where the denominator term $\sum Q_i/B$ equals 0 for the ideal bootstrap according to (4.7). (For the non-ideal bootstrap, $\sum Q_i/B$ approaches 0 at rate $O_p(1/\sqrt{B})$.)

We see that

$$\frac{d{s}_{\alpha }}{d\alpha }{\mid }_{\widehat{\alpha }}={\mathrm{cov}}_{\ast },$$ (4.26)

so from (4.2),

$$\frac{d{s}_{\alpha }}{d\beta }{\mid }_{\widehat{\alpha }}={\widehat{V}}^{-1}{\mathrm{cov}}_{\ast }.$$ (4.27)

Since $\widehat{V}$ is the covariance matrix of ${\widehat{\beta }}^{\ast },$ that is, of $\widehat{\beta }$ under distribution $f_{\alpha =\widehat{\alpha }}$, (4.6) and (4.27) verify $\stackrel{\sim }{\mathrm{sd}}$ in (4.9) as the usual delta-method estimate of standard deviation for s( $\widehat{\beta }$).

Theorem 1 and Corollary 1 can be thought of as special cases of the exponential family theory in this section. The multinomial distribution of Y* (3.12) plays the role of $f_{\widehat{\alpha }}({\widehat{\beta }}^{\ast });\widehat{V}$ in (4.9) becomes $I-1_n{1^{\prime }}_n/n$ (3.3), so that (4.9) becomes (3.4). A technical difference is that the Mult_n(n,p) family (3.13) is singular (that is, concentrated on a n − 1-dimensional subspace of ℛⁿ), making the influence-function argument a little more involved than the parametric delta-function calculations. More seriously, the dimension of the nonparametric multinomial distribution increases with n, while for example, the parametric “Supernova” example of the next section has dimension 10 no matter how many supernovas might be observed. The more elaborate parametric confidence interval calculations of Section 6 failed when adapted for the nonparametric Cholesterol analysis, perhaps because of the comparatively high dimension, 164 versus 10.

5 The Supernova data

Figure 6 concerns a second example we will use to illustrate the parametric bootstrap theory of the previous section, the Supernova data: the absolute magnitude y_i has been determined for n = 39 Type Ia supernovas, yielding the data

Figure 6. The Supernova data.

Absolute magnitudes of n = 39 Type Ia supernovas plotted versus their OLS estimates from the full linear model (5.3); adjusted R² (5.5) equals 0.69.

$$\mathit{y}={(y_1,y_2,\dots ,y_n)}^{\prime }.$$ (5.1)

Each supernova has also had observed a vector of spectral energies x_i measured at p = 10 frequencies,

$${\mathit{x}}_i(=x_{i1},x_{i2},\dots ,x_{i10})$$ (5.2)

for supernova i. The 39 × 10 covariate matrix X, having x_i as its ith row, will be regarded as fixed.

We assume a standard normal linear regression model

$$\mathit{y}=X\alpha +\epsilon ,\epsilon \sim N_{39}(\mathit{O},I),$$ (5.3)

referred to as the full model in what follows. (For convenient discussion, the y_i have been rescaled to make (5.3) appropriate.) It has exponential family form (4.1), p = 10, with natural parameter $\alpha ,\widehat{\beta }=X^{\prime }\mathit{y},$ and $\psi ={\alpha }^{\prime }{X}^{\prime }X\alpha /2.$

Then ${(X^{\prime }X)}^{-1}\widehat{\beta }=\widehat{\alpha },$ the MLE of α, which also equals ${\widehat{\alpha }}_{\mathrm{OLS}}$, the ordinary least squares estimate of α in (5.3), yielding the full-model vector of supernova brightness estimates

$${\widehat{\mathit{\mu }}}_{\mathrm{OLS}}=X{\widehat{\alpha }}_{\mathrm{OLS}}.$$ (5.4)

Figure 6 plots y_i versus its estimate ${\widehat{\mu }}_{\mathrm{OLS},i.}$ The fit looks good, having an unadjusted R² of 0.82. Adjusting for the fact that we have used m = 10 parameters to fit n = 39 data points yields the more realistic value

$$R_{\mathrm{adj}}^2=R^2-2\cdot (1-R^2)\frac{m}{n-m}=0.69;$$ (5.5)

see Remark D.

Type Ia supernovas were used as “standard candles” in the discovery of dark energy and the cosmological expansion of the universe (Perlmutter et al., 1999; Riess et al., 1998). Their standardness assumes a constant absolute magnitude. This is not exactly true, and in practice regression adjustments are made. Our 39 supernovas were close enough to Earth to have their absolute magnitudes ascertained independently. The spectral measurements x, however, can be made for distant Type Ia supernovas, where independent methods fail, the scientific goal being a more accurate estimation function $\widehat{\mu }(\mathit{x})$ for their absolute magnitudes, and improved calibration of cosmic expansion.

We will use the Lasso (Tibshirani, 1996) to select $\widehat{\mu }(x)$. For a given choice of the non-negative “tuning parameter” λ we estimate α by the Lasso criterion

$${\widehat{\alpha }}_{\lambda }=\arg \underset{\alpha }{\min }\left\{{\parallel \mathit{y}-X\alpha \parallel }^2+\lambda {\displaystyle \sum _{k=1}^p\mid {\alpha }_k\mid }\right\};$$ (5.6)

${\widehat{\alpha }}_{\lambda }$ shrinks the components of $\widehat{\alpha }$_OLS toward zero, some of them all the way. As λ decreases from infinity to 0, the number m of non-zero components of ${\widehat{\alpha }}_{\lambda }$ increases from 0 to p. Conveniently enough, it turns out that m also nearly equals the effective degrees of freedom for the selection of ${\widehat{\alpha }}_{\lambda }$ (Efron, Hastie, Johnstone and Tibshirani, 2004). In what follows we will write ${\widehat{\alpha }}_m$ rather than ${\widehat{\alpha }}_{\lambda }$.

Table 4 shows a portion of the Lasso calculations for the Supernova data. Its last column gives $R_{\mathrm{adj}}^2$ (5.5) with R² having the usual form

Table 4.

Lasso model selection for the Supernova data. As the regularization parameter λ in (5.6) decreases from infinity to zero, the number m of non-zero coordinates of ${\widehat{\alpha }}_m$ increases from 0 to 10. The choice m = 7 maximizes the adjusted R² value (5.7), making it the selected model.

λ	m	R2	R2adj
∞	0	0	0
63	1	.17	.12
19.3	3	.74	.70
8.2	5	.79	.73
.496	7	.82	.735	(selected)
.039	9	.82	.71
0	10	.82	.69	(OLS)

$$R^2=1-\frac{{\parallel \mathit{y}-{\widehat{\mathit{\mu }}}_m\parallel }^2}{{\parallel \mathit{y}-\stackrel{-}{y}1\parallel }^2}\left({\dot{\widehat{\mathit{\mu }}}}_m=X{\widehat{\alpha }}_m,\stackrel{-}{y}={\displaystyle \sum y_i/n}\right).$$ (5.7)

The choice $\widehat{m}$ = 7 maximizes $R_{\mathrm{adj}}^2$,

$$\widehat{m}=\arg \underset{m}{\max }\left\{R_{\mathrm{adj}}^2\right\},$$ (5.8)

yielding our selected coefficient vector ${\widehat{\alpha }}_{\widehat{m}}$ and the corresponding vector of supernova estimates

$$\widehat{\mathit{\mu }}=X{\widehat{\alpha }}_{\widehat{m}};$$ (5.9)

note that ${\widehat{\alpha }}_{\widehat{m}}$ is not an OLS estimate.

B = 4000 bootstrap replications ${\widehat{\mathit{\mu }}}^{\ast }$ were computed (again many more than were actually needed): bootstrap samples y^* were drawn using the full OLS model,

$${\mathit{y}}^{\ast }\sim N_{39}({\widehat{\mathit{\mu }}}_{\mathrm{OLS}},\mathit{I});$$ (5.10)

see Remark E. The equivalent of Table 4, now based on data y^*, was calculated, the $R_{\mathrm{adj}}^2$ maximizer ${\widehat{m}}^{\ast }$ and ${\widehat{\alpha }}_{{\widehat{m}}^{\ast }}^{\ast }$ selected, giving

$${\widehat{\mathit{\mu }}}^{\ast }=X{\widehat{\alpha }}_{{\widehat{m}}^{\ast }}^{\ast }.$$ (5.11)

Averaging the 4000 ${\widehat{\mathit{\mu }}}^{\ast }$ vectors yielded the smoothed vector estimates

$$\tilde{\mathit{\mu }}={\displaystyle \sum _{i=1}^B{\widehat{\mathit{\mu }}}_i^{\ast }/B.}$$ (5.12)

Standard deviations ${\stackrel{\sim }{\mathrm{sd}}}_{Bj}$ for supernova j’s smoothed estimate ${\stackrel{\sim }{\mu }}_j$ were then calculated according to (4.15), j = 1,2,…,39 The ratio of standard deviations for ${\stackrel{\sim }{\mathrm{sd}}}_B/{\widehat{\mathrm{sd}}}_B$ each of the 39 supernovas, ranged from 0.87 to 0.98, with an average of 0.93. Jackknife calculations (3.11) showed that B = 800 would have been enough for good accuracy.

At this point it pays to remember that ${\stackrel{\sim }{\mathrm{sd}}}_B$ is a delta-method shortcut version of a full bootstrap standard deviation for the smoothed estimator s(y). We would prefer the latter if not for the computational burden of a second level of bootstrapping. As a check, a full second-level simulation was run, beginning with simulated data vectors ${\mathit{y}}^{\ast }\sim N_{39}({\widehat{\mathit{\mu }}}_{\mathrm{OLS}},I)$ (5.10), and for each y* carrying through calculations of s* and ${\stackrel{\sim }{\mathrm{sd}}}_B^{\ast }$ based on B = 1000 second-level bootstraps. This was done 500 times, yielding 500 values $s_k^{\ast }$ for each of the 39 supernovas, which provided direct bootstrap estimates say ${\stackrel{\sim }{\mathrm{sd}}}_k$ for s_k. The ${\stackrel{\sim }{\mathrm{sd}}}_k$ values averaged about 7.5% larger than the delta-method approximations ${\stackrel{\sim }{\mathrm{sd}}}_{Bk}$ Taking this into account, the reductions in standard deviation due to smoothing were actually quite small, the ratios averaging about 98%; see the end of Remark H.

Returning to the original calculations, model selection was highly variable among the 4000 bootstrap replications. Table 5 shows the percentage of the 4000 replications that selected m non-zero coefficients for ${\widehat{\alpha }}^{\ast }$ in (5.11), m = 1, 2,…, 10, with the original choice m = 7 not quite being modal. Several of the supernovas showed effects like that in Figure 3.

Table 5.

Percentage of the 4000 bootstrap replications selecting m non-zero coefficients for ${\widehat{\alpha }}^{\ast }$ in (5.11), m = 1, 2,…,10. The original choice m = 7 is not quite modal.

m	1	2	3	4	5	6	7	8	9	10
%	0	1	8	13	16	18	18	14	9	2

Model averaging, that is bootstrap smoothing, still has important confidence interval effects even though here it does not substantially reduce standard deviations. This is shown in Figure 7 of the next section, which displays approximate 95% confidence intervals for the 39 supernova magnitudes.

Figure 7.

Approximate 95% confidence limits for the 39 supernova magnitudes μ_k (after subtraction of smoothed estimates ${\tilde{\mu }}_k$ (5.12)); ABC intervals (solid) compared with smoothed standard intervals ${\tilde{\mu }}_k\pm 1.96{\stackrel{\sim }{\mathrm{sd}}}_k$ (dashed). Crosses indicate differences between unsmoothed and smoothed estimates, (5.9) minus (5.12).

Other approaches to bootstrapping Lasso estimates are possible. Chatterjee and Lahiri (2011), referring back to work by Knight and Fu (2000), resample regression residuals rather than using the full parametric bootstrap (5.10). The “m out of n” bootstrap is featured in Hall, Lee and Park (2009). Asymptotic performance, mostly absent here, is a central concern of these papers; also, they focus on estimation of the regression coefficients, in (5.3), a more difficult task than estimating μ = Xα.

6 Better bootstrap confidence intervals

The central tactic of this paper is the use of bootstrap smoothing to convert an erratically behaved model selection-based estimator $t(\cdot )$ into a smoothly varying version $s(\cdot )$ Smoothing makes the good asymptotic properties of the bootstrap, as extensively developed in Hall (1992), more credible for actual applications. This section carries the smoothing theme further, showing how $s(\cdot )$ can be used to form second-order accurate intervals.

The improved confidence intervals depend on the properties of bootstrap samples from exponential families (4.1). We define an “empirical exponential family” ${\widehat{f}}_{\alpha }(\cdot )$ that puts probability

$${\widehat{f}}_{\alpha }({\widehat{\beta }}_i^{\ast })=e^{{(\alpha -\widehat{\alpha })}^{\prime }{\widehat{\beta }}_i^{\ast }-\widehat{\psi }(\alpha )}\frac{1}{B}$$ (6.1)

on bootstrap replication ${\widehat{\beta }}_i^{\ast }$ (4.3) for i = 1, 2,…, B, where

$$\widehat{\psi }(\alpha )=\log \left({\displaystyle \sum _{j=1}^B{e}^{{(\alpha -\widehat{\alpha })}^{\prime }{\widehat{\beta }}_i^{\ast }}}/B\right).$$ (6.2)

Here $\widehat{\alpha }$ is the MLE of in the original family (4.1), $\widehat{\alpha }={\lambda }^{-1}(\widehat{\beta })$ in the notation following (4.2)

The choice of $\alpha =\widehat{\alpha }$ makes ${\widehat{f}}_{\widehat{\alpha }}({\widehat{\beta }}_i^{\ast })=1/B$ for i = 1, 2,…, B; in other words, it yields the empirical probability distribution of the bootstrap sample (4.3) in ℛ^p. Other choices of “tilt” the empirical distribution in direction $\alpha =\widehat{\alpha }$; (6.1) is a direct analogue of the original exponential family (4.1), which can be re-expressed as

$$f_{\alpha }\left({\widehat{\beta }}^{\ast }\right)=e^{{(\alpha -\widehat{\alpha })}^{\prime }{\widehat{\beta }}^{\ast }-(\psi (\alpha )-\psi (\widehat{\alpha }))}{f}_{\widehat{\alpha }}\left({\widehat{\beta }}^{\ast }\right),$$ (6.3)

now with $\widehat{\alpha }$ fixed and ${\widehat{\beta }}^{\ast }$ the random variable. Notice that $\widehat{\psi }\mid (\widehat{\alpha })=0$ in (6.2). Taking this into account, the only di erence between the original family (6.3) and the empirical family (6.1) is the change in support, from ${\widehat{f}}_{\alpha }(\cdot )$ to the empirical probability distribution. Under mild regularity conditions, family ${\widehat{f}}_{\alpha }(\cdot )$ approaches ${\widehat{f}}_{\alpha }(\cdot )$ as the bootstrap sample size B goes to infinity.

As in (4.23)–(4.24), let s_α be the value of the smoothed statistic we would get if bootstrap samples were obtained from f_α rather than $f_{\widehat{\alpha }}$. We can estimate s_α from the original bootstrap samples (4.3) by importance sampling in family (4.1),

$$\begin{array}{lll}{s}_{\alpha }& =& {\displaystyle \sum _{i=1}^B{e}^{{(\alpha -\widehat{\alpha })}^{\prime }{\widehat{\beta }}_i^{\ast }}{t}_i^{\ast }}{\displaystyle /}{\displaystyle \sum _{i=1}^B{e}^{{(\alpha -\widehat{\alpha })}^{\prime }{\widehat{\beta }}_i^{\ast }}}\hfill \\ & =& {\displaystyle \sum _{i=1}^B{\widehat{f}}_{\alpha }({\widehat{\beta }}_i^{\ast })t_i^{\ast }}\hfill \end{array}$$ (6.4)

without requiring any further evaluations of $t(\cdot )$. (Note that ${\widehat{f}}_{\alpha }({\widehat{\beta }}_i^{\ast })$ is proportional to w_i in (4.24).) The main point here is that the smoothed estimate s_α is the expectation of the values $t_i^{\ast },i=1,2,\dots ,B,$ taken with respect to the empirical exponential family (6.1).

A system of approximate confidence intervals enjoys second-order accuracy if its coverage probabilities approach the target value with errors 1/n in the sample size n, rather than at the slower rate $1/\sqrt{n}$ of the standard intervals. The ABC system (“approximate bootstrap confidence” intervals, DiCiccio and Efron, 1992, not to be confused with “approximate Bayesian computation” as in Fearnhead and Prangle, 2012) employs numerical derivatives to produce second-order accurate intervals in exponential families. Its original purpose was to eliminate the need for bootstrap resampling. Here, though, we will apply it to the smoothed statistic $s(\widehat{\beta })=\sum t({\widehat{\beta }}_i^{\ast })/B$ (4.5) in order to avoid a second level of bootstrapping. This is a legitimate use of ABC because we are working in an exponential family, albeit the empirical family (6.1).

Three corrections are needed to improve the smoothed standard interval (2.11) from first- to secondorder accuracy: a non-normality correction obtained from the bootstrap distribution, an acceleration correction of the type mentioned at (3.22), and a bias-correction. ABC carries these out via p+2 numerical second derivatives of ${\widehat{S}}_{\alpha }$ in (6.4), taken at $\alpha =\widehat{\alpha }$, as detailed in Section 2 of DiCiccio and Efron (1992). The computational burden is effectively nil compared with the original bootstrap calculations (4.3).

Figure 7 compares the ABC 95% limits for the supernova brightnesses μ_k,k = 1, 2, . . . , 39, solid lines, with parametric smoothed standard intervals (2.11), dashed lines. (The smoothed estimates ${\stackrel{\sim }{\mu }}_k$ (5.12) have been subtracted from the endpoints in order to put all the intervals on the same display.) There are a few noticeable discrepancies, for supernovas 2, 6, 25, and 27 in particular, but overall the smoothed standard intervals hold up reasonably well.

Smoothing has a moderate effect on the Supernova estimates, as indicated by the values of ${\widehat{\mu }}_k-{\stackrel{\sim }{\mu }}_k$, (5.11) minus (5.12), the crosses in Figure 7. A few of the intervals would be much different if based on the unsmoothed estimates ${\widehat{\mu }}_k$, e.g., supernovas 1, 12, 17, and 28. Remark I says more about the ABC calculations.

As a check on the ABC intervals, the “full simulation” near the end of Section 4, with B = 1000 bootstrap replications for each of 500 trials, was repeated. For each trial, the 1000 bootstraps provided new ABC calculations, from which the “achieved significance level” ${\mathrm{asl}}_k^{\ast }$ of the original smoothed estimate ${\stackrel{\sim }{\mu }}_k$ (5.12) was computed: that is,

$${\mathrm{asl}}_k^{\ast }=\text{bootstrap ABC confidence level for}\left(-\infty ,{\stackrel{\sim }{\mu }}_k\right)$$ (6.5)

If the ABC construction were working perfectly, ${\mathrm{asl}}_k^{\ast }$ would have a uniform distribution,

$${\mathrm{asl}}_k^{\ast }\sim U\left(0,1\right)$$ (6.6)

for k = 1; 2,…, 39.

Table 6 displays quantiles of ${\mathrm{asl}}_k^{\ast }$ in the 500 trials, for seven of the 39 supernovas, k =5, 10, 15, 20, 25, 30, and 35. The results are not perfectly uniform, showing for instance a moderate deficiency of small ${\mathrm{asl}}_k^{\ast }$ values for k = 5, but overall the results are encouraging. A U (0, 1) random variable has mean 0.500 and standard deviation0.289, while all 3500 ${\mathrm{asl}}_k^{\ast }$ values in Table 6 had mean 0.504 and standard deviation 0.284.

Table 6.

Simulation check for ABC intervals; 500 trials, each with B = 1000 bootstrap replications. Columns show quantiles of achieved significance levels ${\mathrm{asl}}_k^{\ast }$ (6.5) for supernovas k = 5, 10,…,35; last column for all seven supernovas combined. It is a reasonable match to the ideal uniform distribution (6.6).

quantile	SN5	SN10	SN15	SN20	SN25	SN30	SN35	ALL
0.025	0.04	0.02	0.04	0.00	0.04	0.03	0.02	0.025
0.05	0.08	0.04	0.06	0.04	0.08	0.06	0.06	0.055
0.1	0.13	0.08	0.11	0.10	0.12	0.10	0.12	0.105
0.16	0.20	0.17	0.18	0.16	0.18	0.18	0.18	0.175
0.5	0.55	0.50	0.54	0.48	0.50	0.48	0.50	0.505
0.84	0.84	0.82	0.82	0.84	0.84	0.84	0.84	0.835
0.9	0.90	0.88	0.90	0.88	0.90	0.90	0.90	0.895
0.95	0.96	0.94	0.96	0.94	0.94	0.94	0.94	0.945
0.975	0.98	0.97	0.98	0.98	0.96	0.98	0.97	0.975

The ABC computations are local, in the sense that the importance sampling estimates $s_{\alpha }$ in (6.4) need only be evaluated for very near $\widehat{\alpha }$. This avoids the familiar peril of importance sampling, that the sampling weights in (6.4) or (4.1) may vary uncontrollably in size.

If one is willing to ignore the peril, full bootstrap standard errors for the smoothed estimates $\stackrel{\sim }{\mu }$ (4.5), rather than the delta-method estimates of Theorem 2, become feasible: in addition to the original parametric bootstrap samples (4.3), we draw J more times, say

$$f_{\widehat{\alpha }}(·)\to {\stackrel{\sim }{\beta }}_1^{\ast },{\stackrel{\sim }{\beta }}_2^{\ast },\dots ,{\stackrel{\sim }{\beta }}_J^{\ast },$$ (6.7)

and compute the corresponding natural parameter estimates ${\stackrel{\sim }{\alpha }}_j^{\ast }={\lambda }^{-1}\left({\stackrel{\sim }{\beta }}_j^{\ast }\right),$ as following (4.2). Each ${\stackrel{\sim }{\alpha }}_j^{\ast }$ gives a bootstrap version of the smoothed statistic $s_{{\stackrel{\sim }{\alpha }}_j^{\ast }}$, using (6.4), from which we calculate the usual bootstrap standard error estimate,

$${\stackrel{\sim }{\mathrm{sd}}}_{\mathrm{boot}}={\left[{\displaystyle \sum _{j=1}^J{\left(s_{{\stackrel{\sim }{\alpha }}_j^{\ast }}-s.\right)}^2}/\left(J-1\right)\right]}^{1/2},$$ (6.8)

Where $s.={\displaystyle \sum s_{{\stackrel{\sim }{\alpha }}_j^{\ast }}}/J$. Once again, no further evaluations of $t(·)$ beyond the original ones in (4.5) are required.

Carrying this out for the Supernova data gave standard errors ${\stackrel{\sim }{\mathrm{sd}}}_{\mathrm{boot}}$ a little smaller than those from Theorem 2, as opposed to the somewhat larger ones found by the full simulation near the end of Section 5. Occasional very large importance sampling weights in (6.4) did seem to be a problem here.

Compromises between the delta method and full bootstrapping are possible. For the normal model (5.3) we have ${\stackrel{\sim }{\beta }}_j^{\ast }\sim N\left(\widehat{\beta },X^{\prime }X\right)$ in (6.7). Instead we might take

$${\widehat{\beta }}_j^{\ast }\sim N\left(\widehat{\beta },c{X}^{\prime }X\right)$$ (6.9)

with c less than 1, placing ${\stackrel{\sim }{\alpha }}_j^{\ast }$ nearer $\widehat{\alpha }$. Then (6.8) must be multiplied by $1/\sqrt{c}$. Doing this with c = 1/9 gave standard error estimates almost the same as those from Theorem 2.

7 Remarks, details, and proofs

This section expands on points raised in the previous discussion.

A. Proof of Corollary 1

With ${\mathit{Y}}^{\ast }=\left(Y_{ij}^{\ast }\right)$ as in (3.2), let $\mathit{X}={\mathit{Y}}^{\ast }-1_B{1^{\prime }}_n=\left(Y_{ij}^{\ast }-1\right)$. For the ideal bootstrap, B = nⁿ,

$${\mathit{X}}^{\prime }\mathit{X}/B=\mathit{I}-{1^{\prime }}_n{1}_n.$$ (7.1)

the multinomial covariance matrix in (3.3). This has (n − 1) non-zero eigenvalues all equaling 1, implying that the singular value decomposition of X is

$$\mathit{X}=\sqrt{B}\mathit{L}{\mathit{R}}^{\prime },$$ (7.2)

Land R orthonormal matrices of dimensions B × (n −1) and n × (n −1). Then the B-vector ${\mathit{U}}^{\ast }=\left(t_i^{\ast }-s_0\right)$ has projected squared length into ℒ(X)

$$\begin{array}{ccc}\hfill {{\mathit{U}}^{\ast }}^{\prime }\mathit{L}{\mathit{L}}^{\prime }{\mathit{U}}^{\ast }& \hfill =\hfill & B{{\mathit{U}}^{\ast }}^{\prime }\frac{\mathit{L}\sqrt{B}{\mathit{R}}^{\prime }\mathit{R}\sqrt{B}{\mathit{L}}^{\prime }}{B^2}{\mathit{U}}^{\ast }\hfill \\ \hfill & \hfill =\hfill & B\left({{\mathit{U}}^{\ast }}^{\prime }\mathit{X}/B\right)\left({\mathit{X}}^{\prime }{\mathit{U}}^{\ast }/B\right)=B{\stackrel{\sim }{\mathrm{sd}}}^2,\hfill \end{array}$$ (7.3)

verifying (3.4).

B. Hájek projection and ANOVA decomposition

For the ideal nonparametric bootstrap of Section 3, define the conditional bootstrap expectations

$$e_j=E_{\ast }\left\{t\left({\mathit{y}}_i^{\ast }\right)\mid y_{ij}^{\ast }=y_j\right\},$$ (7.4)

j = 1, 2, … , n (not depending on k). The bootstrap ANOVA decomposition of Efron (1983, Sect. 7) can be used to derive an orthogonal decomposition of t(y*),

$$t\left({\mathit{y}}_i^{\ast }\right)=s_0+L_i^{\ast }+R_i^{\ast }$$ (7.5)

where $s_0=E_{\ast }\left\{t\left({\mathit{y}}^{\ast }\right)\right\}$ is the ideal smoothed bootstrap estimate, and

$$L_i^{\ast }={\displaystyle \sum _{j=1}^n{Y}_{ij}^{\ast }\left(e_j-s_0\right)},$$ (7.6)

while $R_i^{\ast }$ involves higher-order ANOVA terms such as e_jl− e_j− e_l + s₀ with

$$e_{\mathit{jl}}=E_{\ast }\left\{t_i\left({\mathit{y}}^{\ast }\right)\mid y_{\mathit{ik}}^{\ast }=y_j\mathrm{and}{y}_{\mathit{im}}^{\ast }=y_k\right\}.$$ (7.7)

The terms in (7.5) satisfy $E_{\ast }\left\{L^{\ast }\right\}=E_{\ast }\left\{R^{\ast }\right\}=0$ and are orthogonal, $E_{\ast }\left\{L^{\ast }{R}^{\ast }\right\}=0.$ The bootstrap Hájek projection of t(y*) (Hájek, 1968) is then the first two terms of (7.5), say

$$H_i^{\ast }=s_0+L_i^{\ast }.$$ (7.8)

Moreover,

$$L_i^{\ast }={\displaystyle \sum _{j=1}^n{Y}_{ij}^{\ast }{\mathrm{cov}}_j}$$ (7.9)

from (3.5) and the ratio of smoothed-to-unsmoothed standard deviation (3.10) equals

$${\stackrel{\sim }{\mathrm{sd}}}_B/{\widehat{\mathrm{sd}}}_B={\left[{\mathrm{var}}_{\ast }\left\{L_i^{\ast }\right\}/\left({\mathrm{var}}_{\ast }\left\{L_i^{\ast }\right\}+{\mathrm{var}}_{\ast }\left\{R_i^{\ast }\right\}\right)\right]}^{1/2}.$$ (7.10)

C. Nonparametric bias estimate

There is a nonparametric bias estimate ${\stackrel{\sim }{\mathrm{bias}}}_B$ for the smoothed statistic s(y) (2.8) corresponding to the variability estimate ${\stackrel{\sim }{\mathrm{sd}}}_B$. In terms of T(Y*) and S(p) (3.13)–(3.14), the nonparametric delta method gives

$${\stackrel{\sim }{\mathrm{bias}}}_B=\frac{1}{2}{\displaystyle \sum _{j=1}^n\frac{{\ddot{S}}_j}{n^2}}$$ (7.11)

where ${\ddot{S}}_j$ is the second-order influence value

$${\ddot{S}}_j=\underset{\epsilon \to 0}{\lim }\frac{S\left({\mathit{p}}_0+\epsilon \left({\mathit{\delta }}_j-{\mathit{p}}_0\right)\right)-2S\left({\mathit{p}}_0\right)+S\left({\mathit{p}}_0-\epsilon \left({\mathit{\delta }}_j-{\mathit{p}}_0\right)\right)}{{\epsilon }^2}.$$ (7.12)

See Section 6.6 of Efron (1982).

Without going into details, the Taylor series calculation (3.18)–(3.19) can be carried out one step further, leading to the following result:

$${\stackrel{\sim }{\mathrm{bias}}}_B={\mathrm{cov}}_{\ast }\left(D_i^{\ast },t_i^{\ast }\right)$$ (7.13)

where $D_i^{\ast }={{\displaystyle {\sum }_1^n\left(Y_{\mathit{ij}}^{\ast }-1\right)}}^2.$

This looks like a promising extension of Theorem 1 (3.4)–(3.5). Unfortunately, (7.13) proved unstable when applied to the Cholesterol data, as revealed by jackknife calculations like (3.11). Things are better in parametric settings; see Remark I. There is also some question of what “bias” means with model selection-based estimators; see Remark G.

D. Adjusted R²

Formula (5.5) for $R_{\mathrm{adj}}^{\ast }$, not the usual definition, is motivated by OLS estimation and prediction in a homoskedastic model. We observe

$$\mathit{y}\sim \left(\mathit{\mu },{\sigma }^2\mathit{I}\right)$$ (7.14)

and estimate μ by $\widehat{\mathit{\mu }}=\mathit{My}$, where the n × n symmetric matrix M is idempotent, M² = M. Then ${\sigma }^2={\Vert \mathit{y}-\widehat{\mathit{\mu }}\Vert }^2/\left(n-m\right)$, m the rank of M, is the usual unbiased estimate of σ². Letting y° indicate an independent new copy of y, the expected prediction error of $\widehat{\mathit{\mu }}$ is

$$E\left\{{\Vert \mathit{y}°-\widehat{\mathit{\mu }}\Vert }^2\right\}=E\left\{{\Vert \mathit{y}-\widehat{\mathit{\mu }}\Vert }^2+2m{\widehat{\sigma }}^2\right\}$$ (7.15)

as in (2.6). Finally, the usual definition of R²,

$$R^2={1-{\Vert \mathit{y}-\widehat{\mathit{\mu }}\Vert }^2/\Vert \mathit{y}-\stackrel{\_}{y}1\Vert }^2$$ (7.16)

is adjusted by adding the amount suggested in (7.15),

$$R_{\mathrm{adj}}^2={1-\left\{{\Vert \mathit{y}-\widehat{\mathit{\mu }}\Vert }^2+2m{\widehat{\sigma }}^2\right\}/\Vert \mathit{y}-\stackrel{\_}{y}1\Vert }^2,$$ (7.17)

and this reduces to (5.5).

E. Full-model bootstrapping

The bootstrap replications (5.10) are drawn from the full model, ${\mathit{y}}^{\ast }\sim N_{39}\left({\widehat{\mathit{\mu }}}_{\mathrm{OLS}},\mathit{I}\right),$ rather than say the smoothed Lasso choice (5.12), ${\mathit{y}}^{\ast }\sim N_{39}\left(\stackrel{\sim }{\mathit{\mu }},\mathit{I}\right)$. This follows the general development in Section 4(4.3) and, less obviously, the theory of Sections 2 and 3, where the “full model” is the usual nonparametric one (2.3).

An elementary example, based on Section 10.6 of Hjort and Claeskens (2003), illustrates the dangers of bootstrapping from other than the full model. We observe $y\sim \mathcal{N}\left(\mu ,1\right)$, with MLE $\widehat{\mu }$ = t(y) = y, and consider estimating μ with the shrunken estimator $\widehat{\mu }$ = s(y) = cy, where c is a fixed constant 0 < c < 1, so

$$\stackrel{\sim }{\mu }\sim \mathcal{N}\left(c\mu ,c^2\right).$$ (7.18)

Full-model bootstrapping corresponds to $y^{\ast }\sim \mathcal{N}\left(\widehat{\mu },1\right)$, and yields ${\stackrel{\sim }{\mu }}^{\ast }={\mathit{cy}}^{\ast }\sim \mathcal{N}\left(c\mu ,c^2\right).$ as the bootstrap distribution. However the “model-selected bootstrap” $y^{\ast }\sim \mathcal{N}\left(\stackrel{\sim }{\mu },1\right)$ yields

$${\stackrel{\sim }{\mu }}^{\ast }\sim \mathcal{N}\left(c^2\widehat{\mu },c^2\right),$$ (7.19)

squaring the amount of shrinkage in (7.18).

Returning to the Supernova example, the Lasso is itself a shrinkage technique. Bootstrapping from the Lasso choice $\stackrel{\sim }{\mathit{\mu }}$ would shrink twice, perhaps setting many more of the coordinate estimates to zero.

F. Bias of the smoothed estimate

There is a simple asymptotic expression for the bias of the bootstrap smoothed estimator in exponential families, following DiCiccio and Efron (1992). The schematic diagram of Figure 8 shows the main elements: the observed vector y, expectation μ, generates the bootstrap distribution of y*, indicated by the dashed ellipses. A parameter of interest θ = t(μ) has MLE $\widehat{\theta }$ = t(y). Isoplaths of constant value for t(⋅) are indicated by the solid curves in Figure 8.

Figure 8.

Schematic diagram of large-sample bootstrap estimation. Observed vector y has expectation μ. Ellipses indicate bootstrap distribution of y* given $\widehat{\mathit{\mu }}=\mathit{y}$. Parameter of interest θ = t(μ) is estimated by $\widehat{\theta }$ = t(y). Solid curves indicate surfaces of constant value of t(⋅).

The asymptotic mean and variance of the MLE $\widehat{\theta }$ = t(y) as sample size n grows large is of the form

$$\widehat{\theta }\sim \left(\theta +\frac{b\left(\mathit{\mu }\right)}{n},\frac{c^2\left(\mathit{\mu }\right)}{n}\right)+O_p\left(n^{-3/2}\right).$$ (7.20)

Here the bias b(μ)/n is determined by the curvature of the level surfaces near μ. Then it is not difficult to show that the ideal smoothed bootstrap estimate $\stackrel{\sim }{\theta }={\displaystyle \sum t\left({\mathit{y}}_i^{\ast }\right)/B,B\to \infty }$, has mean and variance

$$\stackrel{\sim }{\theta }\sim \left(\theta +2\frac{b\left(\mathit{\mu }\right)}{n},\frac{c^2\left(\mathit{\mu }\right)}{n}\right)+O_p\left(n^{-3/2}\right).$$ (7.21)

So smoothing doubles the bias without changing variance. This just says that smoothing cannot improve on the MLE $\widehat{\theta }$ in the already smooth asymptotic estimation context of Figure 8.

G. Two types of bias

The term b(μ)/n in (7.20) represents “statistical bias,” the difference between the expected value of t( $\widehat{\mathit{\mu }}$) and t(μ). Model-selection estimators also involve “definitional bias”: we wish to estimate θ = T(μ), but for reasons of robustness or efficiency we employ a different functional $\widehat{\theta }$ = t(y), a homely example being the use of a trimmed mean to estimate an expectation. The ABC bias correction mentioned in Section 6 is correcting the smoothed standard interval $\stackrel{\sim }{\mu }\pm 1.96{\stackrel{\sim }{\mathrm{se}}}_B$ for statistical bias. Definitional bias can be estimated by t(y)−T(y), but this is usually too noisy to be of help. Section 2 of Berk et al. (2012) makes this point nicely (see their discussion of “target estimation”) and I have followed their lead in not trying to account for definitional bias. See also Bühlmann and Yu (2002), Definition 1.2, for an asymptotic statement of what is being estimated by a model-selection procedure.

H. Selection-based estimation

The introduction of model selection into the estimation process disrupts the smooth properties seen in Figure 8. The wedge-shaped regions of Figure 9 indicate different model choices, e.g., linear, quadratic, cubic, etc. regressions for the Cholesterol data. Now the surfaces of constant estimation jump discontinuously as y crosses regional boundaries. Asymptotic properties such as (7.20)–(7.21) are less convincing when the local geometry near the observed y can change abruptly a short distance away.

Figure 9.

Estimation after model selection. The regions indicate different model choices. Now the curves of constant estimation jump discontinuously as y crosses regional boundaries.

The bootstrap ellipses in Figure 9 are at least qualitatively correct for the Cholesterol and Supernova examples, since in both cases a wide bootstrap variety of regions were selected. In this paper, the main purpose of bootstrap smoothing is to put us back into Figure 8, where for example the standard intervals (2.11) are more believable. (Note: Lasso estimates are continuous, though non-differentiable, across region boundaries, giving a picture somewhere between Figure 8 and Figure 9. This might help explain the smooth estimators’ relatively modest reductions in standard error for the Supernova analysis.)

Bagging amounts to replacing the discontinuous isoplaths of θ = t(μ) with smooth ones, say for θ_bag = s(μ). The standard deviations and approximate confidence intervals of this paper apply to θ_bag, ignoring the possible definitional bias.

I. The ABC intervals

The approximate bootstrap confidence limits in Figure 7 were obtained using the ABC_q algorithm, as explained in detail in Section 2 of DiCiccio and Efron (1992). In addition to the acceleration a and bias-correction constant z₀, ABC_q also calculates c_q: in a one-parameter exponential family (4.1), c_q measures the nonlinearity of the parameter of interest θ= t(β) as a function of β, with a similar definition applying in p dimensions. The algorithm involves the calculation of p + 2 numerical second derivatives of s_α (6.4) carried out at α = $\widehat{\alpha }$. Besides a, z₀, and c_q, ABC_q provides an estimate of statistical bias for s_α.

If (α, z₀, c_q) = (0, 0, 0) then the ABC_q intervals match the smoothed standard intervals (2.11). Otherwise, corrections are made in order to achieve second-order accuracy. For instance (a, z₀, c_q) = (0, −0.1, 0) shifts the standard intervals leftwards by 0.1− $\widehat{\sigma }$. For all three constants, values outside of ±0.1 can produce noticeable changes to the intervals.

Table 7 presents summary statistics of a, z₀, c_q, and bias for the 39 smoothed Supernova estimates ${\stackrel{\sim }{\mu }}_k$. The differences between the ABC_q and smoothed standard intervals seen in Figure 7 were primarily due to z₀.

Table 7.

Summary statistics of the ABC_q constants for the 39 smoothed Supernova estimates ${\stackrel{\sim }{\mu }}_k$ (5.12).

	a	z0	cq	bias
mean	.00	.00	.00	.00
stdev	.01	.13	.04	.06
Lowest	–.01	–.21	–.07	–.14
highest	.01	.27	.09	.12

J. Bias correction for ${\stackrel{\sim }{\mathrm{sd}}}_B$

The nonparametric standard deviation estimate ${\stackrel{\sim }{\mathrm{sd}}}_B$ (3.7) is biased upward for the ideal value $\stackrel{\sim }{\mathrm{sd}}$ (3.4), but it is easy to make a correction. Using notation (3.3)– (3.9), define

$$Z_{ij}^{\ast }=\left(Y_{ij}^{\ast }-1\right)\left(t_i^{\ast }-s_0\right).$$ (7.22)

Then $Z_{ij}^{\ast }$ has bootstrap mean cov_j (3.5) and bootstrap variance say ${\mathrm{\Delta }}_j^2$. A sample of B bootstrap replications yields bootstrap moments

$${\widehat{\mathrm{cov}}}_j=\frac{1}{B}{\displaystyle \sum _{i=1}^B{Z}_{ij}^{\ast }{\sim }_{\ast }\left({\mathrm{cov}}_j,{\Delta }_j^2/B\right),}$$ (7.23)

so

$$E_{\ast }{\stackrel{\sim }{\mathrm{sd}}}_B^2={\stackrel{\sim }{\mathrm{sd}}}^2+\frac{1}{B}{\displaystyle \sum _{j=1}^n{\Delta }_j^2.}$$ (7.24)

Therefore the bias-corrected version of ${\stackrel{\sim }{\mathrm{sd}}}_B^2$ is

$${\stackrel{\sim }{\mathrm{sd}}}_B^2-\frac{1}{B^2}{\displaystyle \sum _{j=1}^n{\displaystyle \sum _{i=1}^B{\left(Z_{ij}^{\ast }-{\widehat{\mathrm{cov}}}_j\right)}^2.}}$$ (7.25)

K. Improved estimates of the bagged standard errors

The simulation experiment of Figure 5 can also be regarded as a two-level parametric bootstrap procedure, with the goal of better estimating $\mathrm{sd}\left({\stackrel{\sim }{\mu }}_k\right)$, the bagged estimates’ standard deviations for subjects k = 1, 2 , . . . , 11 in the Cholesterol study. Two possible estimates are shown: (1) the empirical standard deviation $\stackrel{\sim }{\mathrm{Sd}}$ (4.22), solid curve, and (2) the average $\stackrel{\sim }{\mathrm{sd}}$. of the 100 second-level ${\stackrel{\sim }{\mathrm{sd}}}_i$ values (4.15), dashed curve. There are two reasons to prefer the latter.

The first has to do with the sampling error of the standard deviation estimates themselves. This was about 10 times larger for $\stackrel{\sim }{\mathrm{Sd}}$ than $\stackrel{\sim }{\mathrm{sd}}$., e.g., 5.45 ± 0.35 compared to 5.84 ± 0.03 for subject 1. (Note: The two curves in Figure 5 do not di er significantly at any point.)

The second and more important reason has to do with the volitility of model-selection estimates and their standard errors. Let σ(β) denote the standard deviation of a bagged estimator $\widehat{\mu }$ in a parametric model such as (4.16)–(4.17). The unknown true parameter β₀ has yielded the observed value $\widehat{\beta }$ , and then bootstrap values ${\widehat{\beta }}_i^{\ast }$, i = 1, 2, . . . , 100, and second-level bootstraps ${\widehat{\beta }}_{ij}^{\ast }$, j = 1, 2, . . . , 1000. The estimate ${\stackrel{\sim }{\mathrm{sd}}}_{100}$ obtained from the ${\widehat{\beta }}_i^{\ast }$’s (4.15) is a good approximation to $\sigma \left(\widehat{\beta }\right)$. The trouble is that the functional σ(β) is itself volatile, so that $\sigma \left(\widehat{\beta }\right)$ may differ considerably from the “truth” σ(β₀).

This can be seen at the second level in Figure 5, where the dashes indicating ${\stackrel{\sim }{\mathrm{sd}}}_i$ values, i = 1, 2,…, 100, vary considerably. (This is not due to the limitations of using B = 1000 replications; the bootstrap “internal variance” component accounts for only about 30% of the spread.) Broadly speaking, ${\widehat{\beta }}_i^{\ast }$ values that fall close to a regime boundary, say separating the choice of “Cubic” from “Quartic,” had larger values of $\sigma \left({\widehat{\beta }}_i^{\ast }\right)\doteq {\stackrel{\sim }{\mathrm{sd}}}_i$.

The preferred estimate $\stackrel{\sim }{\mathrm{sd}}$ effectively averages $\sigma \left({\widehat{\beta }}_i^{\ast }\right)$ over the parametric choice of ${\widehat{\beta }}_i^{\ast }$ and $\widehat{\beta }$. Another way to say this is that $\stackrel{\sim }{\mathrm{sd}}$. is a flat-prior Bayesian estimate of $\sigma \left({\beta }_0\right)$,given the data $\widehat{\beta }$. See Efron (2012).

Of course $\stackrel{\sim }{\mathrm{sd}}$.requires much more computation than ${\stackrel{\sim }{\mathrm{sd}}}_B$ (4.15). Our 100 × 1000 analysis could be reduced to 50 × 500 without bad effect, but that is still 25000 resamples. In fact, $\widetilde{\mathrm{sd}}$. was not much different from ${\stackrel{\sim }{\mathrm{sd}}}_B$ in this example. The difference was larger in the nonparametric version of Figure 5, which showed substantially greater bias and variability, making the second level of bootstrapping more worthwhile.