Estimation in Partially Linear Models and Numerical Comparisons

Abstract

Partially linear models with local kernel regression are popular non-parametric techniques. However, bandwidth selection in the models is a puzzling topic that has been addressed in literature with the use of undersmoothing and regular smoothing. In an attempt to address the strategy of bandwidth selection, we review profile-kernel based and backfitting methods for partially linear models, and justify why undersmoothing is necessary for backfitting method and why the “optimal” bandwidth works out for profile-kernel based method. We suggest a general computation strategy for estimating nonparametric functions. We also employ the penalized spline method for partially linear models and conduct intensive simulation experiments to explore the numerical performance of the penalized spline method, profile and backfitting methods. A real example is analyzed with the three methods.

1 Introduction

Partially linear models are, in form, special cases of the additive regression models (Hastie and Tibshirani, 1990; Stone, 1985), which assume that the relation between the response variables and the covariates can be represented as

$$Y_i=X_i^{\text{T}}\beta +g(T_i)+{\epsilon }_i,$$ (1)

where X_i are d–vector covariates, T_i are scalar covariates, the function g(·) is unknown, and the model’s errors ε_i are independent, with a conditional mean zero given the covariates. The models allow easier interpretation of the effect of each variables and may be preferable to a completely nonparametric model since the well-known reason “curse of dimensionality”. On the other hand, partially linear models are more flexible than the standard linear model because they combine both parametric and nonparametric components when it is believed that the response variable Y depends on variable X in a linear way but is nonlinearly related to other independent variable T. Another appealing feature of partially linear models is that computation is remarkably easier for partially linear models than for additive models, in which iterative approaches such as a backfitting algorithm (Hastie and Tibishirani, 1990) or marginal integration (Linton and Nielsen, 1995) are necessary. In partially linear models, a closed form of the estimator of β can be explicitly described out even with the backfitting algorithm.

Partially linear models have been paid a great attention in the past decade, since Engle et al. (1986) proposed these models and used them to analyze the relation between electricity usage and average daily temperature. They have been widely studied in the literature. See, for example, the work of Cuzick (1992), Carroll et al. (1997) Severini and Staniswalis (1994), Gao and Anh (1999), and Li (2000), among others. The models have been applied in economics (Schmalensee and Stoker, 1999), biometrics (Zeger and Diggle, 1994), and environmental science (Prada-Sánchez et al., 2000). More recently, Lin and Carroll (2001) considered semiparametric partially generalized linear models for clustered data.

Up to now, several methods have been proposed to consider partially linear models. Engle et al. (1986), Heckman (1986), and Rice (1986) used the smoothing spline technique. Robinson (1988) constructed a feasible least-squares estimator of β by estimating the nonparametric component by a Nadaraya-Waston kernel estimator. Speckman (1988) introduced the idea of a profile least-squares method. Hamilton and Truong (1997) developed the local linear method. Härdle, Liang and Gao (2000) systematically summarized the results obtained to date for partially linear models.

Kernel regression, including local constant (Speckman, 1988) and local linear techniques (Hamilton and Truong, 1997; Opsomer and Ruppert, 1999) has also been used to study the partially linear models. A remarkable characteristic of the kernel-based methods is that under-smoothing has been taken in order to get the root–n estimator of β sometimes (Green et al., 1985; Opsomer and Ruppert, 1999), and this restriction may be canceled (Speckman, 1988; Severini and Staniswalis, 1994). These can be confusing, and it may not be clear for users that which strategy of bandwidth selection is appropriate. We present here a way of clarifying the essential differences by reviewing the profile-kernel based estimator, briefly mentioning backfitting estimator, and analyzing the reasons for applying undersmoothing and regular smoothing. We also suggest an alternative estimator of the nonparametric component and develop the penalized spline with a linear mixed-effects (LME) framework in partially linear models.

The paper is organized as follows. Section 2 reviews the profile-kernel based and backfitting estimators. Empirical-bias bandwidth selection (EBBS) is mentioned. Section 3 develops the penalized spline for partially linear models. Numerical performance of these methods is intensively explored in Section 4. Section 5 investigates a real example to illustrate the approaches.

2 Kernel-based Profile and Backfitting Methods

Note that E(Y|T) = {E(X|T)}^Tβ+g(T). It follows that Y − E(Y|T) ={X−E(X|T)}^Tβ+ε. An intuitive estimator of β may be defined as the least squares estimator after appropriately estimating E(Y|T) and E(X|T).

In what follows, for any random variable (vector) ζ, let Ê(ζ|T) be a kernel regression estimator of E(ζ|T), and let ζ̃ = ζ − E(ζ|T), Σ_X|T = cov{X − E(X|T)}. For example, X̃_i = X_i − E(X_i/T_i). Denote Y = (Y₁, ···, Y_n)^T X, g, and T similarly. Let m_x(t) = E(X|T = t), m_y(t) = E(Y|T = t), and

$$\mathrm{\Psi }(m_x,m_y,\beta ,Y,X,T)=\{X-m_x(T)\}\left[Y-m_y(T)-{\{X-m_x(T)\}}^{\text{T}}\beta \right].$$ (2)

The profile-kernel based estimator β̂_P solves the estimating equations

$$0=\sum _{i=1}^n\mathrm{\Psi }({\widehat{m}}_x,{\widehat{m}}_y,\beta ,Y_i,X_i,T_i),$$

where m̂_x and m̂_y are two kernel estimators of m_x and m_y. There are a host of alternative methods for estimating m_x and m_y, including higher degree local polynomial kernel methods, kernel methods with varying bandwidths, smoothing and regression splines, etc. Specman (1988) studied local constant smoother. We use local linear smoothers with fixed bandwidths in this paper. The same results will apply for any kernel–based method.

We establish the asymptotic normality of β̂_P under the following assumptions.

Assumption 2.1

1. Σ_X|T is a positive-definite matrix, E(ε|X, T) = 0, and E(ε²|X, T) < ∞;

2. The bandwidths estimating m_x(t) and m_y(t) are of order n^−1/5;

3. K(•) is a bounded symmetric density function with compact support and satisfies that ∫K(u)du = 1, ∫K(u)udu = 0 and ∫u²K(u)du = 1;

4. The density function of T, f(t), is bounded away from 0 and has a bounded continuous second derivative;

5. m_y(t), m_x(t), and g(t) have bounded and continuous second derivatives.

Theorem 2.1

Under Assumption 2.1, $\sqrt{n}({\stackrel{\sim }{\beta }}_{\text{P}}-\beta )$ is asymptotically normally distributed with mean zero and covariance matrix ${\mathrm{\sum }}_{\text{X}\mid \text{T}}^{-1}E({\epsilon }^2\stackrel{\sim }{X}{\stackrel{\sim }{X}}^{\text{T}}){\mathrm{\sum }}_{\text{X}\mid \text{T}}^{-1}$.

The proof is given in the Appendix. The result is the same as that of Speckman (1988). If ε is independent of (X, T) and has variance σ², the covariance matrix is ${\mathrm{\sum }}_{\text{X}\mid \text{T}}^{-1}{\sigma }^2$. Checking the proof of Theorem 2.1, we find that, for any estimators m̂_x(t) and m̂_y(t) that satisfy sup_t|m̂_x(t) − m_x(t)| = o_P(n^−1/4) and sup_t|m̂_y(t) − m_y(t)| = o_P(n^−1/4), Theorem 2.1 still holds.

Theorem 2.1 indicates that the profile estimator β̂_P is asymptotically normal for the usual optimal bandwidth n^−1/5. This seems to contradict the conclusion made by Opsomer and Ruppert (1999). Before investigating further, we introduce Opsomer and Ruppert’s estimators, β̂_back, of β and g(t) as follows. A detailed discussion of the two estimators are referred to their article. Although the two estimators are gotten from a backfitting algorithm, there are closed forms and calculation is unnecessarily iterative. However, we still call them “backfitting” estimators in what follows.

Denote ${\mathbf{Z}}_t=\left\{\begin{array}{cc}1& (T_1-t)\\ ⋮& ⋮\\ 1& (T_n-t)\end{array}\right\}$, W_t = diag{K_h(T₁ − t), ···, K_h(T_n − t)} with K_h(·) = 1/hK(·/h), and ${\mathbf{S}}_t={\mathbf{e}}_1^{\text{T}}{({\mathbf{Z}}_t^{\text{T}}{\mathbf{W}}_t{\mathbf{Z}}_t)}^{-1}{\mathbf{Z}}_t^{\text{T}}{\mathbf{W}}_t$ with e₁ = (1, 0)^T, and $\mathbf{S}={({\mathbf{S}}_{T_1}^{\text{T}},\cdots ,{\mathbf{S}}_{T_n}^{\text{T}})}^{\text{T}}$. The backfitting estimators β̂_back and ĝ_back are explicitly expressed as follows:

$$\begin{array}{l}{\widehat{\beta }}_{\text{back}}={\{{\mathbf{X}}^{\text{T}}(\mathbf{I}-\mathbf{S})\mathbf{X}\}}^{-1}{\mathbf{X}}^{\text{T}}(\mathbf{I}-\mathbf{S})\mathbf{Y},\\ {\widehat{\mathbf{g}}}_{\text{back}}=\mathbf{S}(\mathbf{Y}-{\mathbf{X}}^{\text{T}}{\widehat{\beta }}_{\text{back}}),\end{array}$$

where and in the sequel, I denotes the n × n identity matrix.

If Assumption 2.1 holds, except replacement of (b) by h ≈ n^s for −1 < s < −1/4, β_back is still asymptotically normal with the same limit distribution as that of β_P. The proof and discussion are given in Opsomer and Ruppert (1999). This statement indicates that to be root–n consistent, undersmoothing is unavoidable. We analyze this necessity and the unrestriction for β̂_P as follows.

When we calculate the bias of β̂_back, the essential term is X^T(I − S)g/n. Each element of this vector is the same order as h² + o_P(h²). However, the bias of β̂_P is determined by X^T(I − S₁)(I − S₂)g/n, where S₁ and S₂ are two projection smoother matrices like S in β̂_back. Note that each element of the d×n matrix X^T(I−S₁) −{X − E(X|T)}^T is of order h² +o_P(h²), and each element of the n–vector (I − S₂)g is of order h² + o_P(h²). A direct derivation yields that each element of the d–vector X^T(I − S₁)(I − S₂)g/n is of order h⁴ + o_P(h⁴). The biases of β̂_back and β̂_P are therefore of orders h² and h⁴, respectively. The usual “optimal” bandwidth n^−1/5 therefore results that β̂_back is not consistent, but β̂_P is. That is why we need to undersmooth for β̂_back, but undersmoothing is unnecessary for β̂_P.

The statements on bandwidth selection indicate that after obtaining the estimators β̂_P with optimal bandwidths and the estimator β̂_back with undersmoothing, if we directly estimate g(t) by ĝ_P(t) = m̂_y(t) −{m̂_x(t)}^Tβ̂_P and ĝ_back(t) = S_t(Y − Xβ̂_back) with the corresponding bandwidths, the mean squared errors of two nonparametric estimators are of the same order h⁴ +O((nh)⁻¹). The squared bias and variance of ĝ_P(t) are of the same order as O(n^−4/5), whereas the variance of ĝ_back(t) dominates the squared bias of ĝ_back(t) and is of higher order than O(n^−4/5) because the corresponding bandwidth is n^s for s < −1/4. What we suggest here is that regardless whether β̂_P or β̂_back is used, we estimate g(t) by regressing y − X^Tβ̂_P(or β̂_back) on T with optimal bandwidths. This leads to an appropriate estimator of g(t).

When implementing backfitting estimator β̂_back with local linear regression, we adopt the EBBS methods proposed by Ruppert (1997). Its implementation in partially linear models has been suggested by Opsomer and Ruppert (1999) and can be described as follows.

Give a set of bandwidth candidates H = {h₁, ···, h_M} and a d– vector c. Approximate the bias of c^Tβ̂ by a polynomial of h, and regress these M biases at H on h₁, ···, h_M to obtain the bias of c^Tβ̂, say b(h). The final bandwidth is the solution of

$$\underset{h\in \{h_{k_1},\cdots ,h_{M-k_2}\}}{min}\{b^2(h)+{\mathbf{c}}^{\text{T}}{\widehat{\sigma }}^2(h)\mathrm{\sum }(h)\mathbf{c}\},$$

where ${\widehat{\sigma }}^2(h)=n^{-1}{\sum }_{i=1}^n{\{Y_i-X_i^{\text{T}}{\widehat{\beta }}_{\text{back}}(h)-{\widehat{g}}_{\text{back}}(T_i,h)\}}^2$ and Σ(h) = {X^T(I − S)X}⁻¹{X^T(I − S)(I − S)X}{X^T(I − S)X}⁻¹; k₁ and k₂ are two tuning parameters. More details refer to Opsomer and Ruppert (1999).

3 The Penalized Spline

The penalized spline method for nonparametric regression was developed by Eilers and Marx (1996) and Ruppert and Carroll (2000). Brumback, Ruppert and Wand (1999) connected this idea with LME framework. See also Coull et al. (2001), Kammann and Wand (2003), and Wand (2003). In this section, we employ the penalized spline method for partially linear models.

We approximate g(t) by $g(t,\mathit{\gamma })={\gamma }_0+{\gamma }_{1}t+\cdots +{\gamma }_p{t}^p+{\sum }_{k=1}^K{b}_k{(t-{\zeta }_k)}_{+}^p$, where p ≥ 1 is an integer and ζ₁ < · · · < ζ_K are fixed knots, a₊ = max(a, 0). Denote γ = (γ₀, ···, γ_p)^T. Consider Y = X^Tβ + g(T, γ) + ε. The penalized spline estimator ${({\widehat{\beta }}_{\text{PS}}^{\text{T}},{\widehat{\mathit{\gamma }}}_{\text{PS}}^{\text{T}})}^{\text{T}}$ of (β^T, γ^T)^T is defined as the minimizer of

$$\sum _{i=1}^n{\{Y_i-X_i^{\text{T}}{\beta }_i-g(T_i,\mathit{\gamma })\}}^2+\alpha \sum _{k=1}^K{b}_k^2,$$ (3)

where α is a smoothing parameter.

As shown by Brumback et al. (1999), the estimator ${({\widehat{\beta }}_{\text{PS}}^{\text{T}},{\widehat{\mathit{\gamma }}}_{\text{PS}}^{\text{T}})}^{\text{T}}$ based on equation (3) is equivalent to the estimator of β based on an LME model,

$$\begin{array}{l}\mathit{y}=\mathbf{\Lambda }{({\beta }^{\text{T}},{\mathit{\gamma }}^{\text{T}})}^{\text{T}}+\mathit{Zb}+\mathit{\epsilon },\\ \text{where}\mathbf{\Lambda }=\left(\begin{array}{ccccccc}{x}_{11}& \cdots & x_{1d}& 1& T_1& \cdots & T_1^p\\ x_{21}& \cdots & x_{2d}& 1& T_2& \cdots & T_2^p\\ ⋮& ⋮& \ddots & ⋮& & & \\ x_{n1}& \cdots & x_{nd}& 1& T_n& \cdots & T_n^p\end{array}\right),\mathit{Z}=\left\{\begin{array}{ccc}{(T_1-{\zeta }_1)}_{+}^p& \cdots & {(T_1-{\zeta }_K)}_{+}^p\\ {(T_2-{\zeta }_1)}_{+}^p& \cdots & {(T_2-{\zeta }_K)}_{+}^p\\ ⋮& \ddots & ⋮\\ {(T_n-{\zeta }_1)}_{+}^p& \cdots & {(T_n-{\zeta }_K)}_{+}^p\end{array}\right\},\end{array}$$

where $\mathit{b}={(b_1,\cdots ,b_K)}^{\text{T}}\sim (0,{\sigma }_b^2),\mathit{\epsilon }={({\epsilon }_1,\cdots ,{\epsilon }_n)}^{\text{T}}\sim (0,{\sigma }_{\epsilon }^2)$, and $\alpha ={\sigma }_{\epsilon }^2/{\sigma }_b^2$. This fact implies that the penalized spline smoother in the framework of equation (3) is equivalent to a standard LME model. The solution can be obtained through the use of an LME macro available in S-PLUS software.

Assume that Λ is of full rank and we denote it by (X, Λ₂), a direct calculation yields that

$$\begin{array}{l}{\widehat{\beta }}_{\text{PS}}={[{\mathbf{X}}^{\text{T}}{\mathbf{V}}^{-1/2}\{\mathbf{I}-{\mathbf{V}}^{-1/2}{\mathbf{\Lambda }}_2{({\mathbf{\Lambda }}_2^{\text{T}}{\mathbf{V}}^{-1}{\mathbf{\Lambda }}_2)}^{-1}{\mathbf{\Lambda }}_2^{\text{T}}{\mathbf{V}}^{-1/2}\}{\mathbf{V}}^{-1/2}\mathbf{X}]}^{-1}.\\ [{\mathbf{X}}^{\text{T}}{\mathbf{V}}^{-1/2}\{\mathbf{I}-{\mathbf{V}}^{-1/2}{\mathbf{\Lambda }}_2{({\mathbf{\Lambda }}_2^{\text{T}}{\mathbf{V}}^{-1}{\mathbf{\Lambda }}_2)}^{-1}{\mathbf{\Lambda }}_2^{\text{T}}{\mathbf{V}}^{-1/2}\}{\mathbf{V}}^{-1/2}\mathbf{Y}],\end{array}$$

where V = ZZ^T + αI.

The penalty parameter α and the number of knots K must be selected in implementing the penalized spline. For these two parameters, α plays a more important role. However, the formulation of a mixed-effects model automatically derives an estimated value of α. We only need to specify the number of knots K. On the other hand, the estimator β̂_PS is not so sensitive to K as to α. Our computation experience indicates that max(10, n=4) is a good choice of the value of K and that the results are insensitive to different values of K. With this value of K, the knots are at equally spaced sample quantiles of {T_i}. See Ruppert (2002) for a detailed discussion of knot selection.

4 Simulation Experiment

To evaluate the three estimators β̂_back, β̂_P, and β̂_PS, we conduct an intensive simulation experiment to explore their numerical performance. We generate n = 300 data from model (1), and assume that β = (1.3, 0.45)^T. Let T ~ Uniform(0, 1), and g(t) = 4.26{exp(−3.25t) −4 exp(−6.5t) + 3 exp(−9.75t)}.

We consider three cases to see the numerical performance of three methods.

Case 1

X ~ Normal(0, diag(0.3, 0.4)) and ε ~ Normal(0, 1). X and T are independent.

Case 2

X ~ Normal(0, diag(0.3, 0.4)) and ε ~ Normal(0, |t|). X and T are independent.

We use this case to show the effect of heteroscedasticity on the three estimators.

Case 3

x_ij = 0.4T_i + 0.6U_i for j = 1, 2, U ~ Uniform(0, 1), and ε ~ Normal(0, 1). We use this case to explore the effect of correlation between X and T on the three estimators.

At each configuration, 500 independent sets of data are generated. In our simulation and consideration of a real data example later, we use the Epanechnikov kernel function k(u) = 15/16(1 − u²)²I_(|u|≤1). We obtain the estimates of the parametric and nonparametric components for each of three estimation methods. When implementing β̂_P, we estimate m_x(t) and m_y(t) by using the usual local linear regression and select bandwidths by using the plug-in method (Ruppert et al. 1995). When implementing β̂_back, we select bandwidths by using the EBBS method. The estimates of β are summarized in Table 1. We also present the box plots of the parametric estimates for N = 500 in case 1 in Figure 1. Outcomes from case 2 is similar and the boxplot is not reported. The estimates β̂_back from case 3 are more stable than those from cases 1 and 2 and the boxplot is omitted. These results indicate that the estimated values are highly exact. The variance of β̂_P is slightly larger than those of β̂_back and β̂_PS. When the error is not homogeneous (error distribution Normal(0, |t|)), the estimators given in Section 2 are theoretically not efficient. However, the simulation experiment results are still satisfactory.

Table 1.

Results of the simulation study. “estimates” is the simulation mean, “s.e.” is the simulation standard error.

Case		β̂back		β̂P		β̂PS
1	estimates	1.299	0.452	1.297	0.451	1.3	0.45
	s.e.	0.0189	0.0142	0.0304	0.0167	0.0178	0.0124
2	estimates	1.299	0.450	1.296	0.451	1.299	0.449
	s.e.	0.0259	0.0201	0.0333	0.0296	0.0240	0.0187
3	estimates	1.296	0.448	1.32	0.459	1.296	0.451
	s.e.	0.1085	0.0849	0.101	0.082	0.0907	0.071

Figure 1.

Box plots of the estimates (N=500 simulation runs) for backfitting (back), profile, and penalized spline (PS) methods for homogeneous errors. Upper panel: the first element of β; lower panel: the second element of β. The long horizontal lines are the true values.

In most cases, we are more interested in the parameter β and take the g(t) as a nuisance parameter. However, the estimates of the nonparametric function g(t) sometimes provide useful information in practice. A question therefore arises how about the estimates of g(·) obtained according to our suggestion. Figure 2 shows us the estimates of g(t) after selecting appropriately smoothing parameters for estimating β in case 1. The solid, dotted, dashed, and long-dashed lines indicate the true curve and the estimates based on the backfitting, profile, and penalized spline techniques. The estimated curves are indistinguishable from the true curve. Outcomes from cases 2 and 3 are similar and are not reported. When we estimate g(t) by ĝ_back(t) or ĝ_P directly, their biases are larger than those obtained in the present context, especially for the backfitting method (not shown here).

Figure 2.

Estimated curves of the nonparametric components for simulated data. The true curve is shown as an unbroken line and the curves estimated with backfitting, profile, and penalized spline methods are shown by dotted, dashed, long-dashed lines, respectively.

5 Real Data Example

In this section we analyze a real data example by using the three estimation methods. The real data are from a study of the relation between the log-earnings of an individual and personal characteristics (sex, marital status) and measures of a person’s human capital, such as schooling and labor market experience. See Rendtel and Schwarze (1995) for a detailed description of the data. Experience suggests that there is a non-linear relation between log-earnings and labor market experience, which therefore plays the role of the variable T in (1). The wage curve is obtained by including the local unemployment rate as an additional regressor, that may have a nonlinear influence. Rendtel and Schwarze (1995), for instance, estimate g(•) as a function of the local unemployment rate by using smoothing-splines. We study the relation by using the model

$$ln(Y)=X\beta +g(T)+\epsilon ,$$

where X is a dummy variable indicating the level of secondary schooling a person has completed and T is a measure of labor market experience defined as the number of years spent in the labor market and approximated by subtracting (years of schooling + 6) from a person’s age. The estimate of β can be interpreted as the rate of return from obtaining the respective level of secondary schooling. Human capital theory suggests that g(T) should take concave form: rapid human capital accumulation in the early stage of one’s labor market career is associated with rising earnings that peak somewhere during mid-life and flatten thereafter as hours worked and the incentive to invest in human capital decrease.

The estimates of β based on the backfitting, profile-kernel, and penalized spline methods are β̂_back = 0.0942 (s.e. 0.0085), β̂_P = 0.0914 (s.e. 0.0089), and β̂_PS = 0.0912 (s.e. 0.0083). Standard errors are obtained by 500 bootstrap resample. The estimates of g(T) are reported in Figure 3. The results from three methods are similar. We also fit a linear regression model to the data and obtain a mean function ln(y) = 0.594 + 0.0964x + 0.0118t. The estimated coefficient of x is close to those obtained from partially linear modeling. But this linear regression does not reflect the changing trend of the log-earnings over the labor market experience. To investigate the effects of different values of K on the β̂_PS, we also run this data set for K = 2, 5, 8, and 15, and obtain β̂_PS;K=2 = 0.09108 (s.e. 0.00831); β̂_PS;K=5;8;15 = 0.09121 (s.e. 0.00832). The corresponding estimates of g(T) are described in Figure 4. We therefore conclude that the penalized spline estimates are very robust to differing numbers of knots.

Figure 3.

Relation between log-earnings and labor-market experience.

Figure 4.

Penalized spline estimates of the nonparametric component from real data with different numbers of knots.

6 Conclusions and Discussion

In this paper we reviewed the profile-kernel based and backfitting methods and developed a penalized spline method for partially linear models. We analyzed the essential reasons that why the backfitting estimation needs undersmoothing, whereas the profile-kernel based method avoids this restriction, in order to have asymptotically normal estimators. We also suggested a strategy for effectively estimating the nonparametric component. We numerically compared the estimators obtained with backfitting, profile-kernel based, and penalized spline methods.

The plug-in and EBBS methods were used to select bandwidths in the different situations. The main advantages of the penalized spline are that selection of smoothing parameters can be avoided, computation is fast, its numerical performance is comparable to that of other methods. In conclusion, we recommend the penalized spline method because of its robustness in implementation, its computational speediness, and its comparability to other methods. We believe that the penalized spline is methodologically and practically valuable.

Appendix: Proof of Theorems 2.1

We finish the proof of the theorem by using Newey’s (1994) results, although a direct proof is also available. Denote ||•|| a norm for a function, such as Sobolev norm, that is a supremum norm for a function and its derivatives. It follows from Assumption 2.1 that ||m̂_x(•) − m_x(•)|| = o_P(n^−1/4) and m̂_y(•) − m_y(•) = o_P(n^−1/4). This is assumption 5.1(ii) of Newey (1994). Let

$$D(m_x^{\ast }-m_x,m_y^{\ast }-m_y,\beta ,Y,X,T)=\frac{\partial \mathrm{\Psi }}{\partial m_x}(m_x^{\ast }-m_x)+\frac{\partial \mathrm{\Psi }}{\partial m_y}(m_y^{\ast }-m_y).$$

where $\frac{\partial \mathrm{\Psi }}{\partial m_x}$ and $\frac{\partial \mathrm{\Psi }}{\partial m_y}$ are the Frechet derivatives. A direct calculation derives that $E\left(\frac{\partial \mathrm{\Psi }}{\partial m_x}\right)=0$ and $E\left(\frac{\partial \mathrm{\Psi }}{\partial m_y}\right)=0$. Furthermore,

$$\begin{array}{l}\left|\right|\mathrm{\Psi }(m_x^{\ast },m_y^{\ast },\beta ,Y,X,T)-\mathrm{\Psi }(m_x,m_y,\beta ,Y,X,T)-D(m_x^{\ast }-m_x,m_y^{\ast }-m_y,\beta ,Y,X,T)\left|\right|\\ =O_{\text{P}}\left({\left|\right|m_x^{\ast }-m_x\left|\right|}^2+{\left|\right|m_y^{\ast }-m_y\left|\right|}^2\right).\end{array}$$ (A.1)

Equation (A.1) is Newey’s assumption 5.1(i). His assumption 5.2 holds by the expression of D(•, β, Y, X, T). In addition, it follows from the above statements that for any ( $m_x^{\ast },m_y^{\ast }$),

$$E\left\{D(m_x^{\ast }-m_x,m_y^{\ast }-m_y,\beta ,Y,X,T)\right\}=0,$$

thus verifying Newey’s assumption 5.3: his α(T) = 0, according to his discussion just above his Lemma 5.1. By that lemma, it follows that β̂_P has the same distribution as the solution to the equation

$$0=\sum _{i=1}^n\mathrm{\Psi }(m_x,m_y,\beta ,Y_i,X_i,T_i).$$ (A.2)

It is easy to show that the solution of (A.2) has the same limit distribution as described in the statement of Theorem 2.1. This completes the proof.