Aberrant Crypt Foci and Semiparametric Modeling of Correlated Binary Data

Summary

Motivated by the spatial modeling of aberrant crypt foci (ACF) in colon carcinogenesis, we consider binary data with probabilities modeled as the sum of a nonparametric mean plus a latent Gaussian spatial process that accounts for short-range dependencies. The mean is modeled in a general way using regression splines. The mean function can be viewed as a fixed effect and is estimated with a penalty for regularization. With the latent process viewed as another random effect, the model becomes a generalized linear mixed model. In our motivating data set and other applications, the sample size is too large to easily accommodate maximum likelihood or restricted maximum likelihood estimation (REML), so pairwise likelihood, a special case of composite likelihood, is used instead. We develop an asymptotic theory for models that are sufficiently general to be used in a wide variety of applications, including, but not limited to, the problem that motivated this work. The splines have penalty parameters that must converge to zero asymptotically: we derive theory for this along with a data-driven method for selecting the penalty parameter, a method that is shown in simulations to improve greatly upon standard devices, such as likelihood crossvalidation. Finally, we apply the methods to the data from our experiment ACF. We discover an unexpected location for peak formation of ACF.

1. Introduction

This article develops methods for semiparametric regression with possibly nonstationary correlated binary data. We are motivated by an example in colon carcinogenesis where the number of observations per subject is too large to use exact likelihood or restricted likelihood (Sherman, Apanasovich, and Carroll, 2004), so we use composite likelihood. This article extends Heagerty and Lele's (1998) work on composite likelihood to nonparametric specifications of the mean. We develop a new smoothing parameter selector, because there appears to be no previous work on smoothing parameter selection for composite likelihood. In the Web Appendix, the asymptotic theory in Heagerty and Lele (1998) is extended to smoothing parameter estimation. The asymptotic theory we provide is new and should be of general interest. We use increasing domain asymptotics, in which the minimum distance between the sampling points is bounded away from zero and thus the spatial domain of observation is unbounded. This type of asymptotics has been successfully considered in many applications (Cressie, 1993; Heagerty and Lele, 1998; Fuentes, 2002).

Our model has fixed effects for the mean, representing systematic effects, and random effects giving a rather general form of dependence conditional upon the mean. The fixed-effects structure includes partially linear models, single index models, and additive models and uses fixed-knot splines with penalties (Ruppert, Wand, and Carroll, 2003). This structure is new and should be of interest elsewhere.

The problem is motivated by a study in colon carcinogenesis concerning aberrant crypt foci (ACF), colonic crypts that have been changed morphologically by a carcinogen and/or radiation. These ACF are precursors to colon cancer (Bird, 1995; Bird and Good, 2000), and hence of major biological significance. The mean function models the intensity of ACF formation as a function of location in the crypt, whereas the spatial dependency can address the issue of whether and how cells signal to each other. Although suggested by this application, our methodology is sufficiently general and it will apply to a wide variety of problems.

The ACF study has a large number (about 800) of repeated measures per subject and a small number of subjects (about 14). Many papers on longitudinal data address data sets of the opposite type where there are many subjects but relatively few repeated measures per subject. To analyze data such as ours, we provide a theoretical framework for a large number of outcomes on a single subject. This methodology can be applied separately to each subject, as we do in the ACF study.

Our binary mixed model is a type of generalized linear mixed model (GLMM) called a spatial GLMM. Spatial GLMMs (Diggle, Tawn, and Moyeed, 1998) are flexible models for a variety of applications where we have observations of spatially dependent and non-Gaussian random variables. As in a standard GLMM, given the random effects, which they model by a Gaussian random field, the observations are conditionally independent and follow a generalized linear model. Both Bayesian and frequentist methods have been developed for inference and forecasting in spatial GLMMs. Diggle et al. (1998) and Christensen and Waagepetersen (2002) used a Bayesian Markov chain Monte Carlo framework. The computational burden for those methods is high because the number of correlated random effects to be simulated is equal to the number of observations. Maximum likelihood estimation (MLE) in spatial GLMMs generally involve numerical integration of a high-dimensional integral. The integral may be computed by Monte Carlo integration. McCulloch (1997) reviews several Monte Carlo techniques for MLE within GLMMs. Booth and Hobert (1999) described a Monte Carlo expectation-maximization (EM) algorithm for the spatial probit model, whereas Zhang (2002) used a maximum likelihood approach together with Monte Carlo EM algorithm to estimate parameters of a general spatial GLMM. Alternatively the integral may be computed by Laplace's method, which uses a Gaussian approximation of the integrand. In GLMM inference, this method has been used by Breslow and Clayton (1993), who call it penalized quasilikelihood (PQL) and has been implemented in SAS's GLIMMIX (SAS 9.1 from SAS Institute, Inc.). However, PQL is known to be biased (Breslow and Lin, 1995; Lin and Breslow, 1996; McCulloch, 1997; Neuhaus and Segal, 1997). In two examples where the exact MLE can be computed, Neuhaus and Segal find that the exact MLE and the PQL estimator can differ by more that one standard error of the PQL and even Breslow and Lin's (1995) bias correction does not lead to accurate approximation of the exact MLE. Neuhaus and Segal's conclusion is that “the biases exhibited by the Taylor approximation and PQL approaches indicate that these approaches require further development and modification before they should be used in practice.”

Maximum likelihood inference, whether by EM or PQL, requires high-dimensional matrices to be inverted repeatedly. Because matrix inversion is an order O(n³) operation, computational effort increases rapidly with n. Here n is the number of random effects, which in our context is the number of observations. This is the motivation for pairwise likelihood (Heagerty and Lele, 1998; Curriero and Lele, 1999; Nott and Rydén, 1999; Kuk and Nott, 2000; Renard, Molenberghs, and Geys, 2004). Even for ordinary kriging where the spatial process is directly observable and there is no need to integrate latent variables, Curriero and Lele (1999) state that ML or restricted maximum likelihood estimation (REML) “involve inversion of large matrices which can be computationally prohibitive” and use alternative methods. In a simulation study of multilevel probit models, Renard et al. (2004) find that the time to compute the MLE increases rapidly with the number of the random effects per cluster and the time to compute the MLE is much greater than for pairwise likelihood when this number exceeds 3; see their Table 3. Obviously, the crossover point where the pairwise likelihood estimator is less intensive than the MLE is problem specific, but it is noteworthy that the crossover point is very low in this study.

A possible solution to this computational problem is to reduce the number of random effects, for example, with low-rank splines as in Ruppert et al. (2003, Section 13.8). However, this technique is applicable only when the observations are conditionally independent given the random effects determining the mean. In our model, the observations are conditionally dependent, given the mean. The dependencies in the ACF example are short-range and cannot be modeled with only a small number of random effects.

In summary, both Bayesian and maximum likelihood approaches are extremely difficult to implement for large sets of correlated binary data. To gain computational efficiency, we consider composite likelihood (Lindsay, 1988), which is the product of likelihoods for subsets of data, and estimate parameters by maximizing this product. This approach has been used in many problems for correlated binary response data, for example, in spatial models (Heagerty and Lele, 1998). Note that the integrals require the use of low-dimensional numerical integration.

We know of no theoretical results comparing the efficiency of composite to full maximum likelihood, but in an example of Kuk and Nott (2000) pairwise likelihood for clustered data has nearly full asymptotic efficiency. In another simulation study, Renard et al. (2004) found that the efficiency losses of pairwise likelihood relative to the MLE averaged about 10% and ranged from 5% to 18% across three parameters (two fixed effects and a variance component) in four simulation studies. Cox and Reid (2004) studied asymptotic relative efficiency for estimators based on pairwise likelihood in the case of dichotomized normals. They reported a relatively small loss of efficiency. Varin, Høst, and Skare (2005) compared the maximum pairwise likelihood estimator with the approximate MLE based on Laplace's approximation for integration and reported similar values for mean squared errors. Zhao and Joe (2005) showed a relatively small loss of efficiency for models they considered.

Also, in a related context of autologistic models, Sherman et al. (2004) found the maximum pseudolikelihood and generalized pseudolikelihood estimators comparable to the MLE approximated by MCMC.

In Section 2, we describe our estimation methodology and give a brief discussion of asymptotic theory and estimation of standard errors. A very general asymptotic theory, including technical conditions and proofs, is given in Web Appendix A. Section 3 presents our smoothing parameter selectors. A small simulation study is presented in Section 4, followed by the analysis of the ACF experiment in Section 5. Discussion and extensions are given in Section 6.

2. Models

2.1 Binary Mixed Model and General Fixed Effects Structure

Our models are semiparametric GLMMs, with a binary response variable D_i and predictors (Z_i, X_i), all measured within a spatial domain. In our case, we will model the effects of Z_i parametrically and the results of X_i semiparametrically, and as in our example we focus on the case that X is scalar.

To complete the model specification, let ∊_i be independent Normal(0, 1) random variables. Let λ_i denote random effects responsible for possible spatial dependence. For a parameter ρ and a correlation matrix Ω(ρ), the {λ_i} are assumed to be normally distributed with mean 0 and covariance matrix ${\sigma }_{\lambda }^2\Omega \left(\rho \right)$ and independent of the ∊_i. Let μ_i be the mean function incorporating systematic effects. Then the model is defined as D_i = I(μ_i + λ_i + ∊_i > 0), so that

$$\mathrm{pr}(D_i=1\mid {\lambda }_i,{\mu }_i)=\Phi ({\mu }_i+{\lambda }_i),$$ (1)

where Φ(•) is the standard normal cumulative distribution function (CDF).

In the remainder of this section, we describe the structure of the μ_i (Section 2.3), the fixed-knot splines used to model μ_i (Section 2.4), and the composite likelihood method (Section 2.5) to estimate parameters and functions.

2.2 Random Effects Structure

We will consider one approach to modeling the covariance function of the spatial process {λ_i}, although our methods and theoretical techniques apply more generally.

Let d(i, j) be the Euclidian distance between sites i and j. The simplest type of correlation structure is stationary, for example, the Matérn family (Handcock and Stein, 1993; Handcock and Wallis, 1993; Stein, 1999). Thus, the (i, j) element of Ω(ρ) is Ω_ij(ρ) = M{d(i, j), ρ}, where M(•) is a known function with unknown parameter ρ.

There are several choices available for the correlation function, see Stein (1999) for an extensive overview. In this article we work with a parametric family of autocorrelation functions, the Matérn family (Handcock and Stein, 1993; Handcock and Wallis, 1999). Let ${\mathcal{K}}_{\mathcal{V}}\left(•\right)$ be the modified Bessel function of order ν. Then the Matérn correlation at distance d can be written as either K_μ(d/ρ)(d/ρ)^ν2^1-ν/Γ(ν) or its more stable version 2^1-ν(2dν^1/2/ρ)^νK_ν(2dν^1/2/ρ)/Γ(ν). For example, when ν = 5/2, the correlation is exp(-d/ρ) {1 + d/ρ + (1/3)(d/ρ)²}.

2.3 General Fixed Effects Structure

We focus here on the partially linear model for the fixed effects, namely, that for some unknown parameter ζ₀, and an unknown function Λ(·), the fixed effects structure is

$${\mu }_i=Z_i^{\mathrm{T}}{\zeta }_0+\Lambda \left(X_i\right).$$ (2)

Our results will be phrased in terms of the partially linear model (2), but they actually apply to far more general situations, most notably the single-index model $\mathcal{G}\left(Z_i^{\mathrm{T}}{\zeta }_0\right)+\Lambda \left({\mathrm{X}}_i\right)$, where G(·) is an unknown function modeled by splines with its own smoothing parameter, and for identifiability |ζ₀| = 1. However, the partially linear model is important in its own right, applicable to the ACF example, and the results for it are the most transparent.

2.4 Regression Splines and Penalization

In our work, we model an unknown function Λ(•) as a fixed-knot regression spline, which has the representation Λ(x) = B̃^T(x)η₀, where B̃^T(x) = {B₁(x), …, B_k(x)}. For example, B̃(x) might be the B-spline basis functions or the truncated power series basis (Ruppert et al., 2003) of order q with K knots x₁, …,x_K given as ${\tilde{B}}^{\mathrm{T}}\left(x\right)=(1,x,\dots ,x^q,(x-x_1{)}^q+,\dots ,(x-x_K{)}_{+}^q)$, where the subscripted plus sign is the positive part function.

Regression splines with a fixed number of knots have become an increasingly popular means of semiparametric inference (see Ruppert et al. [2003] and references therein). Generally, not many knots are required to capture most fixed effects structures (Ruppert, 2002), and for binary data in particular, capturing very complex structure that cannot be modeled by a low-order basis representation is unlikely to be practical. Of course some sort of smoothing is required. This is generally done either by knot selection devices to greatly lower the dimensionality, or by penalization to achieve smoothness. In this article, we use the latter device, see Section 3 for details. Our general model is a generalization of that of Yu and Ruppert (2002) who also use fixed-knot splines and found that this technique is a numerically stable approach to single-index model.

2.5 Penalized Composite Likelihood Estimation

In composite likelihood estimation, the n observations are split into many subsets of a computationally convenient size k. We use k = 2 in our numerical work and, for ease of notation, will keep k fixed at 2 in the main body of the article. The extension to general k is discussed in the Appendix and proofs in the Web Appendix are for general k. For each subset of observations, the log-likelihood function is computed: the idea of using a subset of observations is to make this calculation easy. Then the various log likelihoods are summed across all the selected subsets, and maximized to get the composite likelihood estimator. Inference for composite likelihood is somewhat tricky because the subsets are not independent sets of data.

In this section, we first set up the formal calculation of a composite likelihood function. We then introduce the penalization that is necessary when using regression splines. Section 3 introduces a method for data-based estimation of the smoothing parameters.

2.5.1 Composite likelihood calculation

Let Φ2(μ₁, μ₂; Δ) be the bivariate standard normal probability of being below μ₁, μ₂ when the correlation matrix is Δ. If ${\mu }_i^{\ast }={\mu }_i∕(1+{\sigma }_{\lambda }^2{)}^{1∕2}$, then marginal probabilities can be expressed as

$$\begin{array}{cc}\hfill & \mathrm{pr}(D_{i_1}=1,D_{i_2}=1\mid {\mu }_{i_1}^{\ast },{\mu }_{i_2}^{\ast },{\sigma }_{\lambda }^2,\rho )\hfill \\ \hfill & ={\Phi }_2\left[{\mu }_{i_1}^{\ast },{\mu }_{i_2}^{\ast };\frac{{\sigma }_{\lambda }^2}{(1+{\sigma }_{\lambda }^2)}{\left\{{\Omega }_{i_j}\left(\rho \right)\right\}}_{i,j\in \{i_1,i_2\}}\right].\hfill \end{array}$$ (3)

The composite likelihood is defined as a sum of log likelihoods based on the marginal distribution of the responses, see below for more details. At least in principle, both the correlation function and the variance of the random effects can be estimated from the marginal component likelihoods.

Because μ_i and ${\mu }_i^{\ast }$ are multiples of one another, for notational simplicity, we will use only one of them. In what follows, we will use only the function ${\mu }_i^{\ast }$. Also, we can now drop the subscript “*” and henceforth ${\mu }_i^{\ast }$ will be denoted by μ_i.

2.5.2 Penalization

Recall from (2) that the fixed effects depend on ζ₀ and the regression spline is written as $\Lambda \left(x\right){\tilde{B}}^{\mathrm{T}}\left(x\right){\eta }_0$. Let β be the collection of the parameters (ζ, η), with true value β₀, and write μ_i = μ_i(β). We need to penalize this function to account for the nonparametric regression, which we do with a smoothing parameter k and a penalty matrix $\mathcal{G}$. The penalty matrix depends on the basis functions used. For example (Ruppert et al., 2003), for the truncated polynomial series basis of order q and K knots defined in Section 2.4, $\mathcal{G}=\text{diag}(0I_q,I_K)$, where I_q is the identity matrix of size q.

In order to estimate the correlation function, we must use at least pairs of observations. Let Θ denote the unknown parameters $({\sigma }_{\lambda }^2,\rho )$ and let θ₀ denote its true value. Define Θ as the collection of all parameters to be estimated, and let θ₀ be its true value.

If we use sets of size 2, then we need to keep track of the values of the sets and their probabilities. To do this, we define

$$\begin{array}{cc}\hfill & Z_{i_1,i_2}^{(j_1,j_2)}=I(D_{i_1}=j_1,D_{i_2}=j_2);\hfill \\ \hfill & {\pi }_{i_1,i_2}^{(j_1,j_2)}=\mathrm{E}(Z_{i_1,i_2}^{(j_1,j_2)}\mid {\mu }_{i_1}^{\ast },{\mu }_{i_2}^{\ast },{\sigma }_{\lambda }^2,\rho ).\hfill \end{array}$$ (4)

Using the marginal probabilities (3), we can write the likelihood at distinct locations (i₁, i₂)as

$${\mathcal{L}}_{i_1,i_2}\left(\Theta \right)=\sum _{j_1,j_2=0}^1{Z}_{i_1,i_2}^{(j_1,j_2)}\log \left\{{\pi }_{i_1,i_2}^{(j_1,j_2)}(\beta ,\theta )\right\}.$$

It is useful to note that ${\mathcal{L}}_{i_1,i_2}\left(\Theta \right)$ is an actual likelihood function, albeit for a reduced set of data.

2.5.3 A penalized composite likelihood

We propose to maximize a weighted penalized composite likelihood. Let w_i1,_i2 be weights, which take nonzero values only if the maximum distance between two locations from a set of locations (i₁, i₂) is less than a specified value apart and 0 otherwise, a choice that is useful to cut down on the size of the summations. It is assumed that w_i1,_i2 does not depend upon (D_i1, D_i2) or any parameters, because otherwise, if the weights depend upon parameters, then our methods will be inconsistent, a well-known fact in generalized least squares that is made explicit by Müller (1999). Here we just used indicators, whereas Guan (2006) proposed to use a data-driven choice of the maximum distance used to select sets and the weight function. However, the choice should be driven by efficiency considerations, and goodness of fit discussed by Guan (2006) is not directly applicable in our case.

For fixed smoothing parameter k, the composite likelihood estimator is denoted by $\widehat{\Theta }\left(\kappa \right)$ and is obtained by maximizing over Θ:

$${\tilde{\mathcal{L}}}_{\mathcal{W}}(\Theta ,\kappa )={\mathcal{W}}^{-1}\sum _{i_1,i_2=1}{w}_{i_1,i_2}{\mathcal{L}}_{i_1,i_2}\left(\Theta \right)-(1∕2)\kappa {\eta }^T\mathcal{G}\eta ,$$ (5)

where $\mathcal{W}={\Sigma }_{i_1,i_2}{w}_{i_1,i_2}$.

2.6 Asymptotic Results and Standard Error Estimation

Web Appendix A gives technical conditions and proofs for models far more general than the partially linear model (2), including single-index models and multiple smoothing parameters. The theory is notable because it describes precisely how the smoothing parameter k must decrease to zero with the sample size. Here we provide a brief description of the main results.

Web Appendix A shows that as long as the smoothing parameter k decreases to zero at a rate faster than ${\mathcal{W}}^{-1∕2}$, then the penalized composite likelihood function ${\tilde{\mathcal{L}}}_{\mathcal{W}}(\Theta ,\kappa )$ given in (5) can be treated as if it were an ordinary composite likelihood function. Thus, if the composite score function is

$${\Psi }_{\mathcal{W}}(\Theta ,\kappa )=\partial {\tilde{\mathcal{L}}}_{\mathcal{W}}(\Theta ,\kappa )∕\partial \Theta ,$$

then $\widehat{\Theta }\left(\kappa \right)$ is asymptotically normally distributed with mean Θ₀, the true value, and a covariance matrix given by Godambe's (1960) information matrix, also known as the sandwich formula:

$$\begin{array}{c}\hfill \mathrm{cov}\left\{\widehat{\Theta }\left(\kappa \right)\right\}\approx {\left[E\left\{\frac{\partial {\Psi }_{\mathcal{W}}(\Theta ,\kappa )}{\partial {\Theta }^{\mathrm{T}}}\right\}\right]}^{-1}\mathrm{cov}\left\{{\Psi }_{\mathcal{W}}(\Theta ,\kappa )\right\}\hfill \\ \hfill \times {\left[E\left\{\frac{\partial {\Psi }_{\mathcal{W}}(\Theta ,\kappa )}{\partial {\Theta }^{\mathrm{T}}}\right\}\right]}^{-1}.\hfill \end{array}$$ (6)

Estimating the covariance matrix given in (6) has a standard and a nonstandard part. The “bread,” that is, the outer matrices, can be estimated simply by plug-in as $[\partial {\Psi }_{\mathcal{W}}\{\widehat{\Theta }\left(\kappa \right),\kappa \}∕\partial {\Theta }^{\mathrm{T}}{]}^{-1}$.

The “meat” of the sandwich, $\text{cov}\left\{{\Psi }_{\mathcal{W}}(\Theta ,\kappa )\right\}$ in (6), is more complex because of the spatial dependence. However, a consistent estimate of it can be well approximated by a Monte Carlo calculation. Specifically, if Θ₀ is the true parameter, then we have a model for the data generation process. Replacing the unknown Θ₀ by the estimated $\widehat{\Theta }\left(\kappa \right)$, we can generate simulated realizations of the data generation process, and hence realizations of ${\Psi }_{\mathcal{W}}(\Theta ,\kappa )$. We then approximate $\text{cov}\left\{{\Psi }_{\mathcal{W}}(\Theta ,\kappa )\right\}$ by the sample covariance matrix of the generated realizations of ${\Psi }_{\mathcal{W}}(\Theta ,\kappa )$.

3. Smoothing Parameter Estimation

It is well known that with dependent data, standard approaches to smoothing parameter selection, such as crossvalidation (CV), lead to undersmoothing, see, for example, Opsomer, Wang, and Yang (2001) for discussion and extensive references. The usual device for numerical response data is either to attempt to select the smoothing parameter in such a way as to minimize asymptotic average mean squared error, or to treat the parameters in the mean as random rather than fixed effects so that the smoothing parameter is a ratio of variance components, and to maximize a resulting mixed model likelihood. Given that in our case the correlated responses are binary and that a likelihood estimate with or without an extra variance component is extremely difficult to compute, it seems, at the very least, useful to explore different approaches.

Let Θ denote all the parameters, let Θ₀ be their true value and let CL(Θ) be a composite log likelihood, as in the first term in (5). Our approach is to adapt the idea of Kullback-Leibler (KL) distance to estimate the smoothing parameters. Define

$$\mathrm{KL}(\Theta ,{\Theta }_0)={\mathrm{E}}_{{\Theta }_0}\{\mathrm{CL}\left({\Theta }_0\right)-\mathrm{CL}\left(\Theta \right)\},$$

where the subscript Θ₀ means that the expectation is taken at the distribution induced by Θ₀. It is technically convenient to work with a symmetrized version of this distance, namely,

$$\mathrm{SKL}(\Theta ,{\Theta }_0)=\mathrm{KL}(\Theta ,{\Theta }_0)+\mathrm{KL}({\Theta }_0,\Theta ).$$

It is easy to see that SKL(Θ, Θ₀) is always nonnegative and equals zero when Θ = Θ₀. In our problem we estimate parameters by maximizing a composite log-likelihood function that depends on a vector of smoothing parameters k. If we plug an estimated $\widehat{\Theta }\left(\kappa \right)$ into this expression we get a random variable, whose expectation we then take to find

$${\mathrm{E}}_{{\Theta }_0}\left[\mathrm{SKL}\{\widehat{\Theta }\left(\kappa \right),{\Theta }_0\}\right].$$ (7)

Our goal is to choose the smoothing parameter so as to minimize (7). More precisely, we will find an asymptotically equivalent version of $\widehat{\Theta }\left(\kappa \right)$, substitute it into (7) and minimize.

The key to the analysis is that in our problem formulation, SKL (Θ, Θ₀) can be computed analytically. We cannot of course compute (7) analytically, but we do show how to compute an asymptotically equivalent version of it, and this allows us to estimate the smoothing parameters. With a slight abuse of notation, for composite likelihood, we have that

$$\begin{array}{cc}\hfill \mathrm{SKL}\left\{\widehat{\Theta }\left(\kappa \right)\right\}& ={\mathcal{W}}^{-1}\sum _{i_1,i_2=1}^n\sum _{j_1,j_2=0}^1{w}_{i_1,i_2}\hfill \\ \hfill & \times [{\pi }_{i_1,i_2}^{(j_1,j_2)}\left({\Theta }_0\right)-{\pi }_{i_1,i_2}^{(j_1,j_2)}\left\{\widehat{\Theta }\left(\kappa \right)\right\}]\hfill \\ \hfill & \times (\log \left\{{\pi }_{i_1,i_2}^{(j_1,j_2)}\left({\Theta }_0\right)\right\}-\log \left[{\pi }_{i_1,i_2}^{(j_1,j_2)}\left\{\widehat{\Theta }\left(\kappa \right)\right\}\right]),\hfill \end{array}$$

where ${\pi }_{i_1,i_2}^{(j_1,j_2)}(\cdot )$ are defined in (4). In the Appendix, we sketch an argument indicating that with this definition of SKL(•), the minimizer of (7) can be derived as follows. Let ${\Sigma }_{\mathcal{W},0}=E\{\partial {\Psi }_{\mathcal{W}}(\Theta ,\kappa )\partial {\Theta }^{\mathrm{T}}\}$ and let Σ_W,0 +Σ_W,c be the interior covariance matrix in the sandwich formula (6). We discussed in Section 2.6 how to estimate these components.

Then, as described in the Appendix, there is a matrix $\mathcal{H}(\tilde{\kappa },{\Theta }_0,{\Sigma }_{\mathcal{W},0},{\Sigma }_{\mathcal{W},c})$ and a vector $\mathcal{J}(\tilde{\kappa },{\Theta }_0,{\Sigma }_{\mathcal{W},0},{\Sigma }_{\mathcal{W},c})$, both defined in the Appendix, such that

$$\kappa ={\mathcal{W}}^{-1}{\mathcal{H}}^{-1}(\kappa ,\Theta ,{\Sigma }_{\mathcal{W},0},{\Sigma }_{\mathcal{W},c})\mathcal{J}(\kappa ,{\Theta }_0,{\Sigma }_{\mathcal{W},0},{\Sigma }_{\mathcal{W},c}).$$

Therefore we estimate κ iteratively such that

$$\begin{array}{cc}\hfill {\kappa }^{\left(l\right)}=& {\mathcal{W}}^{-1}{\mathcal{H}}^{-1}\{{\kappa }^{(l-1)},\widehat{\Theta }\left({\kappa }^{(l-1)}\right),{\widehat{\Sigma }}_{\mathcal{W},0}\left({\kappa }^{(l-1)}\right),{\widehat{\Sigma }}_{\mathcal{W},c}\left({\kappa }^{(l-1)}\right)\}\hfill \\ \hfill & \times \mathcal{J}\{{\kappa }^{(l-1)},\widehat{\Theta }\left({\kappa }^{(l-1)}\right),{\widehat{\Sigma }}_{\mathcal{W},0}\left({\kappa }^{(l-1)}\right),{\widehat{\Sigma }}_{\mathcal{W},c}\left({\kappa }^{(l-1)}\right)\}.\hfill \end{array}$$

We conjecture that the KL criterion, rather than its symmetrized version SKL, will yield an asymptotically equivalent smoothing parameter selector.

4. Simulation Study

We performed a small simulation study to understand in part the properties of our methods, patterning the study after the analysis of the ACF data that will be presented in Section 5. There the covariate Z modeled parametrically is the vertical location of a slice and takes on eight equally spaced values between 0 and 1. The covariate, X_i, is the horizontal distance from the origin, normalized to the unit interval. We assume that the latent spatial process is stationary with a Matérn correlation function of index 5/2 (Stein, 1999), so that ${\Omega }_{ij}\left(\rho \right)=\tilde{\Omega }(d_{ij},\rho )=\exp (-d_{ij}∕\rho )\{1+d_{ij}∕\rho +(d_{ij}∕\rho {)}^2∕3\}$, where d_ij is the Euclidian distance between sites i and j. Mimicking the ACF data, we set n = 800, reflecting a 100 × 8 grid. The function chosen was $\Lambda \left(X\right)=0.5[\sin \left\{2\pi (X_i-0.5)\right\}-1]$ For each correlation structure, performed 1000 simulations. In this example we used 14-knot cubic splines with a truncated polynomial series basis. In keeping with the ACF example, where there were seven animals in a treatment group, we replicated the processes seven times. This requires only a few minor changes in notation.

There is one smoothing parameter here, κ. We used two sets of correlation parameters, chosen so that ${\sigma }_{\lambda }^2=1,\tilde{\Omega }(1,\rho )=0.5$ and ${\sigma }_{\lambda }^2=4,\tilde{\Omega }(1,\rho )=0.6$, reflecting the moderate dependence typical of ACF data and stronger dependence, respectively.

We studied the performance of the algorithms and compared two methods of smoothing parameter selection: (a) the proposed methods based on SKL criterion (Section 3); and (b) the traditional one based on CV (Ruppert et al., 2003). The integrated mean squared errors and squared bias of the probability function for both methods are given in Table 1. Effectively, we see that for estimating the function, both methods are roughly unbiased, but our method has much smaller mean squared errors, with mean squared error efficiencies being roughly 200%. The reason for this is that CV undersmooths the function, leading to increased variability, which shows up in individual data sets.

Table 1.

Results of the simulation to compare crossvalidation with our smoothing parameter selector MASKL. The methods are as follows: “MASKL” refers to the method based on minimization of MASKL (criterion described in Section 3), “CV” refers to the method based on crossvalidation, “ISB” is integrated squared bias and “IMSE” is integrated mean squared error of the probability function. Our method, MASKL, clearly dominates crossvalidation in terms of mean squared error.

Method	ISB(10-2)	IMSE (10-2)
	ψ = 0.5, Ω(ρ) = 0.5
MASKL	0.003	0.246
CV	0.005	0.553
	ψ = 0.8, Ω(ρ) = 0.6
MASKL	0.002	0.469
CV	0.003	1.425

Of course, our algorithm allows us to estimate the correlation along with the mean function (see Table 2). Included in Table 2 are the mean estimates of $\psi ={\sigma }_{\lambda }^2∕(1+{\sigma }_{\lambda }^2)$ and Ω(ρ), along with the 2.5th and 97.5th percentiles over the simulated data sets. The estimators appear to be working satisfactorily.

Table 2.

Results of the simulation for 1000 data sets under the first scenario. The first column is the method: “MASKL” refers to the method based on minimization of MASKL (criterion described in Section 3). Columns 2-4 and 5-7 are mean, 2.5th, 97.5th percentiles for ψ and Ω(ρ), respectively.

Method	$\widehat{\psi }$	$\widehat{\psi }.025$	$\widehat{\psi }.975$	$\Omega \left(\widehat{\rho }\right)$	$\Omega \left(\widehat{\rho }\right).025$	$\Omega \left(\widehat{\rho }\right).975$
ψ = 0.5, Ω(ρ) = 0.5
MASKL	0.440	0.362	0.533	0.570	0.452	0.656
ψ = 0.8, Ω(ρ) = 0.6
MASKL	0.764	0.664	0.972	0.562	0.454	0.758

We also addressed the concern whether the asymptotic theory works with the size of the data set we have. We note that asymptotic formulas for variances of the nonparametric function estimates agree with Monte Carlo variances in our simulations. The overall coverage probability averaged across all X values was 95.2% and 94.4% for the case where ${\sigma }_{\lambda }^2=1,\tilde{\Omega }(1,\rho )=0.5$ and ${\sigma }_{\lambda }^2=4,\tilde{\Omega }(1,\rho )=0.6$, respectively, for the selected level of 95%. Let ${\widehat{\mu }}_{0.5}^{\left(i\right)}$ be an estimated mean at X = 0.5 obtained based on the results from ith simulation run, i = 1, …, 1000. Figure 1 shows Q-Q plots for $\{{\tilde{\mu }}_{0.5}^{\left(i\right)}{\}}_{i=1}^{1000}$ for both simulation scenarios. The p-values for Shapiro-Wilks test for normality were 0.7403 and 0.4201 for both simulations scenarios, respectively. One can clearly see that estimators have a nearly normal distribution.

Figure 1.

Q-Q plots for $\{{\widehat{\mu }}_{0.5}^{\left(i\right)}{\}}_{i=1}^{1000}$ for ψ = 0.5, Ω(ρ) = 0.5 (left panel), and ψ = 0.8, Ω(ρ) = 0.6 (right panel).

Separately, we also studied the case of very strong correlation $({\sigma }_{\lambda }^2=1,\tilde{\Omega }(1,\rho )=0.9)$, which comes closer to an infill asymptotic framework. There was relatively small bias in both the estimated mean function and in the product $\{{\sigma }_{\lambda }^2∕(1+{\sigma }_{\lambda }^2)\}\Omega (1,\rho )$, although there was, as expected, somewhat greater bias in the component parts of this product.

5. Analysis of the ACF Experiment

In the ACF experiments, animals are exposed to a carcinogen, with half of them also exposed to radiation. They are then sacrificed, and images of the colon are obtained by various staining devices. A typical image is given in Figure 2: a color version of this is available at Web Figure A. It is not feasible in practice to measure the exact locations of ACF, so we formed a rectangular grid of locations and recorded (by hand) the existence of an ACF within each location (see Figure 3). Thus the data available to us are the grid of locations along with the indicator of an ACF. Further details on data collection can be found in Apanasovich et al. (2003).

Figure 2.

The image of the colon with aberrant crypts displayed as larger dark and distended regions with raised “bumps.”

Figure 3.

A typical rat colon. This shows how the data are made available to us: the black regions are Peyer's Patches, where aberrant crypt foci are not readable. The number on the horizontal axis means the physical distance from the distal part of the colon in centimeters.

In Figure 2 we see two major types of structures. The small white dots are normal colonic crypts, whose function is to produce cells that line the colon. The larger dark and distended regions are ACF. The response D at a given location is whether there is an ACF or not.

The data are naturally two-dimensional, although in our application, the sections are much longer (X) than wide (Z) by a factor greater than 10. Because the colon strips are not very wide, we modeled the width parametrically and the length nonparametrically, via the partially linear model (2).

A score test for longitudinal/spatial dependence developed by Apanasovich et al. (2003) indicated the presence of spatial dependencies in these data. This finding is interesting because it suggests that damage to the colon is localized regionally, and thus that there may be areas in which greater levels of damage in response to an insult could lead to focused areas of inflammatory responses, or an alteration in the release of signaling molecules that could then affect the regulation of homeostatic mechanisms in colonocytes in adjacent crypts. This localization may help explain why tumors develop from a particular ACF, but not from all ACF formed in response to a carcinogen insult.

Having shown dependence, two questions immediately arise: the extent of the dependence and the nature of the rate of ACF formation depending on the location within the colon. Our hope in this experiment is to identify regions of high ACF formation, and then to see whether regions of high ACF formation are also the regions of high tumor formation predicted by biologists.

In this study we used 14 rats, 7 irradiated and 7 nonirradiated animals that were sacrificed at 6 weeks. Here X_i is the horizontal distance from the distal part of the colon normalized to the unit interval, and Z is the more compact vertical distance. The correlation is modeled with respect to the nominal distance between grid cell, such that the distance between the closest horizontal or vertical neighbors is 1.

We fit a separate partially linear model in each rat, but with the correlation parameters and the smoothing parameter fixed between rats in a treatment group. We formed the composite likelihood using all pairs of sites less than three units apart. For the fixed effects, we used 10-knot cubic splines with the truncated polynomial basis, and the rats were allowed their own smoothing parameters.

In practice, spatial data are observed at a finite number of points and it is not clear which asymptotic framework to use, infill or increasing domain. Physically, we think increasing domain asymptotics makes sense in our example, and of course they have been used in many other contexts to which our methods would apply if splines were used. As the horizontal location we call X increases, we are actually going further out from the proximal (front) part of the colon to the distal (end) of the colon. Thus, neighbors at the proximal part of the colon are 50 times closer in physical distance than they are from locations at the distal region. We also believe that once one takes into account background rate for given rats, there is no reason to think that ACF formation at the front of the colon is highly predictive of ACF formation at the end of the colon. Under increasing domain asymptotics one would expect the correlation to decrease quickly. To test this we ran score tests similar to the one described in Apanasovich et al. (2003) and found that the correlation is not statistically significant at the distance greater than 3 units apart. Moreover, simulations suggested that under increasing domain asymptotics, the distributions of our estimators approximate the finite-sample distributions of those estimators well. Therefore we have reasonably good evidence to support the use of increasing domain asymptotic framework for our data set, and of course more generally this type of asymptotics is often used.

5.1 Results and Model Checks

As described above, Apanasovich et al. (2003) note that a score test indicates spatial correlation. To quantify this, we fit the stationary Matérn correlation model with index 5/2 (see Section 2.2), obtaining ${\widehat{\sigma }}_{\lambda }^2=0.48,\tilde{\Omega }(1,\widehat{\rho })=0.28$ for irradiated animals and ${\widehat{\sigma }}_{\lambda }^2=2.62,\tilde{\Omega }(1,\widehat{\rho })=0.20$ for nonirradiated animals. Additional tests based upon our asymptotic theory showed that the estimates for ψ and $\tilde{\Omega }(1,\rho )$ for irradiated and nonirradiated animals are significantly different from 0.

The probability of ACF formation for each group was calculated as an average of individual probabilities. Figure 4 shows the estimated probability of ACF formation as a function of normalized distance from the distal part of the colon for each of the two groups. The nonsolid lines correspond to confidence limits that we constructed the following way. First, we built pointwise confidence intervals for each rat, then we averaged confidence intervals for rats belonging to the same group. This figure suggests three important possible conclusions. First, the shapes are complex, and certainly neither constant nor linear. Second, overall the irradiated and nonirradiated groups are different in their ACF formation, with the former having higher ACF formation overall.

Figure 4.

Estimated probabilities of ACF formation with 95% confidence limits. The solid line corresponds to the irradiated group and the dashed line corresponds to the nonirradiated group. X is the normalized distance from the distal part of the colon.

We did two model checks. First, our model fits indicate that the irradiated animals have more ACF. We used a simple t-test to compare the total number of ACFs between two groups, with p-value approximately 0, thus confirming this finding.

Second, and of more potential importance, is the finding that ACF formation is not uniform across the colon, and indeed seems greatest near the middle part of the colon for both the nonirradiated and irradiated rats. This was initially surprising to us, because we expected that in rats who were allowed to live longer, most tumors would be found in the distal region, rather than roughly in the middle.

We used t-tests to show that the data support the claim that there is a higher response in the middle as follows. For each rat we computed the mean of ACF indicators for three regions in the colon: low, middle, and upper, which correspond to the normalized horizontal distance from the proximal part of the colon to be equal to [0.0, 0.3], (0.3, 0.8], and (0.8, 1.0], respectively. The rats are independent by design, therefore we combined the results across the rats. Then we ran paired t-tests. For the rats in the irradiated group, the middle region versus the low region had a t-statistic of 6.13, whereas the middle region versus the upper region had a t-statistic of 1.18. For the rats from the nonirradiated group, the middle region versus the low region had a t-statistic of 6.06, whereas the middle region versus the upper region had a t-statistic of 3.99. Therefore, there is empirical evidence that the probability of ACF formation is higher in the middle.

5.2 Selection of Pairs for Composite Likelihood

In our example, in computing composite likelihood we have selected pairs that were less than L units apart with L = 3. This choice was made on the basis of computational feasibility, as well as the fact that even for a relatively high correlation of 0.50 between nearest neighbors, the correlation between sites 3 or more units apart for the chosen correlation structure is less than 0.02.

Interestingly, there is evidence in the literature that using composite likelihood pairs that are much less correlated than other pairs may have a deleterious effect on the performance of composite likelihood. Varin and Vidoni (2007) in their Figure 1 describe an AR(1) process in time series with an autocorrelation of 0.95. They find that the efficiency of composite likelihood reaches its maximum when pairs up to 6 lags apart: these pairs have a correlation ≥0.73. However, when lags up to 20 were considered, that is, a correlation of 0.36 at lag 20, the mean squared error efficiency for estimating the correlation decreased by approximately 25%.

Varin and Vidoni's calculations indicate that essentially un-correlated pairs should not be used when trying to estimate correlation parameters. In our case, this means pairs more than 3 units apart.

To check whether the choice L = 3 was appropriate, we computed estimators using L = 2 and 4. The estimated probabilities of ACF formation for all three cases (L = 2, 3, and 4) are almost the same. However, confidence intervals using L = 2 are 1.23 times wider compared to L = 3 and confidence intervals for L = 4 are only slightly narrower (by the factor 0.98) than for L = 3. The number of terms for L = 4 is almost 1.6 times larger than for L = 3, which increases computational effort, especially when calculating the standard errors. Therefore, we believe that our choice L = 3 is appropriate for the data considered.

6. Discussion and Extensions

Summarizing briefly, we have proposed models for binary data where a fixed effects structure is a combination of parametric and nonparametric models based upon fixed-knot penalized splines. We have focused on the partially linear model, although extensions to single-index models are incorporated into the general theory in Web Appendix A.

The penalized splines have penalty parameters that must converge to zero asymptotically: we derived rates for these parameters that do not lead to an asymptotic bias. We also used the KL distance to develop a data-driven method for selecting a proper amount of smoothing and derived the optimal rate of convergence for penalty parameters by selecting this way. Simulation evidence was positive: even for modeling the mean function our methods did much better than those based on CV. The method is a novel way of performing effective penalization of regression splines in spatial data situations.

The nonparametric method using ideas from Gallant (1987, p. 533) can be implemented for estimating the “meat” of the sandwich. This method is more general and can be applied for problems other than our binary case. This method will be reported elsewhere, but we found that it gave roughly the same answers and small-sample behavior as the exact calculation.

We applied the methods to the data from an experiment on the genesis of colon cancer. We identified regions of high ACF formation that were initially surprising to us but that upon examination of other data actually correspond to regions of high tumor formation. Biologically, this provides a quantification of the localization of ACF formation as precursors to tumors.

APPENDIX

Composite Likelihood with Sets of Size k

Equation (3) is replaced by

$$\begin{array}{cc}\hfill \mathrm{pr}& (D_{i_1}=1,\dots ,D_{i_k}=1\mid {\mu }_{i_1}^{\ast },\dots ,{\mu }_{i_k}^{\ast },{\sigma }_{\lambda }^2,\rho )\hfill \\ \hfill & ={\Phi }_k\left[{\mu }_{i_1}^{\ast },\dots ,{\mu }_{i_k}^{\ast };\frac{{\sigma }_{\lambda }^2}{(1+{\sigma }_{\lambda }^2)}{\left\{{\Omega }_{ij}\left(\rho \right)\right\}}_{i,j\in \{i_1,\dots ,i_k\}}\right].\hfill \end{array}$$ (A.1)

If we use sets of size k, then we need to keep track of the values of the sets and their probabilities. To do this, we define

$$\begin{array}{cc}\hfill & Z_{i_1,\dots ,i_k}^{(j_1,\dots ,j_k)}=I(D_{i_1}=j_1,\dots ,D_{i_k}=j_k);\hfill \\ \hfill & {\pi }_{i_1,\dots ,i_k}^{(j_1,\dots ,j_k)}=\mathrm{E}(Z_{i_1,\dots ,i_k}^{(j_1,\dots ,j_k)}\mid {\mu }_{i_1}^{\ast },\dots ,{\mu }_{i_k}^{\ast },{\sigma }_{\lambda }^2,\rho ).\hfill \end{array}$$ (A.2)

Using the marginal probabilities (A.1), we can write the likelihood at distinct locations (i₁, …, i_k) as

$${\mathcal{L}}_{i_1,\dots ,i_k}\left(\Theta \right)=\sum _{j_1,\dots ,j_k=0}^1{Z}_{i_1,\dots ,i_k}^{(j_1,\dots ,j_k)}\log \left\{{\pi }_{i_1,\dots ,i_k}^{(j_1,\dots ,j_k)}(\beta ,\theta )\right\}.$$

It is useful to note that ${\mathcal{L}}_{i_1,\dots ,i_k}\left(\Theta \right)$ is an actual likelihood function, albeit for a reduced set of data.

We propose to maximize a weighted penalized composite likelihood. Let $w_{i_1,\dots ,i_k}$ be the weights, for example, the indicator that the maximum distance between two locations from a set of locations (i₁, …, i_k) is less than a specified value apart, a choice that is useful to cut down on the size of the summations. It is assumed that $w_{i_1,\dots ,i_k}$ does not depend upon $(D_{i_1},\dots ,D_{i_k})$ or any parameters.

For fixed smoothing parameter, κ, the composite likelihood estimator is denoted by $\widehat{\Theta }\left(\kappa \right)$ and is obtained by maximizing over Θ

$${\tilde{\mathcal{L}}}_{\mathcal{W}}(\Theta ,\kappa )={\mathcal{W}}^{-1}\sum _{i_1,\dots ,i_k=1}{w}_{i_1,\dots ,i_k}{\mathcal{L}}_{i_1,\dots ,i_k}\left(\Theta \right)-(1∕2)\kappa {\eta }^{\mathrm{T}}\mathcal{G}\eta ,$$ (A.3)

where $\mathcal{W}={\Sigma }_{i_1,\dots ,i_k}{w}_{i_1,\dots ,i_k}$.

With a slight abuse of notation, for composite likelihood, we have that

$$\begin{array}{cc}\hfill \mathrm{SKL}\left\{\widehat{\Theta }\left(\kappa \right)\right\}=& {\mathcal{W}}^{-1}\sum _{i_1,\dots ,i_k=1}^n\sum _{j_1,\dots ,j_k=0}^1{w}_{i_1,\dots ,i_k}\hfill \\ \hfill \times & [{\pi }_{i_1,\dots ,i_k}^{(j_1,\dots ,j_k)}\left({\Theta }_0\right)-{\pi }_{i_1,\dots ,i_k}^{(j_1,\dots ,j_k)}\left\{\widehat{\Theta }\left(\kappa \right)\right\}]\hfill \\ \hfill \times & (\log \left\{{\pi }_{i_1,\dots ,i_k}^{(j_1,\dots ,j_k)}\left({\Theta }_0\right)\right\}\hfill \\ \hfill & -\log \left[{\pi }_{i_1,\dots ,i_k}^{(j_1,\dots ,j_k)}\left\{\widehat{\Theta }\left(\kappa \right)\right\}\right]),\hfill \end{array}$$

where ${\pi }_{i_1,\dots ,i_k}^{(j_1,\dots ,j_k)}(\cdot )$ are defined in (A.2).

Smoothing Parameter Selection

The main purpose of this section is to show that smoothing parameters should be of order $O\left({\mathcal{W}}^{-1}\right)$, and to sketch the basic algebra to obtain these smoothing parameters. Here we need to be more specific about the definition of Θ, which we take by convention to be Θ = (η^T, ζ, θ^T)^T.

Define the derivative of ${\mathcal{L}}_{i_1,\dots ,i_k}\left(\Theta \right)$ as ${\mathcal{A}}_{i_1,\dots ,i_k}\left(\Theta \right)$. Define $\mathcal{T}(\kappa ,\Theta )=\kappa \mathcal{M}\Theta$, where $\mathcal{M}=\text{diag}(\mathcal{G},0I_{q\zeta -1})$. Set $\kappa ={\mathcal{W}}^{-\mathcal{V}}c$ for some constant c. Assume that ν ≥ 1/2, then by theorem 1 in Web Appendix A, $\widehat{\Theta }\left(\kappa \right)-{\Theta }_0=O_p\left({\mathcal{W}}^{-1∕2}\right)$.

By Taylor expansion after some algebraic manipulations,

$$\begin{array}{cc}\hfill \mathrm{SKL}=& {\mathcal{W}}^{-1}\sum _{i_1,\dots ,i_k}{\left\{{\mathcal{A}}_{i_1,\dots ,i_k}\left({\Theta }_0\right)\right\}}^{\mathrm{T}}(\widehat{\Theta }-{\Theta }_0)\hfill \\ \hfill & \times {(\widehat{\Theta }-{\Theta }_0)}^{\mathrm{T}}{\mathcal{A}}_{i_1,\dots ,i_k}\left({\Theta }_0\right)+O_p\left(W^{-3∕2}\right).\hfill \end{array}$$

It is easy to see that

$$\begin{array}{cc}\hfill \mathrm{SKL}=& {\mathcal{W}}^{-1}\sum _{i_1,\dots ,i_k}{\left\{{\mathcal{A}}_{i_1,\dots ,i_k=1}^n\left({\Theta }_0\right)\right\}}^{\mathrm{T}}(\widehat{\Theta }\ast -{\Theta }_0)\hfill \\ \hfill & \times {(\widehat{\Theta }\ast -{\Theta }_0)}^{\mathrm{T}}{\mathcal{A}}_{i_1,\dots ,i_k}\left({\Theta }_0\right)+O_p\left(W^{-3∕2}\right),\hfill \end{array}$$

where ${\widehat{\Theta }}^{\ast }={\Theta }_0+\{{\Sigma }_{\mathcal{W},0}+M\left(\kappa \right){\}}^{-1}{\mathcal{W}}^{-1}{\Sigma }_{i_1,\dots ,i_k}{\mathcal{A}}_{i_1,\dots ,i_k}\left({\Theta }_0\right)-\mathcal{T}(\kappa ,{\Theta }_0)\}$. Associated with SKL is its expected value in this asymptotically equivalent version of $\widehat{\Theta }$, namely,

$$\begin{array}{cc}\hfill \text{MASKL}=& {\mathcal{W}}^{-1}\sum _{i_1,\dots ,i_k}\left\{{\mathcal{A}}_{i_1,\dots ,i_k}\left({\Theta }_0\right)\right\}\mathrm{E}\hfill \\ \hfill & \times \left\{{(\widehat{\Theta }\ast -{\Theta }_0)(\widehat{\Theta }\ast -{\Theta }_0)}^{\mathrm{T}}\right\}{\mathcal{A}}_{i_1,\dots ,i_k}\left({\Theta }_0\right).\hfill \end{array}$$

Recall that ${\Sigma }_{\mathcal{W},0}=E\{\partial {\Psi }_{\mathcal{W}}(\Theta ,\kappa )\partial {\Theta }^{\mathrm{T}}\}$ is the exterior noninverted part and ${\Sigma }_{\mathcal{W},0}+{\Sigma }_{\mathcal{W},c}$ is the interior covariance matrix in the sandwich formula (6). We discussed in Section 2.6 how to estimate these components

Note thet ${\mathcal{W}}^{-1}{\Sigma }_{i_1,\dots ,i_k}{\mathcal{A}}_{i_1,\dots ,i_k}\left({\Theta }_0\right)\{{\mathcal{A}}_{i_1,\dots ,i_k}\left({\Theta }_0\right){\}}^{\mathrm{T}}={\Sigma }_{\mathcal{W},0}+o_p\left({\mathcal{W}}^{-1∕2}\right)$. Define $R\left(\kappa \right)=\{{\Sigma }_{\mathcal{W},0}+M\left(\kappa \right){\}}^{-1}$, $S\left(\kappa \right)={\mathcal{W}}^{-1}({\Sigma }_{\mathcal{W},0}+{\Sigma }_{\mathcal{W},c})+\mathcal{T}(\kappa ,{\Theta }_0){\mathcal{T}}^{\mathrm{T}}(\kappa ,{\Theta }_0)$. Then we have that the asymptotically equivalent version is

$$\widetilde{\mathrm{MASKL}}\left(\kappa \right)=\text{trace}\left\{R\left(\kappa \right)S\left(\kappa \right)R\left(\kappa \right){\Sigma }_{\mathcal{W},0}\right\}.$$ (A.4)

It is evident by inspection that in order to minimize (A.4), we must have $\kappa =O\left({\mathcal{W}}^{-1}\right)$ as claimed. The minimizer of (A.4) solves $0=\partial \widetilde{\text{MASKL}}\left(\kappa \right)∕\partial \kappa$. Then there is a matrix $\mathcal{H}(\kappa ,{\Theta }_0,{\Sigma }_{\mathcal{W},0},{\Sigma }_{\mathcal{W},c})$ and a vector $\mathcal{J}(\kappa ,{\Theta }_0,{\Sigma }_{\mathcal{W},0},{\Sigma }_{\mathcal{W},c})$, where ${\mathcal{H}}^{-1}(\cdot )$ and $\mathcal{J}(\cdot )$ are both of order O(1), such that to terms of order o(W^-1), the nontrivial equation for κ is $\kappa ={\mathcal{W}}^{-1}{\mathcal{H}}^{-1}(\kappa ,{\Theta }_0,{\Sigma }_{\mathcal{W},0},{\Sigma }_{\mathcal{W},c})\mathcal{J}(\kappa ,{\Theta }_0,{\Sigma }_{\mathcal{W},0},{\Sigma }_{\mathcal{W},c})=0$. The latter equation can be solved iteratively. However, to terms of the first order, the minimizer of (A.4) solves

$$0=\left\{\frac{\partial \widetilde{\text{MASKL}}(0,0)}{\partial \kappa }\right\}+\left\{\frac{{\partial }^2\widetilde{\text{MASKL}}(0,0)}{\partial \kappa \partial \kappa }\right\}\kappa ,$$

which leads to a linear equation $\kappa ={\mathcal{W}}^{-1}{\tilde{\mathcal{H}}}^{-1}\times ({\Theta }_0,{\Sigma }_{\mathcal{W},0},{\Sigma }_{\mathcal{W},c})\times \tilde{\mathcal{J}}({\Theta }_0,{\Sigma }_{\mathcal{W},0},{\Sigma }_{\mathcal{W},c})$. All terms mentioned above are straightforward to compute and estimate.

7. Supplementary Materials

A Web Appendix containing general asymptotic theory for this problem, as well as a color version of Figure 2, are both available under the Paper Information link at the Biometrics website http://www.biometrics.tibs.org.