Performance of Information Criteria for Spatial Models

Abstract

Model choice is one of the most crucial aspect in any statistical data analysis. It is well known that most models are just an approximation to the true data generating process but among such model approximations it is our goal to select the “best” one. Researchers typically consider a finite number of plausible models in statistical applications and the related statistical inference depends on the chosen model. Hence model comparison is required to identify the “best” model among several such candidate models. This article considers the problem of model selection for spatial data. The issue of model selection for spatial models has been addressed in the literature by the use of traditional information criteria based methods, even though such criteria have been developed based on the assumption of independent observations. We evaluate the performance of some of the popular model selection critera via Monte Carlo simulation experiments using small to moderate samples. In particular, we compare the performance of some of the most popular information criteria such as Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC), and Corrected AIC (AICc) in selecting the true model. The ability of these criteria to select the correct model is evaluated under several scenarios. This comparison is made using various spatial covariance models ranging from stationary isotropic to nonstationary models.

1 Introduction

Model selection is an important part of any statistical analysis. In fact, G.E.P. Box once remarked that “Models, of course, are never true, but fortunately it is only necessary that they be useful” (Box, 1979, p.2). In this sense among all the approximating models, we would like to use the one that is “closest” to the true data generating process. In practice, most researchers often consider a given set of plausible models postulated and formulated based on the background knowledge and preliminary data analysis. As the choice of one model over the another can make a substantial difference in statistical inference, we need a “good” method to select among a set of postulated models. Thus, a model comparison tool is required to identify the “best” model (if possible) among several candidate models. Many researchers have examined this issue and various methods for selecting the “best” model have been suggested. Examples include hypothesis testing, cross validation (Stone, 1974), coeficient of determination, R², Mallows' C_p (Mallows, 1973), and several information criteria.

Information criteria are probably one of the most widely used tools in model selection. However, there have been few investigations of the performance of these criteria in a spatial modeling context. Hoeting et al. (2006) discussed the issue of model selection for geostatistical data. They explored the effect of spatial correlation on variable selection using Akaike Information Criterion (AIC) (Akaike, 1973) in geostatistical models. Their simulation results showed that spatially corrected AIC (AICc) (Sugiura, 1978; Hurvich and Tsai, 1989) outperforms independent AICc which ignores spatial correlation in the variable selection.

In particular, few studies have compared the performance between different information criteria like AIC and Bayesian Information Criterion (BIC) (Schwarz, 1978). Hence, little is known about the relative performance of different information criteria. Currently, no consensus exists on the best criterion for spatial model selection. Of particular interest is how these different criteria perform with various spatial covariance models. We explore this issue via extensive Monte Carlo simulation experiments using a wide variety of spatial models. The purpose of this study is to examine the performance of different information criteria for use in spatial covariance model selection. We compare the performance of traditional information criteria such as AIC, BIC, and AICc. This comparison is made using various spatial covariance models ranging from stationary isotropic to nonstationary models.

The remainder of this article is organized as follows. Section 2 provides brief descriptions of various information criteria used for model selection such as AIC, BIC and AICc. In Section 3, we describe various spatial covariance models such as stationary isotropic and anisotropic, and nonstationary models that will be used as an illustration to generate data from a specific covariance model. Section 4 presents the results from simulations which compare the performance of AIC, BIC and AICc with regard to their ability to identify the true model among various spatial covariance models. Discussions and future research are summarized in Section 5.

2 Information Criteria for Model Selection

Information criteria have played an important role in model selection. These are based on the Kullback-Leibler (K-L) information (Kullback and Leibler, 1951) which is defined as

$$I\left(f,g\right)=\int f(x)log\left(\frac{f(x)}{g(x|\theta )}\right)dx,$$

where θ represents parameters in model g. Here, I(f,g) can be interpreted as the “information lost when the model g is used to approximate full reality or truth f” (Burnham and Anderson, Section 2.1, 2002). Hence, the model that minimizes I(f, g) will be considered as the best model. However, I(f, g) cannot be used directly in model selection because f(x) and θ are not known.

A variety of information criteria have been proposed to be used in model selection. These include AIC, AICc, Takeuchi's Information Criterion (TIC) (Takeuchi, 1976), and QAIC and QAICc (Quasi-likelihood modifications to AIC and AICc) (Lebreton et al., 1992). These criteria are estimates of the relative K-L information between f and g and are based on the concept that a true f may not be included in the set of models being evaluated (Burnham and Anderson, 2002). On the other hand, several criteria such as Bayesian Information Criterion (BIC) (Schwarz, 1978) have been developed assuming that a true f belongs to the set of models being considered as candidate models (Burnham and Anderson, 2002). Deviance Information Criterion (DIC) (Spiegelhalter et al., 2002) is analogous to AIC from a Bayesian perspective.

A substantial advantage in using information criteria is that these are valid even when the models being compared and not nested. Traditional likelihood ratio tests are defined only for nested models, and this represents a limitation in the use of hypothesis testing in model selection.

Most information criteria have a form that consists of two terms. In general, the first term is the negative log-likelihood, multiplied by two, of the data calculated with the maximum likelihood estimates of the parameters. The second term differs between different information criteria. The second term is often interpreted as a penalty for model complexity. Hence, it increases as the number of parameters in the model increases.

In model selection using information criteria, the model that minimizes information criteria is declared as the best model among the set of models under consideration. In the following sections, we describe commonly used information criteria such as AIC, AICc, and BIC. It should be noted that the theoretical validity of the information criteria are generally based on the asymptotic theory. In case of spatial models the theoretical results of Mardia and Marshall (1984) and Sweeting (1980) can be used to establish the consistency and asymptotic normality of the MLE under some strong regularity conditions. These asymptotic results can then be used to justify the use of AIC and/or BIC. However we have not pursued such theoretical investigations in this paper and remains a topic for further studies. The focus of this article is to study the finite sample performances of the information criteria using empirical studies. Even when theoretical results are available, it is generally difficult to determine exactly how large the sample size should be in order to use the large-sample theory. Our results provide some guidelines and insights on the use (and abuse) of information criteria for small to moderate sample sizes.

2.1 Akaike Information Criterion (AIC)

Akaike Information Criterion (AIC) (Akaike, 1973) is one of the most well known information criteria used in model selection. AIC is an estimate of relative, expected K-L information (Kullback and Liebler, 1951) between a fitted model and the true model. AIC is defined as

$$\text{AIC}=-2\text{log}\left[L(\widehat{\theta }|X)\right]+2p,$$ (1)

where logL(θ̂|X) represents the log-likelihood function of the maximum likelihood estimator (MLE), θ̂, given the observed data X, and p is the dimension of the parameter θ. The first term can be interpreted as a measure of lack of model fit, while the second term can be interpreted as a penalty for increasing the dimension of the model. The second term is the asymptotic bias-correction term derived from an asymptotic estimator of relative, expected K-L information (Burham and Anderson, 2002).

In application, we compute AIC for each of the candidate models and select the model with the smallest value of AIC. Models producing smaller values of AIC can be thought of as having a smaller difference from the true model. AIC provides a simple and effective means for the selection of the best approximating model to the true model (Burnham and Anderson, 2002).

With regard to general linear models, AIC is known to perform relatively well for small samples, however the criterion does not tend to select the true model in large samples (Hurvich and Tsai, 1990).

2.2 Corrected Akaike Information Criteria (AICc)

As an approximately unbiased estimator of the expected K-L information of a fitted model, AIC has been shown to be strongly negatively biased in small samples (Sugiura, 1978; Hurvich and Tsai, 1989). Hurvich and Tsai (1989) derived a bias-corrected version of AIC, and termed it as AICc. They argued that AICc should be used in place of AIC, when the dimension of the model is large relative to sample size or when n is small, for any p. The AICc is defined as

$$\begin{array}{l}\text{AICc}=-2\text{log}\left[L(\widehat{\theta }|X)\right]+2p\left(\frac{n}{n-p-1}\right)\\ =-2\text{log}\left[L(\widehat{\theta }|X)\right]+2p+\frac{2p\left(p+1\right)}{n-p-1}\\ =\text{AIC}+\frac{2p\left(p+1\right)}{n-p-1},\end{array}$$ (2)

where n is the sample size and p is the number of parameters in the model. AICc has an additional bias-correction term compared to AIC, which is adjusted to the parameter complexity p and the sample size n. However, if n is large with respect to p, then this additional bias-correction is negligible and AIC should perform well. Burnham and Anderson (2002) advocated the use of AICc, in particular, when the ratio n/p < 40 for the model with the largest value of p. If n/p is sufficiently large, then AIC and AICc are similar and will tend to select the same model. They also mentioned that AICc should be used in practice, because AICc converges to AIC as n gets large, with p fixed.

2.3 Bayesian Information Criterion (BIC)

Along with AIC, Bayesian Information Criterion (BIC) (Schwarz, 1978) is currently among the most commonly used information criteria in model selection. BIC is usually explained in terms of Bayesian theory, especially as an approximation of the Bayes factor, which is the ratio of the marginal likelihoods for two models. Unlike AIC, BIC is not an estimate of relative expected K-L information (Burnham and Anderson, 2002). BIC is defined as

$$\text{BIC}=-2\text{log}\left[L(\widehat{\theta }|X)\right]+p\text{log}(n),$$ (3)

where log [L(θ̂|X)] again represents the log-likelihood function of θ̂, which is the maximum likelihood estimator (MLE) based on the observed data X; p is the number of parameters in the model, and n is the sample size. The first term of BIC is same as that of AIC. However, the second term penalizes the model with increased model complexity, or larger p, and sample size as well. AIC and BIC differ only by the coefficient multiplying the number of parameters, in other words, by how strongly they penalize large models. In general, models chosen by BIC are more parsimonious than those chosen by AIC. As usually used, one computes the BIC for each model and selects the model with the smallest criterion value. In contrast to AIC, BIC tends to choose the true model in large samples. However, BIC has also known to perform poorly in small samples in the context of general linear models (Hurvich and Tsai, 1990).

3 Spatial Models

We consider various geostatistical models that are popularly used for point-referenced data. In particular, we evaluate the performance of information criteria using models that range from stationary (including anisotropic) to nonstationary models.

3.1 Stationary Processes

Consider a random process {Z(s) : s ∈ D}, where D is a fixed subset of ℜ^d. Assume that the random process Z(·) satisfies

$$E(Z(s))=\mu ,\text{for all}s\in D\text{and}$$ (4)

$$\text{Cov}(Z(s_i),Z(s_j))=C(s_i-s_j),\text{for all}{s}_i,s_j\in D.$$ (5)

That is, the mean does not depend on s and the covariance is a function only of the increment s_i − s_j. Then Z(·) is said to be a second-order or weak stationary process. Furthermore, if C(s_i − s_j) is a function of ‖s_i − s_j‖ only, that is, the distance between s_i and s_j, then C(·) is called isotropic. An isotropic process assumes that the correlation structure between sites is circular which indicates that the correlation depends only on the distance between sites.

One frequently used isotropic covariance function is the exponential model. Here the covariance between measurements at two locations is an exponential function of the distance between two locations,

$$\text{Cov}(Z(s_i),Z(s_j))={\sigma }^{2}exp(-\phi \parallel s_i-s_j\parallel )+{\tau }^{2}I(i=j),{\sigma }^2>0,\phi >0,{\tau }^2>0,$$ (6)

where ‖s_i − s_j‖ is the distance between sites s_i and s_j, and I denotes the indicator function. Here σ² and φ are positive parameters called the partial sill and the decay or inverse range parameter, respectively. When i = j, d_ij = 0 and C(d_ii) = Var(Z(s_i)) is often expanded to τ² + σ², where τ² > 0 is called a nugget effect, and τ² + σ² is called the sill.

Many other parametric models for the isotropic covariance function are also commonly used (Schabenberger and Gotway, 2005, Section 2.1). Isotropic processes are popular because a number of relatively simple parametric forms are available.

If dependence between Z(s_i) and Z(s_j) is a function of both the distance and the direction of s_i − s_j, then the process Z is called anisotropic. Hence, the covariance function, C(s_i − s_j) is no longer purely a function of distance between two spatial locations, s_i and s_j.

Sometimes the anisotropy can be corrected by a linear transformation of the increment vector s_i − s_j. This anisotropy is known as geometric anisotropy and gives elliptical contours for the correlation. Specifically, the geometric anisotropy is corrected by (i) a rotation of the coordinate system to align the major and minor axes of the elliptical contours, and (ii) a compression of the major axis to make the contours spherical. Following Schabenberger and Gotway (2005, p.151), the anisotropy matrix A is thus defined as,

$$A\equiv A(\lambda ,\gamma )=\left[\begin{array}{cc}1& 0\\ 0& \lambda \end{array}\right]\left[\begin{array}{cc}cos\gamma & -sin\gamma \\ sin\gamma & cos\gamma \end{array}\right],$$ (7)

where λ and θ are the anisotropy ratio for compression and the anisotropy angle for rotation, respectively. Here λ equals the ratio of the ranges in the directions of the major and minor axes of the elliptical contours. Geometric anisotropy is common for processes that evolve along particular directions. For example, airborne pollution emitted from an industrial plant will likely evolve along the wind directions (Schabenberger and Gotway, 2005, p.151).

In general, geometric anisotropy can be incorporated in the isotropic model by correcting distances. For instance, we can incorporate geometic anisotropy in the exponential model (6),

$$\text{Cov}(Z(s_i),Z(s_j))={\sigma }^{2}exp(-\phi \Vert A(s_i-s_j)\Vert )+{\tau }^{2}I(i=j),$$ (8)

where A is the anisotropy matrix in (7).

3.2 Nonstationary Processes

We consider a class of parametric nonstationary covariance models proposed by Hughes-Oliver et al. (1998). They incorporate nonstationarity in the covariance model driven by a point source, (e.g., the center of a wafer in semiconductor processing). Their covariance model for a point source at location c is

$$\text{Cov}[Z(s_i),Z(s_j)]={\sigma }^{2}exp\{-\phi h_{ij}exp\{\alpha |c_i-c_j|+\beta min[c_i,c_j]\}\}+{\tau }^{2}I(i=j),$$ (9)

where h_ij = ‖s_i − s_j‖, c_i = ‖s_i − c‖, and c_j = ‖s_j−c‖. Here c_i and c_j are the distances of sites s_i and s_j from the point source c, respectively, and α, β ≥ 0. This covariance model is nonstationary because the correlation between sites s_i and s_j depends on the distances between sites and the point source through c_i and c_j.

The covariance model (9) can be thought of as a generalization of the exponential model for an isotropic process. Note that when α = β = 0 in (9), the covariance model (9) reduces to the exponential model (6). Here (9) assumes that the effects of the point source are circular, that is, point source isotropy. Hence the correlation depends only on the distance between sites and on the distance between a site and the point source.

We can also incorporate point source anisotropy in the nonstationary point source isotropic model in a similar way as shown in (8). Schabenberger and Gotway (2005, p.423) presented point source anisotropy incorporated in the model (9),

$$\text{Cov}[Z(s_i),Z(s_j)]={\sigma }^{2}exp\{-\phi h_{ij}^{\ast }exp\{\alpha |c_i^{\ast }-c_j^{\ast }|+\beta min[c_i^{\ast },c_j^{\ast }]\}\}+{\tau }^{2}I(i=j),$$ (10)

where $h_{ij}^{\ast }=\Vert A(s_i-s_j)\Vert ,{\text{c}}_i^{\ast }=\Vert A_c(s_i-c)\Vert ,c_j^{\ast }=\Vert A_c(s_j-c)\Vert$ and A, A_c are the anisotropy matrices in (7). Notice that models (6), (8) and (9) are nested within the model (10) and one may use the likelihood ratio test (LRT) based on Wilks theorem or Wald's test or Rao's score test, to compare each one of the models to the model (10). However, in many practical applications the models need not be nested (e.g., (8) and (9) are not nested) and hence LRT may not be applicable.

4 A Simulation Study

In this simulation study, we evaluate and compare the performance of information criteria presented in Section 2 in selecting the models presented in Section 3. Of particular interest is how these criteria perform with different spatial covariance models. Specifically, we compare the performance of these information criteria with regard to their ability to discriminate the true model under various spatial covariance models, parameter values, and sample sizes.

4.1 Covariance Models

We consider following four different forms of exponential models for spatial covariance functions.

1. Σ₁: Exponential Isotropic Model,

   $${\sum }_{ij}=\text{Cov}[Z(s_i),Z(s_j)]={\sigma }^{2}exp\{-\phi h_{ij}\}+{\tau }^{2}I(i=j),$$

2. Σ₂: Exponential Anisotropic Model,

   $${\sum }_{ij}=\text{Cov}[Z(s_i),Z(s_j)]={\sigma }^{2}exp\{-\phi h_{ij}^{\ast }\}+{\tau }^{2}I(i=j),$$

3. Σ₃: Exponential Point Source Isotropic Model,

   $${\sum }_{ij}=\text{Cov}[Z(s_i),Z(s_j)]={\sigma }^{2}exp\{-\phi h_{ij}exp\{\alpha |c_i-c_j|\}\}+{\tau }^{2}I(i=j),$$

4. Σ₄: Exponential Point Source Anisotropic Model,

   $${\sum }_{ij}=\text{Cov}[Z(s_i),Z(s_j)]={\sigma }^{2}exp\{-\phi h_{ij}^{\ast }exp\{\alpha |c_i^{\ast }-c_j^{\ast }|\}\}+{\tau }^{2}I(i=j),$$

where $h_{ij}=\Vert s_i-s_j\Vert ,h_{ij}^{\ast }=\Vert A(s_i-s_j)\Vert ,c_i=\Vert s_i-c\Vert ,c_i^{\ast }=\Vert A_c(s_i-c)\Vert ,c_j=\Vert s_j-c\Vert ,c_j^{\ast }=\Vert A_c(s_j-c)\Vert ,$ and A, A_c are anisotropy matrices in (7).

We use a Gaussian process with the above covariance models to enable likelihood inference. The most convenient assumption would be a multivariate normal distribution for the observed data. That is, suppose we have observations Z = (Z(s₁), ⋯, Z(s_n))' at known locations s_i, i = 1, ⋯, n. We then assume that

$$Z|\theta \sim N_n\left(0,\sum \left(\theta \right)\right),$$ (11)

where N_n denotes the n-dimensional normal distribution, with mean 0 and covariance (Σ(θ)), where (Σ(θ)) takes one of the four forms described above.

Specifically, we consider the following four different spatial models for our simulation studies:

1. M₁: Stationary Isotropic Model,

   $$Z|\theta \sim N_n(0,{\sum }_1\left(\theta \right)),\theta =({\sigma }^2,\phi ,{\tau }^2),$$

2. M₂: Stationary Anisotropic Model,

   $$Z|\theta \sim N_n(0,{\sum }_2\left(\theta \right)),\theta =({\sigma }^2,\phi ,{\tau }^2,\lambda ,\gamma ),$$

3. M₃: Nonstationary Point Source Isotropic Model,

   $$Z|\theta \sim N_n(0,{\sum }_3\left(\theta \right)),\theta =({\sigma }^2,\phi ,{\tau }^2,\alpha ),$$

4. M₄: Nonstationary Point Source Anisotropic Model,

   $$Z|\theta \sim N_n(0,{\sum }_4\left(\theta \right)),\theta =({\sigma }^2,\phi ,{\tau }^2,\gamma ,\lambda ,\alpha ).$$

Notice that M₁, M₂, and M₃ are nested under M₄. That is, M₁, M₂, and M₃ are special cases of M₄. Specifically, M₄ reduces to M₁ when α = 0 and A = I in the covariance function Σ₄. Also, M₄ reduces to M₂ when α = 0 and M₃ when A = I. However Σ₂ and Σ₃ are not nested and hence use of information criteria would be more useful than using LRT. Table 1 summarizes the number of parameters p in each model along with parameters θ for each model.

Table 1.

Number and list of parameters in each model

Model	p	θ
M1	3	σ2, φ, τ2
M2	5	σ2, φ, τ2, λ, γ
M3	4	σ2, φ, τ2, α
M4	6	σ2, φ, τ2, γ, λ, α

4.2 Data Generation Processes

Using the method presented in Cressie (1993, Section 3.6) to simulate point-referenced data, we simulated the spatial process at n locations, s₁, ⋯, s_n, following a multivariate normal distribution with mean vector E(Z) = 0, and covariance matrix Cov(Z) = Σ_i, i = 1, ⋯, 4, as presented in Section 4.1. We used the Cholesky decomposition which allows the covariance matrix, Σ_i, to be decomposed as the matrix product ${\sum }_i=L_i{L}_i^{\text{'}}$, where L_i is a lower triangular n×n matrix. Then we simulated Z, which satisfies the mean 0 and the covariance Σ_i through the relation Z = L_iε, where ε = (ε(s₁), ⋯, ε(s_n))' and ε(s_i)'s are iid with a standard normal distribution. We also simulated irregularly spaced n locations, s₁, ⋯, s_n, distributed uniformly on the square [0, 100] × [0, 100].

Simulated data were generated under 18 different conditions created by varying four factors of interest: the true model (M = M₁, M₂, M₃, M₄), the true parameter value for nonstationarity (α = 5, 10) and for anisotropy ratio (λ = 5, 10), and sample size (n = 50, 100). From the combinations of the true parameter values in the models, we created nine sets of data as given in Table 2. D₁ was generated from model M₁, D₂₁, D₂₂ from model M₂ with different λ, D₃₁, D₃₂ from model M₃ with different α, and D₄₁, D₄₂, D₄₃, D₄₄ from model M₄ with the combination of different λ and α. We assumed the point source to be located at the origin c = (0, 0) for model M₃ and M₄. These nine data sets were generated with three different sample sizes (n = 50, 100, 200). 100 data sets were replicated for each of the nine scenarios, and hence a total of 2700 data sets were generated for our simulation study.

Table 2.

True parameter values for each data generation process

DGP	α	γ	λ
D1	0	0	1
D21	0	π/4	5
D22	0	π/4	10
D31	5	0	1
D32	10	0	1
D41	5	π/4	5
D42	5	π/4	10
D43	10	π/4	5
D44	10	π/4	10

4.3 Results

First, we compared the covariance functions of nine scenarios with n = 50 given in Table 2 by computing the Frobenius distances between these nine covariance functions. Frobenius distance can be used to measure the distance between two matrices and to indicate the difference between these matrices. Suppose A = {α_ij} and B = {b_ij} are square matrices with the same dimension, then the Frobenius distance between these two matrices is calculated as,

$$F\left(A,B\right)=\sqrt{\sum _{i=1}^n\sum _{j=1}^n{\left(a_{ij}-b_{ij}\right)}^2.}$$ (12)

Thus the closer F(A, B) is to 0, then the more these two matrices A and B are similar. Also notice that F(A, B) = 0 if and only if A = B.

Table 3 presents the Frobenius distances between the covariance functions of nine scenarios generated using sample size n = 50. Here, Σ_i represents the covariance function of D_i, i = 1, 21, 22, 31, 32, 41, 42, 42, 44. The Frobenius distances are small between the covariance functions generated from the same model with different parameter values, in particular, between the covariance functions of nonstationary models. Σ₁ seemed very different from all of other covariance functions. Σ₂₂ was a little closer than Σ₂₁ to the covariance functions of the nonstationary models. That is, the stationary anisotropic model with a large anisotropy ratio (λ = 10) seems to be closer than the stationary anisotropic model with a small anisotropy ratio (λ = 5) to the nonstationary model. The distances of the stationary covariance functions from the nonstationary point source isotropic covariance functions are very similar to those from the nonstationary point source anisotropic covariance functions. Unlike in the stationary models, whether the covariance function is isotropic or anisotropic seemed not to make much difference in the nonstationary models. The stationary anisotropic model appeared closer to the nonstationary models than to the stationary isotropic model. The distances between nonstationary point source isotropic models and nonstationary point source anisotropic models were the smallest among the distances between different models. Similar comparative features are observed between covariance functions generated using the sample of sizes n = 100 and 200.

Table 3.

Frobenius distance between covariance functions of models

	Σ1	Σ21	Σ22	Σ31	Σ32	Σ41	Σ42	Σ43	Σ44
Σ1	0	8.350	9.745	10.972	11.032	11.120	11.132	11.128	11.133
Σ21		0	2.165	4.810	4.887	4.951	4.979	4.970	4.980
Σ22			0	3.407	3.487	3.413	3.448	3.437	3.450
Σ31				0	0.802	1.721	1.694	1.692	1.689
Σ32					0	1.767	1.741	1.707	1.705
Σ41						0	0.302	0.224	0.312
Σ42							0	0.119	0.076
Σ43								0	0.092
Σ44									0

Each of the nine scenarios was modeled using one of four different models, M₁, M₂, M₃, M₄. Each dataset was thus modeled with one correct model and three incorrect models. Models were fit using the R statistical software which uses the optim function to maximize the likelihood or equivalently to minimize −2 log[L(θ)|Z] = log |Σ(θ)|+Z^TΣ(θ)^-1Z. However it should be noted that numerical maximization of the likelihood with larger sample size (e.g. with n > 100) is computationally intensive. The determinant and the inverse of the covariance matrices that are needed to evaluate the log-likelihood were computed in R using the efficient eigen value decomposition method (see the R manual for more details). AIC, AICc, and BIC were calculated for each dataset using the formula given in (1), (2), and (3), respectively. Table 4 presents the penalty used by each criterion for each model. BIC uses a penalty almost twice as large as that of AIC and AICc. The penalties used by AIC and AICc are not much different. Only the penalty of AIC does not depend on the sample size n.

Table 4.

Penalty given by each criterion to four models with three different sample sizes

	n = 50				n = 100				n = 200
	M1	M2	M3	M4	M1	M2	M3	M4	M1	M2	M3	M4
AIC	6	10	8	12	6	10	8	12	6	10	8	12
BIC	11.7	19.5	15.6	23.5	13.8	23.0	18.4	27.6	15.9	26.5	21.2	31.8
AICc	6.5	11.4	8.9	13.9	6.2	10.6	8.4	12.9	6.1	10.3	8.2	12.4

We examined whether AIC, AICc, and BIC can correctly identify the true underlying model when various spatial models are fit to a particular data set which is generated from one of the models fitted. Results are summarized in Tables 5-8. Each table presents the percentage of times one of the four models is chosen by an information criterion. For example, the first row and second column of Table 4 presents the percentage of times that model M₁ is selected based on AICc when D₁ is fitted. The last row of each table represents the total percentage of times that the true model is not picked by the corresponding criteria, which can be called an ‘Error’. In each table, the true model is marked by ‘*’ for convenience.

Table 5.

Percentage of correct decisions when data are generated from M₁

Model Fit	DGP: D1
	n = 50			n = 100			n = 200
	AIC	BIC	AICc	AIC	BIC	AICc	AIC	BIC	AICc
$M_1^{\ast }$	90	99	92	91	100	92	96	100	95
M2	6	0	5	5	0	4	3	0	4
M3	4	1	3	4	0	4	1	0	1
M4	0	0	0	0	0	0	0	0	0
Error	10	1	8	9	0	8	4	0	5

Table 8.

Percentage of correct decisions when data are generated from M₄

Model Fit	DGP: D41
	n = 50			n = 100			n = 200
	AIC	BIC	AICc	AIC	BIC	AICc	AIC	BIC	AICc
M1	44	76	50	11	47	12	15	18	12
M2	6	9	4	24	4	22	5	4	6
M3	46	24	43	57	49	60	65	58	71
$M_4^{\ast }$	4	0	3	8	0	6	15	20	11
Error	96	100	97	92	100	94	85	80	89
Model Fit	DGP: D42
	n = 50			n = 100			n = 200
	AIC	BIC	AICc	AIC	BIC	AICc	AIC	BIC	AICc
M1	41	72	49	9	45	9	19	31	17
M2	8	0	2	21	5	19	11	9	8
M3	52	28	49	56	50	60	47	42	52
$M_4^{\ast }$	2	0	0	14	0	12	23	18	23
Error	98	100	100	86	100	88	77	82	77
Model Fit	DGP: D43
	n = 50			n = 100			n = 200
	AIC	BIC	AICc	AIC	BIC	AICc	AIC	BIC	AICc
M1	34	71	41	14	46	16	19	36	16
M2	8	1	5	19	4	18	8	10	7
M3	58	28	54	57	50	59	49	39	51
$M_4^{\ast }$	0	0	0	10	0	7	24	15	26
Error	100	100	100	90	100	93	76	85	74
Model Fit	DGP: D44
	n = 50			n = 100			n = 200
	AIC	BIC	AICc	AIC	BIC	AICc	AIC	BIC	AICc
M1	42	75	50	11	49	13	19	34	17
M2	8	1	4	16	3	12	7	17	10
M3	49	24	46	64	48	67	53	36	51
$M_4^{\ast }$	1	0	0	9	0	8	21	13	22
Error	99	100	100	91	100	92	79	87	78

We expected the true model to be chosen most of the time, when the data are fitted to various models including the true model. Results from our simulation study indicated that AIC, AICc, and BIC performed well for some specific spatial models, however these criteria performed poorly as well for some other spatial models.

Table 5 presents results for D₁ generated from the stationary isotropic model, M₁. All criteria performed well for n = 50, 100 and n = 200. Especially BIC performed very well. BIC picked the true model 99% of the time for n = 50 and 100% for n = 100, 200. AIC and AICc chose the correct model 90% and 92% of the time for n = 50 and 96% and 95% of the time for n = 200. The performances of AIC and AICc were similar. Note that M₄ was never picked by all criteria. Overall BIC performed better than AIC and AICc in selecting the stationary isotropic model M₁.

Table 6 summarizes the results for D₂₁ and D₂₂. D₂₁ and D₂₂ were generated from the stationary anisotropic model, M₂, with different parameter values for λ = 5 and λ = 10, respectively, and with the same parameter value γ = π/4. Each criterion performed similarly in D₂₁ and D₂₂ except that AIC and AICc picked M₃ more often in D₂₂. All criteria did not perform well when the sample size was n = 50. Especially BIC performed poorly. BIC picked the true model, M₂, only 7% of the time in D₂₁ and 6% in D₂₂. AIC performed better than AICc for n = 50. This is counter to the idea that AICc is designed to perform well for small sample sizes. When n = 50, all criteria more often selected M₁ instead of the true model, M₂. All criteria tended to pick the parsimonious model, that is, the simpler model even though the true model is more complex. As sample size increased to 100 or 200, the performance of all the criteria improved. All criteria selected the correct model M₂ most often except BIC for D₂₁. AIC and AICc were successful in choosing the correct model and the performances of them were similar. For AIC, the success rate of picking the true model increased from 32% to 81% under D₂₁ and from 27% to 85% for D₂. Also, the performances of AICc increased as the errors dropped by 57 (=77-20)% for D₂₁ and 65 (=81-16)% for D₂₂. While AIC and AICc performed well with the sample size n = 200, BIC still tended to pick parsimonious model, M₁. BIC correctly picked the true model 68% for D₂₁ and 72% for D₂₂.

Table 6.

Percentage of correct decisions when data are generated from M₂

Model Fit	DGP: D21
	n = 50			n = 100			n = 200
	AIC	BIC	AICc	AIC	BIC	AICc	AIC	BIC	AICc
M1	50	84	61	20	50	23	12	26	15
$M_2^{\ast }$	32	7	23	71	45	70	81	68	80
M3	10	8	9	4	5	4	4	5	4
M4	8	1	7	5	0	3	3	1	1
Error	68	93	77	29	55	30	19	32	20
Model Fit	DGP: D22
	n = 50			n = 100			n = 200
	AIC	BIC	AICc	AIC	BIC	AICc	AIC	BIC	AICc
M1	49	83	56	14	41	15	10	22	11
$M_2^{\ast }$	27	6	19	69	50	69	85	72	84
M3	21	10	23	8	7	8	4	4	4
M4	3	1	2	9	2	8	1	2	1
Error	73	94	81	31	50	31	15	28	16

We found that the results from D₂₁ and those from D₂₂ were similar but slightly different. When n = 50, all criteria selected M₃ more often in D₂₂ than in D₂₁, and chose M₂ less often in D₂₂ than in D₂₁ except BIC for n = 100. This occurrence makes sense because the covariance function for D₂₂ was closer than the covariance function for D₂₁ to the covariance functions of M₃. However, based on the Frobenius distance given in Table 3, it still does not make sense to choose M₁ more often than other models. When n = 100, all criteria also picked the correct model M₂ less often and picked M₃ and M₄ more often in D₂₂ than in D₂₁. We think it is because that the covariance function of D₂₂ was closer than that of D₂₁ to the covariance functions of M₃ and M₄. BIC picked M₂ more often than M₁ in D₂₂. The reverse occurred in D₂₁. Overall, AIC performed better than AICc and BIC in choosing the stationary anisotropic model M₂ and accuracy improved with sample size.

The results for D₃₁ and D₃₂ are given in Table 7. D₃₁ and D₃₂ were generated from the nonstationary point source isotropic model, M₃, with different parameter values for θ = 5 and θ = 10, respectively, and with the same parameter value λ = 1. Overall, all criteria performed well with the exception of BIC when the sample size was n = 50. For n = 50, BIC picked both the wrong model, M₁, and the correct model, M₃, with almost the same percentages (49% and 50%, respectively) for D₃₁, while the wrong model, M₁, was picked with a higher percentage (61%) for D₃₂. This indicated that BIC tended to select a simpler model, M₁, than the true model, M₃, when n = 50. In contrast, AIC and AICc identified the true model relatively well for n = 50. AIC selected the correct model 73% of the time for both D₃₁ and D₃₂, and AICc chose the true model 70% and 63% of the time for D₃₁ and D₃₂, respectively. AIC performed better than AICc for the small sample size. For n = 100 and 200, the performance of all criteria improved. The performances of AIC and AICc were same, and these criteria performed better than BIC. Overall, the error rates decreased with increasing sample size for both D₃₁ and D₃₂. All criteria performed better in D₃₁ than D₃₂. The performance of each criterion appeared to have a similar pattern under D₃₁ and D₃₂ except that BIC and AICc picked M₁ more often in D₃₁ than in D₃₂ for n = 50. Note that M₄ was never picked by all criteria given all the values of θ and n considered, even though the covariance function of M₄ was closer than that of M₁ and M₂ to the covariance function of M₃ in terms of the Frobenius distance in Table 3.

Table 7.

Percentage of correct decisions when data are generated from M₃

Model Fit	DGP: D31
	n = 50			n = 100			n = 200
	AIC	BIC	AICc	AIC	BIC	AICc	AIC	BIC	AICc
M1	24	49	28	6	17	6	3	9	3
M2	3	1	2	3	1	3	1	1	2
$M_3^{\ast }$	73	50	70	91	82	91	96	90	95
M4	0	0	0	0	0	0	0	0	0
Error	27	50	30	9	18	9	4	10	5
Model Fit	DGP: D32
	n = 50			n = 100			n = 200
	AIC	BIC	AICc	AIC	BIC	AICc	AIC	BIC	AICc
M1	23	61	35	9	18	9	4	9	5
M2	4	0	2	4	0	4	2	2	2
$M_3^{\ast }$	73	39	63	87	82	87	94	89	93
M4	0	0	0	0	0	0	0	0	0
Error	27	61	37	13	18	13	6	11	7

Table 8 illustrates the results for D₄₁, D₄₂, D₄₃, and D₄₄ which were generated from the nonstationary point source anisotropic model, M₄, with different parameter values of θ and λ as shown in Table 2. As given in Table 8, the performances of the criteria did not vary much across four data sets. All criteria performed very poorly in selecting the true model, M₄ (even when n = 200), and tended to choose a simpler model than the real model. In particular, BIC did not pick the true model even one time out of 100 replications when n = 50, 100. Even though AIC performed better than AICc and BIC, AIC did not perform well. AIC only picked the true model less than 5% of the time when n = 50 and less than 15% when n = 100 for all of four data sets. For n = 50, all criteria selected M₁ and M₃ most of the time. BIC picked M₁ more often (more than 70% of the time) than AIC and AICc. AICc picked M₁ more often and selected M₃ less often than AIC. For n = 100, AIC and AICc picked M₃ more often, and BIC picked M₁ and M₃ with similar percentage. The performances of AIC and AICc were similar and improved with increasing sample size. In this case, it seemed that AIC and AICc made more sense than BIC in model selection for M₄. Selecting M₁ more often than other models seems unreasonable based on the Frobenius distance, because the covariance function of M₁ was much different from that of M₄. The covariance functions of M₂ and M₃ were much closer than that of M₁ to the covariance function of M₄ as shown in Table 3. Overall, AIC performed better than AICc and BIC in selecting M4.

5 Discussions and Future Research

We investigated how information criteria such as AIC, AICc, and BIC perform in the spatial model selection problems via simulations. The results are summarized as follows:

• BIC was superior to AIC and AICc when the true model was the stationary isotropic model. When the sample size was large (e.g., n = 200), BIC perfectly picked the true model. BIC also performed very well even though the sample size was small (e.g., n = 50). AIC and AICc also performed well.

• When the true model was the stationary anisotropic model, all criteria did not perform well for n = 50. Especially BIC performed poorly. As n increased to 200, the performance of all criteria improved. AIC and AICc performed well for n = 200, however BIC did not perform well even for the large sample size.

• AIC performed better than AICc and BIC for n = 50, and both AIC and AICc outperformed BIC for n = 100, 200, when the true model was the nonstationary point source isotropic model. BIC picked the stationary isotropic model most often when n = 50, however it picked the correct model most of the time when n = 100, 200. AIC and AICc performed well for all sample sizes. The error rates for all criteria decreased as sample size increased.

• All criteria performed poorly when the true model was the nonstationary point source anisotropic model even with moderately larger sample sizes. AIC performed better than AICc and BIC. BIC never picked the true model even when the sample size was n = 50, 100. In contrast, AIC and AICc picked the true model more often when n = 200.

Our results indicate that the performance of the criteria to select the true model generally improved with increase of sample size, despite differences in performance among the criteria. From the results obtained from simulations, we found that the performance of the criteria depends on sample size and model complexity, but not parameter values. Hence, it would be worthwhile to investigate further simulation studies other stationary and nonstationary models and if possible with much larger sample sizes. It will also be of interest to establish theoretical results on the validity of the use of information criteria for spatial models.