A Third Moment Adjusted Test Statistic for Small Sample Factor Analysis

Abstract

Goodness of fit testing in factor analysis is based on the assumption that the test statistic is asymptotically chi-square; but this property may not hold in small samples even when the factors and errors are normally distributed in the population. Robust methods such as Browne’s asymptotically distribution-free method and Satorra Bentler’s mean scaling statistic were developed under the presumption of non-normality in the factors and errors. This paper finds new application to the case where factors and errors are normally distributed in the population but the skewness of the obtained test statistic is still high due to sampling error in the observed indicators. An extension of Satorra Bentler’s statistic is proposed that not only scales the mean but also adjusts the degrees of freedom based on the skewness of the obtained test statistic in order to improve its robustness under small samples. A simple simulation study shows that this third moment adjusted statistic asymptotically performs on par with previously proposed methods, and at a very small sample size offers superior Type I error rates under a properly specified model. Data from Mardia, Kent and Bibby’s study of students tested for their ability in five content areas that were either open or closed book were used to illustrate the real-world performance of this statistic.

Introduction

Maximum likelihood techniques used in confirmatory factor analysis rely on the assumption of multivariate normality and asymptotic theory. The estimation of parameters involves minimizing the difference between the sample covariance matrix S and the implied covariance matrix Σ(θ) using the fit function (Jöreskog, 1967, 1970)

$$F_{ML}=log\mid \mathrm{\sum }(\theta )\mid +\text{trace}(S{\mathrm{\sum }}^{-1}(\theta ))-log\mid S\mid -p$$ (1)

where p is the number of observed variables. The derivation of F_ML assumes that the vector of observed indicators y is independently sampled from a multivariate normal distribution (Bollen, 1989). Since the observed indicators are a function of parameters in the factor analytic model, non-normality in observed indicators is an implied consequence of non-normality in the distribution of factors and errors. Violations of normality due to excess skewness and kurtosis can still lead to consistent parameter estimates, but the standard errors and goodness of fit chi-square statistics are compromised (Browne, 1984; Bollen, 1989).

The confirmatory factor model begins with y, a vector of observed indicators that depends on Λ, a common factor loading matrix, η, a vector of latent factor scores, and ε, a vector of unique errors. Their linear relationship is defined as

$$y=\mathrm{\Lambda }\eta +\epsilon .$$ (2)

Assuming η is normally distributed and uncorrelated with ε the restricted model covariance of y is

$$\mathrm{\sum }(\theta )=\mathrm{\Lambda }\mathrm{\Phi }{\mathrm{\Lambda }}^{\prime }+\mathrm{\Psi }$$ (3)

where Ψ is a diagonal matrix of unique variances and Φ is a matrix of covariances among latent factor scores.

When the maximum likelihood fit function (1) is used to estimate the vector θ of parameters of the set {Λ, Φ, Ψ}, the goodness of fit chi-square estimator is given as

$$T\equiv n{\widehat{F}}_{ML}$$ (4)

where n is the sample size and F̂_ML is the minimum fit function (1) evaluated at the maximum likelihood estimate θ̂. This is −2 times the logarithm of the Wishart likelihood ratio criterion for testing the null hypothesis of (3) against the alternative hypothesis of unrestricted Σ (Amemiya & Anderson, 1990). If the latent factor scores η and unique errors ε are normally distributed, then T converges in law to a χ²(d) distribution with degrees of freedom

$$d=\frac{1}{2}(p)(p+1)-t$$ (5)

where p is the number of observed variables and t is the number of free parameters in θ. Conceptually, (5) represents the number of non-redundant elements in the covariance matrix S minus the number of free parameters. This χ² approximation is based on the assumption that a) y has no kurtosis, b) the covariance rather than the correlation matrix is analyzed, c) the sample is sufficiently large, and d) H₀ : Σ = Σ (θ) holds exactly (Bollen, 1989). Under violation of the first condition, there is an alternative procedure known as the asymptotically distribution-free (ADF) method (Browne, 1984). Empirically, this method has been shown to require large sample sizes to yield an accurate assessment of model adequacy since it requires the inversion of a sample fourth order moment based matrix that converges slowly to its population counterpart (Hu, Bentler, & Kano, 1992; Curran, West, & Finch, 1996). The current study utilized and extends a class of chi-square correction procedures known as mean scaling and degrees of freedom adjustment methods (Satorra & Bentler, 1988, 1994) that have been shown to perform better than the ADF method under small sample sizes.

Define s = (s₁₁, s₂₁, ···, s_kk)′ as the vector of elements in the lower triangle of the sample covariance matrix S, and σ(θ) = (σ₁₁, σ₂₁, ···, σ_kk)′ the elements of the model restricted covariance matrix. Browne (1974, 1984) showed that minimizing the maximum likelihood fit function described in (1) is asymptotically equivalent to minimizing the quadratic form

$$F(\theta ,V)={[s-\sigma (\theta )]}^{\prime }V[s-\sigma (\theta )].$$ (6)

Define V̂ to be the consistent estimate of V as

$$\widehat{V}=2^{-1}{D}^{\prime }(A_n^{-1}\otimes A_n^{-1})D$$ (7)

where A_n is a matrix that converges to Σ with probability one (e.g., $\widehat{\mathrm{\sum }(\theta )}$), and D is the 0–1 duplication matrix that eliminates the supra-diagonal elements of A_n (Magnus & Neudecker, 1986). Assume that $\sqrt{n}(s-\sigma )$ converges in distribution to N(0, Γ) as n → ∞, where Γ is the asymptotic covariance matrix of s. Typically, Γ is estimated via the covariance matrix (Browne, 1984; Satorra & Bentler, 1994):

$$\widehat{\mathrm{\Gamma }}=\frac{1}{n-1}\sum _{i=1}^n(b_i-\overline{b}){(b_i-\overline{b})}^{\prime }$$ (8)

where b_i = D⁺vec(y_i − ȳ)(y_i − ȳ)′, D⁺ is the Moore-Penrose inverse of the duplication matrix of D described in (7), b̄ and ȳ are the means of b and y, and vec is the operator that vectorizes a matrix column-wise. Let the Jacobian matrix be $\mathrm{\Delta }\equiv \frac{\partial \sigma }{\partial {\theta }^{\prime }}$, which is estimated in the sample by Δ̂, and define

$$\widehat{U}=\widehat{V}-\widehat{V}\widehat{\mathrm{\Delta }}{({\widehat{\mathrm{\Delta }}}^{\prime }\widehat{V}\widehat{\mathrm{\Delta }})}^{-1}{\widehat{\mathrm{\Delta }}}^{\prime }\widehat{V}.$$ (9)

Asymptotically,

$$T\to n{(s-\sigma (\theta ))}^{\prime }U(s-\sigma (\theta ))$$ (10)

where the matrix U is the asymptotic true form of Û. Using Theorem 2.1 of Box (1954)

$$T\stackrel{L}{\to }\sum _{i=1}^d{\lambda }_i{\chi }_i^2(1)$$ (11)

where λ_i’s are the positive eigenvalues of the matrix UΓ. Typically population values are not accessible, and we estimate T using T̂.

Mean Scaling and Moment Adjustment Methods

The Satorra Bentler (1994) scaled chi-square statistic, here called the Mean Scaled statistic, is

$$T_M\equiv \frac{T}{m}$$ (12)

where m ≡ trace(UΓ)/d. Note that for brevity, the robust statistics are described in its population form, but can be implemented by replacing ÛΓ̂ for UΓ.

Satorra and Bentler (1994) also proposed an extension of T_M which involves both scaling the mean and a Saitterwaithe second moment adjustment of the degrees of freedom (Satterthwaite, 1941). This statistic is called the Mean Scaled and Variance Adjusted statistic, and is defined as

$$T_{MV}\equiv \frac{v}{\text{trace}(U\mathrm{\Gamma })}T$$ (13)

where v ≡ [trace(UΓ)]²/trace[(UΓ)²].

Recall in Equation 11 that T asymptotically approaches a quadratic form which is a weighted sum of independent chi-square variates. Based on the cumulant generating function of a central chi-square quadratic form

$$c_k=\sum _{i=1}^d{\lambda }_i^k$$ (14)

where k is an index of its moment; the mean, standard deviation and skewness of T are μ_T = c₁, σ_T = 2c₂, ${\beta }_T=\sqrt{8}(c_3/c_2^{3/2})$ respectively (Liu, Tang, & Zhang, 2009). In practice, sample estimates μ_T̂, σ_T̂, and β_T̂ are used.

The proposed statistic, to be called the Mean Scaled and Skewness Adjusted statistic, is defined as

$$T_{MS}\equiv \frac{v^{\ast }}{\text{trace}(U\mathrm{\Gamma })}T$$ (15)

with estimated degrees of freedom

$$v^{\ast }\equiv \frac{c_2^3}{c_3^2}=\frac{{({\sum }_{i=1}^r{\lambda }_i^2)}^3}{{({\sum }_{i=1}^r{\lambda }_i^3)}^2}=\frac{\text{trace}{[{(U\mathrm{\Gamma })}^2]}^3}{\text{trace}{[{(U\mathrm{\Gamma })}^3]}^2}$$ (16)

where v* is a function of the skewness of T. Motivated by the Pearson-Imhof three-moment central χ² approximation method (Pearson, 1959; Imhof, 1961), T_MS scales the mean as does T_M and T_MV but also adjusts the degrees of freedom such that the asymptotic quadratic form of T has the same skewness as a candidate χ²(v*), the new reference distribution. The goal of modifying the degrees of freedom in addition to scaling the mean is to downwardly adjust the obtained statistic such that its distribution is as close to a central chi-square as possible, despite a potentially large disparity in the population eigenvalues of the matrix UΓ.

Theoretical Conditions for Approaching χ²

Consider the case in the population where the eigenvalues of UΓ are constant, notably if λ_i = λ for i = 1, ···, d. Then Equation 16 simplifies to

$$v^{\ast }=\frac{{({\sum }_{i=1}^d{\lambda }_i^2)}^3}{{({\sum }_{i=1}^d{\lambda }_i^3)}^2}=\frac{{({\sum }_{i=1}^d{\lambda }^2)}^3}{{({\sum }_{i=1}^d{\lambda }^3)}^2}=\frac{{(d{\lambda }^2)}^3}{{(d{\lambda }^3)}^2}=\frac{d^3{\lambda }^6}{d^2{\lambda }^6}=\frac{d^3}{d^2}=d.$$ (17)

Substituting d into Equation 15,

$$T_{MS}=\frac{v^{\ast }}{\text{trace}(U\mathrm{\Gamma })}T=\frac{d}{\text{trace}(U\mathrm{\Gamma })}T=T_M$$ (18)

which makes T_MS equivalent to T_M. In the special case where λ_i = 1 for i = 1, ···, d, the asymptotic distribution of T would approach χ² since ${\sum }_{i=1}^d\lambda {\chi }_i^2(1)={\chi }^2(d)$. Then by modification of Equation 18, T_MS would be equivalent to T. When λ_i is not constant, the distribution of T cannot be determined, and the assumption is that scaling the mean and adjusting the degrees of freedom of T will result in a better approximation of a chi-square variate.

The Effect of Sample Size on the Obtained Test Statistic

The previous sections introduced well-known robust methods and the proposed third moment adjusted statistic based on the assumption of asymptotic theory with given population parameters. In practice however, large sample sizes and population parameters may not be available to the researcher, which means that the properties of these robust methods may not hold. As stated in the introduction, these methods were developed to address the issue of population non-normality in factors and errors and do not directly address the issue of non-normality due to sampling error. Consider the situation in which the population factors and errors are normal but because of small sample size, the eigenvalues of the matrix ÛΓ̂ are not all one and hence the obtained test statistic is not chi-square distributed. The latter half of the paper marks a diversion away from asymptotic theory to show the empirical relation between sample size and the test statistics at hand.

Figure 1 illustrates the effect of small, medium, and large sample size on the distribution of the eigenvalues of ÛΓ̂ from a simulated run of a case where factors and errors were normally distributed. Despite the theoretical equality of eigenvalues in the population, the figure shows that the estimated eigenvalues under n = 25 are markedly more dispersed compared to other sample size conditions, with eigenvalues ranging from 0 to 12 and positive values dropping off at i = 25. This increased dispersion of sample eigenvalues compared to population counterparts has been previously noted for cases when the ratio of estimated parameters to sample size is large (Ledoit & Wolf, 2004). As shown in the figure, the dispersion decreases with moderate sample size (n = 100), with eigenvalues now ranging from 0 to about 3.5. Asymptotically (n = 5000), the positive eigenvalues stabilize at a constant point near one, and drop off to zero at i = 54, the expected degrees of freedom. The increasing dispersion of eigenvalues with decreasing sample size highlights the rationale of using mean scaling and degrees of freedom adjustments for small sample factor analysis.

Figure 1.

Distribution of the ranked eigenvalues of ÛΓ̂ for one sample replicate at n = 25, n = 100, and n = 5000

To demonstrate the relation of sample size to the skewness of the obtained statistic, calculate the skewness using the formula

$${\beta }_{\widehat{T}}=\sqrt{8({\widehat{c_3}}^2/{\widehat{c_2}}^3)}=\sqrt{8/\widehat{v^{\ast }}}.$$ (19)

For a sample replicate of n = 25, the skewness of T̂ was calculated to be 1.01. Given n = 100, the skewness decreased to 0.70, and for n = 5000 it was 0.39, approaching the optimal skewness of 0.38 given that the degrees of freedom of the reference chi-square is 54. Since both the sample size and the estimated degrees of freedom are inversely proportional the skewness of the obtained test statistic T̂, it makes intuitive sense to adjust the estimated degrees of freedom downward when the skewness of the obtained statistic may be higher due to smaller sample size. In the empirical example to follow (Table 2), notice that the degrees of freedom for T_MS are consistently lower than that of other statistics.

Table 2.

Estimates of Test Statistics for the Open-Book Closed-Book Dataset

	Correlated Factor Model		Uncorrelated Factor Model
Test Statistic	Estimate (df)	Probability	Estimate (df)	Probability
Original Sample n = 88
T	2.07 (4)	0.72	50.53 (5)	< 0.01
TM	2.21 (4)	0.70	41.03 (5)	< 0.01
TMV	1.89 (3.42)	0.67	39.33 (4.79)	< 0.01
TMS	1.55 (2.82)	0.64	28.86 (3.52)	< 0.01
Sub-sample n = 15
T	8.73 (4)	0.07	20.77 (5)	< 0.01
TM	10.16 (4)	0.04	23.24 (5)	< 0.01
TMV	7.24 (2.85)	0.06	17.84 (3.84)	< 0.01
TMS	5.50 (2.17)	0.07	11.64 (2.50)	< 0.01

Monte Carlo Simulation

The current study evaluates the performance of two existing robust statistics against the proposed Mean Scaled and Skewness Adjusted statistic, T_MS. Yuan and Bentler (2010) evaluated the Type I and mean-square error of T_M and T_MV under different coefficients of variation in the eigenvalues of UΓ, and found that T_MV performs better than T_M when the disparity of eigenvalues is large. Based on this finding, the current hypothesis proposes that asymptotically, the performance of T_MS would not differ from T_M and T_MV given that the factors and errors are normally distributed in the population. Due to the inverse relationship between sample size and skewness of the obtained test statistic, we hypothesize that a third moment based adjustment of the obtained statistic would be better than a second moment adjustment or mean scaling alone and that the performance of T_MS would improve as sample size decreases. As such, the test statistics T, T_M, T_MV and T_MS were evaluated for their Type I error and empirical power under asymptotic sample size (n = 5000), moderate sample size (n = 100), and very small sample size (n = 25).

Confirmatory Factor Model

A confirmatory factor model with 12 observed variables and three orthogonal factors was used to generate a model based simulation study in R 2.13.0 for Windows (R Development Core Team, 2011). Factors and errors were generated independently and distributed N(0, 1). A simple structure of Λ was used where each set of four observed variables loaded onto a single factor with loadings of 0.80, as shown in (20). Random variates for factor and errors were generated and entered into Equation 3, with Φ = I. Model 1 is defined as the data generation model with λ₅₁ = 0, shown as the loading to the left of the slash in (20). Model 2 is defined as the data generation model in which λ₅₁ = 0.8, shown to the right of the slash. Each analysis was replicated 100 times, since at n = 25, convergence issues made it difficult to perform more replications. The simulated datasets were analyzed using the sem package in R (Fox, 2006) and the mean and scaling statistics obtained by modifying source code from sem.additions (Byrnes, 2010) and CompQuadForm (Duchesne & Lafaye De Micheaux, 2010). The parametrization of sem.additions involved specifying useFit=T so that $A_n=\widehat{\mathrm{\sum }(\theta )}$ in Equation 7, and useGinv=T so that a generalized inverse is used in place of the inverse of Δ̂′V̂Δ̂ (see Equation 9). These parametrizations were chosen so that the computed values of T, T_M and T_MV match those of EQS 6.2 (Bentler, 2006).

$${\mathrm{\Lambda }}^{\prime }=\left[\begin{array}{cccccccccccc}0.8& 0.8& 0.8& 0.8& 0/0.8& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0.8& 0.8& 0.8& 0.8& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0.8& 0.8& 0.8& 0.8\end{array}\right]$$ (20)

Type I Error and Empirical Power

Type I error performance was measured by dividing the observed rejection rate of the null hypothesis by the total number of trials (N = 100) under the correctly specified model (Model 1 generated, Model 1 analyzed). Setting the probability of rejection criterion at α = 0.05,

$$1-P(T\le {\chi }^2(d)\mid \mathrm{\sum }(\theta )={\mathrm{\sum }}_0)=\alpha$$ (21)

where an ideal Type I error rate should approach 5% rejection of the null hypothesis. Under an incorrectly specified model Σ(θ) ≠ Σ₀ (Model 2 generated, Model 1 analyzed), the empirical power is defined as the proportion of rejection of the null hypothesis for N = 100 simulated trials.

Simulation Results

Table 1 shows the results of N = 100 replications of normal factors and errors under asymptotic sample size (n = 5000), moderate sample size (n = 100), and a very small sample size (n = 25) across α = {0.01, 0.05, 0.1}. As expected, at n = 5000 and under correct model specification (Model 1 generated, Model 1 analyzed), the Type I error performance (left columns) was nearly identical across test statistics, with rates ranging from 0.02 to 0.04 at α = 0.05. Under incorrect model specification (Model 1 generated and Model 2 analyzed), empirical power (right column) was excellent, as every statistic rejected the model 100% of the time across alpha levels. The results give evidence that asymptotically, T_MS performs on par with the other statistics in its Type I error and empirical power.

Table 1.

Performance of Test Statistics under N=100 Replications

	Type I Error			Empirical Power
Test Statistic	0.01	0.05	0.10	0.01	0.05	0.10
Asymptotic Performance n = 5000
T	0.00	0.03	0.1	1.00	1.00	1.00
TM	0.00	0.04	0.1	1.00	1.00	1.00
TMV	0.00	0.03	0.1	1.00	1.00	1.00
TMS	0.00	0.02	0.1	1.00	1.00	1.00
Medium Sample Performance n = 100
T	0.03	0.10	0.18	1.00	1.00	1.00
TM	0.05	0.12	0.23	1.00	1.00	1.00
TMV	0.01	0.05	0.10	0.98	1.00	1.00
TMS	0.00	0.03	0.05	0.90	0.99	1.00
Very Small Sample Performance n= 25
T	0.20	0.45	0.58	0.67	0.79	0.88
TM	0.45	0.74	0.82	0.81	0.90	0.95
TMV	0.01	0.19	0.41	0.14	0.56	0.79
TMS	0.00	0.04	0.25	0.00	0.27	0.58

The advantage of T_MS is more evident at a very small sample size (n = 25). Under correct model specification, T_M had the highest Type I error rates at α = 0.05, rejecting the null hypothesis 74% of the time, followed by T at 45%, T_MV at 19%, and finally T_MS, which offered the most optimal rejection rate at 4%. A similar pattern emerged for α = 0.01, except that T_MV performed on par with T_MS. At α = 0.1, all statistics had higher than expected rejection rates, but T_MS still offered the most optimal rate at 25% rejection compared to 82% for T_M. Figure 2 shows the density of p-values for the test statistics considered and a simulated chi-square variate with 54 degrees of freedom under correct model specification at the region of p < 0.05. Compared to the reference chi-square probability density, T_M was overly peaked (rejecting the correctly specified model too frequently at n = 25), whereas T_MS was closest to the reference chi-square probability density. Under incorrect model specification, the lower right part of Table 1 shows that T_M had the most correct rejections of the null hypothesis at 90% and T_MS the least at 27% for α = 0.05. Similar patterns emerged for α = 0.01 and α = 0.1. The numbers suggest that the empirical power of T, T_M and T_MV under a very small sample exceeded that of T_MS when the model is misspecified.

Figure 2.

Density distribution of probabilities for a χ²(54) variate and test statistics at n = 25, N = 100 replications, p ≤ 0.05 under Model 1.

To test the threshold at which the Type I error rate and power of T_MS converges with the other statistics, a third sample size condition at n = 100 was added. The results show that at this moderate sample size, the Type I performance of T_MV begins to approach T_MS (and in fact may have more optimal performance at α = 0.10), while T and T_M continue to have inflated Type I rates. The underpowered phenomenon of T_MS from the much smaller sample size condition (n = 25) disappears as an empirical power of 0.99 is achieved with α = 0.05. The results suggest that across a range of sample sizes T_MS may be a good choice for achieving optimal Type I error rates, although T_MV may be better for moderate sample sizes.

Illustrative Application of T_MS

Analysis of the Open-book Closed-book Dataset

To illustrate the real-world performance of these statistics, Mardia, Kent, and Bibby’s (1980) Open-book Closed-book dataset was chosen. The dataset can be freely downloaded from the Comprehensive R Archive Network (CRAN) as the scor component in the bootstrap R package (Tibshirani & Leisch, 2009; Efron & Tibshirani, 1993). A total of n = 88 students were tested for their ability in five content areas: mechanics (closed book), vectors (closed book), algebra (open book), analysis (open book), and statistics (open book). Following Cai and Lee (2009), the first model was analyzed using a two correlated factor structure, in which the two closed book skills grouped together, and the four remaining open book skills grouped together to create a model with 11 free parameters and four degrees of freedom. As with the simulation study, the data were analyzed using a source code modification of the sem R package and sem.additions extension, and the results of T, T_M, T_MV were verified with EQS 6.2 software using the METHOD = ML, ROBUST specification.

The first model fit the data very well with an estimated RMSEA < 0.01, CFI = 1.00, and χ²(4) = 2.07, p = 0.72. All test statistics showed that the two correlated factor model was retained at p > 0.5. The difference between the degrees of freedom adjusted statistics and the others is evident in the lower estimated degrees of freedom (see Table 2 top left column). The second model fit the same factor loading pattern as the first model except that the factors were uncorrelated. This model did not fit the data well with an estimated RMSEA = 0.324, CFI = 0.76, and χ²(5) = 50.53, p < 0.01. Similar patterns of model rejection were seen for the uncorrelated factor model across all statistics, except that T_MS again resulted in the lowest mean scaling and degrees of freedom after adjustment (see Table 2, top right column). The results show that under a typical data analytic setting, all test statistics perform comparably well.

Analysis of a Sub-sample of the Open-book Closed-book Dataset

The second analysis demonstrates the test statistics at a much smaller sample size. A total of 15 students were randomly selected from the original Open-Book Closed-Book dataset, with the particular order of student cases chosen as: 2, 3, 5, 15, 21, 26, 33, 57, 58, 68, 74, 75, 76, 84, and 86. Table 2 (lower left column) shows that at α = 0.05, all statistics except for T_M retained the correlated factor model, whereas all statistics rejected the uncorrelated factor model at p < 0.01 (Table 2 lower right column). The results are consistent with those from the simulation study in that T_MS was shown to be relatively more conservative in retaining the specified model than T_M at a very small sample size, although the differences between them were not as definitive as in the simulation study. The authors suspect that the efficacy of T_MS may depend on the magnitude of the ratio $\sqrt{d/\widehat{v^{\ast }}}$, which is $\sqrt{4/2.17}=1.35$ in this empirical example, but was greater at $\sqrt{54/7.84}=2.62$ in the simulation study.

Conclusion and Discussion

The current paper proposed a new statistic that scales the mean and adjusts the degrees of freedom of the obtained test statistic to match the skewness of the reference chi-square variate. The rationale is that this adjustment will allow the obtained statistic to approach a central chi-square variate, even under small sample sizes, so that it is possible to obtain p-values by looking up a standard chi-square table. A Monte Carlo simulation study assessed the performance of the proposed statistic compared with existing methods across a range of sample sizes, and an example using the Open-book Closed-book dataset was provided.

The simulation study gives evidence that T_MS performs on par asymptotically with existing methods, and under a very small sample size (n = 25) can offer superior Type I error rates by as much as 1360% when compared to T_M at α = 0.05, and by 260% compared with T_MV. However, the optimum Type I error rate of T_MS under a correctly specified model comes with the cost of being underpowered for a misspecified model at a small sample size. The T_M procedure that currently exists in structural equation modeling programs such as EQS (Bentler, 2006), Mplus (Muthen & Muthen, 2006) and LISREL (Jöreskog, Sorbom, du Toit, & du Toit, 2000) may be a better choice, although the current study suggests that the power of T_MS approaches T_M when sample size reaches n = 100. In sum, it may be beneficial to use the more conservative T_MS when confirming the fit of the model under a very small sample size, but to use the more powerful T_M as a criterion for rejecting an incorrect model. If the researcher finds that the model is rejected by other robust methods but retained by T_MS, it may be a sign that the model is correctly specified, and it may be worth the extra effort to continue recruiting subjects.

The proposed statistic and the methodological design of the paper is not without limitations. First, since the population factors and errors were normally distributed, the theoretical distribution of the test statistic should be exactly chi-square. This is not the situation for which robust statistics were developed, as non-normality of factors and errors is typically assumed. Given the focus of the paper, the case where population eigenvalues are not equal to one was not empirically assessed, although it is reasonable to assume that the current theory would still hold. Second, rather than perform mean scaling and degrees of freedom adjustments to allow the obtained statistic to approach a central chi-square variate, it may possible to test the probability of a mixture chi-square directly (Bentler, 1994; Lo, Mendell, & Rubin, 2001). The drawback of the direct approach is that under small samples, the obtained test statistic may not actually be distributed mixture chi-square, which would lead to incorrect testing conclusions. Third, the current study considered the robustness of T_MS only for continuous data. Lei (2009) showed that for ordinal variables, T_M may reject too frequently at large sample sizes compared to the mean scaled and variance adjusted procedure, and that contrary to our findings for moderately large samples T_M was less powerful. However, the conclusions do not map one-to-one to our current paper, since the variance adjusted procedure was estimated using weighted least squares whereas T_M was estimated using maximum likelihood. In addition, the smallest sample size considered in that paper was n = 100, which is larger than the n = 25 considered in this paper. Finally, the current paper developed its new test statistic under the assumption of complete data with no mean structure. It is possible to extend the current statistic to the case of incomplete data with mean structure using the work of Yuan and Bentler (2000) by adapting their analogous matrices of UΓ to our methodology.