An Entropy-Based Tool to Help the Interpretation of Common-Factor Spaces in Factor Analysis

Abstract

This paper proposes a method for deriving interpretable common factors based on canonical correlation analysis applied to the vectors of common factors and manifest variables in the factor analysis model. First, an entropy-based method for measuring factor contributions is reviewed. Second, the entropy-based contribution measure of the common-factor vector is decomposed into those of canonical common factors, and it is also shown that the importance order of factors is that of their canonical correlation coefficients. Third, the method is applied to derive interpretable common factors. Numerical examples are provided to demonstrate the usefulness of the present approach.

1. Introduction

In factor analysis, extracting interpretable factors is important for practical data analysis. In order to carry it out, methods for factor rotation have been studied, e.g., varimax [1] and orthomax [2] for orthogonal rotations and oblimin [3] and orthoblique [4] for oblique rotations. The basic idea for factor rotation in factor analysis is owed to the criteria of simple structures of factor analysis models by Thurstone [5], and the methods of factor rotation are constructed with respect to maximizations of variations of the squared factor loadings in order to derive simple structures of factor analysis models. Let $X_i$ be manifest variables, let ${\xi }_j$ be latent variables (common factors), let ${\epsilon }_i$ be unique factors related to $X_i$, and finally, let ${\lambda }_{ij}$ be factor loadings that are weights of common factors ${\xi }_j$ to explain $X_i$. Then, the factor analysis model is given as follows:

$$X_i={{\displaystyle \sum }}_{j=1}^m{\lambda }_{ij}{\xi }_j+{\mathsf{\epsilon }}_i, i=1,2,\dots ,p,$$ (1)

where

$$\{\begin{array}{c}\mathrm{E}\left(X_i\right)=\mathrm{E}\left({\mathsf{\epsilon }}_i\right)=0, i=1,2,\dots ,p;\\ \mathrm{E}\left({\xi }_j\right)=0, j=1,2,\dots ,m;\\ \mathrm{Var}\left({\xi }_j\right)=1, j=1,2,\dots ,m; \mathrm{Cov}\left({\xi }_k,{\xi }_l\right)={\varphi }_{kl};\\ \mathrm{Var}\left({\mathsf{\epsilon }}_i\right)={\omega }_i^2>0, i=1,2,\dots ,p;\\ \mathrm{Cov}\left({\mathsf{\epsilon }}_k,{\mathsf{\epsilon }}_l\right)=0, k\ne l.\end{array}$$

To derive simple structures of factor analysis models, for example, in the varimax method, the following variation function of squared factor loadings is maximized with respect to factor loadings:

$$V={\displaystyle \sum }_{i=1}^p{\displaystyle \sum }_{j=1}^m{\left({\lambda }_{ij}^2-\overline{{\lambda }^2}\right)}^2,$$ (2)

where $\overline{{\lambda }^2}=\frac{1}{pm}{\sum }_{i=1}^p{\sum }_{j=1}^m{\lambda }_{ij}^2$. In this sense, the basic factor rotation methods can be viewed as those for exploratively analyzing multidimensional common-factor spaces. The interpretation of factors is made according to manifest variables with large weights in common factors. As far as we know, novel methods for factor rotation have not been investigated except for rotation methods similar to the above basic ones. In real data analyses, manifest variables are usually classified into some groups of variables in advance that may have common factors and concepts for themselves. For example, suppose we have a test battery including the following five subjects: Japanese, English, Social Science, Mathematics, and Natural Science. It is then reasonable to classify the five subjects into two groups, {Japanese, English, Social Science} and {Mathematics, Natural Science}. In such cases, it is meaningful to determine common factors related to the two manifest variable groups. For this objective, it is useful to develop a novel method to derive the common factors based on a factor contribution measure. In conventional methods of factor rotation, for example, as mentioned above, variation function (2) for the varimax method is not related to factor contribution.

An entropy-based method for measuring factor contribution was proposed by [6], and the method can measure factor contributions to manifest variables vectors and can decompose the factor contributions into those of manifest subvectors and individual manifest variables. By using the method, we can derive important common factors related to the manifest subvectors and the manifest variables. The aim of the present paper is to propose a new method for deriving simple structures based on entropy, that is, extracting common factors easy to interpret. In Section 2, an entropy-based method for measuring factor contribution [6] is reviewed to apply its properties for deriving simple structures in factor analysis models. Section 3 discusses canonical correlation analysis between common factors and manifest variables, and the contributions of common factors to the manifest variables are decomposed into components related to the extracted pairs of canonical variables. A numerical example is given to demonstrate the approach. In Section 4, canonical correlation analysis is applied to obtain common factors easy to interpret, and the contributions of the extracted factors are measured. Numerical examples are given to illustrate the present approach, and finally, Section 5 provides discussions and conclusions to summarize the present approach.

2. Entropy-Based Method for Measuring Factor Contributions

First, in order to derive factor contributions, factor analysis model (1) with error terms ${\epsilon }_i, i=1,2,\dots ,p$, which are normally distributed, can be discussed in the framework of generalized linear models (GLMs) [7]. A general path diagram among manifest variables $X_i, i=1,2,\dots ,p$ and common factors ${\xi }_j, j=1,2,\dots ,m$ in the factor analysis model is illustrated in Figure 1. The conditional density functions of manifest variables of $X_i, i=1,2,\dots ,p$, given the factors ${\xi }_j, j=1,2,\dots ,m$, are expressed as follows:

$$\begin{array}{c}{f}_i\left(x_i|\mathit{\xi }\right)=\frac{1}{\sqrt{2\pi {\omega }_i^2}}\exp \left(-\frac{{\left(x_i-{\sum }_{j=1}^m{\lambda }_{ij}{\xi }_j\right)}^2}{2{\omega }_i^2}\right)\\ =\exp \left(\frac{x_i{\sum }_{j=1}^m{\lambda }_{ij}{\xi }_j-\frac{1}{2}{\left({\sum }_{j=1}^m{\lambda }_{ij}{\xi }_j\right)}^2}{{\omega }_i^2}-\frac{x_i^2}{2{\omega }_i^2}-\log \sqrt{2\pi {\omega }_i^2}\right), i=1,2,\dots ,p.\end{array}$$

Let ${\theta }_i={\sum }_{j=1}^m{\lambda }_{ij}{\xi }_j$ and $d\left(x_i,{\omega }_i^2\right)=-\frac{x_i^2}{2{\omega }_i^2}-\log \sqrt{2\pi {\omega }_i^2}$. Then, the above density function is described in a GLM framework as

$$f_i\left(x_i|\mathit{\xi }\right)=\exp \left(\frac{x_i{\theta }_i-\frac{1}{2}{\theta }_i^2}{{\omega }_i^2}+d\left(x_i,{\omega }_i^2\right)\right), i=1,2,\dots ,p.$$ (3)

According to the local independence of the manifest variables in factor analysis model (1), the conditional density function of $\mathit{X}={\left(X_1,X_2,\dots ,X_p\right)}^T$ given $\mathit{\xi }={\left({\xi }_1, {\xi }_2,\dots ,{\xi }_m\right)}^T$ is expressed as

$$f\left(\mathit{x}|\mathit{\xi }\right)={\displaystyle \prod }_{i=1}^p\exp \left(\frac{x_i{\theta }_i-\frac{1}{2}{\theta }_i^2}{{\omega }_i^2}+d\left(x_i,{\omega }_i^2\right)\right)=\exp \left({{\displaystyle \sum }}_{i=1}^p\frac{x_i{\theta }_i-\frac{1}{2}{\theta }_i^2}{{\omega }_i^2}+{{\displaystyle \sum }}_{i=1}^{p}d\left(x_i,{\omega }_i^2\right)\right).$$ (4)

Figure 1.

Path diagram of a general factor analysis model.

Let $g\left(\mathit{\xi }\right)$ be the joint density function of common-factor vector $={\left({\xi }_1, {\xi }_2,\dots ,{\xi }_m\right)}^T$; let $f_i\left(x_i\right)$ be the marginal density functions of $X_i, i=1,2,\dots ,p$; and let us set

$$\mathrm{KL}\left(\mathit{X}, \mathit{\xi }\right)={{\displaystyle \iint }}^{ }f\left(\mathit{x}|\mathit{\xi }\right)g\left(\mathit{\xi }\right)\log \frac{f\left(\mathit{x}|\mathit{\xi }\right)}{f\left(\mathit{x}\right)}d\mathit{x}d\mathit{\xi }+{{\displaystyle \iint }}^{ }f\left(\mathit{x}\right)g\left(\mathit{\xi }\right)\log \frac{f\left(\mathit{x}\right)}{f\left(\mathit{x}|\mathit{\xi }\right)}d\mathit{x}d\mathit{\xi },$$ (5)

$$\begin{array}{c}\mathrm{KL}\left(X_i,\mathit{\xi }\right)={{\displaystyle \iint }}^{ }{f}_i\left(x_i|\mathit{\xi }\right)g\left(\mathit{\xi }\right)\log \frac{f_i\left(x_i|\mathit{\xi }\right)}{f_i\left(x_i\right)}d{x}_{i}d\mathit{\xi }+{{\displaystyle \iint }}^{ }{f}_i\left(x_i\right)g\left(\mathit{\xi }\right)\log \frac{f_i\left(x_i\right)}{f_i\left(x_i|\mathit{\xi }\right)}d{x}_{i}d\mathit{\xi }, \\ i=1,2,\dots ,p.\end{array}$$ (6)

where “KL” stands for “Kullback–Leibler information” [8]. From (3) and (4), we have

$$\mathrm{KL}\left(X_i,\mathit{\xi }\right)=\frac{\mathrm{Cov}\left(X_i,{\theta }_i\right)}{{\omega }_i^2}={\displaystyle \sum }_{j=1}^m\frac{{\lambda }_{ij}\mathrm{Cov}\left(X_i,{\xi }_j\right)}{{\omega }_i^2},i=1,2,\dots ,p;$$ (7)

$$\mathrm{KL}\left(\mathit{X}, \mathit{\xi }\right)={\displaystyle \sum }_{i=1}^p\frac{\mathrm{Cov}\left(X_i,{\theta }_i\right)}{{\omega }_i^2}={\displaystyle \sum }_{i=1}^p{\displaystyle \sum }_{j=1}^m\frac{{\lambda }_{ij}\mathrm{Cov}\left(X_i,{\xi }_j\right)}{{\omega }_i^2}.$$ (8)

The above quantities (7) and (8) are interpreted as the signal-to-noise ratios for dependent variables $X_i$ and predictors ${\theta }_i$; and the signal-to-noise ratio for dependent-variable vectors $\mathit{X}$ and common-factor vector $\mathit{\xi }$, respectively.

From (7) and (8), the following theorem can be derived [6]:

Theorem 1.

In factor analysis model (1), let $\mathit{X}={\left(X_1, X_2,\dots ,X_p\right)}^T$ and $\mathit{\xi }={\left({\xi }_1, {\xi }_2,\dots ,{\xi }_m\right)}^T$ . Then,

$$\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)={\displaystyle \sum }_{i=1}^p\mathrm{KL}\left(X_i,\mathit{\xi }\right).$$

Consistently, the following theorem, which is actually an extended version of Corollary 1 in [6], can be also obtained:

Theorem 2.

Let manifest variable subvectors ${\mathit{X}}^{\left(a\right)}, a=1,2,\dots ,A$ be any decomposition of manifest variable vector $\mathit{X}={\left(X_1, X_2,\dots ,X_p\right)}^T$ . Then,

$$\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)={\displaystyle \sum }_{a=1}^A\mathrm{KL}\left({\mathit{X}}^{\left(a\right)},\mathit{\xi }\right).$$ (9)

Following Eshima et al. [6], the contribution of factor vector $\mathit{\xi }={\left({\xi }_1, {\xi }_2,\dots ,{\xi }_m\right)}^T$ to manifest variable vector $\mathit{X}={\left(X_1, X_2,\dots ,X_p\right)}^T$ is thus defined as

$$\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)=\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right),$$

so that, in Theorem 2, the contributions of factor vector $\mathit{\xi }={\left({\xi }_1, {\xi }_2,\dots ,{\xi }_m\right)}^T$ to manifest variable vectors ${\mathit{X}}^{\left(a\right)}, a=1,2,\dots ,A$ are defined by

$$\mathrm{C}\left(\mathit{\xi }\to {\mathit{X}}^{\left(a\right)}\right)=\mathrm{KL}\left({\mathit{X}}^{\left(a\right)},\mathit{\xi }\right), a=1,2,\dots ,A.$$

Let ${\mathit{\xi }}^{\backslash \mathit{j}}$ be subvectors of all variables ${\xi }_i$ except ${\xi }_j$ from $\mathit{\xi }={\left({\xi }_1, {\xi }_2,\dots ,{\xi }_m\right)}^T$, i.e.,

$${\mathit{\xi }}^{\backslash \mathit{j}}={\left({\xi }_1, {\xi }_2,\dots ,{\xi }_{j-1},{\xi }_{j+1},\dots ,{\xi }_m\right)}^T, j=1,2,\dots ,m;$$

and let $\mathrm{KL}\left(\mathit{X},{\mathit{\xi }}^{\backslash \mathit{j}}|{\xi }_j\right)$ and $\mathrm{KL}\left({\mathit{X}}^{\left(a\right)},{\mathit{\xi }}^{\backslash \mathit{j}}|{\xi }_j\right)$ be the conditional Kullback–Leibler information as defined in (5) and (6). The contributions of common factors ${\xi }_j$ are defined by

$$\mathrm{C}\left({\xi }_j\to \mathit{X}\right)=\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)- \mathrm{KL}\left(\mathit{X},{\mathit{\xi }}^{\backslash \mathit{j}}|{\xi }_j\right),$$

$$\mathrm{C}\left({\xi }_j\to {\mathit{X}}^{\left(a\right)}\right)=\mathrm{KL}\left({\mathit{X}}^{\left(a\right)},\mathit{\xi }\right)-\mathrm{KL}\left({\mathit{X}}^{\left(a\right)},{\mathit{\xi }}^{\backslash \mathit{j}}|{\xi }_j\right), j=1,2,\dots ,m.$$

Remark 1.

Information $\mathrm{KL}\left(\mathit{X},{\mathit{\xi }}^{\backslash \mathit{j}}|{\xi }_j\right)$ and $\mathrm{KL}\left({\mathit{X}}^{\left(a\right)},{\mathit{\xi }}^{\backslash \mathit{j}}|{\xi }_j\right)$ can be expressed by using the conditional covariances $\mathrm{Cov}\left(X_i,{\theta }_i|{\xi }_j\right)$ . For example,

$$\mathrm{KL}\left(\mathit{X},{\mathit{\xi }}^{\backslash \mathit{j}}|{\xi }_j\right)={\displaystyle \sum }_{i=1}^p\frac{\mathrm{Cov}\left(X_i,{\theta }_i|{\xi }_j\right)}{{\omega }_i^2}.$$

Finally, the following decomposition of $\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)$ holds for orthogonal factors ([6], Theorem 3):

Theorem 3.

If the common factors are mutually independent, it follows that

$$\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)={\displaystyle \sum }_{j=1}^m{\displaystyle \sum }_{a=1}^A\mathrm{C}\left({\xi }_j\to {\mathit{X}}^{\left(a\right)}\right)={{\displaystyle \sum }}_{j=1}^m{{\displaystyle \sum }}_{i=1}^p\mathrm{C}\left({\xi }_j\to X_i\right).$$

The entropy coefficient of determination (ECD) [9] between $\mathit{\xi }$ and $\mathit{X}$ is defined by

$$\mathrm{ECD}\left(\mathit{\xi },\mathit{X}\right)=\frac{\mathrm{KL}\left(\mathit{\xi },\mathit{X}\right)}{\mathrm{KL}\left(\mathit{\xi },\mathit{X}\right)+1},$$

so that the total relative contribution of factor vector $\mathit{\xi }$ to manifest variable vector $\mathit{X}$ in entropy can be defined as

$$\widetilde{\mathrm{RC}}\left(\mathit{\xi }\to \mathit{X}\right)=\mathrm{ECD}\left(\mathit{\xi },\mathit{X}\right)=\frac{\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)}{\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)+1},$$

while, for a single factor ${\xi }_j$, two relative contribution ratios can be defined:

$$\begin{array}{c}\mathrm{RC}\left({\xi }_j\to \mathit{X}\right)=\frac{\mathrm{C}\left({\xi }_j\to \mathit{X}\right)}{\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)}=\frac{\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)- \mathrm{KL}\left(\mathit{X},{\mathit{\xi }}^{\backslash \mathit{j}}|{\xi }_j\right)}{\mathrm{KL}\left(\mathit{\xi },\mathit{X}\right)},\\ \widetilde{\mathrm{RC}}\left({\xi }_j\to \mathit{X}\right)=\frac{\mathrm{C}\left({\xi }_j\to \mathit{X}\right)}{\mathrm{KL}\left(\mathit{\xi },\mathit{X}\right)+1}=\frac{\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)- \mathrm{KL}\left(\mathit{X},{\mathit{\xi }}^{\backslash \mathit{j}}|{\xi }_j\right)}{\mathrm{KL}\left(\mathit{\xi },\mathit{X}\right)+1}\end{array}$$

(see [6] for details).

Second, factor analysis model (1) in a general case is discussed. Let $\Sigma$ be the variance–covariance matrix of manifest variable vector $\mathit{X}={\left(X_1,X_2,\dots ,X_p\right)}^T$; let $\mathsf{\Omega }$ be the $p\times p$ variance–covariance matrix of unique factor vector $\mathsf{\epsilon }={\left({\epsilon }_1,{\epsilon }_2,\dots ,{\epsilon }_p\right)}^T$; let $\mathsf{\Lambda }$ be the $p\times m$ factor loading matrix of ${\lambda }_{ij}$; and let $\mathsf{\Phi }$ be the correlation matrix of common-factor vector $\mathit{\xi }={\left({\xi }_1,{\xi }_2,\dots ,{\xi }_m\right)}^T$. Then, model (1) can be expressed as

$$\mathit{X}=\mathsf{\Lambda }\mathit{\xi }+\mathsf{\epsilon }$$

and we have

$$\Sigma =\mathsf{\Lambda }\mathsf{\Phi }{\mathsf{\Lambda }}^T+\mathsf{\Omega }.$$

Now, the above discussion is extended in a general factor analysis model (1) with the following variance–covariance matrix of $\mathit{X}$ and $\mathsf{\epsilon }$:

$$\left(\begin{array}{cc}\mathsf{\Lambda }\mathsf{\Phi }{\mathsf{\Lambda }}^T+\mathsf{\Omega }& \mathsf{\Lambda }\mathsf{\Phi }\\ \mathsf{\Phi }{\mathsf{\Lambda }}^T& \mathsf{\Phi }\end{array}\right).$$ (10)

Let $\mathit{\theta }=\mathsf{\Lambda }\mathit{\xi }$ be the predictor vector of manifest variable vector ${\mathit{X}}^T=\left(X_1,X_2,\dots ,X_p\right)$. Then, the contribution of common-factor vector $\mathit{\xi }$ to manifest variable vector $X$ is defined by the following generalized signal-to-noise ratio:

$$E\left({\mathit{X}}^T{\mathsf{\Omega }}^{-1}\mathit{\theta }\right)=\frac{E\left({\mathit{X}}^T\tilde{\mathsf{\Omega }}\mathsf{\Lambda }\mathsf{\xi }\right)}{\left|\mathsf{\Omega }\right|}=\frac{\mathrm{tr}\tilde{\mathsf{\Omega }}\mathsf{\Lambda }\mathsf{\Phi }{\mathsf{\Lambda }}^T}{\left|\mathsf{\Omega }\right|},$$ (11)

where $\tilde{\mathsf{\Omega }}$ is the cofactor matrix of $\mathsf{\Omega }$. The signal is $\mathrm{tr}\tilde{\mathsf{\Omega }}\mathsf{\Lambda }\mathsf{\Phi }{\mathsf{\Lambda }}^T$ and the noise $\left|\mathsf{\Omega }\right|$, and both are positive. Hence, the above quantity is defined as the explained entropy with the factor analysis model, and the same notation $KL\left(\mathit{X}, \mathit{\xi }\right)$ as above is used, having to do with the Kullback–Leibler information for the factor analysis model with normal distribution errors (4). Similarly, in the general model, as in (9), signal-to-noise ratio (11) is decomposed into

$$\frac{\mathrm{tr}\tilde{\mathsf{\Omega }}\mathsf{\Lambda }\mathsf{\Phi }{\mathsf{\Lambda }}^T}{\left|\mathsf{\Omega }\right|}={\displaystyle \sum }_{i=1}^p\frac{\mathrm{Cov}\left(X_i,{\theta }_i\right)}{{\omega }_i^2}={\displaystyle \sum }_{i=1}^p{\displaystyle \sum }_{j=1}^m\frac{{\lambda }_{ij}\mathrm{Cov}\left(X_i,{\xi }_j\right)}{{\omega }_i^2},$$

so the above theorems hold true as well. Thus, the results mentioned above are applicable to factor analysis models with error terms with non-normal distributions.

3. Canonical Factor Analysis

In order to derive interpretable factors from the common-factor space, we propose taking advantage of the results of canonical correlation analysis applied to manifest variables and common factors. This approach can be referred to as “canonical factor analysis” [10]. In the factor analysis model (1), the variance–covariance matrix of $\mathit{X}={\left(X_1, X_2,\dots ,X_p\right)}^T$ and $\mathit{\xi }={\left({\xi }_1, {\xi }_2,\dots ,{\xi }_m\right)}^T$ is given by (10). Then, we have the following theorem:

Theorem 4.

For canonical correlation coefficients ${\rho }_k, k=1,2,\dots ,m$ between $\mathit{X}$ and $\mathit{\xi }$ in factor analysis model (1) with (10), it follows that

$$\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)={\displaystyle \sum }_{j=1}^m\frac{{\rho }_j^2}{1-{\rho }_j^2}.$$

Proof. 

Let ${\mathit{B}}^{\left(1\right)}$, ${\mathit{B}}^{\left(2\right)}$, and $\mathit{F}$ be $m\times p$, $\left(p-m\right)\times p$, and $m\times m$ matrices, respectively; let ${\mathit{V}}^{\left(1\right)}={\left(V_1,V_2,\dots ,V_m\right)}^T={\mathit{B}}^{\left(1\right)}\mathit{X}$, ${\mathit{V}}^{\left(2\right)}={\mathit{B}}^{\left(2\right)}\mathit{X}$, and $\mathit{\eta }={\left({\eta }_1,{\eta }_2,\dots ,{\eta }_m\right)}^T=\mathit{F}\mathit{\xi }$. It is assumed that $\left(V_j,{\eta }_j\right)$ are the pairs of canonical variables with correlation coefficients ${\rho }_j, j=1,2,\dots ,m$; that matrices $\left(\begin{array}{c}{\mathit{B}}^{\left(1\right)}\\ {\mathit{B}}^{\left(2\right)}\end{array}\right)$ and $\mathit{F}$ are nonsingular; and that ${\mathit{V}}^{\left(1\right)}$ and ${\mathit{V}}^{\left(2\right)}$ are statistically independent. Since all pairs of canonical variables $\left(V_j,{\eta }_j\right)$ and ${\mathit{V}}^{\left(2\right)}$ are mutually independent, we have

$$\mathrm{KL}\left({\mathit{V}}^{\left(2\right)},\mathit{\eta }\right)=0, \mathrm{KL}\left({\mathit{V}}^{\left(1\right)},{\eta }_j\right)=\mathrm{KL}\left(V_j,{\eta }_j\right), j=1,2,\dots ,m.$$

From Theorem 2, it follows that

$$\begin{array}{c}\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)=\mathrm{KL}\left(\mathit{V},\mathit{F}\mathit{\xi }\right)=\mathrm{KL}\left(\left(\begin{array}{c}{\mathit{V}}^{\left(1\right)}\\ {\mathit{V}}^{\left(2\right)}\end{array}\right),\mathit{\eta }\right)=\mathrm{KL}\left({\mathit{V}}^{\left(1\right)},\mathit{\eta }\right)+\mathrm{KL}\left({\mathit{V}}^{\left(2\right)},\mathit{\eta }\right)=\mathrm{KL}\left({\mathit{V}}^{\left(1\right)},\mathit{\eta }\right)={\displaystyle {\displaystyle \sum }_{j=1}^m}\mathrm{KL}\left({\mathit{V}}^{\left(1\right)},{\eta }_j\right)\\ ={\displaystyle {\displaystyle \sum }_{j=1}^m}\mathrm{KL}\left(V_j,{\eta }_j\right)={\displaystyle {\displaystyle \sum }_{j=1}^m}\frac{{\rho }_j^2}{1-{\rho }_j^2}.\end{array}$$

This completes the theorem. □

In the proof of the above theorem, we have

$$\mathrm{KL}\left(\mathit{X},{\eta }_j\right)=\mathrm{KL}\left(V_j,{\eta }_j\right)=\frac{{\rho }_j^2}{1-{\rho }_j^2}, j=1,2,\dots ,m.$$ (12)

It implies that

$$\mathrm{C}\left({\eta }_j\to \mathit{X}\right)=\mathrm{C}\left({\eta }_j\to V_j\right)=\frac{{\rho }_j^2}{1-{\rho }_j^2};$$

$$\widetilde{\mathrm{RC}}\left({\eta }_j\to \mathit{X}\right)=\frac{\mathrm{KL}\left(\mathit{X},{\eta }_j\right)}{\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)+1}=\frac{\mathrm{KL}\left(\mathit{X},{\eta }_j\right)}{\mathrm{KL}\left(\mathit{\eta },\mathit{V}\right)+1}=\frac{\frac{{\rho }_j^2}{1-{\rho }_j^2}}{{\sum }_{a=1}^m\frac{{\rho }_a^2}{1-{\rho }_a^2}+1};$$

$$\mathrm{RC}\left({\eta }_j\to \mathit{X}\right)=\frac{\mathrm{KL}\left(\mathit{X},{\eta }_j\right)}{\mathrm{KL}\left(\mathit{\xi },\mathit{X}\right)}=\frac{\mathrm{KL}\left(V_j,{\eta }_j\right)}{\mathrm{KL}\left(\mathit{V},\mathit{\eta }\right)}=\frac{\frac{{\rho }_j^2}{1-{\rho }_j^2}}{{\sum }_{a=1}^m\frac{{\rho }_a^2}{1-{\rho }_a^2}}, j=1,2,\dots ,m.$$

Theorem 4 shows that the contribution of common-factor vector $\mathit{\xi }$ to manifest variable vector $\mathit{X}$ is decomposed into those of canonical common factors ${\eta }_j$, i.e.,

$$\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)={\displaystyle \sum }_{j=1}^m\mathrm{KL}\left(\mathit{X},{\eta }_j\right)={\displaystyle \sum }_{j=1}^m\mathrm{KL}\left(V_j,{\eta }_j\right), j=1,2,\dots ,m.$$

Let us assume

$$1>{\rho }_1^2\ge {\rho }_2^2\ge \dots \ge {\rho }_m^2\ge 0.$$ (13)

According to the entropy-based criterion in Theorem 4, the order of importance of canonical common factors is that of canonical correlation coefficients. The interpretation of factors ${\eta }_j$ can be made with the corresponding manifest canonical variables $V_j$ and the factor loading matrix of canonical common factors $\mathit{\eta }=\mathit{F}\mathit{\xi }$. For the canonical common factors, the factor loading matrix can be obtained as ${\mathsf{\Lambda }}^{\ast }=\mathsf{\Lambda }{\mathit{F}}^{-1}$. We refer to the canonical correlation analysis in Theorem 4 as canonical factor analysis [10].

Theorem 5.

In factor analysis model (1), for any $p\times p$ and $m\times m$ nonsingular matrices $\mathit{P}$ and $\mathit{Q}$ , the canonical factor analysis between manifest variable vector $\mathit{P}\mathit{X}$ and common-factor vector $\mathit{Q}\mathit{\xi }$ is invariant.

Proof. 

Since the variance–covariance matrix of $\mathit{P}\mathit{X}$ and $\mathit{Q}\mathit{\xi }$ is given by

$$\left(\begin{array}{cc}\mathit{P}& 0\\ 0& \mathit{Q}\end{array}\right)\left(\begin{array}{cc}\Sigma & {\mathsf{\Lambda }}^{\mathrm{T}}\\ \mathsf{\Lambda }& {\mathit{I}}_m\end{array}\right){\left(\begin{array}{cc}\mathit{P}& 0\\ 0& \mathit{Q}\end{array}\right)}^T,$$

the theorem follows. □

Notice that we also have

$$\mathrm{KL}\left(\mathit{P}\mathit{X},\mathit{Q}\mathit{\xi }\right)=\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right).$$

From the above theorem, the results of the canonical factor analysis do not depend on the initial common factors ${\xi }_j$ in factor analysis model (1). For factor analysis model (1), it follows that

$$\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)={\displaystyle \sum }_{j=1}^m\mathrm{KL}\left(V_j,{\eta }_j\right)={\displaystyle \sum }_{i=1}^p\mathrm{KL}\left(X_i,\mathit{\xi }\right),$$

implying that

$$\frac{\mathrm{tr}\tilde{\mathsf{\Omega }}\mathsf{\Lambda }\mathsf{\Phi }{\mathsf{\Lambda }}^T}{\left|\mathsf{\Omega }\right|}={\displaystyle \sum }_{j=1}^m\frac{{\rho }_j^2}{1-{\rho }_j^2} ={\displaystyle \sum }_{i=1}^p\frac{R_i^2}{1-R_i^2},$$

where $R_i$ are the multiple correlation coefficients between manifest variables $X_i$ and factor vector $\mathit{\xi }=\left({\xi }_1,{\xi }_2,\dots ,{\xi }_m\right), i=1,2,\dots ,p$.

Numerical Example 1

Table 1 shows the results of orthogonal factor analysis (varimax method by S-PLUS ver. 8.2) as reported in [6]; the same example is used here to demonstrate the canonical factor analysis mentioned above. In Table 1, manifest variables $X_1, X_2, \mathrm{and} X_3$ are scores in some subjects in the liberal arts, while variables $X_4 \mathrm{and} X_5$ are those in the sciences. We refer to the factors as the initial common factors. In this example, from Table 1, the variance–covariance matrices in (10) are given as follows:

$$\mathsf{\Sigma }=\left(\begin{array}{c}\begin{array}{ccccc}1& 0.54& 0.39& 0.42& 0.36\\ 0.54& 1& 0.49& 0.38& 0.22\\ 0.39& 0.49& 1& 0.21& 0\\ 0.42& 0.38& 0.21& 1& 0.54\\ 0.36& 0.22& 0& 0.54& 1\end{array}\end{array}\right),$$

$$\mathsf{\Phi }=\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right).$$

where covariance matrix ${\mathsf{\Lambda }}^T$ is given in Table 1.

Table 1.

Factor loadings of orthogonal (varimax) factor analysis.

	${\mathit{X}}_1$	${\mathit{X}}_2$	${\mathit{X}}_3$	${\mathit{X}}_4$	${\mathit{X}}_5$
${\xi }_1$	0.60	0.75	0.65	0.32	0.00
${\xi }_2$	0.39	0.24	0.00	0.59	0.92
uniqueness	0.50	0.38	0.58	0.55	0.16

Uniqueness is the proportion of unique factor ${\epsilon }_i$ related to manifest variable $X_i$.

From the above matrices, to obtain the pairs of canonical variables, linear transformation matrices ${\mathit{B}}^{\left(1\right)}$ and $\mathit{F}$ in Theorem 4 are as follows:

$${\mathit{B}}^{\left(1\right)}=\left(\begin{array}{cc}\begin{array}{cc}0.19& 0.20\\ 0.32& 0.58\end{array}& \begin{array}{ccc}0.06& 0.20& 0.94\\ 0.37& 0.00& -0.65\end{array}\end{array}\right),$$

and

$$\mathit{F}=\left(\begin{array}{cc}0.32& 0.95\\ 0.95& -0.32\end{array}\right).$$

By the above matrices, we have the following pairs of canonical variables $\left(V_i,{\eta }_i\right)$ and their squared canonical correlation coefficients ${\rho }_i^2$:

$$\{\begin{array}{c}{V}_1=0.19X_1+0.20X_2+0.06X_3+0.20X_4+0.94X_5,\\ {\eta }_1=0.32{\xi }_1+0.95{\xi }_2,\\ {\rho }_1^2=0.88,\end{array}$$

$$\{\begin{array}{c}{V}_2=0.32X_1+0.58X_2+0.37X_3+0.07X_4-0.65X_5,\\ {\eta }_2=0.95{\xi }_1-0.32{\xi }_2,\\ {\rho }_2^2=0.73.\end{array}$$

According to the above canonical variables, the factor loading for canonical factors ${\eta }_i, i=1,2$ is calculated with the initial loading matrix $\mathsf{\Lambda }$ and the rotation matrix $\mathit{F}$, and we have

$$\begin{array}{c}{\mathsf{\Lambda }}^{\ast T}={\left(\mathsf{\Lambda }{\mathit{F}}^{-1}\right)}^T={\left(\begin{array}{cc}0.32& 0.95\\ 0.95& -0.32\end{array}\right)}^{-1}\left(\begin{array}{cc}\begin{array}{cc}0.6& 0.75\\ 0.39& 0.24\end{array}& \begin{array}{ccc}0.65& 0.32& 0.00\\ 0.00& 0.59& 0.92\end{array}\end{array}\right)\\ =\left(\begin{array}{cc}\begin{array}{cc}0.56& 0.47\\ 0.45& 0.64\end{array}& \begin{array}{ccc}0.21& 0.66& 0.87\\ 0.62& 0.12& -0.29\end{array}\end{array}\right).\end{array}$$

From the above results, the first canonical factor ${\eta }_1$ can be viewed as a general common ability (factor) to solve all five subjects. The second factor ${\eta }_2$ can be regarded as a factor related to subjects in the liberal arts, which is independent of the first canonical factor. In the canonical correlation analysis, the contributions of canonical factors are calculated. Since the multiple correlation coefficient between ${\eta }_1$ and $\mathit{X}={\left(X_1, X_2,\dots ,X_5\right)}^T$ is ${\rho }_1^2=0.88$ and that between ${\eta }_2$ and $\mathit{X}$ is ${\rho }_2^2=0.73$, we have

$$\mathrm{C}\left({\eta }_1\to \mathit{X}\right)=\frac{{\rho }_1^2}{1-{\rho }_1^2}=7.06, \mathrm{C}\left({\eta }_2\to \mathit{X}\right)=\frac{{\rho }_2^2}{1-{\rho }_2^2}=2.70.$$

Let $\mathit{\xi }=\left({\xi }_1,{\xi }_2\right)$. From the above results, we have

$$\begin{array}{c}\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)=\mathrm{KL}\left(\mathit{\xi },\mathit{X}\right)=\mathrm{C}\left({\eta }_1\to \mathit{X}\right)+ \mathrm{C}\left({\eta }_2\to \mathit{X}\right)=9.86,\\ \widetilde{\mathrm{CR}}\left(\mathit{\xi }\to \mathit{X}\right)=\frac{\mathrm{KL}\left(\mathit{\xi },\mathit{X}\right)}{\mathrm{KL}\left(\mathit{\xi },\mathit{X}\right)+1}=0.91\left(=\mathrm{ECD}\left(\mathit{\xi },\mathit{X}\right)\right).\end{array}$$

From this, $91\%$ of the variation of manifest random vector $\mathit{X}$ in entropy is explained by the common latent factors $\mathit{\xi }$. The contribution ratios of canonical common factors are calculated as follows:

$$\mathrm{CR}\left({\eta }_1\to \mathit{X}\right)=\frac{7.06}{7.06+2.70}=0.72, \mathrm{CR}\left({\eta }_2\to \mathit{X}\right)=2.70.$$

The contribution of the first canonical factor is about 2.6 times greater than that of the second one.

4. Deriving Important Common Factors Based on Decomposition of Manifest Variables into Subsets

From (9) in Theorem 2, $\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)$ is decomposed into those for manifest variable subvectors ${\mathit{X}}^{\left(a\right)}$, $\mathrm{KL}\left({\mathit{X}}^{\left(a\right)},\mathit{\xi }\right), a=1,2,\dots ,A$. Thus, we have the following theorem:

Theorem 6.

Let manifest variable vector $\mathit{X}$be decomposed into subvectors${\mathit{X}}^{\left(a\right)}, a=1,2,\dots ,A$. Let ${\rho }_{\left(a\right)j}, j=1,2,\dots ,m_{\left(a\right)}$be the canonical correlation coefficients between manifest variable subvector${\mathit{X}}^{\left(a\right)}$and common-factor vector $\mathit{\xi }$,$a=1,2,\dots ,A$in the factor analysis model (1), where $m_{\left(a\right)}\le \min \left\{\mathrm{dimension} \mathrm{of} {\mathit{X}}^{\left(a\right)},m\right\}$. Then,$\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)$is decomposed into canonical components as follows:

$$\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)={\displaystyle \sum }_{a=1}^A{\displaystyle \sum }_{j=1}^{m_{\left(a\right)}}\frac{{\rho }_{\left(a\right)j}^2}{1-{\rho }_{\left(a\right)j}^2}.$$

Proof. 

For manifest variable vector ${\mathit{X}}^{\left(a\right)}$ and common-factor vector $\mathit{\xi }$, applying canonical correlation analysis, we have $m_{\left(a\right)}$ pairs of canonical variables $\left(V_j^{\left(\alpha \right)},{\eta }_j^{\left(\alpha \right)}\right)$ with squared canonical correlation coefficients ${\rho }_{\left(a\right)j}^2, j=1,2,\dots ,m_{\left(a\right)}$. Then, applying Theorem 4 to $\mathrm{KL}\left({\mathit{X}}^{\left(a\right)},\mathit{\xi }\right)$ it follows that

$$\mathrm{KL}\left({\mathit{X}}^{\left(a\right)},\mathit{\xi }\right)={\displaystyle \sum }_{j=1}^{m_{\left(a\right)}}\mathrm{KL}\left(V_j^{\left(\alpha \right)},{\eta }_j^{\left(\alpha \right)}\right)={\displaystyle \sum }_{j=1}^{m_{\left(a\right)}}\frac{{\rho }_{\left(a\right)j}^2}{1-{\rho }_{\left(a\right)j}^2}, a=1,2,\dots ,A.$$

From Theorem 2, the theorem follows. □

Remark 2.

As shown in the above theorem, the following relations hold:

$$\mathrm{KL}\left({\mathit{X}}^{\left(a\right)},{\eta }_j^{\left(\alpha \right)}\right)=\mathrm{KL}\left(V_j^{\left(\alpha \right)},{\eta }_j^{\left(\alpha \right)}\right)=\frac{{\rho }_{\left(a\right)j}^2}{1-{\rho }_{\left(a\right)j}^2}, j=1,2,\dots ,m_{\left(a\right)}; a=1,2,\dots ,A.$$

In this sense,

$$C\left({\eta }_j^{\left(a\right)}\to {\mathit{X}}^{\left(a\right)}\right)=\frac{{\rho }_{\left(a\right)j}^2}{1-{\rho }_{\left(a\right)j}^2}, j=1,2,\dots ,m_{\left(a\right)}; a=1,2,\dots ,A.$$

To derive important common factors, the above theorem can be used. In many of the data in factor analysis, manifest variables can be classified into subsets that have common concepts (factors) to be measured. For example, in the data used for Table 1, it is meaningful to classify the five variables into two subsets ${\mathit{X}}^{\left(1\right)}=\left(X_1,X_2,X_3\right)$ and ${\mathit{X}}^{\left(2\right)}=\left(X_4,X_5\right)$, where the first subset is related to the liberal arts and the second one is related to the sciences. In $\left({\mathit{X}}^{\left(1\right)},\mathit{\xi }\right)$ and $\left({\mathit{X}}^{\left(2\right)},\mathit{\xi }\right)$, it is possible to derive the latent ability for the liberal arts and that for the sciences, respectively.

4.1. Numerical Example 1 (Continued)

For $\left({\mathit{X}}^{\left(1\right)},\mathit{\xi }\right)$ and $\left({\mathit{X}}^{\left(2\right)},\mathit{\xi }\right)$, two sets of canonical variables are obtained, respectively, as follows:

$$\{\begin{array}{c}{\eta }_1^{\left(1\right)}=0.95{\xi }_1+0.32{\xi }_2, V_1^{\left(1\right)}=0.52X_1+0.76X_2+0.39X_3, {\rho }_{\left(1\right)1}^2=0.77,\\ {\eta }_2^{\left(1\right)}=0.32{\xi }_1-0.95{\xi }_2, V_2^{\left(1\right)}=0.71X_1-0.07X_2-0.71X_3, {\rho }_{\left(1\right)2}^2=0.12,\end{array}$$

$$\{\begin{array}{c}{\eta }_1^{\left(2\right)}=0.06{\xi }_1+1.00{\xi }_2, V_1^{\left(2\right)}=0.18X_4+0.98X_5, {\rho }_{\left(2\right)1}^2=0.97,\\ {\eta }_2^{\left(2\right)}=1.00{\xi }_1-0.06{\xi }_2, V_2^{\left(2\right)}=0.83X_4-0.55X_5, {\rho }_{\left(2\right)2}^2=0.03.\end{array}$$

According to the above canonical variables, we have the following factor contributions:

$$\{\begin{array}{c}\mathrm{C}\left({\eta }_1^{\left(1\right)}\to {\mathit{X}}^{\left(1\right)}\right)=\mathrm{C}\left({\eta }_1^{\left(1\right)}\to V_1^{\left(1\right)}\right)=\frac{0.77}{1-0.77}=3.27, \mathrm{CR}\left({\eta }_1^{\left(1\right)}\to {\mathit{X}}^{\left(1\right)}\right)=0.96,\\ \mathrm{C}\left({\eta }_2^{\left(1\right)}\to {\mathit{X}}^{\left(1\right)}\right)=\frac{0.12}{1-0.12}=0.14. \mathrm{CR}\left({\eta }_2^{\left(1\right)}\to {\mathit{X}}^{\left(1\right)}\right)=0.04;\end{array}$$

$$\{\begin{array}{c}\mathrm{C}\left({\eta }_1^{\left(2\right)}\to {\mathit{X}}^{\left(2\right)}\right)=6.14, \mathrm{CR}\left({\eta }_1^{\left(2\right)}\to {\mathit{X}}^{\left(2\right)}\right)=0.97,\\ \mathrm{C}\left({\eta }_2^{\left(2\right)}\to {\mathit{X}}^{\left(2\right)}\right)=0.17, \mathrm{CR}\left({\eta }_2^{\left(2\right)}\to {\mathit{X}}^{\left(2\right)}\right)=0.03.\end{array}$$

From the above results, canonical factors ${\eta }_1^{\left(1\right)}$ and ${\eta }_1^{\left(2\right)}$ can be interpreted as general common factors for the liberal arts and for the sciences, respectively. By using the factors, the factor loadings are given in Table 2. In this case, Table 2 is similar to Table 1; however, the factor analysis model is oblique and the correlation coefficient between ${\eta }_1^{\left(1\right)}$ and ${\eta }_1^{\left(2\right)}$ is 0.374. The contributions of the factors to manifest variable vector $\mathit{X}=\left(X_1,X_2,X_3,X_4,X_5\right)=\left({\mathit{X}}^{\left(1\right)},{\mathit{X}}^{\left(2\right)}\right)$ are calculated as follows:

$$\{\begin{array}{c}\mathrm{C}\left({\eta }_1^{\left(1\right)}\to \mathit{X}\right)=6.563, \mathrm{CR}\left({\eta }_1^{\left(1\right)}\to \mathit{X}\right)=0.687,\widetilde{\mathrm{CR}}\left({\eta }_1^{\left(1\right)}\to \mathit{X}\right)=0.60, \\ \mathrm{C}\left({\eta }_1^{\left(2\right)}\to \mathit{X}\right)=4.223. \mathrm{CR}\left({\eta }_1^{\left(2\right)}\to \mathit{X}\right)=0.442,\widetilde{\mathrm{CR}}\left({\eta }_1^{\left(2\right)}\to \mathit{X}\right)=0.39.\end{array}$$

In this case, factors ${\eta }_1^{\left(1\right)}$ and ${\eta }_1^{\left(2\right)}$ are correlated, so it follows that

$$\mathrm{CR}\left({\eta }_1^{\left(1\right)}\to \mathit{X}\right)+\mathrm{CR}\left({\eta }_1^{\left(2\right)}\to \mathit{X}\right)=1.129>1.$$

Table 2.

Factor loadings by using canonical common factors ${\eta }_1^{\left(1\right)}$ and ${\eta }_1^{\left(2\right)}$.

	${\mathit{X}}_1$	${\mathit{X}}_2$	${\mathit{X}}_3$	${\mathit{X}}_4$	${\mathit{X}}_5$
${\eta }_1^{\left(1\right)}$	0.62	0.80	0.70	0.31	$-0.06$
${\eta }_1^{\left(2\right)}$	0.19	$-0.02$	$-0.22$	0.49	0.94
uniqueness	0.50	0.38	0.58	0.55	0.16

4.2. Numerical Example 2

Table 3 shows the results of the maximum likelihood factor analysis (orthogonal) for six scores $X_i, i=1,2,\dots ,6$ ([11], pp. 61–65); such results are treated as the initial estimates in the present analysis. In this example, variables are classified into the following three groups: variable $X_1$ is related to the Spearman’s $g$ factor; variables $X_2$, $X_3$, and $X_4$ account for problem-solving ability; and variables $X_5$ and $X_6$ are associated with verbal ability [11]; however, it is difficult to explain the three factors by using Table 3. In this example, the present approach is employed for deriving the three factors. From (10) and Table 3, the correlation matrix of the manifest variables is given as follows:

$$\widehat{\Sigma }=\left(\begin{array}{cc}\begin{array}{ccc}1& 0.417& 0.576\\ 0.417& 1& 0.567\\ 0.576& 0.567& 1\end{array}& \begin{array}{ccc}0.312& 0.576& 0.514\\ 0.306& 0.265& 0.263\\ 0.427& 0.355& 0.354\end{array}\\ \begin{array}{ccc}0.312& 0.306& 0.427\\ 0.576& 0.265& 0.355\\ 0.514& 0.263& 0.354\end{array}& \begin{array}{ccc}1& 0.193& 0.193\\ 0.193& 1& 0.799\\ 0.193& 0.799& 1\end{array}\end{array}\right).$$

Let ${\mathit{X}}^{\left(2\right)}=\left(X_2,X_3,X_4\right)$, let ${\mathit{X}}^{\left(3\right)}=\left(X_5,X_6\right)$, and let $\mathit{\xi }=\left({\xi }_1,{\xi }_2\right)$. Canonical correlation analysis is carried out for $\left(X_1,\mathit{\xi }\right)$, $\left({\mathit{X}}^{\left(2\right)},\mathit{\xi }\right)$, and $\left({\mathit{X}}^{\left(3\right)},\mathit{\xi }\right)$, and we have the following canonical variables, respectively:

$${\eta }_1^{\left(1\right)}=\frac{1}{\sqrt{{0.67}^2+{0.37}^2}}\left(0.64{\xi }_1+0.37{\xi }_2\right)=0.87{\xi }_1+0.50{\xi }_2, V_1^{\left(1\right)}=X_1, {\rho }_{\left(1\right)1}^2=0.55$$

$$\{\begin{array}{c}{\eta }_1^{\left(2\right)}=0.52{\xi }_1+0.85{\xi }_2, V_1^{\left(2\right)}=0.24X_2+0.96X_3+0.14X_4, {\rho }_{\left(2\right)1}^2=0.83,\\ {\eta }_2^{\left(2\right)}=0.85{\xi }_1-0.52{\xi }_2, V_2^{\left(2\right)}=0.81X_2-0.59X_3-0.02X_4, {\rho }_{\left(1\right)2}^2=0.00,\end{array}$$

$$\{\begin{array}{c}{\eta }_1^{\left(3\right)}=0.99{\xi }_1-0.12{\xi }_2, V_1^{\left(3\right)}=0.99X_5+0.11X_6, {\rho }_{\left(3\right)1}^2=0.96,\\ {\eta }_2^{\left(3\right)}=0.12{\xi }_1+0.99{\xi }_2, V_2^{\left(3\right)}=0.64X_5-0.77X_6, {\rho }_{\left(3\right)2}^2=0.01.\end{array}$$

The contributions of canonical factors ${\eta }_i^{\left(k\right)}, i=1,2;k=2.3$ are calculated as follows:

$$\{\begin{array}{c}\mathrm{C}\left({\eta }_1^{\left(2\right)}\to {\mathit{X}}^{\left(2\right)}\right)=\mathrm{C}\left({\eta }_1^{\left(2\right)}\to V_1^{\left(2\right)}\right)=\frac{0.83}{1-0.83}=4.88, \mathrm{CR}\left({\eta }_1^{\left(2\right)}\to {\mathit{X}}^{\left(2\right)}\right)=1.00,\\ \mathrm{C}\left({\eta }_2^{\left(2\right)}\to {\mathit{X}}^{\left(2\right)}\right)=\frac{0.00}{1-0.00}=0.00. \mathrm{CR}\left({\eta }_2^{\left(1\right)}\to {\mathit{X}}^{\left(1\right)}\right)=0.00;\end{array}$$

$$\{\begin{array}{c}\mathrm{C}\left({\eta }_1^{\left(3\right)}\to {\mathit{X}}^{\left(3\right)}\right)=\frac{0.96}{1-0.96}=24.00, \mathrm{CR}\left({\eta }_1^{\left(3\right)}\to {\mathit{X}}^{\left(3\right)}\right)=0.99,\\ \mathrm{C}\left({\eta }_2^{\left(2\right)}\to {\mathit{X}}^{\left(2\right)}\right)=0.01, \mathrm{CR}\left({\eta }_2^{\left(2\right)}\to {\mathit{X}}^{\left(2\right)}\right)=0.01.\end{array}$$

The common factor ${\eta }_1^{\left(1\right)}\left(=g\right)$ can be interpreted as the Spearman’s $g$ factor (general intelligence) and canonical common factors ${\eta }_1^{\left(2\right)}$ and ${\eta }_1^{\left(3\right)}$ can be interpreted as problem-solving ability and verbal ability, respectively. The correlation coefficients between the three factors are given by

$$\mathrm{Corr}\left(g,{\eta }_1^{\left(2\right)}\right)=0.88, \mathrm{Corr}\left(g,{\eta }_1^{\left(3\right)}\right)=0.80, \mathrm{Corr}\left({\eta }_1^{\left(2\right)},{\eta }_1^{\left(3\right)}\right)=0.42.$$

The contributions of the above three factors to manifest variable vector $\mathit{X}=\left(X_1,X_2,X_3,X_4,X_5,X_6\right)$ are computed as follows:

$$\{\begin{array}{ccc}\mathrm{C}\left(g\to \mathit{X}\right)=19.93,& \mathrm{CR}\left(g\to \mathit{X}\right)=0.68& \widetilde{\mathrm{CR}}\left(g\to \mathit{X}\right)=0.66\\ \mathrm{C}\left({\eta }_1^{\left(2\right)}\to \mathit{X}\right)=9.77,& \mathrm{CR}\left({\eta }_1^{\left(2\right)}\to \mathit{X}\right)=0.33,& \widetilde{\mathrm{CR}}\left({\eta }_1^{\left(2\right)}\to \mathit{X}\right)=0.32,\\ \mathrm{C}\left({\eta }_1^{\left(3\right)}\to \mathit{X}\right)=25.02,& \mathrm{CR}\left({\eta }_1^{\left(3\right)}\to \mathit{X}\right)=0.85,& \widetilde{\mathrm{CR}}\left({\eta }_1^{\left(3\right)}\to \mathit{X}\right)=0.82.\end{array}$$

The common-factor space is two-dimensional, and the factor loadings with common factors ${\eta }_1^{\left(2\right)}$ and ${\eta }_1^{\left(3\right)}$ are calculated as in Table 4. The table shows a clear interpretation of the common factors. Thus, the present method is effective for deriving interpretable factors in situations such as that of this example. The expressions of the factor analysis model can also be given by factor vectors $\left(g,{\eta }_1^{\left(2\right)}\right)$ and $\left(g,{\eta }_1^{\left(3\right)}\right)$, respectively. The present method is applicable for any subsets of manifest variables.

Table 3.

The initial maximum likelihood estimates of factor loadings (varimax).

	${\mathit{X}}_1$	${\mathit{X}}_2$	${\mathit{X}}_3$	${\mathit{X}}_4$	${\mathit{X}}_5$	${\mathit{X}}_6$
${\xi }_1$	0.64	0.34	0.46	0.25	0.97	0.82
${\xi }_2$	0.37	0.54	0.76	0.41	$-0.12$	$-0.03$
uniqueness	0.45	0.59	0.21	0.77	0.04	0.33

Table 4.

The factor loadings with common factors ${\eta }_1^{\left(2\right)}$ and ${\eta }_1^{\left(3\right)}$.

	${\mathit{X}}_1$	${\mathit{X}}_2$	${\mathit{X}}_3$	${\mathit{X}}_4$	${\mathit{X}}_5$	${\mathit{X}}_6$
${\eta }_1^{\left(2\right)}$	0.49	0.63	0.89	0.48	$-0.01$	0.07
${\eta }_1^{\left(3\right)}$	0.39	0.01	0.00	0.00	0.98	$0.79$
uniqueness	0.45	0.59	0.21	0.77	0.04	0.33

5. Discussion

In order to find interpretable common factors in factor analysis models, methods of factor rotation are often used. The methods are based on maximizations of variation functions of squares of factor loadings, and orthogonal or oblique factors are applied. The factors derived by the conventional methods may be interpretable; however, it may be more useful to propose a method for detecting interpretable common factors based on factor contribution measurement, i.e., importance of common factors. An entropy-based method for measuring factor contribution [6] can measure the contribution of the common-factor vector to the manifest variable vector, and one can decompose such a contribution into those of single manifest variables (Theorem 1) and into that of some manifest variable subvectors as well (Theorem 2). A characterization in the case of orthogonal factors can be also given (Theorem 3). The paper shows that the most important common factor with respect to entropy can be identified by using canonical correlation analysis between the factor vector and the manifest variable vector (Theorem 4). Theorem 4 shows that the contribution of the common-factor vector to the manifest variable vector can be decomposed into those of canonical factors and that the order of canonical correlation coefficients is that of factor contributions. In most multivariate data, manifest variables can be naturally classified into subsets according to common concepts as in Examples 1 and 2. By using Theorems 2 and 5, canonical correlation analysis can also be applied to derive canonical common factors from subsets of manifest variables and the initial common-factor vector (Theorem 6). According to the analysis, interpretable common factors can be obtained easily, as demonstrated in Examples 1 and 2. In Example 1, Table 1 and Table 2 have similar factor patterns; however, the derived factors in Table 1 are orthogonal and those in Table 2 are oblique. In Example 2, it may be difficult to interpret the factors in Table 3 produced by the varimax method. On the other hand, Table 4, obtained by using the present method, can be interpreted clearly. Finally, according to Theorem 5, the present method produces results that are invariant with respect to linear transformations of common factors, so that the method is independent of the initial common factors. The present method is the first one to derive interpretable factors based on a factor contribution measure, and the interpretable factors can be obtained easily through canonical correlation analysis between manifest variable subvectors and the factor vectors.

Author Contributions

Conceptualization, N.E.; methodology, N.E., C.G.B., M.T., T.K.; formal analysis, N.E.; writing—original draft preparation, N.E.; writing—review and editing, C.G.B.; funding acquisition, T.K. All authors have read and agreed to the published version of the manuscript.

Funding

This research was supported by the Grant-in-aid for Scientific Research 18K11200, Ministry of Education, Culture, Sports, Science, and Technology of Japan.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data sharing is not applicable to this article.

Conflicts of Interest

The authors declare no conflict of interest.