Convergence, Admissibility, and Fit of Alternative Confirmatory Factor Analysis Models for MTMM Data

Abstract

We compared six different analytic models for multitrait–multimethod (MTMM) data in terms of convergence, admissibility, and model fit to 258 samples of previously reported data. Two well-known models, the correlated trait–correlated method (CTCM) and the correlated trait–correlated uniqueness (CTCU) models, were fit for reference purposes in comparison to four other under- or unstudied models, including (a) Rindskopf’s reparameterization of the CTCM (CTCM-R) model, (b) a correlated trait–constrained uncorrelated method model and two of its more general cases, (c) a correlated trait–constrained correlated method model, and (d) a correlated trait–uncorrelated method model. Results show that (a) the CTCM-R model often solved convergence and admissibility problems with the CTCM model at rates equivalent to the CTCU model and (b) constrained models often provided convergent and admissible solutions but significantly worse model fit, indicating that they are often not plausible when analyzing real data. A follow-up simulation study showed that the CTCM-R model also provided the most accurate estimates of the full range of parameters relevant to a confirmatory factor analytic model of MTMM data.

In light of continuing controversy over alternative estimation methods to minimize convergence and admissibility (C&A) problems in the analysis of multitrait–multimethod (MTMM) data, we compared six different analytic models in terms of C&A and goodness-of-fit in a reanalysis of a large sample of previously published MTMM matrices. We fit the correlated trait–correlated method (CTCM; Widaman, 1985) and correlated trait–correlated uniqueness (CTCU; Marsh, 1989) models for reference purposes because their relative advantages and disadvantages are already well known (Lance, Nobel, & Scullen, 2002). Next, we fit a reparameterized version of the CTCM model described by Rindskopf (1983; CTCM-R) that is designed to avoid Heywood cases. We also describe three new models that are adapted from Andrews (1984) and Saris and Andrews (1991) that have been widely applied but which have not been systematically studied otherwise. One of these, a correlated trait–constrained uncorrelated method model (CTCoUM) places invariance constraints on same-method factor loadings and orthogonality constraints on method correlations. The correlated traits–uncorrelated methods (CTUM) and correlated traits–constrained method (CTCoM) models are nested within the CTCoUM model and apply these constraints individually. After a brief review of relevant literature, we describe these models in more detail, the sample of studies that provided the data to compare them, and our findings with respect to their performance in terms of model admissibility and fit to data in 258 samples from previously reported studies. Finally, we report results of a small simulation study that complements the main study’s findings and sheds light on accuracy of these alternative models’ parameter estimates.

Background

One of the main sources of controversy in the MTMM literature has concerned the optimal analytic approach for analyzing MTMM data, and a number of approaches have been proposed. Campbell and Fiske’s (1959) original approach involved subjective comparisons among monotrait–heteromethod, heterotrait–heteromethod, and heterotrait–monomethod correlations to establish inferences regarding convergent validity, discriminant validity, and the presence of method effects. Subsequently, a number of more quantitative approaches to the analysis of MTMM data were proposed, including analysis of variance (e.g., Kavanagh, MacKinney, & Wolins, 1971), path analysis (Schmitt, 1978), and various confirmatory factor analytic (CFA) models. One of the earliest examples of the latter was the CTCM model and its many special cases (Widaman, 1985), but it soon became known that the full CTCM model suffers from C&A problems due to empirical underidentification issues (Brannick & Spector, 1990; Kenny & Kashy, 1992). The CTCU model was proposed as an alternative to the CTCM model (Kenny, 1976; see also Marsh, 1989) and tends to avoid model C&A problems, though it suffers its own conceptual and statistical limitations (Lance et al., 2002). We include it here for comparative purposes because its strengths and limitations relative to the CTCM model are well known. The other models we compare are not new but their performance in terms of convergence, admissibility, and model fit to sample data have either not been studied or have been understudied. One of these is Rindskopf’s (1983) reparameterization of the CTCM model that is designed to avoid Heywood cases. The other three invoke equality (i.e., invariance) and/or orthogonality constraints on method factor correlations. As such, the purpose of the present study was to compare the performance of these latter four understudied CFA models to the performance of two CFA models for MTMM data whose performance is well known (i.e., the CTCM and CTCU models).

Assumptions

In what follows, we assume that the observed Trait-Method Units (TMU_ijs, i.e., variables representing the ith Trait as measured by the jth Method), and Trait (ξ_Ti) and Method (ξ_Mj) factors (where appropriate) are at least approximately normally distributed and mean centered (i.e., E[TMU_ij] = E[ξ_Ti]=[ξ_Mj] = 0) with unit variance and uncorrelated with model disturbances (δ_ijs) (i.e., E[TMU_ij, δ_ij] = E[ξ_Ti, δ_ij] = E[ξ_Mj, δ_ij] = 0 with E[δ_ij] = 0 and $E[{\delta }_{\mathrm{ij}}^2]={\sigma }_{\delta }^2=e^2+s^2$, where e ² and s ² refer to nonsystematic and specific variance components, respectively. As is also typical (Widaman, 1985), we assumed that E[ξ_Ti, ξ_Mj] = 0, where appropriate.

Comparison Models

CTCM Model

The CTCM model is written as

$$\mathrm{MTMM}=\left[\begin{array}{cc}\hfill {\mathbf{\Lambda }}_T\hfill & \hfill {\mathbf{\Lambda }}_M\hfill \end{array}\right]\left[\begin{array}{cc}\hfill {\mathbf{\Phi }}_{\mathrm{TT}\prime }\hfill & \hfill \mathrm{sym}\hfill \\ \multicolumn{1}{c}{0}& \hfill {\mathbf{\Phi }}_{\mathrm{MM}\prime }\hfill \end{array}\right]\left[\begin{array}{c}\hfill {\mathbf{\Lambda }}_T^{\prime }\hfill \\ \multicolumn{1}{c}{{\mathbf{\Lambda }}_M^{\prime }}\end{array}\right]+\mathbf{\Theta }$$

where MTMM is the p×p multitrait–multimethod correlation matrix where typically (but not necessarily) p = T * M (where T refers to the number of Traits and M refers to the number of Methods).¹ Λ _T (p×T) and Λ _M (p×M) contain a priori specified fixed and freely estimated factor loadings connecting the TMU_ij observed variables to their respective Trait and Method factors, Φ _TT′ (T×T) and Φ _MM′ (M×M), are both symmetric matrices that contain freely estimated correlations among the Trait and Method factors, respectively,² and Θ is a diagonal matrix containing estimated uniquenesses (i.e., residuals from the TMU_ijs’ regressions on the appropriate Trait and Method factors). The CTCM model is the most general linear model in Widaman’s (1985) taxonomy (Model 3C) and presents a number of theoretical advantages over alternative models in this taxonomy as it permits the estimation of the full range of parameters contained within the Trait and Method factor space. However, it has long been known to suffer C&A problems associated with empirical underidentification issues (Brannick & Spector, 1990; Kenny & Kashy, 1992). We included the CTCM model here primarily as a reference model against which the performance of the alternative models studied were compared.

The CTCU Model

The CTCU model was originally formulated by Kenny (1976) but was popularized by Marsh (1989) as an analytic alternative model that often avoids C&A problems associated with the CTCM model. The CTCU model may be written as

$$\mathrm{MTMM}={\mathbf{\Lambda }}_T{\mathbf{\Theta }}_{\mathrm{TT}\prime }{\mathbf{\Lambda }}_T^{\prime }+\mathbf{\Theta }$$

where again Λ _T (p×T) and Φ _TT′ (T×T) contain the Trait factor loadings and correlations, respectively. The CTCU model parameterizes Method effects not as Method factors’ effects on variables (as in the CTCM model) but as covariances among the uniquenesses. Thus, ${\sigma }_{\delta }^2=e^2+s^2+{\sigma }_{\mathrm{Mj}}^2$ and E(δ_ij,δ_i′j′) = ${\sigma }_{{\delta }_{\mathrm{ij}}{\delta }_{i\prime j\prime }}$ for all j = j′. As such, Θ is a symmetric matrix containing unique (and Method) variances along the diagonal and covariances among uniquenesses for TMU_ijs that share a common method factor that are structured either as subdiagonals (if Methods are nested within Traits in the MTMM matrix) or as triangular covariance submatrices (if Traits are nested within Methods). Although the CTCU model returns C&A CFA solutions far more frequently than the CTCM model (e.g., Marsh & Bailey, 1991), it suffers from a number of conceptual limitations (Lance et al., 2002), and it produces upwardly biased estimates of Trait factor loadings and correlations when Method factors are correlated in the population (Conway, Lievens, Scullen, & Lance, 2004). As such, we include it here too primarily as a reference model against which the performance of the alternative models studied here are compared.

CTCM-R Model

Rindskopf (1983) presented a reparameterization of the CTCM model that some researchers have used to overcome problems of Heywood cases (i.e., negative estimates of unique variances) that are often encountered using the CTCM model (e.g., Kinicki, McKee-Ryan, Schriesheim, & Carson, 2002; La Du & Tanaka, 1989; Lance, Dawson, Birklebach, & Hoffman, 2010; Nagy, Trautwein, & Lüdtke, 2010). The CTCM-R model may be written as

$$\mathrm{MTMM}=\left[\begin{array}{ccc}\hfill {\mathbf{\Lambda }}_T\hfill & \hfill {\mathbf{\Lambda }}_M\hfill & \hfill {\mathbf{\Theta }}^{1/2}\hfill \end{array}\right]\left[\begin{array}{ccc}\hfill {\mathbf{\Phi }}_{\mathrm{TT}\prime }\hfill & \hfill \hfill & \hfill \mathrm{sym}\hfill \\ \multicolumn{1}{c}{0}& \hfill {\mathbf{\Phi }}_{\mathrm{MM}\prime }\hfill & \hfill \hfill \\ \multicolumn{1}{c}{0}& \hfill 0\hfill & \hfill \mathbf{I}\hfill \end{array}\right]\left[\begin{array}{c}\hfill {\mathbf{\Lambda }}_T^{\prime }\hfill \\ \multicolumn{1}{c}{{\mathbf{\Lambda }}_M^{\prime }}\\ \multicolumn{1}{c}{{\mathbf{\Theta }}^{1/2}}\end{array}\right]$$

where Λ _T (p×T) and Λ _M (p×M) contain the Trait and Method factor loadings, respectively, Θ ^1/2 (p×p) is a diagonal matrix containing the square roots of the TMU_ijs’ uniquenesses and I (p×p) is the identity matrix. This parameterization avoids Heywood cases that are encountered frequently in the estimation of the full CTCM model (Brannick & Spector, 1990) because uniquenesses are calculated as the squared elements in Θ ^1/2 (i.e., regardless of whether individual elements in Θ ^1/2 are positive or negative, their squares are positive by definition), while their orthogonality is maintained by fixing their correlation matrix to I. Inadmissible solutions can still be obtained, however, because estimated Trait and/or Method correlations can exceed |1.0|.

To our knowledge, the performance of the CTCM-R model has been studied only once before. Dillon, Kumar, and Mulani (1987) compared (a) the CTCM-R model, (b) an alternative parameterization presented by Bentler (1976), and (c) simply fixing offending estimates (i.e., Heywood cases) to zero in simple CFA models. Based on their findings, Dillon et al. (1987) had little to recommend for the first two approaches, saying “though these two approaches represent theoretically elegant ways of handling this problem, they have limited practical usefulness compared with the simpler approach in which the offending parameter estimate is fixed at zero” (p. 134). We disagree with respect to the Rindskopf parameterization (Lance, Fan, Siminovsky, Morgan, & Shaikh, 2014). Whereas either approach may contend with Heywood cases, fixing offending estimates to zero is an ad hoc symptom-based approach, while the Rindskopf reparameterization represents a more general and preemptive model-based approach. As such, we include the CTCM-R model here as it has not been studied in the analysis of MTMM data and in theory is guaranteed to solve one of the two inadmissibility problems associated with the analysis of such data.

The CTCoUM Model and Its More General Cases

The CTCoUM model that we propose here may be written as

$$\mathrm{MTMM}=\left[\begin{array}{ccc}\hfill {\mathbf{\Lambda }}_{\mathbf{T}}\hfill & \hfill {\mathbf{\Lambda }}_{\mathbf{M}}^{\mathbf{c}}\hfill & \hfill {\mathbf{\Theta }}^{1\mathbf{/}2}\hfill \end{array}\right]\left[\begin{array}{ccc}\hfill {\mathbf{\Phi }}_{\mathbf{TT}\mathbf{\text{'}}}\hfill & \hfill \hfill & \hfill \mathrm{sym}\hfill \\ \multicolumn{1}{c}{0}& \hfill I\hfill & \hfill \hfill \\ \multicolumn{1}{c}{0}& \hfill 0\hfill & \hfill I\hfill \end{array}\right]\left[\begin{array}{c}\hfill {\mathbf{\Lambda }}_{\mathbf{T}}^{\mathbf{\text{'}}}\hfill \\ \multicolumn{1}{c}{{\mathbf{\Lambda }}_{\mathbf{M}}^{\mathbf{c}\mathbf{\text{'}}}}\\ \multicolumn{1}{c}{{\mathbf{\Theta }}^{1\mathbf{/}2}}\end{array}\right]$$

Note that the general structure of this model is the same as that of the CTCM-R model but with two important exceptions: (a) factor loading estimates are constrained equal to one another within columns of ${\Lambda }_M^c$and (b) method factors are constrained to be orthogonal. This model is essentially a Rindskopf (1983) parameterization of the model proposed by Andrews (1984). According to Andrews, Method effects can be seen as arising from individual differences in interpretations of response options presented within a given method (e.g., individual differences in leniency/severity, differential perceptions of the width of response intervals, extremity or central tendency bias, etc.). As Corten et al. (2002) put it, “[I]ndividuals may differ in the way they use a certain response scale, and it may be expected that the use of such a scale by a single individual is more or less constant across traits” (p. 214; see also Kogovšek, Ferligoj, Coenders, & Saris, 2002; Saris & Aalberts, 2003). As such, and since the measurement method per se remains invariant across the Traits assessed using that method (e.g., a 5-point Likert-type scale remains the same across items representing different constructs), it may be a reasonable assumption and is a testable hypothesis that invariance constraints be imposed on factor loadings within Methods across Traits in MTMM data, by imposing equality constraints within columns of the ${\Lambda }_M^c$ matrix in Equation 4. The specification of orthogonal Methods is motivated by the assumption that researchers will use maximally different (unrelated) methods as called for in Campbell and Fiske’s (1959) original exposition of the MTMM matrix.

Of course, this model is nested within two other models that are more liberal with respect to these constraints but which themselves are also nested within the CTCM-R model: a correlated trait–constrained methods (CTCoM) model and a correlated trait-uncorrelated methods (CTUM) model.

CTCoM Model

In the CTCoM model, Method factor loadings are constrained to be equal to one another within columns of the ${\Lambda }_M^c$ matrix, but the Method factors are themselves allowed to be oblique, rather than being constrained to be orthogonal as in the CTCoUM model. Thus, this model proposes that while Method effects may be homogeneous within Methods, the Methods themselves may be correlated and recent meta-analytic evidence presented by Lance et al. (2010) suggests that correlated methods may be the rule rather than the exception. However, the Lance et al. findings are based on reanalyses of a small number of MTMM studies (k = 18) and to our knowledge this model has not been studied.

CTUM Model

The CTUM model corresponds to a Rindskopf (1983) parameterization of Model 3B′ in Widaman’s (1985) taxonomy, relaxes the Method factor loading equality constraints specified by the CTCoUM model, and provides an omnibus test (as compared with the CTCM-R model) of whether Methods are oblique or orthogonal. Using simulated data, Woehr and Hoffman (2004) ³ showed that the CTUM model resulted in convergent and admissible solutions far more often than did the CTCM model. However, we know of no studies that have compared the CTUM model with other alternative models using real (i.e., not simulated) data. As such, little is known regarding its relevance for real MTMM data.

Study Purpose

In summary, the purpose of the current study was to effect comparisons between two analytic models for MTMM data whose performance in terms of C&A problems are well known (the CTCM and CTCU models) to a set of alternative models, some old, some new, whose performance has not been studied. We chose to conduct the main portion of this study using real, unsimulated, previously published MTMM data because we wished to test the viability of the models we investigated using data that were representative of MTMM data as they actually exist, toward strong external and ecological validity of our findings. However, we do present supplementary simulated data to document the relative accuracy of model parameter estimates provided by each model. Also, we chose to not study some other alternative models because, for example, (a) the CTC(M-1) model (Eid, 2000) is not directly comparable with the models investigated here; (b) the direct-product model (Campbell & O’Connell, 1967) specifies multiplicative Trait × Method relationships, whereas we limited our attention to linear models; and (c) multiple indicator MTMM models (Marsh & Hocevar, 1988; Thomás, Hontangas, & Oliver, 2000) require multiple observed indicators for each T_i×M_j combination and no published data of this sort were available. Table 1 presents a summary of the models we studied here.

Table 1.

Summary of Models Tested.

Model	Trait factor loadings	Trait factor correlations	Method factor loadings	Method factor correlations	Uniquenesses
CTCM	Freely estimated	Freely estimated	Freely estimated	Freely estimated	Freely estimated
CTCU	Freely estimated	Freely estimated	Estimated indirectly as correlated unique-nesses	Methods assumed orthogonal	Confound unique and method variances
CTCM-R	Freely estimated	Freely estimated	Freely estimated	Freely estimated	Estimated as squared unique factor loadings
CTCoUM	Freely estimated	Freely estimated	Constrained to be equal within columns	Constrained to be orthogonal	Estimated as squared unique factor loadings
CTCoM	Freely estimated	Freely estimated	Constrained to be equal within columns	Freely estimated	Estimated as squared unique factor loadings
CTUM	Freely estimated	Freely estimated	Freely estimated	Constrained to be orthogonal	Estimated as squared unique factor loadings

Note. CTCM = correlated trait–correlated method; CTCU = correlated trait–correlated uniqueness; CTCoUM = correlated trait–constrained uncorrelated method model; CTCoM = correlated traits–constrained method; CTUM = correlated traits–uncorrelated methods.

Method

Literature Review

We took several steps to identify published MTMM matrices for reanalysis. First, we conducted a citation search of Campbell and Fiske’s (1959) article to identify studies that had reported an original MTMM matrix. Second, we examined reference lists of previous reviews (e.g., Bowler & Woehr, 2006; Crampton & Wagner, 1994; Lance et al., 2010; Turner, 1981; Williams, Cote, & Buckley, 1989) to identify relevant studies. Third, we contacted authors of studies that did not report the original MTMM matrices to determine their availability.⁴ In all, we identified 318 studies that reported one or more MTMM matrices. The data we describe here are from 258 MTMM matrices reported in 187 studies⁵ (a) that reported at least a three trait–three method (3T3M), 2T4M, or 4T2M matrix,⁶ (b) were positive definite, and (c) whose data resulted in a convergent and admissible solution for either the CTCM, CTCU, or CTCM-R models, as these represented the three baseline models against whose performance the alternative models were compared.

Analyses

We used LISREL-VIII (Jöreskog & Sörbom, 1993) to fit each of the models described earlier and noted whether the model converged to a solution within 1,000 iterations and whether the solution was admissible. Inadmissible solutions contained factor correlations or loadings >|1.00| and/or negative uniquenesses. We recorded in an Excel document, for each matrix and each model tested, (a) sample size; (b) the number of Traits and Methods included in the MTMM matrix; (c) where possible, the Trait and Method type classified according to the taxonomy reported by Lance et al. (2014); (d) whether the model was nonconvergent, inadmissible, or admissible; (e) df and χ²; and (f) the standardized root mean squared residual (SRMSR), the root mean squared error of approximation (RMSEA; Browne & Cudeck, 1993), the Tucker–Lewis index (TLI; Tucker & Lewis, 1973), and Bentler’s (1990) comparative fit index (CFI) as these are commonly reported and recommended overall goodness-of-fit indices (Hu & Bentler, 1998; McDonald & Ho, 2002).

Results

Table 2 summarizes overall study characteristics. The typical MTMM study in our sample had a “modest”N (Mdn = 183, though N ranged considerably from 17 to nearly 40,000), included 3 to 7 Traits (Mdn = 4) and 2 to 4 Methods (Mdn = 3). The MTMM matrices reanalyzed in this study represented wide varieties of (a) Traits, including job performance (k = 34), personality (k = 52), attitudes (k = 40), job or life satisfaction facets (k = 19), assessment center (AC) dimensions (k = 14), job/organizational characteristics (k = 12), academic performance (k = 17), children’s (problem) behavior (k = 25), and core self-evaluations (k = 31), and (b) Methods, including AC exercises (k = 14), raters/rater sources (k = 83), occasions (k = 51), rater source plus objective assessment (k = 19), scale format (k = 51), and alternative test forms (k = 55).⁷

Table 2.

Overall Study Characteristics.

	Median	Range	10th percentile	90th percentile
Sample size	183	17 to 39,923	69	621
Number of traits	4	2 to 13	3	7
Number of methods	3	2 to 6	2	4

Table 3 shows results pertaining to C&A. As expected, the CTCM model displayed significantly lower C&A rates compared with the CTCU model (24% vs. 79%; χ²[1] = 156.39, p < .01) and these rates are consistent with previous research (e.g., Lance, Woehr, & Meade, 2007; Marsh & Bailey, 1991). Note that in these two models only, nonconvergence/inadmissibility issues can stem either from out-of-range factor correlations/loadings or negative uniquenesses. Table 4 shows that the CTCM model returned inadmissible solutions due to estimated factor correlations >|1.00| or Heywood cases or both but that when the CTCU model was inadmissible it was almost always due to the fact that both problems were present.

Table 3.

Model Convergence and Admissibility.

Model	Number/% convergent and admissible	Number/% nonconvergent	Number/% inadmissible
CTCM	61/24%	114/44%	83/32%
CTCU	203/79%	12/5%	43/17%
CTCM-R	217/84%	11/4%	30/12%
CTUM	242/94%	1/<1%	15/6%
CTCoM	107/42%	6/2%	145/56%
CTCoUM	219/85%	0/0%	39/15%

Note. CTCM = correlated trait–correlated method; CTCU = correlated trait–correlated uniqueness; CTCoUM = correlated trait–constrained uncorrelated method model; CTCoM = correlated traits–constrained method; CTUM = correlated traits–uncorrelated methods.

Table 4.

Reasons for CTCM and CTCU Model Inadmissibility.

Model	Number/% factor correlations >|1.0|	Number/% negative uniqueness	Number/% both	Total number of inadmissible solutions
CTCM	24/29%	17/20%	42/51%	83
CTCU	4/9%	0	39/91%	43

Note. CTCM = correlated trait–correlated method; CTCU = correlated trait–correlated uniqueness.

Table 3 shows that the CTCM-R model resulted in an admissible solution far more often than did the CTCM model (84% vs. 24%, respectively; χ²[1] = 189.79, p < .01). This is despite the fact that the CTCM and CTCM-R models are mathematically identical—the correlations between dfs and χ²s for cases in which both models resulted in C&A solutions were 1.00, confirming their identity. The CTCM-R model even surpassed the CTCU model in producing admissible solutions (84% vs. 79%, respectively). The CTUM model resulted in the highest rates of admissible solutions (94%), even surpassing its conceptual cousin the CTCU model. Recall that the CTCoM model imposes equality constraints within columns of Λ _M in the CTCM-R model. The relatively small proportion of admissible solutions for this model (42%) suggests that these restrictions are often implausible. Every inadmissible and nonconvergent model CTCoM contained Method correlations that were estimated >|1.00|, suggesting that these out-of-bounds estimates of Method correlations were symptoms of misspecified equality constraints among Method factor loadings. Perhaps ironically, the CTCoUM model that combines the invariance restrictions within Methods and constrains the Method correlations to orthogonality resulted in one of the highest rates of model admissibility of all the models tested (85%). This was because the offending Method correlations >|1.00| in the CTCoM model were fixed = 0 in the CTCoUM model, thus solving (or rather, obscuring) the C&A problem in the CTCoM model but worsening model fit (see below).

It has been suggested that the CTCU model be invoked only as a “last-ditch” option only in the event that other, more theoretically plausible, models fail to return an admissible solution (Lance et al., 2002). Restricting attention to the 41 samples in which the CTCU model (but not the CTCM or CTCM-R models) returned an admissible solution, we found that the CTCoM, and especially the CTUM, and CTCoUM models often returned admissible solutions (see Table 5). Thus, one or more of these constrained models, which are in some cases perhaps more theoretically defensible than the CTCU model, may still be viable analytic alternatives to the CTCU model even when some parameterization of the CTCM (including even the CTCM-R model) model fails. These might include cases in which Method effects are suspected to be present but may be assumed (or shown empirically) to be independent of one another (the CTUM model), where Method effects may be assumed (or shown empirically) to be homogeneous across Traits (the CTCoM model), or both (the CTCoUM model).

Table 5.

Restricted Models’ C&A Rates for Studies in Which the CTCU (But Not the CTCM and CTCM-R) Model Returned Admissible Solutions.

Model	Number/% convergent and admissible	Number/% nonconvergent	Number/% inadmissible
CTUM	29 (71%)	1 (2%)	11 (27%)
CTCoM	10 (24%)	2 (5%)	29 (71%)
CTCoUM	27 (66%)	0 (0%)	14 (34%)

Note. C&A = convergence and admissibility; CTCM = correlated trait–correlated method; CTCU = correlated trait–correlated uniqueness.

Table 6 summarizes overall model fit. Note first that even though the CTCM and CTCM-R models are identical mathematically the median df and 20% trimmed mean χ² are both larger for the CTCM model (t[276] = 6.17; p < .01, for mean Winsorized χ² statistics). This is because “smaller” MTMM matrices are more prone to empirical underidentification problems that can cause inadmissible solutions as compared with “larger” models that afford greater nominal df. Perhaps more interesting are comparisons between the CTCM-R model and its special cases. At the aggregate level the CTCM-R fit significantly better than all three of them (t[457] = 3.83, p < .01 vs. CTUM; t[322] = 7.91, p < .01, vs. CTCoM; t[434] = 3.09, p < .01, vs. CTCoUM) indicating that orthogonal Methods or invariance constraints within Methods, or both, were often implausible model restrictions. At the individual study level, there were 87 cases in which all four models reached admissible solutions. In 36 (41%) of these cases, the CTCM-R model fit significantly better (in terms of Δχ² tests) than all three alternative models, indicating that neither the invariance constraints on Method factor loadings nor the hypothesis of orthogonal Methods was statistically plausible. In 30 (34%) cases, the fit of the CTCM-R model did not differ from that for the CTUM model but was significantly better than that for the CTCoM and CTCoUM models, indicating that the hypothesis of uncorrelated Method factors was plausible but the invariance constraints on Method factor loadings were not. In the remaining 21 (24%) cases, the Δχ² tests indicated that both sets of restrictions were plausible. Finally, results for the overall goodness-of-fit indices shown in the right hand portion of Table 5 show that they were of little use in distinguishing among models except perhaps for the CTCoUM model, which, on the average, failed to meet currently accepted cutoff values for RMSEA (≤.06) and TLI (≥.95; Hu & Bentler, 1998).

Table 6.

Model Goodness of Fit for Convergent and Admissible Solutions.

Model	k	df	χ2	% p < .01	SRMSR	RMSEA	TLI	CFI
CTCM	61	33	89.81	54%	.041	.048	.98	.99
MIN		5	1.71		0	0	.77	.89
MAX		662	11458.77		.15	.17	1.02	1.00
SD		100.84	1484.80		.03	.04	.05	.02
CTCU	203	15	42.51	35%	.043	.063	.98	.99
MIN		2	0.19		0	0	.68	.76
MAX		572	3294.38		.19	.46	1.86	1.00
SD		57.89	340.58		.03	.07	.26	.04
CTCM-R	217	14	46.99	44%	.044	.048	.98	.99
MIN		5	1.33		0	0	.63	.79
MAX		662	11458.77		.18	.19	1.99	1.00
SD		64.67	816.84		.03	.05	.11	.03
CTUM	242	15	63.43	48%	.055	.061	.97	.99
MIN		6	0.97		.01	0	.53	.76
MAX		672	11982.62		.18	.25	1.88	1.00
SD		67.41	829.32		.03	.05	.13	.04
CTCoM	107	36	95.23	63%	.058	.069	.96	.98
MIN		9	3.80		.02	0	.28	.49
MAX		697	15538.15		.33	.22	1.86	1.00
SD		83.21	1632.40		.09	.05	.18	.09
CTCoUM	219	23	123.15	70%	.064	.082	.93	.96
MIN		10	4.72		.01	0	.18	.48
MAX		707	20033.65		.31	.27	1.86	1.00
SD		74.20	1462.66		.04	.06	.18	.09

Note. CTCM = correlated trait–correlated method; CTCU = correlated trait–correlated uniqueness; SRMSR = standardized root mean squared residual; RMSEA = root mean square error of approximation; TLI = Tucker–Lewis index; CFI = comparative fit index; MIN = minimum observed value; MAX = maximum observed value; SD = standard deviation. First row of entries for each model are k = number of C&A studies; df = median df; χ² = 20% trimmed mean χ² (Wilcox & Keselman, 2003); % p < .01 = percentage of studies whose probability values for estimated χ² statistics were <.01. SRMSR, RMSEA, TLI, and CFI are median standardized root mean squared residuals, respectively.

Simulation Study

One of the limitations of the main portion of this study was that in the analysis of real data one cannot judge the accuracy of parameter estimates, because population values are never known. Consequently, we undertook a small simulation study that would allow us to estimate bias in parameter estimates incurred in the MTMM models studied here. We generated multivariate normal data using R (R Core Development Team, 2012) with the CTCM model as the generating model and manipulated one factor thought to be related to model C&A rate—model size. As such, the two population models were a 3T3M model and a 5T5M MTMM model. These represent minimum and “large” MTMM matrices according to Lance et al.’s (2014) review. Model parameters’ values were varied about the median values presented in Lance et al.’s review so as to represent “typical” values found in the MTMM literature (see Table 7). We selected a sample size of N = 200 (just above the Mdn of 183 reported in Table 2 earlier) with 1,000 replications.

Table 7.

Population Values for Simulation Study.

Population parameters	Mdn from Lance et al. (2014)	Population values
Method loading	.28	.18, .28, .38
Trait loading	.50	.40, .50, .60
Method correlation	.29	.19, .29, .39
Trait correlation	.36	.26, .36, .46

Note. Parameter values were selected equally often in population models so that the mean value equaled Lance et al.’s median value.

Results

Table 8 shows that (as expected) model size had a large effect on the number of C&A solutions achieved by the CTCM model because larger models afford more nominal df. All models that specified orthogonal Method factors achieved very favorable C&A rates (however, see below) as did the CTCM-R model. In general terms, C&A results for the simulated data were consistent with those for previously published data shown in Table 3.

Table 8.

Model Convergence and Admissibility.

	3T3M		5T5M
Model	Number convergent	Number convergent and admissible	Number convergent	Number/Convergent and admissible
CTCM	528	79	574	252
CTCU	995	956	1,000	1,000
CTCM-R	952	643	985	731
CTUM	957	949	979	979
CTCoM	936	307	980	471
CTCoUM	997	986	1,000	1,000

Note. CTCM = correlated trait–correlated method; CTCU = correlated trait–correlated uniqueness; CTCoUM = correlated trait–constrained uncorrelated method model; CTCoM = correlated traits–constrained method; CTUM = correlated traits–uncorrelated methods.

Table 9 shows summary goodness-of-fit data for models that reached C&A solutions. Just as for the real data reported in Table 6, the models tested were largely indistinguishable from one another in terms of popularly reported goodness-of-fit indexes relative to their recommended cutoff criteria (Hu & Bentler, 1998). However, the relative magnitudes of the models' noncentrality parameter estimates clearly indicate which models were not the generating models for both the 3T3M and 5T5M data.

Table 9.

Model Goodness of Fit Means for Convergent and Admissible Solutions.

Model	k	df	χ2	NCP	RMSEA	TLI	CFI
3T3M
CTCM	79	12	6.48	.06	.001	1.07	1.00
CTCU	949	15	42.51	1.93	.063	.98	.99
CTCM-R	643	12	8.46	.31	.003	1.05	1.00
CTUM	949	15	15.49	2.28	.018	.99	.99
CTCoM	307	18	17.74	1.91	.013	1.00	.99
CTCoUM	986	21	22.01	2.84	.017	.99	.99
5T5M
CTCM	252	230	228.79	3.03	.004	1.00	.99
CTCU	1,000	215	230.85	10.20	.011	.99	.99
CTCM-R	731	230	230.75	3.43	.004	1.00	.99
CTUM	979	240	252.96	8.55	.009	.99	.99
CTCoM	471	250	269.86	12.14	.011	.98	.99
CTCoUM	1,000	260	284.02	15.25	.013	.98	.98

Note. CTCM = correlated trait–correlated method; CTCU = correlated trait–correlated uniqueness; CTCoUM = correlated trait–constrained uncorrelated method model; CTCoM = correlated traits–constrained method; CTUM = correlated traits–uncorrelated methods; NCP = noncentrality parameter; RMSEA = root mean square error of approximation; TLI = Tucker–Lewis index; CFI = comparative fit index; k = number of C&A studies; df = median df; χ² = median χ².

Finally, Table 10 shows estimated bias for parameter estimates. Given that the models’ population values were deviated about the medians by ±.10 (see Table 7), we considered bias ≥.10 as representing large estimation bias. Table 10 shows that the CTCM model produced “homogenized” estimates (i.e., underestimated Trait factor loadings and correlations and overestimated Method factor loadings and correlations) in the 3T3M data but that these effects diminished somewhat in the 5T5M data, this despite the fact that the CTCM model was the generating model. Recall, however, from Table 8 that these results are from relatively few samples. The CTCU model produced slightly elevated estimates of Trait factor loadings and correlations as would be expected due to the moderate-to-low population Method factor loadings and correlations (see Conway et al., 2004) but by definition produced highly downwardly biased estimates of Method factor loadings and correlations by virtue of the fact that they are not estimated (i.e., assumed = 0) in this model. The CTCM-R model exhibited a “homogenizing” effect on model parameter estimates similar to that of the CTCM model but to a much lesser degree—CTCM-R model estimates were among the most accurate of any shown in Table 10. While the CTUM and CTCoUM models provided reasonably accurate estimates of Trait factor loadings and correlations they, like the CTCU model, provided severely biased estimates for Method correlations by virtue of their nonestimation. Finally, the CTCoM model produced significant underestimates (overestimates) for Trait (Method) factor loadings in both the 3T3M and 5T5M data sets. In general, the CTCM-R model provided the least biased model estimates for the complete set of parameters that are relevant to a CFA model representation of MTMM data.

Table 10.

Mean Bias and Standard Deviation for Parameter Estimates.

	Trait loadings	Trait correlations	Method loadings	Method correlations
3T3M
CTCM
M	−.042	−.115	.056	.160
SD	.182	.277	.224	.327
CTCU
M	.023	.058	−.280	−.290
SD	.102	.130	0	0
CTCM-R
M	−.028	−.100	.070	.090
SD	.197	.237	.269	.292
CTUM
M	.023	.053	.007	−.290
SD	.104	.128	.285	0
CTCoM
M	−.041	−.169	.171	.198
SD	.189	.226	.136	.381
CTCoUM
M	.023	.056	−.068	−.290
SD	.103	.126	.107	0
5T5M
CTCM
M	−.026	−.061	.033	.124
SD	.121	.165	.166	.268
CTCU
M	.020	.040	−.280	−.290
SD	.071	.096	0	0
CTCM-R
M	−.014	−.025	.015	.047
SD	.124	.151	.195	.268
CTUM
M	.023	.052	−.035	−.290
SD	.075	.097	.197	0
CTCoM
M	−.048	−.151	.128	−.004
SD	.117	.193	.320	.503
CTCoUM
M	.021	.052	.004	−.290
SD	.070	.099	.056	0

Note. M = mean; SD = standard deviation; CTCM = correlated trait–correlated method; CTCU = correlated trait–correlated uniqueness; CTCoUM = correlated trait–constrained uncorrelated method model; CTCoM = correlated traits–constrained method; CTUM = correlated traits–uncorrelated methods.

Recommendations, Discussion, and Conclusions

Results from this study support the following conclusions and recommendations:

1. Among the models studied here, CTCM-R model is often the optimal model as it (a) maintains all the theoretical advantages of its mathematical equivalent CTCM model (Lance et al., 2002) while largely avoiding C&A problems associated with it, (b) generally returns C&A solutions at least as often as any of the other models studied here, (c) generally provides superior model fit as compared to its special cases studied here, and (d) returns very accurate model parameter estimates relative to simulated population values.

2. The CTUM model is a viable and attractive model when the fuller CTCM-R model results are inadmissible and when orthogonal Methods is a plausible hypothesis. However, results presented here (and the fact that the average Method correlation in the sample reported here was .29) indicate that this hypothesis is not routinely plausible and should be tested, when possible, before implementing the CTUM model.

3. Imposition of invariance constraints within Method factors in the CTCM-R model often results in improper solutions caused by estimated Method factor correlations >|1.00| in the CTCoM model. Improper estimates for Method factor correlations are apparently one symptom of misspecified invariance constraints and can be used, in conjunction with statistical tests (i.e., Δχ² tests), to ascertain the plausibility of these constraints. Fixing these offending estimates to zero in the CTCoUM model often yields an admissible solution (because it obscures out-of-bounds values estimated for Method factor correlations) but at the expense of worse model fit (thereby providing additional evidence that this model is inconsistent with the data).

4. The constrained models presented here may serve as attractive alternatives to the CTCU model (which often produces upwardly biased estimates of Trait factor loadings and correlations and severely downwardly biased estimates of Method factor loadings and correlations) when the full CTCM-R model does not hold, and especially if the imposed constraints are plausible theoretically and statistically.

Study Limitations and Directions for Future Research

These conclusions and recommendations are based on the largest sample of MTMM studies ever reported, sampling wide varieties of Traits and Methods and supplemented with simulated data. As such, we view our findings as being ecologically valid and generalizable. Still, there are likely other MTMM studies that would have met our inclusion criteria that our literature search efforts did not detect. As such, we limit our conclusions and recommendations with the realization that the studies reported here represent a sample (albeit a very large one) and not the population of MTMM studies reported in published and unpublished literature.

Second, it could be argued that the CTCM-R model is no longer relevant as Heywood cases can be resolved using inequality constraint options that are available in many currently popular SEM software packages and this is, to some extent, a reasonable claim. Some examples of implementations of inequality constraints are given in the appendix. However, and as we alluded to earlier, most of these are post hoc symptom-based fixes that may disguise other clues to sources of model misfit, whereas the Rindskopf reparameterization of the CTCM model is a preemptive, model-based approach to avoiding Heywood cases.

Finally, we made informed and reasoned decisions as to which MTMM models we would study here and, as a result, eliminated some models from study. Many of the models omitted from our study were simply not theoretically motivated (e.g., many of the other submodels in the taxonomy of Widaman, 1985), were not comparable on common grounds (e.g., the CTC(M-1) and direct product models) or presented insurmountable data challenges (i.e., multiple indicator models). As such, our conclusions and recommendations are confined to the models studied here; future research may extend our work to a broader sample of analytic models for MTMM data.

Conclusion

In conclusion, we recommend the CTCM-R model for the analysis of MTMM data as it (a) maintains the theoretical advantages of its mathematically identical CTCM model but without the C&A problems that the CTCM model often encounters; (b) returns C&A model solutions as often as the CTCU model, which is known to provide upwardly biased estimates for trait factor loadings and correlations when Method correlations are nonzero in the population; and (c) produces estimates for the complete array of parameters relevant to a CFA parameterization of MTMM data relative to the other models studied here. Invariance restrictions on Method factor loadings and orthogonality constraints on Method correlations may also be plausible when they are justified theoretically and empirically.

Appendix

Implementation of Inequality Constraints in Popular SEM Software Packages

Mplus (Muthén & Muthén, 1998-2012) users may include the following syntax:

• MODEL:

• F1@1;

• F2@1;

• MODEL CONSTRAINT;

• F1 WITH F2 < 1.0;

• D1 > 0;

to constrain the correlation between factors F1 and F2 to be less than unity and the uniqueness (D1) to be positive.

Similarly, LISREL (Jöreskog & Sörbom, 1993) users can use interval restrictions syntax of the form:

• IR TD(1,1) >0

• IR PH(2,1) >-1 <1

to constrain a uniqueness to be positive and a factor correlation to be < |1.0|. SAS PROC CALIS users can use the BOUNDS statement to effect constraints such as these. However, Jöreskog and Sörbom (1993) recommend that users “run the problem without the interval restrictions first and then apply only those interval restrictions which are needed. Chi-square and standard errors will be affected if parameter estimates are on the boundary of the interval” (p. 76). To our knowledge, the CTCM-R reparameterization of the mathematically identical CTCM model does not suffer from these limitations.

Third, and as noted in the EQS program manual (Bentler, 2006), “The program automatically constrains variance estimates to be nonnegative, and correlations between variables having fixed variances as lying between −1 and +1” (p. 81). This is a very convenient feature, but we urge caution in invoking it as it may mask important clues (i.e., improper solutions) that the model being fit is inconsistent with the data as was the case with respect to the CTCoM model in this study.