Eliminating Bias in Classify-Analyze Approaches for Latent Class Analysis

Abstract

Despite recent methodological advances in latent class analysis (LCA) and a rapid increase in its application in behavioral research, complex research questions that include latent class variables often must be addressed by classifying individuals into latent classes and treating class membership as known in a subsequent analysis. Traditional approaches to classifying individuals based on posterior probabilities are known to produce attenuated estimates in the analytic model. We propose the use of a more inclusive LCA to generate posterior probabilities; this LCA includes additional variables present in the analytic model. A motivating empirical demonstration is presented, followed by a simulation study to assess the performance of the proposed strategy. Results show that with sufficient measurement quality or sample size, the proposed strategy reduces or eliminates bias.

Recent methodological advances in latent class analysis (LCA) have resulted in a rapid increase in its application in behavioral and psychological research. The latent class model, which posits a mutually exclusive and exhaustive underlying set of latent classes (i.e., subgroups) inferred from multiple categorical observed variables, has been described in detail in a variety of resources (e.g., Clogg, 1995; Collins & Lanza, 2010). LCA has proven to be a useful tool for identifying qualitatively different population subgroups in a variety of disciplines (e.g., substance use: Beseler, Taylor, Kraemer, & Leeman, 2012; delinquency: Barnes, Boutwell, Morris, & Armstrong, 2012; sexual behavior: Haydon, Herring, & Halpern, 2012; physical activity: McDonald et al., 2012).

As application of LCA spreads, more complex scientific questions are being posed about the role latent class membership plays in development. Addressing these questions often requires estimating associations between the latent class variable and other observed variables such as predictors, outcomes, moderators, and mediators. In some cases, these associations can be modeled in the context of the latent class model itself. For example, LCA with covariates (e.g., Collins & Lanza, 2010; Dayton & Macready, 1988) allows multiple predictors of latent class membership to be incorporated directly into the latent class modeling framework; this type of approach is sometimes referred to as a 1-step approach (e.g., Vermunt, 2010).

More recent methodological work has focused on predicting a distal outcome from latent class membership. A model-based approach using kernel density estimation has been proposed (Lanza, Tan, & Bray, 2013), but currently it cannot handle more complex models such as those that include additional covariates predicting the distal outcome. Advances like these are important because, for example, the significance of latent classes is often conveyed by examining the consequences of latent class membership on later developmental outcomes (e.g., Hardigan & Sangasubana, 2010; Nylund, Bellmore, Nishina, & Graham, 2007; Petras & Masyn, 2010; Reinke, Herman, Petras, & Ialongo, 2008; Roberts & Ward, 2011).

Even more complex questions arise when theory posits latent class membership acting as a moderator of important associations, including treatment effects, or as a mediator in a model linking an individual’s earlier experiences to later outcomes. Despite the recent methodological advances discussed above, many of these more complex research questions cannot currently be addressed within the context of the latent class model itself. In many studies, researchers have no alternative to taking a classify-analyze approach for LCA.

Classify-analyze (Clogg, 1995) is a common technique used to circumvent the challenge of not having an LCA model-based solution. This technique involves first classifying individuals into latent classes, and then performing a subsequent analysis treating latent class membership as known (i.e., as a manifest categorical variable) in a larger model of interest. Fundamentally, LCA is a probability-based approach that does not require assignment of individuals to latent classes; this is one of its greatest strengths because it provides a way to account for measurement error in responses to manifest indicators and to remove this measurement error from other estimates of interest. Using classify-analyze to estimate associations between the latent class variable and other manifest variables is straightforward, but also problematic because it often leads to attenuation of these estimates (Bolck, Croon, & Hagenaars, 2004; Vermunt, 2010). Despite this known limitation, recent research has used classify-analyze to relate latent class membership to predictors (e.g., Sutfin, Reboussin, McCoy, & Wolfson, 2009) and outcomes (e.g., Lacourse et al., 2010), as well as to treat latent class membership as a moderator (e.g., Herman, Ostrander, Walkup, Silva, & March, 2007) and a mediator (e.g., Oshri, Rogosch, & Cicchetti, 2012). The primary goal of this study is to demonstrate an improved classify-analyze technique that can reduce, and even eliminate, bias in estimates of the association between a latent class variable and other variables of interest.

The classification of individuals to latent classes is a form of imputation; because an individual’s true class membership cannot be known, the latent class variable is 100% missing. Each individual has a probability of membership in each latent class; these probabilities, which are derived from the latent class measurement model, are known as posterior probabilities. Once the parameters of a latent class model have been estimated, posterior probabilities of membership in each latent class can be obtained for each individual using Bayes’ theorem (e.g., Gelman, Carlin, Stern, & Rubin, 2003; Lanza, Collins, Lemmon, & Schafer, 2007). The posterior probability of membership in a given latent class can be expressed

$$P(C=c\mid Y=y)=\frac{P(C=c)P(Y=y\mid C=c)}{P(Y=y)},$$ (1)

where C represents the latent class variable with c = 1, …, K latent classes, and Y represents an individual’s vector of responses, y, to a set of observed variables. Typically, methods for classifying individuals involve imputing latent class membership for each individual based on his or her own set of posterior probabilities.

Two classify-analyze approaches for LCA are commonly used in practice. Both approaches rely on posterior probabilities to classify individuals. We refer to the first approach as the maximum-probability assignment rule (Nagin, 2005, pg. 80), which assigns individuals to the class for which they have the highest posterior probability of membership (Goodman, 1974, 2007). The subsequent analysis is performed once with latent class membership treated as known. Although this simple method does not take into account uncertainty in class assignment (Clogg, 1995), it minimizes the number of incorrect assignments compared to other approaches (Goodman, 2007).

The second approach is multiple pseudo-class draws (Bandeen-Roche, Miglioretti, Zeger, & Rathouz, 1997). This approach is similar to maximum-probability assignment but accounts for uncertainty in class assignment. Using this method, individuals are randomly classified into latent classes multiple times based on their distributions of posterior probabilities. Often, 20 pseudo-class draws are used; that is, individuals are classified 20 times (Wang, Brown, & Bandeen-Roche, 2005). The subsequent analysis is performed once for each draw (i.e., 20 times) and results are combined across draws using rules derived for multiple imputation for missing data (Rubin, 1987). Importantly, this technique was originally developed as a diagnostic tool to assess model adequacy (Bandeen-Roche et al., 1997; Wang et al., 2005). Researchers have since adopted this technique for use in a very different context, that of multiply imputing latent class membership for use in a subsequent analysis.

For both maximum-probability assignment and multiple pseudo-class draws, the posterior probabilities upon which classification depends are calculated from an LCA with a specified number of latent classes. A typical approach to classification, for example in the case of predicting a distal outcome from a latent class variable, is to (1) determine the optimal number of latent classes by fitting and comparing models that only include the manifest indicators of interest, (2) use the parameter estimates from the selected model and observed data to calculate posterior probabilities of latent class membership for all individuals (i.e., the classification model; see Equation 1), (3) use the posterior probabilities to classify individuals into latent classes using maximum-probability assignment or multiple pseudo-class draws, (4) conduct an analysis to estimate the relation between latent class membership (treated as known) and the distal outcome, for example, by regressing the distal outcome on classification (i.e., the analytic model).

One reason for the known estimate attenuation in the analytic model is that classification error is introduced when individuals are assigned to latent classes (Bolck et al., 2004). Approaches have been proposed to adjust for this classification error; these approaches treat latent class membership as known in the analytic model, which is weighted by the classification error (Bolck et al., 2004; Vermunt, 2010). The most straightforward version of these approaches is referred to as the 3-step approach and has been implemented in Latent GOLD (Vermunt & Magidson, 2005) and Mplus (L. K. Muthén & Muthén, 1998–2012).

However, classification error is not the only reason for estimate attenuation in the analytic model. In both traditional practice and the 3-step approach, other variables of interest are not included in the classification model. From the multiple imputation literature, if these variables are not included in the classification model but are included in the final analytic model, it is expected that the estimated relations between latent class membership and the other variables will be attenuated (e.g., Collins, Schafer, & Kam, 2001; Schafer, 1997). In particular, with a non-inclusive LCA, we expect attenuation to increase as the strength of the true relation increases. This attenuation is recognized as an important issue when using classify-analyze for LCA (Bolck et al., 2004; Clark & Muthén, 2009; Vermunt, 2010).

We propose inclusive LCA as a straightforward solution to this problem. In this approach, all variables to be included in the final analytic model are included as covariates in the classification model. In other words, the LCA used to obtain the posterior probabilities is generalized to include all variables used in the final analytic model, ensuring that the imputation (i.e., classification) model is as general as the analysis model. This parallels recommendations made in the multiple imputation literature (Collins et al., 2001). When covariates are included in the LCA, calculation of posterior probabilities is additionally conditioned on an individual’s vector of responses to the covariates; that is, calculation of the posterior probabilities from Equation 1 is modified to

$$P(C=c\mid Y=y,X=x)=\frac{P(C=c\mid X=x)P(Y=y\mid C=c,X=x)}{P(Y=y\mid X=x)},$$ (2)

where X represents an individual’s vector of responses, x, to a set of observed covariates.

The motivating empirical demonstration in the current study considers the relatively simple case of using latent class membership, risk exposure, to predict the distal outcome binge drinking. In this case, the proposed inclusive LCA would include the distal outcome as a covariate in the latent class model from which the posterior probabilities are calculated. Because the distal outcome is included in both the classification and analytic models, we expect inclusive LCA, regardless of whether classification is based on maximum-probability assignment or multiple pseudo-class draws, to produce a more accurate estimate of the relation between latent class membership and the distal outcome.

Purpose of the Current Study

In the current study, we propose a flexible, straightforward strategy to improve the posterior probabilities on which classify-analyze approaches for LCA are based, so that attenuation in estimated relations can be reduced or eliminated. First, we present a motivating empirical demonstration in which the relation between risk exposure latent class membership and later binge drinking is examined. Second, a simulation study is conducted in order to examine the relative performance of inclusive and non-inclusive LCA.

Empirical Demonstration

We demonstrate the proposed inclusive LCA using the relatively simple case of predicting a distal outcome from latent class membership. In general, however, the inclusive strategy can be extended to more complex models when it is desirable to treat latent class membership as known. Six manifest variables indicating exposure to various risk factors were used to identify the latent class variable, risk exposure. Risk exposure latent class membership then was treated as known and used to predict the distal outcome, binge drinking in the past year; comparisons are drawn from results based on inclusive and non-inclusive LCA. This demonstration, including selection of the subsample of participants and the measures, was based on a latent class model presented by Lanza and Rhoades (2011).

Participants

Data were from Wave I and Wave II of the public-use data from the National Longitudinal Study of Adolescent Health (Add Health; Harris, 2009; Harris et al., 2009). The sample consisted of n = 844 adolescents who were in 8^th grade at Wave I (53% female; mean age = 14.5 years, SD = .86; 72% White, 20% Black, 3% Asian, 5% Other; 11% Hispanic). Only participants who provided data on exposure to at least one risk factor at Wave I and provided data on binge drinking at Wave II (i.e., 9^th grade) were included in the sample.

Measures

Indicators of risk exposure

Measures of the latent class variable, risk exposure, included two indicators of household risk, two indicators of peer risk, and two indicators of neighborhood risk assessed at Wave I, when the participants were in 8^th grade. Adolescents were considered to be at risk for household poverty if their household income-to-needs ratios were below 1.85; they were considered to be at risk for single-parent household if they lived with a parent/caregiver who was widowed, divorced, separated, or never married at the time of assessment. Adolescents were considered to be at risk for peer cigarette use if one or more of their three best friends smoked at least one cigarette per day; similarly, they were considered to be at risk for peer alcohol use if one or more of their three best friends drank alcohol at least once per month. Adolescents were considered to be at risk for neighborhood unemployment if they lived in a census block where the unemployment rate was greater than 10.9% in 1989 (Billy, Wenzlow, & Grady, 1998); they were considered to be at risk for neighborhood poverty if they lived in a census block where at least 23.9% of the households were living below the poverty level in 1989 (Billy et al., 1998).

Binge drinking

The distal outcome, binge drinking, was measured using a single indicator at Wave II, when the participants were in 9^th grade. Adolescents were considered to be past-year binge drinkers if they reported drinking five or more drinks in a row on one or more days in the past 12 months; 24.8% of adolescents reported binge drinking.

Analysis

LCA was used to confirm that the 5-class model identified by Lanza and Rhoades (2011) was optimal for the public-use sample selected for the current demonstration. Then, inclusive and non-inclusive LCA with maximum-probability assignment and multiple (20) pseudo-class draws were used to assign individuals to latent classes; the proportion of adolescents reporting past-year binge drinking given latent class membership was calculated using each approach.

The data were analyzed using SAS V9 software. Inclusive and non-inclusive LCA were conducted with PROC LCA (Lanza, Dziak, Huang, Wagner, & Collins, 2013); PROC LCA and the corresponding users’ guide are available for free download at methodology.psu.edu/downloads.

Results

First, to confirm that the 5-class model was optimal, LCAs with 1–6 classes were compared based on model fit, parsimony, and stability using the Akaike information criterion (AIC; Akaike, 1974), Bayesian information criterion (BIC; Schwartz, 1989), consistent AIC (CAIC; Bozdogan, 1987), adjusted BIC (a-BIC; Sclove, 1987), bootstrap likelihood ratio test (BLRT; McLachlan & Peel, 2000), and G² fit statistic. A summary of the fit criteria and entropy (Celeux & Soromenho, 1996) are shown in Table 1. Solution stability was based on the proportion of times the maximum-likelihood solution was selected out of 1000 random sets of starting values (Solution %). As expected, the 5-class model was selected as optimal.

Table 1.

Model Fit Information for LCAs with 1–6 Latent Classes Based on the Empirical Study

Classes	df	AIC	BIC	CAIC	a-BIC	BLRTa	Entropy	G2	Solution %
1	57	699.20	727.63	733.63	708.58	—	1.00	687.20	100
2	50	365.40	426.99	439.99	385.71	< .001	.71	339.40	69.5
3	43	179.78	274.54	294.54	211.03	< .001	.78	139.78	100
4	36	140.50	268.43	295.43	182.68	< .001	.86	86.50	19.1
5	29	112.83	273.93	307.93	165.95	< .001	.80	44.83	25.8
6	22	113.90	308.16	349.16	177.96	.185	.81	31.90	19.7

Notes. AIC = Akaike information criterion; BIC = Bayesian information criterion; CAIC = consistent AIC; a-BIC = adjusted BIC; BLRT = bootstrap likelihood ratio test; Solution % = proportion of times maximum-likelihood solution was selected out of 1000 random sets of starting values. Dash indicates criterion was not applicable to the model. Bold font indicates selected model.

^aThe BLRT p-value indicates whether a model with k–1 latent classes (i.e., null model) fits as well as a model with k latent classes (i.e., alternative model).

The parameter estimates for the 5-class model are shown in Table 2. The first latent class was labeled Low Risk (prevalence = 41%) because members had low probabilities of exposure to all six risk factors. In comparison, the second latent class was labeled Peer Risk (22%) because members had high probabilities of exposure to peer cigarette use and peer alcohol use, but low probabilities of exposure to the other four risk factors. Using a similar approach, the third, fourth, and fifth latent classes were labeled Economic Risk (19%), Household & Peer Risk (13%), and Multi-Risk (4%), respectively.

Table 2.

Parameter Estimates for Model of Five Risk Exposure Latent Classes and Effect of Latent Class Membership on Binge Drinking Based on the Empirical Study

		Low Risk	Peer Risk	Econ Risk	H&P Risk	Multi-Risk
Latent Class Membership Probabilities		.41	.22	.19	.13	.04
Indicator	Overall Proportion	Item-Response Probabilities
HH Below Poverty	.37	.24	.00	.68	1.0	.47
HH Single-Parent	.29	.15	.14	.49	.58	.52
Peer Cigarette Use	.38	.00	.88	.15	.89	1.0
Peer Alcohol Use	.42	.16	.77	.21	.77	1.0
NH Unemployment	.23	.06	.06	.68	.19	1.0
NH Below Poverty	.24	.01	.03	.81	.22	.97
Effect of Latent Class Membership on Binge Drinking
Non-Inclusive LCA
Maximum-Probability Assignment		.16	.39	.18	.38	.44
Multiple Pseudo-Class Draws		.16	.37	.17	.39	.41
Inclusive LCA
Maximum-Probability Assignment		.11	.42	.12	.60	.36
Multiple Pseudo-Class Draws		.11	.41	.12	.62	.36

Notes. Econ = Economic; H&P = Household & Peer; HH = Household; NH = Neighborhood; Non-Inclusive = Outcome not included in LCA to generate posterior probabilities; Inclusive = Outcome included in LCA to generate posterior probabilities; Maximum-Probability Assignment = Assignment to latent classes based on maximum posterior probability; Multiple Pseudo-Class Draws = 20 assignments to latent classes based on each individual’s distribution of posterior probabilities. Table entries for the item-response probabilities represent probabilities of endorsing the indicators of risk exposure conditional on latent class membership. Table entries for the effects of latent class membership on binge drinking represent the proportions of past-year binge drinking conditional on latent class membership. The overall proportion of participants reporting binge drinking was .25.

Second, posterior probabilities from this 5-class model were retained for the analysis based on non-inclusive LCA. Binge drinking was then added to the latent class model as a covariate in order to generate posterior probabilities for the analysis based on inclusive LCA. Notably, binge drinking was significantly related to latent class membership (2 logL = 88.4, df = 4, p < .0001, Cohen’s w = .42). The odds of membership in the Peer Risk (OR = 5.5), Household & Peer Risk (OR = 13.4), and Multi-Risk (OR = 4.8) latent classes relative to the Low Risk latent class were significantly higher for binge drinkers compared to those who did not binge drink.

Third, using the posterior probabilities from the non-inclusive LCA, individuals were assigned to latent classes (a) once using maximum-probability assignment, and (b) twenty times using multiple pseudo-class draws. Then, using the posterior probabilities from the inclusive LCA, individuals were again assigned to latent classes (a) once using maximum-probability assignment, and (b) twenty times using multiple pseudo-class draws.

Fourth, the effect of risk exposure latent class membership on binge drinking was estimated by calculating the proportion of adolescents reporting past-year binge drinking given (assigned) latent class membership. These estimates, for both inclusive and non-inclusive LCA, are shown in Table 2. As a point of comparison, recall that the overall proportion of adolescents reporting binge drinking was .25. Based on the traditional approach of non-inclusive LCA, results did not differ appreciably between the maximum-probability and multiple pseudo-class draws assignment procedures. This was also the case for the proposed inclusive LCA.

As shown in Table 2, adolescents in the Low Risk and Economic Risk latent classes were less likely to report binge drinking compared to adolescents in the Peer Risk, Household & Peer Risk, and Multi-Risk latent classes. The estimates of the proportions, however, differed substantially depending on whether an inclusive or non-inclusive strategy was used. Estimates based on the traditional non-inclusive LCA tended to be closer to the marginal proportion of .25, as compared to the proposed inclusive strategy. For example, the non-inclusive multiple pseudo-class draws approach estimated that 39% of Household & Peer Risk adolescents reported binge drinking; the corresponding estimate based on inclusive LCA was 62%. This is consistent with our expectation that associations between latent class membership and other variables of interest may be attenuated with a non-inclusive strategy. To assess the performance of inclusive LCA more rigorously, a simulation study was conducted using the empirical demonstration as a basis for its design.

A Monte Carlo Study Comparing Inclusive and Non-Inclusive LCA for Classify-Analyze Approaches

The primary objective of this simulation study was to compare the performance of inclusive and non-inclusive strategies for classify-analyze. Secondary objectives were to compare, for both inclusive and non-inclusive LCA, relative performance of maximum-probability assignment and multiple pseudo-class draws, and to compare the performance of the different approaches under varying conditions of measurement quality, effect size, and sample size.

Experimental Design

Five latent classes, corresponding to Low Risk, Peer Risk, Economic Risk, Household & Peer Risk, and Multi-Risk, were indicated by six binary variables. Latent class membership probabilities were held constant across all conditions: .40 for Low Risk, .20 for Peer Risk, .20 for Economic Risk, .10 for Household & Peer Risk, and .10 for Multi-Risk. Three factors were considered in this study because they were expected to have a substantial impact on the performance of inclusive and non-inclusive LCA: measurement quality, the size of the effect of latent class membership on the distal outcome, and sample size. We conducted a fully-crossed factorial design; a single cell of the simulation represents one combination of measurement quality, effect size, and sample size conditions.

Measurement Quality

Measurement quality was examined because it is directly linked to posterior probability estimates: as measurement quality increases, measurement error decreases and posterior probabilities move closer to 0 and 1, indicating greater confidence in class assignment. By measurement quality we mean the degree of association between the latent class variable and its manifest indicators. Three sets of item-response probabilities were specified to represent different levels of measurement quality: high quality (probabilities of .9 and .1), medium quality (.8 and .2), and low quality (.7 and .3).

Effect Size

Strength of the association between the latent class variable and distal outcome (i.e., effect size) was manipulated. If, as hypothesized, effects estimated using a non-inclusive strategy are attenuated, this attenuation should be more pronounced for stronger relations between latent class membership and the distal outcome. Four Cohen’s w effect sizes were specified to correspond to different strengths of the relation between the latent classes and distal outcome: large effect (.50), medium effect (.30), small effect (.10), and no effect (.00). The true overall proportion of binge drinking differed somewhat across different effect sizes (ranging from .19 to .30) due to the requirements specified in data generation (i.e., latent class prevalences and effect size).

Sample Size

Sample size was examined because it affects the precision of estimates of both the latent class model and the effect of latent class membership on the distal outcome. Two sample sizes were considered: large (n = 800), which approximated that of the empirical demonstration, and small (n = 400).

Analytic Procedure for the Monte Carlo Study

The following procedure was used for each cell of the simulation.

Data Generation

Given (a) the latent class model specified by the latent class membership probabilities and item-response probabilities, (b) the strength of the association between latent classes and distal outcome, and (c) the sample size, random observations were generated by (1) generating a latent class variable from a multinomial distribution specified by the true latent class membership probabilities, (2) generating item responses based on the true item-response probabilities, and (3) generating outcomes based on the multinomial logistic regression model linking latent class membership and the distal outcome. For each set of conditions (i.e., cell), 1000 replicate datasets were generated.

Classification Step

Two latent class models were fit to each replicate dataset: an inclusive LCA that included the distal outcome as a covariate and a non-inclusive LCA (i.e., traditional LCA with no covariates). To ensure model identification in the non-inclusive LCAs, 100 random sets of starting values were used; parameter estimates from the maximum-likelihood solution were used as starting values for the inclusive LCAs. Then, the maximum-probability assignment rule was used to infer class membership based on posterior probabilities from the non-inclusive LCAs, and again based on posterior probabilities from the inclusive LCAs. Finally, 20 pseudo-class draws were used to repeatedly assign class membership based on posterior probabilities from the non-inclusive LCAs, and again based on posterior probabilities from the inclusive LCAs¹.

Analysis Step

For every replicate dataset, the effect of latent class membership on the distal outcome was estimated by calculating the proportion of observations with the distal outcome, conditional on (assigned) latent class membership. This was estimated for each combination of inclusive or non-inclusive LCA and maximum-probability assignment or multiple pseudo-class draws. The average estimated relation between latent class membership and the distal outcome across replicates was compared to the true values. Adequacy of the estimated effect was examined in terms of bias (i.e., mean estimated value minus true value; smaller is better) and root mean square error (RMSE; i.e., $\sqrt{{\mathit{bias}}^2+{SE}^2}$ where SE is the standard error of the 1000 estimated values; smaller is better), which balances bias and variability across replicates.

Results

Inclusive Versus Non-Inclusive LCA

In order to summarize the results concisely, results for only the Household & Peer Risk latent class are presented and discussed². This latent class was small and had the highest prevalence of binge drinking; thus, this set of results is ideal for studying the relative performance of inclusive and non-inclusive LCA. Simulation results for this latent class are shown in Table 3; each cell contains the bias and RMSE for the estimated proportion of adolescents in this class reporting binge drinking. For example, Table 3 shows that for high measurement quality, large effect size, and large sample size, bias in the estimated proportion of adolescents in this class reporting binge drinking was −.115 and −.012 for non-inclusive and inclusive LCA using maximum-probability assignment, respectively. In other words, the proportion of adolescents binge drinking was substantially underestimated using the trraditional classification approach.

Table 3.

Bias (Root Mean Square Error) for Estimate of the Effect of Latent Class Membership on the Distal Outcome for the Household & Peer Risk Latent Class Based on the Monte Carlo Study

		Bias (RMSE)
		Non-Inclusive LCA		Inclusive LCA
Measurement Quality	Effect Size	Max Prob	Pseudo-Class	Max Prob	Pseudo-Class
Large Sample Size (n = 800)
High	Large	−.115 (.132)	−.132 (.146)	−.012 (.087)	−.032 (.086)
	Medium	−.059 (.081)	−.069 (.087)	−.009 (.075)	−.021 (.072)
	Small	−.019 (.055)	−.023 (.053)	.000 (.068)	−.005 (.063)
	No Effect	.000 (.047)	.000 (.043)	−.001 (.061)	.000 (.056)
Med	Large	−.253 (.278)	−.287 (.304)	−.007 (.204)	−.061 (.190)
	Medium	−.136 (.156)	−.156 (.169)	−.021 (.163)	−.047 (.147)
	Small	−.052 (.075)	−.060 (.075)	−.010 (.115)	−.021 (.100)
	No Effect	−.002 (.050)	−.002 (.042)	−.001 (.099)	−.002 (.086)
Low	Large	−.404 (.418)	−.421 (.430)	−.113 (.325)	−.199 (.316)
	Medium	−.228 (.242)	−.236 (.245)	−.090 (.275)	−.136 (.243)
	Small	−.085 (.106)	−.090 (.102)	−.034 (.203)	−.058 (.154)
	No Effect	−.000 (.060)	−.001 (.044)	.016 (.188)	.005 (.131)
Small Sample Size (n = 400)
High	Large	−.119 (.145)	−.134 (.157)	−.034 (.110)	−.045 (.113)
	Medium	−.054 (.093)	−.063 (.096)	−.010 (.101)	−.018 (.097)
	Small	−.020 (.075)	−.024 (.073)	−.004 (.087)	−.008 (.084)
	No Effect	−.002 (.072)	−.003 (.068)	−.001 (.090)	−.002 (.087)
Med	Large	−.294 (.330)	−.317 (.345)	−.099 (.273)	−.136 (.270)
	Medium	−.157 (.188)	−.171 (.194)	−.062 (.202)	−.082 (.189)
	Small	−.064 (.098)	−.069 (.095)	−.034 (.146)	−.042 (.133)
	No Effect	.000 (.069)	−.000 (.060)	.001 (.133)	−.001 (.117)
Low	Large	−.415 (.432)	−.426 (.438)	−.201 (.358)	−.254 (.359)
	Medium	−.226 (.242)	−.233 (.244)	−.116 (.272)	−.148 (.250)
	Small	−.085 (.115)	−.089 (.109)	−.035 (.206)	−.052 (.173)
	No Effect	−.004 (.077)	−.004 (.065)	.006 (.184)	.002 (.154)

Notes. High = High Measurement Quality; Med = Medium Measurement Quality; Low = Low Measurement Quality; Max Prob = Maximum-Probability Assignment; Pseudo-Class = 20 Pseudo-Class Draws. The true proportions of past-year binge drinking conditional on membership in the Household & Peer Risk latent class were .67 for the large effect, .48 for the medium effect, .37 for the small effect, and .30 for the no effect conditions.

For both classification approaches (i.e., maximum-probability assignment and multiple pseudo-class draws), bias was substantially smaller using the proposed inclusive LCA compared to non-inclusive LCA. For example, Table 3 shows that with high measurement quality, large sample size, and maximum-probability assignment, an inclusive strategy resulted in biases of −.012, −.009, and .000 for large, medium, and small effects, respectively, compared to non-inclusive biases of −.115, −.059, and −.019 for large, medium, and small effects, respectively. As expected, relying on a traditional classification approach (i.e., use of a non-inclusive strategy) produced substantially attenuated estimates. The results across all five latent classes were consistent in that, across all simulated conditions, the traditional (non-inclusive) LCA produced estimated proportions of binge drinking biased toward the overall population proportion.

Maximum-Probability Assignment Versus Multiple Pseudo-Class Draws

An important finding is that under all conditions, maximum-probability assignment outperformed multiple pseudo-class draws in terms of bias. This was true for both inclusive LCA and the standard, non-inclusive LCA. For example, Table 3 shows that with high measurement quality, large effect size, and large sample size, bias based on non-inclusive LCA was −.115 for maximum-probability assignment versus −.132 for multiple pseudo-class draws. Similarly, bias based on inclusive LCA was −.012 for maximum-probability assignment versus −.032 for inclusive multiple pseudo-class draws.

The RMSE results are somewhat more complicated, as the RMSE accounts for both bias and random noise in the approaches. Overall, for both inclusive and non-inclusive LCA, the RMSEs for maximum-probability assignment and multiple pseudo-class draws were comparable. However, it is important to note that for both inclusive and non-inclusive strategies, the maximum-probability assignment approach introduced slightly more random noise compared to the multiple pseudo-class draws approach. This was expected given that multiple pseudo-class draws reduce random noise by averaging across multiple imputations of the latent class variable.

Factors Affecting Performance

As measurement quality increased, all approaches were less biased, but inclusive LCA still performed substantially better than non-inclusive LCA. For example, under the condition of a large effect size and large sample size, using inclusive multiple pseudo-class draws, bias decreased from −.199 to −.061 to −.032 when measurement quality increased from low to medium to high (see Table 3). A similar pattern was seen for non-inclusive multiple pseudo-class draws bias (with corresponding biases of −.421 to −.287 to −.132).

As effect size increased (i.e., strength of the relation between latent class membership and distal outcome increased), attenuation of the estimated effect increased, and was particularly pronounced with non-inclusive LCA. For example, using multiple pseudo-class draws under the condition of high measurement quality and a large sample size, bias increased from −.023 to −.069 to −.132 as the effect size increased from small to medium to large for non-inclusive LCA (see Table 3), compared to corresponding biases of −.005 to −.021 to −.032 for inclusive LCA.

As sample size increased, all approaches were slightly less biased, yet even with a large sample size inclusive LCA still performed substantially better than non-inclusive LCA. For example, using inclusive multiple pseudo-class draws, increasing the sample size from small to large decreased bias from −.045 to −.032 for high measurement quality and large effect size (see Table 3). The benefit of a larger sample size was less pronounced for non-inclusive LCA, with a corresponding decrease in bias from −.134 to −.132.

Importantly, increasing measurement quality appeared to decrease the bias more than doubling the sample size. Overall, even with inclusive LCA, results were biased when sample size was small, effect size was large, and measurement quality was low. In contrast, with an inclusive strategy, bias was nearly eliminated with medium or high measurement quality and a large sample size, or high measurement quality and a small sample size.

Discussion

Motivated by an empirical examination of the effect of risk exposure latent class membership on later binge drinking, our Monte Carlo study demonstrated the importance of using an inclusive strategy when implementing classify-analyze approaches for LCA. Based on these results, we recommend the use of an inclusive LCA for obtaining the posterior probabilities when a classify-analyze approach is to be used regardless of the measurement quality, size of the effect of latent class membership, or sample size. Although this Monte Carlo study demonstrated the benefit of using inclusive LCA for the relatively simple case of estimating the effect of latent class membership on a distal outcome, we believe the approach can be extended to a variety of cases where a latent class variable is embedded in a more complex theoretical model.

Addressing the primary objective of the current study, we showed that an inclusive classify-analyze approach for relating latent class membership to other variables of interest can be used to obtain unbiased estimates. This is consistent with the literature on multiple imputation for missing data (e.g., Collins et al., 2001; Schafer, 1997) that emphasizes the importance of imputing data under a model that is at least as general as the analytic model. We reiterate that an inclusive strategy is recommended only when a classify-analyze approach for LCA is necessary. For example, the model for LCA with covariates is well-understood, so addressing questions about predictors of latent class membership need not be addressed using classify-analyze approaches. Addressing the secondary objective, we showed that maximum-probability assignment is less biased than multiple pseudo-class draws regardless of whether inclusive or non-inclusive LCA is used. Importantly, in terms of bias, there was a clear rank-ordering in the performance of the approaches. Traditional non-inclusive multiple-pseudo-class draws performed the worst, followed by non-inclusive maximum-probability assignment. Then, inclusive multiple pseudo-class draws performed better, and inclusive maximum-probability assignment performed the best.

Consideration of the RMSE shows a more complicated pattern of results in terms of tradeoffs between bias and variability. There is no clear winner in terms of RMSE; rather, the relative benefit of using one approach over another depends on the effect size. In particular, as the effect size gets large, both bias and variability increase, but bias increases more rapidly. So, with large effects it is critical to use an inclusive LCA to combat bias despite the increase in variability. For small effects, bias and variability are smaller, and the RMSE is driven more by variability than bias; thus, the less variable non-inclusive multiple pseudo-class draws approach has better RMSEs. Overall, though, we still prefer the inclusive approaches when considering the RMSE because they are relatively less impacted in situations with large effect sizes (a desirable situation in substantive applications).

In sum, two general conclusions can be drawn from the Monte Carlo study. First, inclusive LCA was substantially less biased compared to non-inclusive LCA. As expected, when non-inclusive LCA was used, the effect of latent class membership on the distal outcome was attenuated. Inclusive LCA resulted in unbiased estimates in cases with sufficiently strong measurement quality or sufficiently large sample size. Second, given that an inclusive strategy is used, one can expect comparable performance from multiple pseudo-class draws and the simpler maximum-probability assignment rule.

Classify-analyze approaches are based on posterior probabilities derived from a latent class measurement model. Bias in estimates of the relation between latent class membership and other variables of interest is substantially reduced or, in many cases, eliminated using an inclusive LCA because the posterior probabilities generated from this model take into account information about associations between the latent class variable and other variables, whereas those from a traditional non-inclusive LCA do not. In other words, the imputation model based on inclusive LCA is sufficiently general so that it is compatible with the analytic model.

In the context of latent classes of developmental trajectories (i.e., group-based models of development; Nagin, 2005, or growth mixture models; B. O. Muthén & Sheddon, 1999), it has been suggested that when the mean posterior probability of class membership for individuals assigned to each class exceeds .70, hypothesis tests of differences across classes may be unchanged (Nagin, 2005; Roeder, Lynch, & Nagin, 1999; White, Nagin, Replogle, & Stouthamer-Loeber, 2004). Other methods for quantifying the extent of classification error (i.e., classification uncertainty) have been proposed in the context of assessing the overall quality of a latent class model (Celeux & Soromenho, 1996; Goodman, 2007; Vermunt & Magidson, 2002). In our empirical demonstration, the mean posterior probabilities of membership for individuals assigned to each class ranged from .84 (Household & Peer Risk) to .93 (Economic Risk) for the non-inclusive maximum-probability assignment approach, which showed considerable bias in the simulation study. In comparison, the mean posterior probabilities ranged from .82 (Household & Peer Risk) to .92 (Economic Risk) for the inclusive maximum-probability assignment approach, which performed very well in the simulation study. In sum, regardless of how small classification error is, we have demonstrated that relations between latent class membership and other variables can be attenuated if information about associations between the latent class variable and other variables is not accounted for in the model used to generate the posterior probabilities upon which the classifications are based.

Practical Recommendations

Results of the Monte Carlo study provide overwhelming evidence of superior performance of inclusive LCA over the traditional practice of non-inclusive LCA. Combined with previous work on LCA (e.g., Lanza et al., 2007), the results of the current study suggest a series of steps when using classify-analyze approaches for LCA: (1) determine the optimal number of latent classes by fitting and comparing models without covariates included in the model (i.e., use non-inclusive LCA for model selection); (2) re-fit the latent class model with the other variables of interest included as covariates to produce posterior probabilities; (3) use either maximum-probability assignment or multiple pseudo-class draws, as the analyst prefers, to classify individuals; (4) treat class membership as known to perform the desired analysis. If multiple pseudo-class draws is used, results must be combined across datasets for the final results.

All modern statistical software packages for LCA, including PROC LCA (Lanza et al., 2013), Mplus (L. K. Muthén & Muthén, 1998–2012), and Latent GOLD (Vermunt & Magidson, 2005), provide posterior probabilities of latent class membership using non-inclusive LCA. Thus, simply by including other variables of interest as covariates, inclusive LCA can be conducted with any of these packages.

Limitations and Future Directions

Several limitations of the empirical demonstration and simulation study merit mentioning. First, the sample for the empirical demonstration did not include participants missing data on binge drinking (i.e., the distal outcome), and the final analytic model in the empirical demonstration did not incorporate Add Health’s sampling weights. In turn, the simulation study did not investigate the impact of different missing data mechanisms or rates of missing data on the performance of inclusive and non-inclusive LCA, nor did it investigate the impact of weighting the final analytic model. These are common features of data analyzed in substantive research; thus, careful consideration should be given to addressing these features in future empirical work.

Second, as with any simulation study, conclusions about the results cannot be generalized beyond the set of conditions that were examined. For example, the current study did not investigate the effect of varying the latent class membership probabilities. Perhaps most importantly, this study only examined the relatively simple case of using classify-analyze to predict a single distal outcome from latent class membership. Research questions posing more complex relations, such as a latent class variable as a moderator or mediator, would require that multiple variables be included as covariates in the classification step. Extrapolating from the results here, and considering the known impact of imputing data under a classification model that is more restrictive than the analytic model (e.g., Collins et al., 2001; Schafer, 1997), we believe that failure to include all additional variables of interest, along with relevant interactions, in the estimation of the posterior probabilities could result in significant attenuation of the effects in the subsequent analysis step.

As discussed, inclusive LCA is analogous to multiple imputation, a model-based procedure that requires assumptions about the missing data mechanism, specifically that data are missing at random (MAR; Little & Rubin, 1987; Rubin, 1976). Multiple imputation also typically requires the additional assumption that data are from a multivariate normal distribution. Similarly, the inclusive strategy for imputing latent class membership, a variable that is 100% missing, relies on certain assumptions. In the scenarios examined here, we have assumed (1) independence of the indicators conditional on latent class, (2) a linear relation (on the logit scale) between the latent class variable and any other variables of interest (here, distal outcome), and (3) independence of the indicators and any other variables of interest conditional on latent class.

As research questions regarding the role of latent class membership in developmental processes become more complex, it is increasingly difficult to address all questions within the context of the latent class model. Despite the fact that standard non-inclusive LCA is common practice for conducting classify-analyze in LCA, the approach is known to attenuate estimated relations between the latent class variable and other variables in the final analytic model. The current study confirmed that this attenuation can be substantial, and an effective and straightforward solution was demonstrated: fitting an inclusive LCA to derive posterior probabilities. This strategy does not require special software, and can be readily adopted by scientists to reduce or eliminate bias in the associations between a latent class variable and other variables of interest, opening the door to broader modeling options when a latent class variable is embedded in a complex theoretical model. A careful study of the performance of inclusive LCA in the context of more complex analytic models remains an important topic for future research.