Mediation Analysis with Binary Outcomes: Direct and Indirect Effects of Pro-Alcohol Influences on Alcohol Use Disorders

Abstract

A risk factor or intervention (an independent variable) may influence a substance abuse outcome (the dependent variable) indirectly, by affecting an intervening variable (a mediator) that in turn affects that outcome. Mediation analysis is a statistical method commonly used to examine the interrelations among independent, mediating, and dependent variables to obtain the direct and indirect effects of an independent variable on a continuous dependent variable. However, mediation analysis may also be used with binary outcomes, such as a diagnosis of an alcohol use disorder (AUD). Study 1 demonstrated methods of mediation analysis with binary outcomes by examining the direct and indirect effects of pro-alcohol social influences on an AUD, as a function of: (a) the distribution of the independent variable (binary vs. continuous), (b) the frequency of the outcome (non-rare vs. rare), and (c) the effect metric (probability vs. odds ratio). Study 2 was a Monte Carlo (simulation) study of bias in the indirect effects based on estimates from the first study. These methods have wide applicability in addictions research because many key outcomes are binary, and mediation analysis is frequently used to study the causal mechanisms by which interventions and risk factors affect substance abuse.

1. Introduction

Because the etiology of problematic drinking and an ensuing alcohol use disorder (AUD) is of critical importance in the addictions field (Tarter, 2002), research has often examined risk factors for alcohol abuse (see review by Stone, Becker, Huber, & Catalano, 2012). Such risk factors are examples of independent variables that may impact alcohol abuse indirectly by affecting an intervening variable (called a mediator) that in turn affects alcohol abuse.

Mediation analysis (Hayes, 2013; MacKinnon, 2008; Preacher, & Kelley, 2011) is a statistical method for estimating the direct and indirect effects of independent variables to test theories about the causal mechanisms by which risk factors affect outcomes—and provides information about the active ingredients of an intervention, thereby improving its effectiveness and reducing costs (MacKinnon, 2011). This causal modeling approach is often used in research on addictive behaviors. For example, methodological articles or books on mediation were cited in 8 of the 10 articles in a special section on “mechanisms of behavior change” in the March 2018 issue of Journal of Studies on Alcohol and Drugs, and mediation was a focus of 3 of them (Finney, 2018; O’Rourke & MacKinnon, 2018; Witkiewitz, Roos, Tofighi, & Van Horn, 2018).

Binary outcomes—including use versus abstinence, treatment completion, and diagnosis of a substance abuse disorder or AUD—are common in studies of addictive behaviors (Feingold, Oliveto, Schottenfeld, & Kosten, 2002; MacKinnon & Dwyer, 1993). Effects on binary outcomes are examined with categorical data analysis, such as binary logistic regression, which models outcome probabilities via a logistic or probit regression framework (Agresti, 2002: Hosmer & Lemeshow, 2000),

A commonly used effect size for findings from a binary logistic regression is the odds ratio (OR; Fleiss & Berlin, 2009), which is a categorical analogue of the Cohen’s d used for continuous outcomes (Feingold, 2009). The odds is defined as the probability of an event occurring divided by the probability that the event does not occur, e.g., the probability of maintaining sobriety divided by the probability of relapsing. If, for example, a treatment for alcohol abuse is effective, the odds for an alcohol outcome would differ between the treatment and the control groups following treatment, and the ratio of the odds for the treatment group to the odds for the control group is the OR communicating the degree to which participants in the treatment group have a higher odds of, say, remaining sober. (When the intervention is ineffective, the odds for the two groups would be the same, and the expected value of the OR would be 1.00.)

ORs for the direct and indirect effects on a binary outcome from a mediation analysis can be obtained using varied statistical software, including Mplus and the Medflex package in R (Lange, Vansteelandt., & Bekaert, 2012; Muthén, Muthén, & Asparouhov, 2016; Tchetgen Tchetgen, 2013). This article demonstrates an application of mediation analysis that obtains ORs for the direct and indirect effects of binary and continuous predictors on binary outcomes.

The algorithms used to produce these ORs vary with the prevalence of the outcome. For example, a method proposed by VanderWeele and Vansteelandt (2010) assumes the outcome occurs rarely (e.g., less than 10% of the time), in which case the ORs accurately measure relative risk. ORs can also be obtained for direct and indirect effects under the assumption that the outcome is not rare (Muthén, et al., 2016), although ORs associated with non-rare outcomes are not good measures of relative risk (Cummings, 2009). The indirect effect on a binary outcome can also be expressed in a probability metric.

The methods used to obtain indirect effects on a binary outcome also depend on whether the predictor is binary or continuous because the effects are estimated for a one unit change in the predictor. Thus, there are four types of models for: (a) a continuous predictor and non-rare outcome; (b) a continuous predictor and rare outcome; (c) a binary predictor and non-rare outcome; and (d) a binary predictor and rare outcome. Each of these four models could be estimated with logistic or probit regression. We focus on logistic regression in this article because it easily provides estimates of odds ratios by the exponentiation of logistic regression coefficients.

1.1. Current studies

1.1.1. Study 1: Illustrative analysis

Social influence, such as behaviors by peers (e.g., peer pressure) and romantic partners, are known to affect alcohol use and misuse (Kim, Tiberio, Pears, Capaldi, & Washburn, 2013; Mushquash et al., 2013; Washburn, Capaldi, Kim, & Feingold, 2014). Alcohol-related cognitions, including alcohol expectancies and drinking motives, are commonly hypothesized to be mediators of the association between alcohol risk factors and alcohol abuse (Goldman, Darkes, & Del Boca, 1999; Kuntsche, Knibbe, Engels, & Gmel, 2007; Lammers, Kuntsche, Engels, Wiers, & Kleinjan, 2013; Studer et al., 2016; Urbán, Kökönyei, & Demetrovics, 2008). Thus, it has been hypothesized that the effects of social influences on alcohol outcomes are mediated by alcohol expectancies (e.g., Wood, Read, Palfai, & Stevenson, 2001), and this study examines this hypothesis in a secondary analysis of data from a study that found effects of pro-alcohol social influences and alcohol expectancies on AUDs but did not consider the latter as a mediator (Feingold, Capaldi, & Owen, 2015).

This tutorial assumes familiarity with mediation analysis for continuous outcomes (e.g., knowledge of interpretation of indirect effects) and thus addresses differences in the analysis when the outcome is binary instead of continuous. Readers without that background should use this article in conjunction with an introductory text on mediation (e.g., Hayes, 2013; MacKinnon, 2008). A basic understanding of methods of categorical data analysis, such as logistic regression, is also assumed.

1.1.2. Study 2: Bias in indirect effects

Methodological research on ORs obtained from logistic regression analysis has found ORs are unduly biased in small samples (Nemes, Jonasson, Genell, & Steineck, 2009), which may also be true for ORs for the indirect effects obtained in mediation analysis. Bias in an observed OR means that the OR is expected to be higher or lower than the parameter of which it is an estimate, and bias in its 95% confidence interval (CI) means the interval for the OR is expected to be to the right or left of the true OR. Undue bias in estimates may also be an issue when the indirect effect is expressed in a probability metric.

A common approach for evaluating bias in parameter estimates (e.g., ORs) uses Monte Carlo (i.e., simulation) studies (e.g., Feingold, in press; Hedges, Pustejovskya, & Shadish, 2012), and such simulations have examined bias in indirect effects in mediation analysis (e.g., MacKinnon, Warsi, & Dwyer, 1995; Miočević, O’Rourke, MacKinnon, & Brown, 2018). To investigate bias in the indirect effects obtained in Study 1, Study 2 describes a Monte Carlo simulation for mediation analysis with a binary outcome that used parameter estimates observed in Study 1. The bias in the indirect effect was interpreted using conventional guidelines for evaluating bias in point estimates, SEs, and CIs (Muthén & Muthén, 2002).

1.2. Summary of objectives

There are three primary objectives of Study 1. The first aim is to provide a more practically-focused illustration of methods of mediation analysis with a binary predictor and a binary outcome--that communicates the indirect effect in both probability metrics and ORs--than is available elsewhere (e.g., Muthén, et al., 2016; VanderWeele & Vansteelandt, 2010). Next, the study illustrated methods for use with a continuous predictor and a binary outcome, which was covered in Muthén, et al. (2016) only in an example that was more complex than the models used in this study. The final goal is to provide a guide for presentation of results from a mediation analysis with a binary outcome. The Muthén, et al. text presented all results from their examples via unmodified sections from Mplus output files, which include more statistics (such as thresholds, residual variances, and multiple CIs) than would typically be reported, This article thus provides a template for presentation of direct and indirect effects in text and tables obtained with any statistical software.

There are several objectives of Study 2, which examined parameter bias in the estimates from Study 1. Key research questions relate to the required sample size. Is my sample (or proposed sample) large enough to ensure that my findings (e.g., parameter estimates) will be valid, and is statistical power sufficient to detect them? Monte Carlo analysis is a popular approach for addressing both of these questions.

This Monte Carlo study investigated bias in the indirect effects, and g provides uidance for adequate sample size. The input statements for the Monte Carlo study also serve as a template for investigators planning a study (or preparing a grant application).

2. Study 1

2.1. Method

2.1.1. Participants

A total of 206 9–10-year-old male participants and their families were recruited for participation in the long-term Oregon Youth Study (OYS) from schools in neighborhoods in the Pacific Northwest characterized by higher rates of juvenile delinquency. The participants’ alcohol use was assessed annually or biennially over the next 3 decades (Feingold, Kerr, & Capaldi, 2008). The current study included men who had participated in the OYS through the last wave of data collection, at ages 37–38 years, and for whom AUD diagnostic data were available. The men were mostly non-Hispanic Whites (90%), and 75% were of low socioeconomic status. Starting when the men were late adolescents (ages 17–18 years), they were invited to participate in the OYS–Couples Study, approximately biennially, with a current romantic partner. Substance use-related measures were obtained from partners (with the partners sometimes varying across assessments). All procedures were approved by the Institutional Review Board of the Oregon Social Learning Center.

From the original sample of 206 participants, 5 died prior to participation in the last two assessments, 1 reported treatment but was lacking diagnostic data, and a further 16 did not participate when the final treatment instrument was administered. This resulted in 183 participants, 2 of whom reported treatment but no AUD and were thus eliminated from the current study, resulting in a final N of 181 for the sample used in Feingold et al. (2015)—which compared means between AUD and AUD-free men on alcohol influences and expectancies, as well as between AUD men who had and had not sought treatment. This sample was used for the illustrative analysis in this study, although not all pro-alcohol influence measures were available for all 181 participants.

2.1.2. Measures

2.1.2.1. Independent variables (pro-alcohol social influence measures)

2.1.2.1.1. Male peers’ and female partners’ heavy drinking.

Male peers’ heavy drinking was assessed twice when the OYS men were aged 20–23 and 37–38 years using the OYS men’s responses to the questionnaire item, How many of your friends get drunk once in a while?. Responses were on a 5-point Likert scale, ranging from 0 (none of them) to 4 (all of them), and scores were summed to generate the peer drinking measure for each assessment wave. Scores were then averaged across the two waves to form a continuous measure of influence of male peers’ heavy drinking. For analysis involving a binary predictor and a binary outcome, mean splits were used to dichotomize the continuous alcohol influence measures, and corresponding analyses were also conducted in which these variables were not dichotomized in analysis with a continuous predictor and a binary outcome.

Female romantic partners’ heavy drinking was assessed up to eight times from reports of the men’s partners collected at ages 20–23 to 37–38 years based on the question, Thinking about all types of alcohol, like beer, wine, or hard liquor, how many drinks do you usually have at one time (one occasion)? Responses for each woman at each wave was recoded to a 4-point scale (0 = 0 drinks, 1 = 1–2 drinks, 2 = 3–4 drinks, 3 = 5 or more drinks). These scores for the women’s drinking were then averaged across waves for use in mediation analysis with a continuous predictor, and dichotomized for use in mediation analysis with a binary predictor.

2.1.2.1.2. Male peers’ and female partners’ substance use influence.

At each of the OYS–Couples Study assessments from ages 27–38 years (up to five participations), the romantic partner completed a social influence questionnaire. A male peer was administered the questionnaire on two occasions (OYS men’s ages 23–24 and 27–28 years). The items from each questionnaire formed seven subscales (OYS men’s substance use, partner/peers’ facilitation of men’s substance use, partner/peer substance use, partner/peers’ avoidance of substance use situations, partner/peer organize/enjoy socializing without substances, partners/peer disapproval of substance use, and partner/peers’ attempts to quit substances) that were combined to yield a substance use influence score. Scores were averaged across waves and then dichotomized to yield the social influence binary predictors used in models for binary independent variables.

2.1.2.2. Mediator (alcohol expectancies)

Expectations about the effects of alcohol were assessed with a scale composed of nine self-report items (e.g., alcohol makes people worry less) administered when the participants were ages 9–10 and 11–12 years (αs .61-.65), and with the Modified Drinking Motive Questionnaire—Revised (Grant, Stewart, O’Connor, Blackwell, & Conrod, 2007) when they were ages 35–36 and 37–38 years (αs = .98 at both assessments).

2.1.2.3. Outcome variable (lifetime AUD diagnosis by ages 37–38 years)

The binary outcome measure was lifetime diagnosis of AUDs by ages 37–38 years, as determined by meeting the criteria for an AUD at any assessment wave with any of three diagnostic measures. A structured psychiatric interview was administered to determine lifetime DSM-IV diagnoses for an AUD at three times, when the men were ages 16–17 years, 25–26 years, and 35–36 years. The Diagnostic Interview Schedule for Children (DISC; Costello, Edelbrock, Kalas, Kessler, & Klarik, 1982) was administered once when the participants during late adolescence, and the Composite International Diagnostic Interview (CIDI; World Health Organization, 1997) was administered twice in adulthood. In addition, the men completed the Michigan Alcohol Screening Test (MAST; Selzer, 1971) at each wave, and a score exceeding 12 on that MAST indicates a diagnosis of an AUD (Selzer, 1971). An AUD history was defined as either being diagnosed with an AUD at least once with the DSM-IV (American Psychiatric Association, 2000) criteria using either the DISC or the CIDI, or scoring above 12 on the MAST at least 1 time. (The he MAST was administered at 7 time points from ages 20–21 to 37–38 years.) At first administration, α = .72. About 70% of the participants meet the criteria for a current or lifetime AUD on one of the diagnostic measures at any wave (see Feingold et al., 2015).

2.1.3. Design and analysis

The indirect, direct, and total effects of pro-alcohol influences on a lifetime diagnosis of an AUD by middle age (the binary outcome)—with alcohol expectancies as the mediator—were examined to illustrate the use of methods of mediation analysis with a binary dependent variable. Two different mediation models—a model assuming no interactions between the predictor (x) and the mediator (m), and a Predictor x Mediator interaction model--were estimated for the same mediator (alcohol expectancies) and outcome (AUD diagnosis) for each of four social influence predictors. These influence measures were treated as both continuous and binary (with the latter formed by dichotomization of the naturally continuous predictors) to illustrate differences in the Mplus input used with the two types of distributions of the predictors.

The Mplus input statements were used to conduct the three analyses with scaling of the predictors as binary or continuous, separately for each of the four predictors. Thus, 2 by 2 by 4 equals 16 mediation analyses. The input statements included the MODEL CONSTRAINT command used to determine the ORs for indirect effects under the rare outcomes assumption (presented solely for didactic purposes given the AUD distribution in this study) in addition to ORs obtained under the default non-rare outcome assumptions that were warranted for the study (see Appendix A for an sample input statement). The bootstrap (Efron & Tibshirani, 1993) is commonly used to obtain standard errors (SEs) and confidence intervals (CIs) for the indirect effect in mediation analysis (MacKinnon, Lockwood, & Williams, 2004), and its use is recommended in Muthén et al. (2016).

There was only one change to the input statement in Appendix A when the predictor was continuous (reflecting that estimates are to be obtained for a one standard deviation change in that predictor). The standard deviation and mean of the continuous predictor were computed before running the mediation analysis, to obtain the estimates and OR for a one standard deviation change in the predictor. These two statistics were included, in that order (and separated by a space, in parenthesis at the end of the line in the option for MODEL INDIRECT). The mean of male peer heavy drinking was 2.27, and 3.17 was one standard deviation above the mean. Therefore, the option in MODEL INDIRECT was modified to include these descriptive statistics (see note at bottom of Appendix A).

2.1.4. Results

2.1.4.1. Missing data

There were no missing data on the alcohol expectancies measure, AUD diagnoses, or male peers’ heavy drinking. However, there were missing data on three of the four predictors—female partners’ heavy drinking (n = 3), male peers’ social influence (n = 20), and female partners’ social influence (n = 9)—which was handled by Mplus using listwise deletion (the Mplus default for missing independent variables). Thus, variations in respective effects across measures could be due in part to the slight differences in samples.

2.1.4.2. Model results for indirect and direct effects

2.1.4.2.1. Models with Predictor (influence) x Mediator (alcohol expectancies) interactions

Table 1 reports the results for each mediation analysis that included the Influence x Expectancies interactions. The top half of the table reports the results from models for a binary predictor (the dichotomized alcohol influence measures), and the bottom half reports the corresponding results from models that used the inherently continuous influence predictors.

Table 1.

Model Results from Mediation Analysis with Mediator by Influence (Predictor) Interactions

Models with Binary Influence x Expectancies Interactions
	AUD on Alcex		AUD on Influence		AUD on Mx		Alcex on Influence
Influence	Estimate	Est./SE	Estimate	Est./SE	Estimate	Est./SE	Estimate	Est./SE
Mphed	0.937	2.34*	0.772	2.01*	−0.092	−0.14	0.391	4.43***
Fphed	1.034	2.25*	1.060	2.77 **	−0.062	−0.10	0.165	1.76
Mpinf	1.141	2.36*	1.023	2.47*	−0.509	−0.74	0.255	2.55*
Fpinf	1.137	2.59**	1.595	3.69***	−0.389	−0.54	0.157	1.66
Models with Continuous Influence x Expectancies Interactions
	AUD on Alcex		AUD on Influence		AUD on Mx		Alcex on Influence
Influence	Estimate	Est./SE	Estimate	Est./SE	Estimate	Est./SE	Estimate	Est./SE
Mphed	0.677	0.82	0.732	3.37***	0.043	0.12	0.234	4.58***
Fphed	1.269	1.24	0.592	1.89	−0.183	−0.35	0.154	2.24*
Mpinf	0.969	2.89**	0.568	2.03*	0.017	0.04	0.125	1.78
Fpinf	0.842	2.49*	1.394	3.98***	0.296	0.48	0.217	3.48***

Note. AUD = Alcohol Use Disorder, Alcex = Alcohol Expectancies, AUD on Mx = Influence x Alcohol Expectancies interaction. Ext./SE = Estimate/Standard Error = z, Mphed = Male peers’ heavy drinking, Fphed = Female partner’s heaving drinking, Mpinf = Male peers’ social influence, Fpinf = Female partners’ social influence.

^*p < .05

^**p < .01

^***p < .001.

Table 1 reports the estimates for: (a) the effect of alcohol expectancies on AUD (y on m), (b) the effect of each influence measure on AUD (y on x), (c) the effect of each binary influence measure on the mediator (m on x), and (d) the Predictor (x) by Mediator (m) interaction effect (y on mx). The interaction effects were all very small and not statistically significant. We acknowledge that testing the additive interaction may differ from testing the interaction and estimate effects at specific values of the predictor variables (Ai & Norton, 2003; VanderWeele, 2015), such as at different values of the influence measure.

Note that the ORs for the direct and indirect effects obtained under the rare outcomes assumption were not included in the table because that assumption was not met for the AUD outcome.

2.1.4.2.3. Models without Predictor (influence) x Mediator (alcohol expectancies) interactions

Table 2 reports the results for each mediation analysis that did not include the Influence x Expectancies interaction. As in Table 1, the top half of Table 2 reports the results from models in which the alcohol influence measures were binary, and the bottom half reports the corresponding results from models when the continuous influence measures had not been dichotomized. (The estimates included in this table were specified as parameters in the Monte Carlo analysis in Study 2.)

Table 2.

Model Results from Mediation Analysis without Mediator x Influence (Predictor) Interactions

Models without Binary Influence by Expectancies Interactions
	AUD on Alcex		AUD on Influence		Alcex on Influence
Influence	Estimate	Est./SE	Estimate	Est./SE	Estimate	Est./SE
Mphed	0.897	2.86**	0.778	2.09*	0.391	4.43***
Fphed	1.007	3.37***	1.067	2.84**	0.165	1.76
Mpinf	0.932	2.85**	1.068	2.67**	0.255	2.55*
Fpinf	1.010	3.09**	1.626	3.91***	0.157	1.66
Models without Continuous Influence by Expectancies Interactions
	AUD on Alcex		AUD on Influence		Alcex on Influence
Influence	Estimate	Est./SE	Estimate	Est./SE	Estimate	Est./SE
Mphed	0.771	2.30*	0.722	3.52***	0.234	4.58***
Fphed	0.963	3.22***	0.614	2.05*	0.154	2.24*
Mpinf	0.967	2.96**	0.564	2.17*	0.125	1.78
Fpinf	0.786	2.36*	1.334	4.04***	0.217	3.48***

Note. AUD = Alcohol Use Disorder, Alcex = Alcohol Expectancies, AUD on Mx = Influence x Mediator interaction. Ext./SE = Estimate/Standard Error = z, Mphed = Male peers’ heavy drinking, Fphed = Female partner’s heaving drinking, Mpinf = Male peers’ social influence, Fpinf = Female partners’ social influence.

^*p < .05

^**p < .01

^***p < .001.

As shown in Table 2, there were statistically significant and positive effects of both pro-alcohol social influences and alcohol expectancies on AUDs across all four different measures of influence, irrespective of whether the influence measures were used in their natural continuous form or dichotomized for didactic purposes. However, the effects of the alcohol influence measures on alcohol expectancies were more consistent when those measures had not been dichotomized, perhaps because of the lost information through dichotomization (Cohen, 1983). Five of the eight estimates (regression coefficients) for the effects of influence on expectancies were statistically significant and the other three estimates showed trends (ps < .10).

2.1.5. Estimates and confidence intervals for mediation analysis effects and ORs based on counterfactuals

2.1.5.1. Binary predictors of alcohol influence

The top half of Table 3 reports the results from the mediation analysis with binary influence predictors. The results on the left side of the table (“Effects of Influence on AUDs”) are the effects in the probability metric (e.g., .2 means that the probability of y = 1 increases by .2 when x = 1 compared to x = 0). The results on the right side of the table (“Odds Ratios for AUDs”) are the ORs for the indirect, direct, and total effects of alcohol influence on AUDs. (Note that the ORs for non-rare outcomes included in Table 3 would not have been reported had the AUD been a rare outcome, as the correct ORs would then have been the ORs obtained under the rare outcome assumption obtained with MODEL CONSTRAINT.)

Table 3.

Bootstrap Confidence Intervals for Indirect, Direct, and Total Effects of Alcohol Influence on Alcohol Use Disorders Based on Counterfactuals

Binary Measures of Alcohol Influence
Influence	Effect of Influence on AUD	Odds Ratio for AUD
	Estimate (CI)	Estimate(CI)
Male Peers’ Heavy Drinking
Tot natural IE	.057 (.018, .106)	1.397 (1.130, 1.802)
Pure natural DE	.160 (.019, .299)	2.084 (1.095, 4.376)
Total effect	.217 (.084, .346)	2.912 (1.512, 6.109)
Female Partners’ Heavy Drinking
Tot natural IE	.025 (−.002, .068)	1.169 (.985, 1.501)
Pure natural DE	.208 (.072, .339)	2.689 (1.396, 5.588)
Total effect	.233 (.100, .362)	3.144 (1.628, 6.679)
Male Peers’ Social Influence
Tot natural IE	.035 (.006, .079)	1.253 (1.045, 1.613)
Pure natural DE	.209 (.067, .349)	2.719 (1.370, 5.939)
Total effect	.244 (.108, .380)	3.407 (1.707, 7.660)
Female Partners’ Social Influence
Tot natural IE	.019 (−.004, .050)	1.162 (.970, 1.449)
Pure natural DE	.303 (.173, .430)	4.529 (2.329, 10.287)
Total effect	.322 (.193, .447)	5.264 (2.639, 12.397)
Continuous Measures of Alcohol Influence
Influence Measure	Effect of Influence on AUD	Odds Ratio for AUD
	Estimate (CI)	Estimate(CI)
Male Peers’ Heavy Drinking
Tot natural IE	.021 (.004, .042)	1.171 (1.033, 1.349)
Pure natural DE	.109 (.059, .158)	1.873 (1.366, 2.912)
Total effect	.130 (.083, .176)	2.193 (1.583, 3.237)
Female Partners’ Heavy Drinking
Tot natural IE	.016 (.002, .038)	1.100 (1.012, 1.254)
Pure natural DE	.075 (.012, .137)	1.480 (1.056, 2.276)
Total effect	.090 (.032, .152)	1.627 (1.173, 2.522)
Male Peers’ Social Influence
Tot natural IE	.014 (−.001, .035)	1.093 (.992, 1.241)
Pure natural DE	.077 (.016, .136)	1.510 (1.087, 2.322)
Total effect	.091 (.032, .149)	1.650 (1.178, 2.560)
Female Partners’ Social Influence
Tot natural IE	.013 (.002, .029)	1.131 (1.027, 1.287)
Pure natural DE	.143 (.093, .194)	2.596 (1.765, 4.576)
Total effect	.156 (.107, .206)	2.935 (2.001, 5.126)

Note. CI = 95% confidence interval; Tot natural IE = Total natural indirect effect; Pure natural DE = Pure natural direct effect.

The effects and ORs not reported in parentheses are the estimates, and the 95% CIs for these estimates are in parentheses. The significance of each indirect, direct, and total effect in Table 3 is determined from the bootstrap CIs for the estimate for that effect. For the effects on the left side of the table (in the probability metric), the effects are statistically significant at p < .05 when the reported 95% CI does not include a value of 0. The respective ORs on the right side are statistically significant when the CI does not include a value of 1. (There were no instances where a probability-based effect was significant based on its CIs and the respective OR was not significant based on the CI for the OR, or vice versa.)

The total effects of the binary influence variables on AUDs (ORs = 2.87–5.26, mdn = 3.26) were statistically significant for all the influence predictors. The direct effects (ORs = 2.04–4.53, mdn = 2.69, rows labeled Pure natural DE) were also all statistically significant, and the ORs were only slightly smaller for the direct than for the respective total effects. The small differences between the total and direct effects indicated that the effects of influences on AUDs were mostly direct. Indeed, the indirect effects were small (ORs = 1.16–1.41, mdn = 1.22, see rows labeled Tot natural IE in Mplus output files and in Table 3), but nonetheless were all either statistically significant or narrowly missed significance (ps < .10).

2.1.5.2. Continuous predictors of alcohol influence

The bottom half of Table 3 contains the results when the continuous influence variables had not been dichotomized. The total effects (ORs = 1.63–3.14, mdn = 1.93) of influences on AUDs were not much different than direct effects (ORs = 1.48–2.73, mdn = 1.70) but the indirect effects (ORs = 1.09–1.18, mdn = 1.12) were negligible in magnitude.

3. Study 2

3.1. Method

3.1.1. Monte Carlo specifications for mediation models

Muthén and Muthén (2002) described the use of Mplus to conduct a Monte Carlo analysis to evaluate biases in point estimates and SEs and CIs of the parameter estimates from different statistical models. A Monte Carlo study for mediation analysis models requires specification of parameters for: (a) the effect of the mediator on the outcome (y on m), (b) the effect of the predictor on the outcome (y on x), (c) the effect of the predictor on the mediator (m on x), (d) the residual variance of the mediator, and (e) sample size (Muthén, et al., 2016).

In addition, Mplus input for simulations under the rare outcomes assumption (which was not warranted in the illustrative data from Study 1) requires specifications of the ORs for the direct and indirect effects, and creation of new parameters using the MODEL CONSTRAINT command. The parameter values specified in the current simulations were obtained from the results from the analysis with each social influence predictor from Study 1.

Appendix B presents one of the four Monte Carlo model input statements for a logit model (the default for mediation analysis with a binary outcome when no link is specified) that can be used as a template for other work, and thus includes the specifications of the ORs in MODEL CONSTRAINT that would normally beincluded only when the outcome is rare. Two modifications to the input statement in Appendix B were required to examine a model that includes a continuous predictor. First, the cutpoints specification, which applies only to binary x variables, is not included in the MONTECARLO command when x is continuous. Second, the model indirect statement uses the same modification (i.e., specifying x values of one standard deviation above the mean and the mean of x) used in the illustrative analysis in Study 1.

3.1.2. Examination of bias in the point estimate, SE, and CI for the indirect effect

Relative (percent) bias in the point estimate and SE for each type of indirect effect (probability metric vs. OR) was examined following the steps outlined in Muthén and Muthén (2002), which also provided guidelines for ignorable bias in Monte Carlo analysis: 10% or less for point estimates and 5% or less for SEs. Mplus evaluates CI accuracy with coverage: the proportion of the replications for which the CI contains the true parameter value (Muthén & Muthén, 2002). Perfect coverage for the 95% CI is .95, and acceptable values are in the .91 to .98 range. Bias in each CI can thus be estimated by subtracting the coverage coefficient from .95 (Feingold, in press), with acceptable values in the −.03 to .04 range.

3.2. Results

3.2.1. Description of Table 4

Table 4.

Monte Carlo Analysis of Indirect Effects and ORs in a Mediation Analysis with a Binary Outcome

	Estimates			SEOR			Bias in Estimates, SEOR and CI
Effects of X on Y with Binary Predictors
Measure	N	Pop OR	Avg	SD	Avg	Coverage	Est %	SEOR%	CI
Mphed	181	0.068	0.0688	.0279	.0279	.925	1.18	.00	.025
Fphed	178	0.030	0.0297	.0196	.0202	.934	−1.00	3.06	.016
Mpinf	161	0.042	0.0425	.0231	.0229	.930	1.19	− 87	.020
Fpinf	172	0.021	0.0216	.0161	.0165	.942	.2.86	2.48	.008
Effects of X on Y with Continuous Predictors
Measure	N	Pop OR	Avg	SD	Avg	Coverage	Est %	SEOR%	CI
Mphed	181	.010	.0104	.0069	.0070	.926	4.00	1.45	.024
Fphed	178	.013	.0126	.0064	.0062	.928	−3.08	−3.13	.022
Mpinf	161	.021	.0207	.0103	.0104	.931	−1.43	.97	.019
Fpinf	172	.023	.0233	.0109	.0109	.935	1.30	.00	.015
Odds Ratios (ORs) with Binary Predictors
Measure	N	Pop OR	Avg	SD	Avg	Coverage	Est %	SEOR%	CI
Mphed	181	1.394	1.4116	.1898	.1948	.937	1.62	2.63	.013
Fphed	178	1.168	1.1751	.1215	.1289	.936	.61	6.09	.014
Mpinf	161	1.251	1.2653	.1531	1599	.939	1.14	4.44	.011
Fpinf	172	1.162	1.1724	.1286	.1380	.947	.90	7.33	.003
Odds Ratios (ORs) with Continuous Predictors
Measure	N	Pop OR	Avg	SD	Avg	Coverage	Est %	SEOR%	CI
Mphed	181	1.174	1.1812	.0842	.0867	.959	.61	2.97	.009
Fphed	178	1.101	1.1037	.0458	.0478	.943	.25	4.37	.007
Mpinf	161	1.091	1.0928	.0478	.0495	.936	.16	3.56	.014
Fpinf	172	1.129	1.1315	.0621	.0642	.953	.22	3.38	.003

Note. Mphed = Male peers’ heavy drinking; Fphed = Female partners’ heavy drinking; Mpinf = Male peers’ social influence; Fpinf = Female peers’ social influence; Pop OR = population value for OR for the indirect effect; Estimates Avg = average estimate (OR) across replications; Estimates SD = standard deviation of ORs across replications; SE_OR Avg = average SE_OR across replications; Coverage = 95% coverage for OR.

The simulation results examining the indirect effects are reported in four sections in Table 4: (a) effects in the probability metric from analyses with binary predictors (dichotomized influence measures), (b) effects in the probability metric from analyses with continuous predictors (continuous influence measures), (c) ORs with a binary predictors, and (d) ORs with continuous predictors.

The first column reports the influence measure for which results reported in that row were obtained. The second column reports the number of participants for whom that measure was available. Column 3 reports the population values, which are essentially the same (given rounding errors) as the values reported for the indirect effect in Study 1.

Column 4 (Estimates Avg.) reports the mean of the 1,000 raw probability metric effects and ORs for each analysis. The differences between the means in this column and the respective population values (reported in Column 3) indicate the absolute biases in the estimates. Relative bias was calculated by dividing the absolute bias by the respective parameter (Column 3) and multiplying by 100 to convert the proportion to a percent (Column 8).

Column 5 reports the Estimates SD, which is the standard deviation of the 1,000 estimates over replications, Column 6 (SE Avgs.) report the means of the 1,000 bootstrap SEs. Absolute bias in SE was calculated by subtracting the statistics in Column 6 from the respective Estimates SDs in Column 5. These absolute biases were divided by the Estimates SD and multiplied by 100 to determine the percent bias in SE for each method (see Column 9). The coverage for the parameter estimates is reported in Column 7, and CI biases in the last column.

3.2.2. Bias in estimates, SEs, and CIs for indirect effects

As shown in Table 4, biases in the ORs (always under 2%) were typically smaller than biases in the probability metrics (always 4% or less), but all were well below the 10% threshold for acceptable bias in parameter estimates (Muthén & Muthén, 2002). Biases in the SEs for the ORs (all under 8%) were typically larger than biases in the SEs for the probability-based effects (all 4% or less). However, the biases in the SEs for the ORs were generally below the 5% threshold for ignorable bias in SEs, with the exceptions of the ORs associated with the two dichotomized partner influence measures. These results indicate the sample size used in Study 1 was too small to obtain accurate ORs when the predictor was binary.

4. Discussion

A typical application of mediation analysis would examine whether the effect of a risk factor (e.g., alcohol-abusing friends) on problematical behavior (e.g., alcohol misuse) is mediated by an intervening mechanism (e.g., alcohol expectancies) using a continuous outcome (e.g., alcohol consumption). Because many key substance abuse outcomes (e.g., an AUD) are binary, this tutorial demonstrated methods for obtaining direct and indirect effects on binary outcomes, and illustrated the use of Monte Carlo analysis to evaluate bias in the indirect effects from these models.

4.1. Study 1: Illustrative analysis

The illustrative study examined whether the effects of influences on alcohol misuse were mediated by alcohol expectancies. The largest effects observed were the total effects of the social influence variables on AUD diagnosis, all of which were statistically significant and of notable magnitude. The direct effects of alcohol influence on AUD were also all consistently statistically significant, with the magnitude of effects (e.g., ORs) only slightly smaller than for the respective total effects. By comparison, the indirect effects sometimes failed to attain statistical significance (mainly when the alcohol expectancies mediator was not significantly related to the AUD outcome), suggesting the direct effects may be more potent than the indirect effects (and thus easier to detect). However, this study may have been underpowered to detect indirect effects on all of the influence predictors, particularly for the influences of romantic partners. Although inadequate power could explain the failure of some indirect effects to attain statistically significance, low power would not explain the small effect sizes, including ORs that were only slightly different from 1.00.

Several methodological issues would need to be addressed if this were a substantive study instead of a tutorial, including discussion of potential confounders of the observed indirect (mediated) effect. Because the mediated effects in Study 1 are small, confounders could provide a plausible alternative hypothesis for the results. Additional variables that may serve as confounders of mediation relations would be included as additional predictors in the mediation analysis. A primary confounder to consider is what could be called a redundant measure effect (Brewer, Campbell, & Crano, 1970; MacKinnon & Gonzalez, 2018; MacKinnon, 2008). That is, researchers could say that x, m, and y in this study measured the same thing (alcohol exposure), in which case a model would be tested with a latent variable that is a cause of all three variables, with no posited mediation. Note that this possibility could apply to other studies in the addictions literature examining such hypotheses. However, such a model would not address the mechanism by which social influences affect development of an AUD. Thus, researchers would need to defend their choice of model selection to test the hypothesis that social influences affect AUD, even if only slightly, via alcohol expectancies.

4.2. Study 2: Monte Carlo studies of bias in or for indirect effect

The second study presented an Mplus template for conducting Monte Carlo studies to examine biases in the point estimates, SEs, and CIs for the indirect effects in the probability and OR metrics, and applied it to findings from Study 1. This Monte Carlo study found that bias in the indirect effects, including the SE and CI for those effects, was generally in the acceptable range, indicating the sample size used in Study 1 was adequate for obtaining valid estimates. However, there was unacceptable bias in the SEs for some of the ORs, particularly for the indirect effects of the binary partner influence measures. Thus, for such models, findings might better be communicated only via the probability metric, where SE bias was consistently in the ignorable range. (Because the binary predictors were the continuous measures that had been dichotomized solely for didactic purposes, the unduly biased ORs obtained with the binary measures would not have been reported in a substantive study.)

4.3. Limitations and directions for future research

The illustrative analysis of mediation analysis with a binary outcome extended the study by Feingold et al. (2015), by using the same samples and measures, which found statistically significant total effects of the alcohol influence measures and alcohol expectancies on an AUD. In the previous study, each of the alcohol influence measures and alcohol expectancies were operationalized in two different ways: a single score from the wave in the longitudinal study when first administered (T1) and as an average of each measure across all the waves in which that measure was administered. In a substantive study, it would have been preferable to use the T1 measures of both influence and expectancies in a mediation analysis because mediation assumes both the predictor and the mediator occur before the outcome. However, the illustrative study used the time-averaged measures for each predictor and the mediator as a practical matter because the previous analyses had found the time-averaged scores for both social influence and alcohol expectancies to be stronger and more consistent predictors of an AUD than were the respective T1 scores. These larger effects meant that direct and indirect effects of social influence would more likely to be manifested by using the time-averaged measures.

Although the mediation models with binary outcomes examined in these two studies were simple, mediation analyses found in the substance abuse literature using continuous outcomes have often been complex. For example, the indirect effects of alcohol influence may vary by gender (which could not be examined in the all-male sample used in Study 1). The ORs for the indirect effect would then differ by gender and separate indirect effects (e.g., ORs) would be needed for men and women. This type of mediation analysis is known as moderation of a mediated effect (MacKinnon, 2008). Another form of complex mediation involves the inclusion of multiple mediators in the analysis. Future work could make advancements in the use of these more complex mediation models, already in use with continuous outcomes, in models for binary outcomes.

More complex models may also add control variables—called “c” in Muthén et al. (2016)—so that effects are conditional on these variables (see Chapter 8 in Muthén et al. for an example involving a vaccination study). Future Monte Carlo research on these models with a binary outcome could address issues about sample size needed to find (a) a significant indirect effect and (b) a significant moderation effect. Such analyses would address whether Study 1 was underpowered to obtain indirect effects.

In addition, the interpretation of the OR for the indirect effect depends on whether the outcome is rare. For rare outcomes, the OR is a good measure of relative risk (proportion in exposed group divided by proportion in unexposed group), or in some cases the absolute difference in these proportions—the ideal measure of risk in one group compared to another (Cummings, 2009). The ORs for non-rare common outcomes are not good measures of relative risk, and loglinear models are more appropriate (Valeri & VanderWeele, 2013). The loglinear model produces the rate ratio effect size measure, which might be illustrated and validated (with Monte Carlo analysis) as a third effect metric to be considered in conjunction with the probably metrics and ORs for the indirect effects on non-rare outcomes. Yet, there are valid reasons for using the OR when outcomes are not rare. In particular, reversing the predictor and outcome gives the value of 1/OR. The OR is also clearly linked with estimates from logistic regression.

5. Summary and Conclusions

This article first conducted a mediation analysis to obtain the direct and indirect effects of binary alcohol influence measure on an AUD outcome for a long-term study of AUD. Next, mediation analyses examined the same effects but with continuous counterparts of the alcohol influence measures. Both analyses found consistently statistically significant direct and total effects--but inconsistent indirect effects (particularly regarding partner influences)-- of four different measures of pro-alcohol influence on AUD.

The primary original contribution in the article was made in Study 2, which presented new Mplus input for conducting a Monte Carlo study. The Monte Carlo study evaluated bias in the indirect effects--in both the probability metric and respective ORs--obtained in the models used in Study 1, with the ORs based on common outcomes (as appropriate given the AUD distribution). This study found ignorable biases in both effect metrics given the sample size used in Study 1, with the exception of some of the ORs obtained with a binary predictor.

The methods described and applied in these studies have important value to researchers of addictive behaviors, as binary measures are commonly used as outcomes in the substance abuse literature and mediation analysis is typically used to examine the causal mechanisms by which treatments and risk factors affect such outcomes.

Appendix A. Mplus Input Statements for the Three Mediation Models with Male Peers’ Heavy Drinking as the Predictor, Alcohol Expectancies as the Mediator, and AUD as the Outcome

Logistic Regression Model with and without Rare Outcomes Assumption

TITLE: alcohol expectancies as mediator between mphed and AUD.

DATA: FILE IS mediation2.csv;

VARIABLE:  NAMES ARE aud alcex mphed fphed mpinf fpinf;
           USEVARIABLES =aud alcex mphed;
           MISSING ARE ALL (−999);
           CATEGORICAL = aud;

ANALYSIS:  ESTIMATOR = ML;
           LINK = LOGIT;
           BOOTSTRAP = 10000;

MODEL:     aud ON alcex (beta1)
           mphed (beta2);
           alcex ON mphed (gamma);

MODEL INDIRECT:
             uad IND alcex mphed;

MODEL CONSTRAINT:
             NEW (indirect direct);
             indirect = EXP(beta1*gamma);
             direct = EXP(beta2);

OUTPUT:    TECH1 TECH8 SAMPSTAT
           CINTERVAL(BOOTSTRAP);

PLOT:      TYPE=PLOT3;

Influence x Alcohol Expectancies Interaction Model

TITLE: alcohol expectancies as mediator between mphed and AUD mediation interaction.

DATA: FILE IS mediation2.csv;

VARIABLE:  NAMES ARE aud alcex mphed fphed mpinf fpinf;
           USEVARIABLES =aud alcex mphed mx;
           MISSING ARE ALL (−999);
           CATEGORICAL = aud;

DEFINE:    mx = alcex*mphed;

ANALYSIS:  ESTIMATOR = ML;
           BOOTSTRAP = 10000;

MODEL:     aud ON alcex mphed mx;
           alcex ON mphed;

MODEL INDIRECT:
          aud MOD alcex mx mphed;

OUTPUT:   TECH1 TECH8 SAMPSTAT
          CINTERVAL(BOOTSTRAP);

PLOT:     TYPE=PLOT3;

Note. When mphed was continuous, the option in MODEL DIRECT in each analysis was modified to:

MODEL INDIRECT:
          aud IND alcex mphed (3.17 2.27.

Appendix B. Mplus Input Statement for Monte Carlo Analysis of Mediation Analysis Using a Logistic Model with a Binary Outcome for Illustrative Study

Title:
    Monte Carlo for mediation with binary Y(AUD),
    acontinuous M (expectancies), and binary X
    (Mphed, split 50–50)

Montecarlo:
    names = x m y;
    nobs = 181;
    nreps = 1000;
    cutpoints = x(0);
    generate = y(1);
    categorical = y;

Model Population:
    [x@0]; x@1;
    y on m*.897 x*.778;
    m on x*.391;
    m*.356;
    [y$1*0];

Analysis:
    estimator = ml;
    bootstrap=500;

Model:
    y on m*.897 (beta1)
    x*.778 (beta2);
    m on x*.391 (gamma);
    m*.356;
    [y$1*0];

! VanderWeele & Vansteelandt 2010 approx assuming rare Y:
Model Constraint:
    new(indirect*1.42 direct*2.178);
    indirect = exp(beta1*gamma); ! see RMA book eqn (8.39) with b3=0
    direct = exp(beta2); ! see RMA book eqn (8.41) with b3=0

! correct counterfactual effects not assuming rare Y,
! in probability and odds ratio scales; see RMA book
! pages 313 and 315:
Model Indirect:
    y IND m x.

Note. AUD = alcohol use disorder, Mphed = male peers’ heavy drinking, RMA = “Regression and Mediation Analysis Using Mplus” text (Muthén et al., 2016).