Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2023 Mar 25.
Published in final edited form as: Struct Equ Modeling. 2022 Mar 25;29(6):944–952. doi: 10.1080/10705511.2022.2045203

Teacher’s Corner: An R Shiny App for Sensitivity Analysis for Latent Growth Curve Mediation

Eric S Kruger 1, Davood Tofighi 1, Yu-Yu Hsiao 1, David P MacKinnon 2, M Lee Van Horn 1, Katie Witkiewitz 1
PMCID: PMC9683348  NIHMSID: NIHMS1848235  PMID: 36439330

Abstract

Mechanisms of behavior change are the processes through which interventions are hypothesized to cause changes in outcomes. Latent growth curve mediation models (LGCMM) are recommended for investigating the mechanisms of behavior change because LGCMM models establish temporal precedence of change from the mediator to the outcome variable. The Correlated Augmented Mediation Sensitivity Analyses (CAMSA) App implements sensitivity analysis for LGCMM models to evaluate if a mediating path (mechanism) is robust to potential confounding variables. The CAMSA approach is described and applied to simulated data, and data from a research study exploring a mechanism of change in the treatment of substance use disorder.

Keywords: behavior change, sensitivity analyses, mediation, latent growth curve mediation modeling, confounding

A mechanism of behavior change is defined as the process or processes by which behavior change occurs and, in the context of a specific intervention, it is often defined as the mediational process(es) through which an intervention is effective (Kazdin, 2007). Studying mechanisms of behavior change is critical for the ongoing development and evaluation of behavioral and educational interventions. In substance use disorder treatment research, for example, researchers have considered many different statistical mediators, defined as intervening variables that explain an association between a randomized intervention (independent variable) and the outcome (dependent variables), to support specific mechanisms of change (Longabaugh et al., 2005). Yet, a significant mediating (indirect) relationship might not accurately reflect the process(es) by which change occurs because establishing statistical mediation does not necessarily establish a mechanism of behavior change (Kazdin, 2007), which requires more comprehensive information including robustness of effects to confounding (MacKinnon & Pirlott, 2015).

Based on Hill’s (1971) considerations for causal inference, statistical mediation is an important interim step in establishing a mechanism of behavior change. Statistical mediation includes the joint test of significance of the independent variable (e.g., randomized treatment condition) in predicting a mediating variable or process (a-path), and the significance of the mediator in predicting the outcome (b-path). There are several methods for testing statistical mediation, with the product of coefficients (a × b paths) being the most common (see MacKinnon, 2008).

However, statistical mediation tests may be biased when a confounding variable is omitted from the model (Holland, 1988; Tofighi & Kelley, 2016), see Figure 1. Stated differently, the confounder variable provides an alternative explanation for the mechanism through which the intervention is effective (Valeri & Vanderweele, 2013; VanderWeele, 2010). If the confounding variables are measured, testing for omitted confounders may be possible by evaluating models that include the confounding variables. However, as it is often the case, possible confounders may not be measured. In this case, it is recommended that researchers perform sensitivity analyses to determine the robustness of the results to potential confounding variables (VanderWeele, 2010). Sensitivity analyses test the mediated effect across a broad range of potential confounding effects and if the mediated effect is relatively unchanged, then the mediating path is robust to potential confounding (MacKinnon, 2008).

Figure 1.

Figure 1.

Open in a new tab

Potential Mediation Confounding

Recent guidelines for reporting results from mediation analyses recommend that sensitivity analyses be used to test for potential effects of unmeasured confounding (Lee et al., 2021). Development of sensitivity analyses for confounders on mediated effects is an active area of research (Harring et al., 2017; Valente et al., 2017; VanderWeele, 2010). A method developed by Mauro (1990) adjusted the b-path for confounders of different effect size and was adapted for mediation (Cox et al., 2013). Imai and colleagues (2010) developed a method for investigating unmeasured confounders by varying the correlation of the residuals in the mediator and outcome variables for a single mediator model. Recently, researchers have also extended mediational sensitivity analysis to a multilevel single mediator model (Talloen et al., 2016; Tofighi & Kelly, 2016).

Researchers, teachers, clinicians, and policymakers are frequently interested in whether changes in the mechanism over time mediate the effects of an intervention, because of this, latent growth curve mediation models (LGCMM) are of considerable interest because they evaluate change in a mediator over time. Prior work on sensitivity methods for mediation applied only to single mediator models, but LGCMM have two mediators (a mediator for the latent intercept and slope). In this context, and in other work by our team, we have developed a sensitivity analysis technique, termed Correlated Augmented Model Sensitivity Analysis (CAMSA), to examine the impact of unmeasured confounding in LGCMMs (Hsiao et al., 2019; Tofighi, 2021; Tofighi et al., 2019). Our group has used CAMSA to examine the sensitivity of mindfulness mediating the effects of a mindfulness-based treatment for substance use disorder on substance craving (Hsiao et al., 2019).

CAMSA uses prespecified correlations between the residuals associated with the endogenous variables to model various degrees of hypothesized omitted confounding.1 A limitation to implementing CAMSA is that the method requires considerable statistical programming skills. Therefore, our team developed a web app that implements CAMSA for LGCMMs and the purpose of this manuscript is to describe a step-by-step tutorial of the CAMSA app. The CAMSA app is an important contribution because it provides a convenient application for making implementation of sensitivity analyses of LGCMMs user friendly and accessible to many researchers without extensive statistical programming skills.

Correlated Augmented Mediation Analyses (CAMSA)

CAMSA is a technique that uses the existing mediation model to conduct a sensitivity analysis within a structural equation modeling (SEM) framework and has been described in detail within Tofighi et al. (2019). The aim of CAMSA is to model the biasing impacts of potential omitted confounders by specifying hypothetical correlations between the residuals associated with the endogenous variables, termed confounder correlations. CAMSA can be extended to other models and the description of the methodology in this section is of the general CAMSA approach.

The first step of CAMSA augments the hypothesized model with confounder correlations to form a correlated augmented model. In the second step, the correlated augmented model is estimated iteratively using combinations of confounder correlations—values that are pre-specified over a given range (e.g., r values from −0.5 to 0.5). At each iteration, point estimates and confidence intervals of the mediated effects are estimated as a function of confounder correlations values. Finally, CAMSA sensitivity analyses are summarized and examined through sensitivity plots or tables of the mediated effects at different levels of confounder correlations.

To determine robustness of the mediated effect across levels of confounder correlations, the researcher examines the confidence interval (CI) of the original mediated effect (without confounder correlations) as compared to the CAMSA estimated mediated effect (with confounder correlations) across the range of confounder correlations. In our examples, we have used a 95% CI (α = .05) as a default. Specifically, two features are examined with respect to the CAMSA estimated CI. The first feature to consider is whether the CI of the observed mediated effect is different from the CASMA estimated mediated effect at varying levels of confounder correlations. For example, if the CI of the original observed mediated effect does not overlap with zero, but a portion of the CAMSA estimated mediated effect CI now overlaps with zero, this would indicate that the statistical conclusion of the original observed mediation test is not robust at that level of confounding. The second feature to consider is whether the CAMSA estimated CI no longer overlaps with the original observed mediated effect, which indicates the CAMSA estimated mediated effect is different from the observed mediated effect at that level of confounding. To provide an in-depth overview of the CAMSA method and interpretation of sensitivity plots we present a tutorial of a CAMSA web application and provide results from both simulated and empirical datasets.

CAMSA App Tutorial

We have developed the CAMSA app for LGCMMs because of the substantive interest in examining changes in a mediator over time. LGCMMs have two mediated effects that are estimated by the CAMSA app, the indirect (mediation) paths through the latent intercept and the latent slope. CAMSA uses an SEM framework to compute the omitted confounder effect on the indirect effects of the intercept and slope. The CAMSA model proposed in Tofighi et al. (2019) utilized Mplus (Muthen & Muthen, 1998–2021) to demonstrate the method. In this paper, we developed a user-friendly Shiny App which incorporated the R program (v.4.0.2; R Core Team, 2020) to make the analysis accessible to users. CAMSA uses lavaan (Rossel, 2012) for LGCMM estimation and RMediation (Tofighi & MacKinnon, 2011) for estimation of the indirect (mediated) point estimate and Monte Carlo estimation of the confidence interval. The CAMSA app is located at (https://casaa-unm.shinyapps.io/CAMSA/). The following is a tutorial of the app using a simulated dataset that is provided within the app. The simulated data is based on substantive research that has found naltrexone, an opioid antagonist medication approved for the treatment of alcohol use disorder, may be effective in reducing frequency of drinking via reductions in craving for alcohol (Witkiewitz et al., 2019). The hypothesis for the simulated data was that being randomized to treatment with naltrexone versus a placebo medication (independent variable) affects percent drinking days (outcome), and this effect is mediated by changes in alcohol craving (mediator) which were measured at several timepoints during the study.

Step 1 Upload Data

Users can upload their own data by clicking on the button ‘Data’, and then indicating the location of the file by clicking on the ‘Browse’ button (Supplementary Figure 1). Data should be in a .csv format with missing values specified as NA. To follow along with the remaining steps in this tutorial, click on the ‘Load Example’ button and load the tutorial example data. Users may also inspect the raw data and output generated in this example by going to the CAMSA app Open Science Framework (OSF) project folder (https://osf.io/awzd2/).

Step 2 Specify Variables

After uploading data, users are required to specify predictor (independent variable, “X”), outcome (“Y”) and mediators (“M1…n”). Drop down menus allow for the specification of the predictor and outcome variables. To select the mediator users are required to specify the time coefficients of the LGCMM in the ‘Time Coef’ data entry table (Supplementary Figure 2). For a linear model, time coefficients are based on the interval duration between timepoints. In the example, the independent variable (X) represents individuals randomized to naltrexone treatment or control, the outcome (Y) is percent drinking days, and the mediator (M), craving, was measured at baseline (M_t1), 1-month mid-treatment (M_t2), end-of-treatment after 3-months (M_t4), 2-month follow-up (M_t6), 4-month follow-up (M_t8), 6-month follow-up (M_t10), and 8-month follow-up (M_t12); the corresponding time coefficients are −3, −2, 0, 2, 4, 6, and 8; respectively. Note, the meaning of the intercept is specified by the location of the zero in the ‘Time Coef’ table. In the simulated example we specified the ‘Time Coef’ table such that the intercept corresponded to the level of the mediator at the end-of-treatment (M_t4), which allowed us to model the effect of treatment on end-of-treatment levels of the mediator (intercept), as well as the effect of treatment on change in the mediator over time (slope).

To specify a nonlinear LGCMM, nonlinear time-coefficients (e.g., quadratic) can be used when entering the ‘Time Coef’ values. Any additional variables that are not included in the CAMSA model should remain empty in the ‘Time Coef’ data entry table. Additionally, after loading the data, the first 20 rows of the data and summary statistics for each variable are provided. A warning is provided if any of the variables in the dataset demonstrate a skew ± 2 and kurtosis ± 7 which would indicate the possibility of a violation of multivariate normality (Curran et al., 1996). Because CAMSA assumes multivariate normality with respect to exogenous variables, users should inspect their data prior to running the analyses as non-normality will cause CAMSA estimation errors.

Step 3 Specify the Sensitivity Parameters

After the variables have been specified in Step 2, users will begin specifying the confounder correlations by clicking on the ‘Model’ tab. Confounder correlations are the augmented correlations between residuals associated with the mediators (latent intercept and slope) and the outcome variable. Users select the pre-specified range of confounder correlations (rho parameters ρ1 and ρ2) by moving the slider bars. Confounder correlations represent a range of plausible values that a confounder would induce if the confounder were included in the model.

When possible, the choice of the range of confounder correlation values should be based on expert knowledge or prior research of potential omitted confounders that would influence the mediation path (Tofighi et al., 2019). If such information does not exist, one might use Cohen’s (1988) general guidelines on qualitative labeling of a correlation value as a measure of effect size: ±.1 (small), ± .3 (medium), and ± .5 (large). After selecting a range, the user can specify the increment by which the confounder correlations (rho parameters ρ1 and ρ2) are tested (Supplementary Figure 3), where ρ1 is the confounder correlation for the mediated effect via the intercept and ρ2 is the confounder correlation for the mediated effect via the slope.

The CAMSA app creates a separate model for each combination of ρ parameters, the specification of the range of confounding correlations and the size of the increment determines the number for models that are run (and the time it takes to complete the iterative process of running all the models). In the example, if ρ1 and ρ2 ranges are −0.5 to 0.5 and the increment is set to 0.25, a total of 25 models are estimated. We chose −0.5 to 0.5 values for ρ1 and ρ2 in the simulated example because it covered a plausible range from small to large effect sizes of confounding correlations. It is recommended that users initially test their models with a large increment and then proceed to smaller increments if a more resolution is desired.

The user may also select the alpha by which the CAMSA app calculates the CI, by default this is 0.05 for a 95% two-sided CI. Users may also select two additional options for model specification on this page, first is to estimate the correlation between the intercept and slope, which is checked by default. If unchecked, then this correlation is set to zero. Second, the user may choose to only test confounder correlations where ρ1 = ρ2. In this case, ρ1 slider is used to select the range of correlations for both ρ1 and ρ2.

Once confounding correlations have been specified, clicking on ‘Run Mediation Model’ will display the lavaan model input syntax for the LGCMM as well as the outputs of the indirect estimates of the mediation effects for the latent intercept and slope. Next, the option to ‘Run CAMSA Models’ appears and clicking on this button will begin CAMSA estimation. A progress bar will appear in the bottom right corner of the browser window which displays model estimation progress. When model estimation has concluded this progress bar will disappear and users can proceed.

Step 4 Obtain Results

Clicking on the ‘Results’ tab will display the results in tabular form as well as several options for downloading results from the CAMSA model estimation (Supplementary Figure 4). The results are displayed in tables under the tabs ‘Indirect Intercept’ and ‘Indirect Slope’, column definitions are provided under the area for downloading the ‘CAMSA csv results.’ A compiled text file of the lavaan output may also be downloaded, which can be helpful for more detailed inspection of models to ensure correct estimation as well as analyze errors. Finally, detailed CAMSA results including the indirect estimates at each level of the confounding correlations may be downloaded in a .csv format. The downloaded .csv files also contains additional information regarding errors in estimation and may be used for plotting in other applications. Common estimation errors that will affect CAMSA results include non-positive covariance matrices, non-convergence of maximum likelihood function, and negative variance (Heywood cases). Some information is contained in the .csv if estimation errors have occurred (see Supplementary Table 1). If data is missing (NA) or there is other evidence of estimation issues, then users should inspect the compiled lavaan output text file. Note, that in many cases there may be minimal information as to the source of the error in model estimation.

Step 5 Plot Results

The CAMSA web app provides a basic plotting function that offers an intuitive way to interpret model output. After model estimation has concluded, users can click on the ‘Plots’ tab and then click ‘Plot’ to generate separate plots for the estimated indirect effects of the latent intercept and slope across the pre-specified levels of the confounding correlations. For example, clicking on the ‘Indirect Intercept’ will reveal a series of five plots (Figure 2, panel A). For each plot in panel A, the y-axis is the estimated indirect effect of the latent intercept, the x-axis is the value of confounding correlation of the intercept (ρ1), and each panel is a separate value of confounding correlation of the slope (ρ2). For the indirect slope, the confounding correlations are reversed and ρ2 is now the x-axis and ρ1 is displayed in each panel (Figure 2, panel B). Note, that if the ρ1 and ρ2 parameters are constrained to be equivalent then only five plots are displayed. For each panel, the red horizontal line represents the observed estimated indirect effect, two light blue dashed horizontal lines represent the upper and lower limits of the observed CI, a black horizontal line represents zero mediation effect (B=0), a black smoothed line that is a visualization of the CAMSA estimated mediated effect at each level of the x-axis confounding correlation, and a gray fill area represents the region of the 95% confidence interval. The plotting function automatically selects panels based on the minimum, maximum, and 25th, 50th, and 75th percentiles of the pre-selected confounding correlations. For example, if the range of confounding correlation is −0.5 to 0.5 then −0.5, −0.25, 0, 0.25 and 0.5 are used for panel plots. If a particular iterative model is unable to be estimated, (e.g., due to an error) at a given level of confounding correlation that is to be plotted (e.g., −0.25) then the plotting function plots at the nearest level where estimation was successful (e.g., −0.27). If there are numerous CAMSA models that are not estimated, then there may be panels missing from the output or there may be areas of the graphs that are incomplete. If this is the case, then we recommend that the user carefully evaluate model estimation using the model output. Plots may be downloaded by right-clicking on the image and selecting “Save image as…”.

Figure 2.

Figure 2.

Open in a new tab

CAMSA Latent Intercept Mediation Plots for Simulated Tutorial Data (Alcohol Craving)

Note. (A) = Latent intercept mediation (y-axis) by ρ1 (x-axis) and ρ2 (panel). (B) = Latent slope mediation (y-axis) by ρ2 (x-axis) and ρ1 (panel). Black smoothed line = CAMSA estimated mediated effect, gray fill = the area encompassing the CAMSA estimated confidence interval, horizontal black line = zero mediation effect (B=0), horizontal red line = observed mediated effect, blue dashed lines = upper and lower limits of the observed mediated effect confidence interval.

Step 6 Examining Robustness

To determine robustness to potential omitted confounders researchers should both inspect plots and the data. If the confidence interval of the mediated effect after adjusting for confounding correlations overlaps with zero or no longer overlaps with the estimated effect in the original LGCMM, then there is evidence that the mediated effect may not be robust to omitted confounders at the given level of the confounder correlations (ρ1 and ρ2). Specifically, this would indicate that the statistical conclusion of the original observed mediation test is not robust at the given level of confounding correlation. When analyzing one’s own data, determining robustness to potential omitted confounders will depend on methodological and theoretical factors in the substantive area that is being examined, such as size of the potential omitted confounders (levels of ρ1 and ρ2).

Simulated Example

Observed Latent Growth Mediation Effect

In the simulated example, participants were randomized to either naltrexone medication or placebo medication (independent variable, “X”) and percent days drinking was the outcome (“Y”). The hypothesized mediator (“M”) was alcohol craving which was measured at several time points during the study, as described in Step 2. A LGCMM was specified with mediation paths for the intercept, alcohol craving at the end of treatment, and slope, linear change in alcohol craving throughout out the study. The mediation path of the intercept (levels of alcohol craving at the end of treatment) was significant (β=−0.082, SE=0.036, 95% CI [−0.152, −0.013]), but the slope (change in cravings) was not (β=0.031, SE=0.036, 95% CI [−0.038, 0.102]). Thus, the observed latent growth mediation effect indicated that naltrexone resulted in significantly lower alcohol craving at the end of treatment (intercept), which resulted in fewer percent days drinking at the end of treatment. The effect of naltrexone on percent drinking days was not significantly mediated by change in craving over time (slope).

Interpreting Results of CAMSA App

CAMSA was performed on the LGCMM as specified above in the tutorial and the plotted results are displayed in Figure 2. The CAMSA estimated 95% confidence interval of the mediated (indirect) effect of the intercept (Figure 2, Panel A) did not overlap with zero for any combination of the confounding correlations (ρ1 and ρ2) and overlapped with the original estimate for the mediated effect for most of the confounding correlation values, thus indicating that the mediating path involving the intercept was robust to potential confounding correlations from small to large effect sizes. The results for the slope are also provided (Figure 2, Panel B) but since the original estimated mediated effect was not significant the CAMSA sensitivity plots were not interpreted.

Substantive Example

To further demonstrate the CAMSA web app, we describe an empirical example from a large randomized clinical trial that examined the efficacy of mindfulness-based relapse prevention (MBRP) compared to relapse prevention (RP) and treatment as usual (TAU) as an aftercare treatment for substance use disorder. Participants (n=286) were randomized to receive 8 weeks of MBRP, RP, or TAU, after completing residential or intensive outpatient treatment for a substance use disorder. Details of the trial and the primary outcomes have previously been reported (Bowen et al., 2014).

Methods

For the current analyses, we hypothesized that MBRP (independent variable) would be associated with greater acceptance at 12-months following treatment (outcome) via greater self-efficacy following treatment (mediator). These hypotheses are based on proposed mechanisms of behavior change in mindfulness-based treatments (Witkiewitz et al., 2014), as well as previous work showing self-efficacy to significantly mediate the effects of a mindfulness-based treatments (Spears et al., 2017). Self-efficacy was measured by the Drug Taking Confidence Questionnaire (DTCQ; Sklar, et al., 1997). The DTCQ is a 50-item self-report measure in which participants rate their perceived ability to resist the urge to drink heavily and cope in different situations. Participants rated their confidence on a 6-point Likert-type scale representing percent confidence in their ability to resist the urge (0 = 0% confident and 5 = 100% confident). DTCQ was measured at baseline, after treatment (8 weeks after baseline), and then at 2-, 4- and 6-months after treatment. Acceptance was measured using the Acceptance and Action Questionnaire (AAQ; Hayes, 2004), a 9-item instrument that assess control of private aversive experiences and acceptance of those experiences and the AAQ was measured at 12-months post treatment.

CAMSA was used to test the robustness of the LGCMM of self-efficacy (latent intercept and slope) mediating the effect of treatment on outcomes with confounding correlation values of −0.5 to 0.5 for ρ1 and ρ2 and an increment of 0.02. The specification did not include a correlation between the latent intercept and slope, and all combinations of the sensitivity parameters ρ1 and ρ2 were estimated. We chose a range from small to large effect sizes that may potentially confound the mediating paths in our model as previous research has demonstrated numerous other potential mechanisms (mediators) of behavior change in alcohol treatment studies (Longabaugh et al., 2013; Magill et al., 2015).

Results

Examination of DTCQ and AAQ did not reveal any evidence of non-normality. A self-efficacy mediation effect of the self-efficacy at the end of treatment (latent intercept) was significant (β=0.058, SE=0.032, 95% CI [0.006, 0.131]), but the change in self-efficacy (latent slope) was not (β=−0.049, SE=0.058, 95% CI [−0.183, 0.047]). Results from sensitivity analyses of the mediation paths of latent intercept and slope of self-efficacy using CAMSA are summarized in Figure 3; see OSF project folder for detailed results. The CAMSA sensitivity analyses demonstrated that the 95% confidence interval of the self-efficacy latent intercept mediation path overlapped with zero for ρ1 between 0.04 and 0.42, did not overlap with the original unadjusted estimate (β=0.058) when ρ1 was greater than 0.2, and these findings were consistent across multiple levels of ρ2. Stated more simply, the original mediating path of levels of self efficacy at the end of treatment (latent intercept) was robust to potential confounders when the omitted confounder was less than 0.04.

Figure 3.

Figure 3.

Open in a new tab

CAMSA Latent Intercept Mediation Plots for Self-Efficacy

Note. (A) = Latent intercept mediation (y-axis) by ρ1 (x-axis) and ρ2 (panel). (B) = Latent slope mediation (y-axis) by ρ2 (x-axis) and ρ1 (panel). Black smoothed line = CAMSA estimated mediated effect, gray fill = the area encompassing the CAMSA estimated confidence interval, horizontal black line = zero mediation effect (B=0), horizontal red line = observed mediated effect, blue dashed lines = upper and lower limits of the observed mediated effect confidence interval.

Discussion

Testing the sensitivity of LGCMMs to the no-omitted confounder assumption is critical to advancing our understanding of mechanisms of change in treatment research. The general mediation model assumes there is no confounding, but that is unlikely to be true in most research contexts. CAMSA uses the existing mediation model to conduct sensitivity analysis within a structural equation modeling (SEM) framework for testing the robustness of mediation effects to the no-omitted confounder assumption (Tofighi et al., 2019). The CAMSA app makes the sensitivity analysis more accessible by not requiring extensive statistical programming expertise. The proposed R Shiny App simplifies the process by allowing researchers to input their data and test a range of potential confounding correlations, as well as perform mediation analyses and run model diagnostics.

The substantive example examining the mediation of self-efficacy of mindfulness on acceptance demonstrated a case where it was important to consider a potential violation of the no-omitted confounder assumption. The initial mediation analysis indicated a significant mediation effect via the intercept of self-efficacy, such that individuals who received mindfulness-based relapse prevention compared to the other treatment conditions (relapse prevention and treatment as usual) had significantly higher post-treatment self-efficacy, which was associated with significantly greater acceptance at 12-months following treatment. Change (latent slope) in self-efficacy following treatment did not significantly mediate the effect of treatment on acceptance. Furthermore, CAMSA indicated that the mediation effect at the end of treatment (latent intercept) was not robust to the no-omitted confounder assumption, such that the introduction of a small confounding correlation (ρ1 > 0.04) yielded a mediation effect that was not significantly different from zero and did not overlap with the original estimate. Therefore, while the initial analysis concluded that end of treatment self-efficacy (intercept) mediated the association between receiving the mindfulness-based treatment and acceptance, this relationship does not appear to be robust to potential confounding variables. Future research should be conducted to replicate these results and examine potential confounders from observed data. For example, subjective well-being, stress, loneliness, and sex have each been shown to be associated with mindfulness and self-efficacy, and any one of these variables could be a confounder in the relationship between receiving a mindfulness-based treatment and self-efficacy in predicting acceptance (Jin et al., 2020; Rayan, 2019, Soysa & Wilcomb, 2015).

Limitations and Future Directions

CAMSA has several limitations. First, currently CAMSA does not accommodate LGCMM with a nonrandomized predictor variable, a model with more than three or more covarying mediators, and non-normal mediators and outcomes. Second, the sensitivity parameters (ρ1 and ρ2) are functions of the residual variances of mediators and the correlations between mediators and the outcome. Hence, model estimation errors (e.g., non-converged results) may occur if any of these values (residual variances or correlations) are close to zero, especially for replications testing larger confounding correlations. Third, testing for robustness via sensitivity analysis does not eliminate other sources of bias (e.g., common method bias) on causal interpretations. Therefore, researchers should extensively probe other plausible theoretical relations between variables when determining the robustness of proposed causal relations.

The CAMSA web app performs sensitivity analyses by probing the robustness of a LGCMM to potential omitted confounders (i.e., specificity assumption of causal modeling, a required reporting element for describing mediation findings; Lee et al., 2021). Future research exploring causal mediators in treatment trials should investigate possible confounds. Many studies have proposed mechanisms of behavior change, but few have evaluated the robustness of these mechanisms. The CAMSA app provides a user-friendly tool for researchers for evaluating the robustness of behavior change mechanisms.

Supplementary Material

Supp 1

Funding.

The current study was funded by NIAAA R01 AA025539 (Witkiewitz and Tofighi, MPIs) and NIDA R37DA09757 (MacKinnon, PI).

Footnotes

Disclosure statement. The authors do not have any conflicts of interest.

1

Tofighi et al. (2019) showed that confounder correlations are a function of the effects of a hypothesized omitted confounder on the endogenous variables. They showed equivalence between a LGCMM with correlated residuals and a LGCMM with a hypothesized omitted confounder.

Data availability.

Data (except for raw substantive data) and app code are available at https://osf.io/awzd2/

References

  1. Bowen S, Witkiewitz K, Clifasefi SL, Grow J, Chawla N, Hsu SHSH, Carroll HAHA, Harrop E, Collins SESE, Lustyk MKK, & Larimer MEME (2014). Relative efficacy of mindfulness-based relapse prevention, standard relapse prevention, and treatment as usual for substance use disorders: A randomized clinical trial. JAMA Psychiatry, 71(5), 547–556. 10.1001/jamapsychiatry.2013.4546 [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Cohen J (1988). Set correlation and contingency tables. Applied Psychological Measurement, 12(4), 425–434. 10.1177/014662168801200410 [DOI] [Google Scholar]
  3. Cox MG, Kisbu-Sakarya Y, Miočević M, & MacKinnon DP (2013). Sensitivity plots for confounder bias in the single mediator model. Evaluation Review, 37(5), 405–431. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Curran PJ, West SG, & Finch JF (1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1(1), 16–29. 10.1037/1082-989x.1.1.16 [DOI] [Google Scholar]
  5. Harring JR, McNeish DM, & Hancock GR (2017). Using phantom variables in structural equation modeling to assess model sensitivity to external misspecification. Psychological Methods, 22(4), 616. 10.1037/met0000103 [DOI] [PubMed] [Google Scholar]
  6. Hayes S (2004). Acceptance and commitment therapy, relational frame theory, and the third wave of behavioral and cognitive therapies. Behavior Therapy, 35(4), 639–665. 10.1016/S0005-7894(04)80013-3 [DOI] [PubMed] [Google Scholar]
  7. Hill AB, (1971). Principles of medical statistics (9th Edition) New York: Oxford. [Google Scholar]
  8. Holland PW (1988). Causal inference, path analysis, and recursive structural equations models. Sociological Methodology, 18, 449–484. 10.2307/271055 [DOI] [Google Scholar]
  9. Hsiao YY, Tofighi D, Kruger ES, Lee Van Horn M, MacKinnon DP, & Witkiewitz K (2019). The (lack of) replication of self-reported mindfulness as a mechanism of change in mindfulness-based relapse prevention for substance use disorders. Mindfulness, 10(4), 724–736. 10.1007/s12671-018-1023-z [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Imai K, Keele L, & Yamamoto T (2010). Identification, inference and sensitivity analysis for causal mediation effects. Statistical Science, 51–71. 10.1214/10-sts321 [DOI] [Google Scholar]
  11. Jin Y, Zhang M, Wang Y, & An J (2020). The relationship between trait mindfulness, loneliness, regulatory emotional self-efficacy, and subjective well-being. Personality and Individual Differences, 154, 109650. 10.1016/j.paid.2019.109650 [DOI] [Google Scholar]
  12. Kazdin AE (2007). Mediators and mechanisms of change in psychotherapy research. Annual Review of Clinical Psychology, 3, 1–27. 10.1146/annurev.clinpsy.3.022806.091432 [DOI] [PubMed] [Google Scholar]
  13. Lee H, Cashin AG, Lamb SE, Hopewell S, Vansteelandt S, VanderWeele TJ, MacKinnon DP, Mansell G, Collins GS, Golub RM, McAuley JH, AGReMA group, Localio AR, van Amelsvoort L, Guallar E, Rijnhart J, Goldsmith K, Fairchild AJ, Lewis CC, Kamper SJ, … Henschke N (2021). A Guideline for Reporting Mediation Analyses of Randomized Trials and Observational Studies: The AGReMA Statement. JAMA, 326(11), 1045–1056. 10.1001/jama.2021.14075 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Longabaugh R, Donovan DM, Karno MP, McCrady BS, Morgenstern J, & Tonigan JS (2005). Active ingredients: How and why evidence-based alcohol behavioral treatment treatments work. Alcoholism: Clinical and Experimental Research, 29(2), 235–247. 10.1097/01.alc.0000153541.78005.1f [DOI] [PubMed] [Google Scholar]
  15. Longabaugh R, Magill M, Morgenstern J, & Huebner R (2013). Mechanisms of behavior change in treatment for alcohol and other drug use disorders. In McCrady BS & Epstein EE (Eds.), Addictions: A comprehensive guidebook (p. 572–596). Oxford University Press. [Google Scholar]
  16. MacKinnon DP (2008). Introduction to statistical mediation analysis. Taylor & Francis Group/Lawrence Erlbaum Associates. [Google Scholar]
  17. MacKinnon DP, & Pirlott A, (2015). Statistical approaches to enhancing the causal interpretation of the M to Y relation in mediation analysis. Personality and Social Psychology Review, 19(1), 30–43. 10.1177/1088868314542878 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Magill M, Kiluk BD, McCrady BS, Tonigan JS, & Longabaugh R (2015). Active ingredients of treatment and client mechanisms of change in behavioral treatments for alcohol use disorders: Progress 10 years later. Alcoholism, Clinical and Experimental Research, 39(10), 1852–1862. 10.1111/acer.12848 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Mauro R (1990). Understanding LOVE (left out variables error): A method for estimating the effects of omitted variables. Psychological Bulletin, 108(2), 314. 10.1037/0033-2909.108.2.314 [DOI] [Google Scholar]
  20. Muthén LK, & Muthén BO (1998–2011). Mplus User’s Guide. Sixth Edition. Los Angeles, CA: Muthén & Muthén. http://www.statmodel.com/ [Google Scholar]
  21. R Core Team. (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/ [Google Scholar]
  22. Rayan A (2019). Mindfulness, self-efficacy, and stress among final-year nursing students. Journal of Psychosocial Nursing and Mental Health Services, 57(4), 49–55. 10.3928/02793695-20181031-01 [DOI] [PubMed] [Google Scholar]
  23. Rosseel Y (2012). Lavaan: An R package for structural equation modeling and more. Version 0.5–12 (BETA). Journal of Statistical Software, 48(2), 1–36. https://www.lavaan.ugent.be/ [Google Scholar]
  24. Sklar SM, Annis HM, & Turner NE (1997). Development and validation of the drug-taking confidence questionnaire: A measure of coping self-efficacy. Addictive behaviors, 22(5), 655–670. 10.1016/s0306-4603(97)00006-3 [DOI] [PubMed] [Google Scholar]
  25. Soysa CK, & Wilcomb CJ (2015). Mindfulness, self-compassion, self-efficacy, and gender as predictors of depression, anxiety, stress, and well-being. Mindfulness, 6 (2), 217–226. 10.1007/s12671-013-0247-1 [DOI] [Google Scholar]
  26. Spears CA, Hedeker D, Li L, Wu C, Anderson NK, Houchins SC, Vinci C, Hoover DS, Vidrine JI, Cinciripini PM, Waters AJ, & Wetter DW (2017). Mechanisms underlying mindfulness-based addiction treatment versus cognitive behavioral therapy and usual care for smoking cessation. Journal of Consulting and Clinical Psychology, 85(11), 1029–1040. 10.1037/ccp0000229 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Talloen W, Moerkerke B, Loeys T, De Naeghel J, Van Keer H, & Vansteelandt S (2016). Estimation of indirect effects in the presence of unmeasured confounding for the mediator–outcome relationship in a multilevel 2-1-1 mediation model. Journal of Educational and Behavioral Statistics, 41(4), 359–391. 10.3102/1076998616636855 [DOI] [Google Scholar]
  28. Tofighi D (2021). Sensitivity analysis in nonrandomized longitudinal mediation analysis. Frontiers in Psychology, 12, 5704. 10.3389/fpsyg.2021.755102 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Tofighi D, Hsiao YY, Kruger ES, MacKinnon DP, Lee Van Horn M, & Witkiewitz K (2019). Sensitivity analysis of the no-omitted confounder assumption in latent growth curve mediation models. Structural Equation Modeling, 26(1), 94–109. 10.1080/10705511.2018.1506925 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Tofighi D, & Kelley K (2016). Assessing omitted confounder bias in multilevel mediation models. Multivariate Behavioral Research, 51(1), 86–105. 10.1080/00273171.2015.1105736 [DOI] [PubMed] [Google Scholar]
  31. Tofighi D, & MacKinnon DP (2011). RMediation: An R package for mediation analysis confidence intervals. Behavior Research Methods, 43, 692–700. 10.3758/s13428-011-0076-x [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Valente MJ, Pelham WE III, Smyth H, & MacKinnon DP (2017). Confounding in statistical mediation analysis: What it is and how to address it. Journal of counseling psychology, 64(6), 659. 10.1037/cou0000242 [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Valeri L, & Vanderweele TJ (2013). Mediation analysis allowing for exposure-mediator interactions and causal interpretation: Theoretical assumptions and implementation with SAS and SPSS macros. Psychological Methods, 18(2), 137–150. 10.1037/a0031034 [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. VanderWeele TJ (2010). Bias formulas for sensitivity analysis for direct and indirect effects. Epidemiology, 21(4), 540–551. 10.1097/EDE.0b013e3181df191c [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Witkiewitz K, Bowen S, Harrop EN, Douglas H, Enkema M, & Sedgwick C (2014). Mindfulness-based treatment to prevent addictive behavior relapse: Theoretical models and hypothesized mechanisms of change. Substance Use & Misuse, 49(5), 513–524. 10.3109/10826084.2014.891845 [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Witkiewitz K, Litten RZ, & Leggio L (2019). Advances in the science and treatment of alcohol use disorder. Science Advances, 5(9), eaax4043. 10.1126/sciadv.aax4043 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supp 1
NIHMS1848235-supplement-Supp_1.docx (342.7KB, docx)

Data Availability Statement

Data (except for raw substantive data) and app code are available at https://osf.io/awzd2/

RESOURCES