The (Lack of) Replication of Self-Reported Mindfulness as a Mechanism of Change in Mindfulness-Based Relapse Prevention for Substance Use Disorders

Abstract

The development and evaluation of mindfulness-based interventions for a variety of psychological and medical disorders has grown exponentially over the past 20 years. Yet, calls for increasing the rigor of mindfulness research and recognition of the difficulties of conducting research on the topic of mindfulness have also increased. One of the major difficulties is the measurement of mindfulness, with varying definitions across studies and ambiguity with respect to the meaning of mindfulness. There is also concern about the reproducibility of findings given few attempts at replication. The current secondary analysis addressed the issue of reproducibility and robustness of the construct of self-reported mindfulness across two separate randomized clinical trials of mindfulness-based relapse prevention (MBRP), as an aftercare treatment for substance use disorder. Specifically, we tested the robustness of our previously published findings, which identified a latent construct of mindfulness as a significant mediator of the effect of MBRP on reducing craving following treatment. First, we attempted to replicate the findings in a separate randomized clinical trial of MBRP. Second, we conducted sensitivity analyses to test the assumption of the no-omitted confounder bias in a mediation model. The effect of MBRP on self-reported mindfulness and overall mediation effect failed to replicate in a new sample. The effect of self-reported mindfulness in predicting craving following treatment did replicate and was robust to the no-omitted confounder bias. The results of this work shine a light on the difficulties in the measurement of mindfulness and the importance of examining the robustness of findings.

Introduction

Over the past 2000 years definitions of mindfulness and methods for practicing mindfulness meditation have varied widely. In contemporary scientific literature, the term “mindfulness” is often characterized by a large number of different practices, processes (i.e., states of being), and individual characteristics (i.e., traits) and there is little agreement on definitions of mindfulness. The most commonly used definitions of mindfulness incorporate aspects of practice (e.g., engagement in mindfulness meditation practice) and the mental faculties of awareness, paying attention, and openness/non-reaction/non-judgment of moment-to-moment present moment experience (Davidson and Kaszniak, 2015a; Kabat-Zinn, 2011; Langer, 1989). The distinction between mindfulness as a practice, mindfulness as a state of being, and mindfulness as a trait are often blurred and ambiguous across empirical studies (Van Dam et al., 2018). Different objective and subjective measures of mindfulness are often only modestly correlated or not correlated at all and both state and trait measures of mindfulness do not tend to change (or even decrease) following interventions that are designed to target mindfulness (Bravo et al., 2018; Grossman, 2011; Van Dam et al., 2018). These challenges to measuring mindfulness have contributed to difficulties in studying mindfulness as a mechanism of change in mindfulness-based interventions (MBIs) and to understanding how MBIs are potentially effective in the treatment of psychological and physical health disorders.

The development and evaluation of MBIs for a wide variety of psychological disorders has increased exponentially over the past two decades. Many popular MBIs, including mindfulness-based cognitive therapy for depression (MBCT; Segal, Williams, & Teasdale, 2002), mindfulness-based therapy for insomnia (Ong & Sholtes, 2010), and mindfulness-based relapse prevention for substance use disorder (MBRP; (Bowen et al., 2011a), are adaptations of the mindfulness-based stress reduction (MBSR) course developed by Jon Kabat-Zinn to treat stress, chronic pain, and other chronic medical conditions (Kabat-Zinn, 2013; Kabat-Zinn et al., 1985). Formal mindfulness practices (e.g., sitting meditation, body scan meditation) and training in informal mindfulness practices (e.g., mindfulness in daily life) are core features of these group-based treatment programs that are typically delivered in closed cohorts for 2 to 2.5 hours per week for eight weeks (Dimidjian and Segal, 2015). A number of adaptations of MBIs have been developed that incorporate varying degrees of mindfulness practice and training durations, as well as mobile-based and web-based delivery options (Van Dam et al., 2018).

Recent systematic reviews and meta-analyses have indicated modest efficacy (e.g., small to medium effect sizes versus control conditions) of MBIs for a variety of physical conditions and psychological disorders (Anheyer et al., 2017; Carrière et al., 2017; Grossman et al., 2004; Hopwood and Schutte, 2017; Khoury et al., 2013; Kuyken et al., 2016; Li et al., 2017; Louise et al., 2017; Piet et al., 2012). Yet, the number of studies available for meta-analyses tend to be small, the quality of studies examined tend to be poor, and few studies have been conducted with active comparison conditions (Dimidjian and Segal, 2015; Van Dam et al., 2018). In the treatment of substance use disorder, there have been 16 randomized clinical trials that have examined MBIs versus active comparison conditions and results indicate MBIs produce small-to-large effect sizes on substance use and craving (Li et al., 2017). For example, MBRP has been shown to be more effective than community treatment (Bowen et al., 2014, 2009) and cognitive-behavioral relapse prevention (Bowen et al., 2014; Witkiewitz, Warner, et al., 2014b).

In addition to testing the efficacy of MBIs, it is important to understand why MBIs may be effective and research has begun to focus on the mechanisms of change in MBIs (Alsubaie et al., 2017; Gu et al., 2015; Young et al., 2018). Numerous mechanisms have been proposed, including attentional control, emotion- and self-regulation, trait and state mindfulness, mindfulness practice, changes in brain function and structure, negative affect, and acceptance, among others (Garland, 2016; Hölzel et al., 2011; Hunter-Reel et al., 2009; Tang and Leve, 2016). Recent reviews and meta-analyses have concluded that enhancing mindfulness is one promising transdiagnostic (e.g., not disorder specific) mechanism of change in MBIs (Alsubaie et al., 2017; Gu et al., 2015; van der Velden et al., 2015).

Enhancing mindfulness as a mechanism of change in MBIs is consistent with the notion that practicing mindfulness via both formal and informal mindfulness meditation practices in MBIs could ultimately increase one’s ability to have greater mindfulness in daily life. For example, the primary intentions of MBRP are to increase awareness of internal and external triggers, as well as reactions to situational, affective, and physical discomfort, that could increase craving for substances and relapse to substance use (Witkiewitz and Bowen, 2010). It is hypothesized that increases in mindfulness (i.e., non-judgment, acceptance, and awareness of present moment experience) obtained via training in formal and informal mindfulness practices will provide greater ability to stay with experiences of discomfort (e.g., craving) and ultimately reduce risk of relapse (Bowen et al., 2011b; Witkiewitz, Bowen, et al., 2014a).

Witkiewitz et al (2013) tested this theoretical mechanism of MBRP by examining whether changes in self-reported mindfulness from baseline to posttreatment significantly mediated the association between MBRP and changes in self-reported craving during and following treatment. Results indicated that a mindfulness latent factor, indicated by self-reported measures of mindfulness (the Acting with Awareness and Non-Judgment Subscales of the Five Facet Mindfulness Questionnaire; Baer, Smith, Hopkins, Krietemeyer, & Toney, 2006) and acceptance of present moment experience (as measured by the Acceptance and Action Questionnaire; Hayes et al., 2004 and three additional items of staying with discomfort; Chawla, 2010), significantly mediated the association between MBRP and changes in craving. Individuals who were randomized to MBRP versus an active comparison group reported significant increases in self-reported mindfulness and enhancement of mindfulness was associated with significant reductions in substance craving (Witkiewitz, Bowen, Douglas, & Hsu, 2013).

Witkiewitz et al (2013) provided evidence of mindfulness as a mediator of the association between MBRP and craving outcomes. Yet, even a significant mediating (indirect) relationship might not accurately reflect the true mechanism by which change occurs and establishing mediation is not sufficient to establish a mechanism of change (Kazdin & Nock, 2003). Witkiewitz et al. (2013) established the statistical significance of the mediated effect, but did not provide a replication of the effect or test the robustness of the mediated effect.

There is a large statistical literature on the importance of examining potential confounding and whether there is evidence that assumption of no omitted variables for mediation models is supported (Judd & Kenny, 1981; MacKinnon, 2008; Valeri & Vanderweele, 2013; VanderWeele, 2010). Here it is assumed that there is not some other variable, termed a confounder, that may also explain why the mediator and the outcome are related. As noted by Judd and Kenny (1981) if an omitted variable that causes the mediator and the outcome is omitted from the analyses, then the resulting mediation findings will be biased, “even when a randomized experimental research design has been used” (p. 607). Omitting potential confounders has the potential of producing biased mediation effects (MacKinnon and Pirlott, 2014). While the no-omitted confounder assumption is not directly testable (Holland, 1988), one can instead conduct a sensitivity analysis that assesses robustness of results against varying degrees of the violation of the no-omitted-confounder assumption.

The goal of the current study was to extend the findings of Witkiewitz et al (2013; also Bowen et al 2009) by examining whether the findings replicated in a new sample of individuals who received MBRP and completed all of the same assessments (Bowen et al., 2014) and to test whether the mediation model was robust to the no-omitted-confounder assumption. Bowen and colleagues (2009) evaluated an 8-week trial of mindfulness-based relapse prevention (MBRP) compared to treatment as usual (TAU) in adults with substance use disorder (Study 1) with assessments at baseline, end of intervention, and 2- and 4-months post intervention. Bowen and colleagues (2014) aimed to replicate the findings from Bowen et al (2009) in a larger clinical trial with three conditions, MBRP, TAU, and an active comparison condition, relapse prevention (Study 2) with assessments at baseline and 2-, 4-, 6- and 12-months post intervention. For the purposes of the current secondary analyses we focused on the MBRP and TAU conditions in both Study 1 (n=168) and Study 2 (n=198). The current study also addressed the criticisms of mindfulness research (Van Dam et al., 2018) with respect to measurement and reproducibility by explicitly studying whether the mindfulness construct was measured consistently across studies and whether the mediation findings were reproducible in an independent sample.

Method

Participants

Participants were recruited from a private, nonprofit agency that provided comprehensive care for alcohol and drug use disorders. Participants were included if they were between the ages of 18 to 70, fluent in English, and had completed intensive outpatient or inpatient treatment in the previous two weeks. Exclusion criteria included: psychosis, dementia, imminent suicide risk, significant withdrawal risk, or a need for more intensive treatment. Participants in Study 1 were 63.7% male, 53.6% non-Hispanic white, 29.8% African American, 15.3% multiracial, 9.5% Native American, 5.4% Hispanic or Latino/a, 0.6% Asian American, and 1.3% “other,” with an average age of 40.5 (SD=10.3), 41.3% were unemployed, and 71.6% had a high school diploma. Participants in Study 2 were 75.1% male, 52.8% non-Hispanic white, 24.6% African American, 7.7% multiracial, 5.6% Native American, 7.2% Hispanic or Latino/a, 1.0% Asian American, and 1.0% “other,” with an average age of 38.2 (SD=10.9), 65.5% were unemployed, and 66.2% had a high school diploma.

Procedure

Mindfulness Based Relapse Prevention (MBRP).

The MBRP group received weekly, 2-hour group sessions that followed the MBRP treatment manual (Bowen et al., 2011a). Two therapists facilitated groups of 6 to 10 persons. Each session included meditation practices and exercises related to one of five central themes: “automatic-pilot” and its relationship to relapse, recognizing thoughts and emotions in relation to triggers, integrating mindfulness practices into daily life, practicing the skills in high-risk situations, and the role of thoughts in relapse. Each session began with a 20- to 30-minute guided meditation, followed by experiential exercises, discussions and a review of assignments and individual practice that occurred between sessions. Participants were assigned daily exercises and were given meditation recordings for individual practice between group sessions.

MBRP therapists held professional positions (master’s degrees, in a doctoral training program, or had doctoral degrees in clinical psychology). MBRP therapists participated in several weeks of intensive training, engaged in daily meditation, and received weekly supervision. Additionally, all MBRP therapists had a personal mindfulness practice and experience attending intensive mindfulness meditation retreats. MBRP sessions were audio recorded and fidelity to the MBRP protocol was verified using the MBRP Adherence and Competence Scale (Chawla et al., 2010).

Treatment as Usual (TAU).

The TAU condition received standard outpatient aftercare that was designed to maintain abstinence through a twelve-step, process-oriented format. The primary treatment component was group sessions on specific topics (rational thinking skills, grief and loss, assertiveness, self-esteem, goal setting, effects of alcohol and other drugs on interpersonal relations and experience). TAU groups lasted 1.5 hours and met 1 to 2 times per week based on client need and were not assigned regular homework. TAU therapists were licensed chemical dependency counselors with varying professional degrees.

Measures

Acceptance.

Acceptance was measured using a 9-item version of the Acceptance and Action Questionnaire (AAQ; Hayes et al., 2004) and three additional items were added to the measure to assess acceptance of uncomfortable present moment private experiences (Chawla, 2010). Hayes et al (2004) described the original 9-items of the AAQ. The additional 3 items (all reverse scored) included: (1) When I am upset or anxious, I try to distract myself from feeling that way by engaging in other activities; (2) When I feel discomfort in my body (e.g., pain, tension), I focus on something else or do something to make it go away; and (3) When I feel discomfort in my body (e.g., pain, tension), I tend to get irritated at myself for feeling that way. All 12 items were rated on a 7-point Likert scale (Hayes et al., 2004). Higher scores on the 12-item measure indicated greater acceptance. The internal consistency in Study 1 was Cronbach’s = 0.58 at baseline and α = 0.65 at posttreatment. In Study 2, the internal consistency reliability was Cronbach’s α = 0.54 at baseline and α = 0.63 at posttreatment.

Mindfulness.

The Five Facet Mindfulness Questionnaire (FFMQ) assesses five facets of trait mindfulness (Observe, Describe, Acting with Awareness, Nonjudgement and Nonreactivity) and consists of 39 items rated on a 5-point Likert scale (Baer et al., 2006). The Acting with Awareness (8 items) and Nonjudgment (8 items) subscales were used in the current analyses. In Study 1, the internal consistency reliability of both subscales exceeded Cronbach’s α=.87 at both time points and in Study 2 the internal consistency reliability of both subscales exceeded Cronbach’s α=.84 at both time points.

Alcohol and Drug Craving.

The adapted Penn Alcohol Craving Scale (PACS) measures craving for alcohol or other drugs based on 5-items that assess frequency, intensity and duration of alcohol or other drug craving and an overall rating of craving for the prior week (Flannery et al., 1999). The internal consistency reliability of the PACS exceeded Cronbach’s α =.85 at all time points in Study 1 and Study 2. Study 1 administered the PACS at baseline, midtreatment, posttreatment, and 2- and 4-month follow-ups. Study 2 administered the PACS at baseline, posttreatment, and 2-, 4-, 6-, and 12-month follow-ups. For the current study, we examined craving from baseline through the 4-month follow-up.

Data Analyses

Descriptive statistics were calculated on both mindfulness and alcohol and drug craving indicators, and the TAU and the MBRP groups were compared on the mean scores of these indicators via Cohen’s d. Mean scores of the FFMQ subscales, modified AAQ, and craving measure were used in a series of latent variable models using Mplus version 8.0 (Muthén & Muthén, 1998–2017). All variables in the model had absolute values of skewness and kurtosis less than 3 and 10, respectively, which can be seen as following the normal distribution (Kline, 2016). Maximum likelihood estimation under the assumption that data were missing at random was employed.

For each model, fit indices were examined to assess model fit. A chi-square significance test $\left({\chi }_{ML}^2\right)$ with a p value greater than .05 indicates good model fit. Given ${\chi }_{ML}^2$ is sensitive to sample size, other fit indices were also considered. The comparative fit index (CFI) exceeding .90 (Hu and Bentler, 1999; McDonald and Ho, 2002), the root-mean-square error of approximation (RMSEA) below .08 (Browne and Cudeck, 1993), and the standardized root-mean-square residual (SRMR) below .08 (Hu and Bentler, 1999; Marsh et al., 2004) were considered acceptable fit to the data.

Replication of Study 1 Mediation Model in Study 2

We conducted a multiple group analysis and specified the proposed latent growth curve mediation model (LGCMM) for both studies. As shown in Figure 1, the mindfulness latent factor at the end of treatment was proposed as a mediator of the relationship between MBRP treatment condition (dummy coded as “treatment as usual = 0” and “MBRP = 1”) and changes in craving over time. Baseline mindfulness was included as a predictor of the mindfulness latent factor at posttreatment. Changes in craving over time was modeled by fitting a linear growth model, in which the intercept was specified to denote the mean craving scores at the end of treatment. MBRP treatment condition, treatment hours, and race (dummy coded as “Non-White = 0” and “White = 1”) were included as covariate predictors of the craving growth factors (see Witkiewitz et al., 2013, for details).

Figure 1.

Mediation model of mindfulness-based relapse prevention (MBRP; “treatment”) being associated with craving outcomes via an indirect effect of mindfulness scores at post-treatment. Paths related to the mediation analysis were highlighted in bold. Unstandardized path coefficients and standard errors (in parentheses) for Study 1 and Study 2 were both displayed. *p < .05, **p < .01, ***p < .001.

For the purposes of the sensitivity analyses (see below), we only included the intercept and linear growth factors and not the quadratic growth factor as depicted in Witkiewitz et al (2013). Additionally, we removed the midtreatment craving assessment from Study 1, because the parallel assessment was not available in Study 2. Hence, the model in Figure 1 was fit to both the Study 1 and Study 2 datasets and the results were compared to those in Witkiewitz et al (2013). The indirect effects were estimated by using the product of coefficients method (MacKinnon et al., 2002), in which mediation effects were denoted by multiplying two regression coefficients: the regression of the mediator (i.e., latent mindfulness factor at posttreatment; the a-path) on the treatment conditions; and the regression of the craving growth factors on the mediator (i.e., the b-path). We used the RMediation package (Tofighi and MacKinnon, 2011) to compute 95% CI for the indirect effects using a method that computes the numerical solution to the distribution of the product of the coefficients. Next, we further tested the difference in the a- and b-paths across studies. Wald test statistics were used to evaluate whether coefficients of the a-path and b-path in Study 1 and Study 2 were significantly different.

Invariance Test of Mindfulness Construct at Post-Treatment across Studies

The measurement model of the mediator, which is the mindfulness latent factor at post-treatment, was examined in both studies. As depicted in Witkiewitz et al (2013; see Figure 1), the mindfulness latent factor was measured by three indicators: the Acting with Awareness subscale of the FFMQ, the Non-Judgement subscale of the FFMQ, and the Acceptance measure derived from the AAQ with three additional items. A test of measurement invariance (MI) was conducted to examine whether we could assume the mindfulness latent factor was measuring the same construct between Study 1 and Study 2.

MI was tested by conducting a series of nested confirmatory factor models with constraints to force certain parameters in the models to be invariant between the two studies. As each set of new parameters is tested, the parameters known to be invariant from previous levels are constrained. Thus, the process of assessing measurement invariance is essentially testing of a series of increasingly restrictive hypotheses. These hypotheses included equal model structures (configural invariance), equal factor loadings (metric invariance), equal intercepts (scalar invariance), and equal error variances (residual invariance) (Brown, 2015; Chen et al., 2005; Millsap and Kwok, 2004; Vandenberg and Lance, 2000). Among the four types of MI, metric invariance has been suggested to closely relate to prediction bias (Vandenberg & Lance, 2000); therefore, we focused on testing of metric invariance in the present study. If the metric invariance assumption was violated, a partial metric invariance assumption in which some of the factor loadings were freely estimated was further examined.

In model identification, the covariance structures were identified by employing a marker variable strategy, in which the factor loadings of one of the indicators of each factor were constrained to be one; the mean structures were identified by constraining the latent means in the first group to zero and the intercepts of the first indicators invariant across groups. Each step of measurement invariance was evaluated by conducting a χ² difference test. A non-significant χ² difference test indicates a preference for the more restrictive model. On the other hand, a significant χ² difference test (i.e., p <.05) indicates the additional constraints resulted in a significantly worse model fit and the test of invariance is not supported.

Robustness of Mediation Finding in Study 1

After comparing the mediation effects yielded from the same model in Study 1 and Study 2, we conducted a sensitivity analysis to examine the robustness of these mediation effects in Study 1 (Fritz et al., 2016; Tofighi and Kelley, 2016; Valente et al., 2017). For studies with randomized predictors (e.g., treatment condition in the present study), the confounding effects of omitted variables for the independent variable-mediator are typically ignorable. On the other hand, the mediator in the model is not randomized so the confounding effects for the mediator-outcome relation could be biased by no-omitted confounders. To examine the level of the mediator-outcome confounding on the mediation model we conducted sensitivity analysis using a method developed by Tofighi et al (in press), which is an extension of sensitivity analysis in multilevel mediation analysis (Tofighi and Kelley, 2016). Tofighi et al (in press) showed that the effect of omitted (unobserved) confounders is equivalent to correlations between the residuals associated with the mediators and outcome variables. The sensitivity analysis is then conducted by introducing correlated residuals corresponding with pairs of the mediator and outcome variables, where the values of the correlated residuals are fixed a priori. Each value of correlated residuals between the mediator and the outcome represents a level of confounding.

In the context of the model in Figure 1, we conducted sensitivity analysis to examine concurrent robustness of two mediated effects associated with the latent intercept and slope simultaneously. We considered a wide range of correlation values for each correlation between latent intercept and outcome variable, ρ_intercept and correlation between latent slope and outcome variable, ρ_slope from −0.5 to 0.5. This range of values was chosen to represent the Cohen’s guideline for the magnitude of correlation as effect sizes: ± .1 “small”, ± .3 “moderate”, and ± .5 large. To ease model interpretation, we assumed that confounding would be positive for both intercept and slope or confounding would be negative for both intercept and slope. The annotated Mplus syntax for conducting the sensitivity analyses with latent growth mediation models is presented in the Supplementary Material.

Results

Descriptive Results across Studies

As shown in Table 1, the effects of MBRP, as compared to TAU, on AAQ, FFMQ, and PACS scores at posttreatment were small-to-medium (Cohen’s d range from .08 to .48 in absolute values) in Study 1 and were much smaller in Study 2 (Cohen’s d range from .03 to .21 in absolute values), particularly for the indicators of the mindfulness latent factor (Cohen’s d ≤ .07 in absolute value in Study 2). Importantly, the range of scores were similar across the two studies, however in Study 2 the participants in the MBRP condition had higher acceptance, acting with awareness, and nonjudgment at baseline, as compared to the scores at baseline among the Study 1 participants in the MBRP condition. In terms of the comparison between the two groups across studies at post-treatment, the MBRP group had higher scores on acceptance, acting with awareness, and nonjudgement than the TAU group in Study 1, but the opposite associations between treatment groups and the three mindfulness indicators were observed in Study 2.

Table 1. Descriptive Statistics for Mindfulness and Craving Scores by Treatment Group across Studies.

	Study 1			Study 2
Measure	TAU Mean(SD)	MBRP Mean(SD)	d	TAU Mean(SD)	MBRP Mean(SD)	d
AAQ
Baseline	3.93(0.80)	3.92(0.62)	0.01	3.86(0.74)	3.99(0.54)	−0.20
Post-treatment	3.97(0.84)	4.27(0.65)	−0.40	4.04(0.76)	4.00(0.65)	0.06
FFMQ Act awareness
Baseline	3.46(0.86)	3.27(0.78)	0.24	3.36(0.84)	3.46(0.75)	−0.13
Post-treatment	3.31(0.90)	3.39(0.87)	−0.08	3.47(0.77)	3.42(0.70)	0.07
FFMQ Nonjudgment
Baseline	3.37(0.83)	3.21(0.78)	0.20	3.67(0.84)	3.80(0.85)	−0.15
Post-treatment	3.17(0.77)	3.54(0.86)	−0.46	3.43(0.83)	3.41(0.79)	0.03
PACS
Baseline	1.73 (1.42)	1.55 (1.13)	0.14	1.62(1.32)	1.23 (0.91)	0.34
Post-treatment	1.70 (1.42)	1.12 (1.05)	0.46	1.33 (1.26)	1.10 (1.05)	0.21
2-month follow-up	1.42 (1.49)	0.98 (0.98)	0.35	0.94 (1.07)	1.13 (1.12)	−0.18
4-month follow-up	1.28 (1.50)	1.09 (1.26)	0.14	1.02 (0.98)	0.87 (0.85)	0.16

Note. MBRP = mindfulness based relapse prevention; d = Cohen’s d measure of effect size calculated at each time point using the formula $d=\frac{x_1-x_2}{S{D}_{pool}}$ where $S{D}_{pool}=\sqrt[2]{(S{D}_1^2+S{D}_1^2)/2}$

Table 2 provides the test-retest correlations between baseline and subsequent assessments. All correlations were below .80, indicating the measures were sensitive to state changes and suitable to evaluate short term intervention effects (Paterson et al., 2017). The greatest difference between the MBRP and TAU groups was observed on the AAQ in Study 1.

Table 2. Correlations between Mindfulness and Craving Scores at Baseline and Subsequent Assessments Within Treatment Groups Across Studies.

	Study 1		Study 2
Measure	TAU	MBRP	TAU	MBRP
AAQ
Post-treatment	0.745	0.362	0.590	0.667
FFMQ Act awareness
Post-treatment	0.630	0.695	0.741	0.675
FFMQ Nonjudgment
Post-treatment	0.503	0.595	0.661	0.604
PACS
Post-treatment	0.616	0.520	0.660	0.530
2-month follow-up	0.176	0.395	0.556	0.397
4-month follow-up	0.510	0.433	0.376	0.436

Note. MBRP = mindfulness based relapse prevention. All the test-retest reliability was calculate for baseline comparing to all the other assessment time points.

Replication of Study 1 Mediation Model in Study 2

Due to the exclusion of the quadratic growth factor and the craving indicator at the mid-treatment time point for the current study (as compared to the inclusion of the quadratic growth factor and the craving indicator at mid-treatment in Witkiewitz et al., 2013), both the direct and indirect effects for Study 1 and Study 2 were examined. Results from Study 1 were consistent with results in Witkiewitz et al (2013). The mediation model in Study 1 provided an adequate model fit: ${\chi }_{ML}^2$ (56) = 81.34, p =.02; CFI = .96; RMSEA = .05; SRMR = .07. Table 3 shows the unstandardized and standardized coefficients of the direct and indirect effects. Compared to the control group, the MBRP group had significantly higher posttreatment mindfulness latent factor scores (β =0.45; B(SE) = 0.25(0.11), p = .023). The mindfulness latent factor was associated with the craving estimates at the end of treatment (β =−0.42; B(SE) = −.74 (.19), p < .001), but not significantly associated with the linear slope factor (p=.165). The mindfulness latent factor significantly mediated the association between MBRP and reductions in craving at the end of treatment (β =−0.10; B(SE) = −.18 (.09), p = .049; 95% CI: −0.39, −0.02), but did not significantly mediate the association between treatment conditions and the linear slope (p = .216; 95% CI: - 0.01, 0.10).

Table 3. Unstandardized (B) and Standardized (β) Path Coefficients of MBRP Treatment on Mindfulness and Change in Craving Over Time.

Direct Effects	B (SE)	β
Study 1
MBRP treatment → mindfulness	0.25(0.11)*	0.45
mindfulness → intercept	−0.74(0.19)***	−0.42
mindfulness → slope	0.13(0.10)	0.22
Study 2
MBRP treatment → mindfulness	−0.14(0.07)	−0.12
mindfulness → intercept	−0.64(0.15)***	−0.43
mindfulness → slope	0.10(0.07)	0.21
Indirect Effects	B (SE)	𝛽
Study1
MBRP treatment → mindfulness → intercept	−0.18(0.09)*	−0.10
MBRP treatment → mindfulness → slope	0.03(0.03)	0.05
Study 2
MBRP treatment → mindfulness → intercept	0.09(0.05)	0.05
MBRP treatment → mindfulness → slope	−0.01(0.01)	−0.03

^*p < .05

^**p < .01

^***p < .001.

The mediation model in Study 2 yielded adequate fit: ${\chi }_{ML}^2\left(56\right)=90.92$, p =.002; CFI = .95; RMSEA = .06; SRMR = .08. The latent mindfulness was significantly associated with the level of craving at the end of treatment (β =−.43; B(SE) = −.64(.15), p<.001), but not with the linear growth factor (p=.109). Contrary to the results in Witkiewitz et al (2013), the MBRP group did not have significantly higher latent mindfulness factor scores (β =−.12; B(SE) = −.14 (.07), p=.053). Additionally, the MBRP group seemed to have lower latent mindfulness factor scores than the TAU group. The result indicated that the signs of the effects from treatment group to the latent posttreatment mindfulness factor were different. Indirect effects of mindfulness on the association between treatment condition and craving at the end of treatment (p=.073; 95% CI: - 0.00, 0.20) and the linear slope (p= .213; 95% CI: −0.04, 0.00) were not significant.

Finally, Wald test statistics were obtained to formally test whether the a- and b-paths in studies 1 and 2 were statistically equivalent. Results indicated that the a-path was significantly different across studies (χ² (1) = 9.17, p=0.002), whereas the b-path with mindfulness predicting the intercept of craving (χ² (1) = 0.15, p=0.694) and the b-path with mindfulness predicting the slope of craving (χ² (1) = 0.06, p=0.802) were not significantly different across studies. Thus, results from the replication analyses indicated the overall mediation effect did not replicate in Study 2. The multiple group models indicate differences across studies in the association between treatment and mindfulness (a-path), with a significant association identified in Study 1 and no association in Study 2.

Invariance Test of the Mindfulness Construct across Studies

One potential explanation for the failed replication could be differences in measurement of the mediator. Thus, we next tested whether the mindfulness construct was invariant across studies. For the mindfulness invariance test at posttreatment, the configural invariance models were saturated models (one factor with three indicators) so the χ² test $\left({\chi }_{ML}^2\right)$ values were zero. The metric invariance model results showed poor fit at posttreatment: ${\chi }_{ML}^2\left(2\right)=6.34$, p =.04; CFI .98; RMSEA = .14; SRMR = .10. Compared to the configural invariance model, the factor loadings of the mindfulness latent factors were non-invariant across studies at the posttreatment time point (i.e., metric invariance was not supported at posttreatment). Two partial invariance models with one or two non-invariant indicators were further examined. Chi-square difference test results showed that the configural invariance model is more preferable to the partial invariance models. Hence, the mindfulness latent factor at post-treatment only fulfilled the condition of configural invariance across studies.

Robustness of Mediation finding in Study 1

A second potential explanation for the failed replication could be lack of robustness of the mediated effect. Thus, we next conducted a sensitivity analysis to test the robustness of the mediated effect in Study 1. The mediator-outcome residual correlations related to the craving intercept factor and craving linear slope factor were both considered in the sensitivity analyses. The residual correlations were specified within the range of −0.5 to 0.5 with an increment of 0.10. As noted above, we only considered the condition where the two residual correlations were either both positive or negative to simplify analyses, thus assuming the same direction of confounding on the intercept and slope factors.

Results from the sensitivity analyses show that the 95% confidence interval (CI) of the intercept mediation effects begin to cover zero (non-significant) when ρ_interceptdecreased to-.40 along with ρ_slope decreased to −.30. On the other hand, increasing both ρ_intercept and ρ_slope to as high as .50 still resulted in a significant mediation effect (the 95% CI of the intercept mediation effects did not contain zero). Thus, the significant intercept mediation effects were detected if the magnitude of the mediator-outcome residual correlations were within the range of −.30 to .50 (or higher), indicating the intercept mediation effects were robust to the no-omitted confounder assumption. In terms of the non-significant slope mediation effects, the mediated effects remained non-significant as both ρ_intercept and ρ_slope increased to .5, but changed to significant as ρ_slope decreased to −.30, regardless of the negative ρ_intercept values. Thus, the non-significant slope mediation effects held across a large range of effects of an omitted confounder. A simplified table with conditions when ρ_intercept = ρ_slope is showed in Table 4, in which the unstandardized indirect effects and 95% CI across different pairs of equal mediator-outcome residual correlations (ρ) in Study 1.

Table 4. Indirect Effect Estimates of MBRP Treatment (X) through Mindfulness (M) on Intercept (DV1) and Slopes Craving (DV2). The Relationships Between Mediator-Outcome Residual Correlation (ρ), and the 95% Confidence Interval (CI) in Sensitivity Analysis of Study 1.

ρ	DV1: Latent Intercept		Dv2: Latent Slope
	Estimate	CI	Estimate	CI
−0.5	−0.09	[−0.29, 0.04]	0.08	[0.01, 0.19]
−0.4	−0.12	[−0.30, 0.00]	0.07	[0.00, 0.17]
−0.3	−0.14	[−0.32, −0.01]	0.06	[0.00, 0.15]
−0.2	−0.15	[−0.34, −0.02]	0.05	[−0.00, 0.13]
−0.1	−0.17	[−0.37, −0.02]	0.04	[−0.01, 0.11]
0	−0.18	[−0.39, −0.02]	0.03	[−0.01, 0.10]
0.1	−0.19	[−0.42, −0.02]	0.03	[−0.02, 0.09]
0.2	−0.21	[−0.45, −0.02]	0.02	[−0.03, 0.08]
0.3	−0.22	[−0.48, −0.01]	0.01	[−0.04, 0.07]
0.4	−0.24	[−0.52, −0.01]	0.01	[−0.05, 0.06]
0.5	−0.26	[−0.58, −0.00]	−0.00	[−0.06, 0.06]

Note. ρ represents both the correlation between latent intercept and outcome variable (ρ_intercept) and the correlation between latent slope and outcome variable (ρ_slope ). Only conditions of ρ_intercept = ρ_slope were shown.

Discussion

A previous study by our research group (Witkiewitz et al., 2013) found self-reported mindfulness following treatment, as measured by a latent factor indicated by the Nonjudgment and Acting with Awareness subscales from the Five Facet Mindfulness Questionnaire (FFMQ) and a modified version of the Acceptance and Action Questionnaire (AAQ), significantly mediated the effect of mindfulness-based relapse prevention (MBRP) on substance use craving at the end of treatment. The current secondary analysis study examined whether these prior findings could be replicated in a second randomized clinical trial of MBRP for substance use disorder (Bowen et al., 2014). Results from the secondary analyses indicated the test of mindfulness as a mechanism of behavior change in the association between MBRP and craving failed to replicate in Study 2, due to the failure of the a-path of the mediation effect in a new sample (Study 2; Bowen et al., 2014). Given the failed replication, we also tested the measurement of the construct of mindfulness and the robustness of the mediation effect in Study 1. The mindfulness latent factor established in our prior research (Study 1; Witkiewitz et al., 2013) was not metric invariant across studies. Interestingly, the results from the first sample (Study 1; Bowen et al., 2009; as reported in Witkiewitz et al., 2013) were robust to the no-omitted-confounder assumption of the b-path of the model. Even a large level of confounding, defined as a correlation between the residuals associated with mediator and outcome of ± .40, resulted in a significant mediation effect. Thus, in Study 1, mindfulness at posttreatment significantly mediated the effect of MBRP on craving even when a large amount of confounding was added to the b-path of the mediation model. Yet, MBRP did not significantly predict mindfulness in Study 2 (the a-path) and thus the replication failed in Study 2.

The results from the failed replication may seem surprising. The two studies (Bowen et al., 2014, 2009) were conducted by the same research group, using the same assessment tools, the same therapist supervisors, many of the same therapists for the MBRP condition (TAU therapists were not controlled), and were conducted in the same community treatment agency. The design of the two studies were nearly identical, with the only difference being the years when the studies were conducted (Study 1 participants were recruited from 2007–2008 and Study 2 participants were recruited from 2009–2012), the number of treatment sites (Study 1 recruited from 1 site within the agency and Study 2 recruited from 2 sites within the same agency), and the number of treatment conditions (Study 2 included a cognitive-behavioral relapse prevention condition). The participant characteristics were similar across the two studies.

Yet, the effects of MBRP versus TAU on the measures used as indicators of the latent mindfulness factor were considerably smaller in Study 2. Results of test-retest correlations showed larger difference between the MBRP and TAU groups on AAQ in Study 1 than that in Study 2. Such results could explain why the significant a-path is observed only in Study 1. Another potential explanation for the failure of MBRP versus TAU to have a significant effect on mindfulness (the a-path of the mediation model) may be due to the influence of MBRP on the treatment center immediately following Study 1. Therapists at the treatment agency became more open to mindfulness-based approaches and could have incorporated some mindfulness practices into the treatment milieu (including the TAU condition).

The p-value for the a-path (p = .023) and effect size in Study 1 may also shed some light on the failure of replication of the a-path in Study 2. Goodman (1992) indicates that for an effect with a p-value equal 0.05, the probability of replication is roughly 50%. Interestingly, the b-paths in Study 1 and 2 were successfully replicated at p < .001. Although p-values cannot be treated as effect sizes, these results are consistent with larger effects being more likely to be replicated in subsequent studies (Taylor and Muller, 1996). The relation of the mediator to the outcome, represented by the b-path, is typically based on greater theory and empirical evidence than the a-path, which represents intervention strategies designed to change the mediator.

Implications for Research on “Mindfulness”

The current findings are consistent with concerns about the reproducibility of mindfulness research findings (Van Dam et al., 2018) and the measurement of mindfulness via self-report questionnaires (Gethin, 2011; Grossman, 2011; Grossman and Van Dam, 2011). With respect to reproducibility, the sensitivity analyses presented in the current paper could be a valuable tool for increasing the rigor of mindfulness research by testing the no-omitted confounder assumption when examining mechanisms of mindfulness-based interventions (MBIs). Estimating the sensitivity of a mediation finding to the no-omitted confounder assumption increases rigor by testing robustness of the candidate mediators. The association between self-reported mindfulness and craving (b-path) did replicate across studies and was robust to the no-omitted confounder assumption. The association between mindfulness and craving has been widely studied using correlational and experimental designs (Tapper, 2017), and the results from the current study provide further support for a negative association between trait mindfulness and self-reported craving.

With respect to the measurement of mindfulness, the current study found that the latent mindfulness construct at posttreatment was not invariant at the metric level across studies. In other words, the association between the mindfulness latent factor and the indicators were different across studies. Previous research examining the FFMQ has also identified non-invariance of the FFMQ across time following a mindfulness-based intervention (Gu et al., 2016) and across meditator and non-meditator groups (Van Dam et al., 2009). Likewise the AAQ has been found to have different factor structures across clinical and non-clinical groups (Costa et al., 2014). As noted previously, the measurement of mindfulness has been poorly defined in most studies and mindfulness practice is generally not associated with self-report measures of mindfulness (Manuel et al., 2017). Attempts to measure mindfulness using objective measures (e.g., breath counting; Levinson; Stoll, Kindy, Merry, & Davidson, 2014) or neural measures (e.g., magnetic resonance imaging; Lutz, Jha, Dunne, & Saron, 2015) have been developed, yet these measures also suffer from interpretational difficulties, often produce small effects, and have often not been replicated across studies (Van Dam et al., 2018).

Limitations and Future Directions

The current study is not without limitations. The current study was a secondary analysis that was limited to the data collected in the original studies. For example, the reliability of the modified AAQ was low in both studies and both studies relied exclusively on self-report measures of mindfulness, acceptance, and craving. An additional measurement limitation is the lack of measurement invariance of the mindfulness latent construct across studies. The fact that the latent mindfulness construct was not invariant limits interpretation of the failed replication. Future research may also examine other psychometric approaches (e.g., item response theory) to study the mindfulness factor structure (Medvedev, Siegert, Kersten, & Krägeloh, 2017).

With respect to the sensitivity analyses of the no-omitted-confounder assumption there were also some limitations. The proposed method (Tofighi et al., in press) can be applied while simultaneously testing up to two mediation effects. Hence, we reduced the complexity of the original model that was estimated by Witkiewitz et al. (2013) to reduce the number of latent outcomes (intercept, linear slope, quadratic slope were reduced to only intercept and linear slope). We also tested a limited range of residual correlations from 0 to ± 5 at 0.1 intervals and we only tested residual correlations that had the same direction across intercept and slope (e.g., −.1 and −.1, .2 and .5). Future research using sensitivity analyses with more than two latent outcomes and confounders with different signs would need further methodological work.

It is also important to note that there are other approaches to examine confounding (Cox et al., 2013; Fritz et al., 2016; Valeri and Vanderweele, 2013) and most intervention studies are not powered to detect mediation effects or potential confounders of the mediation effect (MacKinnon, 2008). More accurate assessment of mindfulness could lead to a larger and more robust effect of mindfulness as a mechanism of MBIs. More generally, identifying mechanisms of behavior change is a distillation process whereby mediators are identified by improving our theories and the measurement of constructs to more precisely test the purported mechanisms.

The results of the current study indicate that previously published findings examining a latent mindfulness construct, indicated by self-report measures of mindfulness, as a mechanism of the association between mindfulness-based relapse prevention and craving in a substance use disorder sample (Witkiewitz et al., 2013) did not replicate in a new sample. Furthermore, the latent mindfulness construct was not invariant across two different samples. The lack of measurement invariance of the mindfulness construct is consistent with other studies that have not provided evidence of measurement invariance for mindfulness measures (Costa et al., 2014; Gu et al., 2016; Van Dam et al., 2009). Although being robust to no-omitted confounding, the effects from treatment to the mindfulness latent factor were not replicated across studies. The failure of replication is also consistent with recent concerns raised about the reproducibility and rigor of mindfulness research (Van Dam et al., 2018), and psychological research in general (Lilienfeld, 2017). There has been a tension between Western science and contemplative traditions in evaluating the construct of mindfulness (Davidson and Kaszniak, 2015b; Gethin, 2011; Grossman, 2011; Kabat-Zinn, 2011) and the current study provides evidence that more work needs to be done, particularly in the operationalization of mindfulness as a mechanism of behavior change following MBIs. We hope that the syntax to conduct sensitivity analyses included in the current paper will provide an opportunity for future research to test the robustness of mediation effects and greater rigor in mindfulness research going forward.