Does the Five Facet Mindfulness Questionnaire measure what we think it does? Construct validity evidence from an active controlled randomized clinical trial

Abstract

The current study attempted a rigorous test of the construct validity of a widely used self-report measure of dispositional mindfulness, the Five Facet Mindfulness Questionnaire (FFMQ), within the context of an active controlled randomized trial (n = 130). The trial included three arms: mindfulness-based stress reduction (MBSR), an active control condition that did not include instruction in mindfulness meditation (Health Enhancement Program, HEP), and a waitlist control condition. Partial evidence for the convergent validity of the FFMQ was shown in correlations at baseline between FFMQ facets and measures of psychological symptoms and psychological well-being. In addition, facets of the FFMQ were shown to increase over the course of an MBSR intervention relative to a waitlist control condition. However, the FFMQ failed to show discriminant validity. Specifically, facets of the FFMQ were shown to increase over the course of the HEP intervention relative to the waitlist control condition. MBSR and HEP, in contrast, did not differ in changes in FFMQ score over time. Implications of these findings for the measurement and theory of mindfulness and MBSR are discussed.

In recent decades, mindfulness-based interventions such as mindfulness-based stress reduction (MBSR; Kabat-Zinn, 1990) and mindfulness-based cognitive therapy (MBCT; Segal, Williams, & Teasdale, 2002) have become increasingly widespread (Khoury et al., 2013; Zoogman, Goldberg, Hoyt, & Miller, 2014). As such, it likewise becomes increasingly important to have a clear articulation of what mindfulness is and how it can be measured (Lutz, Jha, Dunne, & Saron, 2015). The most popular strategy for measuring mindfulness has been to employ self-report questionnaires (Baer et al., 2006; Del Re, Flűckiger, Goldberg, & Hoyt, 2013).

One of the most widely used self-report measures of mindfulness that has been employed in hundreds of studies is the Five Facet Mindfulness Questionnaire (FFMQ; Baer et al, 2006). In a series of validation studies, Baer et al. (2006) subjected items from five available mindfulness questionnaires to factor analysis, ultimately creating a 39-item measure intended to capture five facets of mindfulness (Acting with, Describe, Non-judge, Non-react, and Observe). The FFMQ was shown to correlate in expected directions with related constructs (e.g., emotion regulation, neuroticism, alexithymia) and FFMQ facets predicted psychological symptoms (Baer et al., 2006). Support for the validity of the FFMQ has included demonstrating associations between the measure and amount of meditation experience, with facets of the FFMQ shown to mediate the link between meditation experience and psychological wellbeing (Baer et al., 2008). Key evidence for the validity of the FFMQ has also been found in the measure's responsiveness to various forms of mindfulness training, and changes in mindfulness (indexed using the FFMQ and similar self-report measures of mindfulness) have been shown to moderate effect sizes on clinical outcomes in meta-analyses of mindfulness-based interventions (Khoury et al., 2013). Further, amount and quality of mindfulness practice have both been associated with changes in the FFMQ (Goldberg, Del Re, Hoyt & Davis, 2014).

Despite these promising psychometric properties, some have questioned the validity of self-report measures of mindfulness (Lutz et al., 2015). Grossman (2008, 2011) has raised several vital issues, including the importance of defining and measuring mindfulness in Western psychology in a way that does justice to the term's original intention, avoiding biases in measurement development, and recognizing and addressing semantic confusion with item meanings, alongside the typical known biases inherent to self-report methodologies (e.g., social desirability bias). Indeed, self-report measures of dispositional mindfulness have failed to differentiate groups with theoretically distinct levels of mindfulness (e.g., expert meditators versus controls; Antonova, Chadwick, & Kumari, 2015). Item wording (e.g., positive versus negative) has also been shown to impact response tendencies (Van Dam, Earleywine, & Danoff-Burg, 2009, although perhaps not when samples are demographically matched, e.g., Baer, Samuel, & Lykins, 2011). Based on these and other reports, some have questioned the construct validity of these self-report measures.

Construct validity: Convergent and discriminant

Construct validity is perennially a key concern in the development of psychological tests (Crocker & Algina, 2008). In their classic treatise on construct validity, Cronbach and Meehl (1955) note that “construct validation is involved whenever a test is to be interpreted as a measure of some attribute or quality which is not ‘operationally defined’” (p. 282). The construct of mindfulness arguably remains without an agreed upon operational definition (although some have begun attempting to operationalize mindfulness with behavioral measures; Levinson, Stoll, Kindy, Merry, & Davidson, 2014).

Most simply, construct validity reflects the extent to which a test measures what it is intended to measure (Crocker & Algina, 2008). Cronbach and Meehl (1955) identify several means by which the construct validity of a psychological test can be established. Of note, the FFMQ has passed many of these tests (at least in some studies): group differences shown between meditating and non-meditating samples, correlations with related psychological constructs, adequately fitting internal factor structure (Baer et al., 2006, 2008). Cronbach and Meehl (1955) give particular emphasis to experimental manipulation as a means of determining construct validity, and the FFMQ has been shown to be sensitive to training in mindfulness (Khoury et al., 2013).

However, a key test of the FFMQ's construct validity remains largely ignored, specifically the measure's discriminant validity in the context of experimental manipulation. While convergent validity involves demonstrating that a measure is associated with theoretically related constructs, discriminant validity involves demonstrating that a measure is not associated with theoretically unrelated constructs. Campbell and Fiske (1959) identify the importance of discriminant validity, noting that “for the justification of novel trait measures, for the validation of test interpretation, or for the establishment of construct validity, discriminant validation as well as convergent validation is required” (p. 81).

The current study attempts to provide a standard convergent validity test of the FFMQ alongside a rigorous test of its discriminant validity within the context of experimental manipulation. Specifically, we examined changes in FFMQ scores within a randomized controlled trial (RCT) that included a standard MBSR condition, a waitlist (WL) control condition, and an active control condition – Health Enhancement Program (HEP; MacCoon et al., 2012). HEP is a recently designed active control intervention specifically created to control for the non-mindfulness-specific components of MBSR. Unlike MBSR, which teaches mindfulness practices such as sitting meditation and the body scan, HEP includes music therapy, nutrition education, and physical activity. HEP is matched to MBSR in many non-specific features: both interventions have a group format, meet weekly for 2.5 h, include an “all day” component, and assign the same amount of home practice (MacCoon et al., 2012). The development of HEP attempted to match with MBSR non-specific components of treatment (e.g., social support, therapeutic alliance, positive expectancy) proposed to be powerful in facilitating psychological change, including in the context of mindfulness training (Goldberg, Davis, & Hoyt, 2013).

The inclusion of both an active and inactive control condition allows us to address two discriminant validity questions. We can assess directly whether the FFMQ is responsive to change in a mindfulness-based intervention relative to either a waitlist control or a non-mindfulness-based active intervention (HEP). We hypothesized that as a valid measure of the mindfulness construct, (a) the FFMQ should increase after mindfulness-specific training (i.e., MBSR) relative to wait list, and (b) the FFMQ should show a greater response to mindfulness-specific training than a control intervention composed of non-specific elements (HEP).

In combination with an experimentally manipulated discriminant validity test, a standard test of the FFMQ's convergent validity was possible through examining associations between this measure and related constructs at baseline. It was hypothesized that the FFMQ would correlate at baseline with measures of psychological well-being and psychological symptoms.

Method

This study used a randomized controlled design that has been described elsewhere (MacCoon et al., 2012). Participants were recruited for a “health and well-being” study and were randomly assigned to a mindfulness intervention, an active control intervention, or a wait-list control group. The mindfulness intervention consisted of a standard MBSR course (Kabat-Zinn, 1990) and was provided by experienced MBSR instructors. MBSR is an eight-week intervention offered in a group format that includes instruction in both formal (e.g., sitting meditation) and informal (e.g., attentiveness during daily activities) mindfulness practices. These practices are aimed at cultivating a mindful relationship with one's experience, encouraging attentiveness, non-judgment, and curiosity (Kabat-Zinn, 1990). The active control intervention consisted of the Health Enhancement Program (HEP; MacCoon et al., 2012) course and was provided by experienced HEP instructors with no background in mindfulness. HEP was designed specifically to match MBSR as closely as possible while not including mindfulness as an active ingredient. HEP, like MBSR, is administered in a group setting and assigns homework practice. Unlike MBSR, HEP includes mild physical activity, functional movement, nutrition education, diet planning, and music and imagery designed to increase well-being. Both MBSR and HEP were designed to enhance psychological health, while relying on different theoretical pathways of action. Participants randomized to the waitlist control condition received no intervention during the course of the study. Participants completed a battery of questionnaires and were assessed on a number of physiological measures as part of a larger study of which this experiment was a part. The three measures of interest here are the FFMQ, the Psychological Well-being Scale (PWB), and the Symptom Checklist (SCL-90R).

One hundred and thirty participants were initially enrolled in the study (see Consort Diagram, Supplemental Material Figure 1). Of this full sample, 43 were randomized to MBSR, 42 to HEP, and 36 to the control condition. Baseline data were available for n = 125 and pre-post data for n = 106. The sample was predominantly female (n = 79, 63.2%) with a mean age of 48.05 years (SD = 10.74). Participants were excluded if they had used medication for anxiety, depression, or other psychological issues, or had a psychiatric diagnosis in the past year (see MacCoon et al., 2012). Participants were also excluded if they had any history of bipolar or schizophrenic disorders, brain damage or seizures. Baseline psychological symptoms reflect the non-clinical range of functioning (Symptom Checklist-90R mean = 0.17, SD = 0.18).

The Five Facet Mindfulness Questionnaire (FFMQ; Baer et al., 2006) served as our measure of trait mindfulness. The measure includes 39-items that are rated on a 1 to 5 point Likert-type scale assessing five facets. As discussed above, the FFMQ has several strong psychometric features (Baer et al., 2008). Internal consistency reliability estimates for the FFMQ in the current sample were adequate: α = .89, .92, .89, .79, .80, and .92 for Acting with Awareness, Describe, Non-judge, Non-react, Observe, and total scores respectively. All five subscales as well as a total score composed of all items were used in analyses. A higher score on the FFMQ indicates higher levels of mindfulness.

Psychological well-being was assessed using the 18-item version of the Psychological Well-being scale (PWB; Ryff, 1989). This measure is comprised of items from six domains of well-being including autonomy, environmental mastery, personal growth, positive relations with others, purpose in life, and self-acceptance and has previously shown desirable psychometric properties (Ryff & Keyes, 1995). As a means of data reduction, a total score was computed (α = .81) with a higher score indicating higher levels of well-being.

Psychological symptoms were assessed using the Symptom Checklist-90R (SCL-90R; Derogatis, 1994), a widely used symptom checklist. Items focus on specific symptoms associated with various categories of psychological distress. The measure has been shown to possess adequate psychometric properties (Derogatis & Lazarus, 1994) and demonstrated acceptable internal consistency reliability in the current sample (α = .94). The global severity index (GSI), with higher scores reflecting greater distress, was used by averaging all 90 items.

As a test of convergent validity, correlations were examined at baseline between the FFMQ and measures of psychological well-being (PWB) and psychological symptoms (SCL-90R), two constructs theoretically linked to mindfulness. Discriminant validity was tested by examining change over the course of time in the context of experimental manipulation (as Cronbach and Meehl [1955] recommend). Within-group paired t-tests were first used to examine pre-post change on the FFMQ for the MBSR, HEP, and waitlist groups separately. Analysis of covariance models (ANCOVAs) were then used to assess the relative changes in FFMQ across groups, the most direct test of discriminant validity. In these models, post-test FFMQ scores were predicted by group status controlling for pre-test FFMQ scores. Three sets of models were constructed comparing MBSR with the waitlist control, HEP with the waitlist control, and MBSR with HEP.

Results

At baseline, the FFMQ total score and all subscales showed moderately sized correlations in the expected direction with the PWB (rs = .23 to .50, p < .05). The FFMQ total score was significantly correlated with the SCL-90R GSI in the expected direction (r = −.23, p = .008), as was the Non-judge subscale (r = −.36, p < .001), although the remaining subscales were not significantly associated with the SCL-90R (see Supplemental Materials Table 1).

Pre- and post-test means and standard deviations for FFMQ scores are reported for each group in Supplemental Materials Table 2 along with results of within-group paired t-tests. No significant differences were observed between groups at baseline (p > .05). Significant increases in all five FFMQ subscales and total score were noted in the MBSR condition (Cohen's ds = 0.32 to 0.54, p < .05). The HEP group showed increases on the Acting with Awareness, Describe, and Non-judge subscales as well as on the FFMQ total score (ds = 0.29 to 0.66, p < .05). The waitlist control condition showed an increase on the Non-judge subscale as well as the FFMQ total score (ds = 0.61 and 0.24, p < .05).

Analysis of covariance (ANCOVA) models were used to examine change on FFMQ scores in active groups (MBSR and HEP) relative to the waitlist condition (i.e., MBSR vs. WL, HEP vs. WL) as well as in active groups relative to one another (i.e., MBSR vs. HEP, Table 1). When controlling for pre-test levels, the MBSR group showed increases on the FFMQ relative to the waitlist group on the Describe, Non-react, and Observe facets, as well as a marginally significant increase on the total score (ds = .44, .52, .56, and .39 for Describe, Non-react, Observe, and total score respectively). When controlling for pre-test levels, the HEP group showed increases on the FFMQ relative to the waitlist group on the Describe and Observe subscales and on the total score, as well as marginally significant increases on the Acting with Awareness and Non-react subscales (ds = .47, .41, .53, .38, .39 for Describe, Observe, total, Acting with Awareness, and Non-react respectively).

Table 1.

Between-group ANCOVAs contrasting change in FFMQ in MBSR, HEP, and waitlist control groups

FFMQ Subscale	Group	Estimate	SE	t	p	d
Act Aware	MBSR vs. WL	0.83	0.97	0.86	0.395	0.26
Act Aware	HEP vs. WL	1.63	0.87	1.88	.064t	0.38
Act Aware	MBSR vs. HEP	−1.01	0.9	−1.12	0.265	−0.1
Describe	MBSR vs. WL	1.48	0.74	2	.049*	0.44
Describe	HEP vs. WL	1.8	0.61	2.95	.004**	0.47
Describe	MBSR vs. HEP	−0.31	0.73	−0.43	0.67	0.06
Non-judge	MBSR vs. WL	0.2	1.14	0.17	0.864	−0.03
Non-judge	HEP vs. WL	1.49	0.94	1.59	0.117	0.12
Non-judge	MBSR vs. HEP	−1.44	1.06	−1.37	0.176	−0.13
Non-react	MBSR vs. WL	2.4	0.98	2.44	.017*	0.52
Non-react	HEP vs. WL	1.7	0.89	1.91	.060t	0.39
Non-react	MBSR vs. HEP	0.69	0.81	0.86	0.395	0.18
Observe	HEP vs. WL	1.71	0.85	2.02	.048*	0.41
Observe	MBSR vs. WL	2.41	0.93	2.6	.012*	0.56
Observe	MBSR vs. HEP	0.65	0.9	0.72	0.473	0.18
Total	MBSR vs. WL	4.69	2.75	1.71	.092t	0.39
Total	HEP vs. WL	6.56	2.02	3.24	.002**	0.53
Total	MBSR vs. HEP	−2.58	2.65	−0.98	0.333	−0.02

Note: FFMQ = Five Facet Mindfulness Questionnaire (Baer et al., 2006); MBSR = Mindfulness-based stress reduction (Kabat-Zinn, 1994); HEP = Health Enhancement Program (MacCoon et al., 2012); WL = Waitlist control; SE = standard error; t = t-value from ANCOVA models; d = Cohen's d reflects the standardized mean difference in within-group pre-post difference scores (i.e., MBSR vs. WL, MBSR vs. HEP). MBSR vs. WL with 67 degrees of freedom, HEP vs. waitlist with 69 degrees of freedom, MBSR vs. HEP with 67 degrees of freedom.

ns = 34, 36, and 36 for MBSR, HEP, and waitlist groups. Group listed second was coded as the reference group in each set of models (e.g., WL in MBSR vs. WL models).

^t< .10

^*< .05

^**< .01

*** < .001

In contrast, no statistically significant differences were observed between the two intervention groups (MBSR, HEP) in post-test scores controlling for pre-test (Table 1). Cohen's ds reflecting these differences ranged in absolute value from .02 to .18 (Supplemental Materials).

An anonymous reviewer suggested examining whether the intercorrelation between FFMQ subscales changes in response to intervention. In particular, this reviewer noted the possibility that correlations with the Observe subscale (a subscale with known differences in meditating and non-meditating samples; Baer et al., 2008) may increase in the MBSR condition relative to HEP. In order to examine this possibility in a parsimonious way, four regression models were constructed predicting post-treatment Observe scores from the interaction between group (MBSR vs. HEP) and each of the four remaining FFMQ facets (Acting with Awareness, Describe, Non-judge, Non-react) at post-treatment. A significant interaction was detected between group and Acting with Awareness as well as between group and Non-judge, with a direction indicating a stronger positive association between these subscales and the Observe subscale at post-treatment in the MBSR group relative to the HEP group. This finding provides some evidence that responses on the FFMQ are influenced by group membership, despite the absence of mean differences. Further work in a larger sample could productively examine changes in the intercorrelations of FFMQ subscales (and potentially the measure's factor structure) in response to MBSR vs. active comparison conditions.

Discussion

The current study attempted to examine both convergent and discriminant construct validity evidence for a commonly used self-report measure of mindfulness, the FFMQ. Consistent with several previous reports, evidence of convergent validity was seen in moderate sized positive correlations between all FFMQ subscales and total score with a measure of psychological well-being. In partial support of the measure's convergent validity, the FFMQ total score and Non-judge subscale were shown to correlate negatively with a measure of psychological symptoms.

No evidence was found for the FFMQ's discriminant validity, however. Discriminant validity was assessed in the FFMQ's sensitivity to specificity of change in one intervention designed to increase mindfulness (MBSR), one intervention specifically designed to have no effect on this construct (HEP), and a waitlist control condition. Again consistent with previous reports, facets of the FFMQ were shown to increase from pre-test to post-test in response to MBSR, both when examined in the MBSR group alone (i.e., paired t-tests) as well as when examined relative to a waitlist control condition (further evidence of convergent validity). However, it did not appear that this responsiveness was unique to training in mindfulness. Indeed, the HEP condition designed explicitly with mindfulness not included likewise showed pre-post increases on facets of the FFMQ, both when examined in the HEP group alone as well as when examined relative to the waitlist control. Most striking, there were no differences noted in changes on the FFMQ between the MBSR and HEP conditions when compared directly.

The lack of specificity suggested in the current study may not be an outlying occurrence. A recent meta-analysis included k = 20 studies that compared a mindfulness intervention with comparison conditions defined as specific active controls (i.e., control interventions “known or expected to change clinical outcomes,” p 359, Goyal et al., 2014). Somewhat surprisingly, only four of these actively controlled studies reported inclusion of a measure of mindfulness.

Of these, none reported differences in response between mindfulness and active control interventions, as measured on FFMQ, MAAS, or CAMS-R (Goyal et al., 2014).

Cronbach and Meehl (1955) address the complex “implications of negative evidence” (i.e., failing to find evidence of construct validity, p. 295). The three possible interpretations they offer include: (a) the test does not measure the construct, (b) the theory from which the hypothesis arose is flawed, or (c) the experiment is flawed. It is worth considering these three possibilities in the context of the current study.

Regarding the first interpretation, it is possible that the FFMQ does not truly measure the construct of mindfulness, despite the construct validity previously reported for this measure (e.g., Baer et al., 2008). It is possible that behavioral and physiological measures collected in the current design and currently under analysis could be more sensitive to differentially detect changes in MBSR as opposed to HEP. However, independent of these alternative measures, the failure of a well-validated (and widely used) measure of self-reported mindfulness to differentiate between these two interventions, which are intended to differ precisely in their effects on this construct, is important information. The apparent sensitivity of the FFMQ to changes induced by intervention generally (i.e., HEP) contradicted our initial prediction. This may be due to the FFMQ detecting changes in constructs beyond those intended or, as discussed below, due to the HEP intervention producing changes in constructs beyond those intended.

Cronbach and Meehl's second and third possibilities implicate our theoretical understanding of mindfulness (Cronbach and Meehl [1955] point out that the third possibility may be considered a special case of the second, as may be the case here). The relevant interpretation in the current study would be that our interventions somehow failed to relate as expected to the theoretical construct of mindfulness. This could have occurred in one of two ways – either MBSR failed to induce mindfulness or HEP did induce mindfulness. It seems unlikely that MBSR failed to teach mindfulness. MBSR is a well-studied, structured course explicitly designed to teach the practice and cultivation of mindfulness. Further, instructors in the current study were previously involved in an RCT study that successfully demonstrated that an MBSR intervention on healthy participants protects more than HEP against social stress, as measured by immune and inflammatory biomarkers (Rosenkranz et al., 2013). In addition, effects noted on the FFMQ in the MBSR condition relative to the waitlist are comparable to those derived via meta-analysis (i.e., g = .53 for waitlist controlled studies, Khoury et al., 2013).

The possibility that HEP induces mindfulness is intriguing, although also perhaps not the most likely explanation. It is theoretically conceivable that mindfulness, construed as a set of cognitive, affective, and behavioral tendencies towards present-moment awareness, is less dependent on explicit instruction and can be enhanced in more diverse ways than the literature on mindfulness interventions has assumed. Indeed, it may be the case that many psychological interventions increase this mental capacity. In the present design, however, it does not seem unreasonable to suggest that opportunities for learning mindfulness were greater in the MBSR group than the HEP group. The HEP instructors had no background in mindfulness themselves nor did the treatment include any implicit or explicit mindfulness instruction. One would therefore not expect HEP to induce mindfulness any more than a standard aerobics class. Yet, if it is indeed the case that HEP induced mindfulness, the FFMQ may be both valid and unable to differentiate changes in MBSR versus HEP.

Limitations of the current work include the sample size that, although sufficient to detect medium sized effects, does not rule out the presence of small differences in results. Further assessment of changes in mindfulness in large intervention studies, and use of multiple converging measures of the mindfulness construct, would therefore be valuable. In addition, in testing mean differences the current work assumes measurement invariance from pre-test to post-test (Kaplan, 2009); a future study with a larger sample could examine this assumption.

In conclusion, the current study points out an important inconsistency between a widely used measure of mindfulness and the standard theories of mindfulness and MBSR within the context of a moderately large, well-controlled randomized trial. A complication with Cronbach and Meehl's (1955) “negative evidence” is that one cannot definitively determine the source of the difficulty. Cronbach and Meehl (1955) therefore conclude, as we must, that “a fresh body of data” (p. 295) is required to further assess validity.

Our results suggest several fertile areas for future work. Future validation of self-report and behavioral measures of mindfulness should be examined in the context of active control comparisons to disentangle mindfulness from related cognitive and affective constructs and processes. It is important to employ control conditions that contain specific active ingredients intended to be therapeutic and to assess mindfulness in these studies (Goyal et al., 2014). Validity of self-report measures should be further assessed by incorporating convergent measures of the mindfulness construct - behavioral tasks (Levinson et al., 2014); significant other-ratings, observer ratings, teacher ratings; and non-self-report measures of theoretically related constructs (e.g., emotion and attention regulation). More broadly, it will be important to better characterize whether and how trait mindfulness is influenced by non-mindfulness-based psychological interventions (e.g., psychotherapy). As the current study employed a non-clinical population, it would be important to assess whether the discriminant validity of the FFMQ is detected in a clinical sample. It may also be of interest to explore whether the interactions between FFMQ subscales that have been shown to relate to other constructs (Eisenlohr-Moul, Walsh, Charnigo, Lynam, & Baer, 2012; Peters, Eisenlohr-Moul, Upton, & Baer, 2013) function differently following mindfulness training relative to active comparison groups.