Measuring Cognitive Outcomes in Mindfulness-based Intervention Research: A Reflection on Confounding Factors and Methodological Limitations

Abstract

One potential pathway by which mindfulness-based meditation improves health outcomes is through changes in cognitive functioning. Here, we summarize and comment upon three systematic reviews conducted over the last 7 years that have had the goal of identifying the impact of mindfulness on cognitive outcomes. In our analysis, we identified a number of methodological limitations and potential confounding factors that interfere with and limit our ability to interpret the results. In order to gain a granular view of the relationship between mindfulness training and cognition, we report on the following: 1) What do we know? How does mindfulness affect cognition? 2) variable criteria that define an MBI; 2) limitations of assays used to measure cognition; and 3) methodological quality of an MBI trial and reporting of findings. Finally, we offer constructive means for interpretation and recommendations for moving the field of mindfulness research forward regarding effects on cognition.

Introduction

As the field of mindfulness research continues to grow, there is a need to rigorously interrogate the existing data to better understand the advantages and effectiveness of the 8- week Mindfulness-based Intervention (MBI) model and its variations for specific outcomes while also acknowledging its limitations. One potential pathway by which mindfulness-based meditation training may improve health outcomes and promote adaptive behavioral change is through changes in cognitive functioning. Here, we summarize and comment upon three systematic reviews conducted over the last seven years, two of which are currently published [1,2] and one has recently been completed by the authors, all of which had the goal of identifying effects of MBIs on cognitive outcomes. During our systematic review of the existing studies, we identified a number of methodological limitations and potential confounds (e.g., non-specific effects, ceiling effects) that interfere with and limit our ability to make rigorous interpretations about the (actual) effects of mindfulness on cognition. Moreover, confounding factors may also influence both independent (training effects) and dependent variables (measures of cognition) leading to spurious claims. Furthermore, many of these methodological limitations can be generalized to other mindfulness research and psychological research in general. In order to gain a granular view of the relationship between mindfulness training and cognition, we review these issues here and suggest ways to surmount such limitations in the future. The paper is organized into the following sections:1) What do we know? How does mindfulness affect cognition? 2) Variable criteria that define an MBI; 3) Limitations of assays used to measure cognition; and 4) Methodological quality of an MBI trial and reporting of findings.

What do We Know? How does Mindfulness Affect Cognition?

Mindfulness meditation is commonly characterized as a “systematic form of mental training” which may impact certain aspects of cognition. Several theoretical models have been developed to describe the cognitive processes which may underlie meditation or may be impacted by an MBI [3–7]. Two highly cited models [6,7] suggest that mindfulness involves: (1) attention regulation (improvement in stability, alerting, orienting, bias, meta-awareness/monitoring), (2) memory extinction and reconsolidation (memory habits shifting toward more adaptive trajectories), and (3) inhibition/switching [de-centering from ongoing cognitions, inhibitory control (from distraction), flexible engagement on relevant objects of attention and rapid disengagement from objects that are not (or no longer) relevant]. Measuring these cognitive mechanisms and determining that any changes associated with an 8-week intervention can be isolated from non-specific factors have proven challenging.

Chiesa et al. (2011) identified 23 cross-sectional and intervention studies that covered different forms of meditation, ranging from mantra and visualization to Buddhist practitioners of Zen, Shamatha-Vipassana and secularized forms such as Acceptance & Commitment Therapy (ACT), Mindfulness-based Cognitive Therapy (MBCT) and Mindfulness-based Stress Reduction (MBSR). Programs also varied in duration and intensity from a brief 20 min induction (Wenk- Sormaz, 2005) to 3-month intensive meditation retreats requiring 10–12 h of daily practice. This review provides a mix of evidence that mindfulness-based practices improve some aspects of cognition [attention, memory, executive functioning, and miscellaneous]. Out of the 15 prospective controlled studies, the mindfulness-trained groups were found to out-perform a control group in only six of the 17 attention-related outcomes. Many of the non-significant studies were limited by very short duration of training [8,9] or had a clinical population that may be capacity- limited (traumatic brain injury) [10]. Out of the eight cross-sectional studies [11–18], expert meditators (in comparison to controls) were found to have greater attentional capacity in 11 of the 12 assays. In two cases, there was a linear relationship between formal meditation experience and attention-related outcomes [11,18]. Groups of individuals trained in mindfulness also out- performed a control group in six of eight assays of memory (e.g., working memory, memory specificity, retrieval) [19–24], in all three assays of executive functioning [20,24,25], and in 3 further assays of miscellaneous cognition [22,26,27].

Lao et. al (2016) focused on 18 controlled trials of MBCT/MBSR - two of the most extensively employed and evaluated MBIs, published between January 2000 and February 2015. This review found no significant evidence for improvement in attention and executive functions. Some evidence was apparent for improvements in working memory, autobiographical memory specificity, cognitive flexibility and meta-awareness. A total of 15/30 assays were found to reach a threshold Cohen’s d of 0.8 (generally considered a large effect), only 1 of which was related to attention outcomes. The small number of studies and various methodological challenges prevented firm conclusions from being drawn. The authors conclude, “Results call into question the theoretical underpinnings of mindfulness, further highlighting the need for a comprehensive theoretical framework”.

Our recent systematic review currently in preparation was intended to build upon what has already been done and get a more nuanced view of randomized controlled trials (RCTs) of MBIs that included behavioral measures of cognition, either in patient populations or the general public. The review initially aimed to include only those studies that conformed to the standard 8-week MBI format, also referred to as “first-generation” MBIs [28] found in MBSR [29] and MBCT [30]. Using Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, 10,802 potentially relevant studies were screened. Yet, in the process of screening for eligibility criteria with the requisite MBI structure and content and distilling the constructs that were relevant for cognitive outcomes, a total of only eight MBI RCT trials (MBSR n = 6, MBCT n = 2) were identified. Several highly regarded and methodologically sound studies that did not conform to the standard MBI curricula were not included in the review despite the fact that they demonstrated significant impacts of mindfulness on cognition. Given the authors’ expertise in this area, modified inclusion criteria were created by consensus to ensure inclusion of 20 additional studies that used modified MBI formats and investigated cognitive outcomes. Detailed results from this review will be presented elsewhere. In 28 studies from our review, MBIs out-performed the control intervention in 19% of all the cognitive assays.

Although there appears to be some evidence cognition may indeed be impacted by mindfulness, definitive evidence appears to be lacking (see also [31]). One must really understand the nuanced context-specific view and limitations for making broad claims about effects of mindfulness on cognition. The following sections explicate methodological issues and potential confounds we observed in our review that are necessary for consideration before making any claims.

Criteria that Define a Mindfulness-based Intervention

Although there are specific pedagogical differences, the standard MBSR and MBCT protocols contain/include: (1) 26 hours of in-class training through 8 weekly group classes (1.5– 2.5 hours/class) plus one 6 hr. class during the 6th week; (2) Home practice (45 min/day 6 days/wk or 36 hours over 8 wks); (3) Dyad/triad/whole group reflection/sharing about practice experiences inside and outside of class; (4) Four core meditation practices in which a state of mindfulness is maintained [Focused Attention (mindful body scan, mindfulness of breath); Open Monitoring/open awareness (mindfulness of present moment experience); and mindful Hatha yoga] Loving-kindness meditation is sometimes added (see [28]). The standard 8-week model waschosen for the review given its extensive use in clinical centers, thereby facilitating comparison across studies. A large number of initially excluded studies in our review used modified MBI curricula tailored for the general population or specific cohorts (e.g., military, classrooms) in order to test for cognitive changes in healthy individuals not seeking stress reduction. Many of these programs were less than 8 weeks, and most did not use a day-long retreat. There is a common assumption that meditation practice is the core active ingredient of MBIs; therefore, we decided to also include any and all models of curricula delivery that utilized at least two of the core mindfulness-based meditation practices, regardless of their length or composition.

While some studies of MBIs have reported that cognitive outcomes are dependent upon dosage (i.e., practice time) [13,16,21,32–34], others have failed to find an association [15,35–37]. As reported above, cross-sectional studies of long-term meditators with variable amounts of practice time have demonstrated mixed results in cognitive differences from control subjects [13,15,17,19,38–41], suggesting that a linear relationship between dosage and cognitive improvements is unclear. Interestingly, some studies have demonstrated short (e.g., 1 week) mindfulness interventions produce significant cognitive changes [8,20,25,42–44]. For example, Zeidan et al., 2010 demonstrated that 4 × 20 minute mindfulness training sessions (in comparison to an active control condition) promoted significant changes on cognitive tasks that require sustained attention and executive processing efficiency. In contrast, other studies evaluating cognition before and after a single mindfulness state induction (12–15 min) demonstrate mixed results regarding improvement in cognition [45,46], suggesting state inductions may not be sufficient. Therefore, there is no reason to believe/stipulate that the standard 8-week model is adequate or necessary for mindfulness training to achieve effective impact across all individuals. Furthermore, these results also suggest that training-specific effects on different aspects of cognition may exist.

In both contemporary mindfulness and Buddhist pedagogy, repeated meditation practice is thought to increase stable state and trait mindfulness over time [47,48]. Furthermore, much of the scientific literature suggests trait mindfulness plays a mechanistic role in relation to clinical and cognitive outcomes [4,6,7,49]. Although the scientific literature has reported significant correlations between cognitive or clinical outcomes and self-reported trait mindfulness [50,51], the scientific literature for MBIs has not adequately examined the causal relationship between practice time, state and trait mindfulness, and cognitive (or clinical) outcomes. Thus, the act of sitting formally in meditation may not be sufficient for shifting outcomes. Quality, depth of practice, or other non-specific effects encountered in the MBI should also be considered. Other contextual elements of an MBI may also be critical for cognitive outcomes, including reduction of stress or other clinical symptoms as well as enhancement of emotional wellbeing or emotion regulation. Furthermore, there is a strong body of literature demonstrating the power of group process (e.g., interpersonal learning, group cohesiveness) in addressing psychopathology [52], which could also theoretically influence cognitive outcomes. Other elements include the quality of the teacher and his/her ability to embody the teachings (see Crane and Kuyken, this Issue), the therapeutic alliance between teacher and participant, expectations, and reward systems. There is also a paucity of evidence regarding the impact of “informal practice” during normal daily activities (e.g., while washing dishes or when faced with a real-life stressor). Future studies investigating the relationship between formal and informal practice may use techniques such as experience sampling to better inform the field on the factors influencing dosage.

There is growing evidence that different types of meditation engage different brain areas [53–55] and differentially influence psychological and cognitive outcomes [56–58] (also see Singer and Engbert, this issue). For example, Marzetti et al. (2014) demonstrated unique alpha band connectivity pattern interactions between default and frontoparietal brain networks during focused attention (FA) vs. open monitoring (OM) meditation with implications for differential patterns of executive and monitoring functions [57]. Britton et al., 2018 found that an 8-week MBI focused on training FA only engaged mechanistic targets related to attentional control, while a structurally- equivalent MBI focused on OM training only engaged targets related to emotional non-reactivity. Because MBIs contain multiple training techniques and have been demonstrated to influence a wide variety of cognitive functions, it is not feasible to determine the individual influence of each component on any particular cognitive function. Future studies using a single training modality are needed to disambiguate the effects.

In summary, active ingredients of an MBI must be better isolated and evaluated in future studies along with contextual factors in order to determine the specific effects of mindfulness training on cognition. Future research should also consider that modifications to the first- generation 8-week MBI prescription that include shorter classes or more emphasis on movement (i.e., yoga) may be just as efficacious as standard MBIs and may be more relevant to non-clinical populations, especially if there are logistical or population-specific barriers to longer in-class time commitments. For example, Carmody and Baer (2009) observed that clinical outcomes were more strongly correlated with the amount of home yoga practice than with the amount of home sitting meditation practice.

Limitations of Assays used to Measure Cognition

The field of cognitive neuroscience is limited in its existing tools to measure specific cognitive processes and also other more complex or subtle phenomenological changes to one’s general cognition as a result of mindfulness training. An extensive literature on cognitive science has emphasized behavioral measures as preferred assays of cognition due to their objectivity, relative simplicity, and ease of administration, often requiring the participant to only make verbal responses or button presses. Behavioral tasks typically measure differences in reaction times (RTs) between tasks or groups of subjects in an attempt to infer which cognitive processes are occurring [59]. Although there is an extensive literature on methodological work designed to parse apart cognitive processes underlying RT as well as relatively more recent work on computational models of RT tasks (e.g., decision-making tasks), and work on diffusion and accumulator models of RT data, there remains a great deal of variability in rigor of various “cognitive tasks”. For example, two individuals demonstrating similar RTs may be using vastly different strategies to perform the task. Many RT experiments provide little means of determining the processing that occurs between stimulus and response [60], and therefore are not necessarily sufficient to elucidate the exact mechanisms by which the brain is processing information [61].

Complementary use of multi-modal neuroimaging techniques, such as event-related potentials (ERPs) and functional magnetic resonance imaging (fMRI), provide more information about specific neural mechanisms that underlie each cognitive process, can be measured in the absence of overt responses, and provide a means for cross-validation [60,62]. Some behavioral measures have also displayed questionable psychometric properties [63]. In our systematic review, one of the most frequently used tasks was the Stroop test which assesses cognitive interference and inhibition [64]. Studies have demonstrated a lack of convergent validity between different Stroop formats [65], low test-retest reliability [65], and poor internal reliability [66]. The sensitivity and specificity of this measure in identifying deficits in disorders has also been questionable under certain circumstances [67]. Additionally, cognitive tasks including the Stroop may be confounded by intelligence and educational levels [67]. No study in our review reported significant effects on the Stroop task; however, given the poor psychometric properties of the test, the lack of significant effects could be due either to inefficacious training or to the test itself. Most cognitive tests have multiple variations (e.g., variable/fixed ITIs, multiple outcomes, length of task, variable stimuli) with widely varying psychometrics. Therefore, variability in task design and associated findings must be interpreted with respect to the granularity afforded by such methodological attributes of these tasks.

Another issue of concern is the wide variety of cognitive tasks used across studies. Whereas psychopathology has a handful of gold standard measures (e.g., Hamilton and Beck Depression scales), cognitive neuroscience has multiple tests for each theoretical construct. Furthermore, each cognitive test often has multiple variants, and the data from any particular test can be analyzed in a variety of ways which can generate different results. This problem facing the field of cognitive neuroscience prompted NIH to develop a Toolbox of tasks to aid with standardization. Few, if any, studies in our review utilized this toolbox. We suggest inclusion of these tasks in the future to aid reproducibility and facilitate the comparison of results between mindfulness studies as well as in the wider field of cognitive neuroscience.

Some of the existing literature examining effects of mindfulness on cognition suggests that mindfulness training does not exert its largest effects on tasks assessing speed and accuracy [2,68,69]. Rather, mindfulness instructions typically include taking time to observe present- moment experience in a non-judging and decentered way, and therefore may have greater effects on meta-cognitive abilities. Repeated practice of mindfulness skills is likely to influence other qualities of attention, such as interoception [the ability to observe and report upon internal experiences (cognitions, sensations, emotions)], and would function well as a primary outcome in RCTs [30,70,71]. Objective measures of cognition (e.g., EEG, fMRI) have the potential to cross- validate behavioral changes in attentional allocation, efficiency, inhibition, and bias. Further work may be necessary to employ objective measures and identify better metrics for cognitive mechanism engagement and change as a result of practice.

Methodological Quality of an MBI Trial and Reporting of Findings

In our review, several studies were initially excluded because they did not conform to the standard clinical trial methodology. For example, some of the studies did not use a standard randomization strategy. Studies with students in classrooms or military platoons required participants to be randomized by cohort rather than individually. Although this is an acceptable randomization method, it introduces additional confounds which need to be considered when interpreting the data. Other methodological design issues that influenced interpretation of the data included: (1) Clinical studies sometimes include cognitive tasks as secondary outcome measures and were not adequately powered to detect group differences; and (2) Control groups do not match the non-specific contextual elements of the MBI (e.g., group effects, expectation effects, teacher competency, demographics, and education level).

Most MBI studies of cognition to date have been small (n=12–30 subjects/arm) and are likely underpowered to detect real differences in cognitive outcomes unless there is a pre-existing cognitive deficit. When studying cognitively normal populations, there are likely ceiling effects for any significant change in these processes, especially in comparison to an active control intervention that may also confer benefits. Indeed, many studies found no significant between- group differences. Some observed within-group improvements, so it is possible that larger sample sizes may reveal between-group effects. Furthermore, it may take more than 8 weeks for certain cognitive processes to significantly develop, especially in a healthy population. Follow-up studies of cognition are rare. Therefore, future studies should examine how continued practice may contribute to sustained benefits or further improvements longitudinally. Care should be taken when digesting reviews or meta-analyses that do not distinguish between trials focusing on healthy individuals and those focusing on clinical patients.

Similarly, the context of the subjects’ condition or environment needs to be interpreted carefully. For instance, several studies reporting group differences have found that the performance of the controls declined over time, while the scores of the MBI participants remained stable (e.g., [21]). These results are interpreted as the protective effect of MBIs against stress- related cognitive lapses. More studies with non-stressed individuals are needed in order to disambiguate the stress reduction effects of MBIs from actual enhancement of cognition. How and where people practice and learn mindfulness (e.g., consecrated spaces, monastic settings, secular classrooms) may very likely influence outcomes. Yet, such contextual elements have yet to be explored. To further methodological rigor, clear systems are required for assessing and reporting the integrity/fidelity of the intervention (Crane, this issue). It needs to be clear what the subjects are practicing (including informal practice demands), and there must be standard qualifications of the teacher.

How one advertises for a study may influence expectation or placebo effects. For example, one provocative study demonstrated that subjects recruited for a “Brain training and cognitive enhancement study” outperformed individuals recruited to “Participate in a Study” for course credit, despite the fact that all subjects underwent the same training and testing protocols [72]. One possibility is a selection bias in which different types of people might respond to specific “languaging” used in recruitment materials, an observation that we have observed in our own studies. An alternative interpretation is that expectation/placebo effects impact performance. Therefore, recruiting strategies should be reported and considered when comparing MBI studies of cognition.

Conclusions

Challenges remain for properly assessing the effects of mindfulness on cognition. Cross- sectional studies of advanced meditators have clearly demonstrated some cognitive advantages in comparison to matched healthy control subjects. Yet, without longitudinal studies, it remains unclear whether these observed effects are purely correlational and not causal. The data suggest that cognitive outcomes may vary depending on type of meditation practice emphasized relative to the cognitive outcome being measured (see [56]). Because many of the findings in MBI studies are equivocal at best, improved assays and methodological rigor are encouraged to improve cross- validation and clearly estimate the performance edge that mindfulness training offers (if any) in comparison to interventions that include all of the non-specific contextual elements, but not the active ingredient – mindful awareness. We encourage investigators to state a priori hypotheses how cognition may be enhanced, returned to baseline, or preserved from degradation so these studies can be more fairly compared. MBI methodology, including variations in content, context (contemporary MBIs vs. Dharma teachings), delivery format (e.g., direct teacher transmission or mobile apps), or dosage (in formal and informal practice), need to be considered along with the first-generation 8-week model and with respect to the limitations of the population (e.g., interest/motivation, accessibility, baseline cognitive ability). A focus on isolating the active ingredients may reveal the need to measure changes outside the classroom or off the cushion. The challenge is to better isolate the specific mechanisms supporting mindful awareness over non- specific psycho-education support groups, wait-list, and treatment-as-usual groups. Indeed, some of the non-specific effects (e.g., placebo, group effects) may be more specific to the therapeutic model of an MBI and could be incorporated into the model instead of controlled and disregarded as non-specific. Observable cognitive changes may be also be limited by the assays available to measure such change. Novel tasks or variations of cognitive measures need to be constructed (see Hadash and Bernstein, this issue; Van Vugt and Sivakumar, this issue) that target mechanisms specific to mindfulness practice. Finally, replications of data are needed in high-powered studies using appropriately sensitive measures. Given all these recommendations, we feel that measurement of cognition in mindfulness research will likely benefit greatly.

Highlights.

• The effects of MBIs on cognition remain variable and depend on context

• Methodological issues in MBI research must be considered for proper interpretation

• Informed readers should consider variability in criteria that define an MBI

• Informed readers should consider limitations of assays used to measure cognition

• Informed readers should consider variability in methodological quality of MBI trials