Conceptual and Methodological Issues in Research on Mindfulness and Meditation

Abstract

Both basic science and clinical research on mindfulness, meditation, and related constructs has dramatically increased in recent years. However, interpretation of these research results has been challenging. The present article addresses unique conceptual and methodological problems posed by research in this area. Included among the key topics is the role of first person experience and how it can be best studied; the challenges posed by intervention research designs in which true double-blinding is not possible; the nature of control and comparison conditions for research that includes mindfulness or other meditation-based interventions; issues in the adequate description of mindfulness and related trainings and interventions; the question of how mindfulness can be measured; questions regarding what can and cannot be inferred from self-report measures; and considerations regarding the structure of study design and data analyses. Most of these topics are germane to both basic and clinical research studies and have important bearing on the future scientific understanding of mindfulness and meditation.

I. Introduction

Mindfulness meditation and other meditative practices are increasingly popular and a growing number of people world-wide are using them in their personal lives to produce a variety of outcomes ranging from being ten-percent happier (Harris, 2014), to being a little more relaxed and less anxious, to coping with chronic pain and much more. Those who use some form of meditation on a weekly basis in the United States have been estimated to number more than 20 million, based upon a 2007 survey, and likely to be even more today (Barnes, Bloom, & Nahin, 2008). There presently exist a number of popular magazines, books, and websites concerned with meditation, and the spiritual traditions in which meditation practices have played a key role. Meditation and related practices have been taught within non-sectarian contexts, in educational settings, hospitals, and clinics (Duerr, Zajonc, & Dana, 2003; Hart, 2004; Salmon, Santorelli, & Kabat-Zinn, 1998; Shapiro, Brown, & Astin, 2011). There has also been an accelerating increase, particularly since the late 1990s, in publications related to mindfulness (J. Mark G. Williams & Kabat-Zinn, 2011). Despite this proliferation of research, findings are often difficult to interpret, in large part due to the unique conceptual and methodological problems posed by research in this area. It is the purpose of the present paper to review these problems and suggest conceptual and methodological approaches to addressing them. Other reviews and studies focusing on the health impact of meditation (Kok et al., 2013; Ospina et al., 2007) and the problems in the measurement of mindfulness (Brown, West, Loverich, & Biegel, 2011; Grossman, 2011) have appeared in the past 7 years but a comprehensive overview of methodological issues in this area has been lacking.

II. State, trait, procedure

Although the topic—meditation—might be the unifying theme, investigations vary widely in their research targets. Some studies have focused on the state of the mind and brain produced by meditation practices (e.g., (Davidson & McEwen, 2012; Lutz, Greischar, Rawlings, Ricard, & Davidson, 2004; Tang & Posner, 2014). These studies are quite varied and range from research with long-term practitioners (e.g., Lutz, Greischar, Rawlings, Ricard, & Davidson, 2004) to complete novices who are meditating for the first time for just a few days (e.g., Creswell, Pacilio, Lindsay, & Brown, 2014). The focus of these studies is on what happens during the actual practice of a particular form of meditation. Some of the earliest studies in the scientific literature were focused on state effects and simply examined within subject comparisons of the period before, during and immediately after the practice of a particular form of meditation (e.g., Wallace, 1970). Of course, studies that test adept practitioners during the meditation state are not exclusively evaluating state effects since the impact of the specific meditation practice that is implemented will depend upon the cumulative effects of practice and thus, these studies should properly be considered studies of state X trait interactions.

Other studies have contrasted long-term practitioners with various controls on measures obtained outside of formal meditation, in order to probe putative trait effects of practice (i.e., those effects that are assumed to endure; e.g., Lutz, McFarlin, Perlman, Salomons, & Davidson, 2013). These studies are especially important since the goal of any form of meditation practice is to transform everyday life. If state effects during the practice are not accompanied by enduring effects that persist outside of formal practice, there would little point in devoting time to practice. There are two basic strategies for pursuing such questions. One is to compare long-term practitioners to novices, with the assumption that whatever differences are found can be attributed to the long-term meditation practice among the adepts. However, there are also likely important self-selection biases that may be important since the decision to dedicate so much time in one’s life to practice is rare and likely associated with a constellation of pre-existing individual differences. There is no available research on this, given that this would require long-term longitudinal studies to assess the antecedent predictors of long-term practice.

Another way in which trait effects of mindfulness have been studied involves examination of the correlates of variation in scores on questionnaires that putatively measure mindfulness (e.g, Creswell, Way, Eisenberger, & Lieberman, 2007). Although such studies have provided some interesting data on the correlates of mindfulness questionnaires, it is not entirely clear how to interpret these data, since the construct validity of mindfulness questionnaires themselves is problematic. This issue will be treated in a later section.

Mindfulness is sometimes used in the psychological and neuroscientific literature to refer to a procedure engaged in by participants, or a set of instructions provided to participants. As we suggest below, and is articulated in the accompanying article by Lutz et al., there are a variety of practices that get subsumed under the term “mindfulness” and it is likely that they are not doing the same thing. Moreover, the extent to which participants are complying with the instructions is often variable. Furthermore, it is not entirely clear how to measure participant compliance, particularly at early stages of practice. For this reason, investigators will often obtain self-reports of practice frequency and duration (e.g., Soler et al., 2014) even though the quality of their mental states when they are practicing can be quite variable. The use of ecological momentary assessment (e.g., Mrazek, Franklin, Phillips, Baird, & Schooler, 2013) to probe for qualities such as mind-wandering should be considered, as these methods are likely to provide more nuanced and granular depictions of quality of the practitioner’s mental state than retrospective questionnaires (see Section VII). Such measures can be used in conjunction with other outcome measures to examine whether they mediate and/or moderate the impact of the practice on the studied outcome measure.

III. First, second and third person perspectives

Francisco Varela (Lutz, Lachaux, Martinerie, & Varela, 2002; Varela & Shear, 1999) called attention to the importance of first person experience and the distinctions among first, second and third person perspectives in research on the nature of the mind. First-person perspectives refer to those typically measured by reports from the subject her or himself. Third person perspectives are reflected in objective measures made by an experimenter with no prior relationship to the subject. Second-person perspectives involve measures based upon reports on the subject by another individual knowledgeable about the subject. For example, second-person measures could be based on reports from the subject’s spouse or teacher or persons in some other type of close relationship with the subject. If we wish to seriously understand the nature of lived experience from a first-person perspective, Varela argued that we need a refined instrument of introspective access and reasoned that meditation training—literally becoming more familiar with the nature of one’s own mind—was a methodological necessity to adequately capture the subtlety of conscious experience.

A key target of contemplative practice is awareness itself. According to the contemplative traditions, the clarity and range (or spaciousness) of awareness will be impacted by contemplative practice. Moreover, the quality of awareness will in turn have impact on other mental processes such as perception and learning. To investigate these questions will require that we obtain first person measures of experience and third person measures of the processes hypothesized to be impacted by variations in experience. In a classic example of this approach, Varela and his collaborators (Lutz et al., 2002) trained participants to report on their experience in the immediate seconds just prior to the delivery of a stimulus and they found systematic relations between reports of experience and neural activity evoked by the stimulus. This study underscores the value of creatively combining first and third person methods.

In several theoretical articles, Varela and colleagues (e.g., Varela & Shear, 1999;Varela, 1996) argue that meditation training can benefit first person accounts by enabling a more attentive stance toward experience, thus resulting in a more granular description of experience. The implicit claim here is that reports of conscious experience derived from minds that have not had this form of training will be tainted by distraction and thus be compromised with respect to both reliability and validity. An implication of this perspective is that relations between measures of first person experience and third-person measures of brain function should be more closely related for those with contemplative training compared with those who have not received such training. Although this hypothesis has not received systematic study, it can readily be empirically examined.

A related implication of this perspective is that self-reports on mindfulness questionnaires may reflect very different processes at different levels of training. We might expect that among individuals with no or little mindfulness practice, the quality of the data from such reports will be different from that obtained from longer-term practitioners who have developed considerably more familiarity with the nature of their own experience.

IV. The challenge of conducting intervention studies in the absence of double-blind procedures

The classic experimental design to rigorously evaluate the impact of an intervention on a particular outcome was developed in the context of biomedical research and involves the use of double-blind placebo controlled designs. This form of design has been applied to virtually all major forms of medical intervention ranging from pharmaceutical to surgical interventions. Unfortunately, this kind of design simply is not possible with meditation-based interventions because of the obvious fact that participants will know if they are assigned to a meditation condition and thus cannot possibly be kept blind to the nature of the intervention. This fact is partially responsible for the poor quality of clinical trials of meditation that have appeared in the scientific literature and is one important reason why recent meta-analyses of the clinical impact of meditation have reported so few rigorous studies that are judged to be methodologically sound (Goyal et al., 2014). This issue will be addressed in more detail below, but for now it is important to note that active comparison treatments are required.

V. Rich Description of the Intervention

A particular problem for the interpretation and integration of research on mindfulness and other meditation practices and interventions concerns the variety of possible traditions and trainings that have been studied. Meditation practices taught within different traditions, and even the same tradition, vary, as do various interventions labeled as mindfulness-based. Within the published research literature on these practices and trainings, the specific nature of the practice or training under investigation often remains under-specified, making both generalization and comparisons across studies difficult or impossible.

Some practices and trainings involve maintaining mental focus on a particular sensation (e.g., of the breath), while others involve focus upon a sound, or auditory mental image, the silent repetition of particular words or phrases (e.g., as in loving-kindness meditation), a visual object, or a visual mental image. Other approaches attempt to broaden the attentional field without a preference for selection of any focus, releasing attention gently and without judgment whenever it is pulled to any particular mental experience. Lutz, Slagter, Dunne, & Davidson (2008) propose a framework for understanding these different forms of meditation, in their broad distinction between “focused attention” versus “open monitoring” practices. According to Lutz and colleagues, focused attention meditation involves the directing and sustaining of attention on a selected object (e.g., breath sensations), as well as detecting mind wandering (thoughts or other mental phenomena unrelated to the focus), or other distractions. When this is detected, the practice involves disengaging attention from the distraction and gently (without self-judgment regarding the distraction) shifting attention back to the object of focus.

In most traditions in which it is engaged, open monitoring meditation is typically practiced after some stability of attention regulation is achieved via focused attention meditation. Open monitoring, as described by Lutz and colleagues (2008) involves no explicit focus on objects, maintaining an alert “openness” to whatever arises in the mental continuum. It can also involve awareness of the conscious field itself in which mental phenomena arise, something that contemporary psychology would term meta-awareness. This calm, nonreactive awareness includes all sensations, images, thoughts, and feelings, as well as automatic cognitive-emotional interpretations or associations that arise in the stream of consciousness. However, the practitioner does not dwell upon or get lost in these experiences or associations. Rather, they are allowed to enter and pass out of mind while remaining alert and aware of the conscious field itself. Thus, open monitoring meditation emphasizes the self-monitoring skill developed initially through focused attention meditation practice, and cultivates moment-to-moment meta-awareness.

Given the variation between different meditation practices, and even within those that are described as mindfulness-based approaches (Chiesa & Malinowski, 2011), a challenge for reports of research in this area is adequate description of what practice or training was actually performed. This is of particular importance, since different types of meditation training may have differential impact upon various outcome measures (Goyal et al., 2014). If the focus of research is a contemplative practice intervention, the particular practice instructions provided to participants should be described. When such instructions are extensive, they can be provided in an appendix or online resource linked to the paper. It is also important to know whether the intervention is based upon, or derived from a particular contemplative tradition or specific practice within such tradition. This information helps in understanding the source of the intervention, and allows integration of study results into a broader historical and contemporary contemplative practice context. Cross-sectional studies of experienced meditators, in comparison to meditation-naïve persons or novice practitioners, also require detailed description of the particular practice being considered. Especially problematic can be interpretation of studies that combine into a single meditator sample persons who practice within different meditation traditions. In some cases, the particular practice traditions of such mixed-sample participants may be confounded with other variables of interest, such as length or amount of practice experience (e.g., Luders, Clark, Narr, & Toga, 2011), making it impossible to disentangle practice type versus experience/expertise contributions to the variance in dependent measures.

Another meditation research domain for which rich description is particularly important is where intervention or training has been modified for a particular age group. Although systematic efficacy studies are just beginning to emerge, there have been a number of feasibility studies of interventions such as Mindfulness-Based Stress Reduction (MBSR) with children and adolescents (for review, see Burke, 2009). These studies have often provided little detail regarding how the standard MBSR training may have been modified to accommodate developmental considerations, particularly for younger children (see Flook, Goldberg, Pinger, & Davidson, 2015) for one recent example). Such detail is essential for any attempts at replication or comparison across published study results. Such detail in description of age-related modifications in particular contemplative interventions or trainings is also important in research involving older adults. For example, there are now available a small number of studies evaluating MBSR or related mindfulness-based interventions for older-adult caregivers of persons with dementing illness (for review, see Kaszniak, 2011). The reports of some of these studies note that standard training protocols were modified to make them more suitable to these older-adult samples of participants, but typically provide insufficient detail regarding the specific nature of the modifications. A related issue concerns the importance of documenting the age range of study participants. There appears to be broad awareness of this importance for studies involving children and adolescents. However, meditation research involving adult participants may be more variable in this regard. This is despite the fact that there are reliable differences between adult age groups in such dependent variables of interest in contemplative studies, such as those assessing attention (e.g., Clapp, Rubens, Sabharwal, & Gazzaley, 2011; van Leeuwen, Müller, & Melloni, 2009), emotion, and emotion-regulation (for review, see Kaszniak & Menchola, 2012).

Greater detail of description than is typically provided in published studies is also needed in regard to mindfulness or other meditation instructors. Who are the instructors, and what is their training, certification, and practice experience? An important consideration concerns the qualifications of those who teach the meditation intervention employed in a particular study. The Center for Mindfulness in Medicine, Healthcare, and Society at the University of Massachusetts Medical School provides a training program and certifies MBSR instructors (http://www.umassmed.edu/cfm/oasis/index.aspx). Qualifications that are considered in evaluating applicants for this training are personal psychological development, meditation training and regular practice, including silent retreat attendance, yoga or other bodywork training, and professional training and graduate work in a related field (e.g., psychology, education, medicine, etc.). There have also been recent efforts toward establishing organizations for Western meditation teachers (e.g., the American Zen Teachers Association http://www.americanzenteachers.org/). Such organizations often strive to enhance teacher qualifications by fostering dialogue among teachers and providing access to information about meditation practice and teaching. However, teachers within particular meditation tradition are often authorized to teach individually by their own teacher, and there does not presently exist anything comparable to national or regional licensing or certification. In addition, at this point there exists little research on the relationship between meditation teacher characteristics, qualifications, and outcome of meditation training. Such future research would be helpful in guiding selection of teachers for any planned study.

When active intervention control or comparison groups are part of the study design (see discussion below), it is important to obtain comparable measures on instructors. Indices of teacher training, certification, and experience should be provided for both the contemplative and comparison intervention teachers. Other relevant teacher and teaching measures include participant perception of the teachers’ and interventions’ credibility and expectancy. Credibility of interventions can be assessed using a slightly modified version of the Credibility and Expectancy Questionnaire (CEQ; Devilly & Borkovec, 2000). The CEQ is a clinical measure that was designed to assess the expectancy and credibility of interventions perceived by patients in therapy for the treatment of anxiety, and has been used to assess initial equivalence among compared intervention conditions. Minor modifications can made to the CEQ questions wording to make them appropriate to contemplative intervention studies.

Relatedly, there is a need for explicit assessment of fidelity in delivery of a particular meditation or control intervention, for each instructor. This can be accomplished by, for example, videotaping of several exemplar intervention sessions, and submitting these videotapes for quantitative evaluation of fidelity by highly experienced instructors of the particular intervention.

VI. Measuring Mindfulness: Conceptual and Methodological Challenges

The term mindfulness is an English translation of the Pali word sati, having meanings that have been variously translated as including attention, awareness, retention, and discernment. There has been no single meaning of mindfulness on which all scholars have agreed (see Bodhi, 2011; Dreyfus, 2011; Dunne, 2011; Gethin, 2011). Mindfulness meditation practice, as examined in recent research, typically derives from several different Buddhist traditions and involves aspects of both focused attention and open monitoring, as defined by Lutz et al. (2008). In typical mindfulness practice, focused attention initially rests upon an object such as the breath, while a monitoring function detects, and brings awareness to when the mind has wandered, and repeatedly, without judgment, brings it back to the object. As skill in this monitoring aspect develops, the practitioner learns to observe the functioning of his or her own mind in a calm and unattached manner. Such observation is held to allow insight into the causes and conditions of behavior and suffering (Gunaratana, 1993). In open monitoring, the specific object of mindfulness is dropped and the meditator is instructed to be aware of whatever might arise in the present moment. Shapiro & Carlson (2009) note that mindfulness meditation involves intention, attention, and attitude: Intention refers to the personal vision for why meditation is practiced, which may be dynamic and evolving as practice continues. Attention in mindfulness meditation “…is discerning and nonreactive, sustained and concentrated, so that we can see clearly what is arising in the present moment…” (Shapiro & Carlson, 2009, p. 10). Attitude refers to qualities of openness, acceptance, curiosity, and affection in the attention that is brought to present experience.

Jon Kabat-Zinn (1994), the originator of MBSR, defines mindfulness meditation as a process of paying attention on purpose, in the present moment, and non-judgmentally. MBSR is a relatively well-defined, eight-week, systematic training program in which the central component is said to be mindfulness meditation. MBSR is intended to provide a secular approach to teaching people how to use their resources and abilities to respond more effectively to stress, pain, and illness (Kabat-Zinn, 1990).

As different varieties of mindfulness-based practices enter the mainstream of our modern culture, scientists are being called upon to provide measures of mindfulness. The measurement of mindfulness poses many difficult conceptual and methodological challenges. As we noted above, questionnaire measures that request individuals to provide accounts of their own experience depend, in part, on their practice and experience in interrogating their own minds and we view this as a skill that will change with training. Moreover, as individuals become more mindful of their internal experience, including their thoughts and emotions, they may actually become more dysphoric, at least in the short-term, as they notice how chaotic their minds actually are. When their attention is focused externally and they are occupied in various tasks, fewer resources are available to devote to internal awareness and thus directing awareness internally can be unsettling at the outset.

There are two consequences of this perspective. One is that reports of anxiety and dysphoric affect may actually increase in the early stages of practice, because individuals become more acutely aware of this content in their minds. Secondly, the factors that contribute to reports of experience in participants with only brief training in mindfulness practices may include a substantial reliance on demand characteristics since the mental processes for internal attention are not well-developed.

Although self-report measures of mindfulness evidence some validity in their ability to predict beneficial clinical outcomes (Khoury, Sharma, Rush, & Fournier, 2015), alternatives to self-report are worth exploring. What other options are available to measure mindfulness other than self or other report? One option is experience sampling or ecological momentary assessment (EMA; see Dockray et al., 2010). For example, Killingsworth & Gilbert (2010) using EMA procedures to examine mind wandering and its relation to unhappiness. By asking participants to indicate what they are doing at the time of a probe, and to report the degree to which they are attending to what they are doing, a momentary measure of at least some aspects of mindfulness can potentially be obtained. Such measures are likely less subject to the types of biases that plague retrospective self-report methods. Such methods are just beginning to be incorporated into studies on the impact of mindfulness (see Mitchell et al., 2013 for one of the few examples). Many methodological strategies are afforded by experience sampling methods and should be tried in studies of meditation. For example, the time course of emotional responding can be examined by probing with sufficient frequency following an emotional event. Such a strategy would enable determination of the rapidity of recovery following an emotional event, an important constituent of resilience (Schuyler et al., 2012). We would predict that mindfulness meditation should lead to faster recovery following an emotional event, particularly following a negative emotional event. In addition, experience sampling could be used to probe for shifts in the prevalence of particular kinds of emotion. We would predict that emotions such as anger and anxiety should decrease over time with mindfulness practice experience, though the rate at which these negative emotions might decrease is likely variable across individuals and interacts with dispositional characteristics that precede any formal meditation practice.

Experience sampling could also be used to probe for mind wandering, in a manner similar to that used by Killingsworth & Gilbert (2010). High frequency experience sampling may impose considerable burden on participants and thus might not be feasible in certain contexts. An alternative that is intermediate between typical retrospective self-report measures and ecological momentary assessment is the daily reconstruction method of Kahneman and his colleagues ((Kahneman, Krueger, Schkade, Schwarz, & Stone, 2004). With this method participants systematically reconstruct their activities and experiences of the preceding day with procedures designed to reduce recall biases.

Many investigators use various forms of cognitive and attentional tasks to make inferences about mindfulness. In a recent example, Morrison, Goolsarran, Rogers, & Jha (2014) examined a brief program of mindfulness training with university students on measures of attention and working memory. Before and after seven hours of mindfulness training, students were given tasks designed to probe various aspects of attention and working memory. They found gains in overall accuracy on a sustained attention to response task (SART), along with decreased reaction time variability, compared with a wait list control group. They also found increases in self-reported “on-task” performance. They did not find any significant differences between groups on the working memory tasks.

In new work from Davidson’s laboratory, Levinson, Stoll, Kindy, Merry, and Davidson (2014) developed a breath counting task that requires participants to press one button with each breath cycle and a second button every 9^th breath. In a series of studies designed to assess the reliability and validity of this procedure in more than 400 participants, correlations were found between the initial button press and objective measures of respiration rate that exceeded .95. In response to the second type of button press where participants must keep numerical track of their breathing, they tend to make mistakes and reports of mind-wandering are correlated with errors. The number of correct groups of nine breaths that are accurately tracked is modestly correlated with various self-report measures of mindfulness. In addition, long-term meditators were more accurate in breath counting compared with gender and age-matched controls. Further evidence of validity was found in studies that revealed that greater counting accuracy was associated with better mood and less attentional capture by previously rewarded stimuli, an operational measure of non-attachment. Additional evidence is needed to establish the validity of this method and to determine if it is more predictive of certain types of behavior compared with self-reported mindfulness. Based upon the types of issues we surveyed earlier, we would expect that, particularly in novice meditation practitioners, this objective measure will be more valid than self-report measures of mindfulness in predicting other types of outcomes, such as performance on objective measures of attention such as the SART.

Although there have been many studies of mindfulness that use neuroimaging or EEG measures, it is still not clear which parameters of MRI or EEG should be measured and which patterns meaningfully reflect mindfulness. For this reason, we suggest that it would be hazardous to use any biological measure as a proxy for mindfulness at this stage in the development of this field. This would constitute a version of “reverse inference”, a pitfall common to the entire field of neuroimaging. Reverse inference refers to using measures of brain function as proxies for specific psychological functions. Many commentators (e.g., Poldrack, 2011) have cautioned about the hazards of using imaging measures to make inferences about psychological function and in the case of mindfulness, which is not consensually-defined to begin with, those cautions are particularly significant.

VII. Measuring meditation practice time

Estimating the contribution of meditation practice time is itself problematic. An ideal strategy would involve randomized assignment of participants to a particular, or various different, meditation trainings, and assessing longitudinal changes in particular dependent variables at different points in the training. Little such research has been conducted to date, and this approach is generally feasible only for assessing effects of experience over relatively short durations. Teachers within various meditation practice traditions would likely agree (e.g., Goldstein, 2003) that many of the changes associated with meditation may require several years of regular practice to manifest, and several-year longitudinal randomized controlled studies are both difficult to conduct and prohibitively expensive. Consequently, most studies examining meditation training/practice duration are cross-sectional, correlating practitioner length-of-training self-report and various behavioral and biological measures (e.g., Grant, Courtemanche, Duerden, Duncan, & Rainville, 2010; Lutz et al., 2004). However, as Davidson (2010) points out, absent measures at different points along the course of training, it is not possible to disentangle training effects from those contributed by variation in individual predisposition for persisting in such training.

Most meditation practitioners who have been practicing for more than several months and who have received some formal instruction typically practice more than one style of practice. When estimating dosing it is important to estimate the time a practitioner devotes to each specific form of practice in which she regularly engages. Moreover, differentiating between regular home practice and retreat-time practice which is typically considerably more intensive may be an important distinction and if at all possible, should be examined separately.

Davidson (2010) also raises the issue of how best to measure the quantity and quality of practice and training that occurs outside of formal meditation practice periods. Periods of informal practice clearly do occur, as, for example, when the practitioner remembers to attend nonjudgmentally to mental phenomena that arise in a stressful situation. Research participants in cross-sectional studies eliciting self-reports of, for example, total hours of meditation practice, are unlikely to include such informal practice epochs into their estimates. Davidson (2010) notes that such informal practice in the real world may have effects of equal or greater importance to what occurs in formal practice sessions. Experience sampling methods may be particularly helpful in gathering data on informal practice in the context of daily life.

XIII. Age- and cultural-appropriateness of measures

As noted above, studies of meditation include a wide range of ages from children (Black, Milam, & Sussman, 2009; Davidson et al., 2012) to older adults (e.g., Marciniak et al., 2014). Just how mindfulness may be differently expressed at different ages has not been considered. Age appropriate measures of mindfulness and related constructs will ultimately be required and need to be developed, particularly for longitudinal studies in children where rapid developmental changes occur. Similarly, ethnic and cultural variations need to be taken into account in the design of both interventions and in choosing the most relevant targets for outcome measures. For example, in light of the increased prevalence of essential hypertension in African-Americans compared with other ethnic groups, targeting blood pressure as a potential outcome would be clinically relevant for this group. Recent evidence suggests that a simple mindfulness of breathing intervention provided in school over the course of three months may have an impact on ambulatory blood pressure among African American adolescents (Gregoski, Barnes, Tingen, Harshfield, & Treiber, 2011). A survey of more than 20,000 Americans found wide variation among different ethnic groups in the use of prayer, meditation and other spiritual practices for health purposes (Gillum & Griffith, 2010). In future studies, such information will be important to take into account in helping to language mindfulness-based interventions in ways that are acceptable to different ethnic groups, if the goal is increased adherence to the practice among these groups.

IX. Comparison and control groups

The issue of control and comparison groups is a particularly important one in research on meditation. The failure to utilize rigorous control or comparison conditions is a key reason why recent reviews have failed to find much convincing evidence for the clinical efficacy of mindfulness-related interventions (e.g., Goyal et al., 2014). It is not possible to blind the participant to which group they are assigned. Studies evaluating the impact of meditation-based interventions for clinical outcomes that wish to attribute change to the meditation or mindfulness components per se must therefore use more than a wait-list control. Participants in wait-list control conditions know they are in the control group and those assigned to the active meditation condition know they are in the meditation condition. Thus, differential demand characteristics are likely to be present. This is likely important not only in clinical studies but also in basic research studies where long-term practitioners are compared with novice practitioners. Such basic research studies may necessitate other types of designs that include offering financial incentives to novices to perform better on particular types of tasks that are hypothesized to be performed better by meditators (see Brefczynski-Lewis, Lutz, Schaefer, Levinson, & Davidson, 2007 for example of such a design). There is no single best control or comparison condition. To adequately and rigorously address particular types of questions, it is often necessary to include several control or comparison conditions, each designed to rule out a particular form of alternative explanation or mechanism of the hypothesized effect. A particularly powerful design is the “dual-blind” design in which participants are blind to which treatment group is the focus of the research and the examiners collecting the data are ideally also blind in the same way as the participants and/or blind to which intervention each participant has been assigned.

Therefore, the most reasonable strategy for a control condition against which to compare the impact of a mindfulness-based intervention is to utilize a comparison treatment condition that matches the mindfulness intervention in all of the basic non-specific factors. However, it is quite difficult to develop rigorously matched comparison treatments that are matched to the key mindfulness intervention on all of the relevant variables. As modern commentators have noted (e.g., Wampold et al., 1997), there are a host of “non-specific” factors embedded in interventions that often play a key role in producing beneficial change, including the enthusiasm and confidence in the instructor or therapist that the intervention being deployed will promote well-being and will be beneficial. If a comparison treatment is developed and the instructor knows that it is a “control” condition, their enthusiasm for the intervention may pale by comparison with the bona fide mindfulness instructor.

One of the first rigorous trials that met most of the criteria noted above was conducted by Schmidt et al. (2011). In this trial, they examined the impact of Mindfulness-Based Stress Reduction on pain in fibromyalgia. The investigators randomized patients to a three-arm study in which one arm received MBSR, a second arm received an active comparison treatment which consisted of a Progressive Relaxation (Jacobson, 1938) and some light stretching exercises. The third arm was a wait list control. One of the important features of this study was that patients in both of the two active arms of this study were kept blind to which was the experimental intervention.

In an effort to develop a rigorous comparison condition for MBSR, we (MacCoon et al., 2012) developed the Health Enhancement Program (HEP). HEP was structurally matched to MBSR and was also matched on non-specific characteristics. The non-specific factors on which these two interventions were matched included the group format, the professional training of the instructors, the explicit expectation that the intervention is beneficial and can promote well-being and the requirement to engage in daily practice. The table below (from MacCoon et al., 2012)) illustrates the parallels between HEP and MBSR. It should be noted that while HEP is clearly appropriate for middle and older participants, it may not be well-suited to younger (adolescent) participants.

Interventi on content comparison.

MBSR		HEP
In-Class	Homework	In-Class	Homework
Body Scan	Body Scan and light reading	Music Therapy: Relax, listen to music, imagery, and drawing	Relax, listen to music, imagery, and drawing
Sitting Meditation	Body Scan, Sitting Meditation, and light reading	Nutrition Education around Food Guide Pyramid	Planning meals, tracking diet, food labels, journaling
Yoga	Alternate Yoga and Body Scan, and Sitting Meditation	Functional Movement (posture, balance, core movement)	Posture, balance, coordinated movement
Walking Meditation	Walking and other practices	Physical Activity (walk/jog, stretch)	Walking and stretching
All Day (7 h): Work with all practices, Group discussion & exercises	–	“Spa Day” (7 h): Work with all practices, Group discussion & exercises	–

In a controlled trial in which 63 participants were randomized to either MBSR or HEP, we found no group differences between the interventions on measures of anxiety, general distress, hostility, and medical symptoms. MBSR did result in significant reductions in thermal pain ratings, compared with HEP. In related research, we also found biological differences between MBSR and HEP (Rosenkranz et al., 2013). Participants randomized to MBSR displayed a smaller post-stress inflammatory response compared with those assigned to HEP despite equivalent reductions in self-reported psychological distress and physical symptoms. It may be that the non-reactivity cultivated in MBSR led to decreased emotional reactivity to the stressor that then led to a decreased inflammatory response. These findings underscore the complexity of experimental designs in this domain and also highlight the differential sensitivity of various outcome measures to specific aspects of the intervention. The data suggest that self-report measures of distress, negative affectivity and medical symptoms may respond to non-specific features of the intervention. Our comparison intervention—HEP—performed as well as MBSR in reducing these symptoms. However, the fact that measures of pain responsivity and various biological measures differentiated between the interventions underscores the importance of including an array of outcome measures that may be differentially influenced by specific and non-specific features of the interventions.

Another strategy in choosing comparison conditions for mindfulness-based interventions is to use a dismantling strategy where a comparison intervention is developed that contains all of the same ingredients as the original experimental intervention, with one key attribute eliminated. This is the strategy adopted by Williams and colleagues in their recent study of Mindfulness-Based Cognitive Therapy (MBCT) in preventing relapse in patients with recurrent depression (J Mark G Williams et al., 2014). In this study, the investigators developed a Cognitive Psychological Education (CPE) treatment that was matched to mindfulness-based cognitive therapy on all but the mindfulness component. They found that while the two interventions did not differ in affecting the risk of relapse over 12 months, for those patients with greater than the median levels of childhood trauma, the MBCT group outperformed the CPE in decreasing the risk of relapse. These findings again underscore the importance of selecting outcome measures that may be sensitive to relatively subtle differences between groups. They also imply that some of the beneficial change produced by interventions such as MBCT and MBSR occur via mechanisms that are not mindfulness-specific but rather are likely functions of the many non-specific factors that are present in many different forms of psychological treatment.

When considering a proper comparison treatment for a mindfulness-based intervention, the following represent the key desiderata for constructing a rigorous control condition:

1. Structural matching of dosage: The interventions should be equivalent in length.

2. Matching of homework/daily practice: The interventions should require the identical amount of practice.

3. Participants should not know which is the “experimental” intervention: Participants should be blinded to which intervention the experimenter is targeting for study. Examiners who are engaged in data collection should be blind to the intervention to which the participant has been assigned.

4. Expertise and confidence of the instructors should be matched: The comparison intervention should be taught by instructors who are comparably trained in comparison to the mindfulness teachers and who genuinely believe that the intervention they are teaching will produce beneficial change.

In addition to these desiderata, data on the fidelity of the implementation of the interventions, practice logs and data on the confidence and enthusiasm of the instructors should be routinely collected for use in analysis. While these recommendations constitute a high bar for the research community, we believe that it is necessary at this stage in the development of this field to strongly recommend these methodological desiderata in order to address the significant methodological limitations of most extant work in this area.

X. Structure of Study Design

There are a number of study design considerations that are relatively unique to meditation practice or intervention research. One relevant question concerns how research participants should be recruited. Often, for cross-sectional comparisons of experienced meditation practitioners versus novices or non-practitioners, recruitment has been based on samples of convenience. That is, practice groups or centers geographically or otherwise accessible to the investigator are contacted and research participants solicited. In such research, consideration should be given to the question of to which larger group of contemplative practitioners the available sample results might be generalizable. Some meditation groups or centers are part of an established tradition, with practice instruction and other aspects (e.g., rituals and liturgy) closely adhering to the parent tradition. For other groups or centers, practice tradition and associated activities are more idiosyncratic, creating greater constraints on generalizability.

For randomized controlled intervention studies, consideration should also be given to the question of appropriate participant recruitment. What is the nature of the particular intervention being assessed, and is this intervention best suited to persons with particular characteristics (e.g., age, education or occupational background)? What dependent or outcome variables will be measured, and do these suggest particular participant characteristics to be targeted in recruitment? For example, if it is hypothesized that a given meditation intervention might have an effect on real-world multitasking performance and subjective experience, then perhaps recruitment might target a group or groups for whom multitasking demands are prominent in their work environments (Levy, Wobbrock, Kaszniak, & Ostergren, 2012).

Another question related to recruitment involves what participants should be told about the nature of the interventions. For example, if participants in a randomized controlled study are told that one or more of the intervention arms will involve meditation training, then this might set up particular expectations that could influence outcomes. It may be best to avoid such general and excess-meaning burdened terms and stay with simple operational descriptions (e.g., “training in stabilizing attention”).

An important question facing the investigator designing a meditation intervention study concerns the length of the follow-up period that will be instituted. Clearly, decisions concerning follow-up period will be influenced by both cost and feasibility (e.g., the longer the follow-up, the greater the potential attrition, on average). However, investigators might also reflect on whether the intervention under study might be expected to have only transient and short-term effects (e.g., brief or one-time training), or longer-term effects (e.g., training in a contemplative intervention that participants are instructed to continue practicing after completion of the formal instruction phase of the study).

A related question concerns the specification of outcome measurement intervals. What are the most appropriate post-training/intervention follow-up intervals and frequencies? Is the particular meditation intervention under study expected to lead to relatively stable trait-like changes in relevant outcome measures, or is it expected that changes will be fragile and dependent upon particular conditions (e.g., frequency or intensity of continuing practice of what is learned in the intervention)? Answers to these questions will impact decisions about outcome measurement intervals. Relatedly, if outcomes are expected to be dependent upon practice adherence post-training, then it becomes particularly important to systematically assess practice adherence. Retrospective self-report regarding continuing practice frequency or duration may be subject to recall, estimation, and other sources of bias. Diary records, recorded immediately following each continuing practice episode, may be less vulnerable to such biases, although not entirely free of them. When continuing practice is guided by such aids as recorded practice instructions, consideration should be given to approaches that utilize technology to automatically record when the participant is accessing the recordings.

XI. Statistical and Data Analytic Issues

The majority of statistical and other data analytic issues in meditation practice or intervention research are not unique, but similar to those pertaining to any individual difference or intervention studies. What is known about the reliability (e.g., internal consistency, test-retest) and validity (e.g., content, construct, concurrent, predictive) of the outcome measures to be employed? Given the kinds of outcomes that some studies of meditation practice intend to assess ( e.g., changes in the experienced sense of self; c.f., Dambrun & Ricard, 2011), and the relatively recent empirical focus on such questions, there may be limited prior research upon which the investigator can rely for appropriate reliability and validity data.

Another data analytic issue concerns the question of appropriate analysis approach. So-called intent-to-treat analysis includes all participants who are randomized to intervention conditions, regardless of whether each of these participants actually completes all aspects (e.g., all of the meditation training sessions) of the intervention. This contrasts with an approach analyzing data from only those completing all of the intervention protocol. The most comprehensive approach would be to report types of analyses and if they reveal different outcomes, the investigator can then dissect the factors that contribute to the difference and thus provide a more nuanced understanding of what actually occurred.

Randomized assignment to intervention conditions is typically expected to minimize baseline differences between groups that could confound intervention outcome measurement. However, with relatively small group sample sizes, departure from such expectation may occur, and it is therefore important in such circumstances to compare baseline differences between groups. Should significant baseline differences exist, then the investigator is faced with decisions concerning the most appropriate approach. Considerable caution must be exercised, since available approaches (e.g., using regression to residualize post-intervention outcome measures for baseline group differences) must be considered from the perspective of whether the data characteristics meet assumptions of the approach. A detailed discussion of the specifics of such considerations is beyond the scope of the present paper, but numerous resources are available to aid in the relevant decisions (see Armstrong, 2012, for example in cancer research).

A final set of issues regarding statistics and data analysis concerns the need to go beyond the testing of simple main effects and interactions to the assessment of individual differences. Some relevant individual differences involve initial baseline measures, as discussed above, while others may involve such things as frequency and duration of practice. In the later instance, consideration might be given to whether mediator or moderator analyses (Hayes, 2013) are appropriate. Most importantly, a careful inspection of any trial of a mindfulness-based intervention will reveal that one size does not fit all. Such interventions may benefit certain types of individuals more than others. One of the key goals for future research is to systematically examine which types of individuals may benefit most from which types of meditation interventions. Virtually nothing is known about this from a scientific perspective at this point in time and it is a fertile area for future investigation.

XII. Conclusions

We have reviewed a number of critical conceptual and methodological issues that are germane to research on mindfulness-based interventions and other meditation research. We highlighted the need for a rich description of the intervention and how and by whom it is being taught. We have also underscored the complexity of measuring mindfulness and the variations we might expect in measuring mindfulness in novice versus experienced mindfulness practitioners. The possibility of using behavioral indices to measure mindfulness was considered and some promising possibilities are on the horizon. We concluded that using any single physiological or biological measure at this point in time is likely premature since extant data indicate that the construct cannot be captured in any simple single measure. The issue of control groups was discussed and this represents a thorny problem for research in this area. Double-blind placebo controlled trials are not possible for mindfulness interventions. Alternative strategies involving active comparison conditions are needed and several examples were provided, including the dual-blind design in which the participant is unaware of which intervention is the focus of the research and the examiners collecting the data are unaware to which group each participant has been assigned. Finally considerations regarding study design and statistical issues were discussed, some of which are common to other types of clinical and individual differences research.

It is important to underscore the fact that research in this area is still in its infancy though good progress has been made over the past decade. We believe that the quality of research in this area will improve now that more sophisticated designs have recently been published. There are still many questions that remain to be addressed and it is important for both the research community and policy makers to understand that although there is much excitement about this area, there are still very few methodologically rigorous studies that demonstrate the efficacy of mindfulness-based interventions in either the treatment of specific diseases or in the promotion of well-being. With the incorporation of some of the conceptual and methodological desiderata we showcase above, we anticipate a vibrant and productive period for scientific research on meditation in the future.

Biographies