Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Dec 4.
Published in final edited form as: J Contextual Behav Sci. 2014 Jan;3(1):21–26. doi: 10.1016/j.jcbs.2013.11.003

The Acceptance and Action Questionnaire – Stigma (AAQ-S): Developing a measure of psychological flexibility with stigmatizing thoughts

Michael E Levin a,b,*, Jason B Luoma c, Jason Lillis d, Steven C Hayes a, Roger Vilardaga e
PMCID: PMC4255270  NIHMSID: NIHMS608222  PMID: 25485230

Abstract

The current study sought to develop and test the Acceptance and Action Questionnaire – Stigma (AAQ-S), a measure of psychological flexibility with stigmatizing thoughts. A sample of 604 undergraduate students completed an online survey, which included an initial pool of 43 AAQ-S items as well as measures related to psychological flexibility and stigma. Expert judge ratings and factor analysis were used to identify and refine two distinct subscales; psychological flexibility and psychological inflexibility relating to stigmatizing thoughts. Analyses indicated that the AAQ-S psychological flexibility and inflexibility subscales, as well as a combined total score, correlate with other measures of psychological flexibility and stigma in expected ways, and are more predictive of stigma than a general measure of psychological flexibility. Overall, the results suggest that the AAQ-S could be a useful measure in conducting future stigma research.

Keywords: Psychological flexibility, Experiential avoidance, Stigma, Prejudice, Acceptance and Commitment Therapy

1. Introduction

Stigma affects the lives of countless numbers of individuals belonging to groups such as racial minorities (Williams & Mohammed, 2009), sexual minorities (Meyer, 2003), the obese (Puhl & Heuer, 2009), substance abusers (Luoma et al., 2007), and individuals with mental health problems (Rüsch, Angermeyer, & Corrigan, 2005). Enacted stigma, a subtype of stigma involving directly experienced social discrimination, occurs in areas including employment, housing, interpersonal relationships, and reduced access to services (Link & Phelan, 2001). Effective interventions are needed that can reduce enacted stigma. However, it is difficult to target enacted stigma because such a broad range of groups are stigmatized.

A promising approach is to conceptualize stigma as a more general tendency to evaluate and discriminate against others based on their group membership, rather than being specific to attitudes towards any one group in particular (Akrami, Ekehammar, & Bergh, 2011; Bäckström & Björklund, 2007; Lillis & Levin, in press). Consistent with this, research has found stigmatizing attitudes towards various groups are correlated and load onto a single latent variable of generalized prejudice (e.g., Akrami et al., 2011; Bäckström & Björklund, 2007). Thus, one means of addressing enacted stigma in its many forms is to identify the psychological processes that contribute to stigma as a generalized process and that can be effectively targeted through interventions.

One process that might be relevant for stigma reduction is psychological flexibility, the capacity to actively embrace one’s private experiences in the present moment and engage or disengage in patterns of behavior in the service of chosen values (Hayes, Strosahl, & Wilson, 2011). This process, as it applies to stigma, involves a combination of (1) flexible awareness of one’s private experiences in the present moment, including stigmatizing thoughts, (2) defusion from stigmatizing thoughts (e.g., seeing thoughts as just thoughts rather than something literally true), (3) willingness to have stigmatizing thoughts, rather than engaging in ineffective forms of avoidance (e.g., thought suppression, avoiding situations where stigmatizing thoughts occur), (4) relating to oneself and others as distinct from thoughts and feelings about them, (5) clarifying valued patterns of activity in social interactions, and (6) committing to patterns of valued activity with others, even when stigmatizing thoughts and feelings seem to stand in the way (Hayes, Niccolls, Masuda, & Rye, 2002; Lillis & Hayes, 2007; Lillis & Levin, in press).

The psychological flexibility model also highlights key pathological processes including psychological inflexibility, in which one’s actions are rigidly guided by psychological reactions rather than direct contingencies or chosen values, and experiential avoidance, in which one attempts to avoid, escape or otherwise control difficult thoughts and feelings despite the harmful consequences of doing so. Psychological inflexibility and experiential avoidance are associated with a broad range of psychological and behavioral health problems (Hayes, Luoma, Bond, Masuda, & Lillis, 2006; Ruiz, 2010), and may play a central role in stigma as a generalized process. Consistent with this, research has found that a general measure of psychological inflexibility and experiential avoidance is predictive of internalized homophobia (e.g., Gold, Dickstein, Marx, & Lexington, 2009), weight self-stigma (Lillis, Luoma, Levin, & Hayes, 2010) and stigmatizing attitudes towards individuals with mental health problems (e.g., Masuda & Latzman, 2011).

Treatments that target psychological flexibility, such as Acceptance and Commitment Therapy (ACT; Hayes et al., 2011), have been shown to produce meaningful change in clinical outcomes by reducing avoidance and inflexibility (Hayes et al., 2006; Ruiz, 2010). Recently, there have been a series of studies specifically examining the impact of ACT on stigma and prejudice towards various groups. One randomized control trial (RCT) found that substance abuse counselors improved significantly from baseline to 3-month follow up on stigma towards their clients from a 1-day ACT workshop compared to an education control condition (Hayes, Bissett et al., 2004). Furthermore, the impact of ACT on stigma relative to the control condition was mediated by reductions in the believability of stigmatizing thoughts, a core aspect of psychological flexibility (Hayes, Bissett et al., 2004). Another study found that a 75-min ACT class produced significantly improved positive behavioral intentions towards diverse cultural/racial groups at 1-week follow up compared to an education comparison condition (Lillis & Hayes, 2007). Furthermore, the effect of ACT on positive intentions was partially mediated by changes in a measure of acceptance and flexibility towards stigma (Lillis & Hayes, 2007). Finally, a RCT found that ACT produced significant reductions in stigma towards mental illness irrespective of the level of psychological flexibility of participants, but an education comparison condition only produced improvements in stigma among those participants higher in psychological flexibility (Masuda et al., 2007). Overall, these studies suggest that psychological flexibility could be a useful target for reducing stigma directed towards a broad range of groups.

Despite these promising findings, validated measures have not been developed to assess psychological flexibility with stigmatizing thoughts. The measure used by Lillis and Hayes (2007) represented a collection of individual items to assess different features of psychological flexibility, but was not designed to be a standardized scale. The Stigmatizing Attitudes Believability scale, used in the study by Hayes, Bissett et al. (2004), was designed to specifically measure believability of stigmatizing thoughts towards individuals with addiction problems and does not capture other domains of psychological flexibility nor provide a more generally applicable measure for stigma towards various groups. Finally, although the first and second versions of the Acceptance and Action Questionnaire (Hayes et al., 2004; Bond et al., 2011) represent a well validated general measure of psychological flexibility, research has found that domain specific measures of psychological flexibility are more sensitive and applicable when research is focused on a specific and unique problem area (e.g., Gifford et al., 2004; Gregg, Callaghan, Hayes, & Glenn-Lawson, 2007; Luoma, Drake, Kohlenberg, & Hayes, 2011).

Thus, the current study sought to develop and test a new measure of psychological flexibility with stigmatizing thoughts. An online survey of undergraduate students was used to refine the measure and examine its validity in relation to other measures related to psychological flexibility and stigma. This new measure was predicted to correlate with existing measures of psychological flexibility/inflexibility as well as stigma-related measures (such that higher psychological flexibility would relate to lower stigma). If such a measure is found to be reliable and valid, it could provide a useful tool for further research examining the role of psychological flexibility in stigma and for examining the processes of change in stigma interventions.

2. Methods

2.1. Participants and procedures

The study used a sample of 604 undergraduate students participating in an online survey for extra credit in a psychology course at the University of Nevada, Reno. The sample was 67.7% female and the modal age was 18 (M = 20.30, SD = 3.93). The racial distribution of the sample was 70.2% White, 9.8% Asian, 4.6% Black or African American, 1.7% Native Hawaiian/Other Pacific Islander, 1.7% Native American, 7.1% other and 4.9% Multiracial. In addition, 14.8% described their ethnicity as Hispanic or Latino.

Assessments were completed anonymously and participants had the option to skip any question using the option “I choose not to answer.” Ethical approval for the study was provided by the University of Nevada, Reno Internal Review Board.

Within each participant, mean scores were calculated for each variable provided 80% of the items were answered on a given scale or subscale. If a participant answered fewer than 80% of the items on a scale or subscale, then a sum score was not calculated for that variable. Based on this criteria, between 0% and 4% of participants were missing scale scores for specific variables (the social distance scale was the most common missing variable resulting from the 80% cutoff).

2.2. Initial scale development

A pool of 43 items was developed to assess the various aspects of psychological flexibility with stigmatizing thoughts. These aspects included awareness of stigmatizing thoughts, cognitive fusion/defusion with stigmatizing thoughts, experiential avoidance/acceptance of stigmatizing thoughts, distinguishing people from one’s stigmatizing thoughts about them, and stigmatizing thoughts as a barrier to valued/effective action. Consistent with previous research developing domain specific AAQ scales (e.g., Gregg et al., 2007; Luoma et al., 2011), items were created in part by adapting questions from existing measures including the AAQ (Hayes, Strosahl et al., 2004), AAQ-II (Bond et al., 2011) and items created by Lillis and Hayes (2007) in addition to novel items generated by the study authors. Items were framed in terms of general evaluations, judgments, and prejudiced thoughts towards others in order to help ensure that the measure could be used to assess stigma towards a broad range of groups. Item response options were on a 7-point scale from 1 (never true) to 7 (always true).

A sample of five expert judges were invited to rate the initial pool of 43 items through an online survey due to their significant experience conducting research on ACT, psychological flexibility and stigma. All five of the judges agreed to participate, though one judge did not complete the second section of item ratings. Judges were first asked to rate each item in terms of the degree to which it reflects psychological flexibility (or a particular domain of flexibility such as cognitive fusion or experiential avoidance) as it might relate to stigma towards others. Judges were then asked to rate each item again in terms of the quality of the item (e.g., unbiased, unambiguous, clear to a general population). Items were rated from 1 (poor) to 4 (excellent). This method has been used in previous research developing domain specific AAQ scales (e.g., Luoma et al., 2011) and helped to ensure the construct validity and quality of items.

The mean item rating for content was 3.17 (SD = .33, N = 5) and for quality was 3.22 (SD = .47, N = 4). Items were removed if the average content and/or quality rating across judges was below 2.5, the mid-point of the scale. In total, six items were removed due to low ratings on one of these dimensions. For the remaining 37 items, the mean rating for item quality was 3.32 (SD = .36, N = 5) and for quality was 3.24 (SD = .43, N = 4). On average, there was 83% agreement across judges regarding items that were rated as “excellent” or “good” quality as opposed to being rated as “adequate” or “poor.”

2.3. Psychological flexibility measures

2.3.1. Acceptance and Action Questionnaire – II (AAQ-II; Bond et al., 2011)

The AAQ-II is a 7-item measure of psychological inflexibility. Response options range from 1 “Never true” to 7 “Always true.” An example AAQ-II item is “I’m afraid of my feelings.” The AAQ-II has been found to have adequate reliability and validity with college student and clinical samples (Bond et al., 2011). In the current study, Cronbach’s alpha for the AAQ-II was .91.

2.3.2. Experiences Questionnaire (EQ; Fresco et al., 2007)

The 11-item decentering subscale of the EQ was examined in the current study. Decentering is defined as one’s capacity to notice thoughts as just thoughts rather than something literally true about oneself, which is also described as cognitive defusion, an important aspect of psychological flexibility. Responses are given on a 5-point scale ranging from 1 “Never” to 5 “All the time.” An example EQ item is “I remind myself that thoughts aren’t facts.” The measure has been found to have adequate reliability and validity in college student and clinical samples (Fresco et al., 2007). The decentering subscale had a Cronbach’s alpha of .85 in the current study.

2.4. Stigma-related measures

2.4.1. Interpersonal Reactivity Index (IRI; Davis, 1980)

The IRI is a multidimensional 28 item self-report measure that distinguishes four core components of empathy; personal distress, perspective taking, empathic concern, and fantasy. The perspective taking and empathic concern subscales, which assesses the tendency to adopt others’ psychological point of view and to feel sympathy and compassion for others, were examined in the current study. Responses are given on a 4-point scale ranging from 1 “Does not describe me well” to 4 “Describe me very well.” An example empathic concern item is “I would describe myself as a pretty soft-hearted person.” and an example perspective taking item is “Before criticizing somebody, I try to imagine how I would feel if I were in their place.” The IRI has been found to be valid and reliable in previous studies with college students (Davis, 1983). In our sample, the IRI had a Cronbach’s alpha of .77 for empathic concern and .76 for perspective taking.

2.4.2. Bogardus Social Distance Scale (SDS; Bogardus, 1925)

A modified 5-item version of the SDS was used to examine social distance towards a range of stigmatized groups; ethnic minorities, African Americans, Arab Americans, obese individuals, and gay men. For each item, participants are asked to rate the classification they would willingly admit members from the group to on a scale ranging from 1 “Close kinship by marriage” to 7 “Exclude from my country.” The SDS has been found to be a reliable and valid measure of social distance towards a range of stigmatized groups in a number of studies with college students (Parillo & Donoghue, 2005). A summary score was calculated to indicate general social distancing towards stigmatized groups. In the current study, the SDS had a Cronbach’s alpha of .77.

2.4.3. Right Wing Authoritarianism (RWA; Zakrisson, 2005)

The current study used a shortened 15-item version of the RWA, which is designed to measure the tendency to rigidly follow traditional norms and authority as well as be aggressive towards those who do not. Responses are given on a 9-point scale from 1 “Very negative” to 9 “Very positive.” An example RWA item is “Our country needs a powerful leader, in order to destroy the radical and immoral currents prevailing in society today.” RWA has been found to be a robust predictor of stigma towards a wide variety of social groups (Asbrock, Sibley, & Duckitt, 2010). Past research has found the RWA to be a reliable and valid measure with college students (e.g., Asbrock et al., 2010; Zakrisson, 2005). In the current study the RWA had a Cronbach’s alpha of .76.

2.4.4. Social Dominance Orientation (SDO; Sidanius & Pratto, 2001)

The current study used a 16-item version of the SDO, a measure of preference for hierarchy in society. Responses are given on a 9-point scale ranging from 1 “Very negative” to 9 “Very positive.” An example SDO item is “Some groups of people are simply inferior to other groups.” The SDO has also been found to be a strong predictor of stigma towards a number of social groups (Asbrock et al., 2010) and has been found to be a reliable and valid measure with college students (Sidanius & Pratto, 2001). In the current study the SDO had a Cronbach’s alpha of .94.

2.4.5. Scale of Ethnocultural Empathy (SEE; Wang et al., 2003)

The SEE is a 31-item self-report measure of empathy towards individuals belonging to different racial/ethnic groups. Responses are given on a 6-point scale ranging from 1 “Strongly disagree that it describes me” to 6 “Strongly agree that it describes me.” Example items include “I feel supportive of people of other racial and ethnic groups, if I think they are being taken advantage of”, “I feel annoyed when people do not speak English”, and “It is difficult for me to put myself in the shoes of someone who is racially and/or ethnically different from me.” Research indicates adequate reliability and validity with the SEE in college samples (Wang et al., 2003). In the current study the SEE had a Cronbach’s alpha of .93.

3. Results

3.1. Scale refinement

Analyses were conducted with the 37 items recommended by expert judges in the initial phase of scale development. We first examined the distribution of each variable based on recommended procedures (Clark & Watson, 1995) and found that there were no highly skewed or unbalanced items.

An exploratory principal component analysis without rotation was then conducted (Cortina, 1993). The resulting scree plot clearly indicated a three factor solution. Next, a principal component analysis with oblimin rotation was conducted given the expected correlation across factors. However, the resulting component correlation matrix indicated little to no correlation between the three factors (correlation coefficients ranged between −.19 and .13).

We then ran a principle component analysis with varimax rotation and forcing three factors, given the lack of correlation between factors. An iterative process was used in which items with high cross loadings (≥.3 factor loading on a secondary factor) or low loadings on any factor (no factor loadings ≥.45) were removed and the factor analysis was re-run. A cutoff factor loading score of .45 is slightly higher than the recommended cutoff of .40 (Floyd & Widaman, 1995), which we chose in order to help reduce the final number of items for each subscale. In total, seven items were removed in this process due to high cross loadings or low factor loading. Inter-item correlations were then examined for each factor in order to identify highly redundant items, based on the criteria of multiple inter-item correlations above .5 (Clark & Watson, 1995). Two additional items were identified for removal from the first factor through this process.

Preliminary analyses indicated that the third factor performed poorly with regards to construct validity (i.e., failing to correlate, and in some cases correlating in the opposite direction, with conceptually related constructs) and was therefore excluded from further analyses. Principle component analysis with varimax rotation was conducted with the 21 items from factors 1 and 2, forcing two factors. The results showed the same pattern as previous analyses with the 21 items loading onto the same two factors outlined above. This 2-factor solution accounted for 40.5% of the observed variance.

The final two factor principle component analysis with varimax rotation is presented in Table 1. The remaining 21 items each loaded onto one of two distinct factors, with minimal cross loading. Factor 1 contained 11 items assessing psychological inflexibility with stigmatizing thoughts (e.g., the propensity to act on stigmatizing thoughts even when harmful, engaging in rigid and intense patterns of stigmatizing others). Example items for Factor 1 include “I stop doing things that are important to me when it involves someone I don’t like.” and “The bad things I think about others must be true.” Factor 2 contained 10 items assessing psychological flexibility with stigmatizing thoughts (e.g., awareness of stigmatizing thoughts, relating to stigmatizing thoughts as just thoughts, willingness to experience stigmatizing thoughts). Example items for Factor 2 include “I feel that I am aware of my own biases.” and “It’s OK to have friends that I have negative thoughts about from time to time”.

Table 1.

Factor loading results of AAQ-S items based on principle component factor analysis with varimax rotation forcing two factors.

Item Factor
1 2
Psychological Inflexibility Subscale
1. My biases and prejudices affect how I interact with people from different backgrounds. .54 .03
2. I need to reduce my negative thoughts about others in order to have good social interactions. .61 .08
3. I stop doing things that are important to me when it involves someone I don’t like. .57 .22
4. I have trouble letting go of my judgments of others. .70 .05
5. I feel that my prejudicial thoughts are a significant barrier to me being culturally sensitive. .65 .23
6. I have trouble not acting on my negative thoughts about others. .63 .17
7. When I am having negative thoughts about others, I withdraw from people. .63 .10
8. When I have judgments about others, they are very intense. .76 −.02
9. When talking with someone I believe I should act according to how I feel about him/her, even if its negative. .53 −.01
10. I often get caught up in my evaluations of what others are doing wrong. .68 −.08
11. The bad things I think about others must be true. .60 .02
Psychological Flexibility Subscale
12. I feel that I am aware of my own biases.a .07 .70
13. My negative thoughts about others are never a problem in my life.a .12 .58
14. I rarely worry about getting my evaluations towards others under control.a −.03 .50
15. I’m good at noticing when I have a judgment of another person.a .02 .66
16. When I evaluate someone negatively, I am able to recognize that this is just a reaction, not an objective fact.a .15 .66
17. I am aware when judgments about others are passing through my mind.a .15 .71
18. It’s OK to have friends that I have negative thoughts about from time to time.a −.06 .50
19. I don’t struggle with controlling my evaluations about others.a .21 .59
20. When I’m talking with someone I don’t like, I’m aware of my evaluations of them.a −.02 .65
21. I accept that I will sometimes have unpleasant thoughts about other people.a .14 .61
Open in a new tab
a

Reverse scored items.

3.2. Scale characteristics

The Cronbach’s alpha for psychological inflexibility and psychological flexibility subscales were .85 and .82 respectively. Of note, items in the psychological flexibility subscale were reverse scored so that higher scores indicate less flexibility with stigmatizing thoughts. The two subscales were significantly correlated in the expected direction, r(580) = .24, p < .001. Given that the psychological flexibility and inflexibility subscales are theoretically assessing different aspects of a single construct and the observed correlation between subscales, an AAQ-S total score variable combining the items from the two subscales was also calculated. The combined AAQ-S total score had adequate internal consistency (Cronbach’s alpha = .84) and correlated highly and similarly with both the flexibility (r = .75, p < .001) and inflexibility subscales (r = .82, p < .001), as would be expected.

3.3. Validity

The construct validity for the AAQ-S subscales and total score was examined by exploring their relationship to measures of general psychological flexibility and stigma (see Table 2). Significant correlations ranging between .20 and .48 were observed between the AAQ-S and measures of general psychological flexibility. Correlations were in the expected direction such that higher inflexibility/lower flexibility with stigmatizing thoughts was related to higher psychological inflexibility as measured by the AAQ-II and lower decentering as measured by the EQ. The AAQ-S significantly correlated with stigma-related measures with coefficients ranging between .23 and .43, such that higher inflexibility/lower flexibility with stigmatizing thoughts was related to higher social distancing, authoritarianism, and social dominance as well as lower empathy.

Table 2.

Correlation between the AAQ-S and variables related to psychological flexibility and stigma.

Variable AAQ-S Inflexibility AAQ-S Flexibility AAQ-S Total Score AAQ-II
Psychological flexibility
AAQ-II .48*** .20*** .44*** –
EQ −.20*** −.45*** −.40*** −.38***
Stigma
SDS .23*** .30*** .33*** .02
RWA .27*** .30*** .36*** .03
SDO .36*** .31*** .43*** .11**
SEE −.29*** −.30*** −.38*** −.03
IRI-Emp −.29*** −.25*** −.35*** −.06
IRI-PT −.28*** −.33*** −.39*** −.15***
Open in a new tab

AAQ-II = psychological inflexibility, EQ = Decentering, SDS = social distancing, RWA = right wing authoritarianism, SDO = social dominance, SEE = Scale of Ethno-cultural Empathy, IRI-Emp = empathic concern, IRI-PT = perspective taking.

**

p < .01.

***

p < .001.

3.4. Comparison of the AAQ-S and AAQ-II in relation to predictors of stigma

Correlations between the AAQ-II and predictors of stigma were examined to explore whether the AAQ-S was more strongly related to stigma variables compared to a general measure of psychological inflexibility (see Table 2). In support of this hypothesis, the AAQ-II was not significantly related to measures of social distancing, authoritarianism, or empathy (ps > .10). However, the AAQ-II was significantly related to social dominance and perspective taking, such that higher psychological inflexibility was related to greater dominance and lower perspective taking.

Differences in correlation coefficients between the AAQ-II and AAQ-S were examined using recommended procedures (Meng, Rosenthal, & Rubin, 1992). The correlation between the SDO and AAQ-II was significantly lower than the correlation between the SDO and AAQ-S inflexibility (Z = 4.52, p < .001), the SDO and AAQ-S flexibility (Z = 3.57, p < .001) and the SDO and AAQ-S total score (Z = 5.94, p < .001). In addition, the correlation between IRI perspective taking and the AAQ-II was significantly lower than the correlation between perspective taking and AAQ-S inflexibility (Z = 2.32, p = .02), perspective taking and AAQ-S flexibility (Z = 3.26, p < .001) and perspective taking and AAQ-S total score (Z = 4.43, p < .001).

4. Discussion

The current study sought to develop and evaluate a measure of psychological flexibility with stigmatizing thoughts. Expert ratings and factor analysis were used to identify and refine two distinct subscales; psychological flexibility and psychological inflexibility with stigmatizing thoughts. These subscales were found to have adequate internal consistency. The AAQ-S psychological flexibility and inflexibility subscales as well as a combined total score correlated with other measures of general psychological inflexibility and stigma in expected ways. Furthermore, the AAQ-S was more strongly related to stigma than a general measure of psychological inflexibility.

After removing a third factor, which demonstrated poor construct validity, the factor analyses indicated a two factor solution, with items assessing psychological flexibility and inflexibility loading onto distinct factors. This is consistent with findings from other studies with domain specific AAQ scales (e.g., Luoma et al., 2011) and might be attributable to methodological effects related to the positive and negative wording of these two sets of items, which has been found in research on the AAQ-II (Bond et al., 2011). In support of this, higher correlations were observed between the AAQ-S inflexibility subscale and AAQ-II, both of which use negatively worded items, and the AAQ-S flexibility subscale and EQ, both of which use positively worded items. These findings may also suggest that psychological flexibility and inflexibility do not necessarily represent distinct poles on a single dimension. For example, it may be the case that low psychological inflexibility does not always indicate high psychological flexibility and vice versa. If such findings were replicated and extended, it could suggest that including measures assessing aspects of both psychological inflexibility and flexibility would be beneficial in future research.

The current study did not examine whether the AAQ-S is sensitive to acceptance and mindfulness-based interventions such as ACT, which are known to impact psychological flexibility and stigma (Hayes, Bissett et al., 2004; Lillis & Hayes, 2007; Masuda et al., 2007). It is important that future treatment and longitudinal research examine the AAQ-S regarding its sensitivity to intervention effects, utility in accounting for how interventions such as ACT produce meaningful changes in stigma towards others, and longitudinal role in prosocial behavior.

It is possible that the AAQ-S would function differently within an intervention context. Purely psychometric evaluations of instruments are based on a mechanistic and realistic conception of language (e.g., see Borsboom, Mellenbergh, & van Heerden, 2003) not a contextual analysis, as is the case with ACT and its base in the Relational Frame Theory (Hayes, Barnes-Holmes, & Roche, 2001). In a contextual approach, verbal reports can change their functions in particular conditions (Hayes, Barnes-Holmes, & Wilson, 2012). For example, in some contexts reporting that one is accepting of stigmatizing thoughts could indicate a propensity to stigmatize others and agree with one’s stigmatizing thoughts, while in other contexts, such as after an acceptance and mindfulness intervention, it may indicate a willingness to acknowledge stigmatizing thoughts without acting on them or engaging in ineffective attempts to suppress or avoid them. Similarly, within a college setting, a diversity-related class might result in increased scores on the inflexibility scale due to increased awareness of personal biases and how they affect one’s behavior, even though this might actually represent a decrease in psychological inflexibility. Future research would benefit from examining whether particular contextual factors, including acceptance and mindfulness interventions, alter the relationship between the AAQ-S and stigma-related outcomes, in addition to more traditional validity assessment processes.

The observation that the AAQ-S is more strongly correlated with stigma-related measures than the AAQ-II provides support for the potential utility of the AAQ-S over this more general measure of psychological flexibility within stigma research. This is consistent with other research finding that domain specific measures of psychological flexibility can be more sensitive than the general AAQ-II (e.g., Gifford et al., 2004; Gregg et al., 2007; Luoma et al., 2011). Further research on the AAQ-S would benefit from examining whether it is also more sensitive than the AAQ-II in capturing the impact of interventions targeting psychological flexibility with stigmatizing thoughts.

There are limitations to the present study. The AAQ-S was developed and validated using an undergraduate student sample at a single university and it is unclear how these findings will generalize to other populations, particularly those that are more diverse and that are more likely to report stigmatizing others or being stigmatized. The current study relied solely on self-report measures, which are vulnerable to the kinds of self-presentation biases likely to occur when people are asked to report stigma towards others. Further research should examine the relationship of the AAQ-S to measures that are less impacted by these biases, such as implicit cognition and behavioral measures. Assessments were completed as part of an online survey for course credit, which could have resulted in some poor quality data entering the dataset.

In order to ensure that the AAQ-S could be broadly applied in studying stigma, the measure purposefully used general terms such as “judgments”, “evaluations”, and “negative thoughts” in addition to “prejudice” and “bias.” This provides greater flexibility in studying stigma as a generalized process, but could also mean that the AAQ-S may include some variance related to negative attitudes towards people in general. The observed correlations between the AAQ-S and stigma-related measures, in contrast to the relative lack of correlations between the AAQ-II and stigma measures, provide some reassurance that this measure is not simply measuring a negative bias toward others, but further research is important in establishing the construct validity of the AAQ-S. In addition, further research is needed to examine the relationship of the AAQ-S to measures of stigma directed towards specific groups to continue to examine whether these processes indeed apply across a range of stigmatized groups. Of particular applied importance, it remains to be seen whether the AAQ-S is sensitive to acceptance and mindfulness-based interventions for stigma and mediates intervention effects on stigma.

Overall, the findings from the current study suggest that the AAQ-S provides a reliable and valid measure of psychological flexibility/inflexibility with stigmatizing thoughts. Further research examining the role of psychological flexibility in stigma as well as theoretical processes of change in stigma interventions may benefit from the use of the AAQ-S.

Contributor Information

Jason B. Luoma, Email: jbluoma@gmail.com.

Jason Lillis, Email: jasonlillis22@gmail.com.

Steven C. Hayes, Email: stevenchayes@gmail.com.

Roger Vilardaga, Email: roger.vilardaga@gmail.com.

References

  1. Akrami N, Ekehammar B, Bergh R. Generalized prejudice: Common and specific components. Psychological Science. 2011;22:57–59. doi: 10.1177/0956797610390384. [DOI] [PubMed] [Google Scholar]
  2. Asbrock F, Sibley CG, Duckitt J. Right-wing authoritarianism and social dominance orientation and the dimensions of generalized prejudice: A longitudinal test. European Journal of Personality. 2010;24:324–340. [Google Scholar]
  3. Bäckström M, Björklund F. Structural modeling of generalized prejudice: The role of social dominance, authoritarianism, and empathy. Journal of Individual Differences. 2007;28:10–17. [Google Scholar]
  4. Bogardus ES. Social distance and its origins. Sociology and Social Research. 1925;9(216):225. [Google Scholar]
  5. Bond FW, Hayes SC, Baer RA, Carpenter K, Orcutt HK, Waltz T, et al. Preliminary psychometric properties of the Acceptance and Action Questionnaire-II: A revised measure of psychological flexibility and acceptance. Behavior Therapy. 2011;42:676–688. doi: 10.1016/j.beth.2011.03.007. [DOI] [PubMed] [Google Scholar]
  6. Borsboom D, Mellenbergh GJ, van Heerden J. The theoretical status of latent variables. Psychological Review. 2003;110:203–219. doi: 10.1037/0033-295X.110.2.203. [DOI] [PubMed] [Google Scholar]
  7. Clark LA, Watson D. Constructing validity: Basic issues in objective scale development. Psychological Assessment. 1995;7:309–319. doi: 10.1037/pas0000626. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Cortina JM. What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology. 1993;78:98–104. [Google Scholar]
  9. Davis MH. A multidimensional approach to individual differences in empathy. Catalog of Selected Documents in Psychology. 1980;10:1–17. [Google Scholar]
  10. Davis MH. Measuring individual differences in empathy: Evidence for a multidimensional approach. Journal of Personality and Social Psychology. 1983;44:113–126. [Google Scholar]
  11. Floyd FJ, Widaman KF. Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment. 1995;7:286–299. [Google Scholar]
  12. Fresco DM, Moore MT, van Dulmen M, Segal ZV, Teasdale JD, Ma H, et al. Initial psychometric properties of the Experiences Questionnaire: Validation of a self-report measure of decentering. Behavior Therapy. 2007;38:234–246. doi: 10.1016/j.beth.2006.08.003. [DOI] [PubMed] [Google Scholar]
  13. Gifford EV, Kohlenberg BS, Hayes SC, Antonuccio DO, Piasecki M, Rasmussen-Hall ML, et al. Acceptance theory-based treatment for smoking cessation: An initial trial of Acceptance and Commitment Therapy. Behavior Therapy. 2004;35:689–706. [Google Scholar]
  14. Gold SD, Dickstein BD, Marx BP, Lexington JM. Psychological outcomes among lesbian sexual assault survivors: An examination of the roles of internalized homophobia and experiential avoidance. Psychology of Women Quarterly. 2009;33:54–66. [Google Scholar]
  15. Gregg JA, Callaghan GM, Hayes SC, Glenn-Lawson JL. Improving diabetes self-management through acceptance, mindfulness, and values: A randomized controlled trial. Journal of Consulting and Clinical Psychology. 2007;75:336–343. doi: 10.1037/0022-006X.75.2.336. [DOI] [PubMed] [Google Scholar]
  16. Hayes SC, Barnes-Holmes D, Roche B. Relational frame theory: A post-Skinnerian account of human language and cognition. New York: Kluwer Academic/Plenum; 2001. [DOI] [PubMed] [Google Scholar]
  17. Hayes SC, Luoma JB, Bond FW, Masuda A, Lillis J. Acceptance and Commitment Therapy: Model, processes and outcomes. Behavior Research and Therapy. 2006;44:1–25. doi: 10.1016/j.brat.2005.06.006. [DOI] [PubMed] [Google Scholar]
  18. Hayes SC, Strosahl KD, Wilson KG. Acceptance and Commitment Therapy: The process and practice of mindful change. New York: The Guilford Press; 2011. [Google Scholar]
  19. Hayes SC, Barnes-Holmes D, Wilson KG. Contextual Behavioral Science: Creating a science more adequate to the challenge of the human condition. Journal of Contextual Behavioral Science. 2012;1:1–16. [Google Scholar]
  20. Hayes SC, Bissett R, Roget N, Padilla M, Kohlenberg B, Fisher G, et al. The impact of acceptance and commitment training and multicultural training on the stigmatizing attitudes and professional burnout of substance abuse counselors. Behavior Therapy. 2004;35:821–835. [Google Scholar]
  21. Hayes SC, Niccolls R, Masuda A, Rye AK. Prejudice, terrorism and behavior therapy. Cognitive and Behavioral Practice. 2002;9:296–301. [Google Scholar]
  22. Hayes SC, Strosahl K, Wilson KG, Bissett RT, Pistorello J, Toarmino D, et al. Measuring experiential avoidance: A preliminary test of a working model. Psychological Record. 2004;54:553–578. [Google Scholar]
  23. Lillis J, Hayes SC. Applying acceptance, mindfulness, and values to the reduction of prejudice: A pilot study. Behavior Modification. 2007;31:389–411. doi: 10.1177/0145445506298413. [DOI] [PubMed] [Google Scholar]
  24. Lillis J, Levin ME. Acceptance and mindfulness for promoting empathy and undermining cultural bias. In: Masuda A, editor. Mindfulness and acceptance in multicultural competency: A contextual approach to sociocultural diversity in theory and practice. Oakland, CA: New Harbinger; inpress. [Google Scholar]
  25. Lillis J, Luoma J, Levin ME, Hayes SC. Measuring Weight Self Stigma: The Weight Self Stigma Questionnaire. Obesity. 2010;18:971–976. doi: 10.1038/oby.2009.353. [DOI] [PubMed] [Google Scholar]
  26. Link BG, Phelan JC. Conceptualizing stigma. Annual Review of Sociology. 2001;27:363–385. [Google Scholar]
  27. Luoma J, Drake CE, Kohlenberg BS, Hayes SC. Substance abuse and psychological flexibility: The development of a new measure. Addiction Research and Theory. 2011;19:3–13. [Google Scholar]
  28. Luoma JB, Twohig MP, Waltz T, Hayes SC, Roget N, Padilla M, et al. An investigation of stigma in individuals receiving treatment for substance abuse. Addictive Behaviors. 2007;32:1331–1346. doi: 10.1016/j.addbeh.2006.09.008. [DOI] [PubMed] [Google Scholar]
  29. Masuda A, Hayes SC, Fletcher LB, Seignourel PJ, Bunting K, Herbst SA, et al. The impact of Acceptance and Commitment Therapy versus education on stigma toward people with psychological disorders. Behaviour Research and Therapy. 2007;45:2764–2772. doi: 10.1016/j.brat.2007.05.008. [DOI] [PubMed] [Google Scholar]
  30. Masuda A, Latzman RD. Examining associations among factor-analytically derived components of mental health stigma, distress, and psychological flexibility. Personality and Individual Differences. 2011;51:435–438. [Google Scholar]
  31. Meng XL, Rosenthal R, Rubin DB. Comparing correlated correlation coefficients. Psychological Bulletin. 1992;111:172–175. [Google Scholar]
  32. Meyer IH. Prejudice, social stress, and mental health in lesbian, gay, and bisexual populations: Conceptual issues and research evidence. Psychological Bulletin. 2003;129:674–697. doi: 10.1037/0033-2909.129.5.674. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Parillo VN, Donoghue C. Updating the Bogardus social distance studies: A new national survey. The Social Science Journal. 2005;42:257–271. [Google Scholar]
  34. Puhl RM, Heuer CA. The stigma of obesity: A review and update. Obesity. 2009;17:941–964. doi: 10.1038/oby.2008.636. [DOI] [PubMed] [Google Scholar]
  35. Ruiz FJ. A review of Acceptance and Commitment Therapy (ACT) empirical evidence: Correlational, experimental psychopathology, component and outcome studies. International Journal of Psychology and Psychological Therapy. 2010;10:125–162. [Google Scholar]
  36. Rüsch N, Angermeyer MC, Corrigan PW. Mental illness stigma: Concepts, consequences, and initiatives to reduce stigma. European Psychiatry. 2005;20:529–539. doi: 10.1016/j.eurpsy.2005.04.004. [DOI] [PubMed] [Google Scholar]
  37. Sidanius J, Pratto F. Social dominance: An intergroup theory of social hierarchy and oppression. Cambridge: Cambridge University Press; 2001. [Google Scholar]
  38. Wang Y, Davidson MM, Yakushko OF, Bielstein HB, Tan JA, Bleier JK. The scale of ethnocultural empathy: Development, validation, and reliability. Journal of Counseling Psychology. 2003;50:221–234. [Google Scholar]
  39. Williams DR, Mohammed SA. Discrimination and racial disparities in health: Evidence and needed research. Journal of Behavioral Medicine. 2009;32:20–47. doi: 10.1007/s10865-008-9185-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Zakrisson I. Construction of a short version of the right-wing authoritarianism (RWA) scale. Personality and Individual Differences. 2005;39:863–872. [Google Scholar]

RESOURCES