Abstract
Public stigma robs people with mental illnesses from rightful opportunities related to work and other important life goals. Advocates have developed anti-stigma programs meant to address the prejudice and discrimination associated with these conditions. Evidence is now needed to make sense of program impact; this paper looks at measurement issues related to stigma change. Community based participatory research is central to this research and includes the involvement of a diverse collection of stakeholders in all phases of evaluation. Investigators should be cautious about measures vis-à-vis social desirability effects and should directed by social validity of targeted audiences. Conceptual domains with some research support that correspond with assessments include behavior, penetration, psychological perspective, knowledge, and physiological/information processes. These issues are summarized as ten recommendations for evaluation of anti-stigma programs.
Measuring the Impact of Programs that Challenge the Public Stigma of Mental Illness
Advocates of all stripes agree; the stigma of mental illnesses is egregious in effect and must be challenged to broaden life opportunities of adults and children with psychiatric disorders for whom these prospects are deserving. Advocates have partnered with investigators to better understand stigma. As a result, careful research programs have begun to yield insights about what stigma is and how it can be diminished. Still, advocates, needing to address the problem now, have become impatient with science and have put together programs meant to erase mental illness stigma, sometimes ahead of the data. Governments supporting broad-based and large scale stigma change programs include active efforts in Australia, Canada, the UK, as well as dozens of ad hoc programs supported by the World Psychiatric Association. In addition, most of the fifty states in the US now have anti-stigma programs supported by state funds (see the Resource Center supported by the Substance Abuse and Mental Health Services Administration (SAMHSA)).
In light of these efforts and priorities comes the central question of this paper: What is the evidence for anti-stigma programs? There are recent, fairly comprehensive reviews of evidence that describe the impact of anti-stigma programs (e.g., Arboleda-Florez & Sartorius, 2008; Corrigan, 2005; Hinshaw, 2007; Thornicroft, 2006). In this light, Link and colleagues (2004) provided an excellent review of published stigma measures. Absent from the paper, however is consideration of how to integrate these and other measures in the evaluation of stigma change programs. It is not the quantity of empirical work on measurement but its quality that is considered herein. Unfortunately, the prior reviews suggest that the quality of measurement and design of studies assessing anti-stigma programs have been wanting. Hence, the primary reason for this paper: to consider and advance the sophistication and rigor of measures that assess the impact of anti-stigma programs.
Stigma is a complex and multi-layered phenomenon so we begin the paper with a brief but careful discussion of how research has come to know stigma and its relevance to mental illness. This summary is followed by an outline of ways to change stigma, with specific focus on public stigma. This too has become a large arena; so consistent with our earlier work, we focus on three anti-stigma strategies: protest, education, and contact (Corrigan & Penn, 1999). We then consider questions that need to assume center stage of stigma-change research: community based participatory research, social desirability, and external validity. Given the convergence of these ideas, we summarize five domains of assessment: behavior, penetration, psychological perspectives, knowledge, and physiological/information processes. We end with recommendations for future measurement and research.
Where is Clinical Psychology in this Effort?
Clinical psychology has mostly been at the sidelines regarding stigma and stigma change programs, despite research in this area having mushroomed. Our review of the literature from 1998 to 2008 showed journal articles on stigma in general have quadrupled from 178 to 641, with about one third of these focusing on the stigma of mental illnesses (in 2008, 231/641). No more than nine of these articles were found in journals with Clinical Psychology on its masthead (which is approximately 1.4% of articles at most during this time) though we should not assume few papers in clinical psychology journals mean few clinical psychologists working in anti-stigma program research. The issue of stigma would also seem to overlap with community psychology both theoretically and methodologically; 37.8% of Division 27 members (Community Psychology) report their subfield as clinical psychology (APA, 2008). However, frequency of abstracts mentioning stigma in these journals never exceed 2% (the highest was 12 in 2008).
The absence of clinical psychology’s participation in understanding mental health stigma, and in developing and evaluating anti-stigma programs, is unfortunate for several reasons. Clinical psychologists are often leaders in understanding the symptoms and disabilities of mental illnesses. Along with these factors, stigma is now recognized as a significant predictor of course and impact of mental illnesses. Consider, from this perspective, extraordinary epidemiological findings: 50 to 75% of people with psychiatric disorders, who might benefit from mental health services, do not receive them. Equally sobering are findings that suggest from half to two thirds of people drop out of treatment prematurely (cf Corrigan, 2004b for a review of relevant literature). Stigma has been implicated as one of several systemic factors that explain poor care seeking and service use.
Who is the Focus of this Paper?
Stigma and mental illness are unwieldy concepts. One way to make sense of these constructs is to limit our focus on a group which is especially harmed by mental illness stigma: people with serious mental illnesses. We made this choice because of its public health priority and because of the relative wealth of research and recommendations that may guide this paper (Corrigan, 2005). What do we mean by serious? Clinical definitions of disease severity have merged with the idea of psychiatric disability to describe serious mental illnesses (Liberman, 2008), people whose life goals in important spheres (e.g., work, independent living, relationships, wellness) are challenged by mental illnesses. Mental illnesses mean people with Axis I or Axis II disorders in the DSM-IV-TR, other than those primarily representing developmental disabilities, substance abuse disorders, or physical trauma. Prominent among these are the psychoses: schizophrenia, but also major depression and bipolar disorder with psychotic features. Consistent with the zeitgeist, in this paper we alternately refer to people with serious mental illnesses or consumers of clinical services for these illnesses.
From here, the paper examines brief definitions of stigma, distinguishing self and public stigma, and limits the text to change approaches for public stigma. The paper then segues into its central goal; incorporating socially valid instruments that will demonstrate the impact of these programs. Specific ideas circumscribing rigorous measurement follow and are the large portion of the remainder of the paper.
DEFINING STIGMA
Our approach towards understanding stigma is largely informed by research on social cognition and is laid out in Table 1. Stereotypes are general beliefs about the characteristics, attributes, and behaviors of people who are categorizable as a member of particular social group—what we think members of a particular group are like (Corrigan & Kleinlein, 2005; Major & O’Brien, 2005). For example, “Irish Americans are drunks” or, of relevance here, “The mentally ill are dangerous.” By the time members of the general public reach adulthood, they learn and can repeat these stereotypes. Stereotypes are also culturally defined. For example, some of the mental illness stereotypes prominent in the United States markedly differ from those in China (Lee & Kleinman, 2007; Yang, Kleinman et al., 2007). Research has shown that what Americans are likely to label anxiety or depression are expressed more somatically in China (what many Americans call panic is described by many Chinese as heart or GI ailments). This paper describes the stereotypes held in the West.”
Table 1.
The Cognitive and Behavioral Constructions of Mental Illness Stigma.
| CONSTRUCTS | DEFINITIONS | EXAMPLES OF PUBLIC STIGMA | EXAMPLES OF SELF-STIGMA | ||
|---|---|---|---|---|---|
| COGNITIVE | Stereotypes | Beliefs about groups learned while growing up in a specific society |  |
|
Yes, I am dangerous, blameworthy, incompetent and childlike, and hence…
|
| Prejudice | Attitudes about groups. N.B. Prejudice also has emotional aspects. | ||||
| BEHAVIORAL |
Discrimination |
The behavioral consequences of prejudice |
 |
|
Experience the Why Try effect
|
Prejudice is a generalized attitude toward members of a social group—how someone feels about a group (e.g., feeling nervous, scared, or angry around someone with a serious mental illness). Discrimination is behavior directed toward a group based on prejudice. For example, an employer may have two equally qualified applicants who differ in one way— a serious mental illness. The decision not to hire one applicant solely due to that illness is be discrimination. The connection between prejudice and discrimination can be poignant; “an employee with a mental illness might attack my other employees, therefore I am not going to hire him”).
A definition of stigma includes these constructs. Stigma was originally defined by Erving Goffman (1963) as a mark or attribute that reduces an individual “from a whole and usual person to a tainted, discounted one” (p. 3). Crocker, Major, and Steele (1998) suggest that stigma occurs when a person possesses or is believed to possess “some attribute or characteristic that conveys a social identity that is devalued in a particular context” (p. 505). Link and Phelan (2001) addressed the importance of power and discrimination in stigma; high-status individuals (doctors, lawyers, business people) with access to power and resources are not really injured by negative attitudes about their group (e.g., “Lawyers don’t really have to work too hard”). However, the one down status of low power groups results in the same statement being injurious (“People with mental illness don’t really have to work too hard”).
Interestingly, some advocates think “stigma” is a stigmatizing word; namely, it paints the person with a mark or stain. It suggests that the prejudice and discrimination which result from the mark is fundamental to the person. This implies something wrong with the person, and not the community that maintains the stigma. Concerns about stigma as a term was one reason why SAMHSA changed its nationwide, on-line program from the Resource Center to Address Discrimination and Stigma to the Resource Center to Promote Acceptance, Dignity, and Social Inclusion Associated with Mental Health (stopstigma.samhsa,gov). Despite these concerns, our group still decided to stick with “stigma” for now, partly because it signals to a large constituency what is meant by this all encompassing experience. It has more traction when talking to people from the broader population (e.g., people of like mind in other issues related to social justice).
Self-stigma and Public Stigma
Mental illness stigma has been distinguished by two relevant dimensions: self-stigma and public stigma (see Table 1). Self-stigma occurs when people with mental illnesses internalize stereotypes, apply the attitudes to themselves, and suffer diminished self-esteem and lessened self-efficacy (Corrigan, Watson, et al., 2006; Link, Struening, Cullen, Shrout, & Dohrenwend, 1989). One outcome of self-stigma is what we call the “why try effect” (Corrigan & Rusch, 2009), the tendency for individuals to believe stereotypes about themselves suggesting they are unable to meet the demands of a particular task (e.g., a job) and as a result, decide not to attempt this task (forgo applying for or trying hard at one’s job).
In contrast, public stigma occurs when large segments of the general public agree with the negative stereotypes. Stereotypes of mental illnesses include dangerousness (people with mental illnesses are unpredictable and potentially violent), blameworthiness (lack of moral character explains why the person has mental illness), and incompetence (people with mental illnesses are not capable of real work) (Jones et al., 1984; Rabiner, Wells, Struening, & Schmeider, 1983). Public prejudice leads to varied types of discrimination including loss of opportunity (don’t hire someone with mental illnesses because he or she may harm co-workers), taking away self-determination (an authority figure makes decisions about the person’s goals and the type of treatment to attain these goals), and segregation (in the past, this was removing people with mental illness to hospitals but now occurs in the form of psychiatric ghettoes.)
Changing the Public Stigma of Mental Illnesses
Stigma change programs vary for public stigma compared to self-stigma. The research literature on the two domains is vast, with approaches that represent public stigma meaningfully different than self stigma. In addition, erasing the public stigma of mental illnesses has become a top priority for advocates and is the focus of this paper (cf., President George W. Bush’s New Freedom Commission; Hogan, 2003). Thus, we focus on the assessment of stigma change programs aimed at reducing public stigma. Approaches for public stigma change have been divided into three paradigms based on social psychological research especially related to ethnic and gender minorities: protest, education, and contact (Corrigan & Penn, 1999).
Protest
Protest strategies highlight the injustices of various forms of stigma and chastise offenders for their stereotypes and discrimination. “Shame on us all for perpetuating the ideas that people with mental illness are just ‘big kids’ unable to care for themselves.” There is anecdotal evidence to suggest that protest can positively influence harmful behaviors (Wahl, 1995). For example, through protest NAMI StigmaBusters played a prominent role in getting ABC to cancel the program “Wonderland.” Aired originally in 2000, the show portrayed people with mental illnesses as dangerous and unpredictable. In the first episode, a person with mental illness shot at several police officers and stabbed a pregnant psychiatrist in the abdomen with a hypodermic needle. StigmaBusters targeted the show’s sponsor who, in response to gentle concerns about moratoria, directed ABC to pull the program after just two episodes. This is an interesting case but findings about protest effects on behaviors are largely anecdotal, begging for research.
There is, however, some evidence that protest campaigns that ask people to suppress their prejudice can produce an unintended “rebound” in which prejudices about a group remain unchanged or actually become worse (Macrae, Bodenhausen, Milne, & Jetten, 1994; Wegner, Erber, and Bowman, 1993;Wegner & Schneider, 1989). In one set of studies, Macrae and colleagues (1994) found that research participants directed to suppress their stereotypes about skinheads showed greater stereotype activation, greater stereotype use and increased distancing from skinheads. We found that protest about mental health did suppress prejudices where other interventions did not (Corrigan, River, Lundin et al., 2001).
Education
Educational approaches to stigma challenge inaccurate stereotypes about mental illnesses, replacing them with factual information (e.g., contrary to the myth that all people with mental illnesses are incapable of getting a real job, many are able to obtain and maintain these jobs about as well as the rest of the adult population). Educators bestow knowledge that challenges the prejudices and stereotypes of this disability. Educational strategies have included public service announcements, books, flyers, movies, videos, web pages, podcasts, virtual reality, and other audio-visual aids (Finkelstein, Lapshin, & Wasserman, 2008; National Mental Health Awareness Campaign, 2002; SAMHSA, 2009). Some benefits of educational interventions include their low cost and broad reach. Given today’s computer software, an advocacy group can easily produce a PowerPoint show highlighting key myths and facts (Corrigan & Lundin, 2001) that can be disseminated across the internet the very same day. Quite a bit of research has been completed on educational approaches largely showing short term improvement in attitudes related to dangerousness and blame.(cf Arboleda-Florez & Sartorius, 2008).
There are interesting and unintended consequences of some educational programs. Consider programs where mental illness is presented as a brain disorder (Brown & Bradley, 2002; Mann & Himelein, 2008). They are meant to challenge stigmatizing beliefs about people with mental illnesses being responsible or to blame for their illnesses. Typically, these programs attempt to counter beliefs about choice and culpability (e.g., the person decided to be mentally ill) with findings from medical science about the cause of most psychiatric disabilities (serious mental illnesses are largely caused by heredity or in utero teratogen). Consistent with expectations, research does show that educational programs with this kind of focus result in less stigma related to what is known as onset responsibility; people with serious mental illnesses are to blame for originally contracting the condition (Weiner, 1995, 2008). However, research also shows education about biological roots sometimes worsens offset responsibility such as views about disease prognosis (Phelan, 2005; Phelan, Yang, & Cruz-Rojas, 2006). Participants in education interventions are more likely to believe people with mental illnesses are less responsive to treatment, unable to overcome their disabilities because the disease is “hardwired”, and limited to the least of jobs and housing.
Contact
The third strategy for reducing stigma is interpersonal contact with members of the stigmatized group. Members of the general population who meet and interact with people with mental illnesses are likely to show decreased prejudice (Corrigan, 2005). Not all contact is effective however. Consider mental health service providers such as psychologists, psychiatrists, social workers, and nurses. Service providers have a great deal of contact with individuals with mental illnesses, yet they are among the most stigmatizing, an ironic effect given their charge to help people with mental illnesses (Schulze, 2007). Service providers are more likely to endorse stereotypes about people with mental illness being dangerous and lacking responsibility than the general public.
How can this be? Social psychological research has identified factors that seem to moderate contact effects (Allport, 1954; Pettigrew & Tropp, 2000), including equity of status between groups (Aronson et al., 1978; Weigel et al., 1975), one-on-one contact so that interactants can learn of similar interests and potentially cultivate a friendship (Herek & Capitanio, 1996; Levin, van Laar, & Sidanius, 2003), contact that includes a common goal (Cook, 1985), contact that involves something rewarding (Blanchard et al., 1975), and interactions with a person who moderately disconfirms prevailing stereotypes (Blanchard, Weigel, & Cook, 1975). When one considers mental health service providers who often have repeated contact with a person who is agitated and psychotic, these interactions do not enhance positive contact effects. Service providers and the consumers are not of equal status, the consumers often do not disconfirm stereotype (e.g., providers often interact with consumers when they are agitated, ill, etc), and consumers and providers are often not engaged in an activity together that is rewarding or includes a shared goal (Forster, Higgins, & Strack, 2000; Reinke et al., 2004).
Research has directly compared contact with education and protest programs in carefully controlled trials. Participants in stigma change programs that involve interacting with a person with mental illnesses show significant improvements in their attitudes, change that is significantly greater than protest, education or control conditions (Corrigan, et al., 2001). A subsequent study has demonstrated that attitude change after contact is maintained over time and has related change in behavior (Corrigan, et al., 2003).
Targeted and Local Stigma Change
Who should be the object of anti-stigma approaches? Many programs aim to change either an entire population (e.g., all television-watching Americans) or a subsample of important targets (e.g., landlords). Evaluating population-based stigma programs requires assessment of penetration and impact of the intervention; how many people observed the effort and did it measurably change stereotypes, attitudes, or behavior? These reflect two complex methodological issues that are discussed later in the paper. Alternatively, advocates target important and local groups of people to evaluate stigma change (Corrigan, 2004a). Important target groups include employers, landlords, legislators, educators, and health care providers. They are individuals who hold positions that partly reflect power or social control of people with mental illnesses. For example, employers might endorse stereotypes about people with mental illnesses not working hard and landlords might believe people with these illnesses are more likely to damage property. Stereotypes like these can undermine employment and housing opportunities for people with serious mental illness. Messages aimed at a particular target group can be crafted to address their specific concerns. For example, a goal of a contact -based anti-stigma program aimed at employers would be to replace myths about those with mental illnesses with information that most people with mental illnesses can work a regular job, especially with in vivo support.
Stigma change is not only targeted, but also local. Changing stigma is fundamentally a political endeavor, getting one group to change their beliefs and actions about another (Corrigan, Watson, Byrne, & Davis, 2005). Targeting employers to change prejudice and discrimination is good. But challenging the stigma of employers working in the Greater Lawn neighborhood of Chicago is more potent. Describing this community in terms of diversity (e.g., by ethnicity and SES), economic opportunity (availability of jobs), and resources (available mental health or educational programs) exquisitely advance anti-stigma programs. This issue is more fully discussed in a subsequent section on community based participatory research.
A Compelling Agenda
Clearly, identifying and changing the stigma of psychiatric disability has public health significance of the first degree. The US SAMHSA has already invested in anti-stigma programs for mental illnesses, including public service announcement campaigns costing millions of dollars. What is the evidence that population-based programs like these have significant and meaningful effects? Research findings summarized in previous paragraphs suggest stigma change can fall victim to unintended consequences. Education on mental illness as a brain disorder can decrease positive prognoses of mental illness. Appeals to a moral authority to suppress stigma can actually make it worse. Some forms of contact may exacerbate mental illness stereotypes and discrimination. Hence, stigma change is a large public health priority that needs research to guide advocates to the best approaches.
DESIGN AND METHODS CONSIDERATIONS
Insights from services research (Wells, Miranda et al., 2004), social psychology (Morawski, 2000; Stone, Hosoda et al., 2008), and community psychology (Barnes, 1997; Trickett, 2009) provide useful insights for research that evaluates stigma change. In this light, we review an essential ingredient to research of anti-stigma programs: community-based participatory research. Then we review issues relevant to valid measurement: the effect of social desirability and the relationship of social validity to effect size (both are defined later in the paper). Finally, we focus on measurement issues, reviewing important dimensions in developing a new measure or adopting a current one.
Community-Based Participatory Research
Understanding stigma in terms of social justice and power calls for grassroots involvement. Dissatisfaction with the status quo is often championed by consumers of mental health services and hence they need to be included in decisions about stigma programs, which means they need to have an active role in evaluations meant to inform these decisions. Community-Based Participatory Research (CBPR) is the research agenda that includes consumers and other stakeholders to be full partners in all aspects of the evaluation program (Minkler & Wallerstein, 2003; Rogers & Palmer-Erbs, 1994). CBPR rests on two principles: perspective and politic (Minkler & Wallerstein, 2003). First, the diverse backgrounds and varied perspectives inherent in CBPR infuse theoretical understanding and corresponding research design with this diversity. Second, many advocates flex their political power by consuming research findings, integrating them into policy, and using their authority and networks to realize important change.
What then does the CBPR team do? Similar to administrative councils of any human service program, the CBPR team is responsible for all activities related to: research on stigma change programs that are then used to broaden campaigns meant to control prejudice and discrimination. This approach signals a paradigm shift for many researchers (Bogart & Uyeda, 2009; Christopher et al., 2008). Basic to their work under CBPR, social scientists need to engage with other stakeholders in all aspects of effective evaluation. This includes understanding the problem, describing corresponding anti-stigma approaches, delineating methods and measures meant to test the approach, collecting and analyzing the data that emerge from the design, and making sense of the findings. Indeed, “engage the stakeholder” is not sufficient as a descriptor. Partnership is more appropriate; scientist and consumer share all decisions about a study. This means the researcher needs to educate stakeholders about key aspects of the research approach (Bogart & Uyeda, 2009; Chene et al., 2005). However, CBPR is not a unidirectional process suggesting consumers are inferior to investigators. Consumers, in turn, are responsible for educating researchers about the social significance of the goals of the study.
Heretofore, we have been using the generic “diverse stakeholder” as the subject of our discussion. Who is this stakeholder? First would be consumers of mental health services, though consumer is not a homogeneous construct by any means. Advocates with mental illnesses have been alternatively labeled consumer (the person using mental health services), expatient (in part, suggesting no longer needing treatment and, in part, seeking to distance oneself from the mental health system), and survivor (people who have not only overcome their illnesses, but endured the treatment!) (Covell, McCorkle, et al., 2007). Approach to the mental health system, and therefore to mental health stigma, varies by role, with survivors, for example, likely to be more critical and skeptical of the system than those who understand themselves as current consumers of care.
Other stakeholders may need to be included in a CBPR effort. Family members of people with serious mental illnesses often have different priorities than their relatives with the disability. Family takes many forms; often parents, but also siblings, children, and spouses. Service providers may be valuable members of a CBPR team and are likely to vary by discipline and authority (e.g., physicians usually have legal authority on multi-disciplinary teams). If a central strategy for stigma change is targeting persons in powerful roles, then representatives from these groups might also be useful partners in CBPR. For example, it would be important to include employers in an anti-stigma research project on hiring people with mental illnesses. Who better knows the prejudice and discrimination that employers may hold against individuals with mental illnesses than employers themselves?
Diverse also speaks to the demographic of the CBPR. Research suggests stigma and stigma change vary by ethnic group (Farina, 1998). This is a complex relationship however, with effects depending on whether ethnicity reflects the stigmatizer or the stigmatized. Other important demographics are gender, sexual orientation, and SES (Farina, 1998).
Validity of Measures
Link and colleagues (2004) argue that stigma change measures need to demonstrate the kind of psychometric evidence basic to standard psychological research (Anastasi, 1961). These include reliability and validity, but also issues of relevance when specifically measuring stigma change: social desirability and social validity. These qualities of sound measurement are reviewed herein.
Social Desirability
Social desirability is the tendency for people to say what they believe conforms to cultural mores, even if it varies with what they might otherwise report to be their “real belief.” In research studies assessing stigma, this means an interviewee might respond to a stigma assessment in a way that proffers a positive image (e.g., someone who is egalitarian) to others or to oneself (Tourangeau & Yan, 2007). People avoid saying “the mentally ill are all dangerous!” to escape social opprobrium. Attitude measures may be especially susceptible to desirability biases because the real belief is largely private; a research assistant, for example, cannot discern one’s attitudes about a group by watching them. Social desirability is especially troublesome in research designs that include pre- and post-test assessment. Study participants may discern, from the repeated measure, that researchers are expecting a decrease in stereotypes because of the intervening task. As a result, study participants may diminish public endorsement of stereotypes which would lead to an incorrect inference about the anti-stigma program efficacy and/or effectiveness.
There are several ways in which social desirability may be subverted. Building privacy into the administration of stigma measures may help; i.e., allowing participants to complete study materials anonymously. Measurement might include unobtrusive behavior observation by an outside party; external observation should not be reactive to social desirability compared to self-report (e.g., King, Shapiro, Hebl, Singletary, & Turner, 2006). Use of a multi-factor random controlled trial (RCT) can control for social desirability. At its simplest, this would be a two (group: anti-stigma versus no intervention control) by two (trial: pre and post) design. Consistent with social desirability effects, participant responses for those only exposed to the control group may significantly improve. Inclusion of control may lead to direct assessment of social desirability effects. It is a test of the interaction of group and trial effects that support conclusions about the benefits of the anti-stigma approach. Finally, there are assessment strategies that the participant may be unable to discern its connection to pro-social appearances. Reaction time tasks that are expected to reveal implicit prejudices are a good example and are discussed in the next section.
External Validity and Social Validity
Much of the published research on changing mental illness stigma is done with college students which challenges the external validity of the research. This problem is not unique to stigma change research however; efficacy and effectiveness researchers struggle with similar concerns. For example, the first round of rigorous study on clinical interventions is typically efficacious in nature and conducted in well-controlled settings with randomized designs. This kind of rigor narrows questions to relatively homogeneous groups (e.g., adult perspectives about people with major depression) with exclusion criteria that omit significant and important segments of the population (people who abuse alcohol and other substances). In some ways, stigma research on college students is similar to efficacy studies. Using social psychological paradigms, researchers adopt tightly controlled lab settings to manipulate the narrow scope of an anti-stigma approach on a homogeneous sample. Results are rich in internal validity, but limited in generalizability. Clinical research is expanded to broader settings and subject pools when investigations are conducted in the real-world. In like manner, focus on targeted and local stigma change facilitates effectiveness research in this arena. Researchers seek to expand findings from what has carefully been described in social cognition to understand how, for example, employers in a large city or small town landlords respond to anti-stigma programs.
Like comprehensive approaches to clinical and services research, decisions about findings should not be limited to statistical significance, but should also include measures of sizable change, which has also been defined as social validity. Social validity comprises two measurement issues relevant to this paper: social comparison and subjective evaluation (Finn & Sladeczek, 2001; Kazdin, 1997; Kennedy, 2002; Wolf, 1978). One way to make sense of stigma change is to compare impact against the prototypical, social needs of peers. For example, examining the impact of a stigma change strategy on employment makes more sense when we describe the prototypic goal as actual behavior, such as interviewing people with mental illness for available openings. Of more relevance to assessment issues and stigma change is subjective evaluation. This was a fairly radical consideration when posed in the 1970’s but it clearly mirrors contemporary views about empowerment and self-determination. Concerns about social validity of stigma change approaches are based on a consensus of priorities. This can be described at the population level; based on group evaluation, what do people with serious mental illnesses subjectively report to be important goals of stigma change. Research can sometimes provide useful information that describes subjective evaluation. One study of more than 1800 people with serious mental illnesses on a slew of social forces impacting their life goals puts the general notion of discrimination high on the agenda; i.e., findings showed discrimination related to mental illnesses is as prominent or more so compared to racism, ageism, and sexism (Corrigan, Thompson et al., 2003). Results of the survey also highlighted settings and situations in which discrimination is worse; employment was of most concern, being almost twice as frequent as the next issue on the list: housing (29.9%). Wahl (1999) completed a mixed methods program of research to more fully flesh out the troubles of discrimination. As in the previous survey, research participants reported employment as a significant place where stigma occurs. “In job interviews, once they find out I’m on medications, I’m usually dismissed. They don’t give the reasons. But that is the reason.” (p.81). We do not mean to pose a definitive list of priorities but rather to highlight the need for similar, largely qualitative, research that establishes socially valid outcomes for specific stakeholder and target groups.
In trying to make sense of priorities from large samples, “social validity” also reminds us of the importance of targeted stigma change. Employers are important because prejudice and discrimination undermine job opportunities or accommodations that help the person be successful on these jobs. In addition, social validity is salient in relation to local issues. People with diverse backgrounds are likely to prioritize goals differently, in essence defining social validity distinctly. In both cases, the CBPR team needs to survey stakeholders with a vested interest in stigma change in their community.
Direction of effects is also relevant for assessing social validity. Much of the research on stigma change has documented reductions of prejudice and discrimination with the assumption that lessening these phenomena is a salient priority. This approach has its limits however. The purpose of anti-stigma programs is not solely to take away negative thoughts or experiences, not solely to remove beliefs about dangerousness and coercion. Of additional importance are improvements in what we have come to call affirming attitudes and behaviors. Measurement needs to also examine positive attitudes (e.g., with proper support, people with mental illnesses are as able to get and keep jobs just like the overall adult population). Stigma change programs are valuable not just if they reduce employer behaviors related to hiring discrimination, but also increase relevant behaviors such as job interviews and reasonable accommodations.
Five Domains of Assessment
The 2004 paper by Link and colleagues (Link, Yang, Phelan, & Collins, 2004) delineated important ideas about self-report measures of attitudes, emotions, and consciousness raising; these points are incorporated below and attributed to Link et al. where appropriate. Their review focused largely on one of five domains of stigma measures which are summarized in Table 2: attitudes and emotions. Our goal is to expand on their ideas by including assessment approaches in other domains. Table 2 summarizes the costs and benefits of each approach often in terms of addressing social desirability and external validity. We remind the reader that our focus here is solely public stigma. There are measures of self-stigma, family stigma and other experiences which are not included here and need to be considered for research on those phenomena (cf., Corrigan & Watson, 2002; Link et al., 2004 for measures of self-stigma as an example of related research). Our summary starts with behaviors because we believe social priorities identify behavior change as fundamental to assessing real and beneficial improvement. Advocates might concede benefits of attitude change but assert reductions in discrimination or enhancements of affirming behaviors as the essential product of stigma change. The other four domains in Table 2 gain meaning in light of their relevance to behavior change.
Table 2.
Measurements that assess stigma change.
| Measurement Domains | Conceptual Significance | Benefits | Costs |
|---|---|---|---|
| 1. Behavior | Discriminatory: Coercion Segregation Benevolence Affirming: Support Opportunity Allocation Service participation |
- Most conceptually compelling domain - High on priority lists - High face validity |
- Can be resource heavy: a research assistant independently documents a research subject‘s behavior. - Sometimes requires data gathering outside of setting. - Can be assessed by self report which increases social desirability effects. |
| 2. Penetration | Recollection of medium and Message | - Potentially broadest impact | - Expensive - Challenges psychological models of data gathering - Diminished impact on behaviors |
| 3. Attitudes and Emotions | Attitudes and stereotypes Emotions Behavior intentions |
- Easy to self-administer - Good content and face validity - Good reliability and construct validities - Easy to develop and disseminate |
- Susceptible to social desirability - Unclear connection with behavior - Less social validity |
| 4. Knowledge and mental health literacy | Knowledge about disease/disability Knowledge about the treatment Error choice as knowledge measure directly representing stigma |
- Easy to develop this kind of measure. - Directly relates to impact of education programs. - May guide participants seek help in the future. -Less susceptible to social desirability. - Easy to administer |
- Mixed information on education effects. - No direct impact per se on stigmatizing attitudes and behaviors |
| 5. Physiological and information processes | Proxy for arousal and emotion Consciousness and awareness Implicit and explicit information processing |
- Less susceptible to social desirability - Triangulates self-report and knowledge data |
- Need special equipment and skills to administer - Cumbersome for many anti-stigma evaluations - Social and construct validity not clear |
Measuring Behavior Change
Changing behavior is fundamental to the goals of stigma change; one person with mental illness made the point well. “It would be nice if people didn’t think bad things about me but what I really want is for them to stop blocking me from getting back to work.” Behavioral assessment has benefits and costs which are reviewed in Table 2. In vivo observation is frequently posed to be the prototype of behavioral assessment; direct scrutiny of discrete behaviors has large and compelling importance. Given this, two points are important and considered about measuring behavior change. First, the “what” of behavior is more complex than facile tracking of discrete actions. Behaviors can be negative discrimination or positive observations. They can represent long term goals or intermediate responses that help the person achieve these goals. Second is feasibility of using behavioral observation. These data are frequently more resource demanding than the self-administrative instruments that comprised many of the other measures in the Table.
Types of behavior
There are two examples of discriminatory behavior in Table 2. Effective stigma change programs decrease treatments that promote segregation and coercion. By segregation, we once meant state hospitals that distanced people from their community, family, and friends. State hospitals have slowly closed but segregation is still evident (Pescosolido, Wright, & Kikuzawa, 1999). People with psychiatric disabilities are often sent to mental health ghettoes, which are places in large urban areas to where people with serious mental illnesses are indiscriminately bussed and where services for these people are noticeably absent. People with mental illnesses often end up in neighborhoods where those from the lowest SES live. Quality of housing is poor, crime is high, and services are lacking. Coercion is a “should.” People with mental illnesses should see a doctor weekly, should sign up for social security entitlements, should attend nightly current events groups, and should work at sheltered workshops. Coercion also leads to “should nots;” people with mental illnesses should not seek out regular jobs because their disability prevents them from successfully doing so. They should not move out of a halfway house because they are unable to live independently. Some advocates have explicitly called for coercive treatments (like outpatient commitment) to keep people with mental illnesses from harming the public: the dangerousness stereotype (cf the website of the Treatment Advocacy Center, http://www.treatmentadvocacycenter.org). The challenge for scientists is how to outline “shoulds” and “should nots” and how to measure these outlines.
Behavior change is more effective when trying to increase a positive behavior than decrease a negative. Therefore, the bottom of the behavior domain in Table 2 spells out helping and other social behaviors that often arise from effective anti-stigma programs. Non-specific responses include instrumental support (willingness to help a person with serious mental illness on day-to-day tasks like getting to work in the morning) and interpersonal support (becoming friends with a person with serious mental illness). This kind of support leads to more opportunities. The employer, for example, interviews and hires more people with serious mental illnesses. Support at a governmental level yields greater allocation of funds to mental health services. People with mental illnesses are better able to achieve goals when they have access to and participate in evidence-based practices. These practices are most effective when services seek to engage people in key roles like employers and landlords. A final benefit of anti-stigma programs may be increasing the frequency with which a person in key roles participates in these programs such as employers in job development and supported employment.
Feasibility of assessments and related analyses
Getting a job is often cast as a categorical variable: yes or no, did the person find work? This kind of variable restricts the power of statistical analyses. Contrast these to variables that may be continuous in scope because they are relatively micro in perspective; i.e., reflecting a change in the frequency of a discriminatory or helping behavior. Employer behavior could be summed in terms of discrete interactions with job applicants with serious mental illnesses. We would expect to show employers who benefitted from an anti-stigma program to provide more time with the person in the interview, show positive content, and demonstrate supportive interpersonal style. This point indicates an important decision regarding assessment; many assessments rather than one or two, are likely to yielded more powerful data. To do this, triangulation would include employer measures of interviewing, hiring and accommodating actions.
Real-world observation yields data that directly represent interactions reflecting discrimination. In this kind of paradigm, research assistants unobtrusively follow a research participant, counting each time the person manifests the behaviors of interest. Collecting data using behavioral observation is costly and operationally challenging; e.g., project directors need to develop an assessment plan for raters that include the observation template and issues related to maintaining reliability and validity. Ongoing training is needed to show that raters maintain reliable data gathering. Observational data is more compelling when assessments are collected at natural times of activity when these kinds of interviews occur. Therefore, data-gatherers need to be available during a range of instances, another draw on resources.
Demand characteristics like the Hawthorne Effect (Chiesa & Hobbs, 2008) may undermine the validity of real-world observations. Raters sitting off to one side of face-to-face interactions may likely bias employer and applicant behaviors. For example, employers may act more supportive when raters are in the room. One way in which researchers have addressed this problem is videotaping interactions-of-interest. Videotaped information is then carefully coded, forming the basis of subsequent analyses.
Archival data may serve as an alternative platform for behavioral observations. These data are often records reflecting specific behaviors sometimes indicative of stigma change. For example, a research team can find out from personnel records how many employers hired people with mental illnesses in the period after an anti-stigma program. If available, this information can provide highly reliable and valid data, especially when target behavior represents categorical responses: yes or no, did the employer return the phone call of an applicant with mental illness.
Role play tests are sometimes used instead of real-world observations (Lane, Slavin, & Ziv, 2001). Fewer resources are needed because the exchange is more controlled. Research shows role plays are frequently sensitive to behavioral change programs in clinical trials (Bellack et al., 2006). The major concern is validity. Role play-generated behavior is not clearly indicative of real-world activity. There would seem to be less validity in examining employers’ reaction to a confederate than what they do in their private office.
Guerilla observation has been used to assess discriminating behavior; these are situations where a social problem is set up involving an unaware employer. For example, confederates find real job advertisements listed in daily newspapers and then contact the employer. During the phone call, the confederate “accidentally” lets the employer know of his or her mental illness at a randomly selected time during the interview. The confederate might then obtain an index of behaviors by, for example, counting key micro-behaviors. Prominent among these would be amount of time (number of minutes) on the phone or during the interview. This kind of planned disclosure provides an opportunity to sample discriminatory behavior that might not naturally arise during observations. It has been used by housing advocates, who send an African-American married couple to a landlord with available rental property. The approach seems to identify landlords who red line neighborhoods in which their property is situated; they discriminate against people of color. This kind of approach has significant ethical problems however. Experimental manipulations pose risk by placing employers in a civilly actionable position. The confederate could catch an employer failing to comply with the Americans with Disabilities Act so that law suits or other legal recourse may result. In addition, debriefing the employer to a study in which he or she was never consented is likely to generate hostility which, among other problems, may dissuade the employer from future interactions with people with mental illnesses.
Self-monitoring may have a role in behavior assessment (Ickes, Holloway, Stinson, Hoodenpyle, & Graham, 2006). Namely, people report in retrospect how often they completed an indexed behavior; how many times did the employer interview a person with mental illnesses in the past month. Not all respondents are good historians and may over-report real activities. Erroneous recollections will expand as the retrospective period grows (e.g., from one month to one year). These kinds of self-reports also have the same challenges as social desirability on attitude and other psychological perspectives described below. Better behavioral measures may be achieved when research participants are instructed to count each time the indexed behavior occurs in an assessment period. “I want you, the employer, to mark each day the number of times you called a person with mental illness about an available position.”
The Penetration of Social Marketing Programs
Social marketing programs are large scale, often population-based, efforts to decrease prejudice and discrimination of serious mental illness (Corrigan & Gelb, 2007; Kotler, Roberto, & Lee, 2002). Public service announcements (PSAs) are the best known example of social marketing to decrease mental illness stigma. Traditionally, PSAs are well-crafted video- and audio-taped vignettes that challenge stigma; they are then disseminated to television and radio outlets. More recently, video and audio materials have also been uploaded for online use including websites for social networking, blogs, and other well-populated venues. Two points are basic to evaluating PSAs: impact and penetration. Impact has a similar meaning to the behavior, attitudes, and emotions listed in Table 2. Namely, do these measurement domains improve as a result of the anti-stigma PSA? Penetration is more a novel construct here; it represents the frequency of people in a market who have observed and remembered the PSA. Limited journal space does not provide for comprehensive development of penetration research which largely evolved from supra-psychological disciplines such as marketing, sociology, and public health (Broidie et al., 2007). Still, it has significant implications for setting the anti-stigma agenda in the public health sector and is briefly considered here.
We provide an example to illustrate the relevance of penetration to evaluating stigma change. We consider a series of PSAs -- produced by the National Mental Health Awareness Campaign (NMHAC) about ten years ago -- to illustrate penetration. The 30 second advertisement addressed adolescent concerns about depression and suicide using Sister Hazel’s song “Changing Your Mind.” Market penetration can be indexed as the degree to which a customer base is aware of and uses a product (Baldinger Blair, & Echambadi, 2002). In this light, penetration-related assessment is examined in two ways: awareness and representation. First, is the population at whom a stigma program is directed aware of the change approach? For example, how many adolescents recall the NMHAC PSA? Recalling information from ten years ago is not likely to be valid and is an unfair test of PSA penetration. More important is how many people from the desired population recall the PSA while that PSA is active? In June, 2009, SAMHSA had produced and circulated a PSA-based program titled, “What a difference a friend makes.” One way to examine penetration of this program would be to assess samples of the population to determine how many people remember seeing any of its variations (e.g., TV and radio spots or magazine stories). Note that we qualified assessment for PSAs that are “active.” Many PSA’s are time-limited by contractual agreements that require PSAs in the TV and radio industries to be renewed.
Penetration might be assessed in terms of recollection or recognition. When assessing recollection, a research subject is asked to write down everything related to mental illness he or she has seen on television in a particular time period (the last week or month, for example). This is a complex measure of penetration that yields compelling support if a reasonable part of the population recalls the PSA; reasonable depends on access to the population and is a function of coverage and sampling error (Dillman, 2000). Recollection is a more cognitively demanding process than recognition and hence, likely to show limited findings. As a result, recall measurement is frequently followed by recognition tasks. For television, research participants are provided video or audio tracks of PSAs embedded in other similar stimuli (e.g., an advertisement for household cleaning products) and asked to identify those segments they have observed in the past. Finally, subjects who recall or recognize the PSA are asked to report the gist of its message. In principle, it seems unlikely that a PSA will provide positive benefits if the person cannot remember its essential points.
The second goal of assessing penetration is cultural representativeness. Are those who recall a PSA diverse in such important areas as ethnicity, gender or SES? This information provides trenchant markers of external validity; not only does the sample mirror the overall population, but examination of subgroups suggest relative impact across important demographics (Brondolo, Gallo, & Myers, 2009; Courtwright, 2009). Do African Americans, for example, change prejudices and discrimination as a result of the anti-stigma strategy? Or turning the question around, what approaches change African American stigma about people with mental illnesses?
There are also behavioral markers of penetration. For example, SAMHSA’s “What a difference a friend makes!” included a web address to which interested parties might have gone to learn more about mental illness and appropriate services (http://www.whatadifference.org/). Developers of the PSA assumed the public was benefitting more from stigma change when more fully understanding mental health services by travelling to the website. Data from the PSA evaluation included hits to this web address; between 6,000 and 75,000 surfers went there each month in the two years following the PSA’s release (Marshall, 2009). Process variables are also available; in this case, process represents dissemination. Did the PSA package actually get air time or print exposure? An example is time donated by media outlets to show the PSA. This is often reported in dollars; in-kind funds for “What a difference a friend makes!” ranged from $27,000 to as high as $2.5 million depending on the media and community.
We may be oversimplifying penetration by suggesting that change brought about by anti-stigma strategies is only effective when its impact is conscious. Assessing penetration at the population level requires cognitive assessment of recall or recognition measures about which the research participant is aware. Stigma is also measured by implicit or out-of-conscious processes, a point that is better considered in a subsequent section on Information Processing and Physiological Measures. The point here is that future research on penetration may need to include indices of implicit memory to more fully examine PSA impact.
Attitudes and Emotions
Commonly found in a social psychological approach to stigma research is assessment of attitudes and emotions; see Table 2. Stereotypes are statements of belief or attitudes about a group of people usually in a negative light. Emotions straddle cognition and the more basic, physiological processes developed in the last measurement domain at the end of the paper (Peters, Burraston, & Mertz, 2004; Varas-Diaz & Marzan-Rodriguez, 2007). Basic to assessment of many of these constructs is written vignettes and subsequent questions (Link, Yang et al., 2004). These are brief paragraphs that describe a single person (e.g., someone with psychiatric disability such as Ms. Jones) in real-world context (who you see every Sunday at church). Independent variables are manipulated to examine specific questions about stigma. For example, Ms. Jones could be described as a person who lived has a wild lifestyle until the onset of schizophrenia at age 23 (high blame) OR as someone who seemed to have a normal life until psychosis emerged at the same age. The latter description suggests a biological onset which diminishes attributions of responsibility. Studies have also shown more variance is obtained for rating an individual (Ms. Jones) rather than an amorphous class (such as all people with psychiatric disability) (Corrigan, Markowitz et al., 2003; Corrigan, Rowan et al., 2002). Immediately after reading the vignette (which is sometimes presented as a video), research participants are instructed to fill out pencil-and-paper measures of psychological perspectives. They are asked, for example, their impressions of stereotypes about Ms. Jones on a Likert Scale representing relevant ratings of agreement (e.g., 7 = highly agree). Test items parallel the stereotypes identified for psychiatric disability: violent (Is Ms. Jones dangerous?), blameworthy (Ms. Jones caused her mental illness!), and incompetence (Ms. Jones is not capable of getting a real job!). Some of the oldest and most used measures of stigma address beliefs about individuals with mental illness. Two of these are the Opinions of Mental Illness Scale (OMI: Cohen & Struening, 1962; Struening & Cohen, 1963) and the Community Attitudes toward the Mentally Ill scale (CAMI: Taylor et al., 1979, 1981). These measures reveal the same factors as those related to violence, blame, and incompetence.
Prejudices are agreement with the stereotypes accompanied by emotional reaction. For example, people with mental illnesses are to be feared because they are dangerous. They are pitiable because they have no control over their life. Emotional reactions are measured using self-report instruments (Link et al., 2004); semantic differentials, for example, are frequent. Research participants are instructed to rate their response to a vignette actor on a continuum anchored by polar opposites of the affect-of-interest: frightened-calm or acceptance-rejection.
Behavior intention
Behavioral assessment is typically viewed as direct observation, described earlier as a labor intensive task. Social psychologists have alternatively framed behavioral intentions as a relevant and important perspective included here under attitudes and emotions. These may, for example, be assertions by research participants that they will decrease discriminatory behaviors and increase affirming ones, at some future time. “I intend to interview Ms. Jones for a job in the future.” In this light, researchers have framed behavior intention as a self-reported psychological variable and a proxy of behavior, a much easier approach to directly observing action. Using a Likert scale, employers in an anti-stigma program might be asked to respond to questions such as, “How likely are you to interview a person with mental illness?” on a seven point scale (7 is “very much”).
Behavioral intentions, which are proximal to the time when the behavior will occur are more likely to reflect actual change. Reporting on intention to hire Ms. Jones tomorrow is likely to reflect the actual behavior compared to intentions for a response expected more than a week later. Behavioral intentions are developed more thoroughly in the Theory of Reasoned Action (TRA: Azjen& Fishbein 1980, Fishbein & Azjen, 1981) and its more contemporary Theory of Planned Behavior (Azjen & Manstead, 2007). According to these perspectives, behavioral intention is the result of attitudes and subjective norms. Attitudes about the intention reflect the interaction between beliefs and evaluations. Belief is the perceived probability that the intention in fact will lead to overt behavior and evaluation is a good-bad rating on how satisfying the overt behavior would be. Subjective norms are the combination of beliefs obtained from others (what one’s spouse, mother, or co-worker believes about the overt behavior) and the person’s motivation to comply with that person. An employer’s intentions towards hiring a person with mental illness is influenced more by some people (e.g., a business partner) than others (a second cousin). Also note that the importance of a person’s beliefs varies by situation. The business partner is more influential in work settings rather than the family home. This perspective is reminiscent of clinical models of planned change representing the perceived costs and benefits of a goal behavior (Arkowitz, Westra, Miller, & Rollnick, 2008; Prochaska, Wright, & Velicer, 2008).
A useful path model
Weiner’s (1995) model of causal attribution proffers a set of paths that brings together stereotypes, emotions, and discriminatory responses; an example is provided in Figure 1. Weiner argued that attributing personal responsibility for a negative event (e.g., “That person is to blame for his crazy behavior.”) leads to anger (“I'm sick and tired of that kind of irresponsibility!”) and diminished helping behavior (“I'm not going to interview him for a job.”). Conversely, attributing no blame for a harmful event (“He can't help himself; he's mentally ill.”) leads to pity (“That poor man is ravaged by mental illness.”) and the desire to help (“I'll give him a try at part-time job on the loading dock.”). The specific association between causal attribution, mediating anger or pity, and subsequent behavior has been validated on several samples (Corrigan, Markowitz et al., 2003; Dooley, 1995; Graham, Weiner, & Zucker, 1997; Greitemeyer & Weiner, 2003; Menec & Perry, 1998; Reisenzein, 1986; Reyna & Weiner, 2001).
Figure 1.
Two models representing the paths between cognitions, emotions, and behaviors.
Perspectives on dangerousness are perhaps the most common and egregious stereotypes and are noticeably absent from the attribution model of mental illness stigma (Link, Stueve, & Phelan, 1998; Phelan, Link, Stueve, & Pescosolido, 2000). As a result, Figure 1 includes a second path integrating cognition, emotion, and behavior. Beliefs that a person with mental illness is dangerous result in fear, which in turn lead to avoidance (Angermeyer & Matschinger, 1996; Wolff et al., 1996). Employers, landlords, and physicians want to distance themselves from the person with mental illness which leads to poor employment, housing, and health care. Perceived danger and resulting fear can also result in coercion and segregation.
Measurement of Knowledge
According to advocates and researchers examining the impact of education, the stigma of psychiatric disability will decrease as information about the disability is better learned by the population (Pescosolido et al., 2008; Thornicroft et al., 2008). Namely, knowledge about psychiatric stigma and information to challenge it will decrease public prejudices against people with mental illnesses. This may occur because facts directly counter stigmatizing myths. Alternatively, more information may enhance consciousness raising; the public is more aware of mental illness which may lead to a more critical assessment of stigma. An additional benefit of knowledge is its impact on another form of stigma: what has been called “label avoidance” (Corrigan, 2004b; Link & Phelan, 2001). Many people do not pursue mental health interventions in order to avoid the stigmatizing label that accompanies it. People who know what to expect in services may be more likely to remain engaged and not drop out prematurely.
Assessments of knowledge used in stigma research are similar to the kind of test items college students might encounter in an introductory course (El-Gadi, Abduher, & Sammud, 2008; Martiniuk et al., 2007). Multiple choice is most common, but true-false and fill in the blank have also been incorporated into outcome batteries. Knowledge measures are relatively easy to administer and score. In order to expand on information gathering, essay questions have been used to assess knowledge; e.g., what are symptoms of depression? This kind of method is more flexible in the research subject’s response. However, rigorous data from these texts require coding schema and trained raters to yield reliable findings. In addition, responses to open-ended questions can be confounded by laconic style. People who write down many words yield more information than those who provide single word or phrase responses to the question.
Error choice
One measure that is ostensibly knowledge-based but actually a direct measure of stigmatizing attitudes is called the error-choice method (Antonak & Livneh, 1995a,b; Clarke & Crewe, 2000). Research participants are presented a multiple choice test (usually about ten questions) and instructed to circle the correct answer for each. For example, Q1.,What percentage of people with serious mental illnesses are dangerous in their lifetime?
| a. 3% | (+2) | c. 10% | (−1) |
| b. 5% | (+1) | d. 15% | (−2) |
The range of responses to the question are spread out on a continuum with research-based factual information being the central tendency of the continuum; in Q1, the range is 3 to 15% and the average is 7.2% of people with mental illnesses who are dangerous in their lifetime (Corrigan & Watson, 2005; Swanson, Holzer, Ganju, & Jono, 1990). Valence of responses are then proportionately determined above and below the midpoint. Reponses above the midpoint -- (c) 10% or (d) 15% -- signify the respondent as viewing people with mental illnesses as more dangerous than they are which represents a stigmatizing perspective. Alternatively, research participants selecting a response below the midpoint are choosing options contrary to the stigmatizing perspective. Responses are anchored (+2, +1, −1, −2) with negative responses representing the stigmatizing pole. Sum of all scores represent prejudice with more negative scores being more stigmatizing.
One benefit of error-choice approaches is diminished social desirability. Most research participants are unaware the instrument is directly testing prejudice. Error-choice measures are also easy to administer. They can be disseminated to research participants in face-to-face assessments as paper-and-pencil measures or as on-line tasks. Most research participants are familiar with these kinds of tasks and are, therefore, easily oriented to task. Although the measurement strategy has been used to assess stigma related to epilepsy (Antonak & Livneh, 1995a), we are unaware of this assessment for the stigma of people with psychiatric disability.
Assumes there are facts
Presumed in knowledge models is fact; namely, that there is a body of empirically supported information that defines the necessary font of knowledge which, in turn, grounds the true response to questions. Clinical psychology and psychiatry are scientific endeavors, seemingly rich in factual information and fruitful for the development of knowledge tests. Experts and consensus panels have sifted empirical findings and agreed or otherwise defined facts about mental illness. The DSM-IV is an example of such an instrument (American Psychiatric Association, 1994, 2000). For example, experts concur that schizophrenia is evident partly by verbal hallucinations; compare this to people with bipolar disorders who experience depression and manic episodes.
There are limits to a “factual” assumption; consider this example from an error choice item (Q1). Results from the Epidemiological Catchment Area (ECA: Robins & Regier, 1991) study and the National Comorbidity Survey (NCS: Kessler, 1995) suggest the prevalence of dangerous behaviors ranged from 1.9 to 16.0% depending on psychiatric disorder for an average of 7.2% (Corrigan & Watson, 2005; Swanson et al., 1990). The prevalence rate excluded people with alcohol and other substance abuse disorders. Ratios increased two fold or more when the person with psychiatric disorder had a co-morbid substance abuse disorder. These data show that values representing violence per diagnostic group vary across epidemiological studies (Kessler, 1995; Robins & Regier, 1991). These are huge variances that challenge attempts to generate factual answers.
The direction of effect
Knowledge may be included as a dependent or independent variable depending on the research question. It is assumed that people with more knowledge about mental illnesses will decrease their prejudice against the group. In this view, knowledge is a dependent variable that may represent changes due to the anti-stigma program. Hence, knowledge tests are included as measures sensitive to this change. However, stigma may also affect knowledge as an independent variable. Research has shown that people who are prejudiced against a group will adopt a narrow perspective when gathering future information about that group (Fyock & Stangor, 1994). We would expect to show that people who endorse stereotypes about mental illnesses would easily learn information about incompetence, blame, and dangerousness and would seem to ignore data on recovery and self-determination. The relevant variable here is assimilation; how well does a person include “new” information, discordant with current prejudice. Measurement strategies need to be included in the design of anti-stigma evaluations.
Measures of Physiological and Information Processes
Researchers have adapted models from physiological psychology and information processing to better understand stigma. These processes are useful for explaining dynamic qualities of prejudice and discrimination of mental illness but their role as important and measurable phenomena to evaluate stigma change is not yet clear. Emotions were discussed earlier in the paper where it was paired with attitudes. In this section, relevant physiological processes are discussed. Classic research has shown central nervous activity -- most specifically autonomic system processes -- yields a nonspecific sense of arousal which, when combined with cognitive appraisal, provides a description emotion (Schacter & Singer, 1962). More sophisticated models reflecting better psychophysiological theory and methods have come forth in the intervening years. Blascovich and colleagues (Blascovich, Mendes, Hunter, & Lickel, 2000; Blascovich, Mendes, Tomaka, Salomon, & Seery, 2003; Vick, Seery, Blascovich, & Weisbuch, 2008) have framed emotion as threat or challenge motivations. Threat motivations encompass traditional notions about fight or flight; outside dangerous situation, people seemingly over-respond to social intercourse rather than avail it to meet current goals. Challenge motivation has been alternately portrayed as psychological toughness (Dienstbier, 1989); namely, the positive response to stimuli that yield activation of effective responses to the social situation. Threat versus challenge reactions depend on the demands and resources of a situation. People perceive threat and react accordingly when social interactions are understood to be more demanding than available resources; challenge results when resources are deemed satisfactory for demands.
The two forms of motivation vary in neurophysiological markers. Threat is associated with sympathetic adrenal-medullary activation which enhances cardiac performance particularly left ventricular contractility and cardiac output (Mendes, Reis, Seery, & Blascovich, 2003; Weisbuch, Seery, Ambady, & Blascovich, 2009). In contrast, challenge is associated with these processes AND activation of the pituitary-adrenal-cortical axis which inhibits vasodilation and often produces vasoconstriction. Critiques of the Blascovich biopsychosocial model have questioned assessment of cardiovascular reactivity in this way (Wright & Kirby, 2003) though the body of research representing these phenomena are consistent and robust (Blascovich, Mendes, Tomaka, Salomon, & Seery, 2003). Blascovich et al. (2001) defined change in cardiac reactivity as threat expectations, specifically when research subjects were exposed to people with varying stigmas: people with facial birthmarks, African Americans, and low SES. This relationship is expected to similarly describe reactions to mental illness stigma.
Blascovich and colleagues have spent more than a decade developing a complex technology to assess these psychophysiological processes. Their techniques would be untenable in most situations where stigma change would be examined. Feasibility of these measurements is low in real world settings. Electromyographs (EMG), however, may offer an alternative approach with what seem to be more useable instruments. Vanman and associates (1997, 2004) used EMGs of facial muscles to document emotional changes in European American research participants in response to photographic slides of African Americans. One study specifically uses this technology to measure the stigma of mental illness (Graves, Cassisi, & Penn, 2005). It used facial EMGs plus a relatively crude measure of heart reactivity (a photoplethysmographic sensor on the distal phalanx of the left index finger). Research participants were shown pictures of people accompanied with audio-recorded biographies; half of the biographies provided information consistent with a person with serious mental illness. Data analyses showed EMGs and heart rate were associated with the negative affect that corresponds with mental illness biographies.
EMGs and heart rate have been used to track psychophysiological reactions to relevant media sources (Lang, Chung, Lee, Shin, & Schwartz, 2005; Lang, Chung, Lee, & Zhao, 2005; Lang, Shin, Bradley, Wang, Lee, & Potter, 2005; Schneider, Lang, Shin, & Bradley, 2004). Much of this research occurs in carefully controlled laboratories which challenge the ecological validity of typical anti-stigma programs. There are mobile instruments though their feasibility and accuracy is yet to be examined (Costanza, Inverso, Allen, & Maes, 2008). Still, the better portability of EMG, compared to the Blascovich measurements, challenges feasibility in the real world milieu in which stigma change approaches is examined.
Information processing measures
Information processing divides the macro or gestalt of cognition into component events, a superior paradigm to intelligence models, for example, because individual information processes have been used to explain a variety of psychiatric conditions (Buckley, Blanchard, & Neill, 2000; Gotlib & Krasnoperova, 1998; Lee & Shafran, 2004). It has also been a literature that blends nicely with psychophysiological models (Lang, Schwartz, Lee, & Angelini, 2007). More to the point of this paper, information processing theory has been used to understand the stigma of psychiatric disability and the effects of stigma change. One such method assesses recollection of social information. The Life Story Memory Test (LSMT), a measure designed by Macrae and colleagues (Johnston & Macrae, 1994; Macrae, Bodenhausen, Milne et al., 1994), was developed to assess the impact of stigma change strategies on the perception and recollection of information about persons with serious mental illnesses (Corrigan, River, Lundin et al., 2001). Research participants view a 3-minute videotape of an actor labeled “mentally ill” who is telling his life story. Their narrative contained 20 items, ten that were reliably rated by a pilot group (n=29) as negative and stereotypic statements about mental illnesses (e.g., “Sometimes I believe I’m George Washington.”) and ten that were rated as positive statements (e.g., “I work as an engineer.”). These items were randomly ordered and then written into a coherent narrative which the actor recites on the videotape. One minute after viewing each videotape (during which time participants were instructed to complete a nonverbal interference task: to draw a blueprint of their childhood home or elementary school), participants were asked to write down as many of the statements as they could remember. Raters then count the number of positive and negative statements in each participant’s list; interrater reliability is always high (>0.9). Outcome variables include total number of negative recollections (representing stigmatizing biases) and number of positive replies (representing more of an affirmative perspective).
This measure has demonstrated the superiority of contact-based stigma programs compared to educational programs; research participants in the contact-based program recalled more positive and less negative statements (Corrigan, River, Lundin, et al., 2001). It also has shown the benefits of NAMI’s contact-based program called In Our Own Voice (Corrigan, Rafacz et al., in press). The LSMT is especially appealing because its data represent a dynamic product of stigma change; namely, that program participants are better able to assimilate information about people with mental illnesses that is contrary to stigma.
Information processing concepts are distinguished as controlled or automatic. Controlled processes typically require cognitive capacity and are effortful; e.g., recollection as part of LSMT. They are generally conscious, with the relationship between cognition and behavior seemingly manifest. Automatic processing occurs outside of consciousness and uses little or no cognitive capacity. Its seems to be reflexive in practice. For instance, people gently move away from individuals labeled mentally ill without bringing to mind the related stereotype. Social psychologists have distinguished these phenomena into explicit processes; -- obvious and directly measurable -- versus implicit ones. Most of the measures reviewed in this paper are explicit and controlled.
We did allude to unconscious automatic processes as one of the possible outcomes of population-based strategies, somehow dovetailing with penetration. Although automatic penetration has not been incorporated into research, automatic cognitive processes have gained some traction in studying the stigma of psychiatric disability and seems to have potential worth in documenting stigma change. Implicit measures are not so easily assessed because they represent “hidden” events. Social psychologists have developed computer-based reaction time tasks that represent implicit attitudes (Greenwald & Banaji, 1995; Greenwald, Brown et al., 2009; Greenwald, McGhee, & Schwartz, 1998). Investigators have more recently extended the technology to the stigma of mental illnesses with one study examining the relationship between implicit attitudes representing self stigma and recovery-related goals (Rusch, Corrigan, Todd, & Bodenhausen, in press) and others examining the effects of public stigma (Peris, Teachman, & Nosek, 2008; Teachman, Wilson, & Komarovska, 2006). Peris et al. (2008), for example, examined stigma related to mental health practice and found explicit processes were associated with more negative therapeutic processes and implicit stigma predicted over-diagnosing.
Notions about automatic processing are useful because they suggest that people in some situations are unaware of aspects of prejudice that might influence their behavior. In this light, indices of automatic processing may have value as outcome variables. Equally of interest is whether the current collection of change programs influences automatic beliefs related to stigma. Automatic processes may be resilient to education and contact programs. There is an interesting review paper on looking at the effects of interventions on implicit processing by Palfai and Wagner (2004). Although it proffers some interesting principles, specific techniques are lacking. If implicit processes actually have socially valid effects on discrimination, then research needs to find change strategies that influence these kinds of processes.
Consciousness raising
Another cognitive construct that may be understood as information processing is consciousness; whether a person is aware of and can express messages about the egregious effects of mental illness stigma (Laidlaw, 1991). It seems obvious to say that prejudice and discrimination cannot be erased without participants in an anti-stigma program focusing on relevant content. Anti-stigma programs need to increase attentiveness to stigma (Devlin, MacAskill, & Stead, 2007; Quinn & Nifton, 2005). This is easily measured through Likert scale items: “I realize that I stigmatize people with mental illnesses” (7 = strongly agree). Advocates like to promote consciousness not only because it is the first step in challenging stigma, but also because consciousness raising mobilizes the choir. For example, family members and consumers in such groups as the National Alliance on Mental Illness and Mental Health America quickly join anti-stigma efforts once they have been alerted to or reminded of the harm of stigma. Consciousness raising is also valuable to mobilize advocacy-minded people who are not currently engaged in the psychiatric arena. ACLU members, are a good example, people for whom social justice is a primary value and who might direct their energies to the prejudice and discrimination of psychiatric disabilities. Research has identified factors that correspond with the kind of activism that may emerge from consciousness raising; people low on authoritarianism and the Protestant work ethic (Watson, Corrigan & Angell, 2005).
RECOMMENDATIONS FOR FUTURE RESEARCH
Table 3 lists recommendations that evolved from the paper. Research needs to represent an agenda of priorities, with this agenda reflecting the interests and exigencies of the community of relevant stakeholders. This begins with development of a CBPR team, the group with ultimate control of program evaluation. The CBPR team includes local stakeholders with the most prominent group there being people with psychiatric disabilities. Diversity is important when developing CBPR teams. Relevant categories include ethnicity, gender, SES, age, and education.
Table 3.
Recommendations for Measurement of Stigma Change
| 1. Select measures that represent stakeholder priorities about the goal of stigma change. Incorporate a Community-Based Participatory Research Team in all research activities. Consider the social validity of measures. |
| 2. Evaluate stigma change and diversity. Important demographics may include ethnicity, SES, gender, education, and sexual orientation. Diversity may be important mediators through which individuals understand mental illness; e.g., Do African Americans endorse mental illness and violence more than European Americans? Diversity may also be descriptors of the person with mental illness. E.g., Are Latinos with mental illness viewed as more violent than Native Americans with these disorders? |
| 3. Consider measures of behavior change, frequently prioritized as most important by stakeholder groups. Contrast decreasing discriminatory behavior from increasing affirming actions. On e form of affirming behavior is the degree to which research subjects participate in evidence based rehabilitation programs (e.g., the frequency in which an employer participates in a supported employment program). |
| 4. Select measures that reflect the specific interest of targeted and local groups. |
| 5. Choose measures that are less influenced by social desirability. |
| 6. Consider other domains of measurement – attitudes & emotions, knowledge, information and physiological processes – which, given the status of current research, are important only as they further understanding an anti-stigma approach n behavior. |
| 7. Outline how a physiological or information process may help to better explain stigma change. |
| 8. Develop theory-based models of stigma. These may be especially important for measures of attitudes and emotions. |
| 9. Measure penetration for population-based anti-stigma programs. Include both recognition and recall of previous PSAs. |
| 10. Determine whether awareness has been improved after an anti-stigma program. |
CBPR activities are successful when the research design includes measures with social validity. Social validity represents the “so what” effect. So what if instrument X changed! What implications does it actually have for controlling stigma and bettering the lives of people with psychiatric disorders? Behaviors are often cited by stakeholder groups as the most important for stigma change; the most resounding response to “so what!” But not all behavior is on target. Anti-stigma programs are not effective if an employer indiscriminately increases his or her interviews of just anyone. Is an anti-stigma program on mental illness stigma successful if it improves employer behavior of Black men? Probably not. So what about programs for people in wheelchairs, with substance abuse disorders, or with transient depression? The goal is to determine the impact of interviewing people with disabilities AND whether that action leads to an increase in people with mental illnesses obtaining more and better jobs.
Closely wedded to social validity is targeted and local stigma change. Namely, the social validity of measures varies with the group targeted for change. Goals of stigma change vary with the group meant to improve after the stigma program. Employers hire, landlords rent, and legislators allocate. Anti-stigma programs should also be local. This is consistent with a CBPR agenda. Namely, people with local power and responsibility are in the best position to influence the stigma of mental illness in their community. Hence, they need to be intimately involved with setting up and then evaluating the anti-stigma approach. Stigma change research is affected by social desirability. Research participants decrease their endorsement of stereotypes after deducing that an instrument is assessing prejudice and most people do not wish to appear bigoted. Ways to circumvent social desirability including use of factorial interactions and measures which are not patent indicators of bias. Two sets of constructs fall into the latter concern: physiological and information processes. Complex and elegant techniques have evolved to assess specific constructs in these arenas. However, the current body of research is unclear as to how change in this area has social validity. At this point, physiological and information processes serve mostly as mediating concepts which may inform stigma and stigma change models. This realization is consistent with Recommendation 8 in the Table calling for theory-based development of measurement.
Not all stigma change focuses on local and targeted impact. Some anti-stigma programs aim for population effects. Penetration is central to assessing these programs. Recollection and recognition are both functions useful for tracking population-based effects. Penetration suggests an additional important goal of targeted and local change programs. Does it raise consciousness? Does the program rally the choir to join in the anti-stigma efforts?
FINAL CONSIDERATIONS
Public stigma is only one form of prejudice. Self-stigma and label avoidance offer their own paradigms for understanding and evaluating stigma change and hence have their own implications for measurement. Also note; we addressed the issue of stigma and stigma change in terms of social psychological models which are likely to be more available to readers of this journal than those that have developed out of sociology and marketing. Nevertheless, research needs to incorporate other paradigms for models about stigma to mature. Ultimately, advocates want strategies that derail the various kinds of stigmas. The investigator must partner with these stakeholders to provide information about the best way to erase the stigma.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Contributor Information
Patrick W. Corrigan, Illinois Institute of Technology
Jenessa R. Shapiro, UCLA
References
- Ajzen I, Manstead AS. Changing health-related behaviours: An approach based on the theory of planned behaviour. In: Hewstone M, Schut HAW, De Wit JBF, Van Den Bos K, Stroebe MS, editors. The scope of social psychology: Theory and applications. New York, NY, US: Psychology Press; 2007. pp. 43–63. [Google Scholar]
- Ajzen I, Fishbein M. Understanding attitudes and predicting social behavior. Englewood Cliffs, NJ: Prentice-Hall; 1980. [Google Scholar]
- Allport GW. The nature of prejudice. Cambridge, MA: Addison-Wesley; 1954. [Google Scholar]
- American Psychiatric Association. The diagnostic and statistical manual of mental disorders. 4th edition. Washington DC: APA press; 1994. [Google Scholar]
- American Psychiatric Association. The diagnostic and statistical manual of mental disorders. 4th edition, text revision. Washington DC: APA press; 2004. [Google Scholar]
- Anastasi A. Psychological testing. 2nd ed. Oxford, England: Macmillan; 1961. [Google Scholar]
- Angermeyer MC, Matschinger H. The effect of violent attacks by schizophrenic persons on the attitude of the public towards the mentally ill. Social Science & Medicine. 1996;43(12):1721–1728. doi: 10.1016/s0277-9536(96)00065-2. [DOI] [PubMed] [Google Scholar]
- Antonak RF, Livneh H. Development, psychometric analysis, and validation of an error-choice test to measure attitudes toward persons with epilepsy. Rehabilitation Psychology. 1995a;40:25–38. [Google Scholar]
- Antonak RF, Livneh H. Direct and indirect methods to measure attitudes toward persons with disabilities, with an exegesis of the error-choice method. Rehabilitation Psychology. 1995b;40:3–24. [Google Scholar]
- Arboleda-Flórez J, Sartorius N. Understanding the stigma of mental illness: Theory and Interventions. West Sussex, England: Wiley; 2008. [Google Scholar]
- Arkowitz H, Westra HA, Miller WR, Rollnick S, editors. Motivational interviewing in the treatment of psychological problems. New York: Guilford Press; 2008. [Google Scholar]
- Aronson E, Blaney N, Stephin C, Sikes J, Snapp M. The jigsaw classroom. Beverly Hills, CA: Sage Publishing Company; 1978. [Google Scholar]
- Barnes J. Methods of measuring community characteristics. Child Psychology & Psychiatry Review. 1997;2(4):163–169. [Google Scholar]
- Bellack AS, Brown CH, Thomas-Lohrman S. Psychometric characteristics of role-play assessments of social skill in schizophrenia. Behavior Therapy. 2006;37:339–352. doi: 10.1016/j.beth.2006.01.005. [DOI] [PubMed] [Google Scholar]
- Blascovich J, Mendes WB, Hunter SB, Lickel B, Kowai-Bell N. Perceiver threat in social interactions with stigmatized others. Journal of Personality and Social Psychology. 2001;80(2):253–267. doi: 10.1037/0022-3514.80.2.253. [DOI] [PubMed] [Google Scholar]
- Blascovich J, Mendes WB, Tomaka J, Salomon K, Seery M. The robust nature of the biopsychosocial model challenge and threat: A reply to Wright and Kirby. Personality and Social Psychology Review. 2003;7(3):234–243. doi: 10.1207/S15327957PSPR0703_03. [DOI] [PubMed] [Google Scholar]
- Blascovich J, Mendes WB, Hunter SB, Lickel B. Stigma, threat, and social interactions. In: Heatherton TF, Kleck RE, Hebl MR, Hull JG, editors. The social psychology of stigma. New York: Guilford Press; 2000. pp. 307–333. [Google Scholar]
- Bogart LM, Uyeda K. Community-based participatory research: Partnering with communities for effective and sustainable behavioral health interventions. Health Psychology. 2009;28(4):391–393. doi: 10.1037/a0016387. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Brodie RJ, Winklhofer H, Coviello NE, Johnston WJ. Is e-marketing coming of age? An examination of the penetration of e-marketing and firm performance. Journal of Interactive Marketing. 2007;21(1):3–31. [Google Scholar]
- Brondolo E, Gallo LC, Myers HF. Race, racism and health: Disparities, mechanisms, and interventions. Journal of Behavioral Medicine. 2009;32(1):1–8. doi: 10.1007/s10865-008-9190-3. [DOI] [PubMed] [Google Scholar]
- Brown K, Bradley LJ. Reducing the stigma of mental illness. Journal of Mental Health Counseling. 2002;24(1):81–87. [Google Scholar]
- Buckley TC, Blanchard EB, Neill WT. Information processing and PTSD: A review of the empirical literature. Clinical Psychology Review. 2000;20(8):1041–1065. doi: 10.1016/s0272-7358(99)00030-6. [DOI] [PubMed] [Google Scholar]
- Chené R, García L, Goldstrom M, Pino M, Roach DP, Thunderchief W, Waitzkin H. Mental health research in primary care: mandates from a community advisory board. Annals of Family Medicine. 2005;3(1):70–72. doi: 10.1370/afm.260. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Chiesa M, Hobbs S. Making sense of social research: How useful is the Hawthorne Effect? European Journal of Social Psychology. 2008;38(1):67–74. [Google Scholar]
- Cohen J, Struening EL. Opinions about mental illness in the personnel of two large mental hospitals. Journal of Abnormal Psychology. 1962;64:349–360. doi: 10.1037/h0045526. [DOI] [PubMed] [Google Scholar]
- Courtwright AM. Justice, stigma, and the new epidemiology of health disparities. Bioethics. 2009;23(2):90–96. doi: 10.1111/j.1467-8519.2008.00717.x. [DOI] [PubMed] [Google Scholar]
- Christopher S, Watts V, McCormick A, Young S. Building and maintain trust in a community-based participatory research partnership. American Journal of Public Health. 2008;98(8):1398–1406. doi: 10.2105/AJPH.2007.125757. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Clarke NE, Crewe NM. Stakeholder attitudes toward ADA Title I: Development of an indirect measurement method. Rehabilitation Counseling Bulletin. 2000;43:58–65. [Google Scholar]
- Cook SW. Experimenting on social issues: The case of school desegregation. AmericanPsychologist. 1985;40:452–460. doi: 10.1037//0003-066x.40.4.452. [DOI] [PubMed] [Google Scholar]
- Corrigan PW. Target-specific stigma change: A strategy for impacting mental illness stigma. Psychiatric Rehabilitation Journal. 2004a;28(2):113–121. doi: 10.2975/28.2004.113.121. [DOI] [PubMed] [Google Scholar]
- Corrigan PW. How stigma interferes with mental health care. American Psychologist. 2004b;59:614–625. doi: 10.1037/0003-066X.59.7.614. [DOI] [PubMed] [Google Scholar]
- Corrigan PW. Changing stigma through contact. Advances in Schizophrenia and Clinical Psychiatry. 2005;1:54–58. [Google Scholar]
- Corrigan PW, Gelb B. Three programs that use mass approaches to challenge the stigma of mental illness. Psychiatric Services. 2006;57:393–398. doi: 10.1176/appi.ps.57.3.393. [DOI] [PubMed] [Google Scholar]
- Corrigan PW, Kleinlein P. The impact of mental illness stigma. In: Corrigan PW, editor. On the stigma of mental illness: Practical strategies for research and social change. Washington, DC: the American Psychological Association; 2005. pp. 11–44. [Google Scholar]
- Corrigan PW, Lundin RK. Don’t call me nuts! Coping with the stigma of mental illness. Tinley Park, IL: Recovery Press; 2001. [Google Scholar]
- Corrigan PW, Markowitz F, Watson A, Rowan D, Kubiak MA. An attribution model of public discrimination towards persons with mental illness. Journal of Health and Social Behavior. 2003;44:162–179. [PubMed] [Google Scholar]
- Corrigan PW, Penn DL. Lessons from social psychology on discrediting psychiatric stigma. American Psychologist. 1999;54:765–776. doi: 10.1037//0003-066x.54.9.765. [DOI] [PubMed] [Google Scholar]
- Corrigan PW, River LP, Lundin RK, Penn DL, Wasowski KU, Campion J, Mathisen J, Gagnon C, Bergman M, Goldstein H, Kubiak MA. Three strategies for changing attributions about severe mental illness. Schizophrenia Bulletin. 2001;27:187–196. doi: 10.1093/oxfordjournals.schbul.a006865. [DOI] [PubMed] [Google Scholar]
- Corrigan PW, Rowan D, Green A, Lundin R, River P, Uphoff-Wasowski K, et al. Challenging two mental illness stigmas: Personal responsibility and dangerousness. Schizophrenia Bulletin. 2002;28:293–310. doi: 10.1093/oxfordjournals.schbul.a006939. [DOI] [PubMed] [Google Scholar]
- Corrigan PW, Thompson V, Lambert D, Sangster Y, Noel J, Campbell J. Experiences with discrimination by people with mental illness. Psychiatric Services. 2003;54:1105–1110. doi: 10.1176/appi.ps.54.8.1105. [DOI] [PubMed] [Google Scholar]
- Corrigan PW, Watson AC. Understanding the impact of stigma on people with mental illness. World Journal of Psychiatry. 2002;1:16–19. [PMC free article] [PubMed] [Google Scholar]
- Corrigan PW, Watson AC. Findings from the National Comorbidity Survey on the frequency of dangerous behavior in individuals with psychiatric disorders. Psychiatry Research. 2005;136(2–3):153–162. doi: 10.1016/j.psychres.2005.06.005. [DOI] [PubMed] [Google Scholar]
- Corrigan PW, Watson AC, Barr L. The self-stigma of mental illness: implications for self-esteem and self-efficacy. Journal of Social & Clinical Psychology. 2006;25(8):875–884. [Google Scholar]
- Corrigan PW, Watson AC, Byrne P, Davis K. Mental illness stigma: Problem of public health or social justice? Social Work. 2005;50:363–368. doi: 10.1093/sw/50.4.363. [DOI] [PubMed] [Google Scholar]
- Costanza E, Inverso SA, Allen R, Maes P. EMG for subtle, intimate interfaces. In: Lumsden J, editor. Handbook of research on user interface design and evaluation for mobile technology. Hershey, PA: Information Science Reference/IGI Global; 2008. pp. 524–542. [Google Scholar]
- Covell N, McCorkle BH, Weissman E, Summerfelt T, Essock SM. What’s in a name? Terms preferred by service recipients. Administration and Policy in Mental Health and Mental Health Services Research. 2007;34:443–447. doi: 10.1007/s10488-007-0123-1. [DOI] [PubMed] [Google Scholar]
- Crocker J, Major B, Steele C. Social Stigma . In: Gilbert D, Fiske ST, Lindsey G, editors. The handbook of social psychology. New York: McGraw-Hill; 1998. pp. 504–553. [Google Scholar]
- Devlin E, MacAskill S, Stead M. We're still the same people: Developing a mass media campaign to raise awareness and challenge the stigma of dementia. International Journal of Nonprofit & Voluntary Sector Marketing. 2007;12(1):47–58. [Google Scholar]
- Dienstbier RA. Arousal and physiological toughness: Implications for mental and physical health. Psychological Review. 1989;96(1):84–100. doi: 10.1037/0033-295x.96.1.84. [DOI] [PubMed] [Google Scholar]
- Dillman D. Mail and internet surveys: The tailored design method. 2nd edition. New York: John Wiley; 2000. [Google Scholar]
- Dooley PA. Perceptions of the onset controllability of AIDS and helping judgments: An attributional analysis. Journal of Applied Social Psychology. 1995;25(10):858–869. [Google Scholar]
- Dovidio J, Glick P, Rudman L. On the nature of prejudice: Fifty years after Allport. New York: Blackwell; 2005. [Google Scholar]
- EI-Gadi S, Abudher A, Sammud M. HIV-related knowledge and stigma among high school students in Libya. International Journal of STD & AIDS. 2008;19(3):178–183. doi: 10.1258/ijsa.2007.007170. [DOI] [PubMed] [Google Scholar]
- Farina A. Stigma. In: Mueser KT, Tarrier N, editors. Handbook of social functioning in schizophrenia. Needham Heights: Allyn & Bacon; 1998. pp. 247–279. [Google Scholar]
- Finkelstein J, Lapshin O, Wasserman E. Randomized study of different anti-stigma media. Patient Education & Counseling. 2008;71(2):204–214. doi: 10.1016/j.pec.2008.01.002. [DOI] [PubMed] [Google Scholar]
- Finn CA, Skadeczek IE. Assessing the social validity of behavioral interventions: A review of treatment acceptability measures. School Psychology, Quarterly. 2001;16(2):176–206. [Google Scholar]
- Farina A, Thaw J, Lovern JD, Mangone D. People’s reactions to a former mental patient moving to their neighborhood. Journal of Community Psychology. 1974;2(2):108–112. [Google Scholar]
- Fishbein M, Azjen I. Acceptance, yielding, and impact: Cognitive processes in persuasion. In: Petty RE, Ostrom TM, Brock TC, editors. Cognitive responses in persuasion. Hillsdale, NJ: Lawrence Erlbaum; 1981. pp. 339–359. [Google Scholar]
- Forster J, Higgins ET, Strack F. When stereotype disconfirmation is a personal threat: How prejudice and prevention focus moderate incongruency effects. Social Cognition. 2000;18:178–197. [Google Scholar]
- Fyock J, Stangor C. The role of memory biases in stereotype maintenance. British Journal of Social Psychology. 1994;33:331–344. doi: 10.1111/j.2044-8309.1994.tb01029.x. [DOI] [PubMed] [Google Scholar]
- Goffman E. Stigma: Notes on the management of spoiled identity. Englewood Cliffs, NJ: Prentice-Hall; 1963. [Google Scholar]
- Gotlib IH, Krasnoperova E. Biased information processing as a vulnerability factor for depression. Behavior Therapy. 1998;29(4):603–617. [Google Scholar]
- Graham S, Weiner, Zucker Attributional analysis of punishment goals and public reactions to O.J. Simpson. Personality & Social Psychology Bulletin. 1997;23(no. 4):331–346. [Google Scholar]
- Green MF. Schizophrenia from a neurocognitive perspective: Probing the impenetrable darkness. Needham Heights, MA: Allyn & Bacon; 1997. [Google Scholar]
- Greenwald AG, Banaji MR. Implicit social cognition: Attitudes, self-esteem, and stereotypes. Psychological Review. 1995;102(1):4–27. doi: 10.1037/0033-295x.102.1.4. [DOI] [PubMed] [Google Scholar]
- Greenwald AG, McGhee DE, Schwartz JLK. Measuring individual differences in implicit cognition: The Implicit Association Test. In: Fazio RH, Petty RE, editors. Attitudes: Their structure, function, and consequences. New York: Psychology Press; 2008. pp. 109–131. [Google Scholar]
- Graves RE, Cassisi JE, Penn DL. Psychophysiological evaluation of stigma towards schizophrenia. Schizophrenia Research. 2005;76(2–3):317–327. doi: 10.1016/j.schres.2005.02.003. [DOI] [PubMed] [Google Scholar]
- Greitemeyer T, Weiner B. Asymmetrical attributions for approach versus avoidance behavior. Personality and Social Psychology Bulletin. 2003;29:1371–1382. doi: 10.1177/0146167203255766. [DOI] [PubMed] [Google Scholar]
- Hamilton D. Creating mental illness. Nursing History Review. 2004;12:243–245. [Google Scholar]
- Hinshaw SP. The mark of shame: Stigma of mental illness and an agenda for change. Oxford, NY: Oxford University Press; 2007. [Google Scholar]
- Herek GM, Widaman KF, Capitanio JP. When sex equals AIDS: Symbolic stigma and heterosexual adults’ inaccurate beliefs about sexual transmission of AIDS. Social Problems. 2005;52(1):15–37. [Google Scholar]
- Hogan MF. The President's New Freedom Commission: Recommendations to transform mental health care in America. Psychiatric Services. 2003;5(11):1467–1474. doi: 10.1176/appi.ps.54.11.1467. [DOI] [PubMed] [Google Scholar]
- Ickes W, Holloway R, Stinson L, Hoodenpyle T. Self-monitoring in social interaction: The centrality of self-affect. Journal of Personality. 2006;74:1–26. doi: 10.1111/j.1467-6494.2006.00388.x. [DOI] [PubMed] [Google Scholar]
- Johnston LC, Macrae CN. Changing social stereotypes: The case of the information seeker. European Journal of Social Psychology. 1994;24(5):581–592. [Google Scholar]
- Jones EE, Farina A, Hastorf AH, Markus H, Miller DT, Scott RA. Social Stigma: The Psychology of Marked Relationships. New York: Freeman; 1984. [Google Scholar]
- Kazdin A. Assessing the clinical or applied significance of behavior change through social validation. Behavior Modification. 1977;1:427–452. [Google Scholar]
- Kennedy CH. The maintenance of behavior change as an indicator of social validity. Behavior Modification. 2002;26(5):594–604. doi: 10.1177/014544502236652. [DOI] [PubMed] [Google Scholar]
- Kessler RC. The national comorbidity survey: Preliminary results and future directions. International Journal of Methods in Psychiatric Research. 1995;5(2):139–151. [Google Scholar]
- King EB, Shapiro J, Hebl MR, Singletary S, Turner S. The stigma of obesity in customer service: A mechanism of remediation and bottom-line consequences of interpersonal discrimination. Journal of Applied Psychology. 2006;91:579–593. doi: 10.1037/0021-9010.91.3.579. [DOI] [PubMed] [Google Scholar]
- Kotler P, Roberto N, Lee N, editors. Social Marketing: Improving the Quality of Life. Thousand Oaks, CA: Sage; 2002. [Google Scholar]
- Laidlaw TA. Consciousness raising at its best. Canadian Psychology/ Psychologie canadienne. 1991;32(2):151–153. [Google Scholar]
- Lane JL, Slavin S, Ziv A. Simulation in medical education: A review. Simulation & Gaming. 2001;32:297–314. [Google Scholar]
- Lang A, Chung Y, Lee S, Shin M, Schwartz N. It's an arousing, fast-paced kind of world: The effects of age and sensation seeking on the information processing of substance-abuse PSAs. Media Psychology. 2005;7(4):421–454. [Google Scholar]
- Lang A, Chung Y, Lee S, Zhao X. It's the Product: Do Risky Products Compel Attention and Elicit Arousal in Media Users? Health Communication. 2005;17(3):283–300. doi: 10.1207/s15327027hc1703_5. [DOI] [PubMed] [Google Scholar]
- Lang A, Shin M, Bradley SD, Wang Z, Lee S, Potter D. Wait! Don't Turn That Dial! More Excitement to Come! The Effects of Story Length and Production Pacing in Local Television News on Channel Changing Behavior and Information Processing in a Free Choice Environment. Journal of Broadcasting & Electronic Media. 2005;49(1):3–22. [Google Scholar]
- Lang A, Schwartz N, Lee S, Angelini J. Processing radio PSAs: Production pacing, arousing content, and age. Journal of Health Communication. 2007;12(6):581–599. doi: 10.1080/10810730701508708. [DOI] [PubMed] [Google Scholar]
- Lee S, Kleinman A. Are somatoform disorders changing with time? The case of neurasthenia in China? Psychosomatic Medicine. 2007;69(9):846–849. doi: 10.1097/PSY.0b013e31815b0092. [DOI] [PubMed] [Google Scholar]
- Levin S, Van Laar C, Sidanius J. The effects of ingroup and outgroup friendships on ethnic attitudes in college: A longitudinal study. Group Processes and Intergroup Relations. 2003;6:76–92. [Google Scholar]
- Liberman RP. Recovery from disability: Manual of psychiatric rehabilitation. Washington DC: American Psychiatric Publishing Inc.; 2008. [Google Scholar]
- Link BG, Phelan JC. Conceptualizing Stigma. Annual Review of Sociology. 2001;27:363–385. [Google Scholar]
- Link BG, Struening E, Cullen FT, Shrout PE, Dohrenwend BP. A modified labeling theory approach to mental disorders: An empirical assessment. Am Social Rev. 1989;54:400–423. [Google Scholar]
- Link BG, Stueve A, Phelan J. Psychotic symptoms and violent behaviors: probing the components of ‘threat/control-override’ symptoms. Social Psychiatry and Psychiatric Epidemiology. 1998;33:S55–S60. doi: 10.1007/s001270050210. [DOI] [PubMed] [Google Scholar]
- Link BG, Yang L, Phelan J, Collins P. Measuring Mental Illness Stigma. Schizophrenia Bulletin. 2004;30:511–541. doi: 10.1093/oxfordjournals.schbul.a007098. [DOI] [PubMed] [Google Scholar]
- MacRae C, Bondenhausen GV, Milne AB, Jetten J. Out of mind but back in sight: Stereotypes of the rebound. Journal of Personality and Social Psychology. 1994;67:808–817. [Google Scholar]
- Major B, O’Brien L. The social psychology of stigma. In: Fiske ST, editor. Annual Review of Psychology. Vol. 56. Chippewa Falls, WI: Annual Reviews; 2005. [DOI] [PubMed] [Google Scholar]
- Mann CE, Himelein MJ. Putting the person back into psychopathology: An intervention to reduce mental illness stigma in the classroom. Social Psychiatry and Psychiatric Epidemiology. 2008;43(7):545–551. doi: 10.1007/s00127-008-0324-2. [DOI] [PubMed] [Google Scholar]
- Marshall C. personal communication. 2009 July [Google Scholar]
- Martiniuk ALC, Speechley KN, Secco M, Campbell MK. Development and psychometric properties of the thinking about Epilepsy questionnaire assessing children's knowledge and attitudes about epilepsy. Epilepsy & Behavior. 2007;10(4):595–603. doi: 10.1016/j.yebeh.2007.01.011. [DOI] [PubMed] [Google Scholar]
- Menec VH, Perry RP. Reactions to stigmas among Canadian students: Testing attribution-affect-help judgment model. Journal of Social Psychology. 1998;138:443–453. [Google Scholar]
- Mendes WB, Reis HT, Seery MD, Blascovich J. Cardiovascular correlates of emotional expression and suppression: Do content and gender context matter? Journal of Personality and Social Psychology. 2003;84(4):771–792. doi: 10.1037/0022-3514.84.4.771. [DOI] [PubMed] [Google Scholar]
- Minkler M, Wallerstein M. Introduction to community based participatory research. In: Minkler M, Wallerstein N, editors. Community Based Participatory Research for Health. San Francisco: Jossey-Bass; 2003. pp. 3–26. [Google Scholar]
- Morawski J. Social psychology a century ago. American Psychologist. 2000;55:427–431. doi: 10.1037/0003-066X.55.4.427. [DOI] [PubMed] [Google Scholar]
- National Mental Health Association. Alexandria: NMHA; American Opinions on Mental Health Issues. 1999
- Peris TS, Teachman BA, Nosek BA. Implicit and explicit stigma of mental illness: Links to clinical care. Journal of Nervous and Mental Disease. 2008;196(10):752–760. doi: 10.1097/NMD.0b013e3181879dfd. [DOI] [PubMed] [Google Scholar]
- Pescosolido BA, Jensen PS, Martin JK, Olafsdottir S, Fettes D, Perry BL. Public knowledge and assessment of child mental health problems: Findings from the National Stigma Study--Children. Journal of the American Academy of Child & Adolescent Psychiatry. 2008;47(3):339–349. doi: 10.1097/CHI.0b013e318160e3a0. [DOI] [PubMed] [Google Scholar]
- Pescosolido BA, Wright ER, Kikuzawa S. Stakeholder attitudes over time toward the closing of a state hospital. Journal of Behavioral Health Services & Research. 1999;26(3):318–328. doi: 10.1007/BF02287276. [DOI] [PubMed] [Google Scholar]
- Peters E, Burraston B, Mertz CK. An emotion-based model of stigma susceptibility: Appraisals, affective reactivity, and worldviews in the generation of a stigma response. Risk Analysis. 2004;24:1349–1367. doi: 10.1111/j.0272-4332.2004.00531.x. [DOI] [PubMed] [Google Scholar]
- Pettigrew TF, Troop LR. Does intergroup contact reduce prejudice? Recent metaanlytic findings. In: Oskamp S, editor. Reducing prejudice and discrimination. Hillsdale, NJ: Erlbaum; 2000. pp. 93–114. [Google Scholar]
- Phelan JC. Genetization of deviant behavior and consequences for stigma: The case of mental illness. Journal of Health and Social Behavior. 2005;46:307–322. doi: 10.1177/002214650504600401. [DOI] [PubMed] [Google Scholar]
- Phelan JC, Link BG, Stueve A, Pescosolido BA. Public conceptions of mental illness in 1950 and 1996: What is mental illness and is it to be feared? Journal of Health and Social Behavior. 2000;41(2):188–207. [Google Scholar]
- Phelan J, Yang L, Cruz-Rojas R. Effects of attributing serious mental illnesses to genetic causes on orientations to treatment. Psychiatric Services. 2006;57:382–387. doi: 10.1176/appi.ps.57.3.382. [DOI] [PubMed] [Google Scholar]
- Prochaska JO, Wright JA, Velicer WF. Evaluating theories of health behavior change: A hierarchy of criteria applied to the transtheoretical model. Applied Psychology: An International Review. 2008;57(4):561–588. [Google Scholar]
- Quinn N, Knifton L. Promoting recovery and addressing stigma: Mental health awareness through community development in a low-income area. International Journal of Mental Health Promotion. 2005;7(4):37–44. [Google Scholar]
- Rabiner EL, Wells CF, Struening E, Schmeidler J. Toward a holistic model for predicting community reentry performance. American Journal of Orthopsychiatry. 1983;53(2):303–314. [Google Scholar]
- Reinke RR, Corrigan PW, Leonhard CL, Lundin RK, Kubiak MA. Examining two aspects of contact on the stigma of mental illness. Journal of Social and Clinical Psychology. 2004;23(3):377–389. [Google Scholar]
- Reisenzein R. A structural equation analysis of Weiner’s attribution-affect model of helping behavior. Journal of Personality and Social Psychology. 1986;50:1123–1133. [Google Scholar]
- Reyna C, Weiner B. Justice and utility in the classroom: An attributional analysis of the goals of teachers’ punishment and intervention strategies. Journal of Educational Psychology. 2001;93:309–319. [Google Scholar]
- Robins LN, Regier DA. Psychiatric disorders in America. New York: The Free Press; 1991. [Google Scholar]
- Rogers E, Palmer-Erbs V. Participatory action research: Implications for research and evaluation in psychiatric rehabilitation. Psychosocial Rehabilitation Journal. 1994;18:3–12. [Google Scholar]
- Schacter S, Singer JE. Cognitive, social, and psychological determinants of emotional state. Psychological Review. 1962;69:379–399. doi: 10.1037/h0046234. [DOI] [PubMed] [Google Scholar]
- Schneider EF, Lang A, Shin M, Bradley SD. Death with a story: How story impacts emotional, motivational, and physiological responses to first-person shooter video games. Human Communication Research. 2004;30(3):361–375. [Google Scholar]
- Schulze B. Stigma and mental health professionals: A review of the evidence on an intricate relationship. International Review of Psychiatry. 2007;19(2):137–155. doi: 10.1080/09540260701278929. [DOI] [PubMed] [Google Scholar]
- Thornicroft G. Shunned: Discriminating against people with mental illness. Oxford, NY: Oxford University Press; 2006. [Google Scholar]
- Stone DL, Hosoda M, Lukaszewski KM, Phillips TN. Methodological problems associated with research on unfair discrimination against racial minorities. Human Resource Management Review. 2008;18(4):243–258. [Google Scholar]
- Struening EL, Cohen J. Factorial invariance and other psychometric characteristics of five opinions about mental illness factors. Educational and Psychological Measurement. 1963;42:289–298. [Google Scholar]
- Substance Abuse and Mental Health Services Administration, SAMHSA. Results from the 2001 National Household Survey on Drug Abuse: Volume I. Summary of National Findings (Office of Applied Studies, NHSDA Series H-17, DHHS Publication No. SMA02-3758) Rockville, MD: 2002. [Google Scholar]
- Swanson JW, Holzer CE, Ganju VK, Jono RT. Violence and psychiatric disorder in the community: Evidence from the Epidemiologic Catchment Area surveys. Hospital and Community Psychiatry. 1990;41:761–770. doi: 10.1176/ps.41.7.761. [DOI] [PubMed] [Google Scholar]
- Taylor SM, Dear MJ. Scaling community attitudes toward the mentally ill. Schizophrenia Bulletin. 1981;7:225–240. doi: 10.1093/schbul/7.2.225. [DOI] [PubMed] [Google Scholar]
- Taylor SM, Dear MJ, Hall GB. Attitudes toward the mentally ill and reactions to mental health facilities. Social Science and Medicine. 1979;13D:281–290. doi: 10.1016/0160-8002(79)90051-0. [DOI] [PubMed] [Google Scholar]
- Teachman BA, Wilson JG, Komarovskaya I. Implicit and explicit stigma of mental illness in diagnosed and healthy samples. Journal of Social & Clinical Psychology. 2006;25(1):75–95. [Google Scholar]
- Thornicroft G, Brohan E, Kassam A, Lewis-Holmes E. Reducing stigma and discrimination: Candidate interventions. International Journal of Mental Health Systems. 2008;2:3. doi: 10.1186/1752-4458-2-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Tourangeau, Roger, Yan T. Sensitive questions in surveys. Psychological Bulletin. 2007;133(5):859–883. doi: 10.1037/0033-2909.133.5.859. [DOI] [PubMed] [Google Scholar]
- Trickett EJ. Community psychology: Individuals and interventions in community context. Annual Review of Psychology. 2009;60:395–419. doi: 10.1146/annurev.psych.60.110707.163517. [DOI] [PubMed] [Google Scholar]
- Vanman EJ, Paul BY, Ito TA, Miller N. The modern face of prejudice and structural features that moderate the effect of cooperation on affect. Journal of Personality and Social Psychology. 1997;73(5):941–959. doi: 10.1037//0022-3514.73.5.941. [DOI] [PubMed] [Google Scholar]
- Vanman EJ, Saltz JL, Nathan LR, Warren JA. Racial discrimination by low-prejudiced Whites facial movements as implicit measures of attitudes related to behavior. Psychological Science. 2004;15(11):711–714. doi: 10.1111/j.0956-7976.2004.00746.x. [DOI] [PubMed] [Google Scholar]
- Varas-Díaz N, Marzán-Rodríguez M. The emotional aspect of AIDS stigma among health professionals in Puerto Rico. AIDS Care. 2007;19(10):1247–1257. doi: 10.1080/09540120701405403. [DOI] [PubMed] [Google Scholar]
- Vick SB, Seery MD, Blascovich J, Weisbuch M. The effect of gender stereotype activation on challenge and threat motivational states. Journal of Experimental Social Psychology. 2008;44:624–630. [Google Scholar]
- Wahl OF. Media madness: Public images of mental illness. New Brunswick, NJ: Rutgers University Press; 1995. [Google Scholar]
- Wahl OF. Mental health consumers’ experience of stigma. Schizophrenia Bulletin. 1999;25(3):467–478. doi: 10.1093/oxfordjournals.schbul.a033394. [DOI] [PubMed] [Google Scholar]
- Wegner DM, Erber R, Zanakos S. Ironic processes in the mental control of mood and mood related thought. Journal of Personality and Social Psychology. 1993;65:1093–1104. doi: 10.1037//0022-3514.65.6.1093. [DOI] [PubMed] [Google Scholar]
- Wegner DM, Schneider DJ. The war of the ghosts in the machine. In: Uleman J, Bargh J, editors. Unintended thought. New York: Guilford Press; 1989. pp. 287–305. [Google Scholar]
- Weigel RH, Wiser PL, Cook SW. The impact of cooperative learning experiences on cross-ethnic relations and attitude. Journal of Social Issues. 1975;31(1):219–244. [Google Scholar]
- Weiner B. Judgments of responsibility: A foundation for a theory of social conduct. New York: Guilford; 1995. [Google Scholar]
- Weisbuch M, Seery MD, Ambady N, Blascovich J. On the correspondence between physiological and nonverbal responses: Nonverbal behavior accompanying challenge and threat. Journal of Nonverbal Behavior. 2009;33(2):141–148. [Google Scholar]
- Wells K, Miranda J, Bruce ML, Alegria M, Wallerstein N. Bridging community intervention and mental health services research. American Journal of Psychiatry. 2004;161(6):955–963. doi: 10.1176/appi.ajp.161.6.955. [DOI] [PubMed] [Google Scholar]
- Wolf MM. Social validity: The case of subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis. 1978;11:203–214. doi: 10.1901/jaba.1978.11-203. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wolff G, Pathare S, Craig T, Leff J. Community attitudes to mental illness. British Journal of Psychiatry. 1996;168(2):183–190. doi: 10.1192/bjp.168.2.183. [DOI] [PubMed] [Google Scholar]
- Wright RA, Kirby LD. Cardiovascular correlates of challenge and threat appraisals: A critical examination of the biopsychosocial analysis. Personality and Social Psychology Review. 2003;7(3):216–233. doi: 10.1207/S15327957PSPR0703_02. [DOI] [PubMed] [Google Scholar]
- Yang LH, Kleinman A, Link BG, Phelan JC, Lee S, Good B. Culture and stigma: Adding moral experience to stigma theory. Social Science and Medicine. 2007;64(7):1524–1535. doi: 10.1016/j.socscimed.2006.11.013. [DOI] [PubMed] [Google Scholar]