Evaluation of anti-stigma interventions with sixth-grade students: A school-based field experiment

Abstract

OBJECTIVE

Early school-based interventions provide the opportunity to attack stigmatizing attitudes before they are firmly entrenched, in a ubiquitous institutional setting, and thus may provide optimal conditions for reducing mental illness stigma in the overall population. This study evaluates the effectiveness of classroom-based interventions in reducing stigma and increasing understanding of mental illness and positive attitudes toward treatment-seeking among sixth-grade students.

METHODS

The ethnically/racially diverse sample (n=721) was 40% Latino, 26% white, and 24% African American; mean age was 11.5. In a fully-crossed design, classrooms from a school district in Texas were randomly assigned to receive all three, two, one, or none of the following interventions: a power-point- and discussion-based curriculum; contact with two college students who described their experiences with mental illness; and exposure to anti-stigma printed materials. Standard and vignette-based quantitative measures of mental health knowledge and attitudes, social distance, and help-seeking attitudes were assessed pre- and post-intervention.

RESULTS

Printed materials had no significant effects on outcomes and were grouped with the control condition for analysis. For eight of 13 outcomes, the curriculum-only group report significantly more positive outcomes than the control group; the largest between-group differences were seen for stigma awareness and action, recognition of mental illness in the vignettes, and positive orientation to treatment. The contact-alone group report significantly more positive outcomes on three vignette-based measures.

CONCLUSIONS

Results were most promising for a classroom-based curriculum, which can be relatively easily disseminated to and delivered by teachers, offering the potential for broad application in the population.

Stigma critically influences well-being and recovery for people with mental illnesses, affecting employment, income, social ties, quality of life, mastery, self-esteem, depressive symptoms, and access to medical and mental health services. ^1–15 In recognition of this problem, there has been a sustained effort to reduce stigma by educating the public about neurobiological bases of mental illnesses and available treatments, with the assumption that framing mental disorders as medically treatable “illnesses like any other” would reduce stigma. ^16–19 There is clear evidence that the public has adopted this understanding and that mental health treatment is increasingly viewed as beneficial ^20–22 and sought by the public. ²³ Nevertheless, these changes have not been accompanied by stigma reduction. Core aspects of stigma—emotional reactions, stereotypes, and social distance—remain unchanged or have worsened. ^20,21,24 What can explain this discrepancy? Research now shows that biological explanations tend to increase rather than decrease stigma. ^25–27 Thus, it may be necessary to address stigma directly rather than by changing causal beliefs. Stigmatizing attitudes and behaviors may also be harder to change than causal beliefs because they involve emotions and can have personal consequences. Addressing these attitudes before they are firmly set may be a promising approach to reducing stigma,³⁵ with evidence suggesting that stigmatization occurs as early as childhood and adolesence. ^28–35 Therefore, the current study evaluates the effectiveness of a school-based intervention that directly focuses on stigma. Education- and contact-based interventions can reduce stigmatizing attitudes and behavioral intentions. Most studies focus on adults; fewer target adolescents. ^36–40 We located only six studies that targeted pre-high-school youth. ^41–46 These studies, with samples ranging from 185 to 1500 in over 16 US states, found significant reductions in stigma among youths between third and eighth grades. All employed a curriculum of some sort; one also included a contact intervention. Three studies included follow-ups of three to six months; three employed control conditions. We implemented interventions designed to improve knowledge, attitudes, behavioral intentions, and behaviors about mental illness and help seeking among sixth-graders. To develop a stigma intervention that can be broadly disseminated, we evaluated a classroom-based curriculum, which was designed to appeal to teachers and students and easy for teachers to implement without specialized training. We also evaluated the effectiveness of a contact intervention and of saturating classrooms with anti-stigma materials. Several aspects of social and psychological development led us to target sixth-graders rather than younger children. Preadolescents begin to understand that others have thoughts and feelings different from their own, include interpersonal and psychological features in their understanding of themselves and others, and experience heightened social comparison. ⁴⁷ Beyond augmenting a very small body of research, several strengths of the study allow it to meaningfully extend what can be concluded from the existing literature. Previous studies, like ours, employ a teacher-administered curriculum that does not entail extensive teacher training, suggesting that a relatively easily disseminated curriculum can reduce stigma. However, most of the previous studies rely on samples that involved teacher selfselection, allowing the possibility that effects will only be found when teachers favor an antistigma agenda. In our study, self-selection played a role at the school but not the classroom level, likely introducing less bias. Ours is the only study to include a fidelity measure, allowing us to evaluate how faithfully the intervention was enacted in the classroom and whether fidelity was related to outcomes of the intervention. Finally, although our sample came from a single geographic locale, it provided excellent representation of the major racial/ethnic groups in the U.S. All these features raise optimism that any reductions in stigma we find can generalize to a broad population application. Additional strengths introduced in this study include the evaluation of multiple interventions. Three of the previous studies included a control condition, but in all but one case, the control condition was usual instruction. This leaves open the possibility that intervention per se, via novelty, special attention, etc., rather than intervention content, produced attitude change. By testing three interventions, we can compare their effectiveness and attribute change to a particular intervention as opposed to intervention per se. Finally, by including vignettes describing specific disorders, we can assess changes in responses to these disorders as well as the more typical approach that focuses on the generic concept of “mental illness.” Methods Study Design The delivery of the three interventions was independently varied in a fully-crossed design, resulting in eight experimental cells. Sixteen middle schools (with separate zip codes) from an urban school district in Texas were ranked according to performance on a district-wide standardized test. Rankings based on percent of families below the poverty line were nearly identical. We randomly assigned the top eight schools to one of eight cells; the bottom-ranked eight schools were then randomly assigned to a cell in the reverse order, so that, for example, the top and bottom ranked schools were paired. Each cell (two schools in each) was randomly assigned to a study condition (see Table 1). Before the study began, two schools dropped out for non-study-related reasons. The study was repeated during a second academic year with a new set of sixth-grade students in five of the original schools, chosen to have similar demographics to the lost schools.

Table 1.

Intervention groups and their standardized test scores^a

Curriculum	Contact	Materials	% Passed Standardized Test	Mean % Passed
Yes	No	No	School 1 – 79	75
			School 2 – 70
Yes	Yes	No	School 1 – 87	75
			School 2 – 62
Yes	No	Yes	School 1 – 92	76
			School 2 – 59
Yes	Yes	Yes	School 1 – 73	73
			School 2 – 72
No	No	No	School 1 – 79	74
			School 2 – 68
No	Yes	No	School 1 – 86	75
			School 2 – 64
No	No	Yes	School 1 – 76	74
			School 2 – 71
No	Yes	Yes	School 1 – 82	74
			School 2 – 66

^aTwo schools withdrew from the study for non-study-related reasons. Consequently the study was repeated during a second academic year with a new set of 6^th-grade students in five of the original schools, chosen to have similar demographics to the lost schools. Thus a total of 19 classes from 14 schools were included in the study.

Participants

Participants were 721 sixth-grade students. See Table 2 for sociodemographic characteristics of participants and their families.

Table 2.

Personal Characteristics of Study Participants: Total Sample and by Intervention Group

Characteristics			Curriculum		Contact		Curriculum & Contact		Control Group
	N	%	n	%	n	%	n	%	n	%
Mean age in years	12		11		12		12		12
Standard Deviation	0.6		0.6		0.6		0.6		0.6
Total %			210	28	218	29	135	18	188	25
Sex, % female	406	54	109	52	126	58	72	54	101	54
Race/ethnicity, %**
Latino	301	40	55	26	109	50	47	35	92	49
White	195	26	78	37	33	15	58	43	24	13
Black	180	24	52	25	50	23	24	18	51	27
Other group	75	10	25	12	26	12	5	4	21	11
English as primary language at home, % Yes**	526	70	162	77	137	63	107	79	117	62
Contact with mental illness, % None	150	20	33	16	48	22	27	20	43	23
Parent/guardian annual income (<$50,000), %Yes**	518	69	120	57	172	79	81	60	145	77
Parent/guardian education (≥12 years), % Yes**	631	84	191	91	172	79	118	88	152	81

Note:

^**P<0.01 for differences between intervention groups.

Procedures

Pre- and post-test instruments were self-administered in physical education classes on laptop computers from February-May and September-December 2012. Each class received its assigned combination of interventions within one week of pre-testing. Post-test instruments were administered within a week after the intervention. Informed consent of participants and their parents or guardians, following the Helsinki guidelines, was obtained after procedures were fully explained. All students in the classroom were exposed to the assigned intervention(s); only the consented students completed the assessment instruments. The Columbia University Medical Center Institutional Review Board approved the study.

Interventions

Curriculum

Eliminating the Stigma of Differences is a three-module, three-hour curriculum delivered by teachers over a three- to six-day period. Power-point slides provide a platform for classroom discussion. The teacher’s guide provides suggestions for questions to pose and information to convey to the class, as well as suggested in-class exercises and homework assignments. A demonstration video is also included. All materials were extensively pre-tested. Module 1 addresses the bases on which we judge others to be different; the definition, causes and consequences of stigma, including for students themselves; ways to end stigma; definition and description of mental illness; causes of mental illness; treatment for mental illness; barriers to help-seeking; how stigma applies to mental illness; and sharing personal experiences with mentally ill people. Modules 2 and 3 address ADHD, anxiety disorders, depression, and bipolar disorder, and include description of the disorders, discussion of causes and treatments, and stimulation of empathy. Suicide is also discussed. The curriculum employs principles of active learning and the encouragement of empathy throughout.

Contact

Two college students—one male, 27, with a history of bipolar I, and one female, 24, with a history of bipolar II disorder—made a 10-minute in-class presentation (20 minutes altogether) describing onset and course of their symptoms, hospitalizations and treatments, their feelings about the illness, coping strategies, and impact of the illness on social relationships and functioning at school and work. Based on previous research,⁴⁸ the talks were constructed to moderately disconfirm stereotypes of mental illness. The speakers practiced to ensure standardization of the presentations. Teachers moderated the presentations, followed by questions and answers.

Printed Materials

Teachers prominently displayed posters in the classroom for two weeks and provided students with bookmarks. The materials focused on individuals’ personal traits and abilities as opposed to language that labels a person as “mentally ill.” The curriculum and printed materials are accessible at http://www.mentalhealthconnection.org/anti_stigma_materials.php.

Outcome Measures Our primary goals were to reduce stigmatizing attitudes, beliefs, behaviors, and behavioral intentions and increase recognition of mental illnesses and favorable help-seeking attitudes. We assembled a comprehensive assessment package utilizing existing measures with established psychometric properties in children and adolescents, measures extensively tested in adults that we adapted for adolescents, and new items developed for the study. We developed composite scales using exploratory factor analysis. Internal consistency reliability of the scales was adequate to excellent for the overall sample and within gender, race/ethnicity, and socioeconomic status groups. All measures were pilot-tested with a racially/ethnically diverse group of youth in the target age range. Measures are organized in terms of knowledge and attitudes, behavior and behavioral intentions, personal help-seeking attitudes, and vignette-based questions. Following a long tradition in stigma research,^{14, 15, 49} two vignettes were created. One vignette character (Julia) met DSM-IV criteria for bipolar disorder, and the other (David) met criteria for social anxiety disorder. Participants read the vignettes and responded to questions about Julia and David.

See Table 3 for content, scoring, example items and reliability of the outcome measures and Online Supplement eTables 1 to 9 for wording of items, responses, vignettes, and pre-test descriptive statistics for all items.

Table 3.

Description of Outcome Measures

Measure	Number of Items	Scoringa	Cronbach’s Alpha	Example Items
Knowledge and Attitudes Measure
Knowledge and Attitudesb	21	1=strongly agree to	.78	It would be embarrassing to have a mental illness.
		5=strongly disagree		People with a mental illness tend to be violent and dangerous.
Behavior and Behavioral Intentions Measures
Stigma Awareness and Actiond	8	1=occurred in past	.67	I heard people use slang terms about mental illness like “psycho,” “crazy,” or “looney,” to put people down.
		2 weeks; 0=did not
Avoidance and Discomfortd	6	1=occurred in past	.63	I avoided a person who said odd things and behaved in strange ways.
		2 weeks; 0=did not
Social Distanceb	6	1=definitely no to	.89
		4=definitely yes
Social Distance Subscale (less acceptable forms)c	3	1=definitely no to	.81	Would it be okay with you to work on a class project with someone with mental illness?
		4=definitely yes
Social Distance Subscale (more acceptable forms)c	3	1=definitely no to	.72	Would it be okay with you to have someone with mental illness as a neighbor?
		4=definitely yes
Personal Help-Seeking Measure
Personal willingness to seek help	7	1=yes; 0=no	.78	I would talk to my doctor if I were having a mental health problem.
Vignette-Based Measures
Beliefs about vignette characters & their mental health conditions	6	1=not at all likely to	n/a: Analyzed as separate items	Julia/David is in this situation because (s)he is just a bad person.
		4=very likely
				Julia/David is experiencing a mental illness.
				Julia’s/David’s situation will improve with treatment.
Social distance from Vignette characters (combined overall score)	8	1=definitely no to	.92	Would it be ok with you to live next door to Julia/David?
		4=definitely yes
Vignette help-seeking recommendations	12	1=yes; 0=no	.75	Should Julia/David talk to a doctor about her/his problem?

^aAll scales are scored such that a higher score indicates more of the named construct.

^bAdapted from Wahl et al.⁴²

^cThe social distance scale was divided into more and less acceptable forms of contact based on pre- and post-test means on social distance items; we use the overall scale as our primary outcome measure.

^dScales combine attitude/awareness and behavioral items.

Other Variables Personal Contact with Mental Illness was assessed. ³² Scores indicate the most intimate level of contact reported, from 0 (“I have never observed a person with mental illness”) to 7 (“I have a severe mental illness”). Social Desirability Bias was examined with a reliable scale for research on children. ^50,51 Fidelity to the curriculum content, quality of delivery, and level of student engagement was assessed by two observers in each classroom (intraclass correlation = .93), using a 60-item measure (available on request) based on two existing tools with good psychometric properties. ^52,53

Statistical Analysis

Given the experimental design, personal characteristics could not cause self-selection into intervention groups. Still, it is possible for groups to differ at baseline because of imperfect randomization or differential participation. Significant pre-test differences were found between intervention and control groups for race/ethnicity, primary language spoken at home, parent/guardian’s education and income, and level of contact with mental illness. We included pre-test values of the corresponding outcome measures in the main analyses to control for preintervention group differences. We also re-ran our main analyses including the personal characteristics that differed at baseline and social desirability bias to see whether they were significantly related to the outcomes after controlling for pre-test values; they were not. Therefore, final analyses only control for pre-test values of the outcomes.

Using analysis of covariance (ANCOVA), we tested the effects of each intervention on the aforementioned outcomes. Across all analyses, outcomes for the printed-materials-only group did not differ significantly from the no-intervention control group; when combined with curriculum or contact, it did not change the outcomes produced by curriculum or contact alone. Therefore, we combined the “materials only” group with the control group. Those receiving materials plus contact were grouped with contact only, etc. Our analyses thus compare outcomes for four groupings: curriculum only, contact only, curriculum plus contact, and control. P-values less than 0.05 were considered statistically significant.

Results

Participation Rate

A total of 751 students (60% of those invited) agreed to participate. Of these, 721 (96%) completed the study. Loss to follow-up (n=30) did not differ by gender, school, or socioeconomic status but did differ by race/ethnicity, with African American students dropping out at a higher rate, mostly due to moving.

Fidelity to Curriculum Intervention

The mean fidelity score was 187, corresponding to a rating just over 3 out of 4 (the highest rating). We found no evidence of a linear effect of fidelity on the outcome measures, which we attribute to generally good to excellent fidelity and lack of substantial variance on the measure.

Outcome Analyses

Aside from correlations between the overall social distance measure and its two subscales, there were no extremely high correlations between outcome measures, indicating that measures represent distinct aspects of stigma (See Online Supplement eTable 10). The largest correlations were between Knowledge and Attitudes and social distance measures (−.56 to −.61) and between general and vignette-based social distance (.57 to .61). Other correlations ranged from nil to .40.

Table 4 shows that intervention groups significantly differ from each other for 10 of the 13 primary outcomes. Regarding the impact of specific interventions, we note consistent patterns. The curriculum fairly consistently increases recognition of mental illness and positive orientations to help-seeking, including Stigma Awareness and Action, identifying the vignette characters as mentally ill rather than bad, endorsing help-seeking for the vignette characters and for oneself in a similar situation, and increasing optimism about treatment effectiveness. The impact of the curriculum on social distance outcomes was weaker. Avoidance and Discomfort, and social distance from the vignette characters and from “someone with a mental illness” did not differ significantly from the control group, although a subscale of the more acceptable forms of contact (be friends, be neighbors, eat lunch together) did show significant improvement.

Table 4.

Analysis of Covariance Results: Adjusted Means and 95% Confidence Intervals for Curriculum, Contact, Curriculum plus Contact and Control Groups^a

	Curriculum		Contact		Curriculum & Contact		Control Group		F-Value	Df and Overall Sig of Group Diffb
	Adj. Mean	Conf. Limits	Adj. Mean	Conf. Limits	Adj. Mean	Conf. Limits	Adj. Mean	Conf. Limits
Knowledge and Attitudes Measure
Knowledge and Attitudes (1 to 5)	3.6	3.55–3.65	3.54	3.49–3.59	3.66**	3.60–3.72	3.53	3.48–3.58	F=4.33	3,720*
Behavior and Behavioral Intentions Measures
Stigma Awareness and Action (0 to 1)	.16**	.14–.18	.12	.10–.14	.19***	.16–.22	.12	.09–.14	F=6.42	3,716***
Avoidance and Discomfort (0 to 1)	.06	.05–.09	.070	.05–.09	.07	.04–.10	.06	.04–.08	F=.19	3,709
Social Distance (1 to 4)	2.06	1.97–2.14	2.20	2.12–2.28	2.03	1.92–2.14	2.15	2.06–2.24	F=2.98	3,716*
Subscale (less acceptable forms)	2.21	2.12–2.31	2.35	2.26–2.44	2.18	2.06–2.30	2.26	2.16–2.36	F=2.02	3,717
Subscale (more acceptable forms)	1.89*	1.80–1.98	2.07	1.98–2.16	1.87*	1.76–1.99	2.04	1.94–2.13	F=4.28	3,716**
Personal Help-Seeking Measure
Would Seek Help: Self (0 to 1)	.68**	.64–.71	.63	.60–.67	.71***	.67–.70	.60	.56–.64	F=4.76	3,707**
Beliefs About Vignette Characters and their Condition
Julia is a Bad Person (1 to 4)	1.70*	1.60–1.80	1.81	1.71–1.91	1.65*	1.52–1.78	1.85	1.75–1.96	F=2.53	3,703*
Julia Has a Mental Illness (1 to 4)	3.15**	3.05–3.25	3.16**	3.07–3.26	3.29***	3.16–3.42	2.94	2.83–3.04	F=6.43	3,683***
Beliefs About Vignette Characters and their Condition
Julia Will Improve with Treatment (1 to 4)	3.05*	2.96–3.14	2.94	2.83–3.02	3.07*	2.95–3.19	2.88	2.78–2.97	F=3.37	3,684*
David is a Bad Person (1 to 4)	1.73	1.64–1.83	1.83	1.74–1.93	1.68	1.56–1.81	1.75	1.64–1.85	F=1.36	3,693
David Has a Mental Illness (1 to 4)	2.81**	2.70–2.92	2.81**	2.70–2.92	2.83**	2.69–2.97	2.58	2.46–2.70	F=3.88	3,679**
David Will Improve with Treatment (1 to 4)	3.01***	2.91–3.12	2.73	2.63–2.84	2.99***	2.86–3.12	2.61	2.50–2.72	F=11.97	3,685***
Vignette Social Distance Measure
Social Distance from Julia & David (1 to 4)	2.14	2.05–2.23	2.19	2.10–2.28	2.05	1.94–2.17	2.12	2.02–2.22	F=1.24	3,686
Vignette Help-Seeking Recommendations
Julia & David Should Seek Help (0 to 1)	.77*a	.74–.80	.77*a	.74–.80	.79**	.75–.83	.71	.67–.74	F=3.98	3,707**

NOTE: Intervention groups that are significantly different from the control group are indicated by asterisks next to adjust means (*p < .05, **p <.01, ***p < .001) The final column presents the overall of significance indicating whether the four groups are significantly different from each other.

^aAll means are adjusted for the pre-test value of the outcome measure.

^bDegrees of freedom vary due to missing values on the outcome variable of interest.

The impact of contact is more limited. In the 10 instances where the overall effect of intervention is significant, the contact-only group differs significantly from the control group in only three instances—the belief that both vignette characters have a mental illness and that they should seek help. Adjusted means for curriculum only and curriculum plus contact never differ significantly, suggesting that in general contact does not add to the effect of the curriculum.

We also assessed interactions of gender, family income, race/ethnicity and primary language spoken at home with intervention. Of the 52 interactions tested, only 5 were significant at p<.05, and these failed to show any consistent pattern across outcomes: there was one for gender, one for income, one for race/ethnicity and two for language.

Discussion

Mental-illness stigma in the population has proved difficult to change. One promising approach may be to attack stigma at a young age, before negative attitudes become entrenched, but research on interventions with pre-adolescents is limited. We conducted a school-based field experiment with sixth-grade students that evaluated an anti-stigma curriculum, an in-vivo contact intervention, and a social marketing strategy that saturated classrooms with informative messages. We evaluated the impact of the interventions on knowledge and attitudes about mental illness, behavior and behavioral intentions related to mental illness, and help-seeking attitudes. Unfortunately the simplest intervention to apply – saturating classrooms with anti-stigma written materials – had no significant impact on any outcomes.

The contact intervention was less effective than the curriculum, with its impact limited to increasing the tendency to identify the vignette conditions as mental illnesses that should be treated. This is at odds with adult studies, which generally find contact interventions more effective than educational ones in reducing stigma. ³⁶ However, it is consistent with Corrigan et al.’s meta-analysis,³⁶ which revealed that the opposite is true with adolescents. We located only one previous study⁴⁶ with pre-adolescents that used a contact intervention. Whereas the study found reduced stigma following exposure to a lecture combined with personal contact, it did not find predicted effects of varying the degree of stereotype disconfirmation embodied in the contact intervention. Also, because contact was combined in every case with a lecture, it is not clear whether stigma reduction resulted from contact. While enthusiasm for contact interventions is justifiably strong, further evaluation of their effectiveness with young audiences is needed.

The curriculum was the most effective of the three interventions. It delivered a consistently beneficial impact on a diverse set of outcomes, tapping recognition of mental illness, awareness and action related to stigma, personal inclinations to seek help, and more favorable orientations toward the vignette characters and their prospects for improvement. However, the curriculum group did not differ significantly from the control group on avoidance and Discomfort, the belief that David (with social anxiety disorder) is a bad person, and most social distance outcomes. The first two null findings may have been influenced by methodological factors: reported instances of avoidance and discomfort, assessed over a two-week period, as well as the belief that David is a bad person, were very low for all experimental groups pre- and post-test. Our curriculum was less successful at reducing social distance than two previous studies that used pre-adolescent samples. ^{42, 45} It is not clear what accounts for the discrepant findings.

Limitations and Contributions

With any experimental study, recruitment bias can be problematic. Sixty percent of students invited decided to participate in our study. Some non-participating students or their families may hold particularly negative views about mental illness, possibly limiting the generalizability of our findings. Although for non-study-related reasons, two of the originally selected schools dropped out, leading us to return to five schools the following school year. We found no significant differences in respondent characteristics between repeated and non-repeated classrooms. Nevertheless, it is possible that the two lost classrooms differed from the remaining classrooms in terms of respondent characteristics or response to the interventions. The present results are also limited in that they include no long-term follow-up and rely on hypothetical rather than actual help-seeking.

At the same time, our study makes several advances over previous ones, allowing greater confidence in the validity and generalizability of findings: There was no self-selection of teachers into the study or particular intervention conditions. Our fidelity measure, whose inclusion was another advance in the literature, showed fidelity was generally high without selfselection or teacher training, suggesting that neither voluntary involvement nor previous training is necessary for high-quality implementation. The assessment of multiple interventions allowed us to attribute effects to particular interventions rather than intervention per se and led us to conclude that the curriculum was superior to the contact and printed materials interventions. Our sample had excellent representation of the major racial and ethnic groups in the US; consequently, we were able to show that interventions had a similar impact among Latino, African American and non-Hispanic white youth. All these study features raise optimism that stigma reduction interventions can generalize to a broad application in the population and strengthen existing evidence that even brief interventions can reduce stigma and may be transferrable to real-world applications. Future steps should include a direct comparison of curricula employed in the different intervention studies, perhaps combining elements that appear to most effectively address different components of stigma, and working with educators to begin to establish stigma-reduction as part of the regular school curriculum.