Successful Application of a Canadian Mental Health Curriculum Resource by Usual Classroom Teachers in Significantly and Sustainably Improving Student Mental Health Literacy

Abstract

Objective:

To investigate whether the significant and substantive findings from a previous study of youth mental health literacy (MHL) could be replicated using the same methods in another population.

Method:

We examined the impact of a curriculum resource, the Mental Health and High School Curriculum Guide (The Guide), taught by usual classroom teachers on students’ knowledge and attitudes related to mental health and mental illness in Canadian secondary schools. Survey data were collected before, immediately after, and 2 months after implementation of The Guide by teachers in usual classroom teaching. We conducted paired-sample t tests and calculated the Cohen d value to determine outcomes and impact of the curriculum resource application.

Results:

One hundred fourteen students were matched for analysis of knowledge data and 112 students were matched for analysis of attitude data at pre-intervention, post-intervention, and 2-month follow-up time periods. Following classroom exposure to the curriculum resource, students’ knowledge scores increased significantly and substantively, compared with baseline (P < 0.001, d = 1.11), and this was maintained at 2-month follow-up (P < 0.001, d = 0.91). Similar findings for attitude improvement were found (P < 0.001, d = 0.66), and this improvement was maintained at 2-month follow-up (P < 0.001, d = 0.52).

Conclusions:

These findings corroborate those from a previous study conducted in a different location. Taken together these results suggest a simple but effective approach to improving MHL in young people by embedding a classroom resource, delivered by usual classroom teachers in usual school settings.

About 1 in 5 people may experience a mental disorder during adolescence.^1–3 If left unrecognized and untreated, mental disorders can lead to substantial negative outcomes in physical and mental health, academic and vocational achievement, interpersonal relationships, and other important life domains.^2,4–9 Despite this tremendous burden of disability, youth requiring MHC predominantly do not receive it.^10–13

Lack of knowledge, presence of stigma, and limited access to care all serve as barriers to addressing mental disorders and alleviating the burden of disability.^14–18 MHL is an essential component of improving individual and population health and mental health outcomes.^18–23 As most mental disorders occurring over the lifespan can be identified by age 25, it is necessary to enhance MHL in young people to help improve their potential for good mental health outcomes.^{18,21,23–28} Schools provide the ideal location in which to implement interventions that can be demonstrated to improve MHL.^{14–16,23,29–35}

Early definitions of MHL, as “knowledge and belief about mental disorders which aid their recognition, management, and prevention,”19, p 182 have evolved concurrently with advances in understanding of health literacy,^36–39 including incorporation of constructs pertaining to stigma, advocacy, and help seeking,^20,21 and empowerment.⁴⁰ Similarly, stigma theory has evolved to acknowledge the importance of knowledge in stigma reduction.⁴¹ Recently MHL has been defined as comprising 4 components: understanding how to obtain and maintain good mental health; understanding mental disorders and their treatments; decreasing stigma; and enhancing help seeking efficacy (knowing when, where, and how to obtain MHC).^18,23,42

To date, emerging evidence suggests that some school-based programs may improve certain aspects of MHL in secondary school students. A recent systematic review on the effectiveness of 27 school-based MHL programs demonstrates their emerging evidence for knowledge improvement, stigma reduction, and help seeking efficacy enhancement.²² Other recent studies have identified similar positive, but relatively modest, findings in knowledge improvement and stigma reduction.^9,32

Clinical Implications

• Increased MHL in school settings may encourage young people to seek help from MHC providers.

• Embedding MHL interventions in schools may enable and empower teachers and students to have better communications with MHC providers.

Limitations

• Further research using control groups is needed.

• Further validation of the measurement tools used in our study is needed.

The Mental Health and High School Curriculum Guide (The Guide)

The Guide⁴³ is a web-based MHL curriculum resource, developed by mental health and education experts, designed for use in junior high and secondary schools, certified by Curriculum Services Canada, a pan-Canadian curriculum standards and evaluation agency, and endorsed by the Canadian Association for School Health. It was developed in recognition of the increasing awareness of the importance of health literacy as a necessary foundation for improving health, extrapolated into the area of youth mental health. The Guide was field tested in numerous schools across Canada and pilot studies of revised versions were conducted in collaboration with the Department of Education and Early Childhood Development, Province of Nova Scotia.⁴⁴

The Guide contains a teacher self-assessment tool, a teacher self-study module, a student evaluation tool, and 6 classroom ready modules. The modules include learning objectives, lesson plans, classroom-based activities, and teaching resources, including, but not limited to, written materials, animated videos, PowerPoint presentations, ready-touse classroom activities, and links to authoritative and trustworthy online mental health resources. The 6 modules are as follows: the stigma of mental illness, understanding mental health and wellness, understanding mental disorders and their treatments, experiences of mental illness, seeking help and finding support, and the importance of positive mental health.

The Guide is applied by usual classroom teachers who have been trained in its application, in usual classroom teaching, and has been designed to be integrated into existing health or human relationships curriculum. It takes about 10 to 12 classroom hours to implement. Teachers take part in a 1-day training session to become familiar with The Guide and improve their own MHL before implementing it in their classrooms. Currently, teacher training programs on the use of The Guide and use of this resource in grades 9 and 10 classrooms are being applied in numerous Canadian provinces. It is estimated that The Guide is being used in over 1000 schools nationwide.^18,44

Previous studies have reported that the 1-day training of teachers on the classroom application of The Guide significantly and substantively increased teacher mental health knowledge and improved attitudes toward mental health and mental disorders.^8,18 These positive MHL results were achieved not from an expensive, time intensive, additional mental health course provided to teachers, but as a value added outcome of the process of familiarizing teachers about how to implement the resource in their own classroom. Additionally, previous research has demonstrated that classroom teachers’ application of the resource in their own classrooms, without the addition of external mental health interventions, significantly and substantively improved students’ mental health knowledge and attitudes, and that these positive results were sustained over time.^17,28

This approach differs from other types of interventions that use mental health experts as in school educators^45,46 and from standalone programs that are not designed to be embedded into school curriculum, are not based on existing teaching methods used in schools, and do not concurrently address teacher and student MHL.

We sought to determine whether the above reported improvements in student MHL²⁸ could be replicated with a different population. Specifically, we examined the impact of classroom teachers’ application of The Guide on students’ knowledge and attitudes related to mental health and mental illness in 3 schools within the city of Toronto, Canada’s largest metropolitan area.

Methods

A mental health coordinator within the TDSB identified 3 different schools, anonymized from the researchers, who were interested in evaluating the impact of The Guide. Teachers from these 3 schools who had received the 1-day training on the use of The Guide in a session conducted under the auspices of the Ontario Shores Mental Health Centre in late 2012 and who agreed to participate in this evaluation were then recruited. Between January and December 2013, these teachers implemented The Guide as a regular component of Grade 9 health classes. Each teacher applied The Guide in their classroom using their own teaching techniques. The type of course (for example, physical education) was not predetermined and this information was not available for analysis.

For evaluation purposes, school personnel collected anonymous survey materials from students before and after the delivery of The Guide (pre- and posttest), as well as at 2-month follow-up. Our study is a secondary analysis of all the data collected. We compared students’ responses gathered before the implementation of The Guide (pretest), with those gathered immediately following the implementation (posttest), and also with responses gathered at 2-month follow-up. Ethical approval was obtained from the Research Ethics Board of Dalhousie University along with the research committee of the TDSB.

Measures

The knowledge and attitude survey is composed of 2 sections. The first includes 28 items assessing general mental health knowledge, corresponding to material contained in the 6 modules of The Guide. Items are presented in a “true or false” and “do not know” format. Students are instructed to use the “do not know” option if they do not know the answer. The Cronbach alpha for internal consistency for our sample is 0.637. The knowledge questionnaire in our study intentionally includes items of multidimensional structure, as opposed to unidimensionality usually considered when applying this measure. The second section uses 8 items to examine attitudes related to mental disorders or illness. These items include statements about mental disorders and (or) people with a mental illness and ask respondents to select their level of agreement using a 7-point Likert scale (that is, from strongly disagree to strongly agree). The Cronbach alpha for internal consistency for our sample was 0.652. The attitude part of the survey addresses various stigma constructs, such as social distance, and personal and perceived stigma. It includes statements such as, A mentally ill person should not be able to vote in an election and I would be happy to have a person with a mental illness become a close friend (see online eAppendix 1 for full student survey). Both parts of the survey were developed with consultation from a child and adolescent psychiatrist, a clinical psychologist, and a group of educators and students to confirm its face validity. The knowledge survey matches the content of The Guide resource, and the attitude survey was developed to cover 3 major aspects based on a literature review of stigma tools: stigma against the person with mental illness, stigma against causes of mental illness, and stigma against help seeking and treatment.

To ensure the anonymity of the survey, 5 linking questions were predesigned to match participants’ pretest, posttest, and follow-up test. These include the participant’s first pet’s name, birth month, postal code, shoe size, and the last 2 digits of their phone number. Unmatched surveys were excluded from the data analysis.

Statistical Analysis

To determine the significance of observed scores between pretest, posttest, and follow-up, a series of paired-sample t tests were employed to compare changes in the total number of correct scores on survey items pertaining to general mental health knowledge and total attitude scores. A Bonferroni correction was applied to reduce the chance of obtaining false-positive results (that is, type I errors) associated with the use of multiple t tests. Therefore, significance levels of Cronbach α = 0.05 were adjusted to a significance level of 0.00834 (6 t tests). PASW 17.0 (IBM SPSS Inc, Armonk, NY) was used to conduct the analysis.

Results

The total sample size was 175 (89 females, 49 males, and 37 did not identify their sex). One hundred fourteen students were matched for the analysis of knowledge data and 112 were matched for attitudes data at pretest, posttest, and 2-month follow-up.

The knowledge assessment allows for scores ranging from 0 to 28, with higher scores indicating more knowledge. Prior to their exposure to The Guide, students responded correctly to an average of 55.18% (mean 15.45, SD 3.97) of 28 questions pertaining to mental health knowledge. Following exposure to The Guide, correct responses improved to an average of 69.64% (mean 19.50, SD 3.39) of the 28 questions, which was significantly and substantially improved over baseline scores, (t = 12.83, df = 113, P < 0.001; d = 1.11). Students’ knowledge scores at the 2-month follow-up period were also significantly higher (pretest: mean 15.45, SD 3.97; 2-month follow-up: mean 19.11, SD 4.04), compared with baseline knowledge scores (t = 11.18, df = 113, P < 0.001; d = 0.91) (Table 1). There was no significant difference found between posttest and 2-month follow-up.

Table 1.

Change in knowledge scores from pretest, posttest, and follow-up

Pretest	Posttest	Follow-up	t test comparison	Cohen d
15.45 (3.97)	19.50 (3.39)		t = 12.83, df = 113, P < 0.001	1.11
15.45 (3.97)		19.11 (4.04)	t = 11.18, df = 113, P < 0.001	0.91

The d statistics represent the effect size of the training, describing the magnitude of difference between the pretest and posttest or between the pretest and follow-up.

The mental health attitudes assessment allows for scores ranging from 8 to 56, where larger scores are indicative of more positive attitudes. Following exposure to The Guide, students showed significant and substantial improvements in attitude between pretest surveys (mean 42.56, SD 6.08) and posttest surveys (mean 46.42, SD 5.67) (t = 8.54, df = 111, P < 0.001; d = 0.66). These improvements were maintained at the 2-month follow-up (mean 45.86, SD 6.66) (t = 6.2, df = 111, P < 0.001; d = 0.52) (Table 2) when comparing against baseline scores (mean 42.56, SD 6.08). There were no significant differences identified between posttest and 2-month follow-up. Significant differences between pretest and posttest attitude scores remained after the Bonferroni correction was applied.

Table 2.

Change in attitude scores from pretest, posttest, and follow-up

Pretest	Posttest	Follow-up	t test comparison	Cohen d
42.56 (6.08)	46.42 (5.67)		t = 8.54, df = 111, P < 0.001	0.66
42.56 (6.08)		45.86 (6.66)	t = 6.2, df = 111, P < 0.001	0.52

The d statistics represent the effect size of the training, describing the magnitude of difference between the pretest and posttest or between the pretest and follow-up.

Discussion

Our study of an MHL resource (The Guide) applied by usual teachers in usual classroom settings demonstrated statistically significant, substantial and sustained improvements in student’s knowledge and attitudes at 3 schools in the TDSB. As such, it replicates the results obtained in a previous study²⁸ conducted in a rural and suburban population in an urban secondary school sample. These findings are also similar to those described by Milin et al¹⁷ in an RCT study of students in the Ottawa School Board, although in that study help seeking efficacy was also measured with significant positive results.

Taken together, these studies illustrate that it may be possible to make significant improvements in secondary school students’ MHL (knowledge and attitudes) simply by integrating a mental health curriculum resource (The Guide) in usual classroom lessons taught by usual classroom teachers. This approach does not rely on the use of heavily marketed, expensive, externally applied programs,⁴⁷ nor does it rely on using mental health experts as classroom educators as has been applied in some other countries.^45,46 In contrast, it applies pedagogically flexible methods to inform usual classroom teachers in how to use an MHL curriculum resource (The Guide) and uses their professional skills in the classroom delivery of the materials. When teachers receive the training on The Guide their MHL significantly improves and when they subsequently apply it in their classrooms, student MHL substantially improves. Thus a simple curriculum resource, integrated into usual school practices, may be able to improve both teacher and student MHL concurrently. An additional advantage is that the resource can be easily embedded within existing provincial school curriculum and once it is in the curriculum it can be used repeatedly and does not require additional budget allocations to support its implementation. It has the potential to become a relatively inexpensive and self-sustaining approach. As well, since both the teacher and student evaluations of MHL have been researched and the measurement tools are freely and easily available online, education authorities (for example, schools, school boards, and policy makers) can conduct ongoing evaluations of the impact of this resource and approach as part of usual program evaluation without additional and costly research study investment. For example, the resource contains a standard student evaluation that could be collected and reviewed at appropriate intervals and compared with those conducted previously. This open-sourced approach also provides additional MHL materials for community stakeholders, such as mental health professionals and researchers, youth centres, and family organizations that can easily access the information online to help inform their practices and activities.

To our knowledge, this is the only approach in school mental health that concurrently and seamlessly addresses both teacher and student MHL with the same activity. Thus, in contrast to most school mental health interventions,^22,48–50 the application of The Guide builds on existing classroom teaching processes and does not require additional investment in the purchase and application of standalone programs that have not been specifically created to be contextually integrated into school settings.^47,51,52

Of added specific interest, our study and others identified above have demonstrated that simply applying an MHL resource (The Guide) integrated into usual school curriculum demonstrates a highly positive impact on decreasing stigma in students pertaining to mental health and mental illness. This allows for schools to consider an alternative approach to stigma reduction than those that are often applied as standalone interventions.⁵³ However, adding other evidence-based, stigma-reduction interventions to The Guide resource may demonstrate additional impact on improving youth attitudes toward mental health and mental illness. This may be a fruitful area for further study.

Our study did not examine the MHL of teachers trained in the use of The Guide, but their MHL may be considered to be consistent with that found in other research that has examined the impact of this approach on the MHL of teachers.^8,18 In a series of program evaluations on the impact of a single training day on the use of The Guide resource by classroom teachers in other Canadian locations, Kutcher and collegues^8,18 have demonstrated significant improvements in teachers’ knowledge (P < 0.001; Cohen d ranging from 1.5 to 2.9) and attitudes (P < 0.001; Cohen d ≤ 1.2).⁵⁴ Thus this approach is consistent with existing classroom teaching processes and allows for application that can be used across many different school settings.

Our study has certain limitations. It is a pre–post with one 2-month point follow-up survey design that lacks a control group for comparison. The students participating in our study were not randomly selected and thus we are not able to extrapolate the results to the wider TDSB. Additionally, unlike in the Milin et al report,¹⁷ we did not measure help seeking as a result of exposure to The Guide, and, for logistical considerations we were unable to extend the follow-up assessment to a period of longer than 2 months. Also, the internal consistency (Cronbach α = 0.65) of the attitude survey is slightly lower than 0.70 and needs further validation. Factor analysis of both the knowledge and attitudes components of the survey is needed in the future to determine the factor structure of both parts.

Conclusions

Our study reports on the significant and substantial improvements realized in students’ MHL by the application of an MHL resource (The Guide) by usual school teachers into usual classrooms. This MHL resource encompasses mental health promotion, knowledge about mental illness and related treatment options, stigma reduction, and enhancement of help-seeking efficacy, thus combining several aspects pertaining to mental health and mental illness that are often addressed separately. As this approach could fit well within the usual operation of schools worldwide, it may have global applicability because of its flexibility of use within education systems and ease of integration into regular teaching activities. Research is currently under way into testing this possibility in various medium and low-income countries and also other Canadian schools for its effectiveness and generalizability in different settings and contexts. The Guide resource further has the potential to benefit mental health clinicians, as it addresses knowledge and stigma, 2 known predicators of help seeking behaviours.

Future research would benefit from adding measures of help seeking efficacy and a longer follow-up period. Although it is positive to see effects lasting 2 months after in-classroom implementation, it would be valuable to know the impact at 6 and 12 months. These issues are being investigated in an ongoing RCT in a Canadian setting.

Abbreviations

MHC

mental health care

MHL

mental health literacy

RCT

randomized controlled trial

TDSB

Toronto District School Board