Experimental evaluation of a school-based mental health literacy program in two Southeast Asian nations

Abstract

Background.

Low mental health literacy (MHL) is a particular challenge in many low and middle-income countries (LMIC). School-based MHL programs hold promise to increase MHL but lack rigorous research assessing their effectiveness in LMIC. The present study evaluated a school-based MHL program, the ”Mental Health & High School Curriculum Guide” (“The Guide”), implemented separately in two different contexts in Southeast Asia (Vietnam and Cambodia) following adaptations made by the research team.

Methods.

Participants were 80 teachers and 2,539 students from 20 schools in Vietnam (Study 1), and 67 teachers and 275 students in one school in Cambodia (Study 2). In Vietnam, teachers/classrooms were randomized to either The Guide MHL program or a treatment-as-usual control condition, with teachers in the intervention condition receiving a 3-day training in The Guide and implementing the 6-module curriculum in their classrooms. In Cambodia, school staff were randomized to either receive The Guide training or to the control condition; four teachers who received the training implemented the curriculum in select classrooms. In both studies, teachers’ and students’ mental health knowledge and attitudes were assessed at baseline and following completion of the classroom curriculum.

Results.

In Vietnam, 6 of 7 program effects for teachers were significant with some large effects (e.g., teacher Recognition of Mental Health Disorders, R²=.36); effects for both of the student outcomes were significant, but small. Results were similar in Cambodia, with 6 of 9 program effects significant favoring the treatment group; effect sizes in Cambodia were smaller than in Vietnam for teachers/staff but larger for students.

Conclusion.

Findings suggest that with limited adaptation, a teacher-delivered MHL intervention can produce measurable increases in MHL among teachers and students in two Southeast Asian countries. These results support the value of school-based MHL training provided via an inexpensive and teacher-friendly program, embedding MHL into classrooms. Some small effect sizes suggest the importance of additional development and research targeting these particular components.

Introduction

Supporting child mental health¹ is a global challenge, as the prevalence of mental health disorders in children and adolescents is high and apparently increasing worldwide (WHO, 2012), with the majority of mental health disorders experienced over the lifetime first manifesting by late adolescence (WHO, 2018). Mental health conditions are associated with a considerable burden and disability (Vigo, Thornicroft, & Atun, 2016), negatively impacting young peoples’ development, quality of life, and functioning (Fisher & Cabral de Mello, 2011; Schulte-Korne, 2016). Supporting child mental health is critical but particularly challenging in low and middle income countries (LMIC; defined as economies with a per capita gross national income lower than an annually set threshold by the World Bank, n.d.). In most LMIC, there is a significant lack of mental health resources including human resources, policy focus, and mental health infrastructure (Malhotra & Patra, 2014; Patel, Kieling, Maulik, & Divan, 2013; Weiss et al., 2012). These gaps in support generally are more pronounced for children, given the specialist nature of child mental health services and the complexity of coordination across multiple service sectors such as health, protection, and education (Kieling et al., 2011; Patel et al., 2013).

Regardless of the country or context, a central foundation to child—and all—mental health development is mental health literacy (MHL), defined by Jorm (2000) as “knowledge and beliefs about mental disorders and their treatment, which aid their recognition, management or prevention” (p.396). MHL has several components, including (1) understanding how to maintain positive mental health, (2) knowledge about mental health disorders and their treatments, (3) low levels of mental health-related stigma, and (4) high levels of help-seeking efficacy (Kutcher, Wei, & Coniglio, 2016). Taken together, MHL competencies establish a foundation to support self-care, recognition of the need for help in oneself or others, and a social and policy environment that enables individuals to access effective care when needed, thereby improving mental health outcomes (Kutcher, Wei, & Coniglio, 2016). People living in LMIC tend to have lower MHL, which contributes to decreased and delayed treatment seeking (Wei, Hayden, Kutcher, Zygmunt, & McGrath, 2013b). This reflects not merely difficulty in recognizing mental health problems, but also the consequences of cultural beliefs that often imply causal attributions of mental illness to personal failings or supernatural rather than biomedical causes; in many such settings even trained non-mental health medical professionals may have poor understanding and negative attitudes toward mental illness (Ganasen et al., 2011; Mascayano, Armijo, & Yang, 2015).

MHL centrally involves symptom recognition and understanding ways and means of accessing care, and it therefore is important that MHL interventions be integrated into daily life and existing organizational structures (Kutcher, Wei, & Coniglio, 2016). Given that most children spend a significant portion of their day in schools, the educational system represents a logical setting for provision of child mental health services (Wei & Kutcher, 2012; Weist, 2003). School-based mental health services are of particular value in LMIC given LMIC’s low levels of formal mental health infrastructure (Fazel, Patel, Thomas, & Tol, 2014; Murray, Dorsey, & Lewandowski, 2014; Weiss et al., 2012). School-based programs can include prevention, promotion, and treatment services, potentially leveraging task-shifting or task-sharing approaches in which teachers or other non-mental health personnel are trained to provide mental health-related programming (McInnis & Merajver, 2011). In the case of MHL interventions, teachers may be integrated through a “train the trainer” approach in which teachers receive training and then become the trainers themselves. A potential benefit of this approach for students is that by directly serving as the MHL trainer (rather than hosting an external trainer, for example), teachers become positioned as someone with whom students have now had an opening dialog about mental health, thereby potentially reducing barriers to future teacher-student engagement around mental health issues. However, there is a need for effective resources to increase MHL for teachers themselves in LMIC in order to effectively engage teachers as mental health supports for students.

Improving both teacher and student MHL may impact student wellbeing in a number of different ways. For example, teacher stress and student behavior problems tend to co-occur and likely have a bidirectional relationship (Ahnert, Harwardt-Heinecke, Kappler, Eckstein-Madry, & Milatz, 2012; Clunies-Ross, Little, & Kienhuis, 2008; Herman, Hickmon-Rosa, & Reinke, 2018). Raising teachers’ MHL to improve their own self-care and help-seeking can prepare them to provide a more psychologically supportive classroom for their students; likewise, improving students’ abilities to cope with stress may also have positive impacts on classroom climate (Holen, Waaktaar, Lervåg, & Ystgaard, 2013). Teachers are also well placed to identify student mental health needs and make referrals for services, yet most teachers receive little mental health training and feel poorly equipped to take on this role (Graham, Phelps, Maddison, & Fitzgerald, 2011; Mazzer & Rickwood, 2015; Reinke, Stormont, Herman, Puri, & Goel, 2011). In addition, teachers play a critical role by providing a supportive relationship through which a student can seek help, and also by facilitating continued engagement by making appropriate classroom accommodations for students with mental health concerns (Krane, Karlsson, Ness, & Binder, 2016). Programs directly targeting student MHL may also improve student outcomes by encouraging recognition, self-care, and help-seeking (Kelly, Jorm, & Wright, 2007; Weiss, Dao, Dang, & Trung, 2019).

Research indicates positive effects of MHL training for both students and teachers. Across a systematic review of 27 student MHL training articles (Wei, Hayden, Kutcher, Zygmunt, & McGrath, 2013a), and two systematic reviews of teacher MHL training of 16 and eight studies, respectively (Anderson et al., 2018; Yamaguchi et al., 2019), most assessments of MHL training reported significant improvements in student and teacher knowledge and attitudes; there is, however, less evidence for changes in actual help-seeking or helping practices. Two other limitations in this literature identified by these reviews were a lack of full experimental design (e.g., randomization to a treatment vs. control condition) and few studies conducted in LMIC, where needs are highest (Anderson et al., 2018; Wei et al., 2013a; Yamaguchi et al., 2019).

There is a growing body of research, however, that supports the potential for mental health literacy training in LMIC. For example, numerous randomized controlled trials have shown that evidence-based mental health treatments, with appropriate adaptations, can be effective and appropriate in LMIC (Singla et al., 2017). This suggests that MHL training programs with demonstrated effectiveness elsewhere similarly may also be effective in LMIC, but will likely require adaptation and testing in each new setting. Indeed, prior MHL training efforts in Africa and Central America have reported substantial cultural alignment with the original curriculum, but also necessary cultural adaptations such as including additional training modules (Kutcher et al., 2015) as well as changing idioms, examples, and help-seeking strategies to the local context (Ravindran et al., 2018). Thus, additional LMIC-focused MHL research is essential, both in terms of continued curriculum development as well as by answering unresolved questions of effectiveness using more rigorous experimental designs.

Current Study

The primary goal of the current study was to experimentally evaluate the efficacy of an evidence-based MHL program, initially developed in Canada, for use in Vietnam following adaptations made by the research team. The current project also examined the portability of this program to other areas of Southeast Asia, through a pilot efficacy study in neighboring Cambodia. These neighboring countries have many cultural, historical, and economic similarities, but also have significant differences, including differences with regard to cultural explanations for mental illness, historical approaches to mental health services integration, and availability of school-based mental health resources (e.g., Hinton, Pollack, Weiss, & Trung, 2018).

Study 1 reports on the project’s primary focus, the adaptation of the program and assessment tools for Vietnam, and results of the accompanying evaluation. Study 2 reports on extension of the Vietnam program to neighboring Cambodia, and the subsequent pilot testing of the program there. An earlier report of the Cambodian study, with some differences in data analysis, has previously been published focusing on implications specific for Cambodia (Phoeun, Nguyen, Dang, Tran, & Weiss, 2019). Given the limited research on school mental health in Southeast Asia, our purpose in combining the fully-powered Vietnam study and the pilot Cambodian study was twofold: (a) to leverage an opportunity to extend the potential reach of the intervention by developing, testing, and disseminating it in two languages; and (b) to describe our exploration of whether further expansion of this program in Southeast Asia is warranted. We hypothesized that in both countries, at post-intervention teachers and students participating in the MHL program would show greater levels of MHL (i.e., constructive knowledge and supportive attitudes regarding mental health) relative to a control group.

STUDY 1 – VIETNAM EVALUATION

Methods

Setting

Vietnam is a Southeast Asian country with a population of approximately 97 million people, the 15^th largest country in the world (Central Intelligence Agency, 2019). It is classified as a lower-middle-income country by the World Bank and ranks 116^th (out of 189) on the Human Development Index (a composite of life expectancy, education, and per capita income; United Nations Development Programme, 2019). Participation in the educational system is high in Vietnam, with 90% of children attending lower secondary school (grades 6-9), and 71% attending upper secondary school (grades 10-12) (UNICEF, 2019). School-based mental health initiatives thus are a core element of efforts to improve access to care for Vietnamese youth (Dang, Weiss, Lam, & Ho, 2018). As in many LMIC, MHL in Vietnam is low. A recent study in Vietnam found, for instance, that only 32% of college student participants were able to correctly identify depression in a vignette, and the majority of those correctly identifying depression believed that treatment would require psychiatric hospitalization (Nguyen Thai & Nguyen, 2018). In our own work with teachers in Vietnam, we found that the MHL of Vietnamese teachers was approximately 1.7 standard deviations below that of an Australian norm group (Dang et al., 2018).

Study Design and Sample

Our evaluation in Vietnam utilized a pre-post randomized design in twenty secondary schools (randomly selected from a pool of 36 possible schools) in the coastal Vietnamese city of Danang. To reduce cross-group contamination, randomization occurred at the school level, with allocation of 10 schools to the intervention condition and 10 to the control condition. The project targeted 8^th and 9^th grades because the MHL curriculum content was deemed most developmentally similar to the content taught in 8^th and 9^th grade biology in Vietnam. Eighty teachers (two per grade per school) were randomly selected out of a total of 110 available teachers (53 in the intervention schools and 57 in the control schools). In the intervention condition, teachers participated in a three-day training and then delivered the 6-module classroom-based MHL curriculum to their students over five weeks. Teachers and students in control schools received no MHL training. Teachers were not blind to intervention status. Both teachers’ and students’ mental health knowledge and attitudes were assessed before and after the classroom curriculum was implemented. The study was approved by the Institutional Review Board at Vietnam National University (U.S. FWA #00018223).

MHL Intervention

The Mental Health & High School Curriculum Guide ("The Guide"; Kutcher, Wei, McLuckie, & Bullock, 2013) is an evidence-based MHL training program designed to be delivered in junior high and high school classrooms (7^th to 11^th grade) by trained teachers. The Guide was initially developed and tested in Canada, but has been expanded and is currently available in English, French, Spanish, and Bengali versions (www.teenmentalhealth.org). Including both teacher training and classroom-ready materials, The Guide is designed to promote positive mental health, improve knowledge of mental disorders and treatments, decrease stigma, and enhance help-seeking efficacy through six classroom-based modules. The teacher training component includes a 2-day course on MHL, reviewing mental health concepts including adolescent development, brain function, and mental health, classification and concepts of mental disorders, and stigma. A 3^rd day of teacher training then focuses on practical use of The Guide in the classroom, including lesson planning and discussion of teaching strategies. The classroom modules, typically delivered in 60-minute sessions, focus on: (1) The Stigma of Mental Illness; (2) Understanding Mental Health and Mental Illness; (3) Information on Specific Mental Illnesses; (4) Experiences of Mental Illness; (5) Seeking Help and Finding Support; and (6) The Importance of Positive Mental Health. Research has found positive impacts of the program on knowledge and stigma among high school students (Milin et al., 2016) and educators (Kutcher, Wei, McLuckie, & Bullock, 2013). Studies in Malawi (Kutcher et al., 2015), Tanzania (Kutcher et al., 2016; Kutcher et al., 2017) and elsewhere (Ravindran et al., 2018) support the transferability of this program, but the intervention had not previously been adapted and tested in Asia.

The USA Edition: Washington State (2nd ed.) version of The Guide (Kutcher & Wei, 2017) served as the basis for mental health literacy training in the present project. Two stages of cultural adaptation were used for The Guide. The first stage focused on the cultural appropriateness of core procedures and concepts (e.g., of what “mental health” consists and how it is defined), the second focused on the cultural appropriateness of the details of program including examples, specific wording of concepts, and other linguistic issues. In the first stage, the research team (BW, DB, HMD) reviewed the English version of The Guide to identify critical cultural and / or linguistic issues related to the core concepts of the program; none were identified. The manual then was translated to Vietnamese and iteratively edited by DB and HMD. In the second stage of the cultural adaptation process, the Vietnamese version of the manual (“The Guide-VN”) was reviewed by two psychiatrists, five teachers, and three psychologists in Vietnam to identify specific activities or wordings that needed to be adjusted for cultural and contextual relevance. Adaptations included replacement of specific words (e.g., in Vietnamese, the formal word for “mental illness” contains a somewhat stigmatized meaning, similar to “insane” in English, so this term was changed to “mental health problems”), case studies, and examples (e.g., for an activity exploring celebrities’ experiences with mental health problems, Vietnamese celebrities with whom students would be more familiar were used). One structural adaptation was made, to present depression as the first example of mental illness example rather than schizophrenia. This change was made (a) because given the relative prevalence of these two disorders, teachers and students likely would have had more direct experience with depression than schizophrenia, and (b) because common stigmatized perceptions of mental illness as “madness” or “craziness” might be more reinforced if the first mental illness presented was an illness centrally involving psychosis.

Teacher Measures

Teachers’ knowledge and attitudes regarding mental health and related constructs was assessed using the Mental Health Literacy Scale (MHLS; O’Connor & Casey, 2015). The original 35-item MHLS has good internal consistency (α = .87) and test-retest reliability (r = .80), as well as support for its validity in evaluating MHL intervention outcomes (O’Connor & Casey, 2015). The Vietnamese version of the scale, which was previously adapted and validated for use in Vietnam (Dang et al., 2018), was used in the present study. In the Vietnamese version, seven items not relevant for Vietnam were dropped, (e.g., because there is currently no data regarding their validity in Vietnam, “To what extent do you think it is likely that in general in Australia, women are MORE likely to experience a mental illness of any kind compared to men”). The final scale retained 28 items assessing: (1) Recognition of mental health disorders (e.g., “If someone experienced a low mood for two or more weeks, had a loss of pleasure or interest in their normal activities, and experienced changes in their appetite and sleep then to what extent do you think it is likely they have Major Depressive Disorder?”; eight items); (2) mental health Help-Seeking Self-Efficacy (e.g., “I am confident that I know where to seek information about mental illness”; four items); 3) Stigma / Negative Attitudes toward mental illness (e.g., “A mental illness is a sign of personal weakness”; 9 items); and (4) Willingness to Interact with people with mental illness (e.g., “How willing would you be to make friends with someone with a mental illness?”; 7 items). Item responses used a 5-point Likert scale ranging from 0 (“very unlikely” / “strongly disagree” / “definitely unwilling”) to 4 (“very likely” / “strongly agree” / “definitely willing”). Sub-scale scores were calculated as the mean of all answered items, retaining the 0-4 scale range to increase ease of interpretation. For recognition, self-efficacy, and willingness to interact, higher scores are more positive; for stigma, higher scores indicate more stigma towards mental illness. In the current sample, internal consistency was: Recognition α = .72 (T1) and α = .85 (T2); Help-seeking Self-efficacy α = .62 (T1) and α = .83 (T2); Stigma α = .64 (T1) and α = .78 (T2); and Willingness to Interact α = .78 (T1) and α = .92 (T2).

The 21-item Beliefs Towards Mental Illness Scale-BMI (BMI; Hirai & Clum, 2000) was also used to measure negative stereotyped views of mental illness (i.e., stigma) across three domains: (1) the Dangerousness of individuals with mental illness (e.g., “A mentally ill person is more likely to harm others than a normal person”; 5 items); (2) beliefs that individuals with mental illness have Poor Interpersonal and Social Skills (e.g., “ It might be difficult for mentally-ill people to follow social rules such as being punctual or keeping promises”; 10 items); and (3) beliefs about the Incurability of mental illness (e.g., “Psychological disorders are unlikely to be cured, regardless of treatment”; 6 items). Items are rated on a 6-point Likert scale ranging from 0 “completely disagree” to 5 “completely agree”. Scores were calculated as the mean of all included items, retaining the 0-5 scale range. For all subscales, higher scores indicate more negative stereotypical views (i.e., higher stigma) toward mental illness. The BMI had not been previously used in Vietnam, so it was translated and reviewed for necessary adaptations by the same team and following the same approach as described above for The Guide. Internal consistency was: Dangerous α = .42 (T1) and α = .78 (T2); Poor social skills: α = .76 (T1) and α =.87 (T2); and Incurability α = .60 (T2) and α = .64 (T2).

Student Measure

Students completed the 36-item Mental Health Knowledge and Attitude Test developed to accompany The Guide (Kutcher & Wei, 2017), which was reviewed and translated into Vietnamese by the study team along with The Guide. This measure assesses Knowledge, with 28 statements evaluated as true / false / I don’t know (e.g., “Most people who have a mental illness get well and stay well with treatment”). To more accurately assess knowledge and avoid correct responses by chance, if students do not know an answer they are encouraged to select “I don’t know” rather than guess. Knowledge scores are reported as the proportion correct (range: 0-1), with “I don’t know” considered not correct as it indicates a lack of knowledge. Internal consistency for the Knowledge subscale was α = .57 (T1) and α = .69 (T2). An additional eight items assess Stigma (e.g., “It is easy to tell when someone has a mental illness because they usually act in a strange or bizarre way”), with Likert scale responses ranging from 0 “strongly disagree” to 4 “strongly agree”. This 5-point response scale is an adaptation from The Guide’s 7-point version, a decision made by the Vietnam team to maintain consistency with the previously validated MHLS response options. Stigma scores were calculated as the mean of all items, with higher scores indicating more stigma regarding mental illness. Internal consistency for the Stigma subscale was α = .58 (T1) and α = .51 (T2).

Program Implementation Fidelity Assessment

Implementation fidelity was assessed using a 15-item lesson observer fidelity rating form adapted for the current study from existing measures (Dang, Weiss, Nguyen, Tran, & Pollack, 2017; Hahn, Noland, Rayens, & Christie, 2002; Rohrbach, Dent, Skara, Sun, & Sussman, 2007). The form evaluates five implementation domains including Appropriate Content (e.g., “followed lesson objectives”; 3 items); Correct Process (e.g., “followed the order of the lesson”; 3 items); Correct Use of Materials (e.g., “used materials as described in the manual”, 2 items); Quality of Implementation (e.g., “teacher understood concepts”; 4 items); and Student Participation (e.g., “students participated in discussion”; 3 items). Observer ratings are made using a three-point Likert –scale of 0 “no” / “not done”, 1 “partially” / “partially done”, 2 “good” / “completed”, with these ratings respectively reflecting approximating less than 30%, less than 70%, and above 70% of the implementation target achieved. All of the intervention classroom lessons were rated by one of five trained, master’s level (in education or psychology) observers who participated in The Guide implementer training as well as in an additional half-day training and practice for the fidelity observers. Observer assignment to classrooms rotated by week, so that each teacher was rated by multiple observers over the course of the intervention to minimize influence of rater differences.

Procedures

After reviewing the research proposal and granting permission, the Danang Department of Education sent an introduction letter to all schools (n=36) in three Danang school districts. All contacted schools expressed interest and agreed to participate, and then N = 20 were randomly selected and randomly assigned to the intervention (n = 10) or control (n = 10) condition. In each school, teachers and their classrooms were randomly selected for recruitment, and contacted to obtain teachers’ informed consent. As happens not infrequently in Vietnam (e.g., Weiss et al., 2014), all teachers agreed to participate. For each of the two assessments, which lasted about 30 minutes teachers received 50,000 VND (equivalent to about $2.50 USD). Teachers in the intervention group received an additional 500,000 VND ($25 USD) for attending the 3-day teacher training, and 750,000 VND ($35 USD) for teaching the curriculum in their classrooms.

Teachers’ MHL was assessed (a) pre-intervention, prior to their participation in the teacher training program and (b) post-intervention, after implementing The Guide-VN in their classrooms; control group teachers completed assessments on the same time schedule. The student assessment was administered to students one week before The Guide-VN classroom curriculum was implemented in the treatment group classrooms, and again the week after classroom training was completed. For both teachers and students, the T1 and T2 assessments were approximately seven weeks apart. After the initial assessment, teachers in the intervention condition participated in the three-day teacher training workshop led by the Director of the Danang Psychiatric Hospital, a psychiatrist with extensive experience in Western-based psychotherapy, youth and school-based mental health (Dang et al., 2016; 2018). This training workshop was based on the training agenda outlined in the teacher training materials, and incorporated a blend of didactic lectures, case studies, group discussion, and role play.

The intervention teachers then taught The Guide-VN modules to their students during their standard 45-minute “life skills” classes twice per week over a 5-week period. In Vietnam, students are organized into cohorts within grades where all students share the same schedule (i.e., students do not rotate in and out of each other’s classes over the course of the day). During the intervention period, the standard life skills curriculum was replaced with The Guide-VN curriculum in the intervention schools; the students in the control schools received their standard curriculum. Trained research assistants observed each classroom implementation session and immediately afterwards completed the fidelity rating.

Because the MHL curriculum was viewed as comparable to schools’ standard “life skills” curriculum in which all students participate, schools obtained informed consent at the student level through a passive consent process approved by the Vietnam National University IRB. Before beginning the lessons, letters were sent home to parents informing them of the study and giving them the option to decline permission for their child’s participation. Prior to administering the baseline assessment, the study was explained to students directly, and they also were given the option to not participate. In the event that a student declined participation, they still received the lessons as part of the regular classroom instruction but could return the assessment form blank. Students received no incentive for study participation. Data were collected using student ID numbers.

Data Analysis

Scale scores were calculated as the item mean. General Linear Model (GLM) analyses were used for both the teacher and student data, with T2 scores on the MHL-related scales the dependent variables, baseline T1 scores as control variables, and Group (program, vs. no program) as a fixed effect, categorical independent variable. In addition, paired (T1, T2) t-tests were conducted to evaluate within-group change to determine whether between-group differences at T2 reflected improvements in the treatment group vs. worsening in the control group (or both). Cases with missing data were excluded using listwise deletion. For the fidelity ratings, each teacher’s average score for each of the five scales was calculated. The teacher-level averages were then further combined across participating teachers to obtain an overall mean fidelity rating by scale. Scale mean fidelity scores of 1.4 and higher (i.e. 70% of the total maximum mean score of 2) were considered to be satisfactory implementation fidelity, with scores higher than 1.7 (i.e. 85% of the total maximum possible) considered to be high implementation fidelity.

Results

Sample Demographics

A total of N = 80 teachers and N = 3,152 students were contacted for study recruitment. All contacted teachers (100%) provided consent and complete T1/T2 data, resulting in a sample size of 80 teachers (intervention: n = 40; control: n = 40) for analysis. N = 3,000 students (95% of those contacted) provided consent; 1,518 out of 1,698 students in intervention schools (89%) and 1,021 out of 1,302 students in control school (78%) provided complete T1/T2 data for analysis. Teachers were 90% female, with a median age of 36 and median of 12 years of teaching experience. Most had a bachelors (84%) or masters (15%) degree, with one teacher (1%) having a community college degree. Student participants were equally split between 8^th (50%) and 9^th grade (50%), 52% female, with a median age of 15.

Teacher results

Scale scores by intervention group and timepoint are reported in Table 1. At baseline, most notable were relatively low levels of willingness to interact with people with mental illness, and relatively high levels of beliefs about mental illness as incurable and about individuals with mental illness as dangerous. All but one variable (MHLS Stigma) showed significant within-group improvement in the intervention group. MHLS Recognition scores for teachers in the control group significantly worsened from T1 to T2 (2.46 to 2.18, p<.05); no other changes were significant in the control condition.

Table 1.

T1 and T2 mean (SD) scale scores by intervention group for Vietnamese sample

Scale	Timepoint	Treatment	Control
Teacher - MHLS Recognition	T1	2.51 (.50)	2.46 (.55)
	T2	2.89 (.48)***	2.18 (.49)*
Teacher – MHLS Help-seeking Self-efficacy	T1	2.78 (.47)	2.66 (.73)
	T2	3.19 (.86)**	2.51 (.47)
Teacher – MHLS Stigma	T1	1.08 (.43)	1.11 (.52)
	T2	1.05 (.57)	1.31 (.43)
Teacher – MHLS Willingness to Interact	T1	1.79 (.58)	1.46 (.60)
	T2	2.71 (.66)****	1.51 (.55)
Teacher – BMI Dangerousness	T1	2.85 (.63)	2.66 (.70)
	T2	1.82 (.82)****	2.90 (.86)
Teacher – BMI Poor skills	T1	2.04 (.87)	2.10 (.68)
	T2	1.43 (1.06)**	2.22 (.79)
Teacher – BMI Incurable	T1	2.60 (.74)	2.54 (.85)
	T2	2.25 (.83)*	2.48 (.81)
Student – MHL Knowledge	T1	0.46 (.15)	0.49 (.10)
	T2	0.52 (.18)****	0.50 (.11)
Student – Stigma	T1	1.74 (.61)	1.77 (.62)
	T2	1.65 (.54)****	1.74 (.61)

Notes.

¹=Range of the MHLS scales is 0 (“strongly disagree”) to 4 (“strongly agree”); range of BMI scales is 0 (“completely disagree”) to 4 (“completely agree”). Student – MHL Knowledge is proportion correct. Range of the Student – Stigma scale is 0 (“strongly disagree”) to 4 (“strongly agree”).

^*= p<.05

^**= p<.01

^***= p<.001

^****= p<.0001 for within-group T1 to T2 comparison

In the GLM models evaluating the effect of the training, all four of the dependent variables from the MHLS for the teachers showed significant group effects adjusted for T1 scores. In all instances, results favored the experimental group, who reported higher levels of recognition, self-efficacy, and willingness to interact, and lower levels of stigma than the control group at T2 (see Table 2); effect sizes were moderate to large (e.g., R²=.46 for willingness to interact with someone with mental illness). For the Beliefs about Mental Illness (BMI) scale, two of the three subscales showed significant Group effects, with the BMI Incurable subscale being the one scale not showing significant group effects (see Table 2). Again, the significant effects favored the experimental group, with moderate to large effect sizes (e.g., R²=.30 for BMI Dangerous).

Table 2.

Results of inferential analyses for Vietnamese sample

Dependent variable	F test, for effect of Group	ES1	Adjusted2 T2 Mean (SD) Tx	Adjusted2 T2 Mean (SD) Cntl
Teachers - MHLS Recognition	F(1,77)=42.55****	R2=.36	2.89 (.47)	2.18 (.49)
Teacher – MHLS Help-seeking Self-efficacy	F(1,77)= 18.35****	R2=.19	3.18 (.85)	2.52 (.45)
Teacher – MHLS Stigma	F(1,77)=5.46*	R2=.07	1.05 (.55)	1.31 (.44)
Teacher – MHLS Willingness to Interact	F(1,77)=65.48****	R2=.46	2.67 (.64)	1.55 (.55)
Teacher – BMI Dangerousness	F(1,77)=33.05****	R2=.30	1.81 (.82)	2.90 (.85)
Teacher – BMI Poor skills	F(1,77)=14.34***	R2=.16	1.44 (1.00)	2.21 (.78)
Teacher – BMI Incurable	F(1,77)=2.04	R2=.03	2.24 (.78)	2.49 (.76)
Student – MHL Knowledge	F(1,2536)=6.98**	R2<.01	0.52 (.18)	0.50 (.11)
Student – Stigma	F(1,2536)=12.72***	R2<.01	1.65 (.54)	1.74 (.61)

Notes:

¹=effect size is partial eta-squared, controlling for T1 dependent variable.

²Adjusted for T1 score.

^*=p<.05

^**=p<.01

^***=p<.001

^****=p<.0001

Teacher Classroom Implementation Fidelity

Observer ratings, on the 0 to 2 scale, indicated high teacher Quality of Implementation (M = 1.90, SD = 0.14), Correct Use of Materials (M = 1.83, SD = 0.24), Student Participation (M = 1.79, SD = 0.25), and Appropriate Content (M = 1.75, SD = 0.25). Ratings were satisfactory but somewhat lower for Correct Process (M = 1.48, SD = 0.26).

Student results

Student baseline scores indicated low mental health knowledge (slightly less than half of knowledge items were answered correctly) and some evidence of stigma attitudes at baseline (Table 1). In the GLM analyses, students who received The Guide-VN program training showed significantly higher levels of mental health knowledge (adjusted for baseline levels) and lower levels of stigma than the control group at Time 2 (Table 2). Effect sizes for both dependent variables were small, however. No significant within-group changes were observed among control students, whereas students receiving the intervention showed small but statistically significant improvements in both Knowledge and Stigma (see Table 1).

STUDY 2 – CAMBODIA EVALUATION

Methods

Setting

Similar to Vietnam, Cambodia is Southeast Asian lower-middle-income country, with the two countries having close cultural and historical ties. However, Cambodia is substantially smaller and less developed than Vietnam, with a population of roughly 16.5 million (Central Intelligence Agency, 2019) and a human development ranking of 146^th (United Nations Development Programme, 2019). School attendance is lower than in Vietnam, with an estimated 50% of children attending lower secondary school (grades 7-9), and 26% attending upper secondary school (grades 10-12) (UNICEF, 2019). Previous studies suggest a high prevalence of mental health-related problems among both children and adults (Jegannathan, Kullgren, & Deva, 2015; Schunert et al., 2012); in a nationally representative study, prevalence of anxiety, depression, and post-traumatic stress disorder among adults was 27%, 17%, and 8%, respectively (Seponski, Lahar, Khann, Kao, & Schunert, 2019). In that same nationally representative study, 8% of households with children reported that one or more children were having school problems, 12% reported child aggressive behavior, and 5% reported children were experiencing other mental health-related problems (Schunert et al., 2012).

To the best of our knowledge, there have been no prior studies of MHL training in Cambodia, although results from the Cambodian sub-study have been previously reported (Phoeun et al., 2019). Anecdotally, low MHL and an over-reliance on medication have been identified as two major challenges to reducing the burden of mental health disorders in Cambodia (TPO Cambodia, 2015). The origins of Khmer explanatory models for mental illness are found in Buddhist and Hindu religious teachings, beliefs in spirits and astrology, and emphasis on the mind-body connection. Help-seekers often look to traditional methods of healing first, turning to the medical system only after traditional methods are unsuccessful (Schunert et al., 2012).

Study Design and Sample

The Cambodian study was carried out in one school in the capital city Phnom Penh, which was selected due to the lead researcher’s ability to obtain permission from the director. The study was designed to be as similar as possible to Study 1, although the use of a single school introduced differences that resulted in a quasi-experimental, controlled pre-post design. To power the study to detect differences in outcomes at the “teacher” level, both teaching and non-teaching school staff such as nurses, office secretaries, information technology staff, etc., were included. All participants at the “teacher” level (collectively referred to from this point as “staff”) were individually randomized to intervention and control conditions. In contrast to Vietnam where all teachers who participated in The Guide training subsequently delivered the classroom curriculum, in Cambodia only four teachers in the intervention group were selected to teach the student curriculum in their classrooms, as these teachers taught in the target grades in subjects (Khmer language, English, Library) that were most compatible with the content of The Guide. Each teacher taught the curriculum in one class. To avoid contamination, students in 7^th and 11^th grades were assigned to the classroom curriculum and students in 8^th and 10^th grades were assigned to the control group; these assignments were purposive so that the intervention and control groups were counterbalanced for age and developmental level. Students in grades 9 and 12 were not included in this study as they were preparing for examinations. This study was approved by the Cambodian National Ethics Committee for Health Research, National Institute of Public Health (U.S. FWA #00029325).

MHL Intervention

The Cambodian MHL program and assessment questionnaires were based on Khmer translations of the English-language versions of the materials used in Vietnam. These included The Guide as well as the various questionnaires used in the Vietnam study (i.e., the MHLS; the BMI; the student Mental Health Knowledge and Attitude Test), which had not been previously used in Cambodia. Initial translation of The Guide and the assessment materials to Khmer was carried out by the lead researcher in Cambodia (BP), who is a bilingual (English/Khmer) clinical psychologist with more than ten years of experience teaching and practicing in mental health. The translated materials were then distributed to a team of bilingual Cambodian psychologists at the EMDR Cambodia Association for review and feedback. Aside from using Khmer words, idioms, and case studies, no substantive adaptations were made to The Guide or the teacher training materials. The assessment instruments were piloted with 10 staff and 8 students before beginning data collection. As The Guide’s original developmental targets were inclusive of grades 7-11, no developmental modifications were deemed necessary.

Staff measures

Assessment measures were selected to be consistent across the Vietnam and the Cambodian studies, the MHLS and BMI. Internal consistency of MHLS subscales was satisfactory: Recognition (of mental health disorders) α = .60 (T1) and α = .66 (T2); Help-seeking Self-efficacy α = .63 (T1) and α = .74 (T2); Stigma α = .60 (T1) and α = .66 (T2); Willingness to Interact with people with mental illness α = .62 (T1) and α = .79 (T2). On the BMI the Dangerousness subscale showed good internal consistency (α = .77 and α = .84 at T1 and T2, respectively); the other two subscales were not satisfactory: Poor Interpersonal and Social Skills α = .53 (T1) and α = .54 (T2); and Incurability α = .54 (T1) and α = .54 (T2).

Student measure

The same instrument was used to assess student MHL Knowledge and Stigma as in Vietnam, although for the Stigma items the Cambodian team reverted to the original 7-point response scale (from 0 “strongly disagree” to 6 “strongly agree”) to provide more flexibility in student responses. Internal consistency for the student Knowledge scale was α = .50 (T1) and α = .62 (T2), and for the Stigma scale α = .47 (T1) and α = .56 (T2).

Program Implementation Fidelity Assessment

The same observational fidelity assessment was used in both studies. In Cambodia, all classroom lessons were observed by one of two clinical psychology graduate students. As in Vietnam, fidelity observers attended The Guide training, received an additional half-day training specifically for fidelity rating training, and were rotated through the different classrooms over the course of the study.

Procedures

After obtaining permission from the director, BP approached school staff for recruitment and informed consent. Staff (N = 73) were randomly assigned to participate in The Guide training (n = 36), or not (n = 37). Staff in the control and intervention groups completed their T1 assessments on the same schedule, prior to the beginning of the teacher training workshop. All staff training was led by BP. Staff received the equivalent of $5 USD for completing the baseline- and follow-up assessments. Staff in the intervention group received an additional $20 for participation in the 2-day training, and the four teachers who delivered the classroom-based curriculum received a further $10 compensation for their additional classroom contribution.

Students in 7^th and 11^th grade classrooms received The Guide curriculum delivered by one the four trained teachers during regular instructional time, over six weekly sessions (approximately 1 to 1.5 hours per week). Students in the 8^th and 10^th grades received standard instruction, but completed assessments on the same schedule as the intervention group. As in Vietnam, informed consent at the student level was obtained through a passive consent process with a letter sent home to the parents and students given the option to opt out of data collection. Follow-up data collection for both teachers and students took place the week after completion of the full classroom delivery of the MHL curriculum (approximately eight weeks after baseline).

Analysis

Data analysis followed the same procedures as in the Vietnam study. In order that the evaluation of the Cambodian version of The Guide assess program effects as it was generally implemented, the four intervention teachers purposively selected for additional training and classroom implementation were excluded from the analyses. Cases with missing data were excluded using listwise deletion.

Results

Sample Demographics

A total of N = 100 staff were contacted and 80% consented to participate in the study, with 73 (intervention: n = 36; control: n = 37) subsequently returning the baseline assessment. Of those, 67 provided follow-up data, resulting in a sample size of 63 (intervention: n = 30, 83%; control: n = 33; 89%) for analysis after also removing the four implementing teachers. Reasons for staff loss to follow up were unrelated to the project (e.g., change of employment). Staff participants were 78% women, with a median age of 27 and a median of four years of professional experience. More than half had a bachelor’s degree (53%), 3% had a master’s degree, with the remaining reporting either a high school (24%) or junior high school (16%) education. Two thirds were teaching staff (65% of the full sample, 63% after removing the four implementing teachers from analysis), with others in administrative or other non-teaching roles.

Of N = 307 students (98% of those contacted) who provided consent, 302 students provided T1 data (intervention: n = 158; control: n = 144), and 301 students provided T2 data (intervention: n = 157; control: n = 144). However, in some cases students provided inconsistent identifier data that precluded matching of their records, resulting in a final sample size of 275 students (intervention: n = 145, 92%; control: n = 130, 90%) with complete data. Students were majority female (62%), median age of 16, and equally distributed across the participating grades (26% in 7^th, 25% in 8^th, 23% in 10^th, and 27% in 11^th).

Staff results

Scale scores for the Cambodian staff participants are reported by intervention group and timepoint in Table 3. Baseline scores showed similar patterns as in the Vietnam sample, with moderate scores for most scales but notably higher (i.e., more negative) perceptions of the dangerousness and incurability of mental disorders. T-tests show significant within-group improvements in the intervention group for all BMI subscales as well as MHLS Willingness to Interact scale; for the remaining MHLS subscales, non-statistically significant improvements were observed. The control group showed no significant within-group changes.

Table 3.

T1 and T2 mean (SD) scale scores by intervention group for Cambodian sample

Scale	Timepoint	Treatment	Control
Teacher - MHLS Recognition	T1	2.69 (.46)	2.41 (.43)
	T2	2.68 (.50)	2.42 (.40)
Teacher - MHLS Help-seeking Self-efficacy	T1	2.63 (.68)	2.50 (.61)
	T2	2.69 (.80)	2.32 (.67)
Teacher – MHLS Stigma	T1	1 .71 (.40)	1.84 (.48)
	T2	1 .54 (.62)	1.86 (.62)
Teacher – MHLS Willingness to Interact	T1	2.00 (.35)	1.84 (.49)
	T2	2.45 (.39)****	1.81 (.48)
Teacher – BMI Dangerousness	T1	2.81 (.69)	2.90 (.79)
	T2	2.08 (1.00)***	2.93 (.73)
Teacher – BMI Poor skills	T1	1.89 (.74)	2.36 (.71)
	T2	1.54 (.70)*	2.20 (.68)
Teacher – BMI Incurable	T1	2.98 (.74)	2.87 (.72)
	T2	2.60 (.64)**	2.91 (.60)
Student – MHL Knowledge	T1	0.57 (.13)	0.56 (.13)
	T2	0.61 (.14)**	0.54 (.12)
Student – Stigma	T1	4.05 (.69)	4.10 (.86)
	T2	3.39 (.84)****	4.02 (.77)

Notes:

¹=Range of the MHLS scales is 0 (“strongly disagree”) to 4 (“strongly agree”); range of BMI scales is 0 (“completely disagree”) to 4 (“completely agree”). Student – MHL Knowledge is proportion correct. Range of the Student – Stigma scale is 0 (“strongly disagree”) to 6 (“strongly agree”); Means are adjusted for T1 scores.

^*=p<.05

^**=p<.01

^***=p<.001

^****=p<.0001

In the GLM models assessing the effects of the intervention adjusting for T1 scores, only one of the four dependent variables from the MHLS, Willingness to Interact with people with mental illness, showed a significant Group effect, with scores favoring the experimental group F(1,60)=29.59, p<.0001. The other three MHLS scales also favored the experimental group, but failed to reach statistical significance: Recognition of mental health disorders F(1,60)=2.70, p>.10, Help-seeking Self-efficacy F(1,60)=3.48, p<.10, Stigma F(1,60)=3.59, p<.10 (see Table 4). All three BMI subscales showed significant Group effects favoring the experimental group with lower levels of all variables at follow up: BMI Dangerousness F(1,60)=15.30, p<.001, BMI Poor skills F(1,60)=7.08, p<0.01, and BMI Incurability of mental illness F(1,60)=6.37, p<0.05 (Table 4).

Table 4.

Results of inferential analyses for Cambodian sample

Dependent variable	F test, for effect of Group	ES1	Adjusted2 Mean (SD) Tx	Adjusted2 Mean (SD) Cntl
Teachers - MHLS Recognition	F(1,60)=2.70	R2=.04	2.65 (.50)	2.45 (.38)
Teacher – MHLS Help-seeking Self-efficacy	F(1,60)=3.48	R2=.05	2.67 (.74)	2.34 (.64)
Teacher – MHLS Stigma	F(1,60)=3.59	R2=.06	1.55 (.62)	1.85 (.61)
Teacher – MHLS Willingness to Interact	F(1,60)=29.59****	R2=.33	2.44 (.37)	1.83 (.48)
Teacher – BMI Dangerous	F(1,60)=15.30***	R2=.20	2.10 (.92)	2.91 (.71)
Teacher – BMI Poor skills	F(1,60)=7.08**	R2=.11	1.67 (.60)	2.08 (.56)
Teacher – BMI Incurable	F(1,60)=6.37*	R2=.10	2.58 (.54)	2.93 (.56)
Student – MHL Knowledge	F(1,272)=23.09****	R2=.08	0.61 (.13)	0.54 (.12)
Student – Stigma	F(1,272)=41.53****	R2=.13	3.40 (.82)	4.01 (.77)

Notes:

¹=effect size is partial eta-squared, controlling for T1 dependent variable.

²Adjusted for T1 score.

^*=p<.05

^**=p<.01

^***=p<.001

^****=p<.0001

Teacher Classroom Implementation Fidelity

Observer ratings, on the 0 to 2 scale, indicated satisfactory fidelity scores for Appropriate Content (M = 1.67, SD = .36), Correct Process (M = 1.60, SD = .31), Quality of Implementation (M = 1.58, SD = .36), and Correct Use of Materials (M = 1.56, SD = .35). Fidelity ratings for Student Participation (M = 1.35, SD = .31) were slightly below the satisfactory cutoff of 1.40.

Student results

Table 3 reports the baseline and follow-up scores by treatment condition. Students in both groups reported low baseline mental health knowledge and high baseline stigma. Within-group t-tests showed small but significant improvements in both Knowledge and Stigma among intervention students, with no significant changes among control students (Table 3). Adjusting for T1 scores, the GLM models indicated significant Group effects, with both higher Knowledge, F(1,272)=23.09, p<.0001, and lower Stigma F(1,272)=41.53, p<0.0001 in the intervention group at T2 relative to the control group (see Table 4).

DISCUSSION

This paper reports findings from two parallel evaluations examining the effects of the MHL training program The Guide, (Kutcher & Wei, 2017), on school staff and student mental health knowledge and attitudes in Vietnam and Cambodia. Findings suggest that with limited adaptations to the original curriculum, teacher-led classroom instruction using a train-the-trainer model can result in measurable improvements in mental health knowledge and attitudes of both teachers and students. This illustrates the potential of an approach focused on embedding an inexpensive and teacher-optimized MHL training program into existing classroom instruction. In addition to its promise for reaching a large number of staff and students and potentially improving teacher support for students with mental health difficulties, this type of program is relatively easy to integrate into school settings where “life skills” training is popular (Dang et al., 2017).

The present study builds on prior work adapting and testing The Guide for two countries in East Africa, which resulted in “The African Guide” (Kutcher et al., 2015, 2016, 2017). Like that project, we sought to extend study of The Guide beyond a single country in Southeast Asia, and expand methodological rigor through use of a randomized control group. This design also helps to address some methodological limitations identified in prior systematic reviews of teacher mental health training programs (Anderson et al., 2018; Yamaguchi et al., 2019).

Across both sub-studies, the largest teacher effect was for reported willingness to interact with people with mental illness, with lesser effects on stigma and related constructs. This is somewhat surprising, as one might expect that changes in stigma attitudes would be a mechanism for changes in behavior (in this case, to interact with people with mental illness), whereas our study results might be seen as suggesting potential for changes in behavior independent of enduring negative beliefs. Because we did not assess teacher or staff behavior, it is unclear the extent to which reporting of willingness to interact would reflect actual behavior versus a social desirability report bias. However, within all of the subscales assessing stigma, the largest effect sizes were in regards to attitudes about the dangerousness of people with mental illness. It is plausible that increased knowledge about the medical origin of mental illness, paired with decreased views of the dangerousness of those experiencing poor mental health, are effective mechanisms to increase willingness to engage with such individuals even when beliefs about poor social skills and the incurability of their disorders are more resistant to change. This impact on willingness to engage is a particularly critical change for school staff, given their potential role in helping to identify, refer, and provide services to students in need of support.

Although the findings from this study are promising, effect sizes for significant effects were variable, ranging from R²<.01 to R²=.36. Future research focusing on understanding the reasons underlying this variability will be important, and can help shape further adjustment for the program. For instance, among the seven outcomes for the Vietnamese teachers, the BMI Incurable scale was the only one that did not show significant program effects. Understanding why this is the case, and how to best target this domain will be important in order more fully develop teachers’ mental health literacy.

What will be most critical will be understanding why effects for students, particularly in Vietnam, were relatively small. There are several potential reasons for this. First, given a heavy focus on examination-based educational advancement in both countries, students face substantial academic pressures, may have viewed these lessons as less important than their other work, and therefore expended less energy in The Guide classes. Second, at an implementation level, teachers received no ongoing support for delivery of the curriculum in their classrooms. One strength of The Guide curriculum is its detailed instructional materials and lesson plans that support off-the-shelf use. However, previous research suggests that school-based implementers often require additional support beyond brief training in order to maintain fidelity to an intervention (Owens et al., 2014; Wandersman, Chien, & Katz, 2012). Fidelity ratings indicating some challenges in implementation in both countries suggest that ongoing support may have potential value. Given consistent positive findings, future research may benefit from a hybrid implementation-effectiveness design in which the implementation and support strategy, rather than just The Guide itself, is evaluated. For example, a study involving randomization to either “off-the-shelf” vs. participation in a peer supervision group with more systematic assessment of in-class fidelity could be useful.

Effect sizes in Cambodia for staff were smaller than those observed in Vietnam, with some failing to reach statistical significance. This could be due to a number of reasons, ranging from cultural relevance to implementation and study design features. Although minimal adaptations were required in either country, it is possible that more extensive adaptation for the specific Cambodian context would improve results. For example, previous research has documented culturally distinct mental health syndrome presentations in Cambodia (Hinton, Pollack, Pich, Fama, & Barlow, 2005; Hinton, Um, & Ba, 2001), not incorporated into The Guide. Further, what is considered to fall under the domain of “mental health” varies across cultures (Bass, Bolton, & Murray, 2007). It thus is possible that respondents’ conceptualization of what constitutes mental illness is broader than what is addressed in The Guide, varying somewhat across Vietnam and Cambodia. The Guide does not directly address stigma related to, for instance, Cambodian “Khyal wind attacks” (Hinton, Pich, Marques, Nickerson, & Pollack, 2010), potentially resulting in less overall change among the Cambodian sample.

It is also possible that this smaller impact for the Cambodian sample may have been due to the revised staff training and delivery model used in Cambodia relative to the original version of The Guide. Whereas in Vietnam all teachers in the intervention group received the third day of training and subsequently taught The Guide-VN in their classes, in Cambodia most staff did not subsequently deliver the curriculum to students (and therefore did not receive the 3^rd day of training on classroom implementation). Although this final day of training focuses on methods to teach The Guide to students rather than on new MHL content, both learning how to teach and the act of teaching itself likely help to consolidate teachers’ own learning (Koh, Lee, & Lim, 2018). In this study we removed from the analyses the four teachers who implemented the classroom curriculum in order that the sample remain homogeneous in regards to the intervention they received. An earlier examination of the full Cambodian staff sample found that with inclusion of these four teachers all MHLS variables showing significant Group effects, but with only slightly higher effect sizes (Phoeun et al., 2019); it is not clear, however, what aspect of their inclusion (e.g., increased sample size increasing statistical power; higher education among the teachers; more training and more experience implementing) is responsible for the difference . Future research that more clearly details the role of subsequent curriculum delivery in helping produce sustained change among implementers will be helpful to guide decisions around which school personnel should receive training. Likewise, the in-class curriculum was delivered over six weekly 1-1.5-hour lessons in Cambodia over an eight week period, versus twice weekly 45-minute lessons in Vietnam over a five week period. Although the same information was taught in both countries, expanding the content over two class periods may have allowed for more in-depth coverage and understanding of the material. In any case, this smaller impact among staff in Cambodia suggests a need for additional content review and a larger (e.g., involving more than one school) study to more fully evaluate The Guide in Cambodia.

Although the Vietnam and Cambodia sub-studies used similar designs, there were important differences in the designs that may have impacted on results. The Vietnam study only included teachers in the sample whereas in Cambodia, non-teaching school staff, who may have been less motivated than teachers to learn and consider the material, also were trained and evaluated. The Vietnam sub-study used school-level randomization to avoid contamination across conditions whereas the Cambodian sub-study involved allocation at the grade level. Thus, Cambodian control group students may have had interactions with teachers, non-teaching staff, or students who had received the training. Cambodian non-teaching staff such as those participating in the study likely have less contact with students as compared to teachers, but their contact is not restricted to students in one condition. Arguing against this interpretation, however, is the lack of significant within-group changes in the Cambodian control group and the slightly larger effect sizes for student outcomes in Cambodia relative to Vietnam.

Strengths and Limitations

Strengths of the present study include evaluation of the program in two neighboring Southeast Asian countries, and use of a randomized experimental design. Although the Cambodian study was conducted in only one school, this pilot evaluation does strengthen support for the broad value of the program which, although initially adapted for Vietnam, holds potential to be feasible and effective throughout the region. One study limitation is that the baseline assessment occurred after randomization and change among participants who received the program could have been influenced by expectation effects; arguing against this interpretation, however, baseline scores for the program and control groups were comparable.

An additional limitation is that aside from the previous validation of the MHLS for Vietnam, the other measures were not independently validated for these two countries, and in some cases the scales used to assess knowledge and attitudes had low internal consistency, particularly at baseline. There are several possible explanations. First, a number of the subscales used in the present study have a relatively small number of items, which psychometrically can reduce internal consistency, regardless of the data (Rust & Golombok, 2009). In addition, it has been recognized for several decades (e.g., Cleary, 1981) that scales assessing life events and similar constructs tend to have low internal consistencies. Stigma and related constructs may to some extent be similar to life events, in that they likely at least in part originate from direct or indirect interactions or “life events” with individuals with mental illness. Regardless of the underlying cause, it is important to note that internal consistency reflects the degree to which a measure or subscale reflects a single construct, rather than validity (Rust & Golombok, 2009).

Another potential concern is that the instrument used to assess outcomes, particularly at the student level, is closely aligned with the curriculum as it was developed specifically as a companion tool, and thus the broader, more fundamental impact of the program is unclear. This is of less concern at the teacher/staff level, where the instruments used were developed separately from The Guide. Future research should involve supplemental assessment at the student level.

Further, although fidelity observers received substantial training in both The Guide and the use of the observer rating form, inter-rater reliability between observers was not assessed. While the within-teacher impact of variation in observer rating was minimized by deriving an average fidelity rating across multiple raters, it is possible that differences in fidelity ratings between Vietnam and Cambodia could to some extent reflect rater differences rather than objective differences in implementation quality.

Finally, the assessment time frame for these studies was relatively short (approximately eight weeks from pre- to post-test), and we were unable to conduct a longer-term follow-up to evaluate sustained programmatic impacts on knowledge and attitudes. Most importantly, we did not assess behavioral outcomes such as self-care and help-seeking. Knowledge and attitudes generally are conceptualized as intermediate outcomes leading to the ultimate goals of more positive behavior (e.g., towards individuals with mental health challenges) and improved skills (e.g., for mental health help-seeking), all leading to improved functioning. A recent pilot evaluation of a MHL curriculum in Nicaragua did evaluate additional outcomes, finding 12-week improvements in behavior, including adaptive coping, better lifestyle choices, and perceived stress among intervention students relative to controls (Ravindran et al., 2018). These findings are encouraging and support future, more extensive evaluation of the Vietnamese and Cambodian versions to determine if similar effects are achieved.

Conclusions

Integration of mental and behavioral health supports in school settings is a pragmatic approach to address the substantial needs of youth in limited resource settings and is increasingly a focus in LMIC (Wei & Kutcher, 2012). These approaches may increasingly rely on task sharing models that engage teachers rather than mental health professionals in delivery of school-wide mental health programming, increasing accessibility of services and reducing stigma associated with seeking mental health care through health facilities (Dang et al., 2017). However, in areas with low levels of MHL, there is a need to strengthen staff and student understanding of mental illness and mental health to reduce stigma and, ultimate, increase help-seeking. The current studies show consistently positive, although varying in magnitude, improvements in knowledge and attitudes among teachers and students following implementation of a classroom-based MHL program in Cambodia and Vietnam.