Evaluation of a School-Based Program Aimed at Preventing Depressive Symptoms in Adolescents

Abstract

The aim of this pilot study was to evaluate the implementation of a universal school-based cognitive behavioral program whose target is to prevent depressive symptoms in adolescents. The study had a quasi-experimental design with pretest, posttest, and a 1-year follow-up and provides an illustrative calculation for the implementation costs of the intervention. Sixty-two students (aged 14) and seven tutors participated. A majority of the students and all of the tutors were satisfied with the intervention. The students, both females and males, rated their depressed symptoms as significantly lower after the course; and for the females, this was maintained 1-year postintervention. The implementation costs for the initial 2 years were about US$300 per student. Positive effects of a universal school-based cognitive behavioral intervention aiming at preventing depressive symptoms in adolescents were found, especially among females.

Introduction

School has been proposed as a good place for health-promoting interventions (DeSocio & Hootman, 2004). Cognitive behavioral programs are found to have positive effects in preventing depressive symptoms in adolescents (Corrieri et al., 2013). There is, however, a great need to evaluate effective interventions in “real-world” practice in order to investigate their sustainability and feasibility (O’Connell, Boat, & Warner, 2009). The purpose of this article is to describe the first 2 years of the implementation of a school-based prevention program, its effects on depressive symptoms, the costs involved, and the views of the students and tutors.

Background

Promoting well-being and preventing depressive symptoms in adolescence are important school health concerns. Depression in adolescence is a common health issue globally (Patel, Flisher, Hetrick, & McGorry, 2007), with a 1-year prevalence of more than 4% by the end of adolescence (Thapar, Collishaw, Pine, & Thapar, 2012). Preventing depressive symptoms means not only less individual suffering but also fewer societal costs (Kessler, 2012). We are most likely to find adolescents at school, and therefore school may be a good place to implement intervention programs targeting depression (Carnevale, 2013). However, it is essential that the programs be effective not only in study settings but also when implemented in schools and conducted by the regular school staff. The RE-AIM framework is a model used for assessing the public health impact of an intervention (Glasgow, Vogt, & Boles, 1999). It was developed to overlap the gap between interventions found to be effective in study settings but that are difficult to implement effectively in the world of busy, often understaffed community settings. The emphasis on developing clinically significant outcomes has produced a lot of intensive, expensive interventions that are demanding for both students and staff. Low-intensity interventions that are less efficacious but reach a large population may have a more pervasive impact (Glasgow et al., 1999). Unfortunately, there is a serious lack of real-world evaluation of the school-based programs concerning behavioral and emotional issues (see Fact Box 1; O’Connell et al., 2009; The Swedish Council on Health Technology Assessment, 2010).

Prevention programs are categorized as universal (designated for everyone in a population), selective (designated for risk groups), or indicated (designated for those who already have symptoms; Mrazek & Haggerty, 1994). Universal approaches may have advantages over indicated and selective interventions due to higher recruitment rates and less risk of stigmatizing (Corrieri et al., 2013). A cognitive behavioral intervention developed in the United States and used for selective and indicated prevention of depression—“Coping with Stress” (Clarke & Lewinsohn, 1995)—has been adopted to a Swedish setting and used here as a universal school-based intervention, called “Depression in Swedish Adolescents” or DISA (Treutiger & Lindberg, 2013). The theoretical orientation of the intervention is represented by the multifactorial model of depression (Lewinsohn, Hoberman, Teri, & Hautzinger, 1985) in which increased depression is presumed to be the result of multiple etiological elements, such as negative cognitions, stressful events, predisposing vulnerabilities, risk factors (e.g., being female, having a previous history of depression, having depressed parents), and immunities to depression (e.g., high self-esteem, coping skills, high frequency of pleasant events and activities). The DISA course comprises 10 structured group sessions of 1½ hr each, led by school nurses, school social workers, and teachers. Although the Coping with Stress course has been well evaluated in the United States (Clarke et al., 1995, 2001; Garber et al., 2009), only one evaluation has been carried out regarding DISA, showing slightly positive results (Treutiger & Lindberg, 2013). There is a need to investigate the reach, effectiveness, adoption, implementation, and maintenance (Glasgow et al., 1999) of the program.

Aim

The aim was to evaluate a universal school-based cognitive behavioral intervention whose target is to prevent depressive symptoms in adolescents. Specific research questions were as follows:

• Research Question 1: Is there a reduction of depressive symptoms after delivering the intervention, and can this be maintained at the one-year follow-up?

• Research Question 2: Is there a difference in improved depressive symptoms between boys and girls postintervention?

• Research Question 3: What are the students’ and tutors’ experiences of the intervention?

• Research Question 4: How much is the total implementation cost of the intervention?

Method

The study had a quasi-experimental design with pretest, posttest, and a 1-year follow-up (Shadish, Cook, & Campbell, 2002) and provides an illustrative calculation of the implementation costs of the intervention. This study was approved by the Regional Ethical Review Board in Lund (2012/462). Parents, adolescents, and tutors were provided with written and oral information about the study and its voluntary nature. The school principal, parents, and participants gave their written informed consent.

Participants

A power calculation was made a priori with mean change in Center for Epidemiological Studies Depression Scale (CES-D) as the primary outcome. A mean difference of 2.0 was regarded as clinically relevant and for 80% power. The power calculation showed that a total sample of 49 participants were needed (standard deviation [SD]: 5.0; α = .05).

The study was performed in southern Sweden, in a municipality with 2,700 inhabitants and with only one school with 365 students. During the academic years 2010–2011 and 2011–2012, all students in Grade 8 (14 years old) were offered a voluntary psychoeducational course in the regular school context. Of the 68 students, 62 accepted the offer and participated. Six students (two females and four males) who did not participate in the intervention instead took part in the ordinary school schedule, which was physical education in all cases. The tutors were school social workers (n = 3), teachers (n = 3), and a school nurse (n = 1), all of whom had undergone 3 days of training and 1 day of supervision in the method. Every course had two tutors, and the groups consisted of 10–18 students. There were five groups in total: three with girls only and two with boys only.

Measurement

The students answered the questionnaires in the classroom, wrote their comments with paper and pencil, and were instructed not to look at each other’s answers. The tutors were available to offer support if there were problems understanding the questionnaires. The students were asked to complete the CES-D (Radloff, 1977) in the first and the last sessions and again after 12 months. The CES-D is a self-reported measure of the frequency of 20 depressive symptoms. Each item is scored for frequency over the past week, from 0 (rarely or never) to 3 (most or all of the time), with a total possible score range between 0 and 60. The CES-D has been found to be valid and reliable in adolescents (Roberts, Lewinsohn, & Seeley, 1991). Scores equal to or above 16 are considered to indicate a high risk for depressive symptoms (Treutiger & Lindberg, 2013; Young, Mufson, & Davies, 2006).

At the end of the course, the students were encouraged to write comments about the course. The instructions were “To find out what you think about these sessions a little bit more carefully, we would like you to fill out this form. You can fill it out anonymously if you wish.” At the 12-month follow-up, questions about place of birth, family, and economic situation and the question “What did you think about the intervention?” were added. The tutors were also asked to write comments about the intervention at the end of the course, and the tutor instructor provided information about the costs of the intervention.

Procedure

The intervention was offered as a voluntary course in lieu of the regular physical education class. It was given once a week, 1½ hr each time, for a period of 10 weeks, on the same weekday and at the same time for every session. All courses were given in the morning and took place in the school conference room where fruit was served. The program is manual based, with a fixed curriculum for every session (see Fact Box 2). It focuses on training in cognitive-restructuring skills, techniques for modifying irrational and negative self-statements, and ideas to reduce the risk of developing depressive symptoms.

Data Analysis

Frequency analyses were performed to describe the following demographic and background characteristics of the students: gender, age, place of birth, parents living together, and perceived economic situation (The question was “How well off financially do you think your family is?” with six possible answers: Very well, rather well, average, not that well, not well at all, or don’t know). The Friedman test (chi-square) was used for measuring differences in the CES-D scores between preintervention, postintervention, and the 12-month follow-up. Wilcoxon signed-rank tests were used for post hoc analysis. A reduced p value (Bonferroni method) of .0167 was used to control for the risk of mass significance due to multiple comparisons (Field, 2009). The statistical analyses were performed using SPSS, version 17.0.

The written comments from the students and the tutors were analyzed using qualitative content analysis (Krippendorff, 2012). The software program Open Code 4.1 was used. The texts were read several times and then condensed, coded, and grouped into categories. During this process, the authors discussed the analysis to obtain consensus and enhance trustworthiness (Graneheim & Lundman, 2004). Two categories emerged from the students’ comments, and two categories from the tutors’ comments.

Results

Sixty-two students participated in the intervention in Grade 8 (their age thus being 14 years). All but three students were born in Sweden, and 10% of the parents were born abroad. This may be compared with the rest of Sweden in which 15% of residents are born abroad (Statistics Sweden, 2012). About 86% of the students had parents living together and 77% of the students believed that their families were in a good or very good economic situation. The school is located in an area with high mean income, according to Statistics Sweden (2012).

CES-D: Questionnaire About Depressive Symptoms

The students answered the CES-D questionnaire at the start and the end of the program and also after 1 year. At the 1-year follow-up, 55 (89%) students had answered the questionnaire on all three occasions. The CES-D scores of the females significantly changed (improved) from preintervention to postintervention and at the 12-month follow-up (p = .018). When Wilcoxon tests were used to follow-up on this finding, it appeared that the CES-D score did significantly change from the preintervention to the postintervention (p = .031) and from preintervention to 12-month follow-up (p = .041).

This long-term effect did not appear in the male population. Although the CES-D scores for the males did significantly change from preintervention to postintervention and at the 12-month follow-up (p = .008); when Wilcoxon tests were used to follow-up on this finding, it appeared that the CES-D score did significantly change from the preintervention to the postintervention (p = .027) but that this was not maintained from preintervention to the 12-month follow-up (p = .726; Table 1).

Table 1.

Effect of the Program on CES-D.

	Pretest	Posttest	12-Month Follow-Up	p Value, Chi-Square, df
Total				.001, A chi-square 14.510, df = 2
n	61	58	55
Median	8.50	5.00	7.00
Percentiles: 25–75	5.00–15.88	2.25–10.88	2.25–12.00
Mean	10.32	7.29	8.60
SD	7.24	6.51	7.79
Girls				.018, A, B chi-square 7.980, df = 2
n	32	31	29
Median	11.50	6.25	8.50
Percentiles: 25–75	7.75–17.00	3.75–12.75	4.75–12.25
Mean:	13.25	8.83	9.10
SD	7.89	6.62	6.37
Boys				.008, A, chi-square 9.648, df = 2
n	29	27	26
Median	7.00	3.00	5.00
Percentiles: 25–75	4.25–12.00	1.00–10.50	1.00–11.25
Mean	7.40	5.75	8.12
SD	5.19	6.14	9.10

Note. CES-D = Center for Epidemiological Studies Depression Scale; df = degrees of freedom; SD = standard deviation. A = significant differences between preintervention and postintervention; B = significant differences between pre-intervention and 12-month follow-up.

Student Experience

The students were asked to rate their experience (positive, neutral, or negative) with the course. In all, 64% of the students were positive, 9% were negative, and 27% were neutral. More females than males were positive but the difference was not significant. The students were also asked to answer an open question, “What did you think of the intervention?” Two categories emerged from the student comments: “achieving a new way of thinking” and “getting to know each other better.” The category “achieving a new way of thinking” was concerned with the learning outcomes of the program mainly about identifying one’s thoughts and changing negative thoughts to positive ones.

I have got a new way of thinking; I have learned how to change negative thoughts into positive ones. (Female)

It feels good to start the morning with a relaxed, stress-free lesson. It felt good to identify my thoughts, both the negative ones and the positive ones, in order to control them more easily. (Female)

The girls wrote more comments than the boys in response to the open question about what they thought about the DISA course. Most of them had positive comments, but some girls wrote that they did not think that the course gave them anything and some found it boring. The boys used fewer words in their comments but most of them were positive:

It was great to get tips about how to change negative thoughts fast! (Male)

Some boys wrote that the program was unnecessary and boring and did not help them.

“Getting to know each other better” seems to be an important outcome of the course. Although one girl wrote that she did not want to talk about her feelings with the other participants, many students wrote about the communication training as being fun and useful.

The most important thing for me is that we girls in Grade 8 became closer to each other and we got to know one another more deeply. (Female)

It was fun and I learned a lot, and it was good to get to know the others in the group. (Male)

Being a Tutor

Among the tutor comments, two categories emerged, “concerns about practical conditions” and “adapting the program to the group.” All tutors were positive about the program although they also described challenges and difficulties. Comments in the category “concerns about practical conditions” were that the course took a lot of time, but they were given this time by the school administration, which is a requirement for future courses. They found that DISA was needed for both girls and boys, and they found the group size to be important. One of the boys’ groups had 18 participants and they found this too big. Students who scored high (>16) on the CES-D questionnaire were offered individual follow-up with the school nurse. The course was offered as a voluntary class at school, though almost all the students in Grade 8 chose to take the course. They reflected on the importance of good chemistry between the tutors (there were two tutors in each course) and felt that it was good for new tutors to collaborate with someone with more experience. They also found that they were able to conduct the courses more smoothly as they became more experienced. Comments in the category “adapting the program to the group” were concerned with how they had included more games and more communication exercises, as well as a set of cards with pictures of bears expressing different feelings, in order to make the program more appropriate for the audience. They found that some students did not work seriously during the program; these students did not actively participate during the discussions and did not do their homework. The tutors found that the program used words and metaphors about connections between thoughts, emotions, and actions that were difficult for some students to grasp; therefore, they tried to adapt the course to the group.

Illustrative Calculation of the Implementation Costs

Introducing a new program entails allocating resources to train program tutors and to implement the program (Drummond, Sculpher, Torrance, O’Brien, & Stoddart, 2005). We have made an illustrative calculation of costs of the program implementation using the figures from the pilot study regarding time used for training program tutors, preparation, and conducting the 10-week course with five groups of students (see Appendix). In total, the cost was US$300 per student to implement the program (Table 2). Of this, as much as half the costs are related to training tutors, including time and fees. If these tutors repeat the program, the average cost per student will fall. In addition, in this particular setting the program was conducted within the current budget, and no extra staff were employed. Costs other than wages corresponded to about US$50 per student.

Table 2.

Cost and Time Calculation.

Time	Calculation	Cost (US$) per Student
Training of the head instructor	US$2,320 (fee) / 62 students	37
Training head instructor and five tutors	US$34 (wage) × 208 hr/62 students	114
Conducting the intervention	US$34 (wage) × 4 hr/student	136
Materials (paper copies, candles, fruit)		13
Total		300

Discussion

The RE-AIM provides a framework for identifying programs that work in real-world environments (Glasgow et al., 1999). The results will be discussed through the lens of the following factors: reach, effectiveness, adoption, implementation, and maintenance.

Reach

The factor of reach refers to the proportion of the target population that participated in the intervention. All the students in Grade 8 (aged 14) were given the opportunity to participate in the intervention as a voluntary course during the regular physical education curriculum time. Only 6 of the 68 students declined to participate, 2 females and 4 males. Three of them stated that they would rather participate in the ordinary physical education class, and three students did not provide a reason. The school is located in a relatively high-income area, and, therefore, further studies are called for to investigate the reach in areas with a different socioeconomic situation.

Efficacy/Effectiveness

The students scored lower on measures of recent depressive symptoms at the 1-year follow-up, compared with preintervention. However, upon taking a closer look at the material, we found that this improvement was only significant for females. At postintervention, there were significantly less depressive symptoms among the males compared with baseline, but this was not maintained at the 1-year follow-up. However, for the females, the significant improvement was maintained from postintervention to the 1-year follow-up. At baseline, the CES-D scores were better compared with other studies in Sweden (Treutiger & Lindberg, 2013) and in the United States (Clarke et al., 1995). One may ask whether these results have any clinical relevance if the participants already felt good at the start. But the fact that the girls scored significantly better at postintervention and at the 1-year follow-up must be considered a positive effect for them. The only previously published study (Treutiger & Lindberg, 2013) of the DISA program in Sweden did not find significant changes in CES-D scores at the 1-year follow-up.

Overall, students rated the program highly. Only a few (9%) had a negative attitude to the course, whereas 64% were positive and 27% neutral. The majority of the students who left more detailed written comments on the course were positive about it. They found that the course had taught them a new way of thinking, and from the comments, it is possible to understand that they had learned the basics of cognitive behavioral therapy. They also wrote that the course made the group climate better. But one girl, in contrast, wrote that she did not want to talk about her feelings in front of the others in the group. Although the course was a voluntary one, the students had not chosen their group members. Making the group atmosphere comfortable is the responsibility of the tutor. The DISA course is based on the participants discussing their feelings quite openly and for this, a safe atmosphere in the group is needed.

Adoption

The factor adoption refers to the school’s acceptance of the intervention within the organization. This is a real-world study, meaning that the intervention was initiated by the school. The researchers were invited to evaluate the implementation, but the intervention would have been conducted even if this evaluation study had not been performed. The intervention was well accepted by the school, and the tutors found that they received sufficient support from the school administration, that is, enough time to plan and conduct the course. They pointed out that this support was necessary for future courses.

Implementation

All of the tutors were positive toward the course, but they also highlighted some difficulties. For example, there were some students who did not take the course seriously; that is, they did not participate actively in the discussions or did not do their homework. The tutors also noticed that the course materials exceeded the reading and comprehension level of some students. In order to adapt the program to the adolescents, they made some adjustments in the course by, for example, bringing in more games or change discussion topics instead of following the program manual strictly. Because of this, it is difficult to tell whether this study really evaluated the DISA program in general or only the version these tutors used. The main instructor (the one who taught the method to the tutors) claimed that it was not possible to exclude anything from the program but that there was some space to bring in extra exercises in order to make the core content understandable for the students. Questions about program fidelity must be taken into account in all evaluations, but this way of adapting the course might be acceptable, according to Breitenstein, Robbins, and Cowell (2012).

The cost of implementing the program for the initial 2 years was about US$300 per student and course. These costs will fall as the tutors lead more courses. Also, the time the tutors spend in planning the course will decrease with time as the tutors become more used to the program. It is also important to note that the school in this case did not employ any extra personnel in order to implement the DISA intervention but instead used the existing staff. Thus, the extra expenditures for the school may only include the cost of the head course instructor and printing costs for student materials (US$50 per student). There are few calculations of implementation costs in the literature and none about the DISA course. However, one study calculating costs for the Coping with Stress course in the United States found the implementation cost to be US$1632 per student, but this was an indicated program targeting adolescents at risk for future depression, and about 65% of the costs were used to identify at-risk teens and recruit families to the intervention (Lynch et al., 2005). In a pilot study such as this one, it is not possible to draw conclusions about possible Qualitative Adjusted Life Year benefits.

Maintenance

This factor refers to the maintenance of intervention effects in individuals and settings over time. The 1-year follow-up effects were discussed previously under the factor efficacy/effectiveness. The intervention is still being used at the school during the academic year 2013–2014, which would demonstrate certain sustainability.

Limitations

In this study, we learned that the instrument measuring depression, CES-D, was easy for the students to fill out, but in a full-scale study, it would be preferable to add more questions targeting anxiety and quality of life as well. In a full-scale study, a control group containing students not participating in the DISA course is also necessary. More geographic and income diversity is also needed in a full-scale study. The tutors’ claims that it was not always easy to strictly follow the manual suggest there is a need for detailed information in a full-scale study about how the intervention is actually conducted (i.e., program fidelity). The written comments from the students and tutors implied that it might be a good idea to conduct interviews in order to better understand their experiences. Qualitative interviews provide more information than short, written comments (Garmy, Berg & Clausson, 2014). It is not possible to draw any extensive conclusions from such a small study without a control group. However, since the literature contains few evaluations of school-based prevention programs, it is of utmost importance that studies of this sort are employed.

School Nursing Implications

Preventing depressive symptoms in adolescents is an important school nursing issue. With the growing recognition of interdisciplinary interventions, school nurses are in a good place to bring mental health promotion programs to schools (Cowell, 2013). In the cognitive behavioral intervention described in this article, school nurses are often tutors. School nurses need to critically evaluate the programs that are employed, either by conducting the evaluations themselves or by motivating others—for example, principals or politicians—to perform them. Unfortunately, there is a great shortage of evaluations today, which may mean that inefficient programs will continue to be used, and efficient programs will remain unknown and unused. We have to perform evaluations and publish them so that we can learn from each other and thereby provide the best care for our children and adolescents at school.

Conclusion

A majority of the students and all of the tutors were satisfied with the cognitive behavioral intervention preventing depressive symptoms. The students, both females and males, scored lower on measures of recent depressive symptoms after the course; and for the females, this was maintained 1-year postintervention. Many students said they had learned a new way of thinking. The tutors found that the core mission of the course was to give the students tools for the future. The tutors highlighted the advantages of offering the course on a voluntary basis. They also highlighted practical issues, such as the course being quite time consuming, and emphasized the importance of having full support from the school administration when implementing this sort of program.

Author Biographies

Pernilla Garmy, MMS, RN, is a school nurse and a PhD student at Kristianstad University and Lund University, Sweden.

Ulf Jakobsson, PhD, RN, is an associate professor at the Center for Primary Health Care Research, Lund University, Sweden.

Katarina Steen Carlsson, PhD, is a health economist and researcher at Lund University, Sweden.

Agneta Berg, PhD, RNT, is a professor of nursing at Kristianstad University and University West, Sweden.

Eva K. Clausson, PhD, RNT, is a senior lecturer in nursing at Kristianstad University, Sweden.

Appendix

Illustrative Calculation of the Implementation Costs

Introducing a new program implies a need to allocate resources to train program tutors and to implement the program. From a decision-making perspective, we have made an illustrative calculation of the costs of the Depression in Swedish Adolescents (DISA) program implementation using the figures from the pilot study for time spent on training program tutors, preparation, and conducting the 10-week course with five groups of students. Total cost includes the cost of training the head instructor. She took two courses for a total of 48 hr of training: basic training (3 days with 1 day of supervision) at US$870 and head instructor training (3 days with 1 day of supervision) at US$1450. She then gave a basic training course to five colleagues as part of the implementation of the DISA program: a total of 32 hr of instruction (4 days, 8 hr per day). Thus, the training time needed for the tutors who were later to instruct a total of 62 students was, on average, 3.35 hr per student. The students were divided up into five different groups, each with two tutors. Each session required 1½ hours of classroom time and 1½ hours of preparation time for each of the two tutors. Thus, each session costs 3 hr per tutor. Since each course group had 10 sessions of instruction, the tutor time needed for each group (with two tutors per group) was 60 hr. The mean number of students in each course group was 12.4 (62 students in five course groups). The average tutor time spent per student was thus 4 hr. The total cost of the course in terms of tutor time per student was thus 7.35 hr (3.35 hr tutor training time and 4 hr of instruction time). The salary costs were about US$34/hr (including prescribed employer costs). The course material can be obtained for free on the Internet, but the costs for making paper copies and for fruit were about US$13 per student. In total, this gives the sum of US$300 per student for implementing the DISA program once in a context similar to the one in this pilot study. Of this, as much as half the costs are related to the training costs of DISA tutors, including time and course fees. If these tutors repeat the DISA program, the average cost per student will fall. In addition, in this particular setting, the DISA program was conducted within the current budget, and no extra staff were employed. The illustrative calculation was based on the assumption that the opportunity cost of staff time—that is, the value of the tasks that could not be performed because the school nurse or school social worker were engaged with the DISA program—was valued using wages following the human capital approach (Drummond et al., 2005). Costs other than wages corresponded to about US$50 per student.