Hot Idea or Hot Air: A Systematic Review of Evidence for Two Widely Marketed Youth Suicide Prevention Programs and Recommendations for Implementation

Abstract

Introduction:

Youth suicide is highly related to mental disorders. While communities and schools are marketed to with a plethora of suicide prevention programs, they often lack the capacity to choose evidence-based programs.

Methods:

We conducted a systematic review of two youth suicide prevention programs to help determine if the quality of evidence available justifies their wide spread dissemination. We searched Medline, PsycINFO, EMBASE, CINAHL, the Cochrane Library, Campbell Collaboration SPECTR database, SocIndex, Sociological Abstracts, Social Services Abstracts, ERIC, Social Work Abstracts, Research Library, and Web of Science, for relevant studies. We included studies/systematic reviews/meta-analysis that evaluated the effectiveness, cost-effectiveness, and/or safety of Signs of Suicide (SOS) and Yellow Ribbon (YR) suicide prevention programs that target adolescents. We applied the Office of Justice Program What Works Repository (OJP-R) to evaluate the quality of the included studies as effective, effective with reservation, promising, inconclusive evidence, insufficient evidence, and ineffective. Two SOS studies were ranked as “inconclusive evidence” based on the OJP-R. One SOS study was ranked as having “insufficient evidence” on OJP-R. The YR study was ranked as “ineffective” using OJP-R. We only included studies in peer-reviewed journals in English and therefore may have missed reports in grey literature or non-English publications.

Results:

We cannot recommend that schools and communities implement either the SOS or YR suicide prevention programs. Purchasers of these programs should be aware that there is no evidence that their use prevents suicide.

Conclusions:

Academics and organizations should not overstate the positive impacts of suicide prevention interventions when the evidence is lacking.

Introduction

Most mental disorders have their onset prior to age 25 (Kessler et al., 2005). This implies a need for effective early life span mental health promotion, case identification and interventions to increase the possibility of positive outcomes such as better educational and social success, reduced burden of illness, decreased suicide rates and improved physical health (Canadian Council on Learning, 2009; Kutcher, 2011; Waddell, Offord, Shepherd, Hua, & McEwan, 2002; Wei, Kutcher, & Szumilas, 2011).

Currently, community leaders, especially school administrators are faced with a plethora of programs purporting to improve various aspects of youth mental health. While the need for best evidence-based implementations is recognized, (Forman, Olin, Hoagwood, Crowe, & Saka, 2009) many decision makers continue to select and implement heavily marketed and costly programs often lacking appropriate evidence for their effectiveness, cost effectiveness or safety (Hallfors & Godette, 2002). This may be due to the lack of the capacity to critically evaluate marketed programs, lack of an independent best evidence-based source of information to support decision making, or other factors.

This report describes findings of a pilot project that may help establish the foundation of an online Canadian resource of best evidence-based school mental health implementations that applies critical evaluation techniques to evaluate mental health programs marketed to schools (teenmentalhealth.org). Such a resource may assist educators and community decision makers in selecting evidence-based programs for implementation. To best address the needs of Canadian schools, a national advisory committee was established to help guide the development of an evidence-based tool designed to assist educators in making decisions about the purchase and implementation of mental health related programs in schools. Committee members consisted of mental health professionals, researchers, and education administers, such as school principals and superintendents, from Nova Scotia, New Brunswick, Ontario, Manitoba, and British Columbia.

On the basis of suggestions from the national advisory committee, consensus was made among the committee members that we selected two suicide prevention programs: the Signs of Suicide (SOS) Prevention Program and the Yellow Ribbon (YR) Suicide Prevention Program, as the target programs of our pilot project. We assessed their effectiveness, cost-effectiveness and safety, using a systematic review approach recommended by The Cochrane Collaboration (http://www.cochrane.org/). SOS is recommended to schools as a youth suicide prevention program (http://www.mentalhealthscreening.org/programs/youth-prevention-programs/sos/) and is listed on the Substance Abuse and Mental Health Services Administration’s National Registry of Evidence-based Programs and Practices in the United States of America. YR is a youth suicide prevention program listed on the Best Practices Registry of the National Suicide Prevention Resource Center in the United States of America and marked to schools and communities as a “best practice” suicide prevention program (http://www.yellowribbon.org/). In Canada it is marketed as a National Suicide Prevention Strategy (http://www.yellowribbon.ca/gatekeeper.html).

Currently, three published reviews are available addressing the effectiveness of SOS. (Cusimano & Sameem, 2011b; Mann, Apter, Bertolote, Beautrais, Currier, Haas, Hegerl, Lonnqvist, Malone, Marusic, Mehlum, Patton, Phillips, Rutz, Rihmer, Schmidtke, Shaffer, Silverman, Takahashi, Varnik, Wasserman, Yip, & Hendin, 2005b; Katz, Bolton, Katz, Isaak, Tilston-Jones, Sareen; Swampy Cree Suicide Prevention Team, 2013). However, the two reviews by Cusimano et al. (2011) and Mann et al. (2005) did not analyze the overall quality of evidence of effectiveness of included studies nor their methodological characteristics, both necessary components of systematic reviews. The third review (Katz et al., 2013) gave SOS mixed ratings. On the one hand, it recommended SOS because of its reported effect on reduced suicide attempts (grade B), but did not recommend it because of its failure to reduce suicide ideation (grade D). This interpretation is confusing as suicide ideation is considered to be the initial stage along the continuum of suicide attempt and completion (Bridge, Goldstein, & Brent, 2006; Perez, 2005). Further, the review by Katz et al. (2013) did not provide an exhaustive list of studies of SOS, and therefore a more in-depth analysis of SOS research is warranted. Katz et al. (2013) also reviewed YR and rated it as grade D (evidence of troublingly or inconclusive studies of any level). However, authors neither discuss the quality of YR research report, nor did they critically appraise its findings. Thus, while these programs are widely marketed as effective in prevention of youth suicide and are listed on recognized repositories, independent systematic analytics of the evidence in support of them is limited in the public domain.

In the present study, we conducted a systematic review of these programs to help determine if the quality of evidence available justifies their wide spread dissemination in schools and in the community as suicide prevention programs. Our consideration was that a program that is marketed and sold as being a suicide prevention program should have substantial evidence of preventing suicide for that claim to be reasonably made.

Findings of this review are reported based on the recommendations suggested by Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement (http://www.prisma-statement.org/index.htm), which is widely accepted in the health/public health research field.

Methods

Inclusion criteria

We included research studies that evaluated the effectiveness, cost-effectiveness, and/or safety of SOS and YR. We also included published systematic reviews/meta-analyses on suicide prevention since they may include studies addressing the two programs. We only included studies published in peer-reviewed journals in English and we had no time limits. We included studies of any type and with any outcome as long as they evaluated one of the two programs.

Exclusion criteria

Systematic review or meta-analyses dealing with suicides linked to physical illness were excluded.

Search methods

Two team members independently searched a number of databases, including Medline, PsycINFO, EMBASE, CINAHL, the Cochrane Library, Campbell Collaboration SPECTR database, SocIndex, Sociological Abstracts, Social Services Abstracts, ERIC, Social Work Abstracts, Research Library, and Web of Science. The search started with two sets of search terms addressing suicides and suicide interventions with key words such as: suicide, self-harm, self-injury/behaviour/inflict/hatred, suicide attempt; prevention, crisis intervention, health services, health education, program development, program evaluation, experiments, trials, effectiveness, efficacious, cost-effectiveness, safety, and harm effects (see Appendix 1 for complete search strategy and the full search strategy with Medline). The results from the two search terms were combined using the Boolean AND, which were further filtered by key terms “SOS suicide prevention” and “Yellow Ribbon suicide prevention”. Meanwhile, we repeated the first two steps but filtered the results with key words systematic review and meta-analysis, to capture existing reviews/meta-analyses that could have included the two programs.

Study selection and data extraction

Two team members ran the search with the key terms in the identified databases, imported the results into the RefWorks 2.0 database management program, and removed duplicates. We then screened titles and abstracts of imported studies to delete irrelevant studies. Remaining research studies were further assessed for inclusion/exclusion by reading the full text, and systematic reviews/meta-analyses were scanned to check whether the two programs were included and investigated, which resulted in the final studies and systematic reviews/meta-analyses for inclusion for this report.

A data extraction form, developed a priori, for identifying information, study eligibility, study characteristics (baseline characteristics, location, timing, methods, and intervention type and duration), outcome measures and quantitative data was applied to each study.

Critical appraisal

We then applied the Office of Justice Program (OJP) What Works Repository (National Criminal Justice Reference Services, 2005) to help evaluate the quality of the included studies. The OJP What Works Repository (OJP-R) classifies programs into six levels of evidence of effectiveness: effective, effective with reservation, promising, inconclusive evidence, insufficient evidence, and ineffective; and three levels of readiness for dissemination: fully prepared for widespread dissemination, fully prepared for limited dissemination, and not ready for dissemination. Programs are appraised using the following criteria: evidence from randomized controlled trials (RCT), replication with different population and contexts, focus on socially important behaviour outcomes, identification of evidence of enduring effects, and dissemination capacity. Further, assessment occurs against 4 indicators to determine levels of readiness for dissemination: information on training and related support materials; technical assistance support; informational materials; quality control for implementation. The OJP-R helps to determine both the internal and external validity of included studies.

Data analysis

We planned to conduct a meta-analysis if the included studies were ranked as “promising” or above based on The OJPR and there were at least two studies for each intervention with homogeneity in study design, study duration, outcome measures, population sizes, and timeframe.

Results

We did not undertake the planned meta-analysis because of the small number of final studies and lack of common outcome measures among the studies. Our systematic review was therefore restricted to a description and analysis of eligible studies.

Study characteristics

Figure 1 indicates the process of how included studies were obtained, applying the template of flow diagram recommended by PRISMA statement (http://www.prisma-statement.org/statement.htm). Four studies evaluating SOS (Aseltine, 2003; Aseltine & DeMartino, 2004; Aseltine, James, Schilling, & Glanovsky, 2007; Schilling, Lawless, Buchanan, Aseltine, 2014) and one study evaluating YR (Freedenthal, 2010) were eligible for inclusion for this report.

Figure 1.

Flow chart of included studies

Table 1 summarizes the characteristics of included studies. Of three studies evaluating SOS, two are RCTs (Aseltine & DeMartino, 2004; Aseltine et al., 2007; Schilling, Lawless, Buchanan, & Aseltine, 2014) evaluating the effectiveness of SOS in middle and high school students in five aspects: knowledge, attitudes, help-seeking behaviours, self-reported suicide attempts and suicidal ideation. The two RCTs (Aseltine & DeMartino, 2004; Aseltine et al., 2007) are overlapping with one reporting year 1 findings of five schools and the other reporting year 2 findings of nine schools, including the five schools in year 1 and four additional schools. A more recent RCT of the SOS program (Schilling, Lawless, Buchanan, & Aseltine, 2014) focused on middle school students (grades 5–8) with a large proportion of students with parents in the military. Another study (Aseltine, 2003) conducted a process evaluation, addressing self-reported help-seeking behaviours, and perspectives of school staff on cost and benefits of the program. All four studies addressed short-term (3 months following the program) impact of SOS with no outcomes beyond three months reported. The one published study (Freedenthal, 2010) about the YR program investigated its effectiveness on students’ help-seeking behaviours, and also observed short-term impact of the program.

Table 1.

Study characteristics

Author/study	Aseltine 2003	Aseltine & DeMartino 2004	Aseltine et al. 2007	Schilling et al., 2014	Freedenthal 2010
Sample	92 schools (post); 68 schools (follow-up)	5 schools;(n=1073 control; n=1027 treatment)	data from 5 schools reported in Aseltine 2044 + 4 new schools; (n=2094 control; n=2039 treatment)	8 schools; (n=419 control and treatment (pretest); n=386 control and treatment (posttest)	2 schools Teachers: (n=70 pre; n=57 post, treatment; n=98 pre; n=85 post, control) Students: (n=146 treatment)
Country	US	US	US	US	US
Population	School staff	High school students	High school students	Middle school students	High school students and teachers
Design	Survey	RCT (by class)	RCT (by class)	RCT (by school)	Quasi-experimental
Comparability at baseline	Not applicable	No baseline data	No baseline data	Yes	No baseline data
Intervention	SOS program combines curricula to raise awareness of suicide and its related issues with a brief screening for depression and other risk factors associated with suicidal behaviour.	SOS program combines curricula to raise awareness of suicide and its related issues with a brief screening for depression and other risk factors associated with suicidal behaviour.	SOS program combines curricula to raise awareness of suicide and its related issues with a brief screening for depression and other risk factors associated with suicidal behaviour.	Not described	Yellow Ribbon Program includes school wide assemblies, peer leadership training for students, staff training for adult gatekeepers such as high school teachers, community presentations, and local chapters that provide outreach and education.
Duration	5.3 days on average	Not described	2 days	Not described	60 minutes (student training); 1.5 hours (staff training); 50 minutes (school assembly)
Comparison	No	Yes	Yes	Yes	Yes
Follow-up	Post intervention	3 month post intervention	3 months post intervention	3 months post intervention	Post intervention
Measurement validity	Not reported	Not reported	Not reported	Not reported	Not reported
Outcome	Overview of Implementation; Ratings of and Reactions to the Program; Help-seeking behaviours; Costs and benefits of the program	Knowledge; Help-seeking behaviours: Suicide attempts; Suicide ideation	Knowledge; Help-seeking behaviours: Suicide attempts; Suicide ideation	Knowledge; Help-seeking behaviours: Suicide attempts; Suicide ideation	Help-seeking behaviours;

RCT: Randomized Controlled Trial

Although the SOS program (Aseltine, 2003) reported teacher’s opinions on cost and benefits of SOS, we could not find any appropriate cost-effectiveness measures in any study of SOS or YR. We could not find any study of SOS or YR that had been designed to identify whether these programs may cause harmful or negative effects. All five studies were carried out in the United States of America.

Three RCT studies (Aseltine & DeMartino, 2004; Aseltine et al., 2007; Schilling, Lawless, Buchanan, & Aseltine, 2014) of SOS applied similar measures for program effectiveness. The third SOS study (Aseltine, 2003) evaluated the feasibility and safety of the implementation of SOS and assessed self-reported help-seeking behaviours, using different measures from the two RCT studies. All three studies applied measurement tools specifically developed for the intervention but with unknown reliability and validity except for one tool. The attitude measurement tool had a measure of reliability, a reported Cronbach’s α of 0.74 (Aseltine & DeMartino, 2004; Aseltine et al., 2007) and 0.73 (Schilling, Lawless, Buchanan, & Aseltine, 2014), but no measure of validity. The study of YR did not report on the reliability and validity of any measurement tools.

Program effectiveness

Table 2 summarizes the statistical data on the program effectiveness for all five studies. All five studies reported surrogate suicide/self-harm related indicators. Two RCT studies (Aseltine & DeMartino, 2004; Aseltine et al., 2007) concluded that SOS was effective in increasing student knowledge of and attitudes towards depression and suicide and in reducing self-reported suicide attempts among students (p<0.05) (Table 2). The most recent RCT (Schilling, Lawless, Buchanan, & Aseltine, 2014) presented similar conclusions except no change in attitudes. SOS showed no impact on student help-seeking behaviours in either of the RCT studies and help-seeking behaviours actually decreased in the treatment groups though not to a statistically significant degree (p>0.05) (Table 2). Two RCT studies (Aseltine & DeMartino, 2004; Aseltine et al., 2007) measured suicidal ideation and attempts. While they reported SOS effective in reducing self-reported suicide attempts, they failed to demonstrate an impact of SOS on reducing suicidal ideation. The new RCT by Schilling et al. (2014) failed to find any SOS effect on posttest suicidal ideation (b=−0.53, SE=1.01, p>.05) or on any suicidal behavior combined among all participants (b=0.84, SE=0.73; p>.05). However, in spite of these negative results the authors reported that the SOS intervention was positive.

Table 2.

Summary of study results

Author/study	Aseltine 2003	Aseltine & DeMartino 2004	Aseltine et al. 2007	Schilling et al., 2014	Freedenthal 2010
Knowledge	Not assessed	M*=6.49 (C); M=7.18* (T)	M=4.36 (C); M=5.00** (T)	b=0.56 ** (SE=0.25)	Not assessed
Attitudes	Not assessed	M=3.80 (C); M=4.05** (T)	M=3.83 (C); M=3.99** (T)	b=0.13 (SE=0.09)	Not assessed
Help-seeking	60% relative (increase of help-seeking compared to the previous year)	For friends: 13% (C); 11.9% (T) For oneself: 18.7% (C); 15.9% (T)	For friends: 12.4% (C); 12.8% (T) For oneself: 19.5% (C); 18.4% (T)	From friends: b=−0.29 (SE=0.41) From parents: b=−0.31 (SE=0.68) From brother or sister: b=−1.44 (SE=1.12)	Staff data: 4.9% (T); 2.3% (C); (2.6% between-school change) Student data: M=12.0 (pre-intervention) M=10.0** (post-intervention)
Suicide attempts	Not assessed	5.4% (C); 3.6%** (T)	4.5% (C); 3.0%** (T)	Not reported	Student disclosed suicide attempts (Staff data): −7.9% (T); 7.7% (C); (−15.6%*** between-school difference) Student reported peer suicide attempts (Staff data): −12.3% (T); 7.8% (C); (−21.1% between-school difference)
Suicide ideation	Not assessed	12.2% (C); 10.1% (T)	11.5% (C) 10.1% (T)	b=−0.53 (SE=1.01)	Student disclosed suicidal thoughts (staff data): −0.4 (T); 4.5% (C); (−4.9%** between-school difference) Student reported peer suicidal thoughts (staff data): −3.4% (T); −0.9% (C); (−2.5% between-school difference)
Other	Implementation: 74% (in health class); 40%–45% students receiving the entire program Program rating: 64%–87% (very or somewhat effective) Costs and benefits: 19% (adverse effects) 88% (somewhat or very helpful)	Not assessed	Not assessed	Not assessed

M = mean, C = control subjects, T = treatment, b = effect of the SOS program, SE = standard error.

^**p<.05, indicating statistically significant results;

^***p<.001, indicating statistically significant results.

Between-school difference is the change in proportion, from pre-test to post-test at the treatment school minus the change in proportion at the control school. A negative number indicates that the experimental school had a smaller increase or greater decrease in proportion than did the control school.

The fourth SOS study (Aseltine, 2003) identified increased rates (60%) of self-reported student help-seeking behaviours. However, this is a relative and not an absolute increase calculation. The number of students reporting help seeking rose from 6.79 per month one year before the program to 10.63 one month following the program. Calculating this 3.84% increase as a clinically more meaningful number-needed-to-treat (NNT), and assuming that this increase is due entirely to SOS, 26 students would have to be exposed to the program in order to have an additional student seek help in the month following program termination. Teachers rated SOS as either very or somewhat effective, and most teachers (81%) did not think it would have adverse effects, although adverse effects were not measured.

The study of YR (Freedenthal, 2010) collected data from school staff and students in both the experimental and control schools at times corresponding to before and after the intervention in the experimental school. Student help-seeking behaviour was not statistically significant for most help-seeking behaviours and was in the opposite of the expected direction for two measures. School staff reported a statistically significant decrease in student-disclosed suicide attempts of 7.9% from post-test to pre-test in the treatment school compared to an increase of 7.7% student-disclosed suicide attempts in the control school, a total of 15.6% (p<0.001) difference between the treatment and control school (Table 2). Student-disclosed suicidal thoughts decreased 0.4% from before to after in the experimental school while it increased 4.5% in the control school, a total of 4.9% (p<0.05) difference between the treatment and control school (Table 2).

None of the studies were designed or powered to detect differences in suicide death rates and no study reported independent evaluation of self-harm or suicide attempts (such as parental report or health record analysis).

Critical appraisal of included studies

Three SOS studies (Aseltine & DeMartino, 2004; Aseltine et al., 2007; Schilling, Lawless, Buchanan, & Aseltine, 2014) were ranked as “inconclusive evidence” based on the OJPR. One SOS study (Aseltine, 2003) was ranked as having “insufficient evidence” on OJP-R. The YR (Freedenthal, 2010) program was ranked as “ineffective” using OJP-R (Figure 2).

Figure 2.

Level of evidence of included studies

The quality of evidence of the internal validity of these studies was undermined by a number of substantive methodological problems such as: lack of reported sustained effects; lack of at least one external/independent replication; lack of substantial clinical significance (absolute differences between groups, NNTs, or risk ratios); lack of validity and reliability of measurement tools; randomization in only half of the studies; information about intention-to-treat in only half of the studies; lack of information about cost-benefits and program safety.

None of the studies reached the level of the “promising” cut-off OJP-R criterion for readiness of program dissemination. Neither SOS nor Yellow Ribbon met dissemination for readiness using OJP-R criteria (Figure 3).

Figure 3.

Readiness of dissemination for included studies

Discussion

Although both SOS and YR programs are extensively marketed as effective youth suicide prevention programs, neither has been shown to decrease youth suicide rates. Independent critical analysis demonstrates that neither program meets minimal criteria for effectiveness or readiness for dissemination. They should not be marketed as nor considered to be suicide prevention programs before more rigorous studies are conducted.

We applied systematic and methodologically clear criteria to search for relevant studies on the SOS and YR programs and we are confident the included studies are the exhaustive list as reported in the literature. This review only included studies in peer-reviewed journals in English language and therefore may have missed reports in grey literature or non-English publications.

The findings that SOS improves knowledge of and/or attitudes towards suicide and depression are consistent with similar studies evaluating various school or community based suicide prevention programs in children and adolescents, (Cusimano & Sameem, 2011b; Gould, Greenberg, Velting, & Shaffer, 2003; Mann, Apter, Bertolote, Beautrais, Currier, Haas, Hegerl, Lonnqvist, Malone, Marusic, Mehlum, Patton, Phillips, Rutz, Rihmer, Schmidtke, Shaffer, Silverman, Takahashi, Varnik, Wasserman, Yip, & Hendin, 2005a; Ploeg et al., 1996; van et al., 2011; York et al., 2013) and are not unique to SOS. Nor is knowledge about suicide equivalent to prevention of suicide. Furthermore, it is unknown if SOS or YR contributions to improved knowledge and attitudes are significantly greater than, equal to or less than traditional educational interventions such as embedding information about depression and suicide in usual school curriculum or presenting general mental health information to students via other methods. This information is essential for educators and decision makers to have, for if the goal of intervention is to improve knowledge and attitudes pertaining to suicide (as apart from suicide prevention), and similar results in improving knowledge and attitudes may be obtained without the purchase of costly add-on programs, there may be less interest in using either program.

While improvements in knowledge and attitudes are useful, none of the studies reported significant or substantially increased help-seeking behaviours. In fact YR reported significantly lower help-seeking behaviours following the intervention. It is not clear whether this was because students were less likely to seek help or whether there were fewer suicide attempts to be reported. If it was the first case, such findings are consistent with previous research findings (Gould et al., 2003) noting that improvement in knowledge and attitudes may not enhance help-seeking behaviours, nor have any significant impact on decreasing youth suicide rates. Another review (Mann, Apter, Bertolote, Beautrais, Currier, Haas, Hegerl, Lonnqvist, Malone, Marusic, Mehlum, Patton, Phillips, Rutz, Rihmer, Schmidtke, Shaffer, Silverman, Takahashi, Varnik, Wasserman, Yip, & Hendin, 2005a) notes that public education and awareness programs usually demonstrate little or no detectable effects on treatment seeking. Without substantial evidence that these programs have an impact on increasing help seeking behaviour it seems unlikely that they could have any impact on suicide prevention, and should not be promoted as effective suicide prevention programs in the absence of such evidence.

Similarly, proxy measures, such as self-reports of suicide ideation or attempts are not appropriate as evidence for suicide prevention. Epidemiological data demonstrates very high rates of self-reported suicide ideation and high rates of suicide attempts but very low rates of suicide deaths in young people (http://www.cdc.gov/violenceprevention/pdf/Suicide_DataSheet-a.pdf). Thus even highly significant changes in these proxy measures may have little or no impact on the outcome of interest – suicide deaths. Researchers choosing to use proxy measures may wish to consider such indicators as emergency room visits or all cause hospitalizations to assess the impact of suicide prevention programs. Further, although all the SOS studies failed to find reduced suicidal ideations, they did report reduced attempts. The reported discrepancy between suicide attempts and ideations is in contrast with previous reports (Bridge, Goldstein, & Brent, 2006; Perez, 2005) and the authors’ clinical experience suggest that the two measures should be significantly associated.

Additionally, these findings have not been independently replicated by others. Instead, numerous studies and systematic reviews/meta-analyses have concluded that there is no evidence that such suicide prevention programs/interventions can significantly decrease suicide attempts or ideation. (Mann, Apter, Bertolote, Beautrais, Currier, Haas, Hegerl, Lonnqvist, Malone, Marusic, Mehlum, Patton, Phillips, Rutz, Rihmer, Schmidtke, Shaffer, Silverman, Takahashi, Varnik, Wasserman, Yip, & Hendin, 2005a; Ploeg et al., 1996; Robinson, Hetrick, & Martin, 2011; York et al., 2013). A recent systematic review on suicide prevention, postvention, and early intervention (Robinson, Cox, Malone, Williamson, Baldwin, Fletcher, & O’Brien, 2013) further points out that there is very limited evidence that such interventions actually work. The inconsistency between the very positive conclusions as reported by the authors of these studies and the largely negative conclusions of independent investigators conducting reviews of these studies is concerning and warrants further investigation.

None of the studies of SOS or YR conducted a thorough analysis on whether these programs could cause harmful or negative effects except one methodologically weak study (Aseltine, 2003) which reported teachers’ subjective feelings of SOS as not causing harm to students. This raises red flags especially when there is substantial research showing suicide awareness and education may have detrimental effects in youths. (Cusimano & Sameem, 2011a; Gould et al., 2003; Ploeg et al., 1996; van et al., 2011) These include more hopelessness and maladaptive coping strategies following the intervention; (Kalafat & Elias, 1994) more negative reactions among high risk youth, (Shaffer, Vieland, Garland, Rojas, Underwood, Busner, 1990) and normalization of suicide behaviour upon receiving the intervention. (Beautrais et al., 2007) Some studies indicate that this is especially the case among male participants.(Lester, 1992; Ploeg et al., 1996; Pompili et al., 2010).

Even more concerning is that the most recent study of SOS (Schilling, Lawless, Buchanan & Asetine, 2014) which demonstrated an increased number of suicide attempts in the intervention group (n=5 attempts) compared to controls (n=0 attempts) even though the control group had significantly higher reported rates of suicidal ideation and plans PRIOR to the application of the intervention. This serious finding could be interpreted as the SOS program inducing suicide attempts. However, the authors of that study made no mention of this finding in their discussion and concluded that their intervention has promising results for suicide prevention!

No study evaluated costs vs. effectiveness, essential information for those who must make economically-informed decisions about purchase and implementation of programs. Marketed programs should demonstrate cost effectiveness based on sound research.

Additionally, no study demonstrated long-term positive impact nor successful external, independent replication. All studies applied self-report questionnaires without established reliability and validity. This is especially problematic when using self-report to evaluate suicide attempts or ideation because of participants’ tendency to choose socially desirable answers (the Hawthorne effect).

Another significant methodological limitation was the failure to collect baseline and immediate post intervention data. Instead, two RCT studies (Aseltine & DeMartino, 2004; Aseltine et al., 2007) of the SOS program studies reported only three-month post intervention results. Given the small sample sizes, results could be due to pre-existing group differences or to the impact of other group specific factors occurring during the three months after the intervention. The most recent RCT on SOS (Schilling, Lawless, Buchanan, & Aseltine, 2014) improved its methodology by providing baseline data, however, the baseline data indicated significant differences in suicidal ideation and suicide plans between the control and intervention group (p<.05), implying potential significant bias in data analysis. Further, the fact that only two schools were assigned as the control group against 6 schools as the intervention group without specifying how many participants in each group is concerning, as their findings may have been biased due to an imbalanced number of participants in each group. Additionally, there are a number of significant errors in the methods and analysis in this study, including failure to account for clustering of students within classrooms and within schools in the analysis, the creation of a new “suicide behavior” variable that lumps suicide attempt with suicide plans and suicidal ideation and potential bias due to the large number of covariates in the logistic regression model.

Contrary to the marketing of SOS and YR, neither the assessment of study quality nor the application of the OJP-R supports these as ready for dissemination. Indeed, there is no evidence that either of these “suicide prevention” programs decrease suicide rates, or its more common proxy measure, emergency room visits for suicide attempt. In the absence of such evidence it is not appropriate to continue marketing or identifying these as suicide prevention programs.

This review only included studies in peer-reviewed journals in English language and therefore may have missed reports in grey literature or non-English publications. Further, our recommendations on school suicide prevention programs were only based on SOS and YR, and therefore may not be comprehensive as other suicide prevention programs may provide alternative perspectives.

Implications and conclusions

Reduction of youth suicide rates is an important public health challenge that must be addressed using interventions of proven effectiveness, safety and cost-effectiveness. Youth suicide is a complex phenomenon and reduction of youth suicide rates may require a comprehensive approach involving the social determinants of health and multi-sector collaboration among youth serving institutions, health and human services systems, families and communities. Policy and practice are not served by implementing programs that have no evidence for reducing youth suicide rates and that are not known to be safe nor cost-effective. In the best case scenario, resources are wasted implementing an ineffective strategy. In a worst case scenario, interventions may actually cause harm, for example by leading to decreased help-seeking behavior, or as identified but not discussed by the authors of the Shilling et al paper (2014), increasing suicide attempts in one of the SOS intervention groups. Indeed, one may make the argument that selling suicide prevention programs in the absence of substantial evidence that they prevent suicide may transgress ethical marketing principles. It is also imperative that academics and nationally respected organizations take responsibility for not overstating the positive impacts of suicide prevention interventions when the evidence for them is lacking.

In conclusion, we cannot recommend that schools and communities implement either the SOS or YR suicide prevention programs. We recommend that the marketing of these programs stop suggesting that they prevent youth suicide and that policy makers, health care providers, educators and others do not promote or consider them as such. We further recommend that programs that are marketed and sold as preventing youth suicide be subject to rigorous, independent and properly designed and powered research studies to determine their impact on suicide deaths prior to their implementation. Perhaps, given the recent passage of Bill 300 (Federal Framework to Prevent Suicide Act) (Government of Canada, 2012), consideration could be given to both establishing an independent evaluation/regulation arm of Health Canada to assess, evaluate and certify programs that have met acceptable standards for effectiveness, safety and cost-effectiveness and to establish and fund an independent national suicide research center that can independently conduct the rigorous research needed to make those determinations.

Appendix 1. Full search strategies with Medline

1. To detect SOS and YR intervention studies:

   (Suicid* OR “self harm” OR (self N3 (injur* OR behav*)) OR (self N3 (hatred OR mutilate* OR injur* OR inflict*)) OR “Suicide”[Mesh] OR “Suicide, Attempted”[Mesh])

   AND

   (Prevent* OR (interven* N3 (crisis OR crises)) OR (health N3 (service* OR educat*)) OR (program* N3 (develop* OR evaluat*)) OR experiment* OR trial* OR effective* OR efficac*)

   AND

   ((“Signs of Suicide” OR “Yellow Ribbon”) AND “suicide prevention”)

2. To detect systematic reviews or meta-analyses that may have included the two target programs:

   (Suicid* OR “self harm” OR (self N3 (injur* OR behav*)) OR (self N3 (hatred OR mutilate* OR injur* OR inflict*)) OR “Suicide”[Mesh] OR “Suicide, Attempted”[Mesh])

   AND

   (Prevent* OR (interven* N3 (crisis OR crises)) OR (health N3 (service* OR educat*)) OR (program* N3 (develop* OR evaluat*)) OR experiment* OR trial* OR effective* OR efficac*)

   AND

   (“systematic review” OR meta-analysis)