Rethinking Suicide Prevention Research – Moving Beyond Traditional Statistical Significance

Abstract

Suicide is a major public health concern globally, and despite decades of research, there has been a disappointing lack of progress in identifying effective prevention strategies and interventions. We argue over-reliance on traditional statistical significance cutoffs and underreporting of marginal findings may be limiting the clinical benefits of research in the field of suicide prevention and in turn impeding practical progress. The consistent reliance on statistically significant results at p <.05 may limit the visibility of potentially promising results to clinicians making treatment decisions. Expanding awareness of promising interventions – which can then be further scrutinized and subjected to further research – could have an important and needed impact on the field. The American Statistical Association has called upon researchers to view the p-value as continuous, with the call being adopted by leading journals. However, most suicide journals do not have explicit policies around how to use p-values for evaluating the strength of the evidence, and the use of continuous p-values has clearly not been routinely adopted by suicide researchers. We want to call upon suicide researchers to be more open to considering and publishing marginally significant findings that suggest promising trends for suicide prevention strategies and interventions.

Suicide is a major public health concern across the globe (United Nations, 2023; World Health Organization, n.d.-b). The World Health Organization (WHO) estimates that more than 700,000 people die by suicide each year (World Health Organization, 2021), and suicide is a leading cause of death among youth globally (Glenn et al., 2020). However, despite decades of research, there has been a disappointing lack of progress in identifying effective prevention strategies and interventions (Fox et al., 2020; National Institute of Mental Health, 2023). The US surgeon general has described mental health as “the defining public health crisis of our time” as both rates of hospitalization for self-harm and suicide continue to rise (Richtel, 2023) and the Director-General of the WHO has called “on all countries to incorporate proven suicide prevention strategies into national health and education programmes in a sustainable way” (World Health Organization, 2019).

Recognizing the urgency, the US National Institute of Mental Health has invested between $40 and 60 million USD annually since 2018 in research efforts to improve suicide risk screening, assessment, and intervention (Gordon, 2022; National Institute of Mental Health, 2023). Furthermore, the WHO made significant investment in the LIVE LIFE initiative to help countries implement evidence-based interventions for suicide prevention related to: Limiting access to means of suicide, Interacting with media for responsible reporting, Fostering socioemotional life-skills in young people, and Early identification and support to everyone affected by suicide and self-harm (World Health Organization, n.d.-a). Despite these significant investments, progress has been limited. Possible contributors to the limited progress are rigid statistical significance requirements for publishing promising results. In this commentary, we revisit and elevate this important issue, arguing that the over-reliance on traditional statistical significance cutoffs and the underreporting of marginal findings is limiting the clinical benefits of research in the field of suicide prevention and in turn impeding practical progress in the field (Sterne & Smith, 2001).

In spite of the limited research progress in suicide prevention (Fox et al., 2020), clinicians (including specialty mental health and primary care) and institutions (including schools, child welfare and the justice system) must act when they observe an individual at risk for suicide. Given a general lack of consensus and published evidence on optimal treatment approaches, clinicians often implement intervention strategies that are feasible and face valid but lack evidence, as it is better to treat with programs that have weak evidence than to not act at all (Fox et al., 2020; Langford et al., 2013).

How Can Research Be Improved and the Role of the p-Value

How can availability of research findings be improved? Publishing promising results from interventions with marginal effects and from underpowered studies could change the trajectory of suicide prevention. Suicide is hard to predict a priori and relatively rare at the individual level, even in high-risk populations (Knox & Caine, 2005). For example, the 2021 age-adjusted suicide rate among non-Hispanic American Indian/Alaska Natives in the United States was 28.1 per 100,000 versus 14.1 per 100,000 in the US population overall (Stone et al., 2023). In Estonia, the country with the highest youth suicide rate, adolescents die by suicide at a rate of a bit less than 10 per 100,000 people (Glenn et al., 2020).

In light of these low underlying rates, researchers testing the effectiveness of suicide prevention programs must assemble participant groups far larger than researchers studying more common outcomes like depression to achieve similar statistical power. This problem is especially difficult when studying relatively small but high-risk groups, such as indigenous and LGBTQ+ individuals. These groups may benefit the most from successful interventions; however, most suicide prevention programs were developed and tested within majority White, heterosexual, and cisgender communities (Winston et al., 2023). Together, these types of challenges—rare, hard to predict, and different effectiveness for different racial/ethnic, age, or gender populations—mean that most studies are underpowered to obtain p-values at traditional significance levels (e.g., p < .05), even with the most promising interventions. Equivalently, 95% confidence intervals will often include 0 even if the interval is focused on positive values.

The consistent reliance on statistically significant results at p < .05 may limit the visibility of potentially promising results when making clinical treatment decisions (Sterne & Smith, 2001). Successfully preventing suicide is very challenging (Caine, 2013) and limiting knowledge about potentially useful interventions because their impacts did not meet a stringent p < .05 cutoff seems self-defeating. Expanding awareness of promising interventions—which can be further scrutinized and subjected to further research—could have an important and needed impact on the field.

By definition, a p-value quantifies the strength of evidence against a null hypothesis (i.e., that the intervention had no effect) and ranges from 0 to 1. High p-values suggest the data (e.g., the observed regression estimate or summary statistic of interest) are consistent with what we would expect under the null hypothesis, while low p-values suggest inconsistency between the null hypothesis and observed data. Researchers define statistical significance against the null as computing a p-value less than some threshold (often .05). Notably, the p-value computation depends heavily on sample size; for a given effect, p-values decrease with increasing sample size, as larger samples provide more precise estimates and greater statistical power. Very large samples are almost guaranteed to produce statistically significant results regardless of clinical significance, while small sample sizes may not be sufficient to detect even large, clinically meaningful effects. For binary variables, the frequency of the outcome is also influential, with the same relative effects yielding lower p-values with more common (i.e., closer to 50%) outcomes than rarer ones.

Learning From Rare Diseases and Accelerated Approval Processes

To illustrate how envisioning p-values as continuous rather than using stringent cutoffs can advance the science in suicide prevention, we call upon the field of rare disease for helpful insights. First, in terms of power, rare disease studies (e.g., those examining amyotrophic lateral sclerosis (ALS), Huntington’s disease, or rare forms of cancer) can only recruit a small number of participants. These participants have a high probability of negative outcomes without intervention based on an existing diagnosis (e.g., ALS) or genetic testing (e.g., Huntington’s Disease or breast cancer genotyping [BRCA]). In contrast, suicide prevention studies need to recruit a much larger pool of participants because currently, no method exists to identify a person as being prone to suicide with the same level of accuracy that is possible for diagnosing rare diseases. Moreover, each participant within studies examining suicidal behaviors has a relatively small probability of attempting or dying by suicide within the typical study time frame. Both these circumstances, inability to diagnose suicide risk accurately and a small probability of attempting or dying by suicide, make it challenging for some suicide prevention/intervention studies to obtain a sample powered to detect statistically significant effects at p < .05 or showcase a confidence interval that does not include 0 (even if only marginally doing so).

Second, another similarity among rare disease and suicide research is the small effect sizes often found for treatments. Small effect sizes can be highly clinically significant even when they do not meet traditional statistically significant cutoffs. In both rare disease and suicide research, the value we place on the potential to save a life and reduce suffering outweighs the small effects observed.

Accordingly, many regulatory agencies including the US Food and Drug Administration (FDA), the European Medicines Agency (EMA), the UK Medicines & Healthcare products Regulatory Agency (MHRA), and the Pharmaceuticals and Medical Devices Agency (PMDA) in Japan have formal processes in place that allow for the approval of medicines that address unmet medical needs based on less comprehensive data than normally required (European Medicines Agency, n.d.; Temkin & Trinh, 2021; Yamori & Blumenrath, n.d.). For example, the US FDA implements an accelerated approval process that allows expedited access to hundreds of new drugs for individuals with HIV/AIDS, various forms of cancers, and rare diseases with hope these new drugs work to eradicate the disease (Temkin & Trinh, 2021). The accelerated approval process by the US FDA and other regulatory agencies is based on the principle that, for serious and life-threatening diseases or conditions, more uncertainty regarding the clinical benefit of new interventions may be acceptable, given the severity, rarity, or prevalence of the condition and the associated availability, or lack thereof, for life-saving treatments (European Medicines Agency, n.d.; Temkin & Trinh, 2021; Yamori & Blumenrath, n.d.). Accelerated approval allows regulatory agencies to bring the most promising drugs more quickly to market, with lower costs and fewer participants than traditional clinical trials. Furthermore, it allows patients to receive earlier access to potential lifesaving therapies, which can significantly improve quality of life. Similarly, other countries such as the United Kingdom have implemented programs such as the Innovative Licensing and Access Pathway which aim to expedite the development and availability of treatments for patients with serious or life-threatening conditions (Medicines and Healthcare products Regulatory Agency, 2021).

The US FDA, EMA, UK MHRA, and Japanese PMDA, among others, are willing to act on promising trends if there are minimal adverse side effects. Paired with systematic data collection (e.g., rare disease registries or vaccine adverse effects monitoring systems) on the longer-term effects of these interventions, this research pipeline can provide an opportunity for the field to strengthen the evidence base on the effectiveness of new treatments, including their benefits and harms, keeping practitioners informed of important current and future trends. It is critically important to implement continuous monitoring of new interventions since there can still be a risk that the intervention is not beneficial (or even harmful) in different settings and populations. When the statistical evidence for an intervention is not initially strong, ongoing monitoring and evaluation to build the evidence base is necessary.

As can be seen in the US FDA’s accelerated approval process:

FDA approval is not an iron-clad promise of safety or effectiveness but a recognition that the applicant has demonstrated that the product is safe and has provided substantial evidence that the product is efficacious … [and] the benefits of the drug outweigh the risks. … Therefore, approval under this rule requires that the effect shown be clinically meaningful in the judgment of the agency, and of such importance as to outweigh the risks of treatment.

(Temkin & Trinh, 2021)

We would like to underscore that measurement of adverse effects and unintended harms of interventions is critical, and those with evidence of potential harm should not be brought to scale without careful consideration and weighing of the potential harms relative to the benefits (Lilienfeld, 2007). Not following strict p-value thresholds should help with this, especially when results showcased in the 95% confidence interval may indicate the intervention is causing harm in situations where the intervention was originally hypothesized to be beneficial. If an intervention shows potential evidence of harm, we could catch it sooner by being more forgiving in the p-value and carefully examining the 95% confidence interval. In fact, detecting a marginally significant harmful association would allow us to potentially determine sooner rather than later that the intervention should not be tested in future studies.

Viewing the p-Value as Continuous

This relaxation of rigid significance thresholds is hardly a call for less rigorous statistical practice and is not new for researchers in many fields (Greenland et al., 2016; Greenwald et al., 1996; Hurlbert & Lombardi, 2009; Imbens, 2021; Sterne & Smith, 2001). The American Statistical Association (ASA) itself has explicitly called upon researchers to view the p-value as continuous, suggesting it can and should be used to evaluate the strength of the evidence for prevention strategies and interventions (Wasserstein & Lazar, 2016; Yaddanapudi, 2016). Specifically, the ASA statement on p-values states: “Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold” (American Statistical Association, 2016). More recently, an international task force including presidents of the ASA, Institute for Mathematical Statistics (IMS), and International Statistical Institute released a statement on p-values and cautioned that “[s]elective reporting, even the high-lighting of a few persuasive results among those reported, may lead to a distorted view of the evidence” (Benjamini et al., 2021). For many suicide studies, teams often find traditionally statistically significant effects with p-values less than 0.05 that quickly move to marginal (e.g., .10 > p >.05) after controlling for important potential confounders or utilizing propensity score weights to help improve the rigor of the study design (Ayer et al., 2022; Hill et al., 2023; Knowles et al., 2022; Mason et al., 2024; Ruch et al., 2019). This leaves researchers with an uncomfortable decision about how best to interpret findings and which model to report. Ultimately, the best statistical model should be reported. This allows full transparency about the level of uncertainty (e.g., via the 95% confidence interval) without fear of not being able to publish the study findings due to the significance level crossing the p < .05 threshold.

The call from the ASA and others has been adopted by several other fields, with some journals explicitly asking researchers to move away from over-reliance on p-values. For example, the New England Journal of Medicine emphasizes that publications must differentiate between statistical significance and clinical significance and request that submission focus on reporting standard errors and confidence interval estimates to avoid pitfalls associated with traditional hypothesis testing (New England Journal of Medicine, n.d.). Psychological Science emphasizes the use of “new statistics” including effect sizes and confidence intervals to avoid pitfalls associated with traditional hypothesis testing and requires that manuscripts differentiate between statistical significance and clinical significance. However, when reviewing the policies of several high-impact journals that publish suicide related research, most do not have explicit policies around the use of p-values and other metrics for evaluating the strength of the evidence for a particular finding. Additionally, the continuous interpretation of p-values has clearly not been routinely adopted by suicide researchers.

Additionally, careful attention should be paid to defining a clinically significant threshold for a given outcome(s) prior to conducting an experimental or observational study (Ogles et al., 2001). Prior research in the field of suicide prevention has already established how to define clinically significant change for many of the commonly used outcomes in this space (Ogles et al., 2001; Wise, 2004). In general, it is appropriate to set a clinically significant threshold for a given program that is tied to the outcomes of interest. To illustrate, with suicide deaths, it can be useful to consider the proportion of the sample whose lives were saved. Wise (2004) proposes clinical significance be defined in the context of how chronic a condition may be. For example, suicidal ideation can be a chronic, daily experience for certain populations. Thus, we may define an intervention’s clinical significance based on the proportion of the sample whose suicidal ideation improved relative to those who deteriorated (Wise, 2004). Additionally, improvement and deterioration could be defined in several ways, including frequency of ideation and severity of ideation. Defining a clinical significance threshold a priori and including this value as part of the study’s research protocol will help reduce potential bias as well as guard against the risk of researchers cherry picking values to report concerning study findings.

Conclusions and Recommendations

In conclusion, we call upon suicide researchers – including those serving as reviewers and editors – to be more open to considering and publishing marginally significant findings that suggest promising trends for suicide prevention strategies and interventions. This approach aligns with other papers published arguing that suicide prevention is a “winnable battle” (Caine, 2012; Knox et al., 2004) “of global importance” (International Association for Suicide Prevention, n.d.). Marginal findings from suicide prevention and intervention trials are still worth acknowledging and could lead to promising outcomes. Specifically, we recommend the suicide prevention field more intentionally and explicitly (1) consider the p-value as continuous, (2) report effect sizes and 95% confidence intervals, and (3) let the weight of the evidence from these statistics guide interpretation of findings as well as practice. To be clear, we are not suggesting careless acceptance of p-values over .05, but rather that such cutoffs should not pose a barrier to publication and interpretation. These recommendations can be strengthened with the use of preregistration, Registered Reports, and other procedures to ensure decisions about data collection and analysis are made prior to running analyses. These strategies make suicide research less susceptible to bias that could be introduced by authors and journals during the publication process.

As noted by the WHO: “The urgency to act to prevent suicides has been recognized and prioritized at the highest levels” (World Health Organization, n.d.-b). Following these three steps not only recognizes this urgency surrounding suicide but provides a road map for increasing suicide prevention programming that may lead to much needed decreases in these behaviors.

Biographies

Dr. Beth Ann Griffin is a senior statistician at RAND and codirector of RAND/USC Opioid Policy Tools and Information Center. Her statistical research focuses on methods for estimating causal effects using observational data. Her public health research falls into areas including suicide, addiction, mental health, rare diseases, and state policies.

Dr. Gabriel Hassler is an associate statistician at RAND and professor at Pardee RAND Graduate School. He is interested in computational Bayesian statistics and developing flexible statistical models for big data. He has applied his methodological work to HIV/AIDS, SARS-CoV-2/COVID-19 epidemic, and the evolution of numerous other biological organisms.

Dr. Arielle H. Sheftall is Associate Professor at University of Rochester Medical Center. Her research focuses on developmental mechanisms in early-to-middle childhood that confer vulnerability to suicidal behavior. She studies racial disparities in suicidal behavior among youth, aiming to frame targets for early intervention to decrease incidence of first attempts.

Elie Ohana is a statistical analyst at RAND. He researches topics at the intersection of psychology and statistics. His work includes research projects focused on mental health, measure development, education, and military family and veteran wellbeing. His statistical expertise includes survey research, experimental design, multilevel models, and propensity score matching.

Dr. Lynsay Ayer is a senior behavioral and social scientist at RAND and professor at Pardee RAND Graduate School. Her research focuses on preventing and mitigating mental health problems among high-risk populations (system-involved youth, those with limited access to care) and on the prevention of self-injurious thoughts and behaviors/suicide.