Abstract
Background
The National Action Alliance for Suicide Prevention Research Prioritization Task Force (RPTF) has created a prioritized national research agenda with the potential to rapidly and substantially reduce the suicide burden in the U.S. if fully funded and implemented.
Purpose
Viable, sustainable scientific research agendas addressing challenging public health issues such as suicide often need to incorporate perspectives from multiple stakeholder groups (e.g., researchers, policymakers, and other end-users of new knowledge) during an agenda-setting process. The Stakeholder Survey was a web-based survey conducted and analyzed in 2011–2012 to inform the goal-setting step in the RPTF agenda development process. The survey process, and the final list of “aspirational” research goals it produced, are presented here.
Methods
Using a modified Delphi process, diverse constituent groups generated and evaluated candidate research goals addressing pressing suicide prevention research needs.
Results
A total of 716 respondents representing 49 U.S. states and 18 foreign countries provided input that ultimately produced 12 overarching, research-informed aspirational goals aimed at reducing the U.S. suicide burden. Highest-rated goals addressed prevention of subsequent suicidal behavior after an initial attempt, strategies to retain patients in care, improved healthcare provider training, and generating care models that would ensure accessible treatment.
Conclusions
The Stakeholder Survey yielded widely valued research targets. Findings were diverse in focus, type, and current phase of research development but tended to prioritize practical solutions over theoretical advancement. Other complex public health problems requiring input from a broad-based constituency might benefit from web-based tools that facilitate such community input.
Introduction
N early 330,000 Americans lost their lives to suicide in the first decade of the 21st century,1 and the U.S. suicide rate in 2010 was higher than it was in 1950, suggesting the need for a more coordinated and intensive effort to address this critical public health problem.a With support from USDHHS Secretary Kathleen Sebelius and Secretary of Defense Robert Gates, the National Action Alliance for Suicide Prevention was launched in September 2010 to address this challenging public health problem.2 The purpose of this public–private partnership is to optimize suicide prevention efforts in all U.S. at-risk populations. The newly formed Action Alliance immediately developed a task force–based organizational structure to address multiple suicide prevention needs simultaneously. To that end, it formalized a relationship with a previously formed working group,b subsequently identified as the Research Prioritization Task Force (RPTF) to guide its suicide research prioritization efforts.3 The RPTF’s mission was to produce a scientific agenda with the potential to reduce suicide attempts and deaths each by at least 20% in 5 years and 40% or greater in 10 years, if fully funded and implemented.4
In some scientific contexts, it might be appropriate to set a research agenda using only input from groups of scientists with the necessary expertise. However, when addressing complex public health problems, reliance on expert opinion alone may lead to scientific work that sacrifices breadth and scope of perspective for depth of knowledge within narrow or highly specialized domains.5,6 As the mismatch between scope of expertise and the true breadth and complexity of a problem increases, there is also an increased likelihood that expert group input alone will not provide the perspective necessary to generate sufficiently broad-based, practical research targets to adequately meet the challenge.6 In such cases, input from large and diverse constituent groups—if properly structured—may produce more viable research targets and also maximize the potential for identifying practical leverage points through which to address complex social issues.
Inclusion of diverse stakeholder groups in an agenda-setting process can also be critical to long-term success. As noted by members of the National Academy of Science Research Council, agendas that endure often grow out of sustainable knowledge-to-action networks, composed of both researchers and end-users of scientific products.7–9 Such networks are capable of providing enduring support for a scientific enterprise, sustaining it from conceptualization through broad-based implementation.
The RPTF’s final agenda is organized around a set of Aspirational Goals, each of which clearly identifies a practical, measurable endpoint to a specific line of suicide prevention research. The RPTF opted to develop these Aspirational Goals using broad-based input (see accompanying supplement for a diagram of the agenda-building process). To generate, refine, and prioritize a set of goals that would be widely regarded as critical to suicide burden reduction, the RPTF conducted an online Stakeholder Survey. This multi-round process consisted of brief, sequenced questionnaires interspersed with structured opportunities for participant feedback and discussion. The primary purpose of this manuscript is to present the final goals developed via this stakeholder engagement process.
Methods
Survey Description
The four-round Stakeholder Survey was conducted between August 8 and November 11, 2011, and analyzed in 2012. The first round used the Zoomerang.com© platform and subsequent rounds utilized RAND Corporation’s online modified Delphi system, ExpertLens™.10 This exercise consisted of the following:
an initial round (the “idea-generating” process) during which each participant nominated two important suicide prevention research goals and important criteria for assessing the merits of any such goal;
a preliminary prioritization round during which participants rated a list of candidate goals abstracted from prior round submissions on criteria chosen during that prior round (i.e., potential burden reduction, projected ease and speed of real-world uptake, impact on vulnerable population groups, and acceptability to suicidal persons and family members);
a feedback and discussion round during which participants reviewed their ratings compared to group medians and discussed views with other survey participants;
a final goal prioritization round during which participants could change preliminary ratings based on prior round discussion.
The RAND Corporation’s online ExpertLens tool electronically calculated feedback provided to survey participants for these rounds. First-round results were condensed and reworked as described in the online publication How Did We Get to 12 Goals? available at http://actionallianceforsuicideprevention.org/task-force/whatsnew/21. Project oversight was provided by the IRBs of the University of North Texas Health Sciences Center and RAND Corporation, and participants were told that consent was implied with completed survey registration.
Final analytic sample
Potential respondents were individuals whose names appeared on any of a number of relevant organizational lists (e.g., professional associations, academic departments, grantee lists, among others) nominated by members of the RPTF. More than 4,000 individuals were invited to participate, and a total of 716 adults (aged 18 years or older) from 49 U.S. states and 18 foreign countries ultimately registered for the Survey. The sample included suicide prevention researchers (n=215); individuals (patients/consumers) and family members (survivors) who had been directly affected by suicide (n=227); healthcare and other treatment providers (n=175); and policy-makers or administrative decision makers with responsibility for suicide prevention activities (n=99).
Attrition between first and subsequent Survey rounds was fairly even across groups (% group representation in the first round versus % participation in one or more subsequent rounds: survivors, 30.0% vs 30.5%; researchers, 32% vs 33.1%; providers, 24% vs 22.9%; and administrators/policymakers, 14% vs 13.5%). A portion of Survey respondents (n=129) participated in either the initial or final Rating Rounds, but not both. To determine whether input from these individuals could be included in the final analytic data set, the degree of change in scoring across rounds was assessed. By-group and within-participant initial-to-final-round delta scores calculated for individuals who participated in both Rating Rounds (n=231) were not significantly different for any group. Likewise, within-subject analyses suggested that with the exception of the goal proposing improved biological interventions, within-subject goal ratings did not change significantly. Therefore, the final sample included data from individuals who participated in only one of the two Rating Rounds as well as data from individuals who participated in both, yielding a final analytic data set of 511 respondents.
Survey Analyses
After Survey Round 1, initial goal nominations were categorized and tallied within categories, and goals receiving the highest number of nominations were further evaluated, as described online at the RPFT website (http://actionallianceforsuicideprevention.org/task-force/research-prioritization).
Goal prioritization was determined by ordering candidate Aspirational Goals by median summed scores across four rating criteria, and variance around the median was described via the interquartile range.11 Wilcoxon Matched-Pairs Signed-Rank or Mann–Whitney–Wilcoxon tests12,13 were used to detect significant changes in goal prioritization across rounds.
The goals in a research agenda should ideally represent independent scientific domains. To evaluate conceptual overlap among goals as scored by Survey respondents, a principal components analysis (PCA) was performed. Each survey participants’ “final” goal scores were created by tallying ratings across the four study criteria (i.e., potential burden reduction, projected ease and speed of real-world uptake, impact on vulnerable population groups, and acceptability to suicidal persons and family members). These final scores were entered into a PCA model that used orthogonal rotation to assess overlap between goals. A three-step structured decision-making model was used to determine the number of components in the final model.14 First, the eigenvalue of 1.0 criterion rule was considered, but was ultimately discarded because of the large number of factors just above and below that value. Next, common variance accounted for by each rotated component was used to determine the number of components that would be included in a model encompassing at least 80% of all variance. Finally, three interpretability criteria were applied to model results (i.e., within-component conceptual consistency, between-component construct difference, and the simple final structure rotated component pattern). For this exploratory analysis, rotated loadings with absolute values of 0.60 and higher were used to place goals within components.
Results
In total, 89.1% of survey respondents participating in the final three Survey rounds were between the ages of 25 and 64 years, and 89.5% were white (n=511). The overall sample was well educated, with 92.2% holding at least a bachelor’s degree. Women comprised approximately 63% of the overall sample, including 48.2% of the Researcher group and 81.4% of the Survivor group. Because rates of suicide are differentially distributed across the country, geographic representation was also considered. Approximately 23% of the sample resided in each of three U.S. geographic regions—the Midwest, South, or West—and 31% resided in the Northeast. Suicide researchers residing outside the U.S. comprised 19% of this group. Between 52.5% and 60.7% of groups other than Survivors identified themselves as having had close personal experience with a suicidal person.
There was substantial agreement between participant groups on final goal prioritization (Table 1). Prevention of reattempts, enhanced continuity of care, provider and gatekeeper training, and improved affordability, accessibility, and effectiveness of care were the highest research priorities. Between-group comparisons revealed that clinicians tended to rate research aimed at enhanced continuity of care higher than other participants (Z=1.73, p=0.08). Also, compared to other groups, researchers gave lower ratings to psychosocial interventions for those at risk and ways to reduce stigma (Z= −2.32, p=0.02 vs Z=−2.46, p=0.01, respectively). Survivors rated three research goals comparatively higher than other groups, including psychosocial interventions for those at risk (Z=2.32, p=0.02); improved continuity of care (Z=2.22, p=0.03); and stigma reduction (Z=2.46, p=0.02).
Table 1.
Candidate Aspirational Goals | Final prioritized ranking | Median score | Interquartile range |
---|---|---|---|
Aspirational Goal 6: Ensure that people who have attempted suicide can get effective interventions to prevent further attempts. (Prevention of reattempts) | 1 | 32 | 28335 |
Aspirational Goal 9: Ensure that people getting care for suicidal thoughts and behaviors are followed throughout their treatment so they don’t fall through the cracks. (Enhanced continuity of care) | 2 | 32 | 28335 |
Aspirational Goal 7: Ensure that healthcare providers and others in the community are well trained in how to find and treat those at risk. (Provider training) | 3 | 31 | 27334 |
Aspirational Goal 8: Ensure that people at risk for suicidal behavior can access affordable care that works, no matter where they are. (Access to affordable and effective care) | 4 | 31 | 27334 |
Aspirational Goal 4: Ensure that people who are thinking about suicide but have not yet attempted receive interventions to prevent suicidal behavior. (Psychosocial interventions for those at risk) | 5 | 30 | 26334 |
Aspirational Goal 10: Increase help-seeking and referrals for at-risk individuals by decreasing stigma. (Stigma reduction) | 6 | 30 | 25334 |
Aspirational Goal 1: Know what leads to, or protects against, suicidal behavior, and learn how to change those things to prevent suicide. (Risk and protective factor interactions) | 7 | 30 | 26334 |
Aspirational Goal 11: Prevent the emergence of suicidal behavior by developing and delivering the most effective prevention programs to build resilience and reduce risk in broad-based populations. (Population-based risk-reduction and resilience-building.) | 8 | 29 | 25333 |
Aspirational Goal 3: Find ways to assess who is at risk for attempting suicide in the immediate future. (Prediction of imminent risk) | 9 | 29 | 24334 |
Aspirational Goal 5: Find new biological treatments and better ways to use existing treatments to prevent suicidal behavior. (Improved biological interventions) | 10 | 27 | 22331 |
Aspirational Goal 12: Reduce access to lethal means that people use to attempt suicide. (Reduction of access to lethal means) | 11 | 27 | 21332 |
Aspirational Goal 2: Determine the degree of suicide risk (e.g., imminent, near-term, or long-term) among individuals in diverse populations and in diverse settings through feasible and effective screening and assessment approaches. (Population- and setting-based screening) | 12 | 23 | 18329 |
Median score refers to medial of summed scores on four criteria.
A three-factor solution accounted for 50% of dataset variance whereas a seven-factor solution accounted for 80.9% of all variance. In order of total loading, the components from the seven-factor solution were (1) improved treatments/strategies for maintaining at-risk individuals in care; (2) better ways to modify population-level risk; (3) improved ways to encourage help-seeking among at-risk individuals not under care; (4) improved biological treatments; (5) more affordable and accessible care models; (6) improved provider training; and (7) improved treatment for suicide ideators. Factor analysis suggested that the goals related to prevention of reattempts and enhanced continuity of care reduced to the same underlying construct, whereas all other categories listed above were populated by one conceptually discrete Aspirational Goal.
Discussion
The RPTF’s approach to building a research agenda was based on at least two underlying assumptions that deserve close examination. First, the RPTF assumed that the very best suicide prevention research agenda is one in which agenda goals prioritize the high-quality scientific activities with the greatest potential to rapidly and significantly reduce the number of U.S. suicidal acts. Second, the RPTF approach assumed that selecting research goals with this level of impact for a public health challenge as recalcitrant and complex as suicide would require multiple inputs, including broad-based feedback from diverse constituent groups. The RPTF’s Stakeholder Survey was designed to solicit input from four major groups with a strong investment in preventing suicide, including suicide survivors, healthcare providers, policymakers/administrators, and research scientists.
Survey results suggested that substantial numbers of individuals within each of these groups were interested and willing to participate. Respondents opted for a practical, “boots-on-the-ground” research agenda. Aspirational Goals coming out of the process did not appear to include substantial conceptual overlap. Highest-rated research targets included prevention of reattempts, enhanced continuity of care, models for provider and gatekeeper training, and strategies to make care more affordable and accessible. All final aspirational goals suggested a corresponding research pathway leading to an enhanced capacity to reduce suicidal behavior. Even with full funding, the goals vary in terms of both the type of research and time to goal achievement, ranging from those requiring early-phase developmental efforts to those where implementation research is the next step in realizing the aspiration. As a part of the RPTF process, a large team of research scientists worked to build sequenced research pathways by which to achieve each goal, and results from the work on aspirational goal research pathways are presented in the online supplement to this issue of AJPM.
How do these Stakeholder Survey results compare to findings associated with other suicide prevention efforts? Recently, at least three other major initiatives have emphasized the need for additional suicide prevention research. The United Kingdom’s clinical guideline for longer-term management of self-harm15 emphasizes the need for “well-powered” effectiveness trials to assess the utility of specialized healthcare provider training as well as production of validated risk assessment instruments. The U.S. Department of Veteran Affairs’ Evidence-Based Synthesis Program review on risk factors and assessment tools again noted a “striking lack of assessment tool evaluation research” conducted among Veteran and military populations. It called for improved study design in future risk factor research, more work to enhance continuity of care, and increased access to rehospitalization after non-fatal attempts.16 Finally, Australia’s Assessing Cost-Effectiveness in Prevention Project (ACE) highlighted the value of research on both problem-solving therapy and depression screening as potentially cost-effective methods of suicide prevention.17 Taken together, these initiatives address eight of the 12 Stakeholder Survey Aspirational Goals.
Formidable unsolved conceptual, logistic, and ethical barriers not found in other lines of prevention science exist in suicide research, and clinical research paradigms are not easily adapted.18–20 These issues were not addressed directly at this early stage in the agenda development process. In addition, we were unable to find a sampling strategy that would capture the voices of a national suicide prevention constituency in a truly representative manner, and the sample may therefore have been biased in unknown ways. Although most participants in the final three rounds were represented in the analyses, there were a substantial number who completed fewer than four total rounds, and this may likewise have biased results in ways we cannot characterize. A limitation of the central tendencies analytic strategy used here is that it seeks the most commonly voiced Survey suggestions, which may or may not be the most innovative, informed, or promising approaches; additional analyses address Survey findings using other analytic approaches designed to address this concern. The survey methods themselves and online format enhanced accessibility for many but created technical difficulties for others, and further exploration of these logistic issues is scheduled for a future publication.
The RPTF’s agenda-development process represents the first large-scale effort to mount a coordinated suicide prevention research effort in the U.S. It is built on the assumption that stimulating high-yield suicide research will involve more than production of a static document outlining any given set of scientific activities, regardless of how visionary those activities might be. It will require a constituent audience willing to engage in ongoing, national dialogue about research priorities. It is the RPTF’s belief that only through such a process can measurable, results-oriented scientific advancements with substantial burden reduction potential be conceptualized, executed, and ultimately brought to scale.
Supplementary Material
Acknowledgments
The National Action Alliance for Suicide Prevention Research Prioritization Task Force is composed of members from both private organizations and the Federal government and was founded to support progress in suicide prevention research in the U.S. Funding for the Stakeholder Survey was provided by organizations represented on the Task Force, including the American Foundation for Suicide Prevention, Jed David Satow Foundation, American Association of Suicidology, Suicide Awareness Voices of Education, and Saul Feldman (private donor). In addition, the RAND Corporation underwrote part of the Survey. Individuals affiliated with these organizations helped to design and conduct the Survey and analyze results. Beyond these individuals, the authors wish to extend their thanks to Sarah Brown, DrPH, Assistant Professor in the Department of Psychiatry, University of North Texas Health Sciences Center, Fort Worth TX; T. Michael Kashner, PhD, JD, Director of the VA Center of Excellence for Graduate Medical Education and Advanced Biostatistics, Loma Linda CA; and Terrance Savitsky of the RAND Corporation, Santa Monica CA for their generous donation of time and expertise in the development and reporting of the analytic approach used in this paper. In addition, the authors are deeply grateful to those individuals who were willing to serve as moderators during the Discussion Round of this project: Belinda Sims, Mercedes Rubio, Amy Goldstein, Jovier Evans, Peggy West, Benedetto Vitiello, Katrina Bledsoe, Gayle Jaffe, Ann Haas, and Keisha Shropshire. Finally, the authors wish to thank Jacob Solomon of RAND and Trevor Summerfield of the American Foundation for Suicide Prevention for technical assistance with the online survey process. The views presented are those of the authors and do not necessarily represent the views of the NIH or USDHHS.
Dmitry Khodyakov, PhD, worked for RAND Corporation in development of the ExpertLens™ tool and Siddhartha Dalal, PhD, worked for RAND Corporation with oversight for ExpertLens development at the time of the Survey. No financial disclosures were reported by the other authors of this paper.
Appendix. Supplementary data
Supplementary data associated with this article can be found at http://dx.doi.org/10.1016/j.amepre.2014.01.004.
Footnotes
Annual rates of suicide: 1950, 11.3 per 100,000 U.S. citizens; 2010, 12.4 per 100,000 U.S. citizens. Sources: Statistical abstracts of the United States. US Census Bureau: Washington DC, 1950; .CDC. Web-based Injury Statistics Query and Reporting System (WIQARS) [Online database] National Centers for Injury Prevention and Control: Atlanta GA, 2010.
The working group was created in June, 2010 by representatives of the National Council for Suicide Prevention (NCSP), the National Institute of Mental Health (NIMH) and the Substance Abuse and Mental Health Service Administration (SAMHSA) for the purpose of developing an agenda that could guide U.S. suicide prevention research efforts.
References
- 1.CDC. Web-based Injury Statistics Query and Reporting system (WIQARS) doi: 10.1016/j.jsr.2017.01.001. cdc.gov/ncipc/wisqars. [DOI] [PMC free article] [PubMed]
- 2.National Action Alliance for Suicide Prevention. Public-Private Partnership Advancing the National Strategy for Suicide Prevention. dev.actionallianceforsuicideprevention.org/
- 3.National Action Alliance for Suicide Prevention. Research Prioritization Task Force. Washington DC: National Action Alliance for Suicide Prevention; 2011. [Google Scholar]
- 4.Research Task Force. The agenda development process of the U S ’ National Action Alliance for Suicide Prevention Research Prioritization. Task Force Crisis. 2012;34(3):147–55. doi: 10.1027/0227-5910/a000208. [DOI] [PubMed] [Google Scholar]
- 5.Tichy G. The Over-optimism among experts in assessment and foresight. Technol Forecast Soc Chang. 2004;71(4):341–63. [Google Scholar]
- 6.MacRae D, Whittington D. Expert advice for policy choice; analysis and discourse. Washington DC: Georgetown University Press; 1997. [Google Scholar]
- 7.Clark W. Introduction and overview. In: Clark W, Matson P, Dickson N, editors. Linking knowledge with action for sustainable development. Washington DC: National Academy of Science; 2008. [Google Scholar]
- 8.Libel L. The challenge of governing knowledge systems for sustainable development. In: Clark W, Matson P, Dickson N, editors. Linking knowledge with action for sustainable development. Washington DC: National Academy of Sciences; 2008. [Google Scholar]
- 9.Matson P. The challenge of promoting use-inspired fundamental research: connecting science, society and managers. In: Clark W, Matson P, Dickson N, editors. Linking knowledge with action for sustainable development. Washington DC: National Academy of Sciences; 2008. [Google Scholar]
- 10.Dalal S, Khodyakov D, Srinivasan R, Straus S, Adams J. ExpertLens: a system for eliciting opinions from a large pool of non-collocated experts with diverse knowledge. Technol Forecast Soc Chang. 2011;78(8):1426–44. [Google Scholar]
- 11.Dixon W, Massey F. Introduction to statistical analysis. 3. New York: McGraw-Hill; 1969. [Google Scholar]
- 12.Wilcoxon F. Individual comparisons by ranking methods. Biomets Bull. 1945;1(6):80–93. [Google Scholar]
- 13.Siegel S. Non-parametric statistics for the behavioral sciences. New York: McGraw-Hill; 1956. [Google Scholar]
- 14.Principal component analysis. 2012 http://support.sas.com/publishing/pubcat/chaps/55129.pdf.
- 15.National Collaborating Centre for Mental Health. National Clinical Guideline Number 133: Self harm: longer-term management. London England: National Collaborating Centre for Mental Health; 2012. guidance.nice.org.uk/CG133/Guidance/pdf/English. [Google Scholar]
- 16.Haney E, O’Neil M, Carson S, et al. Evidence-based synthesis program. Quality Enhancement Research Initiative. Portland OR: Department of Veterans Affairs Veterans Health Administration; 2012. Suicide risk factors and risk assessment tools: a systematic review. [PubMed] [Google Scholar]
- 17.Vios T, Carter R, Barendregt J, et al. ACE-prevention. Brisbane St. Lucia Queensland Australia: University of Queensland; 2010. Assessing cost-effectiveness in prevention. [Google Scholar]
- 18.Geddes J. Suicide and homicide by people with mental illness: we still don’t know how to prevent most of these deaths. BMJ. 1999;318(7193):1225–6. doi: 10.1136/bmj.318.7193.1225. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Moscicki E. Epidemiologic surveys as tools for studying suicidal behavior: a review. Suicide Life Threat Behav. 1989;19(1):131–46. doi: 10.1111/j.1943-278x.1989.tb00372.x. [DOI] [PubMed] [Google Scholar]
- 20.Pearson J, Stanley B, King C, Fisher C. Intervention research with persons at high risk for suicidality: safety and ethical considerations. J Clin Psych. 2001;62(25S):S17–S26. [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.