Development and Pilot Test of the RAND Suicide Prevention Program Evaluation Toolkit

Short abstract

The RAND Suicide Prevention Program Evaluation Toolkit draws from the scientific literature to guide evaluations of suicide prevention programs. This study is a companion to the toolkit and provides background on its development and testing.

Abstract

Evaluations are critical for assessing the impact of U.S. Department of Defense investments in suicide prevention and can be used as the basis for decisions about whether to sustain or scale up existing efforts. The Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury asked RAND to draw from the scientific literature and create a toolkit to guide future evaluations of suicide prevention programs. The resulting toolkit is designed to help program staff determine whether their programs produce beneficial effects and, ultimately, to guide the responsible allocation of scarce resources. This article summarizes the three complementary methods used to develop the RAND Suicide Prevention Program Evaluation Toolkit: an examination of the peer-reviewed evaluation literature and clinical trials, a review of other evaluation toolkits, and feedback from staff responsible for implementing suicide prevention programs in the Department of Defense. It is intended to serve as both a companion and supplement to the toolkit and offers additional background on its development and testing.

In response to increasing suicide rates among military personnel, the U.S. Department of Defense (DoD) has implemented many policy changes and invested in a number of programs to prevent suicide within its ranks. The DoD Task Force on the Prevention of Suicide by Members of the Armed Forces (2010) and a subsequent RAND report (Ramchand et al., 2011) both recommended that DoD evaluate existing programs and ensure that new programs include an evaluation component when they are implemented. Evaluations are critical for assessing the impact of DoD investments in suicide prevention and can be used as the basis for decisions about whether to sustain, scale up, or discontinue existing efforts. The Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury (DCoE) asked RAND to draw from the scientific literature and create a toolkit to guide future evaluations of DoD-sponsored suicide prevention programs (SPPs). The overall goal of the toolkit is to help those responsible for SPPs determine whether their programs produce beneficial effects and, ultimately, to guide the responsible allocation of scarce resources.

Purpose of This Study

This study summarizes the methods used to develop the RAND Suicide Prevention Program Evaluation Toolkit; it is meant to serve as a companion to the toolkit itself and to provide additional background for those who are interested in learning about the toolkit's development.

Methods Used to Develop the Toolkit

We used three complementary methods to develop the toolkit: a peer-reviewed literature review, a review of other evaluation toolkits, and reviews and feedback by program staff responsible for implementing DoD-sponsored SPPs.

Literature Review

We conducted an extensive examination of peer-reviewed suicide prevention evaluation studies and clinical trials of suicide prevention activities. The literature review consisted of two steps. First, we identified relevant sources through three phases of web-based searches of peer-reviewed literature in content-relevant databases. These sources underwent successive rounds of screening, including a title and abstract review, followed by a full-text review, to exclude irrelevant and unsuitable articles. We focused on sources that described the evaluation of suicide prevention or reduction programs.

Second, we abstracted information from each source selected for inclusion (n = 166). We divided these sources into two tiers. We then coded the first-tier articles, which included evaluation studies of SPPs, and abstracted in a consistent way the pertinent data on both the evaluation details and the outcome measures. Evaluation data included descriptions of suicide prevention/reduction programs, details about evaluation design, and synopses of study findings. Measure data described how study outcomes were assessed and included details about measure administration, scoring, and reliability. Each piece of abstracted information represents a characteristic or quality of a program or measure that was useful to consider when constructing the toolkit.

Second-tier articles—those that were relevant to the toolkit's development but inappropriate for data abstraction (e.g., articles that described an evaluation methodology but that were not actual evaluations of SPPs)—were subsequently reviewed by the first author, who used information from these articles to inform the appropriate sections of the toolkit.

Review of Other Evaluation Toolkits

We used other evaluation toolkits to identify key components that should be included in our toolkit and to develop the format of the toolkit. We first reviewed existing evaluation toolkits to develop an outline and determine the types of tools that they offered. Tools identified included sample measures, checklists with yes/no questions, worksheets with open-ended questions to help guide users through the toolkit (i.e., through planning an evaluation, analyzing data, and using evaluation data to improve the SPP), and designs of prior evaluation studies. We relied, in particular, on the Getting To Outcomes® (GTO) approach because GTO is currently the only evidence-based model and intervention that has been proven to increase programs' ability to conduct self-evaluations (Acosta and Chinman, 2011).

Review and Feedback by Program Staff

We developed an initial draft of the toolkit and shared it with 12 program staff responsible for implementing SPPs in the Air Force, Marine Corps, and National Guard. Program staff were asked to spend five to six hours reading through the toolkit and completing the worksheets, checklists, and templates. We asked them to compile their feedback using a standardized feedback form (see the appendix to this report). The feedback form asked the staff member to indicate the extent to which each chapter of the toolkit met its objectives, whether there were any sections that were not clear or otherwise difficult to understand, and whether he or she felt uncomfortable using the tools provided. Program staff emailed their completed feedback forms to the RAND team and participated in a follow-up conference call to discuss their feedback.

Revisions to Toolkit Based on Program Staff Feedback

Based on program staff feedback, we made several improvements to the toolkit. For example, we added process measures in the body of the toolkit, identified the potential cost of outcome measures, and provided guidance on how to get more information about potential evaluation measures. Although several program staff suggested that the toolkit be converted to a more interactive online product, such a change was outside the scope of the current project. Therefore, no revisions of this nature were made to the toolkit.

Recommendations for Toolkit Dissemination

Based on program staff feedback, the RAND team made three recommendations for DCoE to consider as it develops plans for disseminating the toolkit.

1. Continue to refine the toolkit. Continued monitoring will help ensure that the content remains up-to-date and relevant to users.

2. Consider converting the toolkit to an interactive online format. This format would allow for an autofill feature to prepopulate information, making it unnecessary for program staff to transfer information by hand.

3. Continue partnering with the Suicide Prevention and Risk Reduction Committee and the Defense Suicide Prevention Office to disseminate the toolkit. Dissemination strategies could include webinars to introduce the toolkit, distributing copies of the toolkit at conferences that suicide prevention coordinators may attend, and distributing the toolkit to installation-level contacts via email.

Conclusion

The RAND Suicide Prevention Program Evaluation Toolkit was developed to build the knowledge and skills of individuals responsible for implementing DoD-sponsored SPPs as they self-evaluate those programs. Disseminating the toolkit across DoD will help ensure that ongoing and future SPPs have guidance in determining whether their programs are having beneficial effects and will ultimately help guide the responsible allocation of scarce resources.