Development of the Veterans Crisis Line Caring Letters Suicide Prevention Intervention

Abstract

Objective

To describe the design, implementation, and plans to evaluate the Veterans Crisis Line (VCL) Caring Letters intervention.

Data Sources

Veterans with VCL contact and VHA service utilization.

Study Design

Caring Letters is an evidence‐based post‐acute care suicide prevention intervention in which brief messages are mailed to individuals at high risk of suicide repeatedly over time to communicate that people care about them and are concerned for their well‐being. An effectiveness‐implementation hybrid type 1 trial using the RE‐AIM evaluation framework is underway to examine the use of Caring Letters with veterans who contact the VCL. A team of suicide prevention subject matter experts, researchers, and operational partners from the VCL will evaluate the effects of Caring Letters on clinical outcomes and Department of Veterans Affairs – Veterans Health Administration (VHA) clinical utilization rates and examine facilitators and barriers to implementing the Caring Letters campaign.

Data Collection Methods

Veterans who contact the VCL are linked with national administrative VHA data. Semi‐structured interviews were conducted as part of a qualitative formative evaluation.

Principal Findings

In the first 12 months of the intervention, Caring Letters have been sent to over 100,000 veterans with VCL contact (over 500,000 letters mailed). A formative qualitative evaluation early in implementation revealed a variety of positive veteran perspectives on the intervention.

Conclusions

Partnered program design and evaluation with a high level of stakeholder engagement and participant feedback can result in a rigorous and feasible evaluation plan that improves implementation processes and produces actionable results. The initial results of this evaluation will be used to better inform care in the VHA and, specifically, the VCL.

What is known on this topic

• The Caring Letters Suicide Prevention Intervention has been recommended in some clinical practice guidelines after a psychiatric hospitalization for suicidal ideation or a suicide attempt.

• Although the potential value of the intervention for crisis line callers has been highlighted by the U.S. Surgeon General, Caring Letters have not been tested with this population.

What this study adds

• This is the first test of Caring Letters with crisis line callers.

• This is the largest evaluation of Caring Letters ever conducted.

1. INTRODUCTION

Suicide prevention is one of the Veterans Health Administration's (VHA) highest priorities. Suicide is the 10th leading cause of death in the United States, ¹ and veterans have suicide rates 1.5 times higher than the US non‐veteran adult population. ² As a result, the VHA encourages development of evidence to inform policies about strategies to reduce suicide risk among our nation's veterans. ³ One such strategy involves supporting Veterans Crisis Line (VCL) callers after their call. Since 2007, the VCL has responded to over 5 million calls. ⁴ Callers who are enrolled in VHA care have high rates of suicide within 1 month of the call (797 per 100,000), and rates decline but remain elevated through 12 months (298 per 100,000). ⁵ These rates are much higher than in the general VHA or civilian populations. ² New initiatives to continue suicide prevention efforts after VCL contact may be helpful.

One promising initiative is the Caring Letters Suicide Prevention Intervention, which has been associated with improved mental health outcomes in other settings but has not been previously studied as an intervention to follow crisis line contact. The goal of Caring Letters is to send brief messages that express concern for the well‐being of the individual and serves as a reminder that help is available. ⁶ The recipient receives several letters (commonly about 8) over time (often for a 1‐year period). Because Caring Letters can be mailed from a centralized location and can be administered with relatively few staff, this intervention is particularly well‐suited to reach thousands of veterans located all over the United States, who contact the VCL with varying levels of risk.

In its traditional form, Caring Letters is an evidence‐based intervention for post‐acute care ⁷ that can be paired with other prevention efforts as a part of a comprehensive strategy to prevent suicide. The Caring Letters intervention was first tested in the United States four decades ago ⁸ ; those patients randomized to receive Caring Letters following discharge from inpatient psychiatric treatment had significantly lower rates of suicide for the first 2 years of a 5‐year Caring Letters intervention compared with those who received no further contact after discharge. ⁹ Since this study was published, several clinical trials have tested different versions of the intervention in a variety of countries. An Australian postcard version exhibited mixed results, but was associated with significantly fewer repeat episodes of admission for self‐poisoning compared with controls. ¹⁰ Investigators in Iran adapted these caring cards for use in hospital‐treated self‐poisoning in Tehran. ¹¹ At a 24‐month follow‐up, patients randomized to the intervention had significantly lower rates of any suicidal ideation, and any suicide attempt. ¹² In New Zealand, there were no significant clinical benefits reported for postcards adapted to test the minimum number needed for effectiveness (6 sent over 1 year).

Two other trials reported null results. One was stopped because preliminary analyses reflected greater than expected benefits for the intervention. ¹³ In the final analyses, however, no group differences were noted. The other had a small sample and tested alternate content in the Caring Letters. ¹⁴

Caring Letters is one of the only interventions to ever demonstrate a reduction in suicide mortality in a randomized controlled trial, but the body of literature includes both positive and null trials. This is not uncommon, and interpretation often requires an independent, systematic review that carefully considers the quality of the evidence and provides recommendations. The VA/DoD Clinical Practice Guideline (CPG) for the Assessment and Management of Patients at Risk for Suicide reviewed the evidence for Caring Letters with a panel of multidisciplinary experts who rated the quality of the evidence and provided recommendations based on the strength of the evidence. ⁷ The CPG suggested offering caring communications “in addition to usual care after psychiatric hospitalization for suicidal ideation or a suicide attempt.” ^{7 (p28)} Although the intervention has not been studied for crisis line callers, the U.S. Surgeon General's recent Call to Action to Implement the National Strategy for Suicide Prevention recommended considering the use of Caring Letters when gaps in care may exist, including following crisis line calls. ¹⁵

In response to these recommendations, and with input from veterans, the VCL developed a Caring Letters project to support veterans who contact the VCL. The VCL also partnered with researchers and evaluators to develop evidence on whether Caring Letters reduce the risk of suicide behaviors in this setting – this evidence will inform VHA decisions on choice of suicide prevention strategies targeted to veterans who contact the crisis line. The next sections describe the development of the intervention, the initial roll‐out and formative evaluation results, and plans for analyses of changes in health care use and suicide behaviors associated with Caring Letters.

2. METHODS

2.1. Tailoring Caring Letters to the VCL setting

Draft templates for Caring Letter messages were developed for veterans who contact the VCL that aligned with the evidence base ¹⁶ , ¹⁷ and included recommendations from Caring Letters subject matter experts. Additional efforts were taken to tailor the intervention to VCL users.

Feedback from veteran stakeholders: Two Veteran Engagement/Advisory Boards were briefed on the project and asked to provide feedback related to plans for the intervention delivery, content, and design. While veterans did not come to consensus on some recommendations (i.e., recommendations to use “Hello” and first name vs. “Dear” and full name for salutation), the overall impression of the project was favorable. Some veteran stakeholder suggestions were not feasible to implement based on the scale of the project; for instance, recommendations to include personalization (i.e., handwritten cards) may be preferable but not practical in this case (and not necessary for the intended impact, given previous research ¹⁶ ). However, other feedback was noted and addressed, such as recommendations to keep the letters short and “get to the point up front.” Following revisions based on recommendations from the veteran advisory groups, a veteran staff member at VCL assessed the design and content of the letter templates to verify that they were appropriate.

2.2. Intervention roll out

Study Design: We used an effectiveness‐implementation hybrid type 1 trial ¹⁸ , ¹⁹ using the RE‐AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) ²⁰ , ²¹ evaluation framework to evaluate the effects of Caring Letters on key clinical outcomes while simultaneously examining the implementation of the intervention. ²² As a hybrid type 1 trial, the evaluation's primary goal is to test the effectiveness of a Caring Letter campaign to reduce suicide attempts among veterans who contact the VCL. We will conduct a pre–post analysis and compare outcomes between callers who receive Caring Letters relative to callers in the previous 2 years. We will also randomize veterans who contact the VCL to two Caring Letter signatories to test possible authors for this new population. Data collection for the quantitative effectiveness component of the evaluation is still ongoing at the time of this writing. Below, we focus on the formative evaluation.

The use of the RE‐AIM framework allows for ongoing systematic documentation of key project activities that are aligned with specific measurable objectives and outcomes. Moreover, it enables the study team to identify implementation barriers and facilitators as they emerge, and document adaptations made to improve implementation strategies for project delivery and sustainability. We are using RE‐AIM as our analytic framework to examine five dimensions: reach into the target population; effectiveness of the intervention; adoption by the setting; implementation consistency or fidelity; and maintenance over time. Table 1 summarizes how we define each RE‐AIM dimension reported in this manuscript.

TABLE 1.

Preliminary RE‐AIM dimensions with variable definitions and data sources

Measure	Definition	Data source
Reach	Number of veterans receiving Caring Letters Number of letters mailed Number of undeliverable addresses Number of opt‐outs	Administrative data
Implementation fidelity	Date cards initiated after VCL call (target is within 30 days)	Administrative data
Veteran perspective	Qualitative interviews focused on veteran perspective of Caring Letters	Qualitative interview
Staff perspective	Qualitative interviews focused on project staff perspective of Caring Letters	Qualitative interview

Abbreviation: VCL, Veterans Crisis Line.

The purpose of this project is to support internal implementation and evaluation efforts. This project was reviewed by the authorized program office and meets the criteria for classification as non‐research as described in VHA policy. ²³

Participants: Caring Letters were mailed to all veterans with VCL contact who use VHA services (and therefore have an identifiable address in a VHA database). These veterans were automatically enrolled in Caring Letters. Since veterans can contact the VCL in several ways (e.g., phone call, “warm transfer” from another telephone hotline, text message communication with the VCL), a VCL “caller” was broadly defined to include multiple forms of contact. Civilians, concerned friends, and family members calling on behalf of a veteran, and callers who died prior to mailing of the letters were excluded. Eligible callers were identified weekly, and then data files were generated and sent to a contracted printer who printed and mailed the letters. Individuals who called the VCL again after their initial enrollment were flagged so that they did not receive duplicate sets of letters.

Caring Letters Intervention: Messages were mailed in the form of a flat, one‐sided card (similar to a prior successful study). ²⁴ They were sent monthly for the first 4 months after VCL contact and then every other month thereafter for a total duration of 1 year (messages mailed Months 1, 2, 3, 4, 6, 8, 10, 12); an additional card was mailed for Veterans Day to meet veterans' preferences. ²⁵ The cards and envelopes were designed to resemble a greeting card in size and color (e.g., light blue envelope). A mental health resource card was included with each letter containing information about how to access VHA resources (including VCL).

We randomly assigned veterans to receive cards signed by either a provider or peer signatory. In a traditional Caring Letters intervention, the cards would be signed by a provider with an established relationship with a patient. However, not every caller has an established relationship with a provider, and VCL responders do not have ongoing clinical relationships with veterans. In our evaluation, the provider signatory is a mental health provider working with the VCL who the veteran has not met. For the second approach, the signatory is a VCL peer veteran employee. Given the unique cultural characteristics of veterans, ²⁶ , ²⁷ peer veteran signatories have generated interest in other VHA Caring Letter initiatives and have positive pilot data associated with their use. ²⁸ While these approaches may not have some of the advantages conferred by a personal relationship, they attempt to strike a balance between scalability and consistency with theoretical mechanisms. ¹⁷ Furthermore, in our veteran preferences pilot study, most high‐risk veterans stated that they agreed or strongly agreed that they would like to receive Caring Letters from either of these individuals (and most of the remaining responses were neutral). Both signatories are real people who provided input into the final content used in the Caring Letters. The content of the cards is similar across signatories but has been optimized for each role (Figure 1). Randomization was conducted using permuted block randomization (blocks of four patients to two conditions), stratified by gender.

FIGURE 1.

Examples of peer and provider cards. (A) Peer Card; (B) Provider card [Color figure can be viewed at wileyonlinelibrary.com]

A phone number (VA411) was included in every message that veterans could call if they had questions about the project, VA services, or resources. ²⁹ The number was recently launched as a single access point for VA ²⁹ and was designed for providing resources to veterans who do not know the right number to call for VA‐related questions or assistance. In the Caring Letters intervention, the number was included as a resource to provide the option of live agents who can respond to a wide variety of veteran needs, 24 h a day, 365 days a year.

To opt out of the intervention, veterans could either call the VA411 phone number provided or call VCL directly. Veteran opt‐out requests were added to the master datafile, so no future Caring Letters were mailed.

Given that names and addresses were pulled from veterans' medical records (where information is initially hand‐entered), it was determined early on that these variables needed individual review by staff to ensure that the names and addresses did not contain obvious errors. Errors were corrected upon initial review and algorithms were created to ensure those text errors were not regenerated in future mailings.

To increase efficiency, no return address was included on the letter envelopes. Instead, all undeliverable mailings were sent through the United States Postal Service Secure Destruction program. This program secured all undeliverable Caring Letters, logged the unique bar codes, securely destroyed the letters, and provided the project team with information to update the master datafile to avoid future mailings with an incorrect address. Multiple data sources were used to identify deaths to avoid sending Caring Letters to deceased veterans.

Formative Evaluation: The primary goals of the qualitative formative evaluation were to gain an initial sense of the range of veteran perspectives about Caring Letters as a follow‐up to contact with the VCL and to collect feedback from recipients for improving the intervention.

A post‐implementation (summative) evaluation is also planned for the future. At that time, feedback will be collected on intervention satisfaction, perceived helpfulness of messages, the number of letters received, whether recommended resources were used/helpful, and general feedback and suggestions about the project.

The methods for the formative evaluation are described next.

Recruitment: Veterans enrolled in the early months of the project were sent opt‐in letters describing the purpose of the formative evaluation and inviting them to participate in a telephone interview. These early cohorts were targeted for recruitment at the intervention midpoint as they would have received approximately half the total letters (4–6 letters). Of 20,053 individuals 3067 were female (15.29%) and 16,986 male (84.71%). Using simple random sampling, 75 males and 25 females were selected to receive opt‐in letters. Veterans were provided contact information they could call if they were interested in volunteering or had questions about the project. After a limited response from the initial cohort, a second larger sample was generated (225 males, 75 females). Ten veterans volunteered to participate and seven (N = 7) completed a telephone interview.

Data Collection: A PhD‐level research health investigator with a background in anthropology (THA) and research scientist and Army veteran with a background in social work (JAW) led data collection. On the day of the interview, the researchers called eligible veterans, quickly summarized data collection procedures and participants' rights to decline to answer questions or end the interview at any time and obtained consent to audio record. To ensure consistency in data collection, the researchers used a semi‐structured interview guide. The guide contained questions developed to elicit responses pertaining to five broad domains of interest informed by the goals of the evaluation: (1) the impact of having received Caring Letters; (2) feedback on message content; (3) general perspectives about the usefulness of Caring Letters; (4) feedback on the resource list; and (5) recommendations for improvement.

In addition to collecting feedback from veteran recipients of Caring Letters, the formative evaluation also involved obtaining feedback from project staff. Data were collected during weekly team meetings in the form of minutes and telephone interviews with project staff (N = 4). As with veteran interviews, the researchers used a semi‐structured interview guide. The guide contained questions focused on (1) overall opinion of the Caring Letters project, (2) implementation challenges and how they were overcome, (3) what would have made implementation more successful, (4) project fit with existing VCL work processes, and (5) project alignment with VCL clinical care goals.

Analysis: Audio recordings from participants were transcribed verbatim. Data analysis was led by the PhD‐level research health investigator. The researcher analyzed the transcripts in individual templates using content analysis. ³⁰ The templates contained deductive domains informed by evaluation goals (i.e., impact of receiving Caring Letters, feedback on message content, etc.). The researcher developed inductive categories within each domain on the templates that reflected participants' perspectives about the intervention and subsumed impactful statements made during the interview beneath each category. ³¹ The researcher synthesized data from individual templates into a participant‐by‐domain matrix display. She then used constant comparison to identify the full range of responses for each domain. ³² The matrix had the additional advantage of minimizing bias by allowing the researcher to view all the qualitative data in one document (rather than relying on memory) and ensuring analytic consistency across interviews. ³³

Quantitative Evaluation Plans: Data collection related to a quantitative effectiveness evaluation are still underway. The results from this component of the evaluation will be reported elsewhere. To evaluate the impact of Caring Letters on patient outcomes and utilization rates, we will conduct a pre–post analysis and compare outcomes between callers who receive Caring Letters relative to callers in the previous 2 years. ³⁴ This analysis will attempt to answer the following three questions: (1) Are Caring Letters associated with a reduction in the incidence of VHA documented suicide attempts? (2) Are Caring Letters associated with reductions in the incidence and frequency of VHA psychiatric hospitalizations in the post period? (3) Are Caring Letters associated with increased engagement in VHA healthcare? Outcomes include incidence and frequency of any VHA health care service use, psychiatric hospitalizations, outpatient mental health visits, and emergency department (ED) visits; incidence and frequency of suicidal ideation; incidence and frequency of non‐fatal suicide events; all‐cause mortality; and suicide mortality. We also plan to conduct a formal impact evaluation of the Caring Letter signatory type on outcomes by leveraging the randomized controlled design of the evaluation.

To address these goals, we will examine secondary VHA data collected using records that are part of routine care and project delivery data. Data on calls and symptoms reported by callers will be obtained from the VHA records. Data on veteran sociodemographic and clinical characteristics, VHA health care use, and all‐cause mortality will come from the VHA Corporate Data Warehouse (CDW). Suicide mortality data will be obtained from the VHA Mortality Data Repository. Additional data on suicide attempts and ideation will be obtained from the VHA Office of Mental Health and Suicide Prevention. Pre–post analyses of associations among outcomes and Caring Letters will not be able to completely rule out the possibility that observed differences in outcomes are due to unmeasured temporal changes coinciding with Caring Letters. Letters began to be mailed out to VCL callers in June 2020, only 3 months after the start of COVID‐19‐related lockdowns. Therefore, it will be necessary to separate out changes in outcomes that are due to calls during the pandemic from changes in outcomes that are due to receipt of the letters. Plans to do so are detailed in the Appendix A.

Results of all analyses will be reported to VHA and Department of Veterans Affairs leadership and used to comply with requirements of the Foundations for Evidence‐Based Policymaking Act. ³

We will also examine the budget impact of implementing the Caring Letters project in the VHA from the health system's perspective. Specifically, we will examine how the costs of the intervention, outpatient utilization rates, and inpatient utilization rates impact change over a 12‐month period. We expect inpatient costs to decrease and outpatient costs to increase due to the intervention. We do not expect Caring Letters to serve as a substitute for any existing mental health outreach program.

3. RESULTS

3.1. Reach

The Caring Letters project began enrolling veterans who contacted the VCL on June 11, 2020. In the first 12 months of the VCL Caring Letters project, 102,709 unique veterans contacted the VCL, and 543,353 letters were mailed (Table 2). Veteran or health care provider initiated opt‐outs totaled 36 (representing 0.035% of total enrolled veterans) as of June 2021. Letters will continue to be mailed to veterans in the evaluation cohort through June 2022, at which point final reach metrics will be determined.

TABLE 2.

Reach metrics (as of June 10, 2021)

Metrics	Totals
Unique identifiable veterans who contacted the VCL (randomized to peer or provider letter) a	102,709
Deceased before upload for mail processing	344
First Letter scheduled to be mailed	102,365
Deceased between upload for mail processing and 14 days after scheduled mail date	511
Missing/bad address identified between upload for mail processing and 30 days after scheduled mailing date	150
Veterans with first letter mailed; no evidence of death within 14 days of scheduled mail date, no bad address within 30 days of scheduled mail date	101,704 b , c
Letters mailed (to date)	543,353

^a Contact with VCL, successfully matched to CDW record (enrolled in VHA care).

^b Includes 36 individuals who later opted out of letter receipt. Our primary analyses will include these individuals as part of an intent‐to‐treat analysis.

^c Includes 215 individuals who died between 15 and 30 days after the scheduled mail date for the first letter. Our primary analyses will include these individuals as part of an intent‐to‐treat analysis. Sensitivity of results to their inclusion/exclusion will be evaluated.

3.2. Implementation

Fidelity: In the first year of delivery, the VCL Caring Letters project achieved a high degree of implementation fidelity. Veterans were mailed their first letters on average 26 days from the date of their call. This was below our target 30‐day mark for initiating follow‐up.

3.3. Formative evaluation

The characteristics of veterans who participated in interviews are described in Table 3.

TABLE 3.

Characteristics of veterans interviewed for the formative evaluation

Study ID	Age	Gender	Race	Type of Caring Letter
P02	60	Male	White	Peer
P03	66	Male	White	Peer
P04	55	Male	White	Provider
P06	Declined	Female	African American	Peer
P08	65	Female	White	Peer
P09	49	Male	White	Provider
P10	57	Male	White	Provider

Veteran feedback: During telephone interviews for the formative evaluation, veteran participants described having experienced numerous life challenges that led them to reach out to the VCL for support (e.g., social isolation, personal loss, a lack of social support, trauma from military sexual assault, legal difficulties, and feeling abandoned or betrayed by the VHA). Analysis of the interview transcripts revealed six categories related to the impact of having received the letters, including (1) a sense of security, (2) a renewed faith in the VHA and/or sense of hope, (3) a sense that somebody cares or of being heard, (4) a sense of comradery or of belonging to something bigger, (5) a sense of purpose or meaning, and (6) enjoyment at having received an unexpected surprise. Table 4 contains excerpts from the interviews illustrating each category. As illustrated, the categories were often cross‐cutting (i.e., more than one category emerged during the analysis of the same statement).

TABLE 4.

Inductive categories with excerpts from the formative evaluation veteran interviews

Inductive categories	Illustrative excerpts
Sense of security	“It felt good and comforting to know that my fellow brothers were there for me and the VA system.” (P03) “It's a sense of security knowing that if anything should arise unexpectedly, that all I had to do with just pick up the phone and make one call to help sort it out.” (P10)
Renewed faith in the VHA and/or sense of hope	“Previous to me getting the cards, I had felt abandoned [by the VHA]. I felt refreshed, like the system is working.” (P10) “It might give a person a moment of pause to know that it's not all bad, and there are other people out there, and there's resources.” (P03)
Somebody cares/Being heard	“I do not know what it is for others, but for me it had just a slight bit of somebody actually cares out there.” (P02) “Getting a letter made me feel like maybe someone's finally listening to me.” (P08)
Comradery/Belonging to something bigger	“It felt good and comforting to know that my fellow brothers were there for me and the VA system.” (P03) “He [the Veteran signatory] did remind me of the connected bonds, and I'll tell you something: I kind of like getting the cards.” (P06)
Sense of purpose or meaning	“It gave me a sense of purpose, like: ‘Okay, if I need to get and talk to this person…’” (P02) “I felt encouraged, and I felt refreshed that my life does have meaning, and that there are people charged with making sure my needs are met.” (P10)
Enjoyment	“It's a pleasant and an unexpected surprise to receive that in the mail.” (P10)

Note: (P#) indicates the patient identifier for the quote (also used in Table 2).

Participants expressed a wide range of perspectives regarding the usefulness of receiving Caring Letters after contacting the VCL. A few participants expressed the belief that a letter was insufficient or ineffective as a follow‐up, particularly for veterans in crisis. In this respect, one participant stated, “I think you need to get in there and talk. Actually be with them and talk to them on the phone, one‐on‐one” (P02). Most participants, however, approved of the letters as a follow‐up. For example, one veteran stated, “To me, it was a very adequate and necessary response” (P10). Another said, “It might give a person a moment of pause to know that it's not all bad […] and there's resources, and you don't need to do what you're thinking about doing” (P03).

Two participants described having availed themselves of the resources list. One stated, “I remember at the beginning of my crisis within 10 days I received a notice (Caring Letter)… And I wasn't able to slip through the cracks. I was immediately able to seek out resources” (P10) One unexpected finding was that most participants (five of seven) kept some of the letters and/or the list of resources, including those who were more critical of the intervention (i.e. a participant [P09] who described the cards as “ineffective”). One veteran shared that “I kept them, I appreciate it, and saved 'em for me and for others to view along with all the other condolence cards that were sent [after my wife passed away]” (P03).

Participants were also asked for recommendations to improve the Caring Letters project. These recommendations (Table 5) will be considered further as additional evaluation data are analyzed.

TABLE 5.

Recommendations obtained through veteran interviews

Recommendations
Print the resource list on the back of the card to save VHA resources and decrease the environmental impact of printing letters. (P06)
Include a list of resources for Veterans who just want to talk with someone. (P03)
Have a provider follow‐up with Veterans by telephone if you do not hear from them after the first few letters. (P02)
Use more intensive follow‐up with Veterans who have expressed suicidal thoughts (rather than merely feeling isolated or stressed). (P09)
Continue sending multiple cards to the same veteran to ensure that they “stick.” (P04)
Encourage Veterans to have a friend or family member reach out for additional support on their behalf if they are unable to do so on their own. (P10)

Note: (P#) indicates the patient identifier for the quote (also used in Table 2).

Abbreviation: VHA, Veterans Health Administration.

Project staff feedback: Barriers and facilitators to implementation were identified in formative evaluation interviews with project staff. Barriers to implementation included (1) the perceived simplicity of the intervention (“You put a letter in an envelope and how complicated can that be? But, it's so complex…”), (2) the innovativeness of the project, and (3) cumbersome and inefficient processes (using email solutions to communicate when other information technology solutions would be more efficient). Facilitators to implementation included (1) communication and collaboration, (2) knowledge and skills (including the nature of the team, which includes suicide experts, data management experts, statisticians, qualitative evaluation experts, VCL leaders), and (3) project “fit” with existing processes and priorities.

4. DISCUSSION

To our knowledge, the VCL Caring Letters project represents the first large‐scale implementation of the Caring Letters intervention for a crisis line, an approach that was recently recommended by the US Surgeon General. ¹⁵ As far as we are aware, this is also the largest evaluation of the Caring Letters intervention to date. The initial results of the evaluation demonstrate the tremendous reach of the VCL Caring Letters intervention. In 1 year, over 100,000 veterans at elevated risk of suicide were mailed the intervention. Formative results to date suggest that letters are positively perceived, and summative results will be forthcoming.

The extremely low opt‐out rate strongly implies that the intervention is perceived as at least “neutral” to veterans (if not positive). Every card has a phone number listed that veterans can use to opt out of letters. The opt‐out rate is less than one‐tenth of 1% of veterans.

Key stakeholder interviews with veterans provided qualitative data about the perceived impact of having received Caring Letters from recipients. Although some veterans wondered if a simple letter could help veterans in crisis, others described feeling cared for, experiencing a renewed sense of faith in the VHA or connection to others, and/or using the resources that were included with the cards. Although it was a small sample, it is notable that most veterans (five of seven), including those who were more critical of the intervention, reported saving some cards and/or the resources.

The results of the evaluation also highlight the complexities of data management and the significant work required to ensure smooth implementation processes and intervention fidelity. Caring Letters is often described as a “simple intervention;” while this is surely true compared with other evidence‐based suicide prevention interventions (such as implementing a new evidence‐based psychotherapy), results from the formative evaluation suggest that the complexities and importance of dedicated resources across several disciplines should not be underestimated. Our budget impact analysis will quantify these resources.

Although Caring Letters is an evidence‐based practice for post‐acute care, this is a new population for the use of the intervention, and the two signatories are analogs to the traditional approach in which the recipient has at least met the signatory. While it is encouraging that veteran's cited traditional Caring Letters mechanisms during their interviews in the formative evaluation (e.g., feeling cared for), the results of the full hybrid type 1 evaluation will be critical for determining effectiveness for this patient population. The effectiveness of Caring Letters for veterans is largely unknown. We are aware of only two clinical trials that previously tested Caring Letters in military or veteran populations. ³⁵ , ³⁶ One did not achieve recruitment goals (no firm conclusions about efficacy could be made), ³⁶ and the other reported mixed results (some outcomes positive, some null). ³⁵ Therefore, the full results from the current evaluation are important for the VCL's work with veterans.

5. LIMITATIONS

The evaluation was designed as a mixed‐methods approach, but the intervention will not be completed by our evaluation cohort until June 2022, therefore many key data collection goals are not completed at this time. The formative evaluation sample was small. The goal of these interviews is to obtain a range of perspectives to improve quality improvement (not to recruit a large, generalizable sample). However, the demographics of the veterans who were interviewed was quite restricted (e.g., race) and it is possible we are missing some important perspectives. These data are also limited by the fact that participants only included veterans who volunteered to participate. If perspectives differ for veterans who did not choose to participate, we may be missing information that would be helpful to improving acceptability. A summative evaluation is underway that will include larger and more gender and racially and ethnically diverse sample.

6. CONCLUSIONS

Despite the limitations, the initial results from this evaluation demonstrate the significant reach of VCL Caring Letters, an extremely low opt‐out rate, and some helpful veteran perspectives. The early stages of the evaluation support the feasibility of Caring Letters on a very large scale with this crisis line population. This project has also demonstrated that partnered program design and evaluation with a high level of stakeholder engagement and participant feedback can result in a rigorous and feasible evaluation plan that can be used to improve implementation processes, produce actionable results, and inform VHA policy.

APPENDIX A.

A.1. Randomization

Randomization was conducted using permuted block randomization, stratified by gender. Randomization occurred within blocks of four patients to two conditions. This setup allowed for six permutations (1‐AABB, 2‐ABAB, 3‐ABBA, 4‐BAAB, 5‐BABA, 6‐BBAA).

After callers were successfully matched to CDW data, they were stratified by gender. Within each stratum, a uniform random number was generated in SQL (range 0–1) that corresponded to each block of four patients. If the random number, R, was ≥0 and <1/6, the block of four callers was assigned to permutation 1 (order of assignment to letter type is AABB). If 1/6 ≤ R < 2/6, the block of four callers was assigned to permutation 2 (order of assignment to letter type is ABAB). The same pattern followed for permutations 3–6.

A.2. Analysis 1: Pre–post comparison

We will analyze the differences in proportions of VCL callers who engaged with any outpatient VHA health care before and after the Caring Letters campaign with a chi‐square test. Similarly, chi‐square tests will be used to examine differences in incidence of suicide attempts, incidence of psychiatric hospitalization, and incidence of mental health outpatient visits. Differences in frequency of hospitalizations and mental health outpatient visits will be examined with Wilcoxon rank‐sum tests.

Pre–post analyses of associations among outcomes and Caring Letters will not be able to rule out the possibility that observed differences in outcomes are due to unmeasured temporal changes coinciding with Caring Letters. Letters began to be mailed out to VCL callers in June, 2020, only 3 months after the start of COVID‐19‐related lockdowns. Therefore, it will be necessary to separate out changes in outcomes that are due to calls during the pandemic from changes in outcomes that are due to receipt of the letters. To isolate the effect of the letters from the effect of the pandemic, we explore the use of a differences‐in‐differences‐in‐differences (or ‘triple differences’) approach: we will compare changes from before to after a VCL call among (a) veterans who received letters (called between 6/20 and 6/21), (b) veterans who did not receive letters but called during the beginning of COVID lockdowns (called between 3/20 and 5/20), and (c) veterans who did not receive letters and called a year before COVID lockdowns (allowing us to observe outcomes in a pre‐pandemic period; called between 6/18 and 3/19). Sensitivity of results to inclusion of a fourth group (those who called between 4/19 and 2/20 – who had part of their post‐call period during the beginning of COVID) will be evaluated.

In addition, we will run triple differences analyses that compare changes in outcomes from pre‐to‐post‐crisis line contact among individuals in different phases of the pandemic, such as vaccine availability – we will compare changes in outcomes from pre‐to‐post‐crisis line contact among individuals (a) who received Caring Letters after vaccines were widely available, (b) who received Caring Letters before vaccines were widely available, and (c) who did not receive letters and called before COVID. If we observe differences among groups A and B, this will suggest that vaccine availability or a different temporal confounder are responsible for some of our observed effects.

To understand the impact of changes in COVID‐19 burden on our outcomes, we will include sensitivity analyses that compare outcomes among callers who received Caring Letters by strata of excess mortality of callers residence during the month of enrollment in Caring Letters.

For utilization measures, we will examine changes in monthly rates of outcomes from 12 months before a call to 12 months after a call among those who did and did not receive Caring Letters. For suicide events and mortality data, we will examine changes in quarterly rates of 12 months before a call to 12 months after a call among those who did and did not receive Caring Letters.

A.3. Analysis 2: Comparison of letter signatories

Dichotomous outcomes will be examined with logistic regression and counts of hospitalizations and visits will be examined with zero‐inflated Poisson or negative binomial models. Exploratory analyses of associations among Caring Letters and all‐cause and suicide mortality will be conducted with Cox regression (using time of VCL call as time 0). We will further isolate the impact of Caring Letter signatory on outcomes by controlling for caller sociodemographics and mental health diagnoses.

Reger MA, Lauver MG, Manchester C, et al. Development of the Veterans Crisis Line Caring Letters Suicide Prevention Intervention. Health Serv Res. 2022;57(Suppl. 1):42‐52. doi: 10.1111/1475-6773.13985