Assessing variability and implementation fidelity of suicide prevention safety planning in a regional VA Healthcare System

Abstract

Background

In 2008, the Veterans Health Administration (VHA) implemented the use of safety planning for suicide prevention. A safety plan is a list of strategies, developed collaboratively with a provider, for a patient to use when suicide risk is elevated. Despite the use of safety plans in VHA, little is known about implementation fidelity, the extent to which safety plans are delivered as intended, or patient-level outcomes of safety planning.

Aims

This study aims to explore the implementation fidelity of safety planning in a regional VHA hospital and examine the associations between safety plan quality and completeness with patient outcomes.

Method

A comprehensive chart review was conducted for patients who were flagged as high risk for suicide (N= 200). Completeness and quality were coded, as well as information about patient and provider interactions regarding safety plan use.

Results

Safety plans were mostly complete and of moderate quality, although variability existed, particularly in quality. Limited evidence of follow-up regarding safety planning was found in the medical charts. Higher quality was associated with fewer subsequent psychiatric hospitalizations.

Conclusions

Variability in implementation fidelity and infrequent follow-up suggests a need for additional training and support regarding the use of safety plans for suicide prevention.

Introduction

Suicide among Veterans is a significant public health concern. Within the Veterans Health Administration (VHA), it is estimated that male patients are 1.66 times more likely to die by suicide than general population males, and female patients are 1.87 times more likely to die by suicide than general population females (McCarthy et al., 2009). Though the VHA identifies suicide prevention as a key priority, there is currently limited evidence in support of effective suicide prevention treatments for individuals at high-risk (Mann et al., 2005). For many years, the use of “no-suicide” contracts, verbal or written agreements that a patient makes with a provider to not engage in suicidal behaviors (Weiss, 2001), were a standard of care for suicide prevention. Increasing evidence, however, has found no-suicide contracts to be largely ineffective at preventing suicide attempts (Drew, 2001; Goin, 2003; Miller, Jacobs, & Gutheil, 1998; Range et al., 2002; Simon, 1999). In response to this evidence, the VHA introduced and required the use of safety planning in 2008 as an intervention for Veterans at high risk for suicide.

Safety planning, a component of a cognitive-behavioral therapy for suicide prevention, is a collaborative process in which the provider and patient list strategies for the patient to use when suicide ideation is elevated (Stanley & Brown, 2008). Its purpose is to reduce risk by providing the patient with personalized, appropriate, and specific coping strategies and contacts for use during times of crisis. Safety plans are organized in a series of steps, with steps 1–5 providing hierarchical and temporal methods of mitigating risk during a crisis (e.g., coping strategies, supports to contact), while step 6 outlines strategies that should be implemented prior to ideation in order to make the environment safer (e.g. removing drugs or firearms from the home). The implementation of safety planning in VHA included dissemination of a structured template that outlined core safety plan elements (see Table 1), a manual, and local trainings by suicide prevention coordinators. Additional guidelines were established as a part of the comprehensive strategy for suicide prevention. These guidelines required providers to schedule four outpatient mental health visits after a behavioral flag (which alerts providers when the record is opened) for high-suicide risk is entered into the patient’s electronic medical record, document the plan in the medical record system (Computerized Patient Record System; CPRS), involve the family in the Veteran’s care and safety planning when appropriate, and update the safety plan annually, among others.

Table 1.

Safety Plan template with examples

Step	Boilerplate (low quality)	Some evidence of personalization (moderate quality)	Strong evidence of personalization (high quality)
1. Warning signs (list three items)	Sad	Feel sad, cry	Feeling like I can’t stop crying or can’t stop being sad
2. Internal coping strategies (list three items)	Read	Read a book	Read “The Great Gatsby”
3. People and places that provide distraction (list two people & two places)	People: friend Places: public place	Bob mall	Bob (555)555-5555 Northwest Mall
4. People I can ask for help (list three people)	Friend	Bob	Bob (555)555-5555
5. Professionals and agencies (list two providers, one urgent care service, one Suicide Prevention Coordinator, VA Suicide Prevention Hotline)	VA Suicide Prevention Hotline	Dr. Bob Town Hospital VA Suicide Prevention Hotline	Dr. Bob (555)555-5555 Dr. Mary (333)333-3333 Town Hospital, 100 Hospital Way VA Suicide Prevention Hotline
6. Making my environment safe (list two items)	Alcohol	No alcohol in house	Dump out alcohol from liquor cabinets, garage

Despite these guidelines and their widespread dissemination throughout the VHA, little is known about how safety plans are implemented in routine care. One aspect of implementation, fidelity, comprises two key components: (1) adherence, the degree to which all elements of a treatment are delivered, and (2) competence, the level of skill with which the treatment is delivered (Carroll et al., 2007). While the relationship between fidelity and clinical outcomes is not well established for all interventions, several studies have demonstrated that fidelity to cognitive behavioral treatments of depression is associated with subsequent positive clinical outcomes. For example, Feeley, DeRubeis, & Gelfand (1999) found that greater protocol adherence was associated with subsequent symptom improvement. Similarly, Strunk et al (2010) found that greater therapist competence in the delivery of cognitive behavioral therapy for depression was associated with significant symptom reduction. Thus, measuring fidelity can be useful for understanding practice patterns in routine care setting, and can also facilitate a better understanding of whether, and to what degree, fidelity is associated with desirable treatment outcomes for individuals who are at high risk for suicide.

Given the structured format and collaborative nature of safety planning in the VHA, researchers may examine aspects of implementation fidelity by assessing the details of a Veteran’s medical record. Specific available indicators include: the presence or absence of safety plans, the level of completeness and the overall quality of documented safety plans, and expected outcomes associated with safety planning, including subsequent suicide attempts, psychiatric hospitalizations, and engagement in mental health care. This pilot study aims to examine the implementation fidelity and variability of safety plan use in a regional VHA hospital (VA Boston Healthcare System), and explores the associations between safety plan fidelity and clinical outcomes. We hypothesized that, while the requirement for a documented safety plan for all high-risk Veterans would be largely met, variability of safety plan fidelity would also exist as indicated by adherence to the steps of the safety plan (hereafter referred to as safety plan completeness) and the competence with which safety planning was conducted (hereafter referred to as safety plan quality). We also hypothesized that more complete safety plans with higher quality responses would be associated with more the positive clinical outcomes, such as reduced subsequent suicide attempts and psychiatric hospitalizations, and increased attendance of scheduled outpatient mental health visits.

Method

Sample

This study included 200 Veterans identified in the VHA Suicide Prevention and Application Network (SPAN) database, which was obtained by the research team in July 2013. The SPAN database includes VHA patients who were flagged as high risk for suicide within the last 12 months. Researchers received access to this data for all facilities in the VA Boston Healthcare System. Veterans were identified to be at high risk by a VHA healthcare professional if they met any of the following criteria: (1) engaged in a verified suicide attempt, (2) exhibited serious suicidal ideation requiring an immediate change in treatment plan, such as hospitalization, (3) presented warning signs such as threatening or planning to kill oneself, (4) seeking access to means, or other indicators that, in the clinician and suicide prevention coordinator’s clinical judgment, suggest that the Veteran is at elevated risk for suicide. To be eligible for inclusion in the current study, Veterans in the SPAN database must have received a high-risk flag for suicide in the medical record system. So that outcomes could be assessed over a one-year follow-up period, safety plans were only evaluated if they were completed before January 1, 2013. For the current pilot study, only the first (or index) safety plan in a patient’s chart was evaluated.

Procedure

Data were gathered through a retrospective chart review that assessed over 40 elements of Veteran’s CPRS chart, including, but not limited to, demographic information, mental health diagnoses listed in a designated section of the medical record, history of suicide attempts, psychiatric hospitalizations, and attendance of outpatient mental health appointments. A standardized abstraction tool developed by the investigators of this study, and available by request, was used as the primary method of data collection. Raters (JMG and MLT) were trained in coding procedures and overlapped on a preliminary sample of 15 charts as a part of an initial training phase. These medical records were used as training cases only and were not included in the final sample. During training, researchers met regularly to discuss discrepancies and develop and operationalize clear decision rules for coding. Throughout the remainder of the data collection phase, raters overlapped on an additional 24 cases to assess agreement. These cases were also reviewed and discussed as necessary to maintain rating consistency. Each agreement case had a designated primary rater whose ratings were included in the final analyses. The Institutional Review Board at the VA Boston Healthcare System approved and provided oversight for this study.

Measures

The index safety plan’s overall completeness (e.g., whether each of the six steps of the plan were completed and documented) and quality (e.g., the degree to which the safety plan items reflected multiple, detailed, and personalized responses) were rated. Additionally, each step of the safety plan was rated as an independent item for completeness and quality, with step 3 rated in two parts for “people” (3a) and “places” (3b; see Table 1). To represent overall adherence, completeness of the intervention was scored on a scale ranging from 0–2 (0=not complete, 1=partially complete, 2=complete) for each step. Total completeness was measured by summing the completeness scores for each of the six steps, including 3a and 3b, thus making the range of total completeness scores from 0–14. Quality was scored on a scale ranging from 0–3 (0=blank, 1=boilerplate, 2=some evidence of personalization, 3=highly personalized and specific). Quality of each safety plan was measured by calculating the total of the quality scores for each of the steps (min=0, max=21).

Information on the collaborative process between the Veteran and the provider during development of the safety plan, as well as information on the provider’s follow-up after safety plan creation, was also abstracted through chart review. These ratings included evidence of overall collaboration in creating the safety plan, evidence of a discussion in which the patient received instructions on how to use the safety plan, clinician inquiries about use of the safety plan in follow-up visits, and subsequent updates (prior to the required one-year update) or ongoing use of the safety plan in therapeutic interactions. For the purposes of our analyses, all collaboration and follow-up items were dichotomized into yes/no responses. Explicit evidence of a conversation within a clinical note was required for a “yes” rating.

Demographic and medical data obtained from a Veteran’s medical record was merged with the data obtained from the chart review and included information on age, race, history of mental health diagnoses, and suicide flag data. Additional data were abstracted through clinical notes and records to better understand clinical outcomes after receiving a safety plan. Specifically, clinical notes that were posted within the year after the index safety plan were reviewed to determine whether Veterans attended 4 or more outpatient sessions in the four to eight weeks after the episode of care in which they were identified as high risk (yes/no), had a subsequent psychiatric hospitalization (yes/no), and attempted suicide post-safety plan intervention (yes/no).

Statistical Analyses

Descriptive statistics were calculated to describe sample characteristics, assess safety plan fidelity, and to determine the presence or absence of specific provider-patient collaboration and follow up variables. To assess rater agreement, intraclass correlations were calculated for safety plan completeness and quality scores, and percent agreement was assessed for dichotomous variables. Six logistic regression analyses were performed analyzing total completeness and total quality as individual predictors of three outcomes: subsequent suicide attempt, subsequent psychiatric hospitalization, and attending 4 or more outpatient mental health visits.

Results

Demographics

The overall sample was predominantly white (n=170; 85%) and male (n=173, 86.5%), consistent with the VHA population. Age ranged from 21 to 87 years (M=50.5; SD=14.6). The majority of the sample had a mental health diagnoses, and many participants had more than one diagnosis (see Table 2). Approximately 13.5% percent of the sample had a history of traumatic brain injury (TBI; n=27). Seventy-six percent (n=152) of the patients had attempted suicide prior to being identified as being at high risk for suicide.

Table 2.

Frequency of mental health diagnoses

Diagnosis	N	%
Depressive Disorder	139	69.5
Substance Use Disorder	126	63.0
Posttraumatic Stress Disorder	112	56.0
Tobacco Use Disorder	69	34.5
Anxiety Disorder	50	25.0
Adjustment Disorder	43	21.5
Borderline Personality Disorder	6	3.0

Safety Plan Characteristics

Of the 200 Veterans identified in the SPAN database, twenty (10%) who were identified as being at high risk for suicide did not have safety plans documented in the medical records. The following results reflect the sample of Veterans with safety plans available for review in the medical records (n=180).

Safety plans were completed by suicide prevention coordinators (n= 73), social workers (n=41), nurses (n=25), psychiatrists (n=19), psychologists (n= 17), and other mental health professionals (n=5). Seventy-five percent of plans were completed during an inpatient stay (n=135), while 25% were completed in either general mental health outpatient clinics (n=42) or in specialty outpatient clinics (n=3).

Safety Plan Completeness, Quality, and Provider-Patient Interactions

Rater agreement for fidelity variables was high (completeness ICC = .994; quality ICC = .948). Safety plans were generally mostly complete (completeness M=11.2; SD=2.43), and of moderate quality (quality M=15.5; SD=3.28), meaning that on average, individual safety plan items evidenced some evidence of personalization, but were not highly specific. Some variability, particularly in quality, was evident. Evidence of specific provider-patient collaboration and follow up variables (e.g., documentation of a collaborative process of safety plan development, discussion of how and when to use the plan, follow-up inquiry regarding patients’ use of the plan, and whether it reviewed and utilized in treatment over the course of the year) was not consistently found in the charts (see Table 3). Despite the general mention of safety plans in many treatment plans, 45.6% (n=82) of the patient charts had no explicit evidence of ongoing review or utilization of the safety plan in treatment, 22.8% (n=41) had a generic, copied and pasted statement noting the existence of a safety plan in subsequent chart notes with no other evidence of utilization, 13.9% (n=25) had inconsistent mentions of the use of the safety plan in subsequent clinical notes, and only 17.8% (n=32) of the patients in the sample had ongoing, specific documentation of safety plan utilization during therapeutic encounters. Only 11% (n= 21) of the patients’ safety plans demonstrated the required annual update, and 27.2% (n= 49) had two or more safety plans in their chart, indicating that recommendations to update the safety plan annually or when clinically indicated may not have been followed.

Table 3.

Provider-patient collaboration and follow-up items

Interaction	N	%
Evidence of collaboration	54	30.0
Discussion of how to use Safety Plan	86	47.8
Clinician follow-up to inquire whether the patient has used the Safety Plan	19	10.6
Feedback on how Safety Plan is working	18	10.0
Evidence of ongoing review and use of the Safety Plan	57	29.0
Mention of Safety Plan in subsequent treatment plan	110	55.0
Safety Plan updated annually	21	11.0

Association with Subsequent Outcomes

Among Veterans with safety plans, 17% (n=31) made a subsequent suicide attempt. Results from logistic regression analyses which examine the associations between safety plan fidelity and three outcomes (subsequent suicide attempt, subsequent psychiatric hospitalization, and attending 4 or more outpatient mental health visits) are found in Table 4. A significant association was found between total quality and subsequent hospitalization (odds ratio =.88, 95% confidence interval: .80, .97), indicating that individuals with higher quality safety plans were less likely to be hospitalized in the year after safety planning. Total completeness, however, was not associated with any subsequent patient outcomes.

Table 4.

Associations between quality, completeness, and subsequent patient outcomes

Variable	Subsequent suicide attempt		Subsequent hospitalization		Attending 4 or more outpatient visits
	OR	95% CI	OR	95% CI	OR	95% CI
Total Completeness	.91	[.78, 1.08]	.96	[.85, 1.08]	1.06	[.93, 1.18]
Total Quality	.90	[.79, 1.02]	.88**	[.80, .97]	.96	[.88, 1.05]

Note. OR= odds ratio; CI= confidence interval

Unavailable or Missing Safety Plans

To better understand why some Veterans did not have safety plans in the medical records, the first author examined clinical notes surrounding the flag date. Three patterns were identified: a) the Veteran was never on an inpatient unit within the VA Boston Healthcare System, and did not engage in outpatient care or receive a safety plan in outpatient visits (n=13), b) the Veteran had inpatient and outpatient visits, but never received a plan for reasons that were unclear (n=4), and c) evidence existed the Veteran did receive a safety plan (e.g., specific references to a safety plan in the chart), but it was not included in the charts or was developed several months after the flag was issued (n=3). Due to the low number of Veterans without safety plans, we were unable to compare outcomes for Veterans with and without safety plans.

Discussion

Although fidelity to an intervention is an important implementation outcome (Proctor et al., 2009; Schoenwald et al., 2011), no study in the VHA to date has examined safety plan implementation fidelity or the associations of fidelity components to specific clinical outcomes. As such, the aim of our investigation was to better understand these associations among VHA patients.

While our hypothesis that the majority of individuals in our sample would have documented safety plans was largely confirmed, our investigation identified small amount of Veterans who did not. Most commonly, this was because the Veterans without a safety plan did not receive care on an inpatient psychiatric unit at this VHA hospital, where plans are most commonly completed. There may be several reasons why outpatient providers did not complete safety plans. For example, it is possible that when these patients were seen for outpatient care, providers did not perceive them to be at high risk for suicide, despite the flag in their chart. It is also possible that, because safety plans are more commonly completed on inpatient units than in outpatient care, providers assumed a safety plan was already in the chart, or that they were unaware that in the absence of a documented safety plan, they should implement the safety plan intervention to meet the VHA’s safety plan requirement.

Among those Veterans with safety plans, some variability in safety plan quality was evident, although, on average, safety plans were fairly complete and of moderately good quality. Charts contained little evidence of provider-patient collaboration and follow up items, such as eliciting feedback from the Veteran about whether they used the safety plan or about how well they believed it was working, evidence of ongoing use of safety plans, or updates to the safety plan. Nearly half of the charts contained some evidence that the discussion on how to use the safety plan occurred, but 10% (n=18) of the charts contained any evidence of feedback from the patient in subsequent sessions and only 17.8% (n=32) documented consistent ongoing use of the safety plan as a therapeutic tool.

Because data were collected through a chart review methodology, such interactions between providers and patients may have occurred, but were not documented. Failure to document use of the safety plan as a therapeutic tool suggests that perhaps after the initial creation and documentation, the use of the safety plan as a therapeutic intervention may be limited. The safety plan manual notes that safety plans should be updated as clinically necessary or as new skills are identified (Stanley & Brown, 2008), but three quarters of the charts in our sample did not document updates of the safety plans in the year following the index safety plan. Taken together, these findings suggest that safety plans may not be used as effectively as possible in an ongoing manner as a clinical tool.

Though some safety plans may have been left incomplete to allow patients more time to add new skills or contacts that may be developed in the course of therapeutic work, our study reveals that safety plans were rarely updated to add information or remove items that were out of date. Additionally, because the majority of the safety plans were created on an inpatient unit, there may be a lack of continuity during the transition to outpatient care such that providers who did not create the plans may not follow up on them or use them as a part of their treatment strategy. Further investigation is necessary to determine whether these findings accurately represent the nature of provider-patient interactions around safety planning.

Safety plan completeness was not shown to be associated with subsequent suicide attempts, hospitalizations, or Veteran attendance of 4 or more subsequent outpatient mental health sessions. One possible reason for these findings is that adherence to the safety plan format may play less of a role in predicting patient outcomes. In addition, because the majority of safety plans were fairly complete, it is conceivable that restricted range decreased our ability to predict these outcomes. Safety plan quality was not associated with subsequent suicide attempts or with the likelihood of a patient attending 4 or more outpatient mental health session.

Our investigation found evidence that total quality has a protective association with subsequent hospitalizations, such that higher quality safety plans were associated with reduced likelihood of further or future hospitalization. When experiencing a subsequent crisis, a patient with a higher quality, more personalized safety plan may use their specified coping skills and contacts to reduce potential suicidal thoughts and behaviors, thereby decreasing the need for further hospitalization. However, it is also possible that patients who were better able to participate in the safety planning process and identify coping skills that could help them in the future were less likely to need hospitalization later. Future longitudinal research could explore these possibilities. Furthermore, patients who received a safety plan after a recent suicide attempt are likely to receive a variety of interventions, such as group therapy, individual psychotherapy, and medication, all of which are intended to improve subsequent patient outcomes. Therefore, we cannot report with certainty that subsequent patient outcomes are exclusively the result of safety plan quality, and an examination of the types or outcomes of additional interventions for Veterans at high risk for suicide was beyond the scope of the current investigation.

Limitations

Several limitations associated with this study are important to note, particularly when considering future directions for research. Our study may be limited by our method of chart assessment, although clear definitions and decision rules for each rated item increased reliability and rater agreement. Certain variables, such as education level and socioeconomic status, were not discernable from patient charts, while others were not systematically documented in notes, thus potentially providing inconsistent information. It is likely that many of the provider-patient interactions related to safety planning are not adequately represented by documentation in CPRS, despite the recommendations set forth by the VHA. Future studies of safety plan implementation should utilize additional methodologies, such as provider and patient self-report and direct observation to assess the nature of interactions with patients that may be underreported in clinical documentation. Furthermore, since there are no specific directions for documenting other safety planning interactions, results may reflect different providers’ documentation preferences rather than speak to provider competence. For example, some providers may provide more specific and detailed notes, whereas others might not document collaborative interactions that occurred around safety planning.

This study was limited to a small sample size. A more thorough, prospective analysis of patient outcomes with a larger sample should be undertaken in future studies to better understand the relationship between safety planning and clinical outcomes in VHA. Researchers may wish to explore how safety plan fidelity differs among groups stratified by important variables such as gender, service era, PTSD diagnosis, depression, and substance use diagnoses. Finally, as this pilot study obtained data from only one VA Healthcare System, investigation of safety planning in additional VHA hospitals is an important step for future research.

Recommendations and Conclusions

Despite these limitations and the need for future research, our current findings suggest that enhancing clinical practice around safety planning is warranted. If the majority of safety plans are developed while on inpatient units, provider training should include explicit recommendations for outpatient providers to complete or update safety plans when needed. Provider training could also emphasize that outpatient providers follow up and use safety planning as an ongoing therapeutic process during the course of treatment. Patients may not keep track of the copy of the safety plan that they are provided upon discharge, and they may find that contacts and coping skills that they identified during their inpatient stay may not be as useful as anticipated. Thus, inquiring about whether patients have a copy of the safety plan, whether they have used it and found it helpful, and whether it is useful to spend some time re-evaluating items on the safety plan at the first outpatient visit is important. Not only does it reinforce the notion that the healthcare system believes safety planning to be an important aspect of recovery for individuals who are at high risk for suicide, it can also increase patients’ perception of the safety plan as a useful tool that should be used in an ongoing manner.

Additionally, adding new skills and contacts to the safety plan as the patient identifies effective coping strategies and support networks further reinforce the use of the safety plan as a therapeutic tool while ensuring that an up-to-date and clinically useful safety plan is available in the chart should the patient need to obtain a new copy. Since safety plans are intended to be a therapeutic alternative to no-suicide contracts, emphasis of the importance of the ongoing use of the safety plan is necessary. More explicit directions and suggestions in the safety planning manual for how the safety plan may be used in subsequent meetings with patients may also facilitate improved provider and patient engagement with the safety planning intervention. Clinical reminders in the medical record may also be used to prompt providers to follow up and update the safety plan.

Safety planning in VHA is a relatively new practice that requires further investigation. This study describes preliminary findings regarding current practice patterns and explores their association with clinical outcomes that are important to consider when treating individuals who are at high risk for suicide. These findings have implications for patient care and for continuing quality improvement of VHA. Additionally, given the widespread use of no-suicide contracts in usual care by non-VHA providers, these findings may suggest the promising utility and feasibility of safety plans as an alternative intervention for suicide prevention in other healthcare settings. Future research should also make use of a larger sample size and more direct observation to replicate results and focus on additional factors that may affect the outcomes of safety planning.