An implementation evaluation of “Zero Suicide” using normalization process theory to support high-quality care for patients at risk of suicide

Abstract

Background:

Suicide rates continue to rise across the United States, galvanizing the need for increased suicide prevention and intervention efforts. The Zero Suicide (ZS) model was developed in response to this need and highlights four key clinical functions of high-quality health care for patients at risk of suicide. The goal of this quality improvement study was to understand how six large health care systems operationalized practices to support these functions—identification, engagement, treatment and care transitions.

Methods:

Using a key informant interview guide and data collection template, researchers who were embedded in each health care system cataloged and summarized current and future practices supporting ZS, including, (1) the function addressed; (2) a description of practice intent and mechanism of intervention; (3) the target patient population and service setting; (4) when/how the practice was (or will be) implemented; and (5) whether/how the practice was documented and/or measured. Normalization process theory (NPT), an implementation evaluation framework, was applied to help understand how ZS had been operationalized in routine clinical practices and, specifically, what ZS practices were described by key informants (coherence), the current state of norms/conventions supporting these practices (cognitive participation), how health care teams performed these practices (collective action), and whether/how practices were measured when they occurred (reflexive monitoring).

Results:

The most well-defined and consistently measured ZS practices (current and future) focused on the identification of patients at high risk of suicide. Stakeholders also described numerous engagement and treatment practices, and some practices intended to support care transitions. However, few engagement and transition practices were systematically measured, and few treatment practices were designed specifically for patients at risk of suicide.

Conclusions:

The findings from this study will support large-scale evaluation of the effectiveness of ZS implementation and inform recommendations for implementation of high-quality suicide-related care in health care systems nationwide.

Plain Language Summary

Many individuals see a health care provider prior to death by suicide, therefore health care organizations have an important role to play in suicide prevention. The Zero Suicide model is designed to address four key functions of high-quality care for patients at risk of suicide: (1) identification of suicide risk via routine screening/assessment practices, (2) engagement of patients at risk in care, (3) effective treatment, and (4) care transition support, particularly after hospitalizations for suicide attempts. Researchers embedded in six large health care systems, together caring for nearly 11.5 million patients, are evaluating the effectiveness of the Zero Suicide model for suicide prevention. This evaluation focused on understanding how these systems had implemented clinical practices supporting Zero Suicide. Researchers collected qualitative data from providers, administrators, and support staff in each system who were responsible for implementation of practices supporting Zero Suicide. Normalization process theory, an implementation evaluation framework, was applied following data collection to: (A) help researchers catalog all Zero Suicide practices described, (B) describe the norms/conventions supporting these practices, (C) describe how health care teams were performing these practices, and (D) describe how practices were being measured. The findings from this evaluation will be vital for measuring the effectiveness of different Zero Suicide practices. This work will also provide a blueprint to help health care leaders, providers, and other stakeholders “normalize” new and existing suicide prevention practices in their own organizations.

Introduction

Suicide rates have continued to rise across the United States for the past two decades, thus galvanizing a need for increased prevention and intervention efforts across settings (Caine, 2013; Hogan & Grumet, 2016). Health care systems have a key role to play in suicide prevention, because many individuals see a health care provider prior to death by suicide—nearly a third in the week prior to suicide (Ahmedani et al., 2019), half in the month prior, and the vast majority in the year prior (Ahmedani et al., 2014; Luoma et al., 2002). The Zero Suicide (ZS) model is a quality improvement initiative for safer and more effective suicide care in health care settings that came out of a “call to action” developed by the Office of the US Surgeon General and the National Action Alliance for Suicide Prevention (U.S. Department of Health and Human Services Office of the Surgeon General and National Action Alliance for Suicide Prevention, 2012). The ZS model is not a formal validated treatment protocol with prescribed interventions, but rather a framework of evidence-based practices and implementation strategies designed to transform organizational culture (Hogan & Grumet, 2016). The ZS model strategy defines four essential clinical functions of high-quality care for patients at risk of suicide: identification of high-risk patients, engagement and care management, effective treatment, and supportive care transitions (see Table 1 for Terms and Definitions) (Brodsky et al., 2018; Education Development Center, 2020). Each of these functions is supported by the application of a procedure or practice intended to support a specific function (Jolles et al., 2019). Identifying high-risk patients involves screening and risk assessment, for example, using the nine-item Patient Health Questionnaire (PHQ-9)] (Kroenke et al., 2001), and the Columbia Suicide Severity Rating Scale (C-SSRS) (Posner et al., 2011). Engagement practices may include collaborative crisis response and safety planning interventions to help patients identify coping strategies and other resources during suicidal crises (Bryan et al., 2018; Stanley & Brown, 2012). Effective treatments may include evidence-based therapies targeting intensity and/or frequency of suicidal thoughts and behaviors (Brown & Jager-Hyman, 2014). Supportive care transitions ensure that patients receive uninterrupted care as they move between care settings, particularly from inpatient to outpatient settings following suicide attempt (Chung et al., 2017).

Table 1.

Terms and definitions.

Term	Definition
Determinant	A factor that enables or hinders the clinical practice from achieving the desired effect.
Function	The purpose of a practice. The core 4 clinical functions of high-quality suicide care defined in the ZS Model include identification, engagement, treatment and transition.
Mechanism	Process or event through which a clinical practice operates to affect outcomes.
Practice	The application of a procedure intended to support a specific function (aka “form following function”). This evaluation focuses on clinical practices intended to support the ZS Model.
ZS Model	A framework designed to support system-wide, organizational commitment to high-quality suicide care in healthcare.

ZS: Zero Suicide.

Although the ZS model has great potential to improve the way health care systems care for patients at risk of suicide, and has been widely adopted, there is limited evidence of the effectiveness of a collection of linked, mutually supporting clinical practices for prevention of suicide attempts (fatal and non-fatal). For example, the identification function is supported by research demonstrating that disclosure of suicidal thoughts on the PHQ-9 depression assessment is a strong predictor of subsequent suicide attempts (Louzon et al., 2016; Simon et al., 2013, 2016), but the link between identification of patients experiencing suicidal thoughts and the ability of health care systems to intervene effectively and help prevent suicide attempts is unclear. Similarly, though randomized clinical trials have demonstrated efficacy of some psychotherapeutic interventions for reducing suicide attempts, little is known about the effectiveness of these practices for suicide prevention in real-world settings or among people who have not volunteered to participate in research (Brown & Jager-Hyman, 2014). Moreover, though prior research has demonstrated that systemic, bundled suicide prevention practices are more effective than any single practice (M. J. Coffey et al., 2015; Knox et al., 2003; Martin et al., 2016; While et al., 2012), it is unclear which specific practices, or combination of practices, supporting different ZS functions are most effective (Ahmedani & Vannoy, 2014).

Early examples of implementation of clinical practices related to the ZS model have been encouraging, such as the substantial reductions seen in annual suicide rates following implementation of the Perfect Depression Care program at the Henry Ford Health System (C. E. Coffey et al., 2013; Hampton, 2010). Nevertheless, more rigorous study is needed to demonstrate how specific ZS clinical practices are associated with important patient outcomes, specifically suicide attempt and mortality. Therefore, researchers in six large health care systems are investigating the effectiveness of different clinical practices (and combinations of practices) for preventing suicide attempts and deaths (Yarborough et al., 2019). At the time of this evaluation, these six health care systems had all implemented varying combinations of suicide prevention practices and were routinely collaborating via a formal suicide prevention learning collaborative (Bruschke & Flores, 2020). Therefore, in support of the learning collaborative and the planned investigation of ZS effectiveness (Figure 1), this research team sought to build a blueprint of the practices supporting the four key clinical functions (identify, engage, treat, transition) of the ZS model across health care systems.

Figure 1.

Goals of the planned Zero Suicide evaluation.

Thus, this evaluation focused on understanding how these six health care systems had implemented (or planned to implement) clinical practices supporting the four key clinical functions highlighted in the ZS Model, in support of a broader ZS evaluation. Building a thorough working knowledge of practice variation across systems would be essential for measuring and comparing the effectiveness of different ZS practices (Figure 1). Therefore, we used normalization process theory (NPT), an implementation evaluation framework (McEvoy et al., 2014; Nilsen, 2015), to help explain which individual and interrelated practices had been implemented and how those practices had been operationalized across different health care systems to support ZS. NPT was selected because the framework focuses on the organic process of how complex interventions become normalized in practice, shaped by four determinants—coherence (how the intervention is understood), cognitive participation (how users engage in the intervention), collective action (how the intervention is performed), and reflexive monitoring (how the intervention is monitored over time) (May et al., 2009; May & Finch, 2009). Specifically, these four NPT determinants were applied following key informant data collection to help explain how different clinical practices supporting ZS had become normalized across organizations and care settings.

Methods

Participating health care systems

Six large health care systems, together caring for nearly 11.5 million members in California, Colorado, Michigan, Oregon, and Washington, are implementing clinical practices supporting the ZS model and agreed to participate in the implementation evaluation. All participating health care systems provide both comprehensive health care and insurance coverage to a defined patient population. Each system is described here and additional details are presented in Table 2, using data compiled from 10/2018-9/2019 (Henry Ford Health System, 2019; Kaiser Permanente, 2019). The participating Kaiser Permanente (KP) health care systems in California together served approximately 9 million patient members, followed by Henry Ford Health System in Michigan serving 1.2 million, and KP health care systems in Oregon, Washington and Colorado similarly serving between 600,000–700,00 members each. The Institutional Review Board at each system approved its participation in the ZS evaluation.

Table 2.

Characteristics of participating health care systems and patient-populations (10/1/2018-9/30/2019).

System characteristics	KPSC	KPNC	KPNW	KPCO	HFHS	KPWA
Members	4.6 million	4.4 million	620,000	640,000	1.2 million	700,000
Outpatient Medical Centers/Clinics	231	252	59	34	50	34
Hospitals	15	21	2	0	6	0
Patient characteristics
Female	51.7%	51.6%	52.1%	52.9%	57.4%	53.9%
Age
0–19	23.4%	22.2%	20.9%	19.9%	18.1%	15.9%
20–39	27.4%	27.1%	25.8%	23.6%	21.5%	25.7%
40–64	33.4%	34.3%	34.7%	35.9%	36.6%	39.9%
65+	15.9%	16.4%	18.6%	20.6%	23.8%	18.4%
Insurance
Medicaid	10.4%	7.3%	9.2%	6.8%	0.4%	0.6%
Medicare	13.6%	19.6%	19.6%	21.7%	23.5%	19.6%
Socioeconomic status
Neighborhood income < 25 K	0.8%	1.2%	0.4%	0.6%	6.7%	1.2%
Neighborhood education < 25% college	53.8%	37.6%	39.3%	25.4%	52.9%	38.9%
Race/Ethnicity
Asian	10.9%	20.1%	6.4%	3.5%	4.6%	7.4%
Black/African American	8.2%	6.9%	3.4%	4.2%	28.2%	3.4%
Hispanic/Lantinx	40.9%	21.1%	8.5%	15.9%	2.3%	4.0%
Hawaiian/Pacific Islander	0.8%	1.0%	0.9%	0.3%	0.1%	0.9%
American Indian/Alaska Native	0.3%	0.5%	0.8%	0.6%	0.4%	1.0%
Multiple/Other	0.0%	0.0%	0.4%	3.1%	1.8%	1.0%
Unknown	7.0%	7.1%	8.1%	12.4%	8.7%	34.1%
White	31.9%	43.4%	71.6%	59.9%	54.0%	48.2%
Any Mental Health Diagnosis in 2018	16.2%	14.4%	22.3%	20.3%	16.8%	20.4%

KP: Kaiser Permanente; KPSC: KP Southern California; KPNC: KP Northern California; KPNW: KP Northwest (Oregon/Southern Washington); KPCO: KP Colorado; HFHS: Henry Ford Health System; KPWA: KP Washington.

The characteristics of the different health care system patient populations vary by sociodemographic characteristics associated with suicide, including age, race/ethnicity, income, and education (Nock et al., 2008; Rehkopf & Buka, 2006). For example, at the time of this evaluation, Henry Ford provided care to the oldest patient population and had the largest proportion insured by Medicare (23.5%), while KP Southern California served the youngest population and had the largest proportion insured by Medicaid (10.4%). Henry Ford served the largest proportion of Black/African American patients (28.2%), KP Southern California the largest proportion of Hispanic/Latinx patients (40.9%), and KP Northern California the largest proportion of Asian patients (20.1%).

Suicide prevention learning collaborative

In 2016, Kaiser Permanente Care Management Institute (CMI) , convened a Suicide Prevention Learning Collaborative workgroup, which included clinical, operational, and patient safety stakeholders, as well as peer advisors (KP health plan members and advocates with lived experience from national mental health advocacy organizations), and researchers, including those from all six health care systems partici-pating in the ZS evaluation. The charter of this workgroup included cross-regional collaboration on implementation of best-practices for suicide prevention (Bruschke & Flores, 2020). With oversight from operational mental health leadership, CMI works with members from the suicide prevention learning collaborative to facilitate development new electronic health record (EHR)-based tools, trainings and metrics to support key ZS practices which build from the work described in this evaluation.

Key informant data collection

A key informant meeting guide (see Supplement A) was developed to assist researchers embedded in each health care system (JER, GES, JMB, AB, BHY, KJC, SAS, JW, BKA) to collaborate with key informants and document clinical practices the systems had implemented (or planned to implement) across the different clinical functions of ZS. Prior to data collection, embedded researchers all had some knowledge about suicide prevention initiatives and clinical practices in their health care systems, so data collection was designed to enhance that knowledge. Key informants included local leaders and staff with knowledge of implementation of suicide prevention related processes and protocols in the various clinical settings. A structured data collection template (Miles et al., 2014) was designed to help the researchers catalog details about current and planned suicide prevention practices across care settings (Supplement B). The data collection template did not explicitly assess the domains of the ZS Model focused on implementation strategies (training, leadership, and continuous quality improvement) (Education Development Center, 2020), but these domains were implicitly addressed in questions about how the practices had/or would be implemented. The template was originally developed and tested at KP Colorado and subsequently tested at two other health care systems, with the finalized version made available for widespread use in March 2018. Templates were populated and reviewed iteratively by the embedded researcher and team members at each participating health care system. In parallel, templates were also uploaded to a private web-based file storage platform available to Health care System Research Network members (Health Care Systems Research Network, 2019) and each reviewed in detail by members of the full study team online during routine all-site team meetings (December 2018 through April 2019). Following these presentations, some researchers made additional edits and updates to their templates to reflect changing ZS practices across their systems and uploaded current versions to the shared private website. In August 2019, a health services research (JER) used the most recent version of the template from each health care system to summarize ZS practices across health care systems, including (1) the function addressed; (2) a brief description of the practice intent and mechanism of intervention (e.g., EHR-based clinical decision support tools), standard workflows/processes and health care system policies; (3) the target patient population and service setting; (4) when/how the practice was (or will be) implemented; and (5) whether/how the practice was documented/measured.

Application of NPT

NPT was applied following key informant data collection to help understand how health care systems had implemented different clinical practices, and combinations of practices supporting the ZS model (May et al., 2009; May & Finch, 2009). NPT assumes the way that practices become routinely embedded and integrated into their social contexts—how they become normalized—is not a structured sequence of events, but instead is an organic process shaped by four determinants: coherence, cognitive participation, collective action, and reflexive monitoring (May & Finch, 2009; Nilsen, 2015). To support this evaluation, we first developed a working interpretation of the NPT determinants for application to ZS implementation, guided by two prior studies evaluating implementation of depression care in primary care (Franx et al., 2012; Gunn et al., 2010). Specifically, like Gunn et al. (2010), we developed working definitions for how each of the NPT determinants would apply to health care system implementation of practices (and combinations of practices) supporting the ZS model (Table 3). Specifically, we used NPT to catalog and connect the descriptions of current and future clinical practices to ZS implementation (i.e., creating coherence) across the health care systems, and to facilitate the specification of the workflows/processes (cognitive participation; collective action) and approaches to measuring these workflows/processes (reflexive monitoring). This approach involved using NPT pragmatically to define specific products resulting from our application of NPT to the key informant data (Table 3) to support the goals of the broader ZS evaluation (Figure 1). Specifically, to describe coherence, a health services research (JER), first used the templated key informant data to create a thematic network (Attride-Stirling, 2001) (i.e., affinity diagram) of the existing practices supporting the four key clinical functions of ZS across all six participating health care systems. Embedded researchers from all participating health care systems, responsible for data collection, participated in several rounds of iterative review and refinement to ensure ZS practices were comprehensively and accurately represented in the final version of the thematic network. Second, we summarized the current state of practices supporting ZS across health care systems and created a list of tools (e.g., screeners, assessments, templates) supporting those practices to describe cognitive participation and collective action. Third, to describe reflexive monitoring we built a working knowledge of whether and how ZS practices were being measured and how different practices and combinations of practices could be measured over time. This work also involved conceptually mapping a suicide care continuum, similar to treatment cascades used to evaluate care gaps for patients living with HIV (Kay et al., 2016).

Table 3.

Normalization process theory (NPT) determinants and application to Zero Suicide (ZS) evaluation.

NPT determinant	Definition (May & Finch, 2009)	Application to ZS Evaluation	Product
Coherence	Do people know what the work is?	What clinical practices support the four clinical functions of ZS?	Thematic network
Cognitive Participation	Do people join in to the work?	What norms/conventions support ZS practices?	Summary of current workflows supporting ZS practices
Collective Action  Skillset Workability  Contextual Integration  Interactional Workability  Relational Integration	How do people do the work?	What tools are used to support ZS practices? How?	Summary of clinical decision support tools used across healthcare systems
Reflexive Monitoring	How do we know the work is happening?	How should we measure ZS practices?	Care continuum model

NPT: normalization process theory; ZS: Zero Suicide. z

Results

Key informant data collection and application of NPT

Embedded health care system researchers had all begun populating the implementation templates in collaboration with local key informants by June 2018. An average of 14 key informants per health care system provided data to help populate each template (range 7–26). Key informants included providers, administrators, EHR programmers, and project managers (often known to embedded researchers) who were responsible for supporting implementation of practices supporting ZS in different ways. All health care systems had participated in virtual team review and updated their templates at least once before the designated health services researcher (JER) begin reviewing template data in August 2019. Prior to application of NPT, the health services researcher organized the practices summarized in the templates by function. In some cases, the function was changed (e.g., “treat” to “engage”) to be consistent with the ZS framework definitions, in collaboration with a ZS expert (JGG).

Each product associated with application of NPT to the templated data (Table 3) was created consecutively and is described in detail below, including (1) the catalog of practices understood to support ZS (coherence); (2) the summary of the current state of norms/conventions supporting these practices and how health care teams performed these practices (cognitive participation and collective action); and (3) a list of tools available to measure practices across systems and a model describing combinations of practices available to measure the continuum of suicide care across settings (reflexive monitoring).

Coherence

In Figure 2, we present the thematic network of clinical practices key informants understood to support ZS (coherence), derived from the implementation templates completed by an embedded researcher at each site. The number of current/planned clinical practices described by key informants varied across health care systems (average per health system = 32, range 19–55), as did the level of detail in the description of each practice (Supplement B). All templates included fairly detailed descriptions of practices supporting the identify function (i.e., standard workflows and tools), but practices supporting the engage, treat and transition functions were often less clearly defined. Moreover, often the clinical, operational, and patient safety stakeholders did not explicitly reference the ZS model when they described the current state of health care system practices supporting suicide prevention. Some templates also included descriptions of practices that were not specifically designed to support suicide prevention, but rather mental health and addiction care more generally. For example, substance use disorder screening/assessment, social support, care coordination/outreach, and types of psychotherapy. Relatedly, some of these practices had been in place at most health care systems more than 5 years prior to this evaluation (like depression severity assessment with the PHQ-9 and different types of psychotherapy), while others had been implemented more recently (suicide severity assessment with the C-SSRS).

Figure 2.

Zero Suicide (ZS)-related practices across all participating health care systems cataloged by the clinical function of ZS.

Cognitive participation and collective action

Identification of high-risk patients

Practices described by key informants used to support the identification function of ZS included standard clinical decision support tools, implemented in the EHR, to support suicide risk screening followed by more comprehensive risk assessment. At the time of this evaluation, all health care systems monitor depression symptom severity, including suicidal ideation, among adult patients receiving care in the outpatient mental health specialty setting. All systems also had screening/assessment practices in place for substance use disorders, although the norms/conventions supporting these screening practices varied. Workflow variation across care settings was common and the health care systems had different screening and assessment approaches (e.g., varying criteria for deciding which patients received the PHQ-9). In the primary care setting, population-based screening and suicide risk assessment were less common; screening and assessment were more often done at the discretion of primary care providers and many systems were engaged in planning future implementation of different approaches for more universal screening/assessment. Some systems were also in the process of planning implementation of suicide risk prediction algorithms (Simon et al., 2018) for purposes of enhancing suicide risk assessment and engagement in suicide-related care.

Engagement and care management

Practices designed to support the engagement function of the ZS model consistently included references to development of a safety plan or crisis response plan (Bryan et al., 2018; Stanley & Brown, 2012), which usually included lethal means assessment (i.e., discussions about prescription medications, firearms, and planning how to limit access). Researchers cataloged varying safety planning workflows and documentation practices across sites and providers (Yarborough et al., 2019); however, a shift toward practice standardization was underway. CMI was facilitating the standardization of safety planning by promoting use of a common EHR-based template across health care systems (i.e., KP regions). Key informants also described integrating social support, referral practices, and care coordination programs to engage patients in care for suicidality. The norms/conventions supporting these practices, however, were often not well-defined.

Effective treatment

Embedded health system researchers documented a variety of practices across health care systems that addressed the treatment function of ZS. These included pharmacotherapy; different types of evidence-based psychotherapy available for depression and other mental health disorders, including psychoeducational group-based therapy and cognitive behavioral therapy (CBT) (Hofmann et al., 2012), electroconvulsive therapy (Pagnin et al., 2004; UK ECT Review Group, 2003); and psychiatric hospitalization, which is the current standard of care for patients at high risk of suicide (Brown & Jager-Hyman, 2014). Researchers also cataloged evidence-based psychotherapies being used in some health care systems that were designed specifically for patients at high risk of suicide, including CBT for suicide prevention (CBT-SP) (Stanley et al., 2009) and dialectical behavior therapy (DBT) (Linehan et al., 2006). Generally, mental health specialty providers in all systems were responsible for treatment of patients at risk of suicide, but the specific treatments providers were using were unknown in some health care systems and difficult to discern in others (e.g., differentiating CBT from CBT-SP, and DBT groups from other group-based therapies). Moreover, it was not possible to know exactly what parts of specific therapies patients were receiving (e.g., specific DBT/CBT skills/strategies) (Bryan, 2019; Koerner, 2013).

Supportive care transitions

The practice consistently documented across health care systems designed to support the transition function of ZS was follow-up care after discharge from inpatient psychiatric settings. However, follow-up practices after discharge from the emergency department (ED) setting (without inpatient admission) were varied. Other practices designed to support care transitions included intensive case management programs typically designed to support patients with severe and persistent mental illnesses in outpatient settings (Dieterich et al., 2017). Finally, key informants at two health care systems reported sending caring messages (Motto, 1976; Motto & Bostrom, 2001) following inpatient discharge which addressed the ZS transition function, but the norms/conventions supporting this practices were not well-defined.

Reflexive monitoring

The data from key informants were also used to conceptualize how different practices and combinations of practices could be measured over time to support development of standard process measures and evaluation of ZS effectiveness (Figure 1). First, we summarized the common screening tools/assessments and health care utilization codes that could be extracted from EHRs across systems (Table 4). Specifically, at the time of this evaluation all health care systems were using the PHQ-9 to measure depression severity (Kroenke et al., 2001) and most participating health care systems also implemented a standardized tool for suicide risk assessment (most commonly the C-SSRS) (Madan et al., 2016; Posner et al., 2011) in their mental health and addiction medicine settings. No systems were using the same tools for drug use disorder screening/assessment, but several systems were using a common screening tool for alcohol use disorders (AUDIT-C) (Bradley et al., 2007; Bush et al., 1998). All systems were also in various stages of implementing new EHR tools designed to support consistent documentation of the key components of a safety plan (Bottomley, 2019; Stanley & Brown, 2012). Providers in all systems also had routine procedures for documenting treatment in patient EHRs via diagnosis and procedure codes (i.e., ICD and CPT) associated with health care encounters (CMS.gov Centers for Medicare & Medicaid Services, 2019; World Health Organization, 1992). Finally, follow-up within 7-day and 30-day post-discharge from inpatient psychiatric settings was consistently measured across all health care systems in accordance with national quality metrics measuring (National Committee for Quality Assurance [NCQA], 2019).

Table 4.

ZS practices measured with common tools across health care systems.

ZS function	Practice	Healthcare system						Measured with common tools across systemsa
		1	2	3	4	5	6	No	Yes	How?
Identification	Depression/Suicide Risk Screening	X	X	X	X	X	X		X	PHQ-2, PHQ-9Q9
	Depression Severity Assessment	X	X	X	X	X	X		X	PHQ-9
	Suicide Risk Assessment	X	X	X	X	X	X		X	C-SSRS
	Suicide Risk Prediction Algorithm						X	X
	Substance Use Screening/Assessment	X	X	X	X	X	X		X	AUDIT-C
Engagement	Collaborative Safety Planning	X	X	X	X	X	X		X	bSafety Plan
	Social Support Programs		X					X
	Referrals/“Warm” hand-offs	X	X	X	X	X	X	X
	Care Coordination/Outreach Programs	X	X	X	X	X	X	X
	Care Management Programs	X	X	X	X	X	X	X
Treatment	Safety Plan updates/monitoring	X	X	X	X	X	X		X	bSafety Plan
	Psychotherapy	X	X	X	X	X	X		X	ICD-10
	Pharmacotherapy	X	X	X	X	X	X		X	NDC Code
	Electroconvulsive Therapy	X	X	X	X	X	X		X	CPT Code
	Psychiatric Hospitalization	X	X	X	X	X	X		X	ICD-10
Transition	Follow-up Post Hospitalization	X	X	X	X	X	X		X	FUH HEDIS Measure
	Follow-up Post ED Discharge	X	X	X	X	X	X		X	cEncounter
	Caring Message		X		X			X
	Intensive Case Management (Out-patient)	X		X		X		X

ZS: Zero Suicide; PHQ: Patient Health Questionnaire; C-SSRS: Columbia Suicide Severity Rating Scale; AUDIT-C: Alcohol Use Identification Test Consumption; ICD-10: 10th revision of the International Statistical Classification of Diseases; NDC: National Drug Code; CPT: Current Procedural Terminology; FUH: Follow-Up After Hospitalization for Mental Illness; HEDIS: Healthcare Effectiveness Data and Information Set; ED: emergency department.

^aDefined as more than 1 health care system presently or in the process of being implemented.

^bDiscrete data element(s) implemented to capture EHR documentation.

^cHealthcare encounter (in-person, phone, etc.) EHR documentation.

Next, we defined a conceptual care continuum (i.e., treatment cascade) (Figure 3) to measure combinations of measurable practices supporting the ZS model. In this continuum, primary care patients enter the ZS care pathway at the point of population-based depression screening. Patients receiving mental health specialty care enter the care pathway via routine depression and/or suicide risk severity assessment. Patients seen in emergency or urgent care settings for mental health-related concerns also enter at the point of depression/suicide risk severity assessment. Patients identified as being at risk of suicide via screening/assessment may be engaged in collaborative safety planning and receive one or more types of treatment (e.g., pharmacotherapy, psychotherapy). Finally, patients who received care for suicidality in an inpatient or ED setting receive proactive outreach following discharge.

Figure 3.

Suicide risk care continuum.

Legend: color coding corresponds to the ZS function defined in Figure 2 (blue = identification, green = engagement, orange = treatment, purple = transition).

Discussion

This evaluation used NPT (McEvoy et al., 2014; Nilsen, 2015) to build a fundamental understanding of how six health care systems had implemented ZS practices to support four key clinical functions of high-quality care for patients at risk of suicide, and lays the foundation for health systems nationwide to understand the impact of implementing these practices. Our innovative application of NPT to data from key informants enabled us to comprehensively catalog practices understood to support ZS (coherence); summarize the current state of norms/conventions supporting these practices (cognitive participation) and how health care teams performed these practices (collective action); and define how we know ZS practices are happening and how to measure when they occur (reflexive monitoring). Embedded researchers who all had some knowledge about suicide prevention initiatives and clinical practices in their health care systems strengthened data collection and interpretation. The most well-defined practices focused on the identification function—specifically, the use of common screening/assessment tools (Kroenke et al., 2001; Posner et al., 2011) to identify patients at high risk of suicide attempt. Many potential treatment practice options were also defined, but few treatment practices identified were designed specifically for patients at risk of suicide and it was difficult to differentiate whether and how providers were using various therapy options. In addition, engagement and transition practices were often inconsistently used and measured within health care settings; but health care systems were consistently measuring health care encounters documentation following psychiatric hospitalization (NCQA, 2019) and a shift toward standardized safety planning was underway (Bruschke & Flores, 2020; Stanley & Brown, 2012).

This working knowledge of ZS practice variation will be used to support efforts to continue to improve suicide care across health care systems nationwide (Bruschke & Flores, 2020) and large-scale evaluation of the effectiveness of ZS practices, both alone and as part of a continuum of care for patients at risk of suicide. Specifically, defining an approach for measuring different practices (and combinations of practices) over time via development of standard process metrics (i.e., reflexive monitoring) have enabled participating health care systems, and will enable others nationwide, to undertake their own appraisal processes in support of ZS implementation. CMI worked in collaboration with mental health leaders to develop a standardized set of metrics to track implementation of ZS practices and suicide prevention outcomes (Bruschke & Flores, 2020). These metrics will enable clinical, operational, and patient safety stakeholders to measure whether identification and engagement practices are occurring when expected (i.e., continuous quality improvement) within their own health care systems. The working knowledge of the variation of ZS practice implementation across health care systems will also allow researchers to compare the effectiveness of different approaches (i.e., workflows) used for identification, engagement, and transitioning patients at high risk of suicide. This evaluation will also support treatment metric development; however, as this work demonstrated, the investigation of the real-world effectiveness of specific psychotherapies for suicide prevention may be limited until we are able to more accurately define ways to measure variation in these treatment practices.

Future directions

This project can provide a framework to support ongoing ZS implementation and evaluation in health care systems across the country. Specifically, this evaluation suggests important considerations for integrating or normalizing new practices to support the ZS model. For example, as suicide risk prediction algorithms (Kessler et al., 2017; Simon et al., 2018) are implemented to improve identification and engagement of patients at risk (U.S. Department of Veterans Affairs, 2017), it will be important to consider how to support this practice within the context of existing norms/conventions, workflows, and tools. This evaluation also underscored the potential utility of using standardized definitions and tools to measure existing engagement and transition practices, such as care coordination/outreach and caring message programs. Caring contact programs in particular have potential to be a simple and effective ZS practice, scalable across different types of health care systems (Carter et al., 2005, 2007; Comtois et al., 2019; Hassanian-Moghaddam et al., 2011; Luxton et al., 2012; Motto, 1976; Motto & Bostrom, 2001).

The work presented here will also provide a vital foundation for understanding potential adaptation of practices supporting ZS during environmental disruption and evaluating those changes. For example, in response to the coronavirus disease 2019 (COVID-19) pandemic many US health care organizations rapidly converted to providing many services, including mental health and addiction medicine care, virtually (Wosik et al., 2020). The processes used to identify patients at risk of suicide via virtual screening/assessment practices may be different than prior in-person practices. Moreover, reduced reliance on emergency and inpatient psychiatric settings during the COVID-19 response may also impact suicide rates and our understanding of the effectiveness of mental health care provided in these settings (Loch, 2014; Rabinowitz et al., 1994). When the effects of COVID-19 on suicide become clearer (Reger et al., 2020), understanding how ZS implementation adapted and changed during this time across these health care organizations will be vital for understanding which practices may have helped mitigate potential unintended negative effects of social isolation on suicide outcomes (Ahmed et al., 2020; Fatke et al., 2020; Vieira et al., 2020).

Limitations

This evaluation did not explicitly address the implementation strategy domains of the ZS model—training, leadership, and continuous quality improvement (Education Development Center, 2020). These domains are critical to ensuring that practices supporting the four key clinical functions of ZS we evaluated result in high-quality suicide care. For example, providing training in specific evidence-based interventions provides clinicians with valuable skills and enhances fidelity to the intervention. The presence of a completed safety plan in an EHR does not necessarily mean that the safety plan was completed collaboratively in a way that was meaningful to the patient. Training, leadership, and continuous improvement are implementation strategies critical for ensuring consistent delivery of high-quality care and will be important considerations for the future evaluation of the effectiveness of different ZS practices (and combinations of practices).

This evaluation also did not take variations in practice maturity into account or consider all the ways that clinical ZS practices interact with related interventions in health care systems. For example, screening and assessment for substance use disorders was identified as a ZS identification practice, which makes sense given the strong association between suicide and substance use disorders (Espinet et al., 2019; Wilcox et al., 2004), particularly alcohol use disorders (Bagge et al., 2013; Caetano et al., 2013; Cherpitel et al., 2004; Lejoyeux et al., 2008; Powell et al., 2001; Richards et al., 2020b). However, we did not consider whether and how to measure engagement in care for alcohol and drug use disorders (NCQA, 2020), which could also be useful mechanism for improving suicide-related care (Richards et al., 2020a, 2020b) or addressing suicidal ideation and risky patterns of alcohol use together for purposes of suicide prevention (Kalk et al., 2019).

Last, health care systems participating in this evaluation provide both integrated, comprehensive health care and insurance coverage to a defined member/patient population. This enabled us to define a conceptual care continuum for patients at risk of suicide across care settings (e.g., primary care/mental health specialty, inpatient/outpatient), but this care continuum may not be generalizable to organizations without the same responsibility to provide comprehensive health care to a defined population of members/patients across care settings. Health care organizations may also not have embedded researchers to assist with monitoring ZS implementation progress. However, many organizations have stakeholders invested in suicide prevention (like the clinical, operational, and patient stakeholders who participate in the suicide prevention collaborative within our health care systems) who can utilize the products of this evaluation (e.g., practice descriptions, measurement tools) to inform their own quality improvement processes.

Conclusion

This novel evaluation used NPT in combination with data from key informants to improve our understanding of clinical practices supporting the ZS model and inform recommendations for implementation of high-quality care for patients at risk of suicide in health care systems nationwide. Specifically, NPT was applied to create a catalog and description of practices supporting the four key clinical functions of ZS and model how to measure specific practices. These schemas will be used to inform a large-scale evaluation of the effectiveness of different ZS practices (and combinations of practices) and provide a blueprint to support suicide prevention practice implementation across health care systems, service settings, and patient populations.