A patient-centered outcomes framework for evaluating behavioral health crisis care

Abstract

Objective:

Recent federal investments have accelerated a nationwide transformation of the behavioral health crisis care (BHCC) system, raising pertinent questions about quality and effectiveness. In response to calls to collect more data and define outcomes for assessing BHCC, this study sought to identify patient-centered outcomes from the perspectives of individuals with lived experience of mental health and/or substance use disorders and/or experience delivering BHCC services.

Methods:

Using a multi-phased recruitment approach, qualitative interviews were conducted with 120 stakeholders, including peers with lived experience (n=21), direct service providers (n=47), program directors/managers (n=44), and executive leaders (n=8). All interview transcripts were analyzed using an open coding approach to generate emerging themes capturing measurable components of BHCC and meaningful outcomes of interest to participants.

Results:

A patient-centered outcomes framework for evaluating BHCC was developed by mapping emerging themes to a conceptual framework adapted from Donabedian’s structure-process-outcomes model. A total of 45 themes were identified and grouped into five domains: system structure, system implementation, system utilization, social outcomes, and health outcomes.

Conclusions:

While further efforts are needed to engage patients and providers in the process of developing specific measures, this framework provides a starting point for patient-centered outcomes research on BHCC.

Although more than 2.5 million behavioral health crises occur each year (i.e., 911 calls primarily related to mental health or substance use disorders),¹ the US has historically lacked a coordinated, national approach to behavioral health crisis care (BHCC).² Recently, the US has invested in new infrastructure for three core components of a comprehensive BHCC continuum: crisis call lines, mobile crisis teams, and crisis stabilization facilities.^3–7 For example, the 988 Suicide and Crisis Lifeline, a national emergency call line dedicated to behavioral health crises, launched in July 2022.³ In its first two years, 988 received more than 10 million calls, texts, and chats.⁴ Additionally, since September 2018, the federal government awarded more than $1.7 billion in expansion grants for Certified Community Behavioral Health Clinics,⁸ safety-net clinics that must provide behavioral health services, including mobile crisis teams and crisis stabilization services, to all patients regardless of insurance or ability to pay.^8–10 The rapid implementation of this new infrastructure has raised questions around BHCC quality and effectiveness, motivating calls to collect more data and to identify outcomes that capture the impact of BHCC services on patients.^11–13

To date, efforts to assess BHCC quality and effectiveness have largely focused on performance measures (e.g., number of crisis calls, crisis response times)⁷ while ignoring outcomes, including patient-centered outcomes (PCOs) that BHCC services have the potential to impact.¹³ PCOs are measurable concepts that are important, impactful, and/or meaningful to patients.^14–16 Identifying PCOs requires engaging with patients and providers to incorporate diverse perspectives.^15–17 Including patients and providers has the potential to enhance the relevance of research findings to these end users, and in turn, may improve the uptake of evidence into practice and policy changes that achieve the desired outcomes.^14–18 PCO frameworks have been developed for many types of health services,^14–20 including diagnosis-specific PCO frameworks for certain outpatient behavioral health services (e.g., substance use disorder, autism spectrum disorder).^21–25 However, PCOs of BHCC have not been identified. Given that patients with a diverse range of mental, behavioral, and neurodevelopmental disorders may utilize services across the BHCC continuum, a cross-cutting conceptual framework for evaluating PCOs of BHCC is needed.

The purpose of this study is to identify PCOs of BHCC and to propose a conceptual framework for evaluating the impact of BHCC on PCOs. First, we identify PCOs from the perspectives of individuals with lived experience of mental health and/or substance use disorders (i.e., patients) and/or experience delivering BHCC services (i.e., providers). Second, we propose a conceptual framework for evaluating PCOs of BHCC, developed by mapping emerging themes to an adapted version of Donabedian’s structure-process-outcomes model.^26,27 Establishing a stronger evidence base about which BHCC approaches are most effective is critical to guiding decisions about how to best deliver care to patients in crisis and updating best practice guidelines. Organizational leaders, researchers, and policymakers may use this framework to inform the development of data collection systems, evaluate the impact of BHCC on PCOs, and, ultimately, improve quality of care received by patients.

Conceptual Framework

To guide our development of a conceptual framework for evaluating PCOs of BHCC, we used the Community Health Implementation Evaluation Framework (CHIEF).²⁷ The CHIEF is based on Donabedian’s structure-process-outcome model for evaluating medical care,²⁶ with strategic adaptations designed to overcome criticisms about the classic Donabedian model and provide a more generalizable framework that can be further adapted to a variety of patient-, community-, and population-level programs and interventions.^27,28 Thus, the CHIEF served as an appropriate foundation from which we developed a conceptual framework adapted specifically for evaluating PCOs of BHCC. The process we used to map and compare emerging themes to the CHIEF and adapt the model is described further under Methods (see Analysis section). Below, we briefly provide context about how the CHIEF was derived from Donabedian’s model.

In Donabedian’s classic model, structure reflects resources or inputs needed to provide a service, process reflects actual activities implemented to deliver the service, and outcomes reflect changes in health status that result from receiving services.^26,27 Similarly, in the CHIEF, the structure dimension includes project management activities, resources dedicated to an intervention, and the overall system design. A criticism of the process dimension of Donabedian’s model is that this dimension oversimplifies the mediating mechanisms that influence outcomes.²⁸ To account for mediating factors that influence the relationship between the structure and outcomes dimensions, the process dimension of the CHIEF is subdivided into two unique dimensions: system implementation and system utilization. System implementation reflects the capacity of the system (i.e., actual hours of operation, types of services offered) and public awareness of services. System utilization reflects the number of people who receive the intervention (i.e., utilize the system), as well as key demographic characteristics of the population served.²⁷

Another criticism of the Donabedian model is that a narrow focus on health outcomes limits the applicability of the model to more comprehensive community-based interventions that extend beyond the walls of a hospital and address more upstream social outcomes,²⁷ as is often the case with BHCC. To facilitate the evaluation of a broader range of patient, community, and population-level outcomes, the outcomes dimension of the CHIEF is subdivided into two unique dimensions: social outcomes and health outcomes. Social outcomes represent changes in social determinants of health, which are non-medical factors that have a significant impact on health and well-being.²⁹ Health outcomes represent changes in health status, including both short-term health outcomes (e.g., changes in health behaviors) and long-term health outcomes (e.g., deaths).²⁷ These two dimensions include outcomes that, when identified using patient and provider input, may also represent PCOs.¹⁴

Methods

Study Participants and Recruitment

We used multiple methods to recruit interview participants. First, as part of a larger statewide evaluation of Indiana’s BHCC system, we purposively recruited executive leaders and administrative directors overseeing the delivery and implementation of BHCC services in Indiana. These individuals were identified using a list of contacts from 20 mental health treatment facilities provided by the Indiana Family and Social Services Administration’s Division of Mental Health and Addiction. Second, we asked initial interviewees to provide us with contact information for BHCC team members who provide direct patient care and peer support workers (hereafter, peers). All peers self-identified as having lived experience of mental health, substance use, and/or trauma-related disorders. As individuals who have been successful in the recovery process, and currently deliver BHCC, peers are able to provide both a direct patient perspective based on their own experience as well as an indirect perspective through their interactions with other patients in active crisis.³⁰ Third, for mental health treatment facilities that were willing to participate (n=5), we conducted in-person site visits during which we recruited additional interview participants. Finally, to increase the applicability of our framework across settings and include as many diverse voices and perspectives as possible, we also recruited interview participants involved in BHCC service delivery outside of Indiana using a national Listserv maintained by the National Council for Mental Wellbeing and a snowball sampling approach. As an incentive, all interview participants were offered a $50 e-gift card. The Institutional Review Board at Indiana University approved this study.

Data Collection

We developed a semi-structured interview guide to capture information about BHCC outcomes of importance to interview participants. More specifically, we asked interview participants 1) how they expected BHCC services to impact the community; 2) what impacts of BHCC they have already seen; 3) what outcomes of BHCC they believe are meaningful and/or would like to see more data on; and 4) what success of BHCC services looks like from their perspective. Interviews were conducted between February to September 2024. Given that we used multiple recruitment methods, our data collection procedures included a mix of virtual interviews conducted using teleconferencing software and in-person interviews conducted onsite. We obtained verbal consent to record and transcribe all interviews. Transcripts were professionally transcribed for accuracy and de-identified prior to analysis.

Analyses

To avoid the influence of any preconceived notions about outcomes of importance to interview participants, two researchers (AB and EM) independently reviewed 30 transcripts each and generated lists of all emerging themes using open coding.³¹ As themes emerged, we mapped them to relevant dimensions of the CHIEF using an iterative, constant comparison approach to develop a conceptual framework depicting interrelationships between identified themes. We then reconciled and refined these lists into a codebook with defined themes. After developing our codebook, two trained researchers then coded the same set of 10 transcripts for the presence of all themes defined in our codebook. We assessed inter-rater reliability by examining our percent agreement across all themes and calculating Cohen’s kappa. Across all themes, we achieved 92% agreement with a kappa statistic of 0.7, indicating strong inter-rater reliability.³² As an added check to ensure consistency across coders before coding the remaining transcripts, we reviewed disagreements identified during our inter-rater reliability assessment. In cases where agreement between coders significantly differed on individual themes, we clarified codebook definitions to improve consistency. After completing all coding, we calculated how frequently each theme occurred in our full sample and by participant type. We also extracted quotes to illustrate the most common themes that emerged within the system structure, implementation, and utilization dimensions as well as all social and health outcomes (i.e., PCOs) of BHCC.

Results

In total, we conducted 120 interviews with participants from 44 organizations located in 15 states, including Indiana (n=100), Illinois (n=2), Iowa (n=2), Missouri (n=2), Ohio (n=2), Rhode Island (n=2), Texas (n=2), Alaska (n=1), Colorado (n=1), Kentucky (n=1), Oklahoma (n=1), Pennsylvania (n=1), South Dakota (n=1), Vermont (n=1), and Washington (n=1). Participants represented the following perspectives: Peers with lived experience as patients with mental health and/or substance use disorders (n=21), BHCC service providers (n=47), program directors/managers (n=44), and executive leaders (n=8). All 21 peers (100%) were from Indiana, as were 40 BHCC service providers (85%), 33 directors/managers (75%), and 6 executive leaders (75%). The number of participants per organization ranged from 1–15 (median number of participants per organization: 1; standard deviation: 3.3). Aggregate organizational characteristics are presented in Table 1.

Table 1:

Characteristics of organizations represented by participants (n=44)

Organizational Characteristics	n	%
Facility Type
Community Mental Health Center	24	54.5
State Mental Health Department	16	36.4
State Substance Use Treatment Agency	15	34.1
Certified Community Behavioral Health Clinic	11	25.0
Federally Qualified Health Center	9	20.5
Psychiatric or General Hospital	3	6.8
Other	8	18.2
Facility Setting
Outpatient	42	95.5
Inpatient Hospital	12	27.3
Residential	11	25.0
Facility Operation
Private Nonprofit	38	86.4
Public/Government	5	11.4
Private For Profit	1	2.3
Crisis Services
Crisis Call Line	44	100
Mobile Crisis Team	41	93.2
Crisis Stabilization Facility	30	0.68
Area Served
Metropolitan	36	81.8
Urban	27	61.4
Rural	20	45.5

Note: Facility type, setting, services, and area served are not mutually exclusive. Organizational characteristics were obtained from the Substance Abuse and Mental Health Services Administration’s mental health treatment facility locator tool (https://www.samhsa.gov/find-help) and/or organizational websites. Areas served determined using Rural-Urban Continuum Codes (Codes 1–3= Metropolitan; 4–7= Urban; 8–9= Rural) from the United States Department of Agriculture (https://www.ers.usda.gov/data-products/rural-urban-continuum-codes).

Across all interviews, a total of 45 unique themes emerged. In adapting the CHIEF, all themes were grouped into one of five overarching dimensions: system structure, system implementation, system utilization, social outcomes, and health outcomes (Figure 1). Frequencies of themes across the full sample, as well as by participant type, are presented in Table 2. Below, we describe themes in more detail and provide illustrative quotes for the most common themes within each domain. Additional supporting quotes that illustrate each theme within the social outcome (Table 3) and health outcome (Table 4) domains are also presented.

Figure 1:

A Patient-Centered Outcomes Framework for Evaluating Behavioral Health Crisis Care, adapted from the Community Health Implementation Evaluation Framework (CHIEF)

Table 2:

Frequencies of emerging themes for evaluating behavioral health crisis care services, by participant type

Themes	All (n=120)		Peer/ Patient (n=21)		Executive Leader/ Director/Manager (n= 52)		Service Provider (n=47)
	n	%	n	%	n	%	n	%
System Structure
Established community partnerships (e.g., law enforcement, hospitals, schools, social resources)	17	14.2	3	14.3	9	17.3	5	10.6
Team size and composition	8	6.7	2	9.5	1	1.9	5	10.6
Team member well-being	5	4.2	2	9.5	2	3.8	1	2.1
Team member training and education	4	3.3	1	4.8	1	1.9	2	4.3
Cost-effectiveness of services	7	5.8	0	0.0	2	3.8	5	10.6
System Implementation
Public awareness of crisis services	24	20.0	7	33.3	6	11.5	11	23.4
Referral pathways to crisis services	11	9.2	4	19.0	2	3.8	5	10.6
Availability of crisis team (i.e., actual hours of operation)	8	6.7	1	4.8	0	0.0	7	14.9
Availability of police co-response	16	13.3	4	19.0	7	13.5	5	10.6
Use of a suicide screening tool	5	4.2	2	9.5	2	3.8	1	2.1
System Utilization: Crisis System
Crisis service utilization (e.g., number of crisis calls, mobile deployments, crisis center visits)	25	20.8	5	23.8	12	23.1	8	17.0
Response time (e.g., time to answer call, time from call to patient contact if deployed, patient wait time at crisis stabilization facility)	10	8.3	1	4.8	8	15.4	1	2.1
Length of encounter (e.g., call length, time spent on scene, time with patient)	4	3.3	2	9.5	2	3.8	0	0.0
Follow-up visits completed by crisis team	31	25.8	10	47.6	10	19.2	11	23.4
System Utilization: Patient Information
Patient demographics	20	16.7	2	9.5	10	19.2	8	17.0
Patient baseline status	10	8.3	2	9.5	7	13.5	1	2.1
Patient disposition	15	12.5	5	23.8	5	9.6	5	11.4
Patient experience	8	6.7	2	9.5	2	3.8	4	4.5
Patient satisfaction	5	4.2	1	4.8	4	7.7	0	0.0
System Utilization: Referrals
Referrals made to ongoing treatment for mental health or substance use	46	38.3	11	52.4	17	32.7	18	34.1
Referrals or direct admissions to inpatient treatment when indicated	25	20.8	6	28.6	8	15.4	11	23.4
Referrals made to social resources	22	18.3	7	33.3	9	17.3	6	12.8
Referrals made to medication management	11	9.2	4	19.0	4	7.7	3	6.4
Actual use of service after referral	21	17.5	4	19.0	4	7.7	13	27.7
Social Outcomes: Social Needs
Housing status	33	27.5	8	38.1	13	25.0	12	25.5
Transportation access	19	15.8	6	28.6	6	11.5	7	14.9
Sense of social support	16	13.3	5	23.8	0	0.0	11	23.4
Food security	13	10.8	3	14.3	8	15.4	2	4.3
Employment	9	7.5	1	4.8	3	5.8	5	10.6
Family reunification	9	7.5	4	19.0	2	3.8	3	6.4
Insurance	6	5.0	3	14.3	1	1.9	2	4.3
Safety	6	5.0	1	4.8	1	1.9	4	8.5
Social Outcomes: Preventable Experiences
Criminal justice involvement (e.g., police contact, arrests, jail time)	53	44.2	11	52.4	27	51.9	15	31.9
Child services involvement (i.e., foster care)	8	6.7	3	14.3	1	1.9	3	6.4
Exposure to public or structural stigma	9	7.5	2	9.5	1	1.9	6	12.8
Exposure to traumatic events (e.g., adverse childhood experiences)	2	1.7	0		1	1.9	1	2.1
Health Outcomes: Preventable Service Utilization
Emergency department visits	31	25.8	5	23.8	19	36.5	7	14.9
Inpatient hospitalizations	29	24.2	2	9.5	15	28.8	12	25.5
Emergency medical services (911) calls	9	7.5	2	9.5	5	9.6	2	4.3
Health Outcomes: Short-Term Health
Acute stabilization (i.e., de-escalation)	32	26.7	8	38.1	14	26.9	10	21.3
Mental health symptoms	14	11.7	5	23.8	2	3.8	7	14.9
Substance use behaviors	14	11.7	4	19.0	3	5.8	7	14.9
Health Outcomes: Long-Term Health
Overall well-being	11	9.2	2	9.5	6	11.5	3	6.4
Suicide rates	20	16.7	3	14.3	8	15.4	9	19.1
Overdose rates	5	4.2	0	0.0	2	3.8	3	6.4

Note: Values shown in table represent the total number and percent of participants in each group that mentioned the specified theme. Values are based on a total of 120 participants, including 21 peer support workers with lived experience, 52 executive leaders/directors/managers, and 47 service providers involved in behavioral health crisis care services.

Table 3:

Illustrative quotes capturing themes related to social outcomes

Themes	Quotes
Social Needs
Housing status	“Homelessness. Of course, we’re not able to give everybody a home from our little facility here. We try to help them fill out all the applications and community-based resources for housing like Rapid Rehousing, Section 8, HUD, anything like that… I feel like the homeless community around here might have some of the biggest changes and impacts from what we do here.” “Also I would probably think about—I’d go with housing and mental health and substance use, seeing if they’re keeping—it might be somebody that wasn’t housed that we went on a mobile crisis to. Six months from now, if they kept all of their appointments, seeing if they finally have a stable place to live. Or maybe a job. I don’t know, something like that where I can actually see that what we’re doing is helping them not only mentally, but also being able to feel like they’re successful. Like that they succeeded and see that improvement.”
Transportation access	“A lot of people either struggle with transportation, they struggle with a phone, or they’re just losing their kids and trying to get their life back on track. We’ve signed up so many people for cell phones and gotten them free phones and free phone services. We also transfer people to and from treatment if need be. I’ve transported people to [child services] meetings and met with them with their caseworker and things like that.” “I would like to track to make sure that we’re getting those 24-hour follow-up calls and also continuing to track our referrals, so making sure—’cause we can hand out referrals, but are we helping if they don’t go? So, making sure that if we are providing outside referrals that our clients are getting to them. If not, why? What can we do? Do you need a car ride, or can we take you somehow?”
Sense of social support	“I would say being able to be there for a client in a stressful situation, even if it is just they need someone to talk to… Just trying to make sure that that client feels heard and validated and like someone cares about them and is trying to help them.” “Every time I get involved with somebody, you wouldn’t believe how relieved people get from having somebody there that can just listen to them and be resourceful for them and kind of put everything together for them. I get a lot of, “Okay, I feel better, a huge weight off my shoulders, just because I have somebody here that can actually help us in the area that we need helped in.” I think it’s just a huge weight off people’s shoulders to know that we’re there, not only just to listen and be a peer support, but to actually implement those things, to have connections at this or this place and know how to do this or that.”
Food security	“I’d also like to know to what extent [social determinants] are playing a role in crisis… We know they’re going to the ER because they are hungry or they are unhoused and the weather is bad or they in a couple of cases we’ve had people that decided that, since they didn’t have any utilities, they should go to the ER because their utilities were turned off and they wanted, you know, access to a shower. So, I feel like knowing for sure what [social determinants] play a role in the crisis call would help us to come up with solutions like the one that knows he can go to the hospital and get a box of free food, because that’s going to prevent him from saying that he’s suicidal when he really is just hungry.” “I mean, really, it’s just the connection—the access to care, and resources, right? And so, mobile crisis might respond to somebody who, you know, we assess for social determinants of health as well, so maybe there’s a food insecurity. And so our mobile crisis team might be connecting them with resources for food, or transportation—which is really tough in our rural counties.”
Employment	“Like somebody who’s been homeless for a long time gettin’ a house or somebody who hasn’t been able to work because of their mental illness getting better and then being able to work and hold a job. Maybe somebody will be able to buy a car that wasn’t able to before or learn how to save money. We had somebody that used to come in here all the time and she would just say—every month it was something—I can’t pay this bill, I can’t do this, I can’t do that. It was always I can’t, I can’t, I can’t. Well, we can—you can work here, and you can work there, and we can connect you with these people and they might be able to get you a job.”
Family reunification	“When I first meet ‘em, they’re usually at the bottom—the very bottom of their lives. That’s what we call it, a bottom. And then we see ‘em progress to maybe getting a job or—and stable housing and getting back with their families or getting it—and can—it’s a lotta people—I have—that disconnected with their children because of either addictions or mental health issues. Then, they’re gettn’ their families back, and they’re gettn’ to enjoy all the things that they struggled with when they were in crisis, so yeah.”
Insurance	“They gotta pay this big old hospital bill because they had to take an ambulance to the hospital because they didn’t take their meds. Or maybe they lost their meds because they’re homeless or whatever their situation is. Then you got us, who’s the in between who got two vehicles out here and we can go pick ‘em up, let ‘em sleep for a night to get stable until we can get ‘em to a doctor or get ‘em into their doctor or get their meds. Let’s say they don’t have insurance. We have a care coordinator as well who can get them presumptive eligibility so that then they can get their medicine.”
Safety	“If they can have somebody meet them where they’re at and be able to get them in a hospital or get them even a safety plan and make sure they’re gonna be safe in the future, and make efforts for that, I think that’s success in itself.”
Preventable experiences
Criminal justice involvement (e.g., police contact, arrests, jail time)	“[An] example we’ve had a few times is when police were called to the scene for someone struggling with pretty intense mental health… It’s not often that you hear someone struggling with schizophrenia who wants help, but this person was interested in going to treatment. They had been pretty aggravated, and the police had stated that they had enough to actually take the person to jail, just based on the interactions that they’d had in the community. But they had reached out to us, because they knew that a lot of the behaviors could be stemmed from the individual’s mental health. And so, we were able to go out there and then instead of the person going to jail, we, he was willing to go into an acute stay. So we took him to an acute stay to try to get stabilized.” “We have decreased interactions with police for the clients that we’ve been serving. We’ve had them say several times, “Hey, I’m glad it was you that showed up to make sure I’m okay versus police making sure I was okay.”
Child services involvement (i.e., foster care)	“Child abuse cases in our area for young children. [That is] another area we want to impact.” “We’ve had some crisis calls after families have been involved with [child services] and children have been removed, which is a very traumatic experience. So being able to sit with someone, a few evenings. I was recently informed about one of those runs and just being able to kind of de-escalate and help ensure safety until they needed an appointment the next day, they wanted to come in.”
Exposure to public or structural stigma	“I think that helps with the stigma as well. I think that folks feel safe knowing that they can call and that someone will be there for ‘em at any given time and that they don’t have to call an ambulance and that the sirens aren’t gonna show up in stretchers, police officers aren’t gonna show up.” “To help as many people as we can and break that stigma of they’re no less—they’re no more, but they need help. A mental health crisis is no different than a heart attack, than a diabetic reaction, or whatever that looks like in the medical world. This is the same thing. To be able to have mobile crisis to be of service, it excites me so much that we can have this opportunity to be able to help at least one person.”
Exposure to traumatic events (e.g., adverse childhood experiences)	“One of the things we track is domestic violence, but we are not tracking life, separating lifetime versus current. A lot of times when we are going out, it could be the fact that they just left their significant other. In history too, maybe their first relationship was okay and that person passed, but then since then they’ve been involved in multiple domestic violence situations. As that number increases, that’s going to continue to affect their mental health and make them more at risk.”

Table 4:

Illustrative quotes capturing themes related to health outcomes

Themes	Quotes
Preventable Service Utilization
Emergency department visits	“I would say our kind of main objectives in terms of diversion is yeah, really keeping people out of long emergency room waits when they’re typically not kept and are kind of turned back out onto the street pretty quickly. You know, just really trying to reduce hospital stays, jail stays, police contact. With, you know, the ambulance, reducing 911 calls. I would say that we are trying to provide an alternative response to those outcomes.” “Generally the success is they’re stable and able to remain in their home. We have an 85% rate of stabilized in the community so they don’t have to go to the emergency room… We keep them in their home and that’s our goal. We want them to be able to stay in the community if you know they’re capable.”
Inpatient hospitalizations	“Obviously for us it’s that reduction in hospitalization. Being able to connect them to address their needs on multiple levels because even if you address the mental health side of things, you get them connected to a therapist or medications that only provides so much support if they don’t have transportation, if they don’t have food in the table or they don’t have a roof over their head. Those are still going to be persistent stressors in their life that when added up over time or when something else gets added to their plate, they’re gonna continue to be in crisis. Making sure that you’re viewing them as a whole and addressing all of those needs that they are willing to address.”
Emergency medical services (911) calls	“I feel like a year from today, the impact is going to be noticeable. I feel like people would be more comfortable calling 988 than 911 for a lot of the situations where 911 was the immediate answer there. My uncle is going through a panic attack. It’s something as simple as a panic attack. I don’t know what to do, so they call 911. Now their poor uncle is sitting with an ambulance bill… Somebody needs to come out there and talk to somebody or make sure that they are safe, make sure they have a safety plan in place. I feel like it alleviates the burden on 911 services.”
Short-Term Health Outcomes
Acute stabilization (i.e., de-escalation)	“Our main focus has always been to de-escalate and to really help.” “I get to see people before the instance outreach hopefully deescalate, keep things from getting to that point. Which I think it has made a huge impact with our community. I think there is a need for an outreach team per se, what we do for immediate help.” “I wish there was a way or that we found a way to better track like, was this person actually deescalated during the call?”
Mental health symptoms	“Ideally, you would be able to track who was treated properly, but how do you really measure that? Or, who overcame their problems? Or who’s not struggling anymore? It’s like, I don’t know if you could really track those things. Ideally, those are the kinds of things I would want to know. Was this helpful for the client? Did this help them reduce their anxiety? Or did this help them reduce their depression? Or whatever it is. I don’t know how you go about tracking those things.”
Substance use behaviors	“Outcomes? Seeing those who struggle with mental health actually just getting into services, taking that step. I feel like a lot of it wouldn’t happen if they didn’t know about us or come to us, if they were skeptical about it. We can give them that entryway into getting into services for mental health, especially. I see people who struggle with substance use. They come to us as well, which I then help them get into facilities. Some people will completely turn life habits around after that and stay clean, or try their best. If they relapse or struggle, they come to us, and we help them again with it.” “We have to meet people with where they’re at, and if they do get any kind of help from that right away, that makes me happy. That’s success for me. Especially since we don’t hold grudges per se. We don’t ban people really. It’s people can come back if they relapse and they’re like, “Hey, it’s me again. I know you saw me last month. I relapsed.” We’re like, “Okay, that’s okay. We’re not here to judge you. We’re gonna get you the help. That’s all that matters.” That’s all that matters. That to me, just to see those people coming back and reaching out for help, that’s success. Rather than them being out on the streets and not reachin’ out for help.”
Health Outcomes: Long-Term Health
Overall well-being	“Then it’s the caller’s wellbeing as well. We’re lookin’ at the entire picture there to make sure.” “Now they have someone they can talk to. Because you should understand also that when the kid or the youth is in crisis, the parents are also in crisis. We try to make sure that both of them, their wellbeing is addressed, they are both safe, they are also mentally okay because the mom may get drained with these behaviors and things like that. We want to give them that space that they need, and help to take care of this kid. Yeah, it’s playing a very big role, it’s making that difference.”
Overdose rates	“Overall, decreasing the numbers of overdoses, people that [die by] suicide. I guess decrease maybe in stigma of mental health, just an improvement of everybody’s well-being in the community, improvement of that.”
Suicide rates	“Substance use was really high in our county. Suicide rate is pretty high in our county. So, those were some of the data points we wanted to impact.”

System Structure.

Five themes related to system structure emerged: establishment of community partnerships, team size and composition, team member well-being, team member training and education, and cost-effectiveness of services. Participants most frequently commented on the importance of establishing partnerships with community organizations, including law enforcement, hospitals, schools, and social resources. As one participant stated,

“When you’re dealing with court-involved individuals, there’s also that collaboration there… I think the way I would just describe it, it’s just better information sharing across organizations. And that helps just provide, what is really the best way to support the individual that’s in crisis?”

Participants also commented on the role of team dynamics (e.g., team size, composition, training, education) in ensuring success of BHCC services and the importance of maintaining team member well-being given the nature of the work BHCC service providers perform. As one participant stated,

“We have to be there for each other. We have tough jobs. We have mentally draining jobs, so we have to take care of each other and ourselves before we can take care of the community or take care of others. I would say it’s absolutely a positive impact in our community.”

Finally, participants commented on cost-effectiveness of services and a goal of being able to demonstrate a reduction in costs incurred by both patients and other safety-net service providers in the community. As one participant stated,

“We’re really interested in the cost savings and doing a cost savings analysis so that we can continue to educate our fellow service providers, our legislators, our county commissioners, and our other public servants about the amount of money that these services are saving for our communities.”

System Implementation.

Five themes related to system implementation emerged: public awareness of services, referral pathways to crisis services, availability of crisis teams (i.e., actual hours of operation), availability of police co-response, and use of a suicide screening tool. Participants most frequently commented on public awareness of BHCC services and actions they were taking to ensure that community members are aware of BHCC services as well as efforts to develop referral pathways to increase access. As one participant stated,

“I think that we’re getting out there more. People are more aware of us, so that’s been a change, but I feel like as far as the call volumes and the kind of calls that we’re getting, I think they’re just as strong as they were at the beginning. I think because there is such a great need. I think through education, through the community, and those kind of things, people are becoming more aware that, oh, this is okay that I’m feeling this way and that I need to get help.”

Participants also commented on goals of increasing availability of services and expanding hours of operation. While many facilities were offering BHCC services on a 24-hour basis, others were still in the process of increasing their capacity in order to support around-the-clock service availability. As one participant stated,

“I think we’ve at least put our names out there and shown people that we are here to help and that, if you need assistance, we will be here, especially since we always tell people we’re 24/7/365. This place doesn’t take a day off. If anything, I think we’ve made more of an impact there than anywhere else.”

System Utilization.

Fourteen themes related to utilization emerged, which we further divided into three subcategories: crisis system utilization, patient information, and referrals to additional services. Themes related to crisis system utilization included the use of crisis services (e.g., call lines, mobile teams, and crisis stabilization facilities) and completion of follow-up visits conducted by crisis teams. As one participant stated,

“We want everyone to have outreach within 14 days. So, if you come to [the crisis stabilization unit] or we’ve responded with the mobile crisis team, we will have a peer follow up within 14 days to check in, make referrals to other services, and see if there’s anything else they need.”

Themes related to patient information captured patient demographics, baseline status, and disposition as well as their overall experience and satisfaction with services. In terms of patient experience and satisfaction, participants expressed a desire for better tools to capture this information. As one participant stated,

“We don’t have a satisfaction or a how much did we help in this crisis moment. We don’t have a great way of doing that. We’ve talked about certain tools, but that feels hard to do on a crisis call… We don’t do that very well at this point. From a client perspective, what client satisfaction looks like.”

Within the referral subcategory, the most common theme was referrals made to ongoing mental health and/or substance use treatment. Participants also talked about the ability to facilitate a direct admission or refer an individual to inpatient services when indicated, referrals to social resources, and referrals to medication management. Finally, participants expressed interest in being able to determine whether patients ultimately use services after referral but noted that this information is more difficult to track, especially when the referrals are for an external organization. As one participant stated,

“I would like to track to make sure that we’re getting those 24-hour follow-up calls and also continuing to track our referrals, so making sure—’cause we can hand out referrals but are we helping if they don’t go? So making sure that if we are providing outside referrals that our clients are getting to them.”

Social Outcomes.

Twelve themes related to social outcomes emerged, which we divided into two subcategories: social needs and preventable experiences (Table 2). The most common themes related to social needs were housing status, transportation access, and sense of social support. As one participant stated,

“Also I would probably think about—I’d go with housing and mental health and substance use, seeing if they’re keeping—it might be somebody that wasn’t housed that we went on a mobile crisis to. Six months from now, if they kept all of their appointments, seeing if they finally have a stable place to live. Or maybe a job. I don’t know, something like that where I can actually see that what we’re doing is helping them not only mentally, but also being able to feel like they’re successful. Like that they succeeded and see that improvement.”

As for preventable experiences, participants spoke primarily about preventing criminal justice involvement, as well as preventing traumatic experiences and exposure to public or structural stigma. Preventing criminal justice involvement included preventing contact with police in the first place, preventing arrests, and preventing individuals from serving jail time by providing an alternative option. As one participant stated,

“I think the biggest impact is that we work a lot with our [local] police department. And if our police department is on a call for someone who has been caught trespassing or someone who is in a mental health crisis, rather than taking them to jail or taking them to the hospital, they are calling us and asking us to come and help these people, to assess them, get them somewhere safe so they don’t have to go to jail or the hospital.”

Participants also spoke about goals of preventing child services involvement when possible, preventing exposure to traumatic events such as child abuse and other adverse childhood experiences, and reunifying families in cases where children have been placed in the foster care system. As one participant stated,

“And then the early childhood stuff was the other place [we want to impact], because the foster care rates were really high, as was child abuse rates in our area for young children.”

Health Outcomes.

Nine themes related to health outcomes emerged, which were divided into three subcategories: preventable service utilization, short-term health, and long-term health (Table 3). Participants described preventing repeat crises and reducing utilization of crisis services delivered in more intensive, restrictive, and expensive settings, including ED visits, inpatient hospitalizations, and 911 calls as a primary objective of BHCC. As one participant stated,

“I would say our kind of main objectives in terms of diversion is yeah, really keeping people out of long emergency room waits when they’re typically not kept and are kind of turned back out onto the street pretty quickly. You know, just really trying to reduce hospital stays, jail stays, police contact. With, you know, the ambulance, reducing 911 calls. I would say that we are trying to provide an alternative response to those outcomes.”

Participants also described short-term health outcomes of importance from their perspectives, with a focus on acute stabilization/de-escalation of the crisis moment. Beyond resolving the current crisis, participants also spoke about wanting to reduce mental health symptoms and substance use behaviors among patients who receive BHCC services. As one participant stated,

“Ideally, you would be able to track who was treated properly, but how do you really measure that? Or, who overcame their problems? Or who’s not struggling anymore... Ideally, those are the kinds of things I would want to know. Was this helpful for the client? Did this help them reduce their anxiety? Or did this help them reduce their depression?”

Finally, several participants mentioned long-term health outcomes of importance, including overall well-being, suicide rates, and overdose deaths. As one participant stated,

“Overall, decreasing the numbers of overdoses, people that [die by] suicide. I guess decrease maybe in stigma of mental health, just an improvement of everybody’s well-being in the community, improvement of that.”

Discussion

This study sought to identify PCOs of BHCC by identifying outcomes that matter to individuals with lived experience of mental health and/or substance use disorders (i.e., patients) and/or experience delivering BHCC services (i.e., providers). Through qualitative interviews with 120 patients and providers, we identified 45 themes capturing measurable components of BHCC. We then used emerging themes to adapt a conceptual framework for evaluating the impact of BHCC on PCOs. All themes were grouped into 5 overarching domains: system structure, system implementation, system utilization, social outcomes, and health outcomes.

In terms of BHCC outcomes, we identified 12 social outcomes and 9 health outcomes of importance to participants. Given that these outcomes emerged directly from the perspectives of patients and providers,^14–16 they represent PCOs of BHCC. Notably, while the primary function of BHCC services is to provide acute de-escalation and stabilization of the individual in crisis,⁷ participants viewed the potential impacts of BHCC services to be much broader, encompassing care coordination and referrals to follow-up services for both social and health-related needs. Many participants described goals of preventing utilization of higher levels of care (e.g., emergency department, inpatient hospitalizations) and criminal justice involvement, as well as goals of connecting individuals with follow-up services to address the underlying factors contributing to their crisis episode, including both unmet social needs (e.g., housing, transportation, social support) and untreated mental health and substance use disorders, known contributors to behavioral health crises.^33–35

The PCOs framework derived from themes identified in this study provides a starting point for organizations that offer BHCC services to develop an evaluation plan for assessing quality and effectiveness of care in terms of PCOs. While the social and health outcomes defined in our conceptual framework are the dimensions that reflect PCOs, identifying measures for tracking all dimensions is important to monitor progress and evaluate the impact of structure and process measures (which may differ across sites) on PCOs.²⁷ Importantly, the themes captured in our conceptual framework represent concepts that can be measured in multiple ways. Organizations developing an evaluation plan guided by this framework should define more specific measures to operationalize and capture these concepts.^36,37 In some cases, organizations may already be tracking data on specific measures that capture these constructs, whereas in other cases additional efforts are needed to develop and implement data systems that facilitate measurement of PCOs. For example, as part of the federal demonstration program, Certified Community Behavioral Health Clinics are required to report 21 quality measures capturing best practices in care delivery, patient and family member experiences of care, and patient outcomes (e.g., changes in depression symptoms).^38,39 Thus, certified clinics may already be tracking measures aligned with our framework, whereas other types of mental health treatment facilities may need additional support to feasibly collect data reflecting PCOs. Given calls for quality measures that better capture health outcomes,³⁹ future research should crosswalk existing quality measures with PCOs to identify gaps and opportunities for development of additional quality measures.

This study has strengths and limitations. We sought to capture perspectives from a diverse group of patients and providers involved in BHCC service delivery across a range of settings. While we interviewed participants from 15 different states, most participants were from the midwestern region of the US. Patients and providers in other regions may have different perspectives. Since we did not collect individual demographic characteristics as part of this study, we are unable to assess whether our participants are representative of the workforce and/or population served. Finally, given ethical and logistical barriers (e.g., identifying potential participants, study timeline, available funding) to recruiting patients who have recently experienced a crisis to participate in our study, we purposively recruited peer support workers with lived experience of mental health and/or substance use disorders to provide a patient perspective. As individuals with lived experience as patients who also frequently interact with other patients in active crisis when providing peer support, peer support workers provide a unique end user perspective. However, given that peer support workers are in a different stage of recovery than many patients, their perspective may not be representative of all patients with behavioral health conditions. Thus, future research should seek to incorporate perspectives from more diverse patient populations.

Conclusions

The PCOs framework for evaluating BHCC developed in this study provides a starting point for organizations that offer BHCC services to assess quality and effectiveness of care. While there are numerous potential measures of interest to patients and providers, the social and health outcomes identified in this study should be thought of as a starting point for defining PCOs of BHCC rather than a comprehensive list. Additional efforts are needed to select and prioritize PCOs of interest and to develop, refine, and test the application of more specific measures that operationalize PCOs across various BHCC services and settings. Future research should seek to accomplish these goals through consensus methods and co-production processes that center the voices of patients and providers.

Highlights.

• To inform efforts to assess the quality and effectiveness of behavioral health crisis care, this study developed a patient-centered outcomes framework for evaluating behavioral health crisis care.

• Qualitative interviews were conducted with 120 individuals with lived experience of mental health and/or substance use disorders and/or experience delivering behavioral health crisis care services to identify outcomes of interest to patients and providers.

• Themes captured five conceptual domains: system structure, system implementation, system utilization, social outcomes, and health outcomes. The most common social and health outcomes mentioned by participants were criminal justice involvement, emergency department visits, housing status, and suicide rates.