Use of data-driven decision-making among agencies serving individuals with criminal-legal system involvement: a qualitative study

Abstract

Background

Data-driven decision-making is an implementation strategy to improve system performance and health outcomes for individuals with substance use and mental health needs interacting with criminal-legal systems. We currently have limited understanding of how agencies define and use data in their decision-making processes. A qualitative study was conducted with agency leaders from jails, probation and parole, and community mental health and substance use service agencies to characterize (1) how respondents defined data and data use, (2) current level of data use for decision-making, (3) data capacity needs, and (4) agency perspectives on best practices and their recommendations.

Methods

In-depth interviews were conducted with 68 mental health, substance use, jail, and probation leaders from a representative group of 52 U.S. counties from December 2020-March 2022. Data were analyzed using applied thematic analysis.

Results

Four themes emerged. For defining data and data use, respondents characterized their current level of data use as rudimentary. In terms of current level of data use, individual-level health or legal information needed to coordinate services, and risk prediction scores were used over system measures. Data capacity needs identified as capacity to track and share data across agencies and inadequate data management. In terms of best practices and agency recommendations, participants highlighted the need for data sharing, standardized data management systems, data dashboards, and in-house data expertise.

Conclusion

Findings from this study highlight the limited capacity of counties to implement data-driven decision making in mental health, substance use, and criminal legal settings. Addressing gaps in infrastructure, staffing, and data expertise will require technical assistance and targeted policy incentives. System-level metrics and benchmarks, such as the cascade of care, are essential for evaluating cross-agency performance and improving outcomes for individuals with mental health needs.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-025-13385-2.

Background

United States criminal legal systems, referring to interactions that occur from 911 calls, police interactions, pretrial jail detention, jail incarceration, and community supervision, are the largest provider of mental health care in the country [1]. Individuals with mental health and substance use service needs interact with criminal legal systems more frequently than do members of the general population [2]. Mental health services are at the intersection of the criminal legal system and the community [3, 4]. While in the community, individuals with mental health disorders interact with agencies including community mental health, community substance use services, community corrections and other social service agencies. Understanding these interactions with community agencies and services is critical to creating effective mental health responses to improve mental health, substance use services and criminal legal outcomes.

Arrest and incarceration are not the only ways individuals with mental health service needs interact with criminal legal systems. Even among those who are arrested, the vast majority get released from jail detention back to the community within hours, days or weeks. Community mental health providers serve criminal-legal involved individuals in many contexts: prior to arrest, during community reentry, and throughout community supervision and specialty courts [5], to name a few.

Data-driven decision-making (DDDM) is a strategy [6] used to implement evidence-based programs and policies in health and other settings. This involves the use of routinely collected metrics, facts, and trends to inform strategic decisions about service delivery and programming [6]. Data-driven decision-making has emerged as a key approach for implementation and quality improvement [7–9]. Data-driven decision-making requires the ability to measure and track service delivery metrics on a regular basis to make and evaluate decisions about services, policies, and programs and guide ongoing policy and practice improvements. DDDM has shown to improve service delivery and contribute to successful treatment referral [10, 11]. While data-driven decision making (DDDM) is recommended for use in criminal legal settings, its adoption beyond early adopters—jurisdictions that have served as pioneers in integrating DDDM practices—remains insufficiently documented in the literature.

Since it gained popularity in recent years, the use of data-driven decision-making in mental health services to optimize public health resources has grown tremendously [7]. Nevertheless, evidence from other settings suggest that several internal and inter-agency level challenges to data-driven decision making persist [12]. These include lack of infrastructure to track metrics, incomplete and fragmented data, poor data quality, limited availability and inaccessibility of data, to name a few [12]. Privacy laws such as Health Insurance Portability and Accountability Act (HIPAA), Code of Federal Regulations (CFR 42; addiction treatment). data security issues, and lack of familiarity of clinicians with criminal legal processes can also be barriers to tracking metrics across agencies [13]. Nevertheless, barriers experienced by mental health and criminal legal agencies are not reported in the literature.

The Stepping Up Initiative [14] is a national effort to help counties reduce the number of individuals with mental illness who are in jail by improving access to community mental health care. The Initiative recommends data-driven decision-making as a centerpiece of its approach. Recommended steps include having agencies create common definitions of and track four key metrics across community mental health, community substance use, jail, and probation agencies in their counties: (1) number of jail bookings of individuals with serious mental illness, (2) length of stay in jail, (3) connections to treatment in the community, and (4) recidivism. Over 550 counties participate in Stepping Up, pledging to attempt to define and track these metrics. The Implementation Mechanisms for Justice and Behavioral Health (I.M. Justice BH) study assesses the data-driven decision-making processes used in Stepping Up and matched comparison counties through qualitative and quantitative surveys at 3 time points (baseline, 18 and 36 months) [15].

This study analyzed the baseline qualitative interviews from 68 mental health, substance use, jail, and probation representatives in 52 U.S. counties to characterize (1) how respondents defined data and data use, (2) current level of data use for decision-making, (3) data capacity needs, and (4) agency perspectives on best practices and their recommendations. The study also aims to identify possible priorities for technical assistance.

Methods

Population and sample

The sampling procedures for this study were nested within the larger Implementation Mechanisms for Stepping Up (I.M. Stepping UP) study baseline quantitative survey [15]. We used weighted stratified random sampling to ensure that the qualitative data was representative of the quantitative sample, which was weighted to be representative of all U.S. counties. The study statistician randomly selected a U.S. nationally representative subset of 60 counties for the qualitative survey based on the following criteria: county size (small [population less than 250,000], medium [population between 250,000 and 750,000], and large [population over 750,000]), their participation in the Stepping Up initiative (yes or no), and agencies to be interviewed (i.e. mental health, substance use, jail, probation). Our qualitative interviewers contacted the assigned agency for each county. If primary respondents refused participation or could not be reached, our team contacted representatives from the other agencies in the same county.

Participants

Study staff reached out to agencies from the 60 representative counties. Participants from 52 counties, representing all corners of the country, were successfully enrolled. Qualitative interviews were conducted with participants representing (1) community mental health agencies, (2) community substance use providers, (3) jail administration and (4) community corrections (probation and parole officers).

Data collection

Data were collected from December 2020-March 2022. Consenting participants were contacted and virtual qualitative interviews were conducted over a HIPAA compliant Zoom meeting. In-depth interviews were conducted with all participants. When requested by the original participant contact, interviews were conducted with multiple participants representing different roles in that county. In order to facilitate discussion and equal contribution, no interview session included more than three participants. About two-thirds of the interviews included a facilitator as well as a note taker. Interviews lasted from 60 to 90 minutes. All interviews were recorded, and audio files were later uploaded to the study secure server.

Interviews

All interviews used the same qualitative topic guide developed for this study for interview facilitation (see supplementary file 1). A team of interviewers (1 male, 3 female) were trained by three experienced qualitative researchers with expertise in the study design, implementation science and work with the study population. The interviewers were masters- or PhD-level researchers with prior qualitative research experience with the study population. Training covered: (1) study goals and research questions, (2) mental health and the criminal legal system, (3) implementation science, (4) qualitative research methods including interview facilitation skills and (5) practice interviews using the topic guide. Debriefs were written after every interview and used to track implementation of the topic guide, facilitation experiences, and key data points, including the recurrence of key concepts. The team met weekly to discuss data collection, including data saturation.

Contextualizing data-driven decision-making

To ensure participant understanding of the term “data-driven decision-making,” we presented an operational definition as “the use of information about admission rates, length of stay, presenting complaints, health or justice outcomes, return to jail, hospitalization, costs, patient satisfaction, waiting times, etc. to make decisions about budgeting, staffing, medicine supply and management, services offered, or new programs.” Then we asked participants questions about their county-/agency-level experiences of how they obtained data, how they used data, and what data capacity needs they may have.

Data analysis

Data were transcribed verbatim by a professional transcription service. Our team developed a transcription guideline which was shared with the transcription service. Transcripts were reviewed for accuracy and completeness by the study team. Once the transcripts were cleaned, they were imported into NVivo data analysis software. The qualitative codes were developed by using a priori concepts drawn from the study research questions and interview guide (deductive codes). Data-driven decision-making capacity and how it developed were a priori codes drawn from study hypotheses [15]. As needed, additional codes were added to capture concepts raised by the participants (inductive codes). All transcripts were coded independently by two coders. The coders met to review their coded transcripts together and worked to establish consensus. Applied thematic analysis was conducted whereby the coders used the codes to label passages within the transcripts [16]. After all the transcripts were coded, data were extracted and the team created summaries of the key themes of the study pertaining to the above definition of data-driven decision-making.

Results

The 68 county-level administrators included in this study represent mental health agencies (N = 22), substance use service agencies (N = 9), probation (N = 3) and jail (N = 18). About a third of the 52 interviews included multiple respondents (up to three) representing different agencies within the same county, resulting in total of 68 participants. We interviewed 42 participants from Stepping up counties and 26 from comparison (non-Stepping up) counties. Table 1 presents composition of the study participants.

Table 1.

Composition of participants

Interviewees	68 (100%)
Number of counties	52 (100%)
Agencies
Jail	18 (35%)
Probation	3 (6%)
Mental health	22 (42%)
Substance use	9 (17%)
County size
Small	39 (75%)
Medium	10 (19%)
Large	3 (6%)

Four themes emerged including (1) defining data and data use, (2) current level of data use to guide data driven decision making, (3) data capacity needs, and (4) agency perspectives on best practices and agency recommendations.

Defining data and data use

Participants’ understandings of data-driven decision-making, regardless of the agency type or the size of the county, was considerably different from the operational definition provided by the interviewers (Table 2). During the interviews, we presented an operational definition of data-driven decision-making (see above and Table 2), however, participants discussed different kinds of, and terms for, data use, with “needs assessment”, or “data monitoring” used most commonly. Participants from criminal legal agencies used the term “risk assessment” when discussing “data,” referring to information from a tool that they use to characterize the public safety risk of the individual (i.e. likely future involvement with the criminal legal system) as well as criminogenic needs (i.e. substance use, antisocial values, antisocial attitudes, family supports, etc.); this information is usually used to classify the risk level of individuals being detained—it is not a measure of agency-level performance.

Table 2.

Definitions and operationalization

Operational Definition of Data-driven decision Making (DDDM)	Terms used by participants	Level of use
Use of information such as admission rates, length of stay, presenting complaints, health or justice outcomes, return to jail, hospitalization, costs, patient satisfaction, waiting times, etc. to make decisions about budgeting, staffing, medicine supply and management, services offered, or new programs	Risk assessment	Individual level health and criminal legal risk and need factors
	Needs assessment	Agency and individual level health and criminogenic factors
	Data monitoring	Agency level

A jail administrator also said:

Interviewer: What kind of specific data points do you have access to, and how do you use those to inform your decision-making?

Participant: Even as we’re looking at recidivism data and the risk data, someone can—maybe they didn’t go to jail, but they’ve been picked up by police three times in the last six months. We aren’t gonna know that. J-17.

A mental health agency leader said the following:

Interviewer: How does your county or your agency use data or information to make decisions about services for justice involved individuals?

Participant: Data. I mean, we are required to assess individuals with the adult needs, ANSA, Adult Needs Surveys something Acute. That’s the ANSA. Based on that, so first they have to be determined eligible, even if they don’t automatically come into our service. MH-49.

Hardly any (N = 4) of the participants made reference to tracking agency-level trends in performance metrics. Some agencies collected data as program deliverables but did not analyze or utilize the data further to make decisions.

Participants did not address how data were used to make decisions about staffing, services or new programs, although some participants discussed how individual-level data helps forecast future demand for new facilities (e.g., the need to develop new housing units for individuals who will be coming out of jail in the future). Table 2 below presents definitions and operationalization.

Current level of data use and DDDM

Data collection and sharing

 Across agency types, most respondents characterized their data as rudimentary and highly fragmented. Participants cited unreliable and/or inconsistent data collection practices within or across agencies as barriers for data use. This was particularly true for data collected by outside agencies as described below.

…what I would like to have is more uniform data, so when we get data from other agencies, we look at it and not exactly sure what it means, or maybe we wouldn’t capture the data the same way they are because we come up with what-ifs, so a more universal definition of what’s being collected and what needs to be collected. MH-42.

Among those who collect data, both electronic and print data were available. However, a trend towards automation of data was reported especially for mental health and substance use services data. Several counties reported implementing a web-based data infrastructure. However, many counties reported that their data were unusable because they were crude, incomplete, and/or required significant cleaning before use:

Yeah, so the EMS data is, I actually did a great job of digging into it last fall and haven’t had a chance to look at it again. It’s pretty complicated. He sends me just summaries of the calls that they have. I can filter it by behavioral health issues”. SU-30.

Mental health agencies also leveraged their connections with other county-government agencies to have access to data: it helps to be a county employee ’cause I can get county EMS data, which gives us a good bit of information, especially about overdoses and Narcan usage.” MH-8.

Among participants from mental health and substance use services agencies, the most commonly reported reasons for data use were to: (1) assess whether a program was working; (2) determine new priorities; (3) understand what impacts the agency is making on key outcomes such as reducing recidivism, emergency room use, and/or suicide attempt, to name a few; (4) make funding decisions such as where to allocate money and what grants to seek; and (5) identify potential areas of need. In contrast to that variety of uses in mental health and substance use, participants from criminal legal agencies (jails and probation) did not report using data to make decisions, although there was a general interest in tracking recidivism.

There was variation in how individuals discussed using data. For those few counties (~ 10 out of 52 interviewed) that systematically used agency-level data to make decisions, they did so to make decisions regarding hiring, funding applications, and program expansions. For example, some counties monitored data trends to determine if programs led to intended outcomes.

I look at trends, I look at numbers, [to ask] is this program growing, shrinking, staying the same? They [managers] know they see the numbers whether it’s billing numbers or whether it’s appointments or number of people showing up….if I say, “How many clients do you have?” They’ll tell me, “They’re 900, 903.” They know that. … What I would like to give them is something that would be saying, “Okay, if you see this pattern in somebody, they’re more likely to relapse,” or, “If they have six weeks of positive urine drug screens, that means you need to do it [intervene].” … What would be cool to know: is there anything that you could see that leads people to having negative urine drug screens? SU-41.

However, other mental health agencies indicated that their data use did not necessarily inform initiating new services. For services that are covered under Medicaid funding, there is little to no room for program modifications.

Nature of data and kinds of decisions it informs

 As opposed to collecting and monitoring agency-level data for quality improvement, when asked about data, agencies often discussed collecting or sharing individual-level health or legal information to coordinate services. For example, mental health agencies extensively mentioned sharing information about individuals to help triage cases. Mental health agencies interacted with the criminal legal agencies (courts) when they were requested to submit data pertaining to the mental health of individuals going through the court system. They mentioned that the court has formal or informal agreements with other agencies serving the same population. Individual-level data collected from mental health agencies were used to determine their mental health status or to redirect individuals to mental health services. There were instances where the jails and courts also shared release dates with community corrections and health agencies to ensure that individuals have prompt access to services upon reentry. There were also cases where specific information regarding an individual’s health and housing situation were collected to determine release conditions including ability to reintegrate to general population or ability to connect with services upon release. These data were also used to get individuals involved with local services and/or for probation decisions.

Compared to criminal legal agencies, mental health and substance use agencies seemed to have a more structured approach to tracking and using performance metrics to make program decisions about budgeting, medication supply, quality of service provision, grant seeking and staffing. Criminal legal agency performance metrics were limited in terms of type and purpose of data collected (mainly to track recidivism and length of stay).

Some participants from mental health and substance use agencies tried to use trends in data (e.g. increases in the number of alcohol-related hospitalization and drug use, overdose, types of diagnosis and referrals) to define program priorities. Despite identifying trends and needs, participants reported that their findings did not necessarily lead to new programs or services, as funding availability usually determined whether new programs could be implemented.

… I know how it would be used, but because there’s some interventions that we just don’t have in place, we have the ability to track them [the needs], or we have the ability to track outcomes, but we don’t have the interventions to see if that’s impacting the outcomes. SU-47.

Some mental health and substance use service agencies mentioned having dedicated inhouse data expertise. For example, one agency has an office called Continuous Quality Improvement (CQI) which provides quarterly reports on metrics including number of individuals placed in mental health medical units, new intakes for acute substance use problems and suicidality.

While most agencies used individual-level data, a few used system-level metrics to make decisions. These included, number of individuals arrested for mental health-related reasons, average length of stay, number of bookings, and average in-custody time. Moreover, mental health administrators mentioned that they monitored the number of crises calls and call volume to determine the need for staffing.

Metrics of interest

 Agencies differed in their primary metrics of interest. For example, probation and parole leaders had great interest in tracking and documenting recidivism. Mental health and substance use service agency leaders expressed interest in outcomes of mental health and substance use treatment.

“Here’s one of those aspirational goals of mine. … I would like to have one year and three-year outcome data on the individuals that were treated for drug and alcohol addiction because… we do a lot of self-reporting, our agencies do calls and backup letters and emails and stuff to get their data, but what we don’t have is, what does that look like the year out? What does it look like three years out and what can we do differently if those numbers are as I suspect they are, not as good as they are at the end of three months or nine months post-treatment. What could we do differently to reinforce or help that aftercare aspect so that people aren’t facing the same issues they had three years ago.” MH-27.

Some of the mental health agencies had access to short-term treatment or criminal legal outcomes data for the populations they serve. However, they admitted that tracking longitudinal outcomes data was a challenge because of lack of follow up assessments. Agencies with longitudinal data were more likely to modify their services to fit the outcomes they want to improve.

Leaders from mental health and substance use services agencies mentioned their interest in individual-level data characterizing who is in jails, service needs, experiences of homelessness, nature of interactions with jail staff, and time spent with law enforcement officers. They expressed interest in individual-level outcomes:

“I think one of the most helpful data points is whether or not a person is already actively engaged with social services and hasn’t opened an active work group. That’s a fairly excellent data point I should say, to help us know where to start with coordination and figuring out how to best help and support and reengage that person with their existing services. Then the second most helpful the historical involvement, whether or not we’ve ever had an open work group with that individual, or if we’ve had a number of contexts with that individual that never resulted in formally opening some sort of case management service…Whether or not they have current or historical involvement with our social service delivery system has been a very beneficial data point for us to utilize in knowing where to begin with the service coordination and service” MH-12.

Data capacity needs

Participants were asked about their current information and data capacity needs. Most of the counties reported that there is a great interest in tracking data. However, they underscored that the capacity to track and share data across agencies remains limited.

Easier avenues for data sharing

 Agencies in some of the counties seem to work in siloes and have limited interagency data sharing arrangements.

We’re constantly crunching numbers in my office. I would say on the downside of things, I think we still continue to work in siloes and pockets when it comes to data sharing, right? And to cross-compare meaning making of that data, and how one area leads to another, but being an evidence-based decision-making county, data is our backbone for many of the services… especially when we’re looking at alternative responses for those that are involved in the criminal justice system. We’re constantly looking at data. J-28.

Most of the participants acknowledged that they have struggled to collect or gain access to data collected by other agencies. Participants from smaller counties, however, noted that being a small county often facilitates their access to data because finding out what is happening in the county was said to be facilitated by smaller size and frequent interactions.

For some of the counties, limited/restricted access to data was repeatedly mentioned as were concerns related to data capacity (sharing and analyzing). Almost all agency leaders endorsed that access to Medicaid data would give them a big-picture analysis of their services. However, they also felt limited by policy restrictions regarding access to Medicaid data. They shared that they have access to data for services they paid for only; excluding data from services paid for by Medicaid or other third-party payers.

“The data system is split. I don’t get what’s provided by Medicaid, or even stripped-down data without name identifiers and that kind of thing. There’s just Medicaid data I can’t get. I don’t know necessarily hospital admissions, if it’s Medicaid and that kind of thing. That is a gap in our system, and it is a hinderance to my board being able to make a decision about adding beds, adding services, that kind of thing. I have access only to what we paid for. That is a small piece when you look at Medicaid, and person third-party payers, and that kind of things.” MH-16.

Some jail respondents wished to have data on the number of missed appointments and reasons for missed appointments for substance use and mental health services. The need for a follow up data for individuals referred to mental health services after their release was also mentioned.

“It would just be nice if we could share more information, I would think, with our drug and alcohol, and our mental health. I know obviously, we come down into some type of violations, but obviously just knowing attendance for—even if we could just track the individuals that were on re-entry, just not everybody that’s gonna leave the jail, but it would be a good portion of the individuals.” J-2.

Participants said having follow up data would help them identify and respond to the needs of people who might “fall through the cracks” as noted by multiple participants.

Standardized electronic data management system to enable data sharing

 Some other jail respondents indicated the need for a uniform/standardized data management system. Currently, different agencies collect data in different formats, use different definitions, and measure ideas like recidivism, relapse, housing instability differently. These discrepancies make understanding and interpreting data from other agencies a difficult endeavor.

“Our behavioral healthcare system is set up via county. It’s not centralized at a state level. The state controls a lot of what happens to us on a local level, but we’re responsible to plan for what happens in our local community’s. It’s really hard to plan for a local community when you don’t have all your information. I can’t make a reasonable business decision about the expenditure of dollars if I’m not sure exactly what I’m spending them on. It’s sort of like shooting a fish with my eyes shut in a barrel five blocks down the street. I mean, that’s what it feels like sometimes….I make some presumptions; I use SAMHSA data to do prevalence rates about what I think is happening here based upon other federal information and other reporting. Again, the further away you get from it, the less reliable and valid that information is.” S-38.

Jail administrators indicated their challenges with respect to data sharing are due to the hand-written nature of their data which makes data sharing and data extraction challenging. Some respondents from jails and probation also said they do not collect data due to lack of data collection infrastructure. Mental health providers stated that they are only able to collect data from individuals involved in the public system (such as on Medicaid, Medicare) but not those that use other insurance. They said that they do not have access to data on individuals who are privately insured. This data is critical for them because most commercial funders do not cover some components of recovery services.

Agency perspectives on best practices and agency recommendations

Participants shared their perspectives on current data capacity needs and their recommendations. Table 3 summarizes agency reports of current data capacity needs, best practices and recommendations.

Table 3.

Agency-identified data capacity, best practices and recommendations

Current data capacity	Current best practices	Recommendations
• Rudimentary levels of data use	• Growing interest in tracking trends data	• More access to ‘big picture” data
• Interest in individual level health and criminal legal outcomes only	• Interagency data sharing agreements	• Funding for a dedicated data expertise
• Inadequate access to data collected by outside agencies	• Small (individual-level) efforts to monitor trends in data	• Assign inhouse data person
• Hand-written, inaccessible data- difficult to process/share	• Involving community partners to collect data	• Create a centralized, quality, accessible data dashboard
• Restrictive policies against data sharing	• Efforts to automate data
	• Use of data to make decisions about staffing and funding

Agencies used creative avenues to access data collected by others. For example, some mental health and substance use services agencies mentioned that they involve their community partners to collect data for them or work to maintain good relationships with other agencies to gain access to data collected by outside agencies.

Others relied on their relationships with external agencies to have a fuller picture of relevant data:

“…if we need to look at admission data, sometimes we’ll pull data that we can get from our managed care organization to look at the broader—picture because we’re only tracking those clients access, our services at the single county authority, which is a fraction of the whole. We do try and look at the data that’s available to us through managed care, in combination with our own data because it’s important to be able to see that broader picture.” SU-30.

County-level or state-level data dashboards

 Although agencies have specific data from the populations they serve, they often experience difficulties trying to access county-level or state-level data. To address this, participants suggested the need for a centralized dashboard where they could submit their data and could access what had been submitted by other agencies working with the same populations in their county. There was an overall optimism among respondents that a data dashboard accessible to the probation, the welfare system, the police department, and the mental health and substance use providers might improve access to and utilization of data.

Some participants proposed using a county-wide unique identifier where a person’s need and current services (interactions with agencies) can be tracked wherever they go. However, they also acknowledged the limitations that come with privacy laws including HIPAA. They proposed putting in place proper data sharing agreements with full understanding of how to make it happen while respecting the HIPAA laws. Some of these providers shared examples where they successfully worked around the privacy laws to share data between agencies.

Overall, across the agencies, there is a general consensus that there is a lack of a good quality, accessible and centralized data repository.

In-house data analyst

 The need for a dedicated in-house data person was also highlighted by some of the participants.

“We don’t have a person that’s full-time data guy or gal that could do that. We do try to keep track of all this information—I guess maybe software, it’d be nice if there was a computer program that would track these people a little bit better.….we have our local jail Zuercher records that keeps our data and our medications, and our type of information for our jail, but it doesn’t communicate with our local Department of Social Services. It doesn’t communicate—and then theirs doesn’t communicate back with us. Human Service Agency, their software, there’s issues with communicating back and forth. That would be a need, too.” J-38.

Discussion

In this study, we explored the use of data-driven decision-making practices among mental health, substance use services and criminal legal agencies (jail and probation). Most agencies described their data use as limited without agency-level metrics. Instead, the focus was on obtaining and using individual-level health and criminal legal data to describe the clients and to assist with the disposition of individual clients. Overall, participants reported rudimentary levels of data use, policy restrictions, inadequate access to data and poor data capturing.

Most of the existing evidence on the value of data-driven decision-making comes from quality improvement literature and the discipline of precision medicine. These reflect small initiatives that targeted using data to make clinical decisions for individuals with serious mental illness. Nevertheless, these efforts largely emphasize the use of technologies and big data to improve aggregate mental health outcomes [17–20]. Our participants expressed their interest in individual-level health and criminogenic prediction algorithms that would allow them to monitor intermediate-level outcomes like treatment participation and functioning. Some studies raised the issue of ethical concerns pertaining to automated decision-making using big data for mental health and criminogenic factors [21–23]. Collectively the body of literature promotes the use of data-driven decision-making without emphasizing the practical uses as benchmark of system progresses.

Given that a tremendous variation exists in what is considered to be data, a need exists to clarify and incentive the use of benchmark metrics. Most participants did not embrace the operationalized definition that was used to describe DDDM as an ongoing tracking of performance metrics of agencies. Participants tended to talk about sharing information about an individual client to triage services provided to the client. It was clear that the study participants did not understand the cascade of care metrics to assess system performance [24] but instead referred to individual clients. More attention is needed to help jurisdictions value system benchmarks or metrics through incentives that might be used to improve system performance. For example, if agencies were incentivized financially to transition clients from treatment in jail to the community, then it is likely that more agencies would develop and use performance metrics. Embedding performance metrics into contracts, reimbursement from Medicaid or other funding sources, and/or performance-based funding initiatives would place a value of system performance [25]. These approaches could move the needle regarding the use of the cascade of care or other system performance tools.

Another benefit of fiscal incentives for using system performance is that attention can be given to the system functioning instead of the progress of individual clients. The current data environment promotes silos in that each agency is focused on their own processes and procedures without attention to how their processes impact other agencies or the ability of clients to transition from one program or service to another [26]. For example, individual agencies might use different screening or assessment tools to diagnose SUD or MH needs. This contributes to each agency performing their own screening and/or assessment, as well as each agency deciding the eligibility for services/programs. A systems approach would streamline the screening and referral process by having multiple agencies use the same tools and determine program eligibility criteria. This would ease data sharing requirements as well as allow the system to promote interagency policies and practices [27].

Clarifying the performance metrics (with incentives to generate the metrics and report them) has the collateral benefit of assisting organizations to identify the variables that they need. As noted in our results, there was a lack of consensus on what data elements are relevant to collect, how a particular metric is operationalized, how data is retrieved and calculated [28]. Having a clear definition of what constitutes data element has the potential to facilitate implementation by having clearly defined metrics [29]. From the quality improvement literature, it is apparent that clarifying the agency level performance metrics improves access to and quality of mental health care [12, 30]. Previous studies reported the presence of a robust performance monitoring in the areas of provider-patient ratios, staffing vacancies, outpatient caseloads and patient-provider satisfaction levels has improved clinical performance and mental health care [12, 31]. Clarity would also promote the accessibility of key data elements and overcome the challenges to access data including restrictive privacy and Medicaid policies, incomplete data, lack of common definitions, and lack of a uniform data-capturing system.

Another policy incentive to facilitate data driven decision making is to have data dashboards built into e-health or electronic health records systems. These data dashboards can help to build the organizational data capacity for data expertise. These dashboards can clearly identify the metrics to assess the progress made both within and across agencies. Of course, as revealed in this study, the data dashboards should be accompanied by training on how to use the data to assess performance and improve system functionality including the use of quality improvement processes. Helping counties to implement such systems could also advance and create strong interagency collaborations with clearer goals regarding service provisions [32].

A major barrier to implementing system benchmarks, interagency data dashboards, and shared metrics such as the cascade of care is the challenge of data confidentiality. Regulations such as HIPAA and 42 CFR Part 2 restrict the sharing of individual-level information without client consent, limiting cross-agency data sharing and coordination. However, some jurisdictions have developed strategies to navigate these confidentiality regulations. For example, states such as Massachusetts, Indiana, and Alaska have established data warehouses that integrate data across agencies at the state level, allowing for analysis while maintaining compliance with privacy laws [26, 33]. Researchers and policy makers are usually required to go to a restricted site to use the data. Or the states prepare metrics to share with agencies and/or policy-makers to illustrate system functioning issues. These data warehouses are novel for advancing the use of DDDM by providing the systems with key benchmarks.

Findings from this qualitative work underscore the importance of counties’ investment in data infrastructure to improve service delivery and bolster system performance. There is a national need to improve mental health data infrastructure [34–36]. Agencies serving individuals in the criminal legal system could benefit from implementing data portals that allow longitudinal tracking of data [37]. Examining the trajectories, particularly the use of services, medications, and interventions, to affect health outcomes can assist with more knowledge about effective service delivery systems.

Finally, a policy initiative focused on improving the data capacity of jurisdictions is needed. Most participants indicated that they did not have the expertise to collect, manage, analyze and report data due to the available staff capabilities. More emphasis should be placed on providing incentives to improve data staff. For example, in the current Medicaid 1115 waivers for reentry, states are required to submit data and reports to Center for Medicaid Services (CMS) on a regular basis to track key metrics [38]. A template is provided to assist states in understanding what data to collect and the metrics to assess the impact of the waiver. CMS also provides specialized fundings to improve electronic health record and management information systems to monitor the use of the 1115 waivers.

Strengths and limitations

This study has several strengths including a large, nationally representative sample size of over 519 counties in a mixed method study involving interviews and quantitative surveys. This study used a diverse pool of participants and a rigorous design. The study does have some weaknesses worth noting. All of interviews were conducted virtually. Therefore, we were unable to capture some non-verbal cues. We also did not conduct member checking due to the structure of the study, particularly the multi-site nature of the data collection, the number of agencies involved, and the timeline of the broader mixed methods evaluation. To mitigate this, we used a rigorous analytic process that included independent coding by multiple team members, consensus-building discussions, and regular summaries to ensure consistency and transparency in our interpretation of the data.

Conclusions

This study explored how data-driven decision making (DDDM) is used as an implementation strategy in mental health, substance use and criminal legal settings. While DDDM is promoted to improve outcomes by helping agencies identify and track key metrics, few counties in our sample had the capacity to do so. Challenges included limited understanding of metrics, lack of data infrastructure, and insufficient staffing. Despite the Stepping Up Initiative providing a clear set of indicators, many counties were not effectively using data systems. Findings point to the need for technical assistance—such as staff training, standardized data tools, centralized dashboards, and hiring or developing in-house data expertise. In addition to these supports, policy and funding incentives are needed to shift the focus from tracking individual clients to evaluating system-level performance across screening, referral, treatment, and outcomes. Coordinating between criminal legal and health systems is complex due to competing goals. Using benchmarks like the cascade of care can help track whether these systems are improving outcomes for individuals with mental health needs.

Supplementary Information

Below is the link to the electronic supplementary material.

Abbreviations

DDDM Data

Driven decision-making

HIPAA

Health insurance portability and accountability act

CFR 42

Code of federal regulations 42

I M Justice BH

Implementation mechanisms for justice and behavioral health

Author contributions

MH conceived the idea. RKR, JEJ and MH trained the interviewers and coders. MH, HE, BM conducted the interviews. MH, JW, JEJ, RKR and BM conducted the qualitative analysis. MH drafted the manuscript. FST, RS, GB and KJC contributed to different parts of the manuscript. All authors reviewed and approved the manuscript.

Funding

The I.M. Stepping Up study was supported by the National Institute of Mental Health (R01 MH118680, MPI Taxman and Johnson). The contents of this publication are solely the responsibility of the authors and do not necessarily represent the official views of NIMH.

Data availability

No datasets were generated or analysed during the current study.

Declarations

Ethics approval and consent to participate

The I.M. Stepping Up Study was approved by Michigan State University Biomedical IRB. The need for informed consent was exempted by Michigan State University Biomedical IRB. All study staff also completed the Michigan State University Human Subject trainings. The study was conducted in accordance with the ethical principles outlined in the Declaration of Helsinki.

Consent for publication

Not applicable

Competing interests

The authors declare no competing interests.