What’s Risk Got to Do with it: Judges’ and Probation Officers’ Understanding and Use of Juvenile Risk Assessments in Making Residential Placement Decisions

Abstract

Objectives:

This hypothetical-vignette-based experiment was designed to better understand judges’ and probation officers’ interpretations and use of juvenile risk assessment tools in their decision-making around restrictive sanctions and confinement of youths on the basis of the youths’ risk level and race.

Hypotheses:

We expected that estimates of the probability of juvenile recidivism would significantly mediate the relationship between a categorical risk descriptor and decisions regarding ordering confinement of youths. We also hypothesized that youths’ race would serve as a significant moderator in the model.

Method:

Judicial and probation staff (N = 309) read a two-part vignette about a youth who was arrested for the first time; in this vignette, race (Black, White) and risk level (low, moderate, high, very high) of the youth was varied. Participants were asked to estimate the likelihood that the youth would recidivate in the following year and their likelihood of ordering or recommending residential placement.

Results:

Although we found no simple, significant relationship between risk level and confinement decisions, judicial and probation staff estimated higher likelihoods of recidivism as risk-level categories increased and ordered out-of-home placements at increased rates as their estimations of the youth’s likelihood of recidivation increased. The youth’s race did not moderate the model.

Conclusion:

The greater the probability of recidivism, the more likely each judge or probation officer was to order or recommend out-of-home placement. However, importantly, legal decision-makers appeared to apply categorical risk assessment data to their confinement decisions using their own interpretations of risk category rather than being guided empirically on the basis of risk-level categories.

More than 40,000 youths are confined in out-of-home justice facilities, including detention centers, secure long-term placements, and adult jails and prisons, across the United States (Hockenberry & Sladky, 2018). Although this number represents a significant decrease from the year 2000, two thirds of confined youths are held in the most restrictive settings (e.g., locked facilities, those that use physical restraints such as hand or leg cuffs) for longer than 1 month, and approximately one quarter of all confined youths are held for more than 6 consecutive months (Sawyer, 2019). Legally involved youths can be placed in confinement through multiple mechanisms; most commonly, they are held prior to an adjudicatory or a disposition hearing (i.e., detention), placed in a secure or nonsecure postadjudication facility following disposition, or placed in detention or a placement facility after violating the conditions of probation or being charged with a new offense (Sedlak & Bruce, 2017). Juvenile or family court judges often preside over detention, adjudicatory, dispositional, and review hearings of youths involved in the juvenile legal system. Judges have the discretionary power to make decisions about the confinement of youths at different time points across case processing based, in part, on information, reports, or recommendations made by probation officers at the time of the hearing, including information gleaned from risk assessment measures (Sarri et al., 2001; Shook & Sarri, 2007).

Risk assessments tools evaluate a youth’s potential for future offending or violence and are used widely at various points in case processing, including at intake, at disposition, and throughout probation (Garrett & Monahan, 2020). Probation officers are typically responsible for administering such risk assessments and providing reports and recommendations to judges during youths’ hearings. Although these tools provide information to justice system personnel (e.g., judges, probation officers) about a youth’s risk for recidivism and need for rehabilitation, it is unclear how judges—particularly juvenile court judges—and probation staff use risk assessment information in their decision-making and recommendations. Therefore, the present study examined judicial understanding (or misunderstanding) of risk assessment information and how judges’ and probation officers’ interpretations of risk information impacts their decisions to place or recommend placing youths in out-of-home facilities.

Juvenile Risk Assessments

The use of risk assessment tools in the juvenile legal system has grown significantly, from 30% of jurisdictions using these measures in 1990 to more than 86% in 2003 (National Center for Juvenile Justice, 2006); as of 2017, 38 states had adopted a statewide, uniform risk assessment measure, whereas the other 12 states reported using risk assessment tools that differed across regions or jurisdictions within the state. States and jurisdictions use their identified uniform risk assessment tools systemwide (e.g., across probation divisions and within juvenile court) to assist with dispositional and probation-related decisions (DeMatteo et al., 2016; Juvenile Justice Geography, Policy, Practice & Statistics, 2017; Vincent, Guy, & Grisso, 2012). Importantly, the instruments used vary in how they estimate risk (e.g., actuarial, structured professional judgment; Hester, 2020). Risk assessment measures also vary on the outcome they were designed to appraise. Although most manuals specify that the tool appraises risk of recidivism, individual assessments, validation studies, and research using these measures operationalize recidivism in different ways, predicting such varied outcomes as self-reported reoffending behavior, rearrest, or reconviction (Schwalbe, 2007). Furthermore, the recidivistic behaviors identified and predicted by risk assessments vary; some risk assessments appraise specific types of recidivism, such as violent reoffending (e.g., Structured Assessment of Violence Risk in Youth [SAVRY]; Borum et al., 2006) or sexual reoffending (e.g., Juvenile Sex Offender Assessment Protocol II [J-SOAP II]; Prentky & Righthand, 2003), whereas others target general reoffending (e.g., Ohio Youth Assessment System [OYAS]; Lovins & Latessa, 2013; Youth Level of Service/Case Management Inventory [YLS/CMI]; Hoge & Andrews, 2011). It is therefore important to recognize that the outcome of a risk assessment (e.g., level of risk) may vary depending on the tool administered. This is further emphasized by recent research examining the validity and utility of risk assessment tools for youths of color.

Reliability and Validity of Juvenile Risk Assessments Across Races

The reliability and validity of the many juvenile risk assessment measures differ substantially (Baird et al., 2013; Schwalbe, 2007). Several are used widely across jurisdictions and initially showed empirical support in their appraisal of recidivism, including the YLS/CMI 2.0 (Hoge & Andrews, 2011), SAVRY (Borum et al., 2006), and OYAS (Latessa et al., 2009; Lovins & Latessa, 2013). However, more recently, scholars and policy makers have questioned the validity of these risk assessments when used with youths of color; research has found that many risk assessments invalidly appraise risk for youths of color, overestimating their probability of recidivism (e.g., Campbell et al., 2020; Onifade et al., 2009; Viljoen et al., 2019). The various ways in which risk assessment measures operationalize recidivism (e.g., rearrest vs. self-reported reoffending) can exacerbate these concerns of validity. Given that youths of color, particularly Black youths, are at greater risk of arrest compared with White youths, in part because of heightened police presence in communities of color (Crutchfield et al., 2012; Fagan et al., 2016), risk assessment tools with rearrest as the outcome may not accurately predict the levels of reoffending for Black youths. Further, researchers and policy makers have expressed concerns about the role of risk assessment instruments in exacerbating racial and ethnic disparities by considering static (e.g., arrest history) and dynamic (e.g., family involvement) risk factors that may lead to higher scores for Black youths than White youths and therefore contribute to harsher sanctions for youths of color (e.g., Mallett & Stoddard-Dare, 2010; Maurutto & Hannah-Moffat, 2007; Skeem & Lowenkamp, 2016). But other advocates and researchers have advocated for the use of established risk assessment tools to reduce racial bias (Graham & Lowery, 2004; Thompson, 2017), as some literature has demonstrated significant differences in risk appraisals across races (McCafferty, 2018; Olver et al., 2014). Given the nuanced findings related to race, researchers have called for further investigation of risk assessment and race, particularly on how the use of risk information may create or protect against inequitable decisions from juvenile legal personnel (Skeem & Lowenkamp, 2016; Viljoen et al, 2019).

Heuristics and Decision-Making

Little published research has examined the extent to which judges and probation officers accurately appraise communicated risk information and are aware of the variability in outcomes, validity, and potential bias across risk assessment instruments (Garrett & Monahan, 2020). Additionally, it is unclear whether judges use the categorical risk information (i.e., low, medium, high risk) typically provided by such tools as intended or as a problematic heuristic when making decisions, such as the need for out-of-home placement. Heuristics are shortcuts that allow a decision-maker to evaluate and determine a course of action without substantial consideration of all available information or alternatives (Shah & Oppenheimer, 2008; Simon, 1990). Given their large caseloads of youths (Butts et al., 2009), judges in juvenile court may rely on their heuristics about risk information (Hester, 2020; Rachlinski & Wistrich, 2017). But judges may fail to consider the complexities of the risk assessment data, validity of particular risk assessment tools, and applicability of specific measures to particular populations. Relatedly, social science research has demonstrated that perceptions of risk are often biased (e.g., Finkel, 2008; Gigerenzer, 2006). Judges (Viscusi, 1999), probation officers (Perrault et al., 2012), and jurors (Krauss et al., 2018) have all been shown to overestimate risk of negative outcomes or recidivism in various contexts. Taken together, a lack of clear understanding on risk assessment data may result in an inaccurate understanding of the risk information provided (Garrett & Monahan, 2020)—and, possibly, misinformed release and confinement decisions.

Impact of Judicial Decision-Making

Juvenile and family court judges make decisions at various points in case processing, at least in part on the basis of reports made by probation staff, that have potentially far-reaching consequences for youths. Though risk assessment information is provided to assist with case-processing decisions, judges and probation officers can override the risk level generated by the structured instruments and make decisions on the basis of their subjective risk impressions and alternative information, ignoring the results of the measures and ridding the decision-making process of structured risk-relevant information (Garrett & Monahan, 2020; Papp, 2019; Shook & Sarri, 2007). These overrides often result in youths receiving harsher and more restrictive sanctions (e.g., electronic monitoring) or out-of-home placements (Papp, 2019).

Restrictive sanctions are associated with negative outcomes for youths (for reviews, see Mendel, 2011, and Weisburd, 2015), including higher probability of future offending (e.g., Gatti et al., 2009), low self-efficacy and mental health concerns (e.g., Cuevas et al., 2017), loss of educational progress and interruption or termination of schooling (e.g., Keeley, 2006), sexual victimization (e.g., Heaton et al., 2012), and increased costs to families (e.g., Campos-Bui et al., 2017). Though these negative consequences are well understood by researchers, more than 10,000 youths were held in long-term secure facilities on any given day in 2019, including approximately 1,300 youths ordered to residential placement because of technical violations of probation, status offenses, or drug use (Sawyer, 2019).

The Present Study

Given that judges have discretionary power to make decisions about a youth’s disposition in or out of the community on the basis, at least in part, of risk assessment information provided in recommendations made by juvenile probation officers, it is important to understand judges’ and probation staff’s interpretations and use of juvenile risk assessment instruments. Specifically, the present study assessed (a) the extent to which judges’ and probation officers’ judgments and estimates of risk probability align with categorical risk information (e.g., low, moderate, high, very high) described by a risk assessment tool and (b) how these estimates influence their decision-making.

We predicted that estimates of the probability of juvenile recidivism would significantly mediate the relationship between a categorical risk descriptor (low, moderate, high, very high) and decisions regarding ordering confinement of youths. We hypothesized that each of the pathways within the model would also be significant: (a) Judges and probation officers would be significantly more likely to order restrictive sanctions, including placement, for a youth with a higher risk level than for a youth with a lower risk level; (b) decision-makers would estimate the risk of recidivism as significantly higher for very-high-risk than for high-risk youths and significantly higher for high-risk than for medium- and low-risk youths; and (c) estimated probabilities of recidivism would significantly affect decisions about sanctions, such that the higher judges’ and probation officers’ estimates of recidivism (controlling for risk category), the more likely they will be to order or recommend confinement. We also predicted that youths’ race would serve as a significant moderator in the model. Specifically, we expected that both the relationship between categorical risk level and high estimates of recidivism probability and between categorical risk level and decision-makers’ likelihood of ordering confinement would be significantly stronger for White youths than Black youths, as decision-makers may be more affected by implicit or explicit biases when faced with a decision about a Black youth (Wistrich & Rachlinski, 2017).

Method

Participants

Participants were 309 judges, magistrates, juvenile court officers, or juvenile probation officers in the United States who had heard or worked on at least 20 juvenile delinquency cases in their tenure. Participants primarily identified as male (51.1%) or female (48.2%); 0.7% preferred not to self-describe their gender. Participants were mostly White/Caucasian (83.3%) and non-Hispanic (95.0%); participants also identified as Black (8.2%), Indian/Alaska Native (1.8%), Asian (1.4%), Hawaiian/Pacific Islander (0.3%%), multiracial (0.7%), and “other” (4.3%). Participants ranged in age from 23 to 68 years old (M = 45, SD = 9.94). Most participants identified as probation officers (79.2%); judges and magistrates represented 15.2% of the participants. On average, participants have served in their position for 13 years (SD = 8.80). Participants classified the jurisdictions in which they work as urban (37.6%), rural (35.5%), and suburban (26.9%) and were from 26 states, most often Ohio (36.2%), Pennsylvania (31.7%), and Utah (8.3%). Participants did not differ across conditions (i.e., risk level and race of the youth in the vignette reviewed) in terms of demographics (i.e., participant race, gender, job type). The samples across randomized vignette conditions did not differ significantly by geographic location, χ²(75, N = 290) = 63.99, p = .814, V = 0.47.

Design

This study employed a 4 (risk level: low vs. medium vs. high vs. very high) × 2 (race: Black vs. White) between-subjects design.

Measures and Procedure

The research team recruited participants via email on professional LISTSERVs, including a national LISTSERV of court- and legal-system professionals, between December 2020 and June 2021. We randomly assigned participants to vignette groups via the Qualtrics online survey platform (https://www.qualtrics.com/) and asked them to read a two-part vignette describing a 15-year-old male appearing in juvenile court. We then asked them to complete a series of surveys, created specifically for this study, regarding the participant’s beliefs about the youth’s recidivism risk, conclusions about sanctions that should be ordered, knowledge about risk assessment tools used in the participant’s jurisdiction, and demographic characteristics.

We included attention checks to identify participants whose data should be excluded; specifically, we asked about the gender, age, race, and risk level of the youth. Two hundred twenty-one participants were excluded because they failed to successfully answer at least one attention check question related to the youth’s race (n = 132), the youth’s age (n = 115), the youth’s gender (n = 70), or the youth’s risk level (n = 123). Group-level analyses revealed that significantly more participants who read the vignette about a Black youth failed the attention check question about race than did those who read about a White youth, χ²(2, N = 533) = 58.44, p < .001, V = 0.33. Additionally, significantly more participants who read a vignette about a youth with a very-high-risk level failed or skipped the attention check question about risk compared with those who read vignettes about youths of other risk levels, χ²(4, N = 533) = 106.03, p < .001, V = .45.

Participants did not receive compensation for participation in the study because of varying state and jurisdictional rules on receiving monetary compensation for research as a jurisdictional employee. The study was conducted in accordance with the guidelines of Drexel University Institutional Review Board (Protocol #2009008074A001).

Vignette

There were eight versions of the vignette (see the online supplemental materials). The research team modeled the structure and content of the vignettes after those used in previous studies on juvenile legal system practitioners’ opinions of youths (e.g., Applegate et al., 2000; Vidal & Skeem, 2007). The eight vignettes were identical except for variation in the race and risk level of the youth specified. Each vignette provided the name (Michael) and demographic information of the youth, including his age (15) and race (Black or White). Additionally, it described information related to his case, including his charge (simple assault), arrest history (first arrest), and risk-level category (low, moderate, high, or very high). The second part of the vignette presented a brief biopsychosocial description (e.g., family life, education, behavioral concerns, mental health history) and indicated that Michael violates his probation by screening positive for marijuana use. Biopsychosocial details reflected common characteristics of adjudicated youths in placement (Sedlak & Bruce, 2010). We selected the specific charge and the type of probation violation to establish case details that would result in substantial discretion in judges’ disposition decisions, generating the possibility of a wide range of potential responses from judges across the country (Office of Juvenile Justice and Delinquency Prevention, 2020). Risk categories were taken from the YLS/CMI, an empirically validated assessment measure commonly used throughout the country at intake and review hearings (e.g., Urquhart & Viljoen, 2014). The YLS/CMI’s predictive validity for general reoffending (defined as a rearrest within 1 year) aligns with other, comparable risk assessment tools (e.g., OYAS, SAVRY; Schwalbe, 2007).

Impressions of Recidivism Risk Survey

After reading Part 1 of the vignette describing the youth at his disposition hearing, participants completed several questions regarding the likelihood that the youth would recidivate as well as the likelihood that the youth would violate probation. This survey included questions asking participants to rate the likelihood that the youth will commit another crime and, separately, violate probation on a 5-point scale (from 1, Not at all likely, to 5, Extremely likely), as well as to estimate the probability of these two outcomes as a percentage (i.e., “What is the probability that Michael will commit another crime?” “What is the probability that Michael will violate his probation?”). Questions on related topics (e.g., probability of committing another crime as a percentage and likelihood of committing another crime as a Likert response) were highly correlated: Questions relating to the youth committing another crime in the following year and those relating to the youth violating probation were positively and highly correlated (r = .73 and r = .78, respectively).

Sanction Recommendation Survey

After reading Part 2 of the vignette, judicial participants estimated how likely they would be to revoke probation and place the youth in a residential justice-related facility (e.g., secure placement, detention); they also indicated what specific sanction (e.g., warning, intervention, GPS monitoring, placement) they would provide if this youth were in their courtroom. Probation officer participants answered questions related to how likely they would be to recommend probation revocation and placement as well as the specific sanctions they would recommend to the judge at the youth’s next hearing. Questions targeting recommending revoking probation and placing the youth using a percentage estimate and Likert response were highly correlated (r = .80 and r = .79, respectively).

Jurisdictional Risk Assessment Survey

Participants then answered questions regarding the use of risk assessment measures in their jurisdiction, how often information provided from risk assessment measures is presented to them in court, and their impressions of risk assessments’ accuracy (from 1, not at all accurate, to 5, extremely accurate). Participants were also asked to estimate how likely a generic youth (i.e., not the youth in the vignette provided) would be to commit another crime and, separately, how likely a generic youth would be to commit a violent crime, given a risk-level category (i.e., low, moderate, high, very high risk). Cronbach’s α for the construct related to these recidivism estimates indicated that the internal consistency was high (α = .89).

Demographics Survey

Participants answered a series of demographic questions so we could gather information about their age, gender, race/ethnicity, job title, length of time serving in their role, and state and jurisdiction of employment. We did not collect any more directly identifying information.

Method of Analysis

To examine the primary aims of this study, we conducted a moderated mediation analysis using the PROCESS macro (Hayes, 2022) for SPSS that employs Hayes’s (2022) bootstrapping method to examine the indirect pathways from youths’ risk level to judicial decision about confinement through juvenile legal system staff’s estimates of recidivism risk, with youths’ race as a moderating variable. We examined the specific pathways of interest (from youths’ risk level to judicial decision, youths’ risk level to estimates of recidivism risk, and estimates of recidivism risk to judicial decision on confinement), as well as the moderating effect of race on the relationships between youths’ risk level and judicial decision and between youths’ risk level and estimates of recidivism risk using the regression coefficients provided for each pathway. Because there were no significant differences among the groups’ demographic information and because we wanted to simplify our model, we did not include any covariates in the model but instead looked for descriptive differences among groups on outcome variables. We also conducted a χ² test of independence to examine the differences between risk-level categories on recommended categorical sanction type. Finally, we conducted descriptive analyses to describe the overall sample of participants; χ² tests of independence were conducted to identify differences between the groups of participants on the basis of the vignette read.

Power Analysis

For the primary mediation strategy, we conducted an a priori power analysis using MedPower (Kenney, 2017), a mediation-specific power computation calculator. Three hundred eighty-six participants were required to produce a power of .80 to detect a small effect size (β=0.14) with an α of .05 in the direct (c′) and indirect (a, b) pathways. Given that bootstrapping is appropriate for use with sample sizes smaller than 386, we deemed the number of recruited participants sufficient to detect the effect of the moderated mediation (Preacher et al., 2007; Rucker et al., 2011; Schoemann et al., 2017).

Results

According to participants, the most common risk assessment tools used in their jurisdictions were the OYAS (36.7%) and the YLS/CMI 2.0 (31.8%). Participants also reported using the Positive Achievement Change Tool (PACT; Baglivio, 2017; 5.9%), the Youth Assessment and Screening Instrument (YASI; Jones et al., 2016; 3.8%), the Protective and Risk Assessment/Pre-Screen Risk Assessment (PRA/PRSA; Utah Commission on Criminal & Juvenile Justice, n.d.; 3.8%), another risk assessment measure (9.3%), or a combination of multiple risk assessment tools (6.9%). Thirty-three participants (12.5%) noted that reports from risk assessment tools used in their jurisdiction specified the percentage likelihood that a youth would recidivate; among this group of participants, more than half (54.5%, n = 18) identified working in states that use risk assessment tools for which data on percentage likelihood of recidivation are not available (e.g., Pennsylvania, Ohio). Because most risk assessment tools being used in participants’ jurisdictions include only three risk categories (i.e., low, moderate, high) to describe youths, we collapsed into a single group for experimental analyses those participants who read vignettes describing a youth scoring in the high- or very-high-risk category. For analyses examining participants’ reactions to a generic youth (i.e., not described in a vignette and given only the risk-level category), the high- and very-high-risk categories were kept distinct.

Estimates of a Youth’s Probability of Recidivism and Violation of Probation

Collapsing across vignettes—from low risk to high/very-high risk—we found that participants estimated that the youth in the vignette had a 38.2% (SD = 22.60) likelihood of committing another crime in the following year. Participants’ estimates of the youth’s likelihood of committing another crime in the following year differed significantly with the youth’s identified risk level, F(3, 296) = 71.72, p < .001, η² = .04, 95% confidence interval (CI) [.03, .09], with participants rating the probability of reoffending as higher at each higher risk level. They estimated that youths identified as low risk had a 17.0% (SD = 11.69) probability of recidivating, youths identified as moderate risk had a 37.3% (SD = 17.64) chance, youths identified as high risk had a 51.3% (SD = 19.96) chance, and youths identified as very high risk had a 52.9% (SD = 19.81) chance. See Figure 1 for ranges of responses. Participants estimated a higher probability of recidivism for the White youth (M = 41.14%, SD = 23.41) than the Black youth (M = 35.81%, SD = 21.68) across risk levels, F(1, 298) = 4.17, p = .042, η² = .014, 95% CI [.00, .05]. Similarly, participants’ perceptions of the likelihood that the youth would violate probation differed by the youth’s identified risk level, F(2, 296) = 78.33, p < .001, η² = .35, 95% CI [.26, .42]. See Table 1. Participants estimated a higher probability of a probation violation for the White youth than the Black youth in the vignettes across risk-level categories, F(1, 297) = 6.77, p = .01, η² = .02, 95% CI [.00, .07].

Figure 1.

Estimations of Recidivism Probabilities by Youths’ Assigned Risk Level.

Table 1.

Descriptive Statistics for Estimates of Vignette Youth’s Probability of Recidivism and Probation Violation

	Recidivism				Probation violation
	Probability (%)		Scalea		Probability (%)		Scalea
Youth’s risk level	M	SD	M	SD	M	SD	M	SD
Low	17.05	11.69	1.70	0.67	17.30	14.47	1.75	0.85
Moderate	37.30	17.63	3.08	0.90	36.36	18.66	2.90	0.88
High	51.27	19.97	3.56	0.87	49.23	18.60	3.43	0.84
Very high	54.94	19.82	3.69	0.81	48.56	19.87	3.22	0.89
High/very high collapsed	52.69	19.91	3.61	0.85	48.97	19.04	3.35	0.81
Total	38.21	22.60	2.93	1.13	36.61	21.93	2.78	1.08

^aThe scale ran from 1, Very Unlikely, to 5, Very Likely.

Participants were asked to estimate the likelihood of general and violent recidivism for a generic youth (i.e., separate from their vignette assignments) who scored at the low, moderate, high, and very-high levels on a risk assessment tool. As risk level increased, estimates of recidivism increased. Estimated probabilities of violent recidivism (low: M = 11.16%, SD = 11.11; moderate: M = 24.53%, SD = 15.05; high: M = 43.29%, SD = 20.20; very high: M = 56.54%, SD = 23.50) were lower than estimated probabilities of general recidivism (low: M = 18.07%, SD = 12.07; moderate: M = 37.30%, SD = 13.74; high: M = 60.52%, SD = 18.87; very high: M = 73.13%, SD = 19.92) across risk levels. Probation staff estimated significantly higher recidivism probabilities than judges did for moderate-risk, t(447) = 2.63, p = .004, and high-risk, t(447) = 1.89, p = .03, youths. Probation staff also estimated a significantly higher probability of violent recidivism than did judges at each risk level (see Table 2 for statistics).

Table 2.

Descriptive Statistics for Estimates of a Generic Youth’s Likelihood of General and Violent Recidivism

Youth’s risk level	Overall		Judges/magistrates		Probation staff/other
	M	SD	M	SD	M	SD	t	p
General recidivism (%)
Low	18.07	12.07	16.13	9.80	18.44	12.51	−1.44	.075
Moderate	37.30	13.74	33.26	12.26	37.98	13.96	−2.63	.004
High	60.52	18.87	56.45	18.51	61.14	18.98	−1.89	.030
Very high	73.13	19.92	69.54	21.65	73.80	19.64	−1.61	.054
Violent recidivism (%)
Low	11.16	11.11	8.06	7.35	11.73	11.68	−2.44	.008
Moderate	24.53	15.05	18.75	12.30	25.66	15.39	−3.50	< .001
High	43.29	20.20	37.83	19.64	44.15	20.34	−2.38	.009
Very high	56.54	23.50	49.88	25.52	57.60	23.04	−2.48	.007

Note. Significant p values (i.e., > .05) are given in boldface.

Likelihood of Probation Revocation and Placement

Across conditions, participants indicated that they would be unlikely to revoke or recommend revocation of the youth’s probation following a positive drug screening for marijuana (M = 1.32, SD = 0.74) and estimated a 10.5% chance that they would revoke the youth’s probation (SD = 17.40). Participants indicated that, instead, they would provide an alternative sanction, most commonly in the form of a substance use evaluation/treatment (20.5% of participants), a warning (11.9%), additional drug testing (5.4%), a mental health assessment/treatment (4.5%), or use of a structured problem-solving tool (e.g., Carey Guides/Brief Intervention ToolS (BITS; Carey Group, 2016; 2.9%)—and, most often, a combination of two or more of these sanctions (58.0%), which typically included either a substance use or a mental health evaluation/treatment. Recommended sanction type did not differ as a function of the youth’s risk level, χ²(12, N = 312) = 16.37, p = .175, V = .18, or the youth’s race, χ²(6, N = 312) = 7.90, p = .246, V = .25.

Primary Aims: Effects of Risk Level on Participants’ Impressions of Recidivism and Likelihood for Placement Decisions

Results revealed no significant relationship in the direct path (c′) between a youth’s assigned risk level and participants’ likelihood of revoking probation and placing the youth outside the home, b = −1.53, 95% CI [−5.34, 2.29], SE = 1.94, p = .43. However, participants’ estimates of a youth’s likelihood of recidivation significantly mediated the relationship between the risk level assigned to the youth and their likelihood of revoking probation and placing the youth outside the home, b = 3.50, SE = 1.11, 95% CI [1.46, 5.74] p = .110, R² = .04. Risk level significantly predicted estimated probabilities of recidivism, b = 18.52, 95% CI [15.74, 21.30], SE = 1.24, p < .001, R² = .45, with participants estimating higher likelihoods of recidivism with each higher risk level. Additionally, the estimated-probability-of-recidivism mediator was significantly and positively related to the likelihood of probation revocation and placement, b = 0.19, 95% CI [0.05, 0.33], SE = 0.07, p = .0073, R² = .04, indicating that participants’ likelihood of ordering out-of-home placement increased as their estimations of the likelihood that the youth would recidivate increased. See Table 3 and Figures 1 and 2.

Table 3.

Mediation Model: Direct and Indirect Effects of Vignette Risk Category on Sanction Decision Through Estimates of Probability of Recidivism for a Hypothetical Youth

Path	b	SE	95% CI	p
Risk category to sanction decision	−1.53	1.94	[–5.34, 2.29]	.43
Risk category to recidivism estimate	18.52	1.41	[15.74, 21.30]	< .0001
Recidivism estimate to sanction decision	0.19	0.70	[0.05, 0.33]	.007
Total effect	−1.53	1.94	[–5.34, 2.28]
Indirect effect	3.50	1.11	[1.46, 5.74]

Note. CI = confidence interval.

Figure 2.

Mediation Pathway Diagram.

Moderated mediation analyses revealed that the race of the youth did not significantly moderate the mediation model, b = −0.49, 95% CI [−1.74, 0.47], SE = 0.55, p = .47, R² = .002. The youth’s race significantly moderated the strength of the mediation between the youth’s assigned risk level and likelihood of revocation and placement through the participants’ estimates of recidivism for White, b = 3.98, 95% CI [1.69, 6.92], SE = 1.32, and Black, b = 3.50, 95% CI [1.54, 5.80], SE = 1.11, youths. However, there were no significant differences between the moderating effects of White and Black youths, suggesting that participants did not rate the probability of recidivation for White and Black youths differently. Taken together, although the strength of the indirect effects was not significantly different for White and Black youths, the model in which participants’ estimates of recidivism probability significantly explained their placement decisions on the basis of the youth’s risk level held true for both White and Black youths. See Table 4 and Figure 3.

Table 4.

Moderated Mediation Model: Direct and Indirect Effects of Youth Race Moderating the Mediation of Vignette Risk Category on Sanction Decision Through Estimates of Probability of Recidivism for a Hypothetical Youth

Path	b	SE	95% CI	p
Risk category to sanction decision	−5.08	5.12	[–15.17, 5.00]	.32
Risk category to recidivism estimate	22.49	4.71	[–23.14, 20.22]	< .0001
Recidivism estimate to sanction decision	0.20	0.07	[0.06, 0.33]	.006
Youth race to sanction decision	−3.11	6.89	[–16.69, 10.47]
Youth race to recidivism estimate	0.44	6.67	[–12.72, 13.59]
Risk Category × Youth Race to sanction decision	2.11	2.94	[–3.67, 7.90]	.47
Risk Category × Youth Race to recidivism estimate	−2.45	2.84	[–8.04, 3.15]	.3896
Youth race
White	−2.97	2.65	[–8.20, 2.26]
Black	−0.86	2.26	[–5.32, 3.60]
Youth race: conditional indirect effect
White	3.98	1.32	[1.69, 6.92]
Black	3.50	1.11	[1.54, 5.80]

Note. CI = confidence interval.

Figure 3.

Moderated Mediation Pathway Diagram.

Discussion

Overall, this study suggests that although judicial staff and probation officers report using information provided by risk assessment tools when making case-planning decisions about legally involved youths, risk category alone did not guide placement decisions in predictable ways. More important, instead, were legal decision-makers’ interpretations of risk category, with estimates of reoffending ranging from 0% to 50% for low-risk youths to 0% to 90% for high-risk youths, in making confinement decisions. The relationships between risk category and recidivism estimate and between recidivism estimate and confinement decision followed predictable patterns: The higher the risk level, the greater the probability of recidivism and the more likely each judge or probation officer was to order or recommend out-of-home placement. However, given the wide range of interpretations based on few categories of risk and the limited information provided, our results suggest that risk assessment outcomes often act as a heuristic for legal decision-makers’ understanding of future delinquent behavior that is distinctive to and inconsistent across individual judicial and probation staff.

Importantly, the youth’s race also played a role in how decision-makers interpreted risk levels. This study suggests that decision-makers take a youth’s race into consideration when interpreting risk-level information and that the direct relationship between risk level and interpretations of risk exists for both Black and White youths.

Effects of Risk-Level Interpretations on Outcomes for Youths

Judges and probation officers’ wide range of interpretations of youths’ likelihoods for recidivism, and the decisions they make on the basis of these misunderstandings, have immense power to affect the outcomes of justice-involved youths. It is challenging to accurately appraise recidivism rates for youths at various risk levels because of how recidivism is operationalized (e.g., self-reported reoffending, rearrest, and reincarceration), which can be impacted by a myriad of factors (e.g., community policing, jurisdictional reporting). However, some research examining youth rearrests has reported that the average rates range from 16% to 40% within the 2 to 3 years following adjudication of youths with no prior legal system involvement; youths who have previously been incarcerated have higher rearrest rates, up to 80% following their release from incarceration (Mendel, 2011). Importantly, though, these arrest rates are for general offending and can include relatively minor offenses, such as marijuana possession, underage drinking, and truancy. Limited published research exists on rearrest specifically for violent offenses, which are more concerning to legal decision-makers. However, since 2005, violent crimes committed by youths have steadily decreased and now represent less than one third of all youth arrests (Hockenberry & Puzzanchera, 2021). Although the specific probability of general or violent reoffending is not available for youths in specific risk categories, these categories do appear to be related to decision-makers’ beliefs about a youth’s probability of reoffending. Thus, these influential perceptions—and the subsequent case-processing decisions—are based on idiosyncratic beliefs about probabilities that vary widely from decision-maker to decision-maker and many times appear to substantially overestimate chances of recidivism, particularly of violent recidivism.

Previous literature on risk appraisals with judges, probation officers, and jurors has indicated that people have perceptual biases that lead them to overestimate the risk of low-probability events (e.g., Krauss et al., 2018; Perrault et al., 2012; Viscusi, 1999), though there has been a lack of research directly examining these perceptions in juvenile legal settings. Although this study provides evidence of wide ranges of estimates of reoffending and violence, including some underestimation of risk, it is notable that it adds to the body of research suggesting that risk is overestimated in other areas of criminal decision-making, as we found that a number of decision-makers overestimated recidivism risk, even for low-likelihood events (i.e., future violent crimes). Furthermore, the various estimates seem to significantly affect decisions regarding sanctions and out-of-home placement, which have substantial implications for youths’ short- and long-term life outcomes. For example, although legal decision-makers were less likely to order or recommend placement of low-risk youths than higher-risk youths, some judges and probation staff believed that even low-risk youths had up to a 50% likelihood of recidivism, which could lead them to decide that these youths should be in an out-of-home placement. For moderate-risk youths, these reoffending probability estimates ranged from 5% to 85%—even if inaccurate, the high ends of these estimates are reason enough to consider ordering or recommending placement of the youth in a secure facility to protect the public.

Research has indicated that providing severe or harsh sanctions, such as incarceration, to low-risk youths can significantly increase the youth’s probability of recidivism; it can exacerbate or contribute to other serious negative outcomes, including increased mental health concerns and educational attainment difficulties (Cullen & Johnson, 2016; Lambie & Randell, 2013; Lowenkamp & Latessa, 2004). Therefore, the overestimates of risk associated with descriptive risk categories may lead to the unwarranted detention or placement of youths who will then suffer the collateral consequences of such a decision. Taken together, such wide-ranging understanding of the probability of recidivism, based solely on risk-level category, may increase the number of youths removed from the community in a misguided effort to secure public safety. Relatedly, research using case data suggests that decision-makers may have even greater difficulty appreciating youths’ specific areas of criminogenic risk and need when making treatment referral decisions than when using categorical risk information in dispositional decisions (Petkus et al., 2022).

In contrast, some decision-makers estimated a low probability of recidivism for higher-risk youths, which may lead to judges and probation officers overlooking significant needs that high-risk youths may have that require more intensive intervention. Youths scoring in the high- and very-high-risk categories on risk assessment measures often have mental or behavioral health (e.g., substance use), psychosocial, or environmental needs (Andrews et al., 2011) that, without intervention, can lead to a number of negative outcomes, including deeper penetration into the legal system (see Brogan et al., 2015, for more discussion; see also Dowden & Andrews, 2010; Lipsey, 2009). Thus, by failing to guide decision-makers toward appropriate, targeted interventions for youths at any level of risk and need, the misunderstanding of domain-specific risk/needs data can further increase the possibility of negative outcomes.

Increased Potential for Bias

Although categorical risk levels may have been designed to provide a useful shorthand to understanding youths’ criminogenic risks and needs, the idiosyncratic interpretation of these categories creates substantial risk of bias in the application of risk assessment results. When risk assessment tools are used without fidelity to their developed purpose (and without consideration of the data-driven research supporting these tools), they lose their ability to reduce potential implicit biases and inequitable decisions (Vincent, Guy, & Grisso, 2012). Instead, judges’ and probation officers’ individual implicit or explicit biases about a youth or their previous delinquent behavior could impact the decisions they make about disposition or consequences for violations of probation and lead to inequitable decisions for that youth.

Given this potential for bias, extensive efforts have been exerted to ensure appropriate implementation and fidelity when assessing risk, with great success in promoting reliable administration and scoring of risk assessment tools (e.g., Vincent et al., 2016). However, results from the present study indicate the need for more training and education around the interpretation and use of risk assessment data in order to ensure reliable, valid, consistent, and fair decisions about youths’ case processing.

Unpredictability in Court Process

This study suggests one reason why many youths and families report case-processing decisions as procedurally unjust (Knoche, 2020; Tatar et al., 2012). Life-altering decisions regarding confinement appear to be made—at least in part—on the basis of individual decision-makers’ idiosyncratic interpretations of a youth’s potential risk of committing another crime in the following year, impressions that do not appear to be grounded in accurate data or to reflect realistic appraisals. This may explain one reason why youths and families report an inability to predict the potential consequences or outcomes of any given court hearing or of meetings with probation officers (e.g., Hazel et al., 2002; Huerter & Saltzman, 1992).

Relatedly, probation officers tended to estimate higher probabilities of general and violent recidivism than did judges. This discrepancy in risk perception may contribute to the lack of predictability that youths and families sometimes experience, especially given that they may receive different feedback from probation officers and judges about how they are doing under supervision. These results may be explained by the differences in probation officers’ and judges’ levels of contact with a youth (i.e., amount of time spent with a youth) and the youth’s community (e.g., family, school, mental and behavioral health providers). With more regular contact, probation staff may be more likely to learn about or infer behaviors that can put a youth at higher risk for reoffending (e.g., contact with negative peers, drug use, technical violations of probation). Thus, the probation staff in this study may have been influenced by their own experiences with youths on their caseloads, thereby estimating higher risk than judges, who may learn about behaviors only if they are documented in written reports or in motions filed for probation violations.

Violations of probation are among the most common reasons for detention or out-of-home placement of youths (Sawyer, 2019). However, the present study’s findings suggest that there may be a shift away from overly punitive and incarceration-focused sanctions for youths who violate probation and have identified treatment needs. Although violations vary in nature and severity, most juvenile legal system personnel recommended alternative interventions (e.g., drug/alcohol counseling, substance use/mental health treatment) for a youth who violated their probation conditions by using marijuana. Targeted interventions are more likely than detention or incarceration to lead to positive behavior change among young people (Henggeler & Schoenwald, 2011). Only a very small group of participants believed that probation revocation and placement was the most appropriate response to a positive marijuana screening. However, estimates of probation violations and recidivism were extremely similar by risk category, suggesting that participants may conflate these separate events. Although examples of probation violations often include rearrests and new charges, many probation violations are much more minor—drug use, missed school days, or missed mental health treatment appointments (NeMoyer et al., 2014). Therefore, it is concerning that judges’ and probation officers’ estimates of the likelihood of probation violations were almost identical to their estimates of recidivism.

Recent research has noted that matching a targeted intervention to criminogenic need is inconsistent despite the intended use of risk assessment tools to identify risks and needs of youths (Petkus et al., 2022). This research suggests that training and support for personnel in the juvenile legal system may be needed nationally. However, the present study demonstrated that most legal staff aim to consider the needs of youths when providing a consequence for misbehavior, demonstrating that the nature of the training may need to vary across jurisdictions. Personnel in the juvenile legal system across states may benefit from training on risk data interpretation, but trainers should recognize that the starting point for interpretation may differ by state or even by local jurisdiction.

Role of Youth’s Race

The findings related to the role that a youth’s race plays in placement decisions add to previous literature on the complex and nuanced ways in which race is a factor in dispositional decision-making for legally involved youths. Previous research indicates that a youth’s race can influence judicial decision-making, but the way in which it does so depends significantly on the conditions of the situation. For example, Barrett and colleagues (2010) found that White youths were more likely to be adjudicated as delinquent than Black youths, especially for least and most severe offenses. However, Black youths were more likely to be prosecuted for intermediate-level offenses (e.g., misdemeanors) and more likely to be incarcerated regardless of the offense type (Barrett & Katsiyannis, 2017). Although the current study did not find significant differences in sanctioning decisions for Black and White youths, the potential for disparate application of risk category information still exists.

The present study’s results do not reflect the real-world disproportionality seen in decisions made about youths of color compared with White youths. The distinction between real-world and vignette-based research findings may occur because, in research studies, participants may be very aware of their responses and concerned about perpetuating implicit biases, especially given the frequency of trainings to reduce such bias (National Consortium on Racial and Ethnic Fairness in the Courts, 2014; National Council for Juvenile and Family Court Judges, 2019). Alternatively, participants may assume that White youths present with greater criminogenic needs and therefore have a higher risk of reoffending than Black youths. Given the disproportionately small number of White youths compared with youths of color in the legal system, participants may believe that White youths are not as commonly arrested, and thus, these White youths must have surpassed a higher threshold for arrest and processing (e.g., crime is more serious, youths have higher criminogenic risk factors) than the threshold that youths of color may have passed in order to be entered into the justice system.

Implications for Policy and Practice

This study provides information to policy makers who support the use of practical guidelines (e.g., bench cards, flowcharts) to assist in dispositional decision-making. Providing staff with a road map (e.g., Garrett & Monahan, 2020; National Council of Juvenile and Family Court Judges & National Center for Juvenile Justice, 2021; Vincent, Guy, Gershenson, & McCabe, 2012) for decision-making based on accurate understanding of the relationship between risk level and reoffending (both general and violent), criminogenic risks and needs, and community supports may increase the consistency of dispositional decisions and reduce the unnecessary and harmful removal of youths from their homes and communities. It is also important that a guiding road map specifies and distinguish between percentile interpretation and actuarial approaches to recidivism risk, especially because some participants in the present study appear to have conflated a youth’s risk-score percentile (i.e., how their risk score compares with risk scores of a normative group) with probability of recidivism. For example, despite a lack of published research demonstrating the specific probabilities of recidivism (as a percentage) for youths with a given risk assessment score or within a specific risk-level category, some participants identified that their counties provide the percentage likelihood of recidivism associated with the risk score for youths on their caseload. Given this, a road map may reduce the possibility for such mistakes.

Finally, this study has implications for clinical practice. When forensic evaluators assess a youth’s potential risk of dangerousness or of reoffending, they should not only provide their findings but also provide appropriate psychoeducation about the risk assessment tool, including the meaning of categorical risk information, the lack of existing data on the probability of juvenile reoffending (particularly of violent reoffending) based on risk level on the various instruments, and the frequent overestimates of risk probability.

Limitations

Several limitations of this study are important to acknowledge. First, there are potential limits to the study’s generalizability. Although we made efforts to obtain a large and representative sample from across the United States (e.g., by using national and statewide LISTSERVs), the number of participants from the states represented varied significantly, with a large proportion of the sample from the mid-Atlantic and midwestern regions and very few participants from states with large numbers of juvenile legal system personnel (e.g., California, Texas). However, limited data exist on the demographic breakdown of juvenile court judges and juvenile probation officers across the country; therefore, it is difficult to ascertain whether this study’s sample was demographically representative of the overall population of juvenile legal decision-makers.

Second, some discussion of the ecological validity of this study is necessary. Although using a vignette as the basis of the survey reduced potential noise in the data and allowed for an experimental design, the study did not address multiple factors that likely influence decision-making about youth placement. For instance, the direct relationship between risk category and placement decision is likely affected by other variables not included in the study, such as family involvement in the court process, and other behavioral or criminogenic need considerations. To consider another integral example, jurisdictions are heterogeneous in their access to evidence-based and cost-effective community-based services (Hamilton et al., 2007; Lipsey et al., 2010). Given this, it is likely that such availability, or lack thereof, plays a role in the decisions that judges and probation officers make about interventions for legally involved youths. Though the present study did not gather information related to available alternatives to detention or placement, this is an important avenue for future research. Still, the simplicity of this experimental design allowed for clean interpretation of the data and produced strong internal validity.

Last, judicial and probation staff were asked about their likelihood of confining a youth following a violation of probation for marijuana use. Although this is a common violation and is still considered a violation in many states in which marijuana is not an arrestable offense, there have been concerted efforts to educate judges and probation officers on more effective interventions for substance use (e.g., substance use counseling) than using confinement as a sanction (e.g., Miller & Aleem, 2012; Office of Juvenile Justice and Delinquency Prevention, 2022). Indeed, participants in this study overwhelmingly identified responses more directly tied to addressing marijuana use (i.e., drug and alcohol assessments/counseling, general mental health interventions, and decision-making interventions). Therefore, this study’s use of a positive marijuana screening as the example of a probation violation may have limited our ability to ascertain the probability of judicial staff using confinement as a sanction for a violation of probation.

Future Directions

Future research can build on this study by reducing the methodological limitations. Specifically, the implications of the present study would be supported by additional research using various types of probation violations, including those with multiple degrees of severity (e.g., curfew violation, failure to appear, rearrest). The present study would be further supported by a replication with a larger sample of judicial decision-makers in order to determine the relationships among risk level, interpretations of risk information, and placement decisions for this specific integral population. Additionally, future research should examine judges’ and probation officers’ interpretation and decision-making on the basis of the specific types of risks and needs (e.g., education, mental health, family functioning) identified via risk assessment measures. Recent research has begun to examine decision-making based on youths’ risk levels (e.g., Petkus et al., 2022), and future research should continue to evaluate this with disposition, sanction, and detention decisions to ascertain the extent to which research findings reflect real-world decision-making. Finally, education about risk assessment information should be studied to better understand whether training increases the knowledge of, improves accuracy of interpretations made by, and enhances the legal decision-making of legal system personnel.

Future research should also prioritize identifying the probabilities of recidivism—both general and violent—among youths with various risk scores and with the different risk-level categories. As observed in this study, judges’ and probation officers’ understanding of recidivism likelihood varied widely within each risk category. Therefore, without empirical data offering better information on the likelihood of a youth reoffending—and particularly of violent reoffending—these idiosyncratic interpretations will likely continue to produce unpredictable and poorly grounded decisions about youths across the multiple phases of juvenile legal case processing.

Conclusion

The findings of the present study indicate that juvenile legal decision-makers use their idiosyncratic and skewed impressions of risk probability to guide their high-stakes decision-making about the detention and out-of-home placement of youths. Results of this study suggest that policy makers and juvenile legal system administrators should increase the training and education on the accurate interpretation and appropriate application of risk assessment results for people who use these data to inform their legal decisions. Additionally, they might institute structural supports to (a) enhance the consistency, predictability, and fairness of decisions regarding youths’ dispositions; (b) decrease the number of youths unnecessarily removed from the community in an effort to ostensibly protect public safety; and (c) decrease the number of youths receiving overly restrictive interventions for noncompliance. Together, these efforts could translate into more accurate and appropriate use of risk assessment data by legal decision-makers, resulting in less restrictive, community-based dispositions and interventions for youths without compromising public safety.

Public Significance Statement.

Judges and probation officers used their own understanding and interpretation of specific recidivism risk levels when making decisions about whether a youth should be placed away from home or in detention. Legal decision makers should be aware of the potential for misinterpreting risk assessment results, as such misunderstanding can lead to significant consequences to the youth or the safety of the public.