Assessing the Use and Impact of Points and Rewards across Four Federal Probation Districts: A Contingency Management Approach

Abstract

Contingency management (CM) is a well-acknowledged behavioral approach for incentivizing changes in behavior and attitudes. A version of CM was piloted in four federal probation districts to determine whether systematically awarding points and rewards for key behaviors could be implemented and impact recidivism rates. A case controlled match design was conducted with a CM sample (referred to as Justice Steps (JSTEPS)) (n=128) who were individually matched to a comparison sample (n=128) on six variables related to recidivism. Analyses compared the number of technical violations and new arrests between JSTEPS participants and a historical comparison sample. Using Kaplan-Meier survival analysis, results indicate JSTEPS sites using early CM rewarding strategies tend have delayed recidivism than others. A research agenda is outlined.

Introduction

Contingency Management (CM) is an evidence-based treatment using principles of operant conditioning and behavioral strategies demonstrating strong positive results in substance abuse (Fitzsimmons et al., 2015; Shearer, Tie, &, Byford 2015; Rash, Stitzer, & Weinstock, 2017) and other settings. CM has been studied extensively in substance abuse treatment programs as a means of reinforcing desired behaviors, such as negative drug tests and engagement in treatment services (Petry, DePhilipps, Rash, Drapkin, & McKay, 2014). When using CM, clinicians or staff identify desired behaviors, assign values to the observed behaviors and deliver rewards when an individual achieves the desired behavior or earns a certain number of points. Rewards can be delivered in the form of tangible (e.g., gift cards) or intangible (e.g., verbal praise) incentives (Kirby et al., 2006), and both are found to be equally effective (Rash, Stitzer, & Weinstock, 2017). CM is a regimented approach that requires providing swift and certain positive reinforcements (rewards) to observed behavior(s).

In justice settings, rewards have mixed results depending on the different techniques for delivering rewards and/or sanctions, or some combination of the two, and the way in which the methods are measured (Wodahl et al., 2011; Hawkins & Kleiman, 2009; Lattimore, et. al., 2016; O’Donnell, et al., 2014; Mowen, Wodahl, & Garland, 2018). Some justice settings have examined the use of rewards as part of behavioral interventions and found that rewards enhance better outcomes (Wodahl et al., 2011). A few studies find positive benefits in using deterrence strategies of punishments (Hawken, & Kleiman, 2009; Kilmer, et al., 2013). While the use of a matrix of graduated sanctions and incentives is recommended (NIC, 2004), its use is not very widespread even in justice settings (Rudes et al., 2012; Lattimore, et. al., 2016). The matrix approach, and various studies on incentives and/or sanctions, have found varied impact depending on how sanctions and/or rewards are measured in the study. While the justice setting prefers punishments delivered as sanctions or negative reinforcements, positive reinforcements have gained traction as part of the evidence-based practices approach where the use of incentives has been identified as a tool to garner engagement and traction in addressing community supervision goals (Taxman, Shephardson, & Byrne, 2004; NIC, 2004; Taxman, 2008, to name a few). In non-justice settings such as housing and mental health services, rewards enhance stabilizing housing and mental health functionality (Rash, Stitzer, & Weinstock, 2017). In substance abuse treatment settings, rewards enhance negative drug tests and positive treatment outcomes (Rash, Stitzer, & Weinstock, 2017).

In the Justice Steps (JSTEPS) study, researchers worked with four federal probation districts and various probation initiatives housed within these districts (i.e. problem-solving courts, a halfway house, and standard probation face-to-face contacts) to: 1) determine the acceptability, feasibility, fit, uptake, and sustainability of CM protocols in justice settings (see Rudes et al., 2013; Portillo, Rudes, &, Taxman 2014); 2) learn more about the working relationships of justice partners within problem-solving courts (Portillo, Rudes, &, Taxman 2014); and, 3) implement a quality improvement process involving Plan-Do-Study-Act (PDSA) cycle to facilitate the implementation of CM (Taxman et al., 2012). Prior studies have documented the implementation of the CM framework in these settings which varies considerably (see Rudes et al., 2012). This paper is focused on a topic that is not explored in prior work--the goal of examining the different implementation features of CM on recidivism outcomes. The following paper discusses the literature on CM and this research study. A case-controlled match design is used to assess the impact of CM on probationer level outcomes by examining recidivism rates. To conclude, the policy implications and future research agenda is outlined.

Background

CM is a frequently studied as a behavioral modification technique to prevent relapse among substance abusing populations (see Kirby et al., 2012; Stitzer & Petry, 2015), although its use has been expanded to examine smoking cessation, housing stabilization, chronic disease management, and other health care outcomes (Rash, Stitzer, & Weinstock, 2017). More recent studies examine the use of a version of contingency management in justice settings in halfway houses, prisons, probation, problem-solving courts, parole, etc. to foster positive behaviors. An overview of contingency management (CM) is discussed followed by a review of the literature on the effectiveness of CM in health and justice settings.

Contingency management

CM is based on operate conditioning methods to alter behavior by using positive reinforcements to alter an individual’s behavior in a constructive way (Petry, 2000). CM strategies can increase an individual’s level of engagement in changing behavior for the better or to encourage a set of desired behaviors (Taxman et al., 2010; Wodahl et al., 2011). CM innovations usually adhere to three basic principles (Higgins, Budney, &, Bickel 1994a): 1) the treatment provider creates an environment where target behaviors such as drug abstinence are quickly observed, making the behavior change easily recognized and measurable; 2) reinforcers are used to acknowledge that the person achieved the targeted behavior; and 3) if the individual does not achieve the targeted behavior, reinforcements are not provided (Petry, 2000). The purpose of positive reinforcements is to help stimulate dopamine (the reward center of the brain) to restructure behavior and responses.

Positive reinforcements often refer to: 1) material rewards such as gift cards, vouchers, or some monetary based system; 2) social rewards including verbal or written praise; and, 3) a combination of the two. The desired behavior change refers to either stopping a negative behavior (i.e. using drugs, drinking alcohol, smoking cigarettes, etc.) or starting/reinforcing a positive behavior (i.e. attending treatment, working, going to school, abstinence, etc.). Voucher-based systems which provide for points or allow for the accumulation of points to give out a voucher or reward do equally as well as prize-based systems (referred to as fishbowl) where an individual obtains a prize for meeting said target behaviors (Rash, Stitzer, & Weinstock, 2017).

CM in substance abuse treatment settings reduces substance use, for a broad array of substance use behaviors including cocaine (Burch, Rash, & Petry, 2017); opioids (Hatzler & Rabun, 2013); alcohol (Cooney et al., 2016); and nicotine (Raiff, 2012). CM is equally effective for a range of substances (Lussier, Heil, Mongeon, Badger, & Higgins, 2006; Griffith, Rowan-Szal, Roak, Simpson, 2000). CM also increases treatment attention and retention (Stitzer and Petry, 2006) and it appears to be effective for a wide range of substance abusers regardless of race or gender, comorbid disorders, medical comorbidities, history of prior SUD treatment or other conditions (Rash, Stitzer, &Weinstock, 2017). Rash, Stitzer, & Weinstock (2017) document how CM has been used to achieve other desirable outcomes such as improvements in the housing status of those that are homeless or in recovery housing programs and increases in smoking cessation, medication adherence, exercise and high-risk sexual behaviors. Collectively, positive reinforcers have a small to moderate effect in changing behaviors in a number of domains. For justice involved clients, CM offers the potential for using a different method to achieved desired behavioral change.

Contingency management in criminal justice settings

In many ways, CM is a strategy that should be consistent with the justice setting since the focus is on swift and certain responses to specific behaviors such as drug use, employment, and following the rules of supervision. During the 1990’s, many parole and probation agencies considered the use of graduated or administrative sanctions to address noncompliance to conditions of release. The premise of graduated sanctions is that the response to a behavior should be graded to the severity of the behavior, and there should be consistency of responses within an agency to common behavioral issues. Results from the use of administrative sanctions are mixed in terms of willingness of probation and/or parole officers to use the sanction matrices, the frequency by which the officers follow the sanction matrices, and the support that officers have in structured administrative sanctions (Turner, et al., 2012; Rudes, 2012; Steiner, et al., 2012). Most efforts emphasize punishments for negative behaviors, although incentives for rewarding positive behaviors are generally mentioned. The Hawaii HOPE presented a modified approach to graduated sanctions with the use of a judge to dispense the sanctions at weekly court hearings, use of frequent and scheduled drug testing, and opportunities for the judge to swiftly respond to the client’s behavior. This is known as swift and certain, given that the approach is to have a structured response with an authority figure (judge). The original study of using swift and certain responses to drug using behavior in court found that this reduced rearrest (Hawken & Kleiman, 2009). The judge dispensed primarily sanctions but also provided for incentives. Hamilton and colleagues (2016) evaluated a modified version of HOPE dispensed by probation officers using an administrative sanction guideline that allowed officers to use up to 30 days in jail for different noncompliant acts. The probation officers guidelines reduced arrests, incarceration and costs. The majority of the emphasis was on sanctions, and little attention was given to incentives.

Other replication studies of the Hawaii HOPE model have not found that the swift, certain and fair responses reduce recidivism, and in fact it may increase recidivism. The Delaware Do Your Time experimental study found that the protocol did not impact recidivism, but the officers and court did not follow the protocol and there was low fidelity to consistently use sanctions to address non-compliant behavior such as positive drug tests (O’Connell, et al., 2016). The four-site HOPE randomized replication study did not find support for the deterrence-based approach. In this replication, an emphasis was placed on implementing the Hawaii HOPE model to fidelity which meant that probationers were consistently drug tested, appeared in court if there was an infraction, and sanctions provided if there was an infraction, including short periods of jail (under 30 days). The HOPE model was tested in Saline County, Arkansas, Essex County, Massachusetts, Clackamas County, Oregon, and Tarrant County, Texas—all were new to implementing the model. The implementation process achieved these benchmarks except for the swift response, the four sites had difficulty having the individual appear in court for the sanction within a seven day window. Sites had a difficult time achieving swiftness in responses, and most took an average of seven days. The delayed response to positive drug tests and other infractions illustrated some of the challenges of implementing HOPE with fidelity. This well-designed RCT reported that the HOPE probationers had similar outcomes as the control group on a variety of recidivism measures (arrest, revocation, arrest/revocation, and reconviction) but revocations were higher in two sites and reconvictions higher in one site. One site did observe a reduction in drug arrests for the HOPE participants. The emphasis was on sanctions or a deterrence-based strategy of sanctions which does not appear to change behavior (Lattimore, et. al., 2016). Incentives were not tested in the replication HOPE study.

Some studies have explored the use of monetary vouchers for justice-involved individuals. Prendergast and colleagues (2006) studied allowed participants in drug court to earn $10 as rewards for certain behaviors such as negative drug tests and completed treatment plan. The monetary voucher had no effect on retention in treatment or negative drug test results. Marlowe and colleagues (2008) examined the differences among drug court, drug court plus increasing incentives (gift certificates), and drug court plus high-magnitude gift certificates on attendance, advancing through drug court, and drug negative urine samples. They found no differences among the groups on any outcome measures. Hall and colleagues (2009) did not find that enhancing drug court rewards with monetary values improved results. In another study, Prendergast and colleagues (2015) did not find that monetary rewards in drug court for young marijuana users on probation had an impact on increases to initiation or retention in treatment for parolees. It also reduced positive drug tests.

Recent attention has been given to how rewards and/or sanctions as responses are delivered in justice settings to better understand the effect. In an experimental study of rewards coupled with sanctions for high-risk parolees, the sole use of sanctions does not appear to be sufficient to motivate probationers to meet target behaviors related to treatment and drug use. Rewards did have the impact of reducing positive drug tests, particularly for those that used marijuana (Friedmann et al., 2008). The model called for rewards to be delivered by officers and treatment providers at each probation visit if the individual met key target behaviors related to attendance at required meetings and some life stability goals (i.e. employment, housing, etc.); this occurred 90% of the time (Taxman & Friedman, 2009). The Steppin’ Up model did not reduce recidivism (Friedmann et al., 2008).

Examining the use of rewards and sanctions as part of an intensive supervision program in Wyoming, researchers found that the use of sanctions, as measured by the total number of sanctions received (mean=2.7) resulted in adverse events of more failures on ISP. The more sanctions delivered, the more often the individual was reported a failure on supervision. The study found that rewards, measured as the total number of rewards provided during the period of ISP (mean=3.5) was related to positive program completion. When the proportion of rewards to sanctions was greater than 2, probationers tended to complete ISP successfully. But, when sanctions outnumbered rewards, probationers tended to be unsuccessful in ISP. The study focused on successful completion of ISP but it did not explore outcomes on rearrest or reconviction (Wodahl, Garland Culhane, and McCarty, 2011).

Mowen, Wodahl and Garland (2018) reanalyzed the Serious and Violent Offender Initiative (SVORI) data to understand how the use of sanctions and rewards affected relapse and recidivism. Sanctions was defined as the number of sanctions given (mean=.218) and reprimands were given (.24) during the SVORI time period. Rewards were defined as the number of praises received (mean=.425) and the number of reductions in requirements (mean=.21). A series of lagged models were constructed which found that the use of praise reduces positive drug tests and offending behavior. The number of sanctions and use of reprimands has q counter effect--increasing drug use and offending. Models that included both sanctions and rewards found that the interaction between the number of incentives and sanctions reduces offending but has no impact on drug use. This non-experimental study found that the way in rewards and sanctions are used has an impact: 1) how officers reward individuals matters; 2) when sanctions and rewards are offered concurrently by officers also matters; and 3) the ratio of rewards to sanctions has differential impacts on outcomes. The inconsistent findings suggest that more attention needs to be given to developing a knowledge base around effective delivering of rewards in a justice environment that will productive positive outcomes. One area that needs more attention is the attitude of justice actors who deliver rewards and sanctions. These actors may not feel comfortable with using incentives in a punitive setting or rewarding people for doing what is required (Sinha et al., 2003; Murphy, Rhodes, &, Taxman 2012).

Swift and certain rewards in treatment settings

Focusing on reinforcing positive behavior is the key aspect of CM, but the time to administer the sanctions is also a critical factor. Studies indicate that reinforcement schedules can range from immediate (McCaul et al., 1984), delayed (Milby et al., 1978), and a combination of immediate and delayed (Calsyn et al., 1994). In a meta-analysis of voucher-based reinforcement therapy (VBRT), Lussier et al. (2006) found that the effectiveness of VBRT depended on the time frame until the reward was administered. The study found that VBRT was given from two weeks to 12 or more weeks. They conclude, “delivering the voucher immediately after verifying abstinence (same clinic visit) rather than later resulted in significantly larger effect sizes” (Lussier et al., 2006, p. 200). Similarly, Griffith et al. (2000) conducted a meta-analysis of CM in outpatient methadone treatment. Using urinalysis testing to measure drug use during treatment, CM was found to be more effective when the delivery of reinforcement was immediate rather than delayed. Griffith et al. (2000, p. 61) concludes, “[c]ontingency management had a less favorable effect if the interval was delayed over 24 hours (mean r = 0.19) than if it was more immediate (mean r = 0.56).” The Hawaii HOPE model had individuals appear before the court within a few days and this appeared to deliver results but the replication study achieving this milestone and did not generate the same positive outcomes. When using CM approaches, especially in criminal justice settings, reinforcements offered early in the process also appears to be critical in promoting better outcomes rather than delaying the rewarding or scheduling. This is similar to operant conditioning where providing early reinforcement is better than delaying (Domjan, 1996).

JSTEPS study design

The JSTEPS study launched in January 2010 as an implementation study where quality improvement processes were used to design and test a CM protocol in probation settings. Prior to the design phase for the CM procedure in each site, staff from the four districts were asked to complete a baseline survey of their opinions on punishment and rehabilitation, use of rewards, and general operations in their offices and/or specialized courts (see Murphy, Rhodes & Taxman, 2012 for survey findings). Qualitative researchers conducted site visits in all four sites to understand the supervision context. During these visits, researchers collected data on district personnel and operations through non-participant observation and qualitative, semi-structured interviews (Rudes, et al., 2013). During the 12-month utilization period of CM by probation officers, the researchers collected data on the use of the CM and the probation outcomes.

JSTEPS Study Sites

Four federal probation agencies volunteered to be in the study. The actual name of the district cannot be provided due to confidentiality. Each probation district selected a program or initiative within their agency for the CM implementation study including problem solving court (n=3), half-way house (n=1), or standard supervision (n=1). All of the sites had an implementation team—the sites that included a problem solving court included the judge, U.S. Attorney, Public Defender, probation officer, and treatment provider; the halfway house included the judge, public defender, and probation officer; and the office that focused on standard probation used a cross-section or the probation officers. Two of the federal districts have used some type of incentive systems in the past and two courts had worked on interagency initiatives for over two years. The other districts were relatively new to addressing interorganizational dynamics. For more information about the formative nature of the development of CM, refer to Rudes, et al., 2012.

JSTEPS Study Procedures

Contingency management had to be adapted for a probation setting. The research team identified nine core principles for JSTEPS CM: 1) provide positive incentives to probationers using a point system that allocates points for each positive behavior; 2) establish clear guidelines about the target behaviors that probationers could earn points for; 3) emphasize abstinence as a key objective; 4) provide incentives early in the CM program by giving rewards for accumulation of small amounts of points; 5) use point escalation to promote sustained good performance on supervision such as negative drug tests or continued employment; 6) integrate point system into the normal operation of probation such as in phase systems; 7) use adding bonus points to reinforce incentives for positive behavior; 8) require no more than three target behaviors at a time to make the behavior management process feasible; and 9) choose the areas where the probationer having difficulty and shift that towards positive by rewarding his or her efforts to improve (Taxman et al., 2010). These were the desired features of the JSTEPS CM protocol. Each site developed their own CM procedure and adapted it to their own individual setting.

In the JSTEPS system of graduated rewards and sanctions, behaviors are separated into color-coded categories to facilitate comprehension. These colors are used to visualize behavior categories in JSTEPS associated software and to provide a visualization of the target behaviors. “Red” behaviors refer to criminal acts. This is an area where no points can be earned but the person can be sanctioned. “Orange” behaviors emphasize abstinence from drugs and alcohol. The study team recommended placing a strong emphasis on drug and alcohol using behaviors (i.e., orange) because they are easily measurable, and the majority of probationers in the programs had some histories of substance abuse. “Yellow” behaviors refer to attendance at required meetings intended to support abstinence; some are specific to people in recovery are attending AA/NA meetings attending treatment sessions, attending a probation visit or being on time for court. “Green” behaviors refer to stabilizing behaviors that support recovery, long-term goals of being drug-and crime-free. Sites were able to select any pro-social activity(s) to specify these behaviors such as employment, housing, parenting, caretaking, civic duties, community services or other behaviors that probationers’ can relate to. Several sites used this category to promote employment. A designated JSTEPS software program allowed sites to select up to nine behaviors to track, each of which can be assigned a point value. Table 1 lists the point and incentive schemes that sites eventually implemented (see Rudes et al., 2012 for a discussion of the implementation process).

Table 1.

Description of point and incentive systems by site.

	Site One	Site Two	Site Three	Site Four
Points for Drug and Alcohol Abstinence (Orange)	1 to 10 (escalating), plus bonuses	1 or more (escalating without limit), plus bonuses	0 to 8 (tracked but not points awarded or escalating), plus bonuses	1 to 10 (escalating), plus bonuses
Points for Behaviors that Support Abstinence (Yellow)	1 (no escalation or bonuses)	1, 2, or 5, based on type of activity	0 (tracked but no points awarded)	1 to 10 (escalating) for attending treatment, plus bonus for starting treatment; 1 (no escalation or bonuses) for other attendance
Points for Behaviors that Support Recovery (Green)	1, 2, or 3, based on type of behavior (no escalation or bonuses)	2 or 5 (no escalation), based on type of behavior, bonus for obtaining a job	1, 5, 10, or 20, based on type of behavior (no escalation or bonuses)	1, 5, 10, 20, 25, 50, or 75, based on type of behavior (no escalation), bonus for completing job readiness training
Point Levels to Earn Rewards	100, 250, 400, 550, 700	Every 10 points	Every 10–125 points	50, 200, 500, 750, 1000
Court Type	Specialized	Specialized	Specialized	Standard Probation

Sites were given broad discretion in determining the amount and type of rewards. The instructional materials developed by the study team included information on both social and material rewards, and sites were encouraged to incorporate both. Social rewards include verbal and written praise whereas material rewards can include decreases in reporting requirements, many of which are built into problem-solving courts. The actual rewards distributed by sites did not show a great deal of variance; most sites surveyed participants to determine what material rewards would be most meaningful to them. The most popular selections were gift cards of different values. Some sites offered more tailored rewards, such as a club membership for a participant who played chess or movie tickets.

The JSTEPS study used a mixed-method longitudinal designed to assess the implementation and use of CM in the four probation settings. Overall, the four probation settings found CM to be both acceptable and feasible but several barriers inhibited the full implementation of the CM strategy by officers. Typically in the substance abuse literature, CM is usually designed to focus on one behavior (usually abstinence or treatment retention). In probation, probationers have a number of requirements (an average of 17, see Taxman, 2012). The CM protocols had to identify which behaviors to reward and which to not reward in a positive incentive scheme. This proved to be challenging since some sites could not differentiate among the varying conditions of supervision and had a difficult time identifying a set of behaviors that were more desirable than others. Some sites also had a difficult time minimizing the number of behaviors that a person could be sanctioned for. The tendency to overidentify behaviors to be incentived and/or sanctioned was a major barrier to CM implementation.

The adapted CM protocols developed by the study sites encountered a number of challenges. These include: 1) probation agencies could not select just one behavior to reinforce but the sites tended to have an average of nine behaviors or multiple target behaviors; 2) the more behaviors and/or target behaviors that the probationer had to abide by, the difficult it was for the probationers to earn rewards or be incentivized; 3) the probation agencies lacked consensus on the concept of swiftness of response to positive behaviors, and the length of time varied from seven days to 30 days or longer; and, 4) some of the agencies (and staff) placed greater emphasis on the use of sanctions instead of rewards (Rudes et al., 2012). The agencies designed the CM protocol but often individual probation officers adapted or changed the CM schedule with their assigned probationers. Some sites gave probationers a list of the behaviors that would result in the reward and the schedule for attaining rewards. Some officers did not share the target behaviors or incentives with probationers.

A further examination of the challenges posed to implementing CM by site are detailed here. In Site One, probation officers and the reentry court team members wanted to individualize all target behaviors, sanctions, and rewards; for example, one probationer’s “green” target behavior could be attending school, while another’s was to fix his vehicle. This was consistent with how that reentry court is structured, but it made implementation of CM (and use of the JSTEPS tool) complex. There were inconsistent point structures. Site Two experienced leadership changes with a retiring probation chief, and the presiding judge and public defender in the drug court were ambivalent about rewards. This meant that the new probation chief did not have good support for CM. In Site Three, the reentry court was a very new program, and the team had difficulty defining the target behaviors they wanted to incentivize. The officers did not want to share this list of target behaviors with probationers which resulted in a lack of transparency; probationers had a difficult time earning points. Site Four tested CM for a portion of their caseload (every officer had only five probationers on their caseload involved on CM which limited the use of CM by the officers). While this provided a large sample size, the structure meant that POs would only use a CM approach with a fraction of their caseloads and officers had difficulty implementing CM. The site had limited funds for rewards and was distributing them across a large sample. The small dollar value of the reward made it difficult to provide meaningful rewards within budget. Finally, a fifth site had to cease being in the study due to an employee that was selling drugs to probationers. (This site was considered to be contaminated since the probationers were arrested and violated. This site was not included in the analyses.) The impact study is detailed below.

Methods

This study assesses whether JSTEPS had an impact on probationer outcomes on supervision. For this study, data were collected on points and rewards distributed to each JSTEPS study participant. A case-controlled match design was used to create a comparison group. The quasi-experimental design allowed us to select a comparison group that had the same characteristics as probationers in the JSTEPS group. An individual matched design is closest to randomization when it is not possible to randomize (Weisburd, Lum, & Petrosino, 2001). It was not possible to randomize due to the small number of probation officers involved in the JSTEPS study and the difficulty of randomizing probationers. The match procedure ensures that the probationers are similar in characteristics and that they are supervised by the same officers as the JSTEPS probationers to reduce selection bias and potential contamination issues.

Probationer outcome data: Matching JSTEPS and comparison probationers

The population from which the JSTEPS and comparison sample was selected is of probationers supervised at the four sites in the JSTEPS study. The JSTEPS probationers were supervised by select officers that were part of the project and the comparison group was also supervised by said officers. It was important to ensure that the officers were the same due to the wide discretion that officers have regarding how they supervise probationers and when they use sanctions with probationers. The JSTEPS study sites were selected due to their prior willingness to implement evidence-based practices, and they represented geographical diversity. For example, one site was located in the Northeast, one in the Midwest, one in the West, and one in the South. Probationer data were collected on JSTEPS and comparison probationers through the Administrative Office of the U.S. Courts (AO) using the Probation/Pretrial Services Automated Case Tracking System (PACTS) data were used which contained criminal history information for federal probationers including FBI arrest history data. The final sample was 128 for JSTEPS probationers and 128 for comparison probationers.

Case control matching variables

The PACTS system had limited data on characteristics of probationers. The researchers identified six control variables were used as a part of the matching process for JSTEPS and comparison probationers, these variables were selected due to their relationship with recidivism and to ensure that comparison group probationers were from the same district. The variables included district, race, gender, age, score from the standardized Risk Prediction Index (RPI) scale used by the AO, and length of time on probation. The probationers’ RPI score was calculated based on: 1) the probationers’ age at the start of supervision; 2) how many times the probationer was arrested prior to the instant offense; 3) if a weapon was used in the commission of the instant offense; 4) if the probationer was employed at the start of supervision; 5) if the probationer had a history of illegal drug usage or alcohol abuse; 6) if the probationer ever absconded from a previous period of supervision; 7) if the probationer had a college degree; and 8) if the probationer was living with a spouse and/or children at the start of supervision. The RPI is a validated risk tool (Eaglin et al., 1997). Finally, length of time on probation before beginning JSTEPS was calculated to ensure that the probationers were in a similar timeframe of their probation status. (Length of time was determined by subtracting the start date of JSTEPS from the date both JSTEPS and comparison probationers began supervision, in months). Table 2 presents the distribution of the final sample by matching variables. Comparisons using independent sample t-tests and chi-squared tests indicate that no significant differences exist between the JSTEPS and comparison probationers on the matching variables.

Table 2.

Percentages and means of matching variables by site.

	Site 1		Site 2		Site 3		Site 4		All Sites a
Vars.	JSTEPS	Comparison	JSTEPS	Comparison	JSTEPS	Comparison	JSTEPS	Comparison	JSTEPS	Comparison
											t	χ 2
Race / Ethnicity (%)
White	--	--	46.7	46.7	19	19	39.3	39.3	40.0	40.0	0.0
Black	--	--	50	50	81	81	59	59	44.2	44.2	0.0
Other	100	100	3.3	3.3	--	--	1.6	1.6	15.8	15.8	0.0
Gender (%)
Male	56.3	56.3	83.3	83.3	85.7	85.7	83.6	83.6	81.8	81.8	0.0
Female	43.8	43.8	16.7	16.7	14.3	14.3	16.4	16.4	18.2	18.2	0.0
Age	41.1	42.2	37.0	36.7	34.7	35.1	37.9	39.1	38.1	38.8	−0.7
Risk prediction index	6.6	5.0	6.3	6.1	7.3	6.7	6.3	6.2	6.4	6.0	1.8
Time on probation	.8	2.0	8.6	8.6	16.7	17.0	1.3	2.0	7.0	7.5	-0.5

N = 256

^aNo statistically significant differences exist between JSTEPS and comparison groups on matching variables for all sites.

Outcome measures

The impact of the JSTEPS program was evaluated on one outcome: recidivism. Recidivism was a binary variable, with a “1” representing any indication of a new arrest or technical violation during the observation period within one year after placement on supervision. Also, the date of the event from the onset of probation was also used in survival analyses. Both arrest data and technical violation data were collected from offenders’ FBI arrest histories. The combined measure was used because the JSTEPS process invites officers to be more aware of the actions of the probationers which could lead to enhanced violations such as those observed in other intensive supervision experiments (Petersilia & Turner 1993; Taxman, 2002).

JSTEPS points data

The study team provided a secure online website to sites to track probationer behaviors, points, rewards, and sanctions. Sites were trained on the tool and asked to use it to track probationers’ adherence to the selected behaviors. The tool calculates the points earned, creates a behavior management contract, graphs probationer progress, and alerts the user when a probationer is due for a reward or sanction. Figure 1 is a screen shot from the home page of the JSTEPS software. Though the tool is designed to be updated with each meeting with probationers, sites used it with varying frequency and fidelity to the model. Some sites elected to track in a different format, such as Excel spreadsheet given that they had to do double entry into the PACTS and JSTEPS system.

Figure 1.

Screenshot from JSTEPS software.

Data analysis

Points data

Data was obtained on the use of CM and how many points were earned by each JSTEPS participant and provided in Table 3. The variables of interest included: 1) date probationer initially entered into JSTEPS software (considered program start date) or excel sheet; 2) number of contracts; 3) number of points earned in each category (red, orange, yellow, and green); 4) compliance in each behavior category; 5) reward level probationer reached; and 6) dates of each follow-up contract. Data were analyzed to assess both how well individual probationers were performing and whether sites were using the tool as intended (time between contracts). The analyses of points data for this paper include descriptive statistics on: 1) mean number of contracts per probationer in each site; 2) mean number of points earned per probationer; and 3) distribution of points across behaviors.

Table 3.

Summary of points and rewards earned by clients.

	Site One	Site Two	Site Three	Site Four
Number of Subjects Enrolled	15	20	15*	57
Mean Number of Contracts (per client)	15	9.3	7.2*	4
Mean Total Points Earned (per client)	25*	15.8	4.5	94
Mean Abstinence Points (per client)	19*	15	6*	41
Total Rewards Earned	Not reported	0	5	20
Mean Time to First Reward	30 days	NA	45 days	33 days

^*Because of limited reporting/use of the JSTEPS Tool, this numbers may not accurately reflect contracts and points earned by clients

CM patterns

Some JSTEPS sites elected to provide rewards early in the probation process while others would only reward the individuals after they had accumulated a certain number of points. Early rewarding refers to sites 1 and 4 where probationers with rewards for various behaviors within the first month of supervision. Delayed reward was used by the other two sites that provided rewards after the first month of supervision. This timing represents the swift portion of the CM process. Comparison probationers were coded as a 3 if they were not provided with any rewards.

Probationer outcome data

Crosstabulations were used to demonstrate the percentage of technical violations and new arrests between JSTEPS and comparison probationers by site. Additionally, Kaplan-Meier survival analysis was used to demonstrate the average number of days (up to one year) until recidivism for the study participants. The time variable was calculated by subtracting the JSTEPS and comparison offenders’ actual end date of probation from the date the probationers began JSTEPS. The status variable was recidivism (i.e., technical violation or new arrest), and the factor variable was whether an offender was in the JSTEPS or comparison group.

The Kaplan-Meier survival analysis examined the point schedules on recidivism rates. Mean survival times in days, standard errors, and 95% confidence intervals (CI) demonstrate differences between the groups. Kaplan-Meier creates three different statistics: 1) the log-rank; 2) the Breslow; and 3) Tarone-Ware. Each statistic has a commonality in that they “compare the number of terminal events with the expected number for each time interval in the study” (Wilson & Davis 2006, p. 319). The log-rank statistic weights each interval between the indicator of type of reward structure: early rewarding JSTEPS, delayed rewards JSTEPS or no rewards.

Findings

Points outcomes

Data on points earned and rewards distributed over a six-month period for the JSTEPS sites were examined, as shown in Table 3. Officers used the CM on eight (8) to 57 probationers across the four sites. The mean number of behavioral contracts per probationer ranged from 4 to 9.3, and the mean total points earned ranged from 4.5 to 94.0. Even within sites, there was a large range in total points earned per probationer; for example, in Site Four, the total points earned ranged from 0 to 367. The range in use of CM is affected by frequency of contacts with probationers (most officers see their probationers monthly) and the value assigned to each target behavior as per the site schedule.

Compliance rates varied greatly by site, though sites appeared to see the greatest compliance rates with abstinence behaviors. Most sites focused on substance-abusing probationers. The rates of compliance with abstinence were 97% to 99% for four sites. Compliance rates for attendance behaviors ranged from 41% to 98%, and compliance with long-term goals ranged from 18% to 85%.

The distribution of points was examined around the three categories of behavior and the number of rewards dispersed to probationers, as shown in Table 3. In Site One, the team emphasized abstinence from drugs and alcohol, assigning escalating points to negative drug tests but not to attendance or stabilizing behaviors. The mean total points per probationer was 25, with the mean number of abstinence points at 19. At Site Four, points were heavily concentrated in both abstinence behaviors. This site saw a mean number of 94 points per probationer, with a mean of 41 for abstinence, 9 for attendance, and 59 for progress toward long-term goals.

With regard to rewards, Site Four reported the largest number of rewards earned and distributed. We compared the average time to first reward for each site, and found that Sites One and Four rewarded quickly, typically within 30 and 33 days, respectively. Like the point systems, the reward frequency is driven by a number of factors, including the site-designated reward threshold (that is, number of points needed to earn a reward, which varied from 10 points for the Sites Three (reentry court) and 125 points for Site Two), probationer compliance with target behaviors, and the site-specific point systems.

Recidivism outcomes

Crosstabulations (Table 4) were conducted on technical violations and new arrests for JSTEPS probationers and the comparison group for each site. The results of the crosstab of technical violations yielded no statistically significant differences between the overall JSTEPS and comparison groups, or each site individually. For example, at Site 2, both JSTEPS probationers and comparison probationers were likely to receive a technical violation in the one-year time period (3.3% each). At Site Three, 4.8% of JSTEPS probationers received a technical violation, compared to none of the comparison probationers; 6.3% of JSTEPS probationers at Site One received a technical violation in the one-year time period versus none of the comparison probationers. At Site Four, 8.2% of comparison probationers received a technical violation as compared to 4.9% of JSTEPS probationers.

Table 4.

Percentages of technical violations (TV) and new arrests (NA) by site ^a.

	Site 1 (N=32)		Site 2 (N=60)		Site 3 (N=42)		Site 4 (N=122)
Vars.	TV	NA	TV	NA	TV	NA	TV	NA
JSTEPS	6.3	6.3	3.3	--	4.8	--	4.9	2.6
Comparison	--	6.3	3.3	--	--	--	8.2	3.3

^aNo statistically significant differences exist between JSTEPS and comparison groups on technical violations and new arrests.

The analysis of new arrests indicates no bivariate statistically significant differences. For example, Site One had an equal distribution of 6.3% new arrests for the JSTEPS and comparison probationers. However, Site Four had a slightly higher percentage of comparison probationers with new arrests, 3.3%, compared to 2.6% for JSTEPS probationers.

Table 5 presents the results from the Kaplan-Meier survival analysis for the pooled data. Both Sites One and Four rewarded earlier compared to Sites Two and Three. The early rewarding sites (One and Four) experienced a longer mean survival time until recidivism compared to the delayed reward JSTEPS sites (Two and Three). The early rewarding JSTEPS mean survival time was 345.71 days compared to 317.41 days for the delayed JSTEPS group and 327.70 for the comparison group. The log-rank statistic was approaching significance (p=.059) across the three conditions which suggests that the rewarding early trends in the right direction. Figure 2 shows a visual representation of Table 5.

Table 5.

Kaplan-Meier survival analysis of time to recidivism (in days).

New Arrest/Technical Violation	Mean Survival Time (days)	Standard Error	95% Confidence Interval	Model Log-Rank Statistic
Early Reward JSTEPS	345.71	6.18	333.60, 357.83
Delayed Reward JSTEPS	317.41	12.88	292.17, 342.65	5.67 †
Comparison	327.70	7.11	313.76, 341.65

^†p = .059

Figure 2.

Time to technical violation/new arrest in days.

Discussion

The results of this study explore the impact of a CM JSTEPS protocol on probationer outcomes, with attention to the impact of providing for early rewards on recidivism outcomes as compared to delayed or no rewards. JSTEPS represents the use of rewards in justice settings and therefore might change the interaction with probationers and officers and/or judges (depending on who delivers the reward). Using data collected through the Administrative Office of the U. S. Courts (AO), a sample of JSTEPS probationers (n=128) was matched to comparison sample on six matching variables that are related to recidivism. Overall, CM does not appear to have an impact on recidivism. But, the process of providing early rewards in a CM protocol appears to warrant further review. The findings approach significance (p=.059 with a small sample of 128 in each group) which indicates that the timing of the delivering of rewards is worth further study. Positive reinforcements during the early months of probation may be a factor in having an impact on the length of time until a recidivist event occurred. This is significant because it means that officers have a greater period of time to work with probationers and reduce their likelihood of recidivism. This finding is similar to the general finding in CM literature that early rewards yields better results.

JSTEPS (CM in justice settings) requires more frequent interactions with officers which may offer the opportunity to increase the technical violations for failing to comply with probation requirements (not criminal behavior), as is common in many intensive supervision studies (see Taxman, 2002) and the HOPE replication study (see Lattimore, et al., 2016). At Site Two of the JSTEPS sites had more technical violations than the comparison group-- Site Three, 4.8% of JSTEPS probationers received a technical violation compared to none of the comparison probationers and Site One, 6.3% of JSTEPS probationers received a technical violation in the one-year time period versus none of the comparison probationers. The slightly higher levels of technical violations experienced by JSTEPS probationers in these three sites may be due to sites placing more emphasis on sanctions over rewards, which is similar to recent findings from the HOPE replication study (see Lattimore et al., 2016) and other justice studies. This parallels one challenge that scholars have found in the use of rewards that the system has a difficult time mixing sanctions and rewards (Wodhal, Garland, Culhane, & McCarty, 2011; Rudes et al. 2012; Portillo et al., 2014; and, Mowen, Wodahl & Garland, 2018). The ratio of incentives to sanctions is important to monitor to ensure that the system is emphasizing incentives to achieve positive gains in probationer performance.

The analysis of new arrests indicates no statistically significant differences if we merely look at the percentage that are rearrested. But, attention to the length of time to arrest reveals a pattern that suggest that CM may have an impact. Our study found that probationer recidivism outcomes are somewhat lower compared to national trends of federal probationers, (e.g., 13% receive new arrests) (Administrative Office of the Courts 2010). The results of the Kaplan-Meier survival analyses demonstrate that a type of JSTEPS CM procedure that focuses on early rewards might have promise in reducing technical violations and arrests. Overall probationers in the early reward JSTEPS sites took longer to achieve negative outcomes compared to the delayed JSTEPS group or the no reward/comparison group. That is, at Sites One and Four, the opposite results were discovered in the survival analyses: the JSTEPS groups’ mean survival time until recidivating was longer than the comparison groups. In Sites One and Four, rewarding early produced a mean survival time of 345.71 days versus 317.41 days for delayed JSTEPS, Sites Two and Three, and 327.70 days for the comparison group. It appears that rewarding early produces more positive outcomes thus adhering to merely using the tenets of operant conditioning without attention to timing (Griffith et al., 2000; Lussier et al., 2006, Skinner, 1938).

Limitations

This study has several limitations, some are driven by the fact that the primary aim of the study was to examine implementation, not probationer outcomes. A few JSTEPS sites focused on implementation in specialized programs (i.e., problem-solving courts) which means that probationers included in the study were different than standard supervision, as well as the activities of the programming. For example, probationers in problem-solving courts were often subject to more reporting requirements and more intensive supervision than standard probation, thus increasing their likelihood of “getting caught” (Grattet, Petersilia, & Lin, 2008). Additionally, enrollment in these specialized programs indicates these probationers may have greater needs (e.g., serious substance dependency or gang involvement). In some jurisdictions, enrollment in the specialized court is the “last resort” before revocation. Given that the case-controlled match was limited to demographic and criminal justice risk data, it is possible that the probationers in the comparison group may or may not have been from specialized programs. There is no official documentation of participation in such programming. As a result, it may be unlikely that the comparison sample were subject to more reporting requirements or had greater needs. The RPI risk score was used to control for these differences but there may be other unmeasured differences among the sample. The study did not have access to sanction data since this is not typically recorded.

During the initial implementation of JSTEPS, training was limited to two learning sessions, telephone technical assistance (TA), and very limited in-person TA. This limited TA was due to the research team’s desire to provide broad leeway to sites to observe how the implementation would unfold. It can be argued that a result of this hands-off approach is that the criminal justice actors responsible for using the JSTEPS initiative were left with a limited capacity in understanding the use of CM. CM is a complex intervention, and sites may have had limited understanding of its intricacies, as well as limited time to devote to its implementation. The learning sessions were augmented with principles about desired CM practices and booster sessions to enhance their knowledge base of this type of innovation (Fixsen et al., 2005; Taxman et al., 2014).

Finally, CM within a criminal justice agency may not be standalone program as tested in this study. CM can be coupled with evidence-based interventions such as cognitive behavioral therapy for substance abuse (Hartzler, Lash, and Roll 2012). Treating CM as an enhancement to another treatment may strengthen the intervention (Lussier et al., 2006; Griffith et al., 2000).

Conclusion

Overall, this study adds to the understanding of the importance of positive rewards in justice settings, and the value of using a structured system for delivering operate conditioning that focuses on early rewards. Prior findings of the JSTEPS initiative focused primarily on implementation outcomes (Rudes et al., 2012) while this study examines probationer outcomes. This is more akin to an implementation effectiveness study. The JSTEPS study used a mixed methods approach to examine the design, uptake, and utilization of JSTEPS. Since each JSTEPS site developed and issued rewards to select probationers, using rewards was an acceptable practice to these criminal justice actors. The use of CM can be incorporated into both specialized and standard probation practice. The degree of fidelity to the CM principles of swift and certain responses for targeted behaviors were difficult to achieve due to the extensive nature of probation expectations, yet it was possible. Probation places a bar on individuals and the consequences can be revocation or return to prison. CM tries to reshape behavior, but it is difficult to deliver CM when there are a vast number of target behaviors of interest. CM is known to work better with limited behaviors such as abstinence, treatment attendance, housing stability and so on. Probation supervision has vast expectations in various areas which makes it both difficult for offers to use a CM protocol and also for CM to deliver the same impact when there are often conflicting behaviors. Nevertheless, this study shows promise from CM when it is delivered swiftly to respond to positive actions by probationers. This study advances our understanding of graduated or administrative sanctions or deterrence-based models by illustrating the value of early rewarding. More research is needed about the use of rewards and incentives in probation settings including prioritizing expected behavioral outcomes and testing CM delivery mechanisms such as the proportion of sanctions to rewards, the type of rewards used in different color zones (yellow and green). And, how these different delivery mechanisms affect the therapeutic alliance, which is important to client experiences on supervision (Ricciardelli, 2018). This study adds to a growing body of literature on the importance of positive environments to achieve successful outcomes from probation.