Promotive, Mixed, and Risk Effects of Individual Items Comprising the SAPROF Assessment Tool With Justice-Involved Youth

Abstract

The Structured Assessment of PROtective Factors for violence risk (SAPROF) is a widely used structured professional judgment (SPJ) tool. Its indices have predictive validity regarding desistance from future violence in adult correctional/forensic psychiatric populations. Although not intended for applied use with youth, SAPROF items lend themselves to an investigation of whether their operationalizations capture only strengths or also risks. With 229 justice-involved male adolescents followed for a fixed 3-year period, promotive, risk, and mixed effects were found. Most SAPROF items exerted a mixed effect, being associated with higher and lower likelihoods of violent and any reoffending at opposite ends of their trichotomous ratings. Summing items weighted using their promotive and risk odds ratios produced statistically significant improvements in predictive accuracy, improvements found also with a cross-validation sample of 171 justice-involved youth. The nature of strengths and implications for the development of SPJ tools and training in their use were discussed.

The past decade and more has seen the development of a growing number of assessment tools that include or focus exclusively on what has been widely but imprecisely termed protective factors in forensic mental health/correctional/youth-justice practice (de Ruiter & Nicholls, 2011; Langton, 2020; Langton et al., 2022; Langton & Worling, 2015), with notable examples representing what is known as the structured professional judgment (SPJ) approach (Douglas, 2019). Among these strength-based SPJ tools, various indices from the Structured Assessment of PROtective Factors (SAPROF; de Vogel et al., 2012; i.e., its summed total score, summed scores for its rationally grouped item sets of Internal, Motivational, and External factors, and summary judgments) have been shown to predict the absence of future violence in samples from a variety of adult populations, including samples with histories of sexual offending (cf. Coupland & Olver, 2020; de Vries Robbé, de Vogel, Douglas, & Nijman, 2015; de Vries Robbé, de Vogel, Koster, & Bogaerts, 2015; de Vries Robbé et al., 2016; de Vries Robbé et al., 2021; Haines et al., 2018; Neil et al., 2020; Yoon et al., 2018).

But interest in the utility of the SAPROF with justice-involved youth has been limited, likely due in large part to the availability since 2015 of a version for use with youth, the Structured Assessment of PROtective Factors—Youth Version (SAPROF-YV; de Vries Robbé, Geers, et al., 2015). This, despite the possibility that further investigation might be illuminating, for example, in terms of identifying constructs (or at least SPJ item-specific operationalizations of constructs) that predict outcomes across developmental periods (distinct from those that are specific to a developmental period). More importantly in a basic sense, investigation of any SPJ item-specific operationalizations of constructs afford a means of advancing understanding of what is meant by (or can be demonstrated for individual items purported to be) protective factors or “strengths.” In this study, this is attempted with a sample of justice-involved youth using Farrington et al.’s (2016) approach and their rules of thumb for describing variables (odds ratios [OR] ≥ 1.7 and percentage point differences ≥ 10 from the base rate). The inclusive term “strength” will be used hereafter to refer to variables that exert one or more types of effect (e.g., a main effect or, in conjunction with another variable, an additive, multiplicative effect, or moderating effect) that lower the probability of (re-)offending (Langton et al., 2022).

Among the few studies to have investigated the predictive accuracy of the SAPROF with justice-involved youth, the findings have been mixed (cf. Klein et al., 2015, Langton et al., in press; Zeng et al., 2015; both of which involved samples of adolescents with sexual offenses). Rather than focusing on predictive accuracy, by calculating the area under the curve (AUC) of the receiver–operating characteristic (ROC; Mossman, 1994; Swets et al., 2000) or Harrell’s C (when follow-up time is unequal within the sample; Hanson, 2022; Harrell, 2015), the primary objective of the current study was to investigate the individual items of the SAPROF using the approach described by Farrington et al. (2016; see also Farrington et al., 2008; Farrington & Ttofi, 2011). Farrington et al. (2016) focused instead on exploring the nature of the association between different score ranges on variables for the Cambridge Study in Delinquent Development (CSDD) at-risk boys aged 8 to 10 years old and criminal convictions by the time they had reached 18 years of age. Specifically, Farrington et al. divided focal variables’ scores into quartiles (a top quartile, middle half, and bottom quartile) and used OR, with an OR≥1.7 considered “substantial” (p. 66) and an increase or decrease in percentage points ≥10 from the base rate of convictions (p. 66) at age 18 to determine whether each variable was a risk factor, promotive factor, or a mixed factor, with all three types found in their dataset.

Following Farrington et al. (2016), to label a variable a risk factor, the risk OR (calculated using those convicted/not convicted in the quartile of scores representing the “worst” or most problematic range on a variable and those convicted/not convicted in the rest of the range on that variable, the “rest”) would be ≥1.7. The promotive OR (calculated using those convicted/not convicted in the quartile of scores representing the opposite quartile or “best” range on a variable and those convicted/not convicted in rest of the range on that variable, the “rest”) would be < 1.7. Also, the percentage of the subset with the “worst” quartile of scores who had a criminal conviction by the age of 18 would be ≥10 percentage points above the base rate for the sample, but the percentage of the subset with the “best” quartile of scores who had a criminal conviction by the age of 18 would be <10 percentage points below the base rate. For a promotive factor, the promotive OR (calculated using those convicted/not convicted in the “best” quartile of scores on a variable and those convicted/not convicted in the “rest”) would be ≥1.7 but the risk OR (calculated using those convicted/not convicted in the “worst” quartile of scores on a variable and those convicted/not convicted in the “rest”) would be <1.7. As well, the percentage of the subset with the “best” quartile of scores who had a criminal conviction by the age of 18 would be ≥10 percentage points below the base rate for the sample but the percentage of the subset with the “worst” quartile of scores who had a criminal conviction by the age of 18 would be <10 percentage points above the base rate. For a mixed factor, both the risk OR and the promotive OR would be ≥1.7 and the percentages of the subset with the “best” and “worst” quartiles of scores who had a criminal conviction by the age of 18 would be ≥10 percentage points below and above, respectively, the base rate for the sample.

With this approach, it is possible to determine whether the main effect of a specific operationalization of a variable confers a risk effect (making an adverse outcome more likely, and indicating a unipolar operationalization of a construct), a promotive effect (making an adverse event less likely, and also indicating a unipolar operationalization), or a mixed effect (making the adverse outcome more or less likely depending on which of two poles on the variable the score falls, essentially a bipolar operationalization of a construct even if not intended as such). Yet there has been a relative neglect of the approach of Farrington and his colleagues (2008, 2016), Farrington and Ttofi (2011), in investigating purported strengths associated with the absence of new offending in recidivism-desistance prediction research with applied assessment tools. This is curious, given the aforementioned lack of consensus in the literature about what is meant by key terms, among them protective factor and strength (Langton et al., 2022). The primary objective of the current study was to undertake such an investigation of the SAPROF items. The trichotomous coding of the variables explicitly operationalized in the SAPROF and other SPJ assessment tools certainly lends itself to investigations of the sort advanced by Farrington et al. (2016).

One notable example of such work is Li et al.’s (2019) investigation of the nature of items comprising the SAPROF-YV. For an outcome of probation noncompletion (rather than new offending), Li et al. calculated two ORs for each of the 13 items. To produce a promotive OR, they used a “promotive category” (score of 2 on the item, denoting the presence of the strength) and an “intermediate category” (score of 1 of the item, denoting the possible or partial presence of the strength). To produce a risk OR, they used a “hazard category” (score of 0, denoting the absence of the strength) and an “intermediate category” (score of 1). This follows the approach Farrington and Ttofi (2011) took calculating a pair of ORs (promotive and risk ORs) for each of the CSDD variables using the “best” quartile scores on each variable and the middle half of scores (promotive OR), and the “worst” quartile of scores on the variable and the middle half of scores (risk OR). But it contrasts with Farrington et al.’s (2016) approach of using the “best” quartile versus the “rest” for their promotive OR, and the “worst” quartile versus the “rest” for their risk OR. Li et al. interpreted their findings primarily on the basis of statistical significance of the ORs, advising caution about their classifications for six of the 13 SAPROF-YV items due to a cell count ≤1. Despite all the items being intended by the developers to be protective factors (strengths), Li et al. labeled three as “hazard factors,” seven as “mixed protective factors,” two as having a “weak protective effect,” and only one as a “promotive factor.”

A secondary objective of the current study was to determine whether the use of the SAPROF neglects relevant information (i.e, potential risk) when its items are combined in a mechanical manner (i.e., summed; see Sawyer, 1966, for a general discussion of types of data collection and ways of combining information in prediction methods). This would be shown if any SAPROF items coded as 0 (which ostensibly indicates that a purported strength is absent) actually confer a risk effect. That risk effect would be ignored in a simple summing of the items. More broadly, it would raise the question of what an assessor using the SPJ approach is making of items coded as 0 as they undertake the part of the assessment process that is more opaque: reaching (here, with the SAPROF), first a final protection judgment (of low, moderate, or high) and then an integrative final risk judgment (of low, moderate, or high). It would also raise the question of what an assessor using an SPJ approach is making of items coded as 0 as they identify targets for treatment and make recommendations for intervention planning.

To be clear, the developers of the SAPROF advocate use of the tool in conjunction with an SPJ risk assessment tool, the Historical, Clinical Risk–20 (HCR-20; Douglas et al., 2013) or a related tool, rather than rely on the use of the SAPROF alone. They advise both of these SPJ tools to be used to reach final judgments rather than a simple summing of item ratings (de Vogel et al., 2012, p. 27; although the developers and other research groups have reported indices for summed totals too). Rather than representing a critique, investigations, such as the current study, of SPJ items that can be shown to exert risk (or promotive) or mixed effects instead of their expected promotive (or risk) effect can inform a richer conceptualization of variables comprising these tools and their specific operationalizations of the underlying constructs. This a necessary next step before researchers can address the questions about what SPJ assessors are making of items coded as 0 when, rather than representing the absence of strength (for a tool such as the SAPROF) or the absence of risk (for a tool such as the HCR-20), the 0 for some items confers the converse effect (i.e., a risk effect for a purported strength item or a strength effect for a purported risk item).

In the present study, exploratory analyses were undertaken to determine whether the individual SAPROF items would exert promotive, risk, or mixed effects, using Farrington et al.’s (2016) rules of thumb of ORs ≥1.7 and new offense rates ≥10 percentage points different from base rates. Also, it was hypothesized that in the prediction of new offenses, the AUCs for the summation of SAPROF items weighted according to their promotive ORs and any risk ORs found would be statistically significantly higher than the summation of SAPROF item sets using their 0, 1, or 2 ratings (because the latter necessarily excludes consideration of the possible presence of risk effects) in the subset of the sample used to generate the ORs and also in a cross-validation subset of the sample.

Method

Procedure

Research ethics clearance was secured from the first author’s institutional affiliations and permissions obtained from the relevant Ministries. Details reported below include how the sample size was determined, all data exclusions, all manipulations, and all measures in the study. The archived case files for the sample were accessed; these contained all available assessment reports written by professionals involved in each case as well as school and police/court documentation. Almost all cases had only one comprehensive mental health-and-risk assessment report completed with the youth and all were in the community at the time of that assessment. It was the date of that report that was used as the start of the at-risk period. Only those materials on file before the beginning of the follow-up period at risk for re-offense were included in the version of each case file prepared for coding. No information about recidivism outcomes was contained in these files. In cases in which there were discrepancies between, or changes in, professionals’ opinions about risks and strengths in the materials, such discrepancies were resolved by consensus with the first and third authors, with precedence given to the final assessment report if there was more than one report before the beginning of the follow-up period.

Participants

Beginning with 617 adolescents referred for specialized services (for youth who had sexually abused others) in a major urban area in Southern Ontario from the late 1980s up to 2014, the small number of female adolescents (n = 15) precluded meaningful analyses with this subset, so these youth were removed from the sample. Also removed were 105 youth for whom the responsible ministry was unable to locate the archived case files or for which the archived case files provided were found upon audit to contain insufficient information to code the majority of the study variables. This left 497 male adolescents, aged 12 to 18.99 years old, with a documented sexual offense committed before the age of 18 for which an archived case file was available for coding.

For the present study, the sample was divided into two subsets using a release/at-risk date of April 2003; those at risk from the late 1980s up to April 2003 (n = 174) and those at risk from this date forward (n = 323). In April 2003 the Youth Criminal Justice Act (2002) came into effect in Canada. Although it was not possible to code changes in the processing of cases in the sample resulting from this legislation, the simple division of the sample into two subsets afforded a means to control in a broad and basic manner for differences within the sample arising from the impact of the change in legislation.

Of the subset of 323 male adolescents between the ages of 12 and 18.99 at risk from April 2003 onward, only those for whom the SAPROF could be coded and for whom a fixed follow-up of 3 years was available were used, resulting in an n of 229 for the analyses that follow. Following Harris et al. (2003), the use of this 3-year fixed follow-up period meant adolescents who did not re-offend but did not have 3 full years of opportunity to do so were removed from these analyses, and adolescents who did re-offend but did so after the 3-year point in their follow-up were reclassified as nonrecidivists for these analyses; this provided a snapshot of re-offense status for each youth at their own 3-year point in the follow-up.

Of these 229, 8% had one or more prior convictions for a violent (nonsexual) offense, 1% had two or more prior convictions for a sexual offense, 13% had one prior sexual offense conviction, and 86% had no prior sexual offense conviction. Just under 24% had five or more prior acts of nonviolent offending, 43% had one to five, and just under 34% had none. The mean age of this subset at the start of the follow-up period (i.e., the date the final assessment was completed or the date of release, whichever was later) was 16.03 years old (SD = 1.51 years; range = 12.30–18.90 years old). Data on ethnic origins were available for 95 of these 229 participants; using Statistics Canada categories, 36% of these were of European origin, 22% were of Caribbean origin, 13% were of African origin (Central, North, South, East, West), 14% were of Asian origin (South, East and South East, West Central and Middle Eastern), 7% were of First Nations, Indigenous, Inuit, or Métis origin, and 8% were of Latin, Central and South American origin. Ethnicity and other diversity issues are not further investigated in this study but they are examined with this sample in a separate study.

Of the subset of 174 male adolescents between the ages of 12 and 19 at risk before April 2003, only those for whom the SAPROF could be coded and for whom a fixed follow-up of 3 years was available were used, resulting in an n of 171. The ROC analyses described below involved these 171. Of these, 12% had one or more prior convictions for a violent (nonsexual) offense, 4% had two or more prior convictions for a sexual offense, 19% had one prior sexual offense conviction, and 77% had none. Just under 21% had five or more prior acts of nonviolent offending, just under 50% had one to five, and 29% had none. The mean age of this subset at the start of the follow-up period was 15.73 years old (SD = 1.32 years; range = 12.04–18.88 years old). Data on ethnic origins were available for 85 participants; using Statistics Canada categories, 47% of these were of European origin, 25% were of Caribbean origin, 9% were of Latin, Central and South American origin, 8% were of African origin (Central, North, South, East, and West), 7% were of First Nations, Indigenous, Inuit, or Métis origin, and 4% were of Asian origin (South, East and South East, West Central, and Middle Eastern).

Measures

SAPROF

SAPROF items were coded as per the manual (de Vogel et al., 2012), without knowledge of recidivism outcomes, and used in the logistic regression analyses (see Data Analytic Strategy, below). This was undertaken as part of a larger study of the comparative predictive validity of tools with justice-involved adolescents (Langton et al., in press). In that study, which included consideration of developmental issues, the SAPROF was included as one of the tools developed for use with adults. All items were coded for the 229 youth or 228 of them (for some items the information contained in some of the case files was insufficient to score a single item); the one exception was the Intelligence item for which information was available to score the item for 142 youth.

To check interrater reliability, intraclass correlation coefficients (ICCs) were calculated using a subset of 23 participants’ cases, coded independently by three raters (see Procedure, above). As reported in Langton et al. (in press), the ICC for the summed total for the SAPROF was .77, which falls in the range described as “excellent” by Cicchetti (1994). ICCs for the individual items fell in the ranges described as “fair” and “good” by Cicchetti, with one falling in the “excellent” range.

Outcomes

Four official sources of information were used to generate as comprehensive a measure of official recidivism as possible: The Canadian Police Information Centre records, a national database of criminal convictions provided by the Royal Canadian Mounted Police; data from the youth and adult offender tracking information systems provided by the Ontario Ministry of Community Safety and Correctional Services; and case files provided by the Ontario Ministry of Children and Youth Services. Outcomes were dichotomously coded. New offenses were coded if documented in the follow-up period as convictions in any of the first three sources or an officially confirmed new incident in the fourth source. A new violent (including sexual) offense (and the absence of) was used as the dependent variable because it is the outcome for which the SAPROF was intended to structure applied assessment work. Findings with a second outcome, any new offense (and the absence of) are also reported (data are given in Supplemental Tables 1 and 2) because an inclusive outcome of this kind is of comparative interest given its inclusion in many investigations in the recidivism-desistance prediction research with applied assessment practices.

The percentage agreement between pairs of the four sources for a new violent offense was between 81% and 88%, indicating that the outcome for as many as one in five cases was inconsistent between sources for this category of offending. The percentage agreement between pairs of the four sources for any new offenses was between 75% and 85%, indicating that the outcome for as many as one in four cases was inconsistent between sources for this category of offending. These percentage agreements confirmed the importance of using multiple sources to detect all officially recorded new offenses.

Data Analytic Strategy

For the exploratory analyses, logistic regression analyses were run using the subset of 229 (those at risk from April 2003 onward) to calculate risk ORs (worst score, 0, vs. scores of 1 or 2) and promotive ORs (best score, 2, vs. scores of 0 or 1) for each of the SAPROF items for both outcomes. ROC analyses were carried out to test the hypothesis concerning predictive accuracy, six indices of strength were generated by summing sets of SAPROF items. The first was a summed total for all coded items excluding Items 6 (Work) and 8 (Financial Management) because their operationalizations in the manual were deemed to be developmentally inapplicable to an adolescent sample, and Item 12 (Medication) because this item was coded as not applicable, per the manual, for almost all participants; this represented a modified SAPROF total. In a separate study evaluating the predictive accuracy of various SAPROF totals using these data (Langton et al., in press), the summed total of the External factor items did not predict either new offense outcome used in this study so a second modified SAPROF total with the External factor items omitted evaluated in that study was also used for this study when comparing AUCs. The third and fourth indices were the sums of subsets of the items, now selected and weighted based on their ORs (described below). The fifth and sixth indices used the same subsets of items used in the OR-weighted totals but were the sums of those items as originally coded (0, 1, or 2).

For the weighted summations (the third and fourth indices), only those items for which the risk OR and/or the promotive OR was ≥1.7 for each outcome separately, calculated using the subset of the sample at risk from April 2003 onwards, were included; this produced two slightly different nine-item scales, one intended to predict each outcome. The predictive validity of these two scales was tested by calculating their AUCs for the two subsets of the sample separately, with the subset at risk before April 2003 providing an indication of the generalizability of the weightings made using the ORs for the subset at risk from April 2003 onwards.

Results

Among the 229 adolescents at risk from April 2003 onward, 30% committed a new offense of any kind and 19% of 224 of these adolescents committed a new violent (including sexual) offense in the fixed 3-year follow-up period. The number of adolescents decreases slightly from the more inclusive outcome to the less inclusive outcome because, for some adolescents, their conviction, counted in the more inclusive any new offense category (e.g., a conviction for a Break-and-Enter), resulted in less than 3 years of time-at-risk and therefore their exclusion (because of time spent back in custody, per a custodial sentence for the Break-and-Enter) from analyses with the less inclusive new violent (including sexual) offense category.

Among the 171 adolescents at risk from April 2003 onward, 27% committed a new violent (including sexual) offense in the fixed 3-year follow-up period and 34% of 169 of these adolescents committed a new offense of any kind. The number of adolescents decreases slightly for the more inclusive outcome because discrepancies in the files for two youths who committed a new nonviolent offense made it impossible to determine whether those offenses had occurred within the fixed 3-year follow-up; so these two youths were excluded from the analyses with this outcome. But neither of these youths committed a new violent (including sexual) offense at any point in their follow-up and so were included in the analyses for that outcome.

Promotive, Risk, and Mixed Effects

A New Violent Offense

The percentage of participants with a new violent (including sexual) offense for each score on the items is given along with the promotive and risk ORs for each item in Table 1. Only two items, Empathy and Self-Control, could be construed as having a promotive effect based on their pairs of ORs. The promotive OR for Empathy was 1.72, p = .401, and its risk OR was 1.63, p = .200. The recidivism rates among those with the lowest score on this item (denoting the absence of this strength) and those with the highest score on this item (denoting the presence of this strength) only differed from the base rate of 19% by ±6. Taken together, these data could be described as a mixed effect but the Cochran–Armitage linear trend test was not statistically significant: χ² = 1.999, 1 df, p = .157. Also, the Self-Control item could be described as having a promotive effect, with a promotive OR of 3.27, p = .260, and a risk OR of 1.66, p = .140, and recidivism rates for those with scores of 0, 1, or 2 of 23%, 17%, and 7%, respectively. A case for a mixed effect could be made here based on the risk OR of 1.66, although, again, the Cochran–Armitage linear trend test was not statistically significant: χ² = 2.877, 1 df, p = .089.

Table 1.

Percent Reoffended and Odds Ratios for SAPROF Items for a New Violent (Including Sexual) Offense Using a 3-Year Fixed Follow-Up

SAPROF item				Odds ratios
	Percent reoffended				95% CI for Promotive OR			95% CI for Risk OR
	Worst Score	Middle Score	Best Score	Prom OR	LL	UL	Risk OR	LL	UL	Effect
Internal factors
1. Intelligence	26	12	23	0.74	0.19	2.92	2.21	0.93	5.25	Risk
2. Secure Attachment in Childhood	23	17	16	1.25	0.56	2.81	1.44	0.73	2.85	—
3. Empathy	25	18	13	1.72	0.49	6.05	1.63	0.77	3.45	Promotive
4. Coping	25	11	0 a	2.73	0.15	50.25	2.98**	1.31	6.80	Mixed
5. Self-control	23	17	7	3.27	0.42	25.70	1.66	0.85	3.25	Promotive
Motivational factors
7. Leisure activities	21	19	15	1.48	0.58	3.78	1.24	0.63	2.44	—
9. Motivation for treatment	27	20	9	2.97*	1.11	7.99	2.01	1.00	4.06	Mixed
10. Attitudes toward authority	27	23	10	3.20**	1.40	7.28	1.98*	1.00	3.89	Mixed
11. Life goals	18	20	17	1.14	0.37	3.56	0.88	0.43	1.81	—
External factors
13. Social network	31	19	14	1.74	0.78	3.86	2.15	0.96	4.80	Mixed
14. Intimate relationship	19	36	0 a	1.25	0.06	26.59	0.51	0.17	1.55	—
15. Professional care	16	24	21	0.88	0.43	1.82	0.65	0.33	1.30	—
16. Living circumstances	0 a	18	25	0.61	0.30	1.23	0.31	0.02	5.62	Paradoxical risk
17. External control	19	15	29	0.48*	0.23	0.99	0.98	0.46	2.10	Paradoxical risk

Note. Base rate for a new violent (including sexual) offense was 19%. Promotive ORs calculated using the worst and middle scores combined as the reference category. Risk ORs calculated using the best and middle scores combined as the reference category. SAPROF = Structured Assessment of PROtective Factors; OR = odds ratio. CI = confidence interval; LL = lower limit; UL = upper limit.

^aNo youth with this score reoffended so 0.5 was added to each cell in the relevant contingency table to calculate the OR.

^*p < .05. **p < .01. ***p < .001.

A single item, Intelligence, could be described as exerting a risk effect, with a promotive OR of 0.74, p = .672 and a risk OR of 2.21, p = .071. But the recidivism rate for those with scores of 0, 1, or 2 was 26%, 12%, and 23%, respectively, consistent with some elevation in risk for both those without this strength and those with this strength, making it harder to categorize the effect.

The item External Control exhibited a paradoxical risk effect, with a promotive OR of 0.48, p = .047, and a risk OR of 0.98, p = .959. Those with this strength present (score of 2) were 2.08 times more likely to reoffend violently than those with a score of 0 or 1. The recidivism rate for those with scores of 0, 1, or 2 was 19%, 15%, and 29%, respectively, reflecting the elevation in risk for those with this strength present. The item Living Circumstances similarly could be described as exerting a paradoxical risk effect. No youth with a score of 0 on this item reoffended so the Haldane-Anscombe correction was applied, adding .5 to each cell in the contingency table, to generate a risk OR; although neither OR was statistically significant, those with this strength present (score of 2) were 1.64 times more likely to reoffend than those with a score of 0 or 1 and those with this strength absent (score of 0) were 3.22 times less likely to reoffend than those with a score of 1 or 2. The recidivism rate for those with scores of 0, 1, or 2 was 0%, 18%, and 25%, respectively, reflecting the unexpected elevation and decrease in risk for the scores associated with strength and risk, respectively, on this item. A paradoxical mixed effect description was not quite justified on the basis of the promotive OR and a nonstatistically significant Cochran–Armitage linear trend test: χ² = 2.776, 1 df, p = .096.

Five items (Secure Attachment in Childhood, Leisure Activities, Life Goals, Intimate Relationship, and Professional Care) exerted no clear effect. The remaining four items could be described as having a mixed effect (Coping, Motivation for Treatment, Attitudes Towards Authority, and Social Network). For example, Motivation for Treatment had a promotive OR of 2.97, p = .031, and a risk OR of 2.01, p = .052, with recidivism rates for those with scores of 0, 1, or 2 of 27%, 20%, and 9%, respectively, and statistically significant Cochran–Armitage linear trend test: χ² = 6.473, 1 df, p = .011).

Any New Offense

Findings were broadly similar but not identical for this outcome. Effects were clear for 10 of the 14 items. Based on pairs of ORs and changes from the base rate of 30%, two items exerted a promotive effect, two a risk effect, four could be described as having a mixed effect, and the same two items had a paradoxical risk effect (see Figure 1 and Supplemental Table 1).

Figure 1.

Percentage With Any New Offense for SAPROF Items Conferring Four Types of Effect.

Note. Base rate for any new offense in the sample was 30%. SAFPROF = Structured Assessment of PROtective Factors for violence risk.

Predictive Accuracy of Original and Weighted Summations of SAPROF Items

The finding that some SAPROF items exerted a risk or mixed effect prompted testing of the hypothesis concerning the AUCs for the summation of SAPROF items selected and weighted according to promotive and risk ORs ≥1.7.

A New Violent Offense

For this outcome, the items included were weighted as follows: For the item Intelligence, 0 was recoded as −1 (indicating risk), 1 was recoded as 0, and 2 was recoded as 0 (indicating no strength). Empathy ratings were recoded as 0 = 0 (no risk), 1 = 0, and 2 = 1 (strength). Items 4, 5, 9, 10, and 13 were recoded as 0 = −1 (risk), 1 = 0, and 2 = 1 (strength). Items 16 and 17 were recoded as 0 = 0 (no risk), 1 = 0, and 2 = −1 (indicating a paradoxical effect for the presence of this strength). This 9-item scale is hereafter referred to as the “OR-weighted SAPROF violent total.”

With the subset at risk from April 2003 onward, the AUC for the OR-weighted SAPROF violent total (.71, p < .001, 95% confidence interval [CI]: [.62, .79]) was not statistically significantly higher than the AUCs for a simple summing of the same nine items (.64, p = .005, 95% CI [.54, .73]): z = 1.55, p = .061 (1-tailed) or the modified SAPROF total (without the External factors items; .65, p < .001, 95% CI [.57, .74]): z = 1.45, p = .074 (1-tailed) but it was statistically significantly higher than the AUC for the modified SAPROF total (.60, p = .035, 95% CI [.51, .70]): z = 2.08, p = .019 (1-tailed). (The latter two AUCs were previously reported in Langton et al., in press.)

With the subset at risk before April 2003, the AUC for the OR-weighted SAPROF violent total (.66, p = .001, 95% CI [.57, .75]) was statistically significantly higher than those for a simple summing of the same nine items (.59, p = .058, 95% CI [.50, .69]): z = 1.66, p = .048 (1-tailed), the modified SAPROF total (.57, p = .135, 95% CI [.48, .66]): z = 1.95, p = .025 (1-tailed), and the modified SAPROF total (without the External factors items; .60, p = .034, 95% CI [.51, .69]): z = 1.99, p = .023 (1-tailed).

Any New Offense

The AUC for the OR-weighted SAPROF any total was statistically significantly higher than those for five of the six comparisons with the various SAPROF indices (see Supplemental Table 2).

Post Hoc Analyses

The paradoxical risk effect for two items (Living Circumstances and External Control) prompted tests of whether youth with scores of 2 (indicating strength) on these items were at high(er) risk of re-offense than their lower scoring counterparts. The absence or presence of one or more prior charges or convictions for a sexual offense was used as the index of risk. Living Circumstances scores of those without prior charges or convictions for a sexual offense (X = 1.20, Mdn = 1.00) were lower than those with one or more prior charges or convictions for a sexual offense (X = 1.59, Mdn = 2.00). A Mann–Whitney U test indicated that this difference was statistically significant: U(N_{no priors} = 196, N_priors = 32) = 4301.00, z = 4.18, p < .001. External Control scores of those without prior charges or convictions for a sexual offense (X = 0.89, Mdn = 1.00) were lower than those with one or more prior charges or convictions for a sexual offense (X = 1.53, Mdn = 2.00). A Mann–Whitney U test indicated that this difference was statistically significant: U(N_{no priors} = 196, N_priors = 32) = 4,647.50, z = 4.76, p < .001.

Discussion

Despite having been developed for use with adults, the operationalizations of constructs tapped by SAPROF items were shown to be helpful in investigating the nature of strengths with a sample of justice-involved youth using Farrington et al.’s (2016) approach and their rules of thumb for describing variables (ORs ≥ 1.7 and percentage point differences ≥ 10 from the base rate). Of the 14 items investigated, only three exerted the promotive effect for one or both outcomes that would be expected with being a purported protective factor. The presence of risk effects (as part of a mixed effect or a simple risk effect) for some SAPROF items is incongruent with their construal as protective factors or strengths. But, in fact, the criterion given for the item Intelligence for a score of 0 could be taken to reflect risk (“Intelligence below average”de Vogel et al., 2012, p. 41), and the presence of risk could be inferred for other item scores of 0 that explicitly only indicate the absence of the protective factor (about which further comments are offered below). Most items’ effects across the two outcomes were consistent although variables’ effects have been shown to vary according to outcome (Langton et al., 2022). Illustrating the added value of information represented by some item scores of 0 concerning risk effects, two OR-weighted 9-item summed SAPROF totals were shown to be statistically significantly more accurate in predicting outcomes than some of the original SAPROF summed totals.

Effect Sizes and Comparison Groups

It is worth noting that if, instead of an OR cutoff of 1.7 or greater, ≥ 1.4 is used, in the present study the effects of some items could be reclassified as having a promotive effect (e.g., Leisure Activities, for the outcome of a new violent offense), a risk effect (e.g., Secure Attachment in Childhood, for a new violent offense), or a mixed effect (e.g., Intelligence, for any new offense). Farrington et al. (2016) described their OR cutoff of 1.7 as “substantial” (p. 66) and cited Cohen (1996) who noted that “Although conventions with regard to OR as effect sizes are not well established, the field of epidemiology tends to regard OR of 2.0 or more as fairly large” (p. 136). Also focused on epidemiological studies, Chen et al. (2010) determined that an OR of 1.68 is equivalent to a Cohen’s d effect size of 0.20 (a small effect size) when the rate of the outcome of interest is 1% in the nonexposed group but that lower ORs correspond to a Cohen’s d of 0.20 as the rate of the outcome of interest in the nonexposed group increases; so an OR of 1.46 is equivalent to a Cohen’s d of 0.20 when that rate is 10%. ORs of 1.44, 2.48, and 4.27 correspond to Cohen’s d effect sizes of 0.20 (small), 0.50 (medium), and 0.80 (large) using Lenhard and Lenhard (2016).

As well, some of the effects of SAPROF items would be reclassified if the promotive ORs were calculated by comparing the new offense rate for those with the “best” score (2, indicating the presence of strength) with the rate for those with the “middle” score (1, indicating the strength may be present or present to some extent) and risk ORs were calculated by comparing the new offense rate for those with the “worst” score (0, indicating the absence of strength) again with the “middle” score. For example, for any new offense, Coping would be reclassified from having a mixed effect to a risk effect; and both Motivation for Treatment and Attitude toward Authority would be reclassified from having a mixed effect to a promotive effect; for a new violent offense, Self-Control, Motivation for Treatment, and Attitude toward Authority would all be reclassified from having a mixed effect to a promotive effect. Social Network would be reclassified from having a mixed effect to a risk effect. Note that External Control and Living Circumstances would both remain classified as exerting a paradoxical risk effect for both outcomes (about which further consideration follows below).

The Nature of Strengths

Whether changing the OR cutoff or comparing rates for “best” or “worst” scores with the rate of the “rest” or an intermediate/middle score, the intuitive expectation that SAPROF items would all exert a promotive effect was not consistent with the data. SAPROF items that were shown to have a mixed effect (e.g., the Motivation for Treatment item and the Attitudes Toward Authority item for both outcomes) confirmed that the operationalization of these variables intended to tap strength in their domains did indeed capture strength at one end of the range (scores of 2). But it also showed that the absence of this strength (scores of 0) either conferred a risk effect or that individuals with a score of 0 included those without this strength, conferring no effect, and those with risk in the same or a conceptually related domain, conferring the detected risk effect. Consider, if the focal strength, for example, a “prosocial and supportive social network” is not present for an individual (Social Network, item 13), a 0 is given for the item. But as de Vogel et al. (2012) noted in the coding material for this item, “. . . if an individual has network members who have a positive attitude towards crime or antisocial behavior, this may lead to increased risk” (p. 69). In the present sample, those youth who associated with antisocial or criminally-involved peers would have also received a score of 0, simply denoting the absence of strength, although the risk would also be present (and evident per the risk ORs for both outcomes). This is consistent with findings that peer influence is a robust effect across a range of behaviors (Giletta et al., 2021) and can have independent risk and strength effects on reoffending (Mowen & Boman, 2018), although the associations are likely complex (Yim, 2021).

With variables intended as unidimensional operationalizations of strengths, such as those items comprising the SAPROF, it is not possible to determine if it is the absence of the strength, as operationalized, that confers the risk effect, or if the score of 0 actually captures the presence of risk in the same or a conceptually related domain, as discussed above. The same question arises with some unidimensional operationalizations of risk (i.e., does the absence of a specific risk, a score of 0 on a risk item, confer a strength effect or merely denote its absence with no effect). With bipolar operationalizations of constructs as items in other tools (e.g., Barnoski, 2004; Viljoen et al., 2014; Worling, 2017), this important question could be addressed, but that research has yet to be undertaken.

The lack of clarity here is of particular interest because, as Langton et al. (2022) have discussed, there remains in the field a lack of consensus about how strengths should be understood. Consider just three of the various ways strengths have been discussed in the literature. One is that strength is simply the absence of risk (Baird, 2009; Harris & Rice, 2015). This view cannot be investigated with SAPROF data because none of this tool’s items are operationalized as purported risk items (for which a 0, indicating the absence of that risk, could be coded). Another view is that strength should be construed as simply the extension of a continuum with risk at the opposite pole (Harris & Rice, 2015), which is a conceptualization compatible with Farrington et al.’s “mixed factors” and is consistent with findings reported by Farrington and his colleagues as well as by Li et al. (2019) and in the current study. A third is that strengths (or at least specific operationalizations of purported strength constructs) are distinct or exert an effect independent from risks (see, for example, Mowen & Boman, 2018), which could be understood as an example of Farrington et al.’s “promotive factors.” There was limited evidence for SPJ items reported in Li et al. and in the current study. With this third view, the question arises as to whether or when a strength is a unidimensional construct for which there are no risk pole, at least not one for which there is a demonstrable effect for a specific outcome (rather than a unidimensional operationalization of a construct for which there would be “complimentary” risk pole, with a demonstrable effect, but that risk pole is simply not part of that specific operationalization). As Langton et al. (2022) demonstrated, there is evidence to suggest these various conceptualizations and others are all relevant in understanding various strength effects. But what does this mean for applied practice with risk-focused and strength-focused SPJ tools?

As noted above, the SAPROF was explicitly intended to be used in applied practice in conjunction with a risk-focused SPJ tool (the HCR-20) and for assessors to reach a “final protection judgment” using the SAPROF and a “integrative final risk judgment” using the SAPROF and HCR-20, rather than emphasize the use of summed total of items or item subsets. So failure to explicitly consider scores of 0 that confer a risk effect (pending replication that some items consistently do that, whether as part of a mixed effect or simply a risk effect as was found in the present sample) is perhaps not problematic in applied practice when risk items are being explicitly considered as part of a separate tool. But are assessors still inferring a strength effect for purported protective factor items actually conferring only a risk effect and what are the implications of doing that?

Indeed, the findings reported here may represent a partial explanation of why the summed total of the SAPROF External factor items failed to predict outcomes in this sample (Langton et al., in press) and in various other studies (Coupland & Olver, 2020; de Vries Robbé et al., 2016; Yoon et al., 2018). Each of the External factor items would be expected to exert a promotive effect. But in the present sample, there were notable exceptions. One of these items, Social Network, conferred a mixed effect (for a new violent offense) and a risk effect (for any new offense), evident based on its promotive and risk ORs. Two additional items, External Control and Living Circumstances, exerted a paradoxical risk effect (scores of 2, indicating the strength was the presence, being associated with a higher likelihood of both outcomes). If replicated, the implications for applying these operationalizations of these constructs in practice are problematic.

Consider, the External Control, Living Circumstances, and Professional Care items might be confounded by risks because those individuals receiving higher levels of external control and professional care and subject to intensively supervised living circumstances would presumably warrant those higher levels of service based on perceived/assessed risk (which would render these items proxies for risk in analyses such as those reported here). Consistent with this, post hoc analyses with these data showed that youth with higher scores on both the External Control and the Living Circumstances items were, indeed, at higher risk (determined using a number of prior sexual offenses). The extra resources/services, reflecting increased professional involvement, tapped by those items would be provided/implemented by case-involved professionals to manage or lower perceived/assessed risk in the case (and such resources/services may well be shown to lower dynamic risk over time with a longitudinal design). Indeed, we would expect high levels of external control to reduce the likelihood of recidivism (facilitate desistance) over time in individuals at higher risk of re-offending (broadly consistent with the empirically supported Risk and Needs Principles of the Risk-Needs-Responsivity model; Andrews et al., 1990; Bonta & Andrews, 2017). But, as just noted, we would also expect that individuals at higher risk of re-offending would be subject to higher levels of external control initially. So, assessed once and at the beginning of the period that an individual is at risk (e.g., when released from custody), External Control might represent risk (as it did in the present sample, exerting a paradoxical risk effect). But it might exert a strength effect based on changes it is shown to bring about in (an index of specified) dynamic risk over time. Without further empirical investigation, however, it will be a challenge for an assessor to know how to incorporate the information provided by items such as this one in an assessment that will accurately inform intervention planning.

The results of the direct statistical comparisons of AUCs in the present study do suggest that mechanical incorporation of information about risk captured in some items’ 0 scores, demonstrated with ORs and changes from base rates, can result in statistically significantly more accurate predictions than a simple summing of items that are all assumed to confer only a strength effect. (Changes from base rates have been used as a basis for the weighting of predictors in recidivism prediction research for decades; Harris et al., 1993; Nuffield, 1982). But the present findings are intended only to be illustrative of the potential value of investigating the nature of specific operationalizations of focal constructs using Farrington et al.’s (2016) approach. They are insufficient, it must be noted, as a basis for any amendment to the developers’ explicit directions in the use of the SAPROF in real-world cases (see de Vogel et al., 2012, pp. 21–29).

Limitations

The study is not without limitations. The study involved male adolescents only and was archival in nature (although coding of independent variables was undertaken without knowledge of recidivism outcomes to partially mitigate concerns over the use of archived case files). All of the adolescents had committed at least one sexual offense whether or not they also had offenses of a nonsexual nature in their criminal history; this might limit the generalizability of the findings to more general samples of justice-involved youth or to distinct offense-defined groups. For the adolescents in the subset at risk from April 2003 onward, the earliest cases received assessment services nearly 20 years ago. For the subset at risk before April 2003, that timeframe began even earlier. As such, the generalizability of these findings may be very limited. A related concern is that the materials in the archived case files would not have been written with the SAPROF items explicitly in mind although many of the constructs would be expected to have been considered by the clinical report writers. The challenges of coding/rating from archived case files is evidenced in part by the range of ICCs for items in this study, which may have lowered some of the effect sizes obtained. As well, the SAPROF was not developed for applied use with adolescents, so the implications of the findings are of conceptual and methodological significance rather than applied utility. As such, again, these findings may be shown in future work with the SAPROF to lack generalizability, whether with adults or with adolescents.

Relatedly, a focus on the SAPROF, all the items for which are unidimensional operationalizations of purported strengths (protective factors), meant it was not possible to distinguish between the effect of the absence of strength and the possible presence of risk in the same or conceptually relevant domain. It was also not possible, given the complete absence of purported risk items (risk factors) in the SAPROF, to distinguish between the effect of the absence of risk and the possible presence of strength in the same or conceptually relevant domain. We have studies underway with different tools to address some of these limitations.

What is clear is that application of Farrington et al.’s (2016) approach represents a systematic line of inquiry that prompts both an explicit consideration of what is meant by “strength” (or labels such as a promotive factor or protective factor) and grounds attempts to clarify in an empirical manner what effect a variable, given its specific operationalization, exerts on a specific outcome in a sample from a specific population. The present study used items comprising an SPJ tool designed for use with adults with a sample of justice-involved youth, which allows some preliminary inferences too about which constructs might exert an effect on reoffending among adolescents although these constructs are operationalized for use with adults (notwithstanding the importance of developmental considerations here as others have observed; see, for example, Langton et al., in press; Ralston & Epperson, 2013; Viljoen et al., 2012). The development and training in the use of SPJ tools may benefit from the research of this kind that demonstrates how their individual items can be understood to work in the assessment of risks and strengths in recidivism-desistance prediction research and in applied assessment and treatment practices.