Evaluation of a Suicide Risk Assessment Tool in a Large Sample of Detained Youth

Abstract

Objectives

We evaluated the six-item Inmate Security Assessment (ISA) tool used among detained youth in Manitoba, Canada.

Method

Two hundred and forty-one recorded self-harm incidents among all incarcerated youth occurred between January 1, 2005 and December 31, 2010 (N=5102). The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (PLR) and negative likelihood ratio (NLR) for three categories of suicide risk (high, medium, and low) as well as each of the six suicide risk evaluation indicators were determined. Receiver operating characteristic (ROC) curves and area under the curve (AUC) calculations for the three suicide risk levels and the six indicators were created.

Results

Having at least a low suicide risk level (93.8%) or at least one suicide risk factor (94.6%) provided high sensitivity. Specificity was high if an individual had at least a medium suicide risk level (94.2%) or at least three suicide risk indicators (96.7%). The PPV was low (8.9–16.2%) and the NPV was high (94.9–99.3%) for all suicide risk levels. The most sensitive risk factor for self-harm was a prior history of suicidal behavior or a family history of suicide (94.6%). All risk indicators had a low PPV (7.4–23.1%) and a high NPV (95.4–99.5%). A very low NLR was found for those without prior suicidal behavior or a family history of suicide (0.107). The AUC was 0.719 (95%CI = 0.692–0.746), indicating a fair test.

Conclusion

The ISA is a moderately accurate tool for identifying risk for self-harm in detained youth.

Introduction

Suicide is the leading cause of death by detained youth worldwide (Gallagher and Dobin, 2006; Hockenberry et al., 2006). In one UK-based study, rates of suicide among detained boys ages 15 to 17 were 18 times higher than age-matched peers in the general population (Fazel et al., 2003). Similarly, suicidal ideation and self-reported suicide attempts tend to be higher among incarcerated adolescents than the general population (Freedenthal et al., 2007; Morris et al., 1995; Centers for Disease Control, 2012). Greater entrenchment in the youth justice system is associated with greater risk as suicidal ideation and attempts are more prevalent among post-adjudicated youth than pre-adjudicated youth (Stokes et al., 2015). Juveniles in the justice system employ more violent means and have a higher rate of death by suicide (Penn et al., 2003).

Multiple factors may explain the increased risk for suicide in detained youth. Similar to community-based peers, youth in the juvenile justice system with a history of depression, sexual abuse, and trauma have higher prevalence of suicidal ideation and self-harm (Chapman and Ford, 2008). In addition, detained youth have increased rates of mental disorders, substance abuse, and history of prior suicide attempts compared to the age-equivalent general population (Wasserman and McReynolds, 2006; Radeloff et al., 2015). Exposure to violence and stressful neighborhood characteristics may also contribute to suicidal behavior among young offenders (Howard et al., 2003). Detained youth who have experienced adversities including sexual abuse, homelessness, running away, and substance abuse, were almost eight times more likely to report having ever attempted suicide than those who did not (Bhatta et al., 2014). Psychological trauma early in life can result in substance use problems, isolation or deviant peer groups, problems with the law, and despair that can lead to risky or self-harming behavior (Ford et al., 2006). The detention environment itself, involving separation from supports, loss of control and perspective, and feelings of shame and guilt may also contribute to a high risk of suicide (Radeloff et al., 2015). Detained youth may be subject to conditions that exacerbate the risk of suicide, such as mechanical restraints, use of restraints for discipline, and locked cells (Gallagher and Dobrin, 2006).

The recognition of youth at risk for self-harm, regardless of intent, is instrumental to preventing young people from untimely death and disability. Self-harm incidents may be followed by acts with greater suicidal intent (Wilkinson et al., 2011). Deliberate self-harm desensitizes people to pain and is associated with emotional distress, which then increases risk for suicidal ideation along with the capability to act on suicidal desire (Klonsky, 2014). The majority of adolescents who self-harm do not reach the attention of medical services (Madge et al., 2008; Kidger et al., 2012) and do not seek help before they harm themselves (Hawton et al., 2002). Early self-harm is common in those who die by suicide in late adolescence or early adulthood (Fortune et al, 2007). The act of coming into custody provides an opportunity to help a literal ‘captive’ audience. Identifying those at elevated risk for self-harm may help reduce suicide in this population. Few tools used to screen for suicide risk have been validated in juvenile justice populations (Stokes et al., 2015). The Suicidal Ideation Questionnaire (Reynolds, 1987) and the Suicidal Behaviors Questionnaire-Revised (SBQ-R) (Cotton et al., 1995) have been applied in youth though neither have been studied extensively in adolescents (National Action Alliance for Suicide Prevention, 2013).

This paper focuses on the Inmate Security Assessment (ISA) tool used at the youth correctional facilities in Manitoba, Canada. This screening measure is used universally within 24 hours of detainment for all youth entering a correctional facility in Manitoba. Screening occurs upon entry into custody, regardless of whether the case involves remand or sentencing. Mental health services may be provided to youth partly based on results of this screening measure. The tool also supplies information that influences the level of observation assigned to the individual youth, which could include constant supervision or five minute observation checks. This tool would categorize youth into 4 categories of suicide risk: no evidence, low, medium, or high risk. Cut-off norms for the risk categories were previously established through a test study of 1000 assessments (e.g., the cut-off level for very high should ensure that no more than five percent of the population receive this score), though this study did not examine the validity or reliability of the tool (Hannah-Moffat & Maurutto, 2003). The instrument could also be repeated several times throughout an individual’s detention period at the discretion of staff.

Our previous study described the demographic and custodial variables associated with self-harm that were found in this same population (Casiano et al., 2016). In that study, we found that being a girl, attaining less than Grade 11 education, being older than 15 years, and having involvement in child welfare were associated with self-harm during incarceration. Custodial factors linked to self-harm included more severe crimes, younger age at first incarceration, longer sentences, disruptive institutional behavior, and attempting escape from custody. Events of self-harm tended to occur earlier in the course of custodial admission, and the bulk of self-harm occurred within 30 days of incarceration. However, new onset self-harm events occurred throughout the custodial stay, with 5.1% of self-harm occurring after one year in detention.

The ISA tool has existed within Manitoba corrections for decades and was designed to help identify at-risk individuals, but as of yet has not been validated. The tool is meant to screen for risk throughout the detention period. The aim of the study included an evaluation of how well the tool identified individuals at risk for self-harm during the custodial period. A major advantage of our study is that it utilized self-harm data obtained by staff input rather than relying on detainee report, which has typically been used in other published studies. Self-report is subject to recall bias and possible over- or under-reporting; as well, the responses of teens carry the risk of being affected by social desirability (Brausch & Gutierrez, 2010).

Method

Sample

Data were obtained from the Manitoba Government Department of Justice Corrections Information System database in Canada. As per the usual protocol in conducting research with governmental information, identifying data for participants was removed. The files of youth aged 12 to 18 years at the time of their offense who were in custody sometime between January 1, 2005 and December 31, 2010 (N = 5,102) were examined. All detainees were included in the study as the ISA was utilized as part of routine care. Participants included those who were in custody on January 1, 2005 along with new cases who were detained throughout the study period. Individuals were followed throughout their custodial stay or until the end date of the study. There was no missing data.

Procedure

A search for keywords within the database of records of daily events and incident reports was conducted including the terms “suicide”, “self-harm”, and “self-injury”. If any of those keywords appeared in the record, the individual was coded as having self-harmed in the detainment period.

The Inmate Security Assessment (ISA) tool employed six questions to determine level of risk for detained youth. Youth were characterized as high, medium, or low suicide risk following the administration of the instrument. There were six indicators, which classified youth into four suicide risk groups: high, medium, low, or no evidence (see Appendix A). The computer software would automatically assign a risk level based on the combination of the six yes/no indicator responses.

Analyses

We followed typical protocols in the evaluation of diagnostic tests (Parfrey et al., 2009; Banoo et al., 2008; Haynes et al., 2006; Weinstein et al., 2005; Van den Bruel et al., 2007). For this study, we included all youth incarcerated between January 1, 2005 and December 31, 2010 (N=5102). If a participant was detained more than once, only the data from their most recent entry into corrections was used. We analyzed the data by calculating the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (PLR) and negative likelihood ratio (NLR) of each of the 3 risk levels (“high”, “medium”, and “low”) as well as for each of the 6 indicators. Sensitivity was defined as the ratio between true individuals who were identified as being at risk for self-harm and did engage in self-harm divided by all the people who engaged in self-harm behavior. Specificity was defined as the ratio between those whom the tool correctly identified as not being at risk for self-harm and all those who did not self-harm. The positive predictive value (PPV) was defined as the ratio between the individuals correctly identified by the instrument as being at risk for self-harm compared to all those who were identified as being at risk. The negative predictive value (NPV) was defined as the ratio between those correctly identified by the tool as having no risk for self-harm compared to all those with a negative test. For our study, the positive likelihood ratio (PLR) was defined as the probability of an individual who self-harmed having been identified with the instrument divided by the probability of an individual who did not self-harm having a positive test using the instrument. Similarly, the negative likelihood ratio (NLR) was defined as the probability of an individual who engaged in self-harm having a negative screen with the tool divided by the probability of an individual who did not engage in self-harm having a negative screen. We then created receiver operating characteristic (ROC) curves for the three suicide risk levels and the six indicators, and also calculated the area under the curve (AUC) for each of the ROC curves. Kuder-Richardson Formula 20 (KR-20) internal reliability was calculated for the six-items. An exploratory factor analysis with oblique rotation was also undertaken to determine the convergent validity of the six-items to measure a single construct.

The project was granted judicial approval prior to its commencement and ethics approval was also received from the Human Research Ethics Board at the University of Manitoba.

Results

Table 1 provides the number of individuals who engaged in self-harm behavior during the study period based on their ISA classification and number of risk indicators present. Fifteen people with no identifiable risk engaged in self-harm over the time period. Thirty-seven individuals who were at low or no risk engaged in self-harm, and 234 with high risk engaged in self-harm. With regard to total number of suicide risk items, 228 individuals with more than one variable engaged in self-harm while five people with all six suicide risk variables engaged in self-harm. Six people with all 6 suicide risk indicators did not engage in self-harm.

Table 1.

Suicide Risk Score and Self-Harm during Current Custodial Period

		Self-harm during current custodial period
		Yes (n)	No (n)
Suicide risk (no vs low/med/high)	Low/Med/High	225	2295
	No	15	2062
Suicide risk (no/low vs med/high)	Med/High	37	253
	No/Low	203	4104
Suicide risk (no/low/med vs high)	High	6	31
	No/Low/Med	234	4326
Suicide risk score ≥ 1	Yes	228	2809
	No	13	2106
Suicide risk score ≥ 2	Yes	61	513
	No	180	4402
Suicide risk score ≥ 3	Yes	24	162
	No	217	4753
Suicide risk score ≥ 4	Yes	15	57
	No	226	4858
Suicide risk score ≥ 5	Yes	9	21
	No	232	4894
Suicide risk score ≥ 6	Yes	5	6
	No	236	4909

In table 2, we studied the ability of the classification properties of the tool to confer risk for self-harm during the study period. Having at least a low suicide risk level (93.8%) or having at least one suicide risk factor (94.6%) provided a high level of sensitivity. The specificity was very high if an individual had a medium suicide risk level (94.2%) or at least three suicide risk indicators (96.7%), regardless if they were primary or secondary (see Appendix A for a list of primary and secondary indicators). The positive predictive value (PPV) was low for all suicide risk levels (0.09–0.16) and throughout all suicide risk factor levels (0.08–0.5). Similarly, the negative predictive value (NPV) was high for all suicide risk levels (0.95–0.99) and remained elevated regardless of the number of suicide risk indicators (0.95–0.99). The positive likelihood ratio (PLR) increased throughout the suicide risk levels, going as high as 3.51 for a high suicide risk level, indicating a 15% increased probability of self-harm (McGee, 2002). The PLR also increased to very high levels as the number of suicide risk indicators increased, going to a high of 17.00, indicating an increased probability of harm above 45% (McGee, 2002). The negative likelihood ratio (NLR) went to a low of 0.13 for a low suicide risk level, indicating a 30% decreased probability of self-harm (McGee, 2002). Similarly, the NLR went down to 0.13 when an individual had only one suicide risk factor, indicating a 30% decreased probability of self-harm (McGee, 2002).

Table 2.

Analysis of suicide risk scores

	Sensitivity	Specificity	PPV	NPV	Positive likelihood ratio	Negative likelihood ratio
Suicide risk level
Low	0.94	0.47	0.09	0.99	1.78	0.13
Medium	0.15	0.94	0.13	0.95	2.66	0.90
High	0.03	0.99	0.16	0.95	3.51	0.98
Suicide risk level total score
≥ 1	0.95	0.43	0.08	0.99	1.66	0.13
≥ 2	0.25	0.90	0.11	0.96	2.43	0.83
≥ 3	0.10	0.97	0.13	0.96	3.02	0.93
≥ 4	0.06	0.99	0.21	0.96	5.37	0.95
≥ 5	0.04	1.00	0.30	0.96	8.74	0.97
≥ 6	0.02	0.999	0.46	0.95	17.00	0.98

PPV = Positive Predictive Value; NPV = Negative Predictive Value

In table 3, we repeated the sensitivity, specificity, PPV, NPV, PLR, and NLR calculations with the individual suicide risk indicators included in the tool. The most sensitive risk factor for self-harm was a previous suicide attempt or a prior history of suicide in oneself or others (94.6%). Other risk indicators such as hopelessness (9.5%), suicide plan at intake (2.5%), or feeling alone (4.6%) were not highly sensitive for the event of self-harm. Several highly specific risk indicators for self-harm were identified, including hopelessness (95.3%), suicidal thoughts at intake (98.1%), a suicide plan at intake (99.6%), and feeling alone (97.2%). These risk indicators each had a low PPV (with a range between 0.23 to 0.07), and similarly, all risk indicators had high NPVs (with a range between 1.00 to 0.95). Both expressing hopelessness (PLR 2.02) and suicidal thoughts at intake (PLR 3.99) indicated an increased probability of self-harm of more than 15%, while expressing a suicide plan at intake (PLR 6.12) conferred a 30% increased probability of self-harm (McGee, 2002). A very low NLR was found for those with a prior history of exposure to suicide (0.11), indicating a reduced risk of self-harm of about 30% if an individual did not have such an experience (McGee, 2002).

Table 3.

Analysis of specific indicators within the Inmate Security Assessment Tool

Sensitivity	Specificity	PPV	NPV	Positive likelihood ratio	Negative likelihood ratio
Loss/stress	0.23	0.87	0.08	0.96	1.78	0.88
Hopelessness	0.10	0.95	0.09	0.96	2.02	0.95
Prior suicide history (self or others)	0.95	0.50	0.09	1.00	1.90	0.11
Suicidal thoughts at intake	0.08	0.98	0.16	0.96	3.99	0.94
Suicide plan at intake	0.03	0.996	0.23	0.95	6.12	0.98
Alone	0.05	0.97	0.07	0.95	1.64	0.98

PPV = Positive Predictive Value; NPV = Negative Predictive Value

In figure 1, we created a receiver operating characteristic (ROC) curve based on the suicide risk level. The sensitivity was 0.94 and specificity was 0.47. The area under the curve (AUC) was 0.72 (95%CI = 0.69–0.75), indicating a fair test. In figure 2, we created a ROC based on the number of suicide risk indicators present. We found a sensitivity of 0.95 and a specificity of 0.43. The AUC was significant at 0.71 (95%CI = 0.68–0.74), indicating a fair test.

Figure 1.

Receiver Operating Characteristic Curve (plotting sensitivity and specificity) for the Inmate Security Assessment – score based on Suicide Risk Level.

Area Under Curve = 0.72 (95% CI = 0.69–0.75)

Optimum cut-off is ≥ 1 (indicating low to high) with sensitivity = 0.94 and specificity = 0.47.

Figure 2.

Receiver Operating Characteristic Curve for identifying sensitivity and specificity for the Inmate Security Assessment - Score based on number of risk items endorsed.

Area Under Curve = 0.71 (95% CI = 0.68–0.74)

Optimum cut-off is ≥ 1 with sensitivity = 0.95 and specificity = 0.43.

The Cronbach’s alpha was equal to 0.592 for the scale, indicating poor reliability of the items as an overarching scale to predict future self-harm. Inter-item correlations were also low, indicating low convergent validity of the items to measure a single construct. An exploratory factor analysis of the 6 items indicated a two-factor solution, rather than a single factor solution, with the two factors explaining 51.8% of the variance.

Discussion

This study has multiple strengths. It included a large sample size and test data was derived from the population among which it is implemented, which is necessary to obtain the most accurate and meaningful results (Strassle et al., 2012; Van den Bruel et al., 2007; Parfrey et al., 2009). The test results utilized staff report of self-harm instead of detainee information, which minimized recall bias. There was no lost data as the tool was applied universally to all participants as part of their routine care.

The test parameters demonstrated no significant problems with the instrument. This tool showed good sensitivity and moderate specificity, which is a trade-off found with most diagnostic tests (Haynes et al., 2006; Greiner et al., 2000). In the tool, we would want high sensitivity so as to not miss an important event like suicide. However, we would ideally also have high specificity, so that we would not unnecessarily utilize expensive, intrusive or difficult-to-implement safety protocols. The tool appropriately increased or decreased the probability of self-harm dependent on the suicide risk level or number of risk indicators present based on likelihood ratio calculations. The AUCs for the suicide risk level and the number of indicators both provided information that the tool was a fair and moderately accurate test (Greiner et al., 2000; Weinstein et al., 2005).

The positive predictive value (PPV) was low for all suicide risk levels and the negative predictive value (NPV) was high for all suicide risk levels. As the prevalence of self-harm has a fairly low base rate, the PPV was expected to be low and the NPV was expected to be high, even when using a test with high sensitivity and specificity (Akobeng, 2006; Glaros and Kline, 1988). While critics of suicide risk assessment tools have argued that low PPV leads to inappropriate allocation of sparse resources (Hatcher, 2007), our youth were already in custody receiving intense support. Although the reliability and some measures of validity of the scale were quite poor, the primary purpose of our study was to examine the scale in its current form as a possible predictor of self-harm among incarcerated youth, since this was how it was being implemented in our province.

Several limitations in this study were noted. We had no ability to ascertain the intent of self-harm behavior. The ISA was not developed with empirical evidence, though this is the case in other traditional risk assessment tools (Roos et al., 2013). Mental disorder was not studied. The prior suicide history indicator did not distinguish between suicide attempts by the youth versus by others known to the respondent. Clarifying the contributions of these two differing variables within the same indicator would be important in a future iteration of the tool given that this indicator was overwhelmingly the strongest single risk predictor of self-harm. As the prevalence of the behavior influences screening tests, caution is warranted in extending our findings to different populations.

Another issue with our study was the administration of the instrument by front-line staff. Although a collaborative, therapeutic alliance between clinician and patient is important when conducting an assessment (Fowler, 2012), front-line staff do not typically serve that purpose, instead focusing on security and containment. Self-harm events may not have been captured by staff, although this possibility was offset by the highly supervised nature of a correctional facility. Staff may not have filled out the tool correctly or may have shown bias in recording responses. It is possible that some of the protocols that existed to support those at risk for self-harm influenced the results. For example, expressing a suicide plan at intake would have led to one-to-one supervision with staff, likely preventing self-harm. This supervision would occur until staff determined that there was decreased risk. Therefore, it is possible that the results of this study were an underestimate of the true level of risk.

Our study is subject to flaws inherent with current suicide risk prediction methods. There has been a traditional reliance on subjectively reported information about the respondent’s thoughts and feelings, which can be misleading when assessing the risk of suicide (Bolton et al., 2015). Alternative assessment measures, such as computer-based implicit association tests, have been studied in adults (Nock et al., 2010) and youth (Cha et al., 2016), though not in detained adolescents. One of the most predictive variables for self-harm is a history of previous self-harm, but new cases of self-harm are very hard to predict given its low incidence (Bolton et al., 2015). Additional challenges in juvenile detainees are the developmental nature of adolescence, as different factors may contribute to suicide risk at different ages or developmental stages of adolescence, the course of adolescents’ mental status which varies over short periods of time, and the standard for determining validity, as suicides are rare (Grisso, 2004).

Despite these issues, our study provides important contributions to the field. It included staff-documented incidents of self-harm, and did not rely on self-report, which is more often utilized in these studies. The instrument meets the suggestions made by the National Action Alliance for Suicide Prevention (2013) for optimal screening instruments, which include the ability to be managed by non-mental health professionals with minimal in-service training on administration and scoring; requiring less than 15–20 minutes to administer and score; and being developed for use for screening in juvenile justice settings. Operational characteristic analysis was beyond the scope of this study, but would include variables such as the time taken to perform the test, its ease of use, cost, user acceptability and the stability of the test under user conditions (Banoo et al., 2008; Parfrey et al., 2009).

Conclusions

The juvenile justice system has two general responsibilities for suicide prevention: the first is to ensure the safety of young people while they are in the system’s custody and the second is to facilitate rehabilitation and treatment that will prevent further delinquency and promote positive youth development (National Action Alliance for Suicide Prevention, 2013; Youth Criminal Justice Act, 2002). Based on our analysis, the ISA tool provided a fair and moderately accurate measurement of risk of self-harm among detained youth.

The ultimate goal of screening for suicide risk is to improve outcomes and reduce rates of morbidity and mortality from self-harm. Screening and assessment should be part of a comprehensive suicide prevention program including identification, evaluation, communication, housing in a safe environment, follow-up, and treatment planning (National Action Alliance for Suicide Prevention, 2013). In the detained youth population, which is known to have greater rates of mental illness (Teplin et al., 2002) and adversity (Ford et al., 2013), the ISA tool is a fair screening instrument. It does not, however, take into account the multiple factors that can lead an individual to suicidal ideation and self-harm. Future directions of research would include the creation of a prediction algorithm for suicide attempts similar to that created for depression (Wang et al., 2014). The 6 indicators within the tool could also be weighted in order to more accurately determine risk of self-harm. The analysis of another instrument could also be used alongside the ISA in order to identify mental health issues that could increase the risk for suicide and self-harm. One possible tool is the Massachusetts Youth Screening Instrument – Second Version (MAYSI-2), which was developed specifically for the juvenile justice population (McCoy et al., 2014; Wasserman et al, 2014; Gilbert et al., 2014). Lennox (2015) demonstrated an AUC of 0.71 (fair performance) for suicidal ideation using the MAYSI-2. Given the limitations of the ISA tool, further study is warranted to optimize risk screening for young people in custody.

It is recommended that further work be done to determine modifiable factors that lead to the cascade of suicidal thoughts and behaviors in detained youth. Because the needs of juvenile detainees differ greatly from those of detained adults, age-specific approaches and prevention programs are needed (Radeloff et al., 2015). For example, understanding how exposure to suicide and self-harm affects youth could help mitigate the risk of social contagion (Hawton et al., 2012). The development of new psychosocial and potentially pharmacological interventions to reduce self-harm is important. The act of a youth coming into custody is an opportunity for evaluation and provision of care. Treatment of depression and anxiety in adolescence may prevent incident self-harm in adulthood (Moran et al., 2012). Identification and appropriate intervention is imperative to improving the outcomes of some of our society’s most vulnerable and marginalized individuals.

Appendix A. The Inmate Security Assessment Tool

Primary Indicators

• Current suicide plan

• Current suicidal thoughts

• Feeling alone

• Prior suicide history – includes either a previous attempt by the respondent or knowledge of a history of suicide in others

Secondary Indicators

• Feelings of loss or stress

• Feelings of hopelessness or helplessness

Classification of Risk

• No evidence – all primary and secondary indicators are negative

• Low Risk – history of previous suicide attempt or both secondary indicators present

• Medium Risk – suicidal ideation present (but no clear plan); or previous history of suicide attempt, youth believe they are alone or lack resources, and presence of one or more secondary indicators

• High Risk – clear and current plan to attempt suicide; or have plan combined with one or more primary or secondary indicators of suicide