Development of the SaFETy Score: A Clinical Screening Tool for Predicting Future Firearm Violence Risk

Abstract

Background

Interpersonal firearm violence among youth is a substantial public health problem, and emergency department (ED) physicians require a clinical screening tool to identify high-risk youth.

Objective

To derive a clinically feasible risk index for firearm violence.

Design

24-month prospective cohort study.

Setting

Urban, level 1 ED.

Participants

Substance-using youths, aged 14 to 24 years, seeking ED care for an assault-related injury and a proportionately sampled group of non–assault-injured youth enrolled from September 2009 through December 2011.

Measurements

Firearm violence (victimization/perpetration) and validated questionnaire items.

Results

A total of 599 youths were enrolled, and presence/absence of future firearm violence during follow-up could be ascertained in 483 (52.2% were positive). The sample was randomly split into training (75%) and post–score-construction validation (25%) sets. Using elastic-net penalized logistic regression, 118 baseline predictors were jointly analyzed; the most predictive variables fell predominantly into 4 domains: violence victimization, community exposure, peer influences, and fighting. By selection of 1 item from each domain, the 10-point SaFETy (Serious fighting, Friend weapon carrying, community Environment, and firearm Threats) score was derived. SaFETy was associated with firearm violence in the validation set (odds ratio [OR], 1.47 [95% CI, 1.23 to 1.79]); this association remained (OR, 1.44 [CI, 1.20 to 1.76]) after adjustment for reason for ED visit. In 5 risk strata observed in the training data, firearm violence rates in the validation set were 18.2% (2 of 11), 40.0% (18 of 45), 55.8% (24 of 43), 81.3% (13 of 16), and 100.0% (6 of 6), respectively.

Limitations

The study was conducted in a single ED and involved substance-using youths. SaFETy was not externally validated.

Conclusion

The SaFETy score is a 4-item score based on clinically feasible questionnaire items and is associated with firearm violence. Although broader validation is required, SaFETy shows potential to guide resource allocation for prevention of firearm violence.

Primary Funding Source

National Institute on Drug Abuse R01024646.

Firearm violence has been identified by health and legal professionals as a critical public health problem (1). Homicide is the third leading cause of death in the United States among youth aged 15 to 24 years, with more than 86% of these deaths due to firearms (2). Furthermore, firearm violence results in substantial monetary cost; for example, medical and work-loss costs of nonfatal firearm injuries treated in U.S. emergency departments were estimated to exceed $2.9 billion in 2010 (3). Mitigating this public health issue requires novel hospital and community-based interventions that are focused on at-risk youth, especially those in urban communities. Urban emergency departments (EDs) have been identified as a critical access point for identifying and intervening with such youth (4). Firearm violence encompasses interpersonal, self-directed, and unintentional firearm-related incidents, but in this study we focus on interpersonal firearm violence, which we refer to simply as “firearm violence” throughout.

Although previous ED-based research (5) has identified risk factors associated with firearm violence involvement among high-risk youth, the field of hospital and ED-based youth violence prevention programs lacks a short, clinically relevant screening tool that can be applied as part of routine clinical care in urban settings. Such a tool could play a key role in determining where to focus prevention or intervention efforts. Youth identified during an ED visit, particularly violently injured youth, are at elevated risk for future firearm violence (5) and thus would benefit most from early intervention, including case management and therapeutic services. Previous screening tools for youth violence (6–8) primarily focused on primary care settings, lack a specific focus on firearm violence, or are too lengthy for practical use in a busy ED setting. Furthermore, research on the construction of violence screening tools (6, 8) has been limited by small sample sizes and has not considered out-of-sample predictive power in devising the screen. Developing an ED/hospital-based clinical screening tool that is focused on assessing risk for future firearm violence will enable ED and hospital health systems to better focus prevention resources on patients at the highest risk.

In the current study, we seek to develop a clinical screening tool for future risk for firearm violence by examining data collected as part of a 2-year prospective study of youth aged 14 to 24 years seeking ED care. First, we used machine learning methods to determine which variables measured at the baseline of a 2-year longitudinal study were most predictive of future firearm violence. Second, on the basis of breadth and clinical feasibility, we selected 4 items from among the most predictive variables. Third, we developed cut-points and assigned point values to each level based on their relative effects, resulting in the SaFETy (Serious fighting, Friend weapon carrying, community Environment, and firearm Threats) score. Finally, we examined the relationship between the SaFETy score and rates of future firearm violence within training and internal validation data sets.

METHODS

Study Design and Setting

Data were collected during the Flint Youth Injury study (9–11), a 2-year prospective cohort study of assault-injured youth (age 14 to 24 years) with any drug use in the past 6 months and a comparison group of non–assault-injured, drug-using youth seeking ED care at a level 1 trauma center in Flint, Michigan. The parent study focused on service needs and utilization among substance (predominantly marijuana) users. Although this potentially limits generalizability, we note that most youth who seek care for assault injuries in this setting are substance users (9). Patients were recruited from December 2009 through September 2011, 24 hours per day on Thursday through Monday and from 5 a.m. to 2 a.m. on Tuesday and Wednesday. Youth who sought care for sexual assault, child abuse, suicidal ideation or attempt, or any conditions that preclude consent (such as altered mental status) were excluded. Institutional review boards at the University of Michigan and Hurley Medical Center approved the study. A National Institutes of Health (NIH) Certificate of Confidentiality (COC) was obtained.

Potential participants were ascertained through electronic patient logs and approached by research assistants in waiting or treatment areas. All assault-injured youth, including those who were initially unstable but stabilized with 72 hours of presentation, were approached and screened for study eligibility. In sequence, the next available age group (14 to 17, 18 to 20, and 21 to 24 years) and sex-matched, non–assault-injured ED entrant was screened for the comparison group. Those providing consent (or assent with parental consent for those younger than age 18 years) privately self-administered the screening survey using a tablet device and received a $1.00 gift for participation. Individuals who self-reported drug use in the past 6 months (98% used marijuana) were considered eligible and consented to the subsequent 2-year longitudinal study. Appendix Figure 1 (available at Annals.org) shows a flow chart of the original study. Remuneration was $20 for completion of a subsequent self-administered baseline survey. Follow-up assessments were conducted at 6, 12, 18, and 24 months, and participants were compensated $35, $40, $40, and $50 for each sequential follow-up. Baseline characteristics (9) and 2-year outcomes (5, 10) are reported elsewhere.

Measures

The following measures were assessed:

The outcome variable was a binary indicator of firearm violence (victimization, perpetration, firearm injury requiring medical care, or firearm death) during the 24-month follow-up period, ascertained through a composite of self-report, medical chart review, and vital records databases (see Carter and colleagues [5] for greater detail). Both peer and partner firearm violence was included. Although the dynamics of peer and partner violence differ, we justify combining them by noting the large overlap between victims (12, 13) and perpetrators (14–16) of peer and partner violence.

Candidate predictor variables were taken from baseline self-report surveys; in addition to age, sex, and reason for ED visit (assault-injured/non–assault-injured), we included 115 survey items. Other variables that were measured but judged unlikely to be assessed accurately and truthfully (for example, serious violence perpetration) without an NIH Certificate of Confidentiality were not considered. See the Supplement (available at Annals.org) for question wording and response options for all items described below.

1. Violence items (13 items) from the National Longitudinal Study on Adolescent Health (17) captured the frequency of received threats/violence, perpetrated threats, fighting, and carrying a weapon while intoxicated in the past 6 months.

2. Partner aggression (13 items) was assessed with Conflict Tactics Scale items (18), which measured the frequency of partner violence victimization in the past 6 months.

3. Nonpartner aggression (13 items) was assessed with questions modified from the Conflict Tactics Scale (18), measuring the frequency of nonpartner violence victimization in the past 6 months.

4. Community violence exposure (5 items) included assessment of the frequency of exposure to violence and neighborhood crime in the past 6 months (19).

5. Mental health (12 items) was measured with the Brief Symptom Inventory checklist (20), which assessed severity of depression and anxiety in the past week.

6. Drug and alcohol efficacy (16 items) assessed confidence in avoiding drug (8 items) or alcohol (8 items) use in various situations (21, 22).

7. Alcohol use (10 items) was assessed with the Alcohol Use Disorders Identification Test (AUDIT), which measures the frequency of alcohol consumption and alcohol-related consequences in the past 6 months (23, 24).

8. Peer influences (11 items) included items from the Flint Adolescent Study (25) regarding the number of friends providing positive (4 items) and negative (7 items) influences; positive items were reverse coded.

9. Parental behavior (10 items) included items from the Flint Adolescent Study (25) assessing parental support (6 items) and level of parental drug/alcohol use (4 items); parental support was reverse coded.

10. Retaliatory attitudes (7 items) included items assessing willingness to engage in violent retaliation; higher scores indicate greater willingness (26, 27).

11. Fight self-efficacy (5 items) assessed perceived ability to avoid conflicts (28).

Statistical Analysis

We first randomly split the data into a training set (75%) and a validation set (25%), ensuring that the prevalence of firearm violence was equivalent in each. In determining variable importance, developing cut-points, and assigning point values, we used only the training data. Multiple imputation by chained equations (29), implemented in the R statistical software package mice (29), was used to impute missing outcomes (there were no missing values in the predictors measured at baseline) for inclusion in the training data. Variable importance analyses and determination of risk score contributions were based on estimates pooled across 50 multiply imputed data sets.

To determine variable importance, we used elastic-net penalized logistic regression, a common machine learning approach to binary classification, which shrinks the regression coefficient estimates to improve out-of-sample prediction. The elastic net (30) penalizes both the absolute value of the coefficients, which performs automatic variable selection by shrinking irrelevant coefficients to zero (31), and the squared size of the coefficient, which limits the effect of collinearity (32). The elastic net is particularly applicable when the number of predictors is large relative to the sample size and when interest lies in variable selection wherein the effect of collinearity is reduced (30). To evaluate out-of-sample prediction accuracy, we used leave-one-out cross-validation (LOOCV). In LOOCV, the model is fit to the entire sample, minus 1 point, and is used to predict the excluded case; this is repeated so each point is left out once. The level of coefficient shrinkage that minimizes the LOOCV error rate was chosen. Model fitting and cross-validation were performed by using the R statistical software package glmnet (33).

After determining the optimally penalized model, we ranked predictor importance by using the size of the standardized regression coefficient estimates. Using the variable importance rankings and the variable domains, we selected 4 variables that were predictive and covered distinct content. We justify the use of 4 items by noting that exploration with 3 item scores (not shown) indicated that the 4-item score yielded a more thorough risk gradient (in the training data), and this was not notably improved by adding a fifth item; because clinical feasibility is paramount, item scores of 6 or more were not explored. Cut-points for each item were chosen by cycling through all possible categorizations of each variable, fitting a logistic regression model of future firearm violence (in the training set) using the categorized variables, and choosing the categorization that minimized the finite-sample–corrected Akaike information criteria (34). To avoid overfitting the training data, a maximum of 3 categories were considered for each variable and overly small categories (<20 people) were not considered. Finally, we determined score contributions by 1) entering the categorized predictors into a single logistic regression model and 2) scaling by the minimum regression coefficient and rounding to an integer (as in reference 35). Properties of the risk score in the validation set (sensitivity, specificity, and odds ratio [OR] with firearm violence) were examined and stratified by assault-injured/non-assault-injured group.

Role of the Funding Source

Our funding sources had no role in the design, conduct, or analysis of our study or the decision to submit the manuscript for publication.

RESULTS

Sample Characteristics

In total, 599 youth (349 assault-injured and 250 non-assault-injured) participated in the study. Follow-up rates were 85.3%, 83.7%, 84.2%, and 85.3% at 6, 12, 18, and 24 months. Among participants, 483 (80.6%) could be definitively classified as having been involved with firearm violence (n = 252 [52.2%]) or not (n = 231 [47.8%]) during the follow-up period. Of the participants, 57.3% were male and 62.5% were African American; the average age at baseline was 19.9 years (SD, 2.4 years); greater detail is published elsewhere (36). One fourth of those with (n = 63) and those without (n = 58) firearm violence were randomly removed for post-score-construction validation; there were no significant demographic differences between the training and validation data.

Variable Importance Analysis

Appendix Figure 2 (available at Annals.org) shows the relative size of the top 20 largest standardized coefficients. Table 1 lists the selected items and their univariate associations with firearm violence. Selected items largely fell into 4 domains that were observed post hoc: 1) violence victimization (peer and partner), 2) community violence exposure, 3) peer/family influences, and 4) fighting.

Table 1.

Highest-Ranked Prognostic Factors for Future Firearm Violence

Prognostic Factor	Response Type*	Importance Rank	Timeframe	Odds Ratio (95% CI)†	Standardized Odds Ratio‡
Received threats
Someone pulled a gun on you	Frequency (0–6)	1	6 mo	2.44 (1.89–3.20)	2.90
Someone used a gun on you	Frequency (0–6)	2	6 mo	3.31 (1.98–5.52)	2.22
Someone pulled a knife on you	Frequency (0–6)	6	6 mo	1.89 (1.40–2.54)	1.81
Someone shot you	Frequency (0–6)	9	6 mo	2.92 (1.69–5.06)	1.73
Someone threw something at you	Frequency (0–6)	12	6 mo	1.58 (1.21–2.07)	1.58
Someone cut/stabbed you	Frequency (0–6)	14	6 mo	2.22 (1.47–3.34)	1.73
Community
I have seen someone shot	Frequency (0–3)	3	6 mo	1.87 (1.44–2.42)	1.78
I have heard guns shot	Frequency (0–3)	5	6 mo	1.55 (1.27–1.90)	1.61
Seen gangs in neighborhood	Frequency (0–3)	18	6 mo	1.31 (1.11–1.55)	1.40
My house was broken into	Frequency (0–3)	20	6 mo	1.70 (1.23–2.34)	1.45
Friends
My friends carry weapons	Number (1–5)	10	Current	1.56 (1.26–1.92)	1.62
My friends smoke marijuana	Number (1–5)	15	Current	1.27 (1.07–1.50)	1.34
Friend legal trouble (drug-related)	Number (1–5)	17	Current	1.63 (1.25–2.12)	1.53
Partner violence
Partner used a knife on you	Frequency (0–6)	13	6 mo	3.46 (1.46–8.24)	2.37
Fighting
Been in a serious fight	Frequency (0–6)	4	6 mo	1.46 (1.25–1.71)	1.76
Put someone in the hospital	Frequency (0–6)	7	6 mo	1.71 (1.35–2.16)	1.85
Drank before fighting	Frequency (0–6)	8	6 mo	1.78 (1.33–2.38)	1.72
Other
Understand another’s point of view	Agree (1–5)	11	6 mo	1.38 (1.16–1.64)	1.49
Today’s ED visit for violent injury	Yes/no	16	Current	1.89 (1.23–2.89)	NA
Unable to stop drinking	Frequency (0–4)	19	6 mo	1.56 (1.10–2.20)	1.39

ED = emergency department; NA = not available.

^*Frequency (0–6) measures frequency on a 7-point scale from 0 (never) to 6 (20+ times). Frequency (0–3) measures frequency on a 7-point scale from 0 (never) to 3 (many times). Frequency (0–4) measures frequency on a 5-point scale from 0 (never) to 4 (daily). Number (1–5) measures frequency on a 5-point scale from 1 (none) to 5 (all). Agree (1–5) measures agreement on a 5-point scale from 1 (very true) to 5 (not true). Yes/no denotes a binary (1/0) indicator.

^†CIs with lower bounds of 1.00 are entirely above 1.00.

^‡Standardized odds ratios were those obtained by using the standardized predictors.

Risk Score Construction

We narrowed the 20 variables to 4 items from different domains to construct the SaFETy score. From violence victimization, we chose the highest-ranking item: being threatened with a firearm. For practical use this could be combined with the number 2–ranked and number 9–ranked items into “threatened or shot you” because of similar content. “Friend weapon carrying” was chosen from the peer influence domain. Among the community violence items, we chose the lower-ranking item–frequency of hearing gunshots–because of the greater likelihood of a truthful response in a clinical setting (compared with the frequency of seeing someone shot). Similarly, we chose “frequency of being in a serious fight” over “frequency of putting someone in the hospital” from the fighting domain.

Table 2 shows the derived cut-points and weights for each category, and how the selected items correspond to the SaFETy mnemonic. “Received firearm threats” was divided into “never,” “once,” and “2+ times”; “hearing gunshots” was divided into “less than many times” and “many times”; “fight frequency” was stratified into “never,” “1–5 times,” and “6+ times”; and “friend weapon carrying” was collapsed into whether or not “many, most, or all” friends carry weapons. The largest weights corresponded to received firearm violence threats and high-frequency serious fighting.

Table 2.

Rules for Calculation of the SaFETy Score

Mnemonic	Category	Question/Scale Levels	SaFETy Contribution
S	Serious Fighting	In the past 6 mo, including today, how often did you get into a serious physical fight?
		0 (never)	0
		1 (once)	1
		2 (twice)	1
		3 (3–5 times)	1
		4+ (6 or more times)	4
F	Friend Weapon Carrying	How many of your friends have carried a knife, razor, or gun?
		1 (none)	0
		2 (some)	0
		3+ (many, most, or all)	1
E	Community Environment	In the past 6 mo, how often have you heard guns being shot?
		0 (never)	0
		1 (once or twice)	0
		2 (a few times)	0
		3 (many times)	1
T	Firearm Threats	How often, in the past 6 mo, including today, has someone pulled a gun on you?
		0 (never)	0
		1 (once)	3
		2+ (twice or more)	4

SaFETy = Serious fighting, Friend weapon carrying, community Environment, and firearm Threats.

Risk Score Performance

In the validation set, a 1-point increase in SaFETy score corresponded to higher risk for firearm violence (OR, 1.47 [95% CI, 1.23 to 1.79]). The area under the receiver-operating characteristic curve was 0.73, indicating reasonable out-of-sample discriminatory power. The Figure shows the distribution of SaFETy scores among those with and without firearm violence in the validation set. Table 3 shows the sensitivity and specificity of SaFETy score in the validation set for each cut-point between 0 and 10 (Appendix Table 1, available at Annals.org, shows training sample results). Informal examination of the training data indicated 5 risk strata: SaFETy scores of 0, 1 to 2, 3 to 5, 6 to 8, and 9 to 10; the same risk gradient is apparent in the validation set (Appendix Figure 3, available at Annals.org), with each level corresponding to future firearm violence rates of 18.2% (2 of 11), 40.0% (18 of 45), 55.8% (24 of 43), 81.3% (13 of 16), and 100.0% (6 of 6), respectively.

Figure.

Distribution of SaFETy scores among youth with and without firearm violence during the follow-up period in the validation data.

SaFETy = Serious fighting, Friend weapon carrying, community Environment, and firearm Threats.

Table 3.

Sensitivity and Specificity for SaFETy Score Thresholds Between 1 and 10 in the Validation Set

Threshold	Sensitivity, n/N (%) (95% CI)	Specificity, n/N (%) (95% CI)
1	61/63 (96.8 [88.0–99.4])	9/58 (15.5 [7.8–27.9])
2	53/63 (84.1 [72.3–91.7])	27/58 (46.6 [33.5–60.0])
3	43/63 (68.3 [55.2–79.1])	36/58 (62.1 [48.3–74.2])
4	36/63 (57.1 [44.1–69.3])	40/58 (69.0 [55.3–80.1])
5	32/63 (50.8 [38.0–63.5])	48/58 (82.8 [70.1–91.0])
6	19/63 (30.2 [19.6–43.2])	55/58 (94.8 [84.7–98.7])
7	7/63 (11.1 [5.0–22.2])	57/58 (98.3 [89.5–100.0])
8	6/63 (9.5 [3.9–20.2])	58/58 (100.0 [92.3–100.0])
9	6/63 (9.5 [3.9–20.2])	58/58 (100.0 [92.3–100.0])
10	5/63 (6.3 [2.1–16.3])	58/58 (100.0 [92.3–100.0])

SaFETy = Serious fighting, Friend weapon carrying, community Environment, and firearm Threats.

Sensitivity Analysis

Because membership in the assault-injured group is itself associated with future firearm violence (Appendix Tables 2 and 3, available at Annals.org), we present several stratified analyses. First, we conducted an omnibus test comparing the model with all 20 of the top variables that do versus do not allow the effects to vary by assault-injured/non–assault-injured group, which yielded a nonsignificant result (P = 0.09). Appendix Tables 4 and 5 (available at Annals.org) show univariate associations between each variable and firearm violence, stratified by assault-injured/non–assault-injured group. Second, we estimated the joint effects of SaFETy score and assault-injured group membership (Appendix Table 3) and found that 1) with SaFETy score included in the model, assault-injured group membership was not significant and 2) the effect of SaFETy score was similar after the inclusion of the assault-injured group. Appendix Tables 6 and 7 (available at Annals.org) show frequency distributions of SaFETy scores and sensitivity/specificity estimates, respectively, stratified by assault-injured group (corresponding training sample results are shown in Appendix Tables 8 and 9, available at Annals.org).

DISCUSSION

In the current study, we used machine learning methods and data from a prospective cohort of ED patients to identify factors predictive of future firearm violence that could be incorporated into a brief clinical screening tool for ED use. This data set is unique in that high-risk urban youth were followed successfully over 2 years and that the interviews capture not only firearm violence that resulted in injury but incidents that resulted in near-injury (for example, being threatened with a gun). SaFETy clearly defines a gradient for future firearm violence risk in this population; this steady increase (rather than a sharp increase at 1 point) inhibits determination of a threshold with strong combined sensitivity/specificity but creates a strong basis for allocation of prevention resources.

Coupling risk stratification with effective prevention tools is an important potential use of the SaFETy score. Very-high-risk individuals (e.g., those with a SaFETy score ≥6) may represent sensible candidates for entry into resource-intensive programs (for example, 1-year wraparound programs), whereas individuals in the middle range (a SaFETy score ≥1 but ≤5) may benefit from graduated levels of targeted interventions designed to interrupt a negative trajectory. Programs focusing on primary prevention may be appropriate for lower-scoring (a SaFETy score of 0) individuals. Although SaFETy has predictive power among both assault-injured and non–assault-injured youth, it is most applicable among non–assault-injured youth because there is no other current means of stratifying risk in that group. Furthermore, given the excess risk for future victimization among those presenting with assault injury, particularly firearm injuries (37), prevention resources should also be considered for this group, even among those with low SaFETy scores. Previous violence prevention programs have been shown to be cost-effective (38–42), specifically with regard to the costs of treating repeated violent injuries (38, 39) and preventing incarceration due to violence-related offenses (38, 40). Given that the average ED visit for a firearm assault costs $1200 and average inpatient costs approach $24 000 (43), even a moderately effective prevention program directed at individuals in higher-risk strata would be cost-effective.

The items selected for the screening tool confer strong face validity to the data-derived prediction tool. Two items—history of receiving gun threats and hearing gun violence in one’s community–confirm the importance of previous violence exposure in the risk for future firearm violence. In addition, peer influences, whose importance is most pronounced during the period of adolescence and emerging adulthood (44), is an important prognostic factor, in the form of friend weapon carrying. The emergence of serious fighting as a strong predictor agrees with prior violence screening tools (7) and underscores the role of impulse control and aggression in firearm violence. These results highlight the broader importance of incorporating community and peer factors into prevention programs, in addition to addressing psychological distress stemming from victimization history, consistent with trauma-informed care practices (45).

We also note that male sex, which is consistently identified as a risk factor for gun violence (5, 46, 47), was not predictive of future gun violence. This suggests that sex differences in firearm violence risk may be subsumed by other risk exposure (such as affiliation with high-risk peers). Similarly, depression and anxiety were not identified as important prognostic factors, suggesting that their association with firearm violence may also be largely subsumed by proximal exposures to high-risk community violence. We note that self-reported ability to understand another’s point of view was highly associated with future gun violence but was not selected for the final screen because of concerns about its accurate assessment and wording that may be misunderstood in a brief clinical screen.

The results here are encouraging because they rely only on items considered feasible to ascertain in a clinical setting. The feasibility of screening on the basis of such measures as “Have you shot someone in the last 6 months?” is limited when the respondents may not perceive themselves as having the same degree of confidentiality as in a clinical study with an NIH Certificate of Confidentiality. In addition, that question is not likely to be asked or answered in a way that is perceived as nonjudgmental or nonincriminating by staff or patients. Asking youth about peer behavior that they are more likely to report–and is not incriminating–has been done for other risk behavior tools, such as the National Institute on Alcohol Abuse and Alcoholism alcohol screening tool for adolescents (48). None of the measures selected specifically ask respondents to 1) incriminate themselves, 2) incriminate any specific person, or 3) embarrass themselves. Although a superior predictive tool may be derived by including a broader class of measures, this gain may be offset by reduced willingness to answer or response accuracy.

Our study had several limitations. First, our sample is limited to a single urban ED; validation in other high-risk populations is required. Second, the analytic sample is limited to drug-using youth. Although we cannot ascertain the efficacy of this screen in non–drug-using youth, we note that substance use has been linked to both gun carrying (36) and violence (49) and that a large majority of individuals screened into the study because of marijuana use. To mitigate this limitation, clinicians could first inquire about marijuana use in the past 6 months, a standard history and physical examination question. Alternatively, given the lack of valid screening instruments for firearm violence and the knowledge that this is the leading cause of death for youth in urban communities, it would be clinically reasonable to suggest that high-scoring urban youth, even those who have not used marijuana in the past 6 months, warrant preventive services. However, future research is needed to validate this tool among non–substance-using youth.

Third, self-reported data were a large component of identifying those with versus those without firearm violence during the follow-up period, which is a concern for underreporting of firearm violence. This limitation is partly mitigated by the use of full validated scales, such as the Conflict Tactics Scale (18), which were privately administered on a tablet. Fourth, our mental health assessments focused only on depression and anxiety. Because such symptoms as suspiciousness, delusions, and extreme anger have been linked to violence and gun carrying (50, 51), future work is needed to assess their power to predict future gun violence. Finally, our missing-data imputation relies on the untestable missing-at-random assumption. Noting the high follow-up rate and that no covariates that made up the SaFETy score differed significantly in terms of missing versus nonmissing cases, the role of nonrandom attrition was likely minimal.

In conclusion, we used machine learning methods to determine the most important predictors of future firearm violence in a high-risk ED sample. This is the first scale to provide risk stratification for firearm violence and the first developed in and specifically for an ED setting (rather than primary care). Previous risk scores were developed to predict related but distinct behaviors, such as nonspecific violent injury (6, 8) and firearm carrying (7). The common thread between SaFETy and previous scales is the importance of fighting (6–8) and received threats (7) as prognostic factors. The SaFETy instrument, which can be administered in 1 to 2 minutes, defines a gradient of future firearm violence risk that can be adapted to a variety of settings. Emergency departments have been previously used as opportunities for identifying high-risk individuals for other types of violence (52, 53), but the current lack of an easily administered screening tool for firearm violence has limited our ability to harness the same opportunity for firearm violence. Our results suggest that SaFETy fills this gap.

From University of Michigan School of Medicine and University of Michigan School of Public Health Ann Arbor, Michigan; Centers for Disease Control and Prevention, Atlanta, Georgia; and Hurley Medical Center, Flint, Michigan.

Appendix

Appendix Figure 1.

Flint Youth Injury study flow chart.

ED = emergency department.

Appendix Figure 2.

Variable importance determined by the predictive model, expressed as standardized regression coefficients (divided by the minimum so that the smallest score is 1).

POV = point of view.

Appendix Figure 3.

Future firearm violence rates in the validation data set (dashed line) in 5 risk strata identified by using the training data set (solid line).

SaFETy = Serious fighting, Friend weapon carrying, community Environment, and firearm Threats.

Appendix Table 1.

Sensitivity and Specificity for SaFETy Score Thresholds Between 1 and 10 in the Training Set

Threshold	Sensitivity (95% CI)	Specificity (95% CI)
1	186/189 = 98.4% (95.1%–99.6%)	29/173 = 16.8% (11.7%–23.4%)
2	160/189 = 84.7% (78.5%–89.3%)	77/173 = 44.5% (37.0%–52.2%)
3	132/189 = 69.8% (62.7%–76.2%)	125/173 = 72.3% (64.9%–78.6%)
4	122/189 = 64.6% (57.2%–71.3%)	134/173 = 77.5% (70.4%–83.3%)
5	109/189 = 57.7% (50.3%–64.7%)	140/173 = 80.9% (74.1%–86.3%)
6	74/189 = 39.2% (32.2%–46.5%)	162/173 = 93.6% (88.6%–96.6%)
7	35/189 = 18.5% (13.4%–25.0%)	167/173 = 96.5% (92.3%–98.6%)
8	19/189 = 10.1% (6.3%–15.5%)	171/173 = 98.8% (95.4%–99.8%)
9	15/189 = 7.9% (4.7%–13.0%)	173/173 = 100.0% (97.3%–100.0%)
10	10/189 = 5.3% (2.7%–9.8%)	173/173 = 100.0% (97.3%–100.0%)

SaFETy = Serious fighting, Friend weapon carrying, community Environment, and firearm Threats.

Appendix Table 2.

Joint Distribution of Violent Injury and Future Firearm Violence Status

Status	AI	Non-AI	Total
Future firearm violence	167	85	252
No future firearm violence	116	115	231
Unknown	66	50	116
Total	349	250	599

AI = assault-injured.

Appendix Table 3.

Relationship Between Future Firearm Violence and SaFETy Score and Assault Injury Presentation

Model	Validation (95% CI)	Training (95% CI)
1: Assault injury	2.14 (1.03–4.45)	1.89 (1.24–2.89)
2: SaFETY score	1.47 (1.23–1.79)	1.56 (1.41–1.75)
3
Assault injury	1.49 (0.67–3.32)	1.23 (0.75–2.00)
SaFETY score	1.44 (1.20–1.76)	1.54 (1.39–1.73)

SaFETy = Serious fighting, Friend weapon carrying, community Environment, and firearm Threats.

Appendix Table 4.

Description of Highest-Ranked Factors for Future Firearm Violence Among Those Presenting for Violent Injury

Factor	Response Type†	Importance Rank	Timeframe	Odds Ratio (95% CI)*	Standardized Odds Ratio
Received threats
Someone pulled a gun on you	Freq (0–6)	1	6 mo	1.91 (1.41–2.61)	2.17
Someone used a gun on you	Freq (0–6)	2	6 mo	2.44 (1.40–4.26)	1.81
Someone pulled a knife on you	Freq (0–6)	6	6 mo	1.37 (0.99–1.67)	1.34
Someone shot you	Freq (0–6)	9	6 mo	2.37 (1.27–4.43)	1.56
Someone threw something at you	Freq (0–6)	12	6 mo	1.41 (1.03–1.92)	1.41
Someone cut/stabbed you	Freq (0–6)	14	6 mo	1.80 (1.15–2.83)	1.50
Community
I have seen someone shot	Freq (0–3)	3	6 mo	1.92 (1.34–2.73)	1.82
I have heard guns shot	Freq (0–3)	5	6 mo	1.28 (0.99–1.67)	1.31
Seen gangs in neighborhood	Fred (0–3)	18	6 mo	1.19 (0.95–1.49)	1.24
My house was broken into	Freq (0–3)	20	6 mo	1.31 (0.86–1.99)	1.23
Friends
My friends carry weapons	Number (1–5)	10	Current	1.44 (1.09–1.91)	1.50
My friends smoke marijuana	Number (1–5)	15	Current	1.24 (0.99–1.54)	1.30
Friend legal trouble (drug-related)	Number (1–5)	17	Current	1.43 (1.00–2.05)	1.37
Partner violence
Partner used a knife on you	Freq (0–6)	13	6 mo	3.20 (1.07–9.61)	2.24
Fighting
Been in a serious fight	Freq (0–6)	4	6 mo	1.21 (0.99–1.47)	1.32
Put someone in the hospital	Freq (0–6)	7	6 mo	1.33 (1.00–1.77)	1.39
Drank before fighting	Freq (0–6)	8	6 mo	1.37 (1.00–1.88)	1.35
Other
Understand another’s point of view	Agree (1–5)	11	6 mo	1.37 (1.10–1.72)	1.48
Today’s ED visit for violent injury	Yes/No	16	Current	NA	NA
Unable to stop drinking	Freq (0–4)	19	6 mo	1.53 (0.94–2.50)	1.20

ED = emergency department; Freq = frequency; NA = not available.

^*CIs with lower bounds of 1.00 are entirely above 1.00.

^†Freq (0–6) measures frequency on a 7-point scale from 0 (never) to 6 (≥20 times); Freq (0–3) measures frequency on a 7-point scale from 0 (never) to 3 (many times); Freq (0–4) measures frequency on a 5-point scale from 0 (never) to 4 (daily); Number (1–5) measures frequency on a 5-point scale from 1 (none) to 5 (All); Agree (1–5) measures agreement on a 5-point scale from 1 (very true) to 5 (not true); Yes/No denotes a binary (1/0) indicator.

Appendix Table 5.

Description of Highest-Ranked Factors for Future Firearm Violence Among Those Not Presenting for Violent Injury

Factor	Response Type†	Importance Rank	Timeframe	Odds Ratio (95% CI)*	Standardized Odds Ratio
Received threats
Someone pulled a gun on you	Freq (0–6)	1	6 mo	3.82 (2.14–6.85)	4.96
Someone used a gun on you	Freq (0–6)	2	6 mo	7.03 (1.69–29.16)	3.66
Someone pulled a knife on you	Freq (0–6)	6	6 mo	4.25 (2.09–8.63)	3.87
Someone shot you	Freq (0–6)	9	6 mo	3.54 (1.06–11.79)	1.91
Someone threw something at you	Freq (0–6)	12	6 mo	1.85 (1.13–3.04)	1.85
Someone cut/stabbed you	Freq (0–6)	14	6 mo	3.44 (1.29–9.17)	2.33
Community
I have seen someone shot	Freq (0–3)	3	6 mo	1.75 (1.20–2.55)	1.67
I have heard guns shot	Freq (0–3)	5	6 mo	2.01 (1.41–2.85)	2.12
Seen gangs in neighborhood	Fred (0–3)	18	6 mo	1.53 (1.17–1.99)	1.70
My house was broken into	Freq (0–3)	20	6 mo	2.40 (1.43–4.01)	1.84
Friends
My friends carry weapons	Number (1–5)	10	Current	1.83 (1.30–2.56)	1.94
My friends smoke marijuana	Number (1–5)	15	Current	1.38 (1.05–1.81)	1.49
Friend legal trouble (drug-related)	Number (1–5)	17	Current	2.00 (1.32–3.03)	1.84
Partner Violence
Partner used a knife on you	Freq (0–6)	13	6 mo	3.73 (0.94–14.84)	2.49
Fighting
Been in a serious fight	Freq (0–6)	4	6 mo	1.80 (1.36–2.37)	2.41
Put someone in the hospital	Freq (0–6)	7	6 mo	2.62 (1.65–4.16)	3.02
Drank before fighting	Freq (0–6)	8	6 mo	3.37 (1.71–6.67)	3.13
Other
Understand another’s point of view	Agree (1–5)	11	6 mo	1.29 (0.97–1.73)	1.38
Today’s ED visit for violent injury	Yes/No	16	Current	NA	NA
Unable to stop drinking	Freq (0–4)	19	6 mo	1.68 (1.02–2.75)	1.46

ED = emergency department; Freq = frequency; NA = not available; OR = odds ratio.

^*CIs with lower bounds of 1.00 are entirely above 1.00.

^†Freq (0–6) measures frequency on a 7-point scale from 0 (never) to 6 (≥20 times); Freq (0–3) measures frequency on a 7-point scale from 0 (never) to 3 (many times); Freq (0–4) measures frequency on a 5-point scale from 0 (never) to 4 (daily); Number (1–5) measures frequency on a 5-point scale from 1 (none) to 5 (All); Agree (1–5) measures agreement on a 5-point scale from 1 (very true) to 5 (not true); Yes/No denotes a binary (1/0) indicator.

Appendix Table 6.

Sensitivity and Specificity in the Validation Set, Stratified by AI or Non-AI Group (95% CIs)

Threshold	Sensitivity (AI)	Specificity (AI)	Sensitivity (Non-AI)	Specificity (Non-AI)
1	39/41 = 95.1% (82.2–99.2%)	0/27 = 0.0% (0.0%–15.5%)	22/22 = 100.0% (81.5%–100.0%)	9/31 = 29.0% (14.9%–48.2%)
2	36/41 = 87.8% (73.0–95.4)	7/27 = 25.9% (11.9%–46.6%)	17/22 = 77.3% (54.2%–91.3%)	20/31 = 64.5% (45.4%–80.2%)
3	31/41 = 75.6% (59.4–87.1%)	13/27 = 48.1% (29.2%–67.6%)	12/22 = 54.5% (32.7%–74.9%)	23/31 = 74.2% (55.1%–87.5%)
4	26/41 = 63.4% (46.9–77.4)	16/27 = 59.3% (39.0%–77.0%)	10/22 = 45.5% (25.1%–67.3%)	24/31 = 77.4% (58.5%–89.7%)
5	23/41 = 56.1% (39.9%–71.2%)	21/27 = 77.8% (57.3%–90.6%)	9/22 = 40.9% (21.5%–63.3%)	27/31 = 87.1% (69.2%–95.8%)
6	12/41 = 29.3% (16.6%–45.7%)	26/27 = 96.3% (79.1%–99.8%)	7/22 = 31.8% (14.7%–54.9%)	29/31 = 93.5% (77.2%–98.9%)
7	5/41 = 12.2% (4.6%–27.0%)	27/27 = 100.0% (84.5%–100.0%)	2/22 = 9.1% (1.6%–30.6%)	30/31 = 96.8% (81.5%–99.8%)
8	5/41 = 12.2% (4.6%–27.0%)	27/27 = 100.0% (84.5%–100.0%)	1/22 = 4.5% (0.2%–24.9%)	31/31 = 100.0% (86.3%–100.0%)
9	5/41 = 12.2% (4.6%–27.0%)	27/27 = 100.0% (84.5%–100.0%)	1/22 = 4.5% (0.2%–24.9%)	31/31 = 100.0% (86.3%–100.0%)
10	3/41 = 7.3% (1.9%–21.0%)	27/27 = 100.0% (84.5%–100.0%)	1/22 = 4.5% (0.2%–24.9%)	31/31 = 100.0% (86.3%–100.0%)

AI = assault-injured.

Appendix Table 7.

Frequency Tables of the SaFETy Score in the Validation Data Set, Stratified by Group Membership

Score	Non-AI Group	AI Group
0	9 (17.0%)	2 (2.9%)
1	16 (30.2%)	10 (14.7%)
2	8 (15.1%)	11 (16.2%)
3	3 (5.7%)	8 (11.8%)
4	4 (7.5%)	8 (11.8%)
5	4 (7.5%)	16 (23.5%)
6	6 (11.3%)	8 (11.8%)
7	2 (3.8%)	0 (0.0%)
8	0 (0.0%)	0 (0.0%)
9	0 (0.0%)	2 (2.9%)
10	1 (1.9%)	3 (4.4%)
Totals	53 (100.0%)	68 (100.0%)

AI = assault-injured

SaFETy = Serious fighting, Friend weapon carrying, community Environment, and firearm Threats.

Appendix Table 8.

Sensitivity and Specificity in the Training Set, Stratified by AI or Non-AI Group (95% CIs)

Threshold	Sensitivity (AI)	Specificity (AI)	Sensitivity (Non-AI)	Specificity (Non-AI)
1	126/126 = 100.0% (96.3%–100.0%)	5/89 = 5.6% (2.1%–13.2%)	60/63 = 95.2% (85.8%–98.8%)	24/84 = 28.6% (19.5%–39.6%)
2	113/126 = 89.7% (82.7%–94.2%)	24/89 = 27.0% (18.4%–37.6%)	47/63 = 74.6% (61.8%–84.4%)	53/84 = 63.1% (51.8%–73.2%)
3	93/126 = 73.8% (65.1%–81.0%)	54/89 = 60.7% (49.7%–70.7%)	39/63 = 61.9% (48.8%–73.6%)	71/84 = 84.5% (74.6%–91.2%)
4	86/126 = 68.3% (59.3%–76.1%)	60/89 = 67.4% (56.6%–76.8%)	36/63 = 57.1% (44.1%–69.3%)	74/84 = 88.1% (78.8%–93.8%)
5	75/126 = 59.5% (50.4%–68.1%)	61/89 = 68.5% (57.7%–77.7%)	34/63 = 54.0% (41.0%–66.4%)	79/84 = 94.0% (86.0%–97.8%)
6	46/126 = 36.5% (28.3%–45.6%)	80/89 = 89.9% (81.2%–95.0%)	28/63 = 44.4% (32.1%–57.4%)	82/84 = 97.6% (90.9%–99.6%)
7	22/126 = 17.5% (11.5%–25.5%)	84/89 = 94.4% (86.8%–97.9%)	13/63 = 20.6% (11.9%–33.0%)	83/84 = 98.8% (92.6%–99.9%)
8	10/126 = 7.9% (4.1%–14.5%)	87/89 = 97.8% (91.4%–99.6%)	9/63 = 14.3% (7.1%–25.9%)	84/84 = 100.0% (94.6%–100.0%)
9	8/126 = 6.3% (3.0%–12.5%)	89/89 = 100.0% (94.8%–100.0%)	7/63 = 11.1% (5.0%–22.2%)	84/84 = 100.0% (94.6%–100.0%)
10	5/126 = 4.0% (1.5%–9.5%)	89/89 = 100.0% (94.8%–100.0%)	5/63 = 7.9% (3.0%–18.3%)	84/84 = 100.0% (94.6%–100.0%)

AI = assault-injured.

Appendix Table 9.

Frequency Tables of the SaFETy Score in the Training Data Set, Stratified by Group Membership

Score	Non-AI Group	AI Group
0	27 (18.4%)	5 (2.3%)
1	42 (28.6%)	32 (14.9%)
2	26 (17.7%)	50 (23.3%)
3	6 (4.1%)	13 (6.0%)
4	7 (4.8%)	12 (5.6%)
5	9 (6.1%)	48 (22.3%)
6	16 (10.9%)	28 (13.0%)
7	5 (3.4%)	15 (7.0%)
8	2 (1.4%)	4 (1.9%)
9	2 (1.4%)	3 (1.4%)
10	5 (3.4%)	5 (2.3%)
Total	147 (100.0%)	215 (100.0%)

AI = assault-injured

SaFETy = Serious fighting, Friend weapon carrying, community Environment, and firearm Threats.