Aggressive Behavior Risk Assessment Tool for Emergency Medical Services

Abstract

Objectives

Emergency medical services (EMS) clinicians are at risk of on-the-job violence that can result in serious injuries. A majority of such violence is associated with underlying behavioral health emergencies. However, no simple screening tools have been validated for use in prehospital settings. This study aimed to examine the utility of the 7-item Aggressive Behavior Risk Assessment Tool for Emergency Departments (ABRAT-ED) modified for prehospital settings in identifying patients at risk for violence.

Methods

This prospective study included adult and pediatric patients aged ≥10 years transported by 2 private EMS agencies between March 11 and April 21, 2024. EMS clinicians completed the modified ABRAT-ED before initiating transport and captured any subsequent physical assault, physical threat, or sexual harassment using the Violent Event Severity Tool. Logistic regression and receiver operating characteristics (ROC) analysis were performed.

Results

Of 11,292 EMS dispatches, 9,024 patients were evaluable, and 105 had ≥1 violent event (1.16 %). The logistic regression model resulted in a 3-item ABRAT-EMS, including agitation, confusion, and aggressive/threatening behaviors. ROC analysis showed the area under the curve of 0.89 (95% CI, 0.85-0.94; P < .001). At the cutoff score of 1, the sensitivity and specificity were 82.9% (95% CI, 74.0%-89.3%) and 88.1% (95% CI, 87.4%-88.8%), respectively, indicating that the ABRAT-EMS would correctly identify 82.9% of violent patients while correctly identifying 88.1% of nonviolent patients.

Conclusion

The ABRAT-EMS is a brief 3-item checklist with satisfactory sensitivity and specificity for identifying potentially violent patients during EMS encounters. The use of ABRAT-EMS can help EMS clinicians identify potentially violent patients and allow the implementation of proactive, targeted violence prevention measures.

The Bottom Line.

This prospective study of 9024 patients examined the utility of the 7-item Aggressive Behavior Risk Assessment Tool for Emergency Departments (ABRAT-ED) modified for screening patients at risk of violence during emergency medical services (EMS) encounters. The resulting 3-item ABRAT-EMS showed a sensitivity of 82.9% and a specificity of 88.1% for identifying patients at risk of violence in prehospital settings. Early identification of such patients may allow proactive implementation of targeted violence prevention measures. ABRAT-EMS and Violent Event Severity Tool could be useful components of workplace violence prevention programs.

1. Introduction

1.1. Background

Emergency medical services (EMS) clinicians experience a high rate of on-the-job violence that can result in serious injuries. Most of such violence is associated with underlying psychiatric disorders, substance use, delirium, or other medical conditions precipitating physical assault, physical threat, or sexual harassment against EMS clinicians.1, 2, 3, 4, 5 According to the US Bureau of Labor Statistics, violence-related injuries involving days away from work for EMS clinicians have markedly increased from 5.2 per 10,000 EMS clinicians in 2007 to 37.8 per 10,000 in 2020.⁶ Furthermore, the rate of injuries was almost 6-fold higher for EMS clinicians than for all US workers and 64% higher than that for nurses.

Despite the high rates of patient violence and related injuries, underreporting is common, which hinders accurate assessments of violence rates and outcomes of preventive measures.^7,8 Underreporting may be due to the misperception that violence is a part of the job, empathy for the patients in distress, as well as the lack of clear policies and reporting procedures.9, 10, 11, 12 A survey of 2171 EMS clinicians reported that two-thirds of the EMS agencies had no written policies and procedures regarding patient violence, whereas almost half reported not receiving adequate training.¹³ Patient violence can cause significant impacts on EMS clinicians’ physical and mental health, including physical injuries, posttraumatic stress disorder, anxiety, and burnout.^10,14, 15, 16 Therefore, there is an urgent need to address patient violence against EMS clinicians and implement effective violence prevention strategies.^13,17,18 Rapid identification of potentially violent patients before violent events occur is the first step in protecting the safety of EMS clinicians and their patients.^1,19

1.2. Importance

The National Association of State EMS Officials recommends using violence risk assessment tools to help guide the management of patients exhibiting violent behaviors.²⁰ The recommended tools include the Richmond Agitation Sedation Scale (RASS), the Altered Mental Status Scale (AMSS), and the Behavioral Activity Rating Scale (BARS). The RASS was initially developed to assess the level of consciousness and detect delirium among patients in intensive care units, with scores ranging from –5 (unarousable) to +4 (combative).^21,22 The AMSS scores mental responsiveness from –4 (unresponsive to mild stimulus) to +4 (combative/violent).²³ The BARS assesses agitation among patients with psychiatric emergencies, ranging from 1 (difficult or unable to rouse) to 7 (violent, requires restraints).^24,25 Although these tools are useful for assessing the patient’s level of consciousness, none of them have been validated for identifying patients at risk of violence before the occurrence of actual violent events in prehospital settings.

Although emergency departments (EDs) and psychiatric units are the most common health care settings for patient violence, many of the same patients are encountered by EMS clinicians in prehospital settings.26, 27, 28 The Aggressive Behavior Risk Assessment Tool for ED (ABRAT-ED) has been previously tested and validated with satisfactory sensitivity and specificity in identifying patients at high risk of violence among 10,554 patients visiting EDs.²⁹ In addition, a prospective interventional study of 77,424 ED patients showed that signage posting for the high-risk patients identified with ABRAT-ED screening resulted in a 27% reduction in violence compared with baseline.³⁰ However, it is not known whether the validated ABRAT-ED would be useful during the EMS encounters.

1.3. Goals of This Investigation

The purpose of this study was to determine the utility of the 7-item ABRAT-ED modified for prehospital settings to identify patients at risk of violence during EMS encounters and to revise the tool. The specific aims were to determine the prevalence of violent events and their severities, select a parsimonious set of modified ABRAT-ED items useful for EMS (ABRAT-EMS), and estimate its sensitivity and specificity for identifying potentially violent patients. In the current study, the violent events included physical assault, physical threat, and sexual harassment. The prevalence of violent events was defined as the number of events per 100 EMS encounters.

2. Methods

2.1. Study Design

A prospective study was conducted among patients transported by 2 private EMS agencies between March 11 and April 21, 2024. A survey comprising the modified ABRAT-ED items and violent events was created on the Qualtrics online platform (Qualtrics) linked to a quick response code. The EMS clinicians used company-issued cell phones or iPads to enter the data. Before starting the data collection, EMS clinicians at both sites received education via PowerPoint slides and a flier from each site’s representatives. The site representatives were available to provide clarification or answer questions during the data collection phase, and regular meetings were held to discuss progress. In addition, the EMS clinicians were reminded to complete the data collection checklists via text messages, emails, face-to-face interactions, posters, and stickers throughout the EMS stations and vehicles.

The Institutional Review Board at Point Loma Nazarene University approved this study protocol. A waiver of written informed consent was granted because this study aimed to identify potentially violent patients, which could not be carried out without a waiver. In addition, the patient assessment data for the study were part of the routine EMS clinical evaluations. After the study completion, the de-identified patient data were retrieved in compliance with the Health Insurance Portability and Accountability Act Privacy Rule.

2.2. Setting and Participants

All adult and pediatric patients aged ≥10 years receiving EMS care at 2 sites associated with Global Medical Response, a private company that provides emergency medical services, were included in the study. The first site was an American Medical Response in Rochester, New York, with 175 EMS clinicians who respond to 79,000 requests for service annually, most of which were 911 emergency services. A second site was A Better Choice Ambulance in Phoenix, Arizona, with 130 EMS clinicians who respond to an annual 27,000 requests, mainly interfacility transfers with some 911 emergency services.

2.3. Measurements

This study used the 7-item ABRAT-ED checklist modified for prehospital settings to identify patients at risk of violence. The original 7-item ABRAT-ED demonstrated satisfactory sensitivity and specificity for identifying potentially violent patients in a study of 10,554 patients visiting EDs.²⁹ This yes-no 7-item checklist is composed of 2 historical indicators of mental illness and aggression; 4 behavioral indicators of agitation, staring, confusion, and aggressive/threatening behaviors; and 1 reason-for-ED visit item with 4 mutually exclusive choices, including suicidal ideation/suicide attempt, behavioral health concern, psychiatric evaluation, and aggressive behavior. The sensitivity and specificity of the original ABRAT-ED at the cutoff score of 4 for identifying patients at high risk for violence were 70.1% and 98.9%, respectively. In the current study, the reason-for-ED visit item was modified as the dispatch reason, and the psychiatric evaluation was replaced with drug/alcohol use, resulting in the modified 7-item ABRAT-ED.

The current study used a modified Violent Event Severity Tool (VEST) that includes physical assault, physical threat, and sexual harassment for the collection of violent events or their absence.³¹ Also, an option for entering other types of violence in text format was allowed. The modified VEST items focused on the most critical types of violence to minimize the data collection burden. These violent event types were derived from the revised definition of workplace violence from The Joint Commission.³² The VEST provides operational definitions of each violence type and severity to objectively capture the violent events and 4 levels of severity for each type of violence, ranging from grade 1 (mild) to grade 4 (life-threatening). For example, physical assault is defined as violence involving physical contact with intent to cause bodily harm. The grade 1 physical assault is associated with no physical injury, whereas grade 4 corresponds to life-threatening physical injury. The patient age and incident complaint data were collected from the EMS Patient Care Report (PCR) database.

2.4. Data Collection Procedures

Following dispatch and arrival at the scene, the EMS clinicians performed routine patient assessments, including the modified ABRAT-ED, before loading the patient into the ambulance. Modified VEST data were completed after arrival at the destination and transfer of care. The ABRAT and VEST data were matched with the de-identified PCR data using the unique run number and unit call sign.

2.5. Statistical Analysis

Descriptive statistics of means, ranges, frequencies, and percentages were calculated to summarize patient ages, incident complaints, ABRAT-ED item results, and violent event data from VEST. Each ABRAT-ED item and violent event were coded as dichotomous variables (yes = 1; no = 0). Kendall's tau correlation procedure was first performed to select a parsimonious set of predictors for violence from ABRAT-ED items. Then, the statistically significant potential predictors were entered into a multivariable logistic regression model with backward elimination to remove nonsignificant ABRAT-ED items. The odds ratio (OR) associated with each remaining ABRAT-ED item was used to assign the weighting value to calculate violence risk summation scores. The final parsimonious set of predictors for violence with assigned weights was named ABRAT-EMS.

The receiver operating characteristics (ROC) analysis was then performed to determine the cutoff scores for the ABRAT-EMS in predicting violent events and estimate the corresponding sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). In the ROC analysis, the area under the curve (AUC) estimates the ability of the ABRAT-EMS to distinguish between violent and nonviolent patients. The SPSS version 29.0 (IBM Corporation) was used for all statistical analyses, and the significance level was set at P value <.05.

3. Results

3.1. Characteristics of Study Participants

A total of 9024 out of 11,292 patient dispatches were evaluable for data analyses following the exclusion of 2268 patients due to missing ABRAT-ED and VEST assessments and other reasons (Fig 1). The patients with missing ABRAT-ED and VEST assessments had characteristics similar to those of the evaluable patients. The ABRAT-ED and VEST data completion rate was 83.3%. Of the evaluable patients, 105 had ≥1 violent events, resulting in a violent event rate of 1.16%.

Figure 1.

Study flow chart.

Table 1 shows the patient characteristics and ABRAT-ED data for the overall population as well as for the patients with violent events. The mean age for the patients with violent events was slightly younger than the overall population (42 vs 47 years). The most common incident complaints were psychiatric, abnormal behavior, suicidal, or suicide attempts for the overall population as well as for the patients with violent events. The patient’s sex, race, and ethnicity data were not collected in this study because of the institutional policy. A higher proportion of the patients with violent events had positive ABRAT-ED items compared with the overall patient population, including mental illness history (54.3% vs 23.0%), aggressive/threatening behaviors (71.4% vs 2.4%), and agitation (74.3% vs 7.6%). A small fraction of the patients (3.5%) had missing age and dispatch information due to the inability to match the ABRAT-ED and VEST data with the PCR data.

Table 1.

Patient characteristics (N = 9024).

Characteristic variables	Overall population (N = 9024)	Patients with violent events (n = 105)
Age, y, mean ± SD	47 ± 20	42 ± 16
PCR incident complaints (≥1%)
Psychiatric/abn. behavior/suicidal/suicide attempt	2155 (23.9)	35 (33.3)
Sick person	1037 (11.5)	6 (5.7)
Breathing problem	651 (7.2)	8 (7.6)
Abdominal pain/problems	520 (5.8)	0
Chest pain	488 (5.4)	4 (3.8)
Pain	426 (4.7)	2 (1.9)
Fall	422 (4.7)	3 (2.9)
Overdose/poisoning/ingestion	219 (2.4)	8 (7.6)
Dizziness/syncope/fainting	207 (2.3)	2 (1.9)
Seizure/convulsions	193 (2.1)	7 (6.7)
Nausea/vomiting	174 (1.9)	0
Unknown problem/person down	161 (1.8)	5 (4.8)
Altered mental status	154 (1.7)	6 (5.7)
Transfer/interfacility/palliative care	151 (1.7)	0
Traffic/transportation/motorcycle event	149 (1.7)	1 (1.0)
Medical transport	135 (1.5)	0
Traumatic injury	119 (1.3)	1 (1.0)
Assault	115 (1.3)	1 (1.0)
Stroke/CVA	106 (1.2)	1 (1.0)
Diabetic problem	105 (1.2)	1 (1.0)
Hemorrhage/laceration	98 (1.1)	2 (1.9)
Modified ABRAT-ED items
History of mental illness	2077 (23.0)	57 (54.3)
History of aggression	557 (6.2)	30 (28.6)
Aggressive/threatening	219 (2.4)	75 (71.4)
Agitation	688 (7.6)	78 (74.3)
Confusion	566 (6.3)	27 (25.7)
Staring	411 (4.6)	34 (32.4)
Dispatch reasons
Behavioral health concern	1826 (20.2)	46 (43.8)
Suicidal ideation/suicide attempt	998 (11.1)	16 (15.2)
Alcohol/drug abuse	697 (7.7)	28 (26.7)
Aggressive behavior	444 (4.9)	36 (34.3)

Values are expressed as n (%) unless indicated otherwise.

abn., abnormal; ABRAT-ED, Aggressive Behavior Risk Assessment Tool for Emergency Departments; CVA, cerebrovascular accident; PCR, patient care report.

Among patients with violent events, 91, 59, and 13 patients exhibited various grades of physical threats, physical assaults, and sexual harassment, respectively, with 66 patients exhibiting more than 1 type of violence (Fig 2). Among 59 physical assaults, 41 were grade 1 or mild assaults causing no physical injury, 12 were grade 2 resulting in physical injury that did not require medical attention, and 6 were grade 3 associated with physical injury requiring medical attention. No grade 4 or life-threatening physical assaults occurred in the current study.

Figure 2.

Violence types and severity (n = 105). Note: The percentages are the proportions of grades within each type of violence. Sixty-six patients exhibited more than one type of violence and no grade 4 (life-threatening) event occurred during the study.

3.2. Sensitivity and Specificity of the ABRAT-EMS

Table 2 presents the results of bivariate Kendall tau tests showing correlations between ABRAT-ED items, age, and violent events. All ABRAT-ED items except suicidal ideation/suicide attempt correlated significantly with the violent events. These significant items were entered into the multivariable logistic regression model, which resulted in only 3 items emerging as predictors of violent events: aggressive/threatening (OR, 43; 95% CI, 23-80), agitation (OR, 5.2; 95% CI, 2.8-9.9), and confusion (OR, 1.8; 95% CI, 1.0-3.2). Based on the OR from the model, item-weight values were assigned for calculating violence risk summation scores, as shown in Table 3. Thus, the parsimonious set of 3 items with assigned weights that best predicts violent events was named ABRAT-EMS. As an example, for an agitated patient exhibiting aggressive/threatening behavior, the ABRAT-EMS summation score would be 6 (ie, 2 + 4).

Table 2.

Bivariate correlations with violent events (N = 9024).

Variables	r
Age	–0.02∗
History
Mental illness	0.08∗∗∗
Aggression	0.10∗∗∗
Behavior
Aggressive/threatening	0.49∗∗∗
Agitation	0.27∗∗∗
Confusion	0.09∗∗∗
Staring	0.15∗∗∗
Dispatch reason
Behavioral health concern	0.06∗∗∗
Suicidal ideation/suicide attempt	0.01
Alcohol/drug abuse	0.08∗∗∗
Aggressive behavior	0.15∗∗∗

∗P < .05; ∗∗P < .01; ∗∗∗P < .001 by bivariate correlation with Kendall tau test.

Table 3.

Multivariable logistic regression model for predicting violent events (N = 9024).

Predictors	OR	95% CI	P	Weight value
Aggressive/threatening	43	23-80	<.001	4
Agitation	5.2	2.8-9.9	<.001	2
Confusion	1.8	1.0-3.2	.039	1

OR, odds ratio.

The percentages of patients with violent events at various ABRAT-EMS scores are shown in Figure 3. Among 7877 patients with an ABRAT-EMS score of zero, only 18 were violent (0.2%). Of the 928 patients with ABRAT-EMS scores between 1 and 3, 12 were violent (1.3%). In contrast, 75 of 219 patients with an ABRAT-EMS score of ≥4 became violent (34.2 %). As shown in Figure 4, the AUC from the ROC analysis for the ABRAT-EMS was 0.89 (95% CI, 0.85-0.94; P < .001).

Figure 3.

Percentage of violent patients vs ABRAT-EMS scores. Note. The ratios shown are violent patients/total patients at various ABRAT-EMS scores. ABRAT-EMS, Aggressive Behavior Risk Assessment Tool for Emergency Medical Services.

Figure 4.

Receiver operating characteristics (ROC) curve: ABRAT-EMS. The diagonal reference line represents the line of no discrimination with an AUC (area under the curve) of 0.5. The solid curve represents the ABRAT-EMS with an AUC of 0.89 (95% CI, 0.85-0.94; P < .001). ABRAT-EMS, ABRAT for Emergency Medical Services.

The sensitivities, specificities, PPVs, and NPVs at various cutoff scores of ABRAT-EMS are shown in Table 4. The cutoff score of 1 has a sensitivity and specificity of 82.9% and 88.1%, respectively. It indicates that the ABRAT-EMS would correctly identify 82.9% of violent patients (true positive) while correctly identifying 88.1% of nonviolent patients (true negative). At an alternate cutoff score of 4, the PPV and NPV were 34.2% and 99.7%, respectively. The PPV of 34.2% indicates that about one-third of the patients with scores of ≥4 will become violent. In contrast, the NPV of 99.7% indicates that almost all the patients with the ABRAT-EMS scores <4 will be nonviolent.

Table 4.

Characteristics of the summation score cutoffs for ABRAT-EMS.

Score cutoff	Sensitivity (%) (95% CI)	Specificity (%) (95% CI)	PPV (%) (95% CI)	NPV (%) (95% CI)
1	82.9 (74.0-89.3)	88.1 (87.4-88.8)	7.6 (6.2-9.3)	99.8 (99.6-99.9)
2	80.0 (70.8-86.9)	92.7 (92.2-93.3)	11.4 (9.3-14.1)	99.7 (99.6-99.8)
3	74.3 (64.7-82.1)	97.3 (96.9-97.6)	24.3 (19.8-29.4)	99.7 (99.5-99.8)
4	71.4 (61.7-79.6)	98.4 (98.1-98.6)	34.2 (28.1-41.0)	99.7 (99.5-99.8)
5	65.7 (55.7-74.5)	98.7 (98.4-99.0)	37.9 (30.9-45.4)	99.6 (99.4-99.7)
6	65.7 (55.7-74.5)	98.8 (98.6-99.0)	39.4 (32.2-47.1)	99.6 (99.4-99.7)
7	20.0 (13.1-29.2)	99.8 (99.6-99.9)	50.0 (34.4-65.6)	99.1 (98.8-99.2)

ABRAT-EMS, Aggressive Behavior Risk Assessment Tool for Emergency Medical Services; NPV, negative predictive value; PPV, positive predictive value.

4. Limitations

The current study has several limitations. First, although this study was based on data from more than 9000 patients, only 105 patients had violent events, which limits the precision of the reported sensitivity and specificity. Second, the data were collected for only 6 weeks in early spring, which may limit the applicability of study findings to other seasons. Third, only private EMS agencies were included in the study and the results may not be generalizable to public EMS agencies. Finally, the study findings may not be generalizable to other settings because it was conducted in 2 states in the USA, associated with a single agency. Further studies are needed to confirm the findings as well as to determine whether proactive interventions targeting the small subset of patients at high risk of violence could reduce violent events.

4. Discussion

The ABRAT-EMS is a brief risk assessment checklist with satisfactory sensitivity and specificity for identifying patients at risk for violence in prehospital settings. This simple 3-item checklist comprises 3 behavioral indicators of agitation, confusion, and aggressive/ threatening behaviors that EMS clinicians can rapidly assess shortly after encountering a patient on the scene. The availability of such a validated screening tool to identify potentially violent patients could allow the implementation of targeted interventions feasible for enhanced safety of EMS clinicians and patients. In contrast to the ABRAT-ED, the ABRAT-EMS does not include 2 historical items and the dispatch-reason item. We speculate that this is due to difficulty obtaining accurate historical patient data or reliable reasons for dispatch during short emergency encounters.

For calculating the ABRAT-EMS summation score, a weight value ranging from 1 to 4 was assigned to each item based on the OR, which reflects the strength of the item’s association with violent events. The ROC analysis based on the ABRAT-EMS scores showed an AUC of 0.89, which indicates a high likelihood of discriminating between violent and nonviolent patients. An AUC of 1.00 indicates a perfect discriminant ability, whereas 0.50 indicates no discriminant ability.³³

For the classification of violence risks, we suggest 3 categories of violence risks based on the ABRAT-EMS scores: high risk (scores of ≥4), medium risk (scores of 1-3), and low risk (scores of 0) for violence. For the high-risk category with scores of ≥4, a sensitivity of 71.4% implies the detection of almost three-quarters of violent patients. In the current study, 75 patients were violent out of 219 patients with ABRAT-EMS scores ≥4 (34.2%). For this group of high-risk patients, EMS clinicians could consider applying appropriate violence preventive measures. In contrast, only 18 of 7877 patients (0.2%) with ABRAT-EMS scores of 0 became violent, and no preventive measures are likely needed. The medium-risk patients with ABRAT-EMS scores of 1 to 3 have some risk of violence (1.3%), and de-escalation techniques or reassessments may be appropriate.

The National Institute for Occupational Safety and Health recommends that EMS employers establish policies and implement workplace violence prevention programs for the safety of EMS clinicians.³² To accurately monitor the effects of such programs, it is essential to use objective and standardized tools to collect violent events and their severities.³⁴ The VEST was used in the current study to meet such needs. The overall incidence rate of patient violence was 1.16%, indicating that slightly more than 1 of 100 patients cared for by EMS clinicians became violent. This incidence rate is consistent with 1.2% to 1.7% reported in previous ED studies.^29,35

In conclusion, this study of more than 9000 patients showed the utility of ABRAT-EMS in identifying potentially violent patients with satisfactory sensitivity and specificity. It is a brief 3-item violence risk assessment checklist that can be easily and rapidly completed on arrival at the scene of an EMS call. The combined use of ABRAT-EMS and VEST may allow rapid identification of high-risk patients, implementation of focused mitigation measures, and monitoring of the effects of such violence prevention programs.

Author Contributions

SCK, KD, and BC conceived and designed the study. CY, GW, AM, ED, CB, SB, and SK contributed to the study design. KD, BC, CY, GW, AM, ED, CB, and SB contributed to the data collection. SCK and SK performed statistical analysis and interpretation. SCK drafted the manuscript, and all authors contributed to reviewing and editing the final draft.

Funding and Support

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Conflict of Interest

SCK is the copyright holder of the Aggressive Behavior Risk Assessment Tools (ABRAT-ED, ABRAT-EMS) and the Violent Event Severity Tool (VEST). She is the sole proprietor of Son Chae Kim, PhD, LLC. Other authors declare no conflict of interest.