A review of performance assessment tools for rescuer response in opioid overdose simulations and training programs

Highlights

• •This literature review included 31 articles that assessed rescuer-centered outcome measures.
• •There are no validated outcome measures for measuring proficiency of opioid overdose response.
• •The comparison of existing programs is difficult without validated proficiency measures.

Abstract

Since the 1990s, more than 600 overdose response training and education programs have been implemented to train participants to respond to an opioid overdose in the United States. Given this substantial investment in overdose response training, valid assessment of a potential rescuers’ proficiency in responding to an opioid overdose is important. The aim of this article is to review the current state of the literature on outcome measures utilized in opioid overdose response training. Thirty-one articles published between 2014 and 2020 met inclusion criteria. The reviewed articles targeted laypersons, healthcare providers, and first responders. The assessment tools included five validated questionnaires, fifteen non-validated questionnaires, and nine non-validated simulation-based checklists (e.g., completion of critical tasks and time to completion). Validated multiple choice knowledge assessment tools were commonly used to assess the outcomes of training programs. It is unknown how scores on these assessment tools may correlate with actual rescuer performance responding to an overdose. Seven studies reported ceiling effects most likely attributed to participants’ background medical knowledge or experience. The inclusion of simulation-based outcome measures of performance, including the commission of critical errors and the time to naloxone administration, provides better insight into rescuer skill proficiency.

1. Introduction

In 2018, 67,367 Americans died from drug overdose, a 4.1% decline from 2017 (NIDA, 2019, Hedegaard et al., 2020). However, there was a 10% increase in drug overdose deaths involving synthetic opioids such as fentanyl (Hedegaard et al., 2020, Han et al., 2019).

Naloxone is an emergency antidote for opioid overdose. A competitive antagonist of the mu opioid receptor, naloxone can reverse the effects of an overdose within 2–5 min by displacing the opioid agonist (Skolnick, 2018). Narcan® (a single-step nasal spray) and Evzio® (an auto-injector providing voice-instruction) are naloxone formulations approved by the FDA for layperson usage. Prefilled naloxone syringes with mucosal atomizers and vials/syringes for IM injection are improvised naloxone products also distributed in communities (pg. 9) (Jiang, 2018).

With the goal of reducing mortality, overdose response training programs, including the distribution of naloxone to laypersons, were implemented in Europe and Australia beginning in the 1990 s (Sporer and Kral, 2007, Dwyer et al., 2018). In 1999, naloxone was first distributed to non-medical providers in the United States through underground programs in Chicago and San Francisco (McDonald et al., 2017). Lambdin et al. (2018) reported that, at the end of 2014, 8% of United States counties (259outof3,142) had an overdose response training program according to the most recent information chronicled by the Harm Reduction Coalition (Lambdin et al., 2018). Since then, training programs have proliferated in the United States, although statistics describing the number and activity of current programs are not available. Programs are designed and initiated through various state, private, and nonprofit organizations such as syringe exchange clinics, homeless shelters, emergency medical services, social service agencies, libraries, emergency departments, health care providers, substance use disorder treatment programs, and pharmacies (Weiner et al., 2019). Multiple studies have demonstrated that naloxone in the community can decrease opioid overdose deaths (Weiner et al., 2019, Walley et al., 2013).

Overdose response training programs were designed to help participants identify overdoses and provide appropriate emergency response (Lewis et al., 2017). Balancing accessibility with rigor of instruction, they vary in duration and format, from 10-minute web-based educational programs to eight hour in-person hands-on instruction (Education, 2019, Seal et al., 2005). Different organizations have developed resources and training guidelines for communities to utilize, including the 2015 American Heart Association (AHA) overdose response algorithm (Lavonas et al., 2015), the 2018 overdose response toolkit developed by the Substance Abuse and Mental Health Services Administration (SAMHSA) (SAMHSA. SAMHSA Opioid Overdose Prevention TOOLKIT Opioid Use Disorder Facts Five Essential Steps for First Responders Information for Prescribers Safety Advice for Patients amp; Family Members Recovering From Opioid Overdose [Internet]., 2018), and Prescribe to Prevent (https://prescribetoprevent.org/) developed by clinical and research experts (PrescribeToPrevent – Prescribe Naloxone, Save a Life [Internet]. [cited, 2020).

In determining the most effective use of overdose response resources, including the most appropriate overdose response training or naloxone preparation, organizations are faced with competing priorities and limited evidence for relative value. Previous researchers have pointed out the limitations of current research on overdose response training effectiveness, including study quality and a lack of randomized trials (Clark et al., 2014, Orkin et al., 2015, Orkin et al., 2019). Controversies remain. Despite the AHA guidelines, in 2016 the New York State Department of Public Health reported that there was insufficient data to recommend that programs train participants in compression-only cardiopulmonary resuscitation (CPR) and/or rescue breathing (Stancliff et al., 2016). Latkin et al. (2019) reported that 54% (n = 316) of participants, with drug use experience, felt they needed more training to respond to an overdose (Latkin et al., 2019).

The authors undertake a review of the literature reporting on assessment tools used to evaluate rescuer-centered outcomes regarding rescuer response to an opioid overdose. This includes studies that directly evaluated one or more opioid overdose response programs and studies that examined other components of opioid overdose response, such as choice of naloxone preparation available to the rescuer. Since the last published review by (Clark et al., 2014), almost 4× more additional articles on overdose response training have been published. Given how quickly the science on overdose response training is evolving, an updated review and summary could be helpful. The purpose of the review is to describe the outcomes measures that have been used and, if feasible, make a comparison between the utility of those outcome measures to determine if a set of common variables related to proficient opioid overdose response may emerge.

1.1. Methods

Two research staff followed PRISMA guidelines and separately screened titles and abstracts for inclusion (Fig. 1) (Moher et al., 2009). PubMed, MEDLINE, and PscyhINFO online databases were searched using the Boolean search terms: (opiate OR opioid) AND (overdose education) AND (naloxone administration). Additionally, a second search using the Boolean search query: (opiate OR opioid) AND (overdose) AND (simulation) AND (naloxone). A manual review of relevant references cited in the included studies was also conducted to identify additional articles for consideration. We included original peer-reviewed articles published between 2014 and 2020 that incorporated a pre-to-post assessment of rescuer-centered outcomes using validated questionnaires, non-validated questionnaires, and/or studies that included simulation-based outcome measures.

Fig. 1.

PRISMA Flow Chart of Articles Included (Aizen et al., 2018, Cash et al., 2018, Coleman, 2018, Dunn et al., 2016, Dunn et al., 2017, Dwyer et al., 2015, Espelt et al., 2017, Faul et al., 2015, Faul et al., 2017, Giglio et al., 2015, Gulec et al., 2018, Haffajee et al., 2019, Heavey et al., 2018, Hill et al., 2018, Jacobson et al., 2018, Jawa et al., 2020, Jones et al., 2014, Keane et al., 2018, Keenan et al., 2017, Kilwein et al., 2019, Kim et al., 2019, Kirane et al., 2016, Lewis et al., 2016, Madah-Amiri et al., 2017, Morris and Kleinman, 2020, Mueller et al., 2015, Nandakumar et al., 2019, Neale et al., 2019, Nielsen et al., 2016, Noveloso et al., 2020, Oliva and Bounthavong, 2017, Panther et al., 2017, Peckham and Boggs, 2016, Raffa et al., 2017, Rando et al., 2015, Ryan and Dunne, 2018, Rzasa Lynn and Galinkin, 2018, Salerno et al., 2018, Schartel et al., 2018, Sumner et al., 2016, Taylor et al., 2018, Weiner et al., 2017, Williams et al., 2019).

F.E. and A.S. assessed the quality of the included studies using a scale adapted from the Clark et al. (2014) review (Clark et al., 2014). Scores for each of the seven categories were based on a 0–1 scale, and a score of 0 or 0.5 was given if the study did not include or sufficiently describe (respectively) the methodology for each category.

2. Results

2.1. Identification and description of articles

By searching PubMed, MEDLINE, and PsychINFO, 380 articles were retrieved after removing duplicates. Thirty-one articles were included in the final comparative analysis (Fig. 1). Nine studies incorporated simulation-based outcome measures (Edwards et al., 2015, Eggleston et al., 2018, Eggleston et al., 2019, Franko et al., 2019, Goldberg et al., 2018, Kobayashi et al., 2017, Kim et al., 2016, Krieter et al., 2016, McDermott and Collins, 2012). Study quality scores ranged from 5 to 7 (mean = 5.9, median = 6, mode = 7) for the opioid overdose response training studies (Table 1). Nine opioid overdose response training studies received a perfect score of seven. Seven out of nine of the studies that received a perfect score included simulation.

Table 1.

Quality Ratings of Reviewed Articles.

Authors (Publication Year)	Research Questions/Hypothesis Are Clear and Appropriate	Sample Size is Stated	Randomization	Attrition Rate is Recorded and Discussed	Data Analysis is Described and Rigorous	Results are Clearly Described	Reproducibility	Total Score
Ashrafioun et al. (2016)	1	1	0	0.5	1	1	1	5.5
Behar et al. (2015)	0.5	1	0	1	1	1	0.5	5
Bergeria et al. (2019)	1	1	1	1	1	1	1	7
Berland et al. (2017)	1	1	0	1	1	0.5	1	5.5
Crocker et al. (2019)	1	1	0	1	1	1	1	6
Dahlem et al. (2017)	0.5	1	0	1	1	1	0	4.5
Dietze et al. (2018)	1	1	0	0.5	1	1	0	4.5
Dion et al. (2016)	1	1	0	1	1	0.5	0	4.5
Edwards et al. (2015)	1	1	1	1	1	1	1	7
Eggleston et al. (2018)	1	1	1	1	1	1	1	7
Eggleston et al. (2019)	1	1	1	1	1	1	1	7
Franko et al. (2019)	1	1	1	1	1	1	1	7
Giordano et al. (2020)	1	1	1	1	1	1	1	7
Goldberg et al. (2018)	1	1	0	1	1	1	1	6
Hargraves et al. (2019)	1	1	0	1	0.5	1	0.5	5
Heavey et al. (2018)	1	1	0	1	1	1	0.5	5.5
Huhn et al. (2018)	1	1	1	1	1	1	1	7
Kim et al. (2016)	1	1	0	1	1	1	1	6
Klimas et al. (2015)	1	1	0	1	1	1	0.5	5.5
Kobayashi et al. (2018)	1	1	1	1	1	0.5	1	6.5
Krieter et al. (2016)	1	1	1	1	1	1	1	7
Kwon et al. (Kwon et al., 2020)	0.5	1	0	1	1	1	1	5.5
McDermott et al. (2012)	1	1	1	1	1	1	1	7
Monteiro et al. (2017)	0.5	1	0	1	1	1	1	5.5
Pade et al. (2016)	0.5	1	0	1	1	1	0	4.5
Petterson et al. (2017)	1	1	0	1	1	1	0.5	5.5
Pietrusza et al. (2018)	0.5	1	0	1	1	1	1	5.5
Saucier et al. (2016)	1	1	0	1	1	1	1	6
Wagner et al. (2016)	1	1	0	1	1	1	1	6
Williams et al. (2014)	1	1	1	1	1	1	0.5	6.5
Zhang et al. (2018)	1	1	0	1	1	1	1	6

Receipt of a quality appraisal score of 1 was tabulated as follows; Category 1, stated aims, research questions, or hypotheses, and was not a quality improvement study; Category 2, sample size must be stated; Category 3, included randomization methodology; Category 4, stated number of participants in analysis; Category 5, provided the type of statistical tests ran; Category 6, provided figures/tables, and scores for assessment scale items; Category 7, provided scales/items.

2.1.1. Target populations

Rescuer-centered outcome measures were reported evaluating laypersons (n = 2169), first responders (EMTs/LEOs) (n = 687) and healthcare professionals (n = 668) (Table 2). The average and median age of participants was 35.8 (SD = 8.2) years and 47.7 (SD = 5.8) for the 31 included studies, respectively. Women made up roughly 45.5% of the participants in the included studies. The race of most participants across the included studies was white. Four studies occurred outside of the U.S. (McDermott and Collins, 2012, Dietze et al., 2018, Klimas et al., 2015, Petterson and Madah-Amiri, 2017).

Table 2.

Target populations of opioid overdose response training.

	Studies without simulation-based outcome measures	Studies with simulation-based outcome measures
Laypersons (Lay)	9	6
At-Risk Population	n = 1123	n = 85
Family/Friends	n = 408	n = 553
Total	n = 1531	n = 638
First Responders (FR)	5	1
EMTs	n = 117	n = 18
Law Enforcement Officers/Firefighters	n = 552	n = 0
Total	n = 669	n = 18
Healthcare Professionals (HCPs)	8	2
Licensed	n = 164	n = 23
Trainees	n = 348	n = 133
Total	n = 512	n = 156

One study, Ashrafioun et al. (2016), targeted healthcare professionals and laypersons.

2.1.2. Training curricula and duration

Twenty-two studies assessed participants’ knowledge, attitude, and confidence after administering a training program, while nine studies used simulation-based outcome measures (Table 2). Across these studies training included risk factors for an opioid overdose, recognition of an opioid overdose, and naloxone administration (Table 3). Five studies provided overdose prevention in conjunction with basic life support (BLS) training or required previous CPR training by the participants (Klimas et al., 2015, Berland et al., 2017, Dion, 2016, Wagner et al., 2016, Zhang et al., 2018). Rescue breathing was included in ten studies (Kobayashi et al., 2017, Dietze et al., 2018, Petterson and Madah-Amiri, 2017, Bergeria et al., 2019, Heavey et al., 2018, Huhn et al., 2018, Pade et al., 2017, Saucier et al., 2016, Williams et al., 2014, Giordano et al., 2020). Twenty-one studies provided participants with demonstrations on how to assemble the naloxone preparation utilized (Edwards et al., 2015, Franko et al., 2019, Kobayashi et al., 2017, McDermott and Collins, 2012, Klimas et al., 2015, Petterson and Madah-Amiri, 2017, Berland et al., 2017, Dion, 2016, Wagner et al., 2016, Pade et al., 2017, Saucier et al., 2016, Williams et al., 2014, Giordano et al., 2020, Ashrafioun et al., 2016, Behar et al., 2015, Crocker et al., 2019, Dahlem et al., 2017, Hargraves et al., 2019, Pietrusza et al., 2018, Kwon et al., 2020, Monteiro et al., 2017). None of the included studies, except for the studies in conjunction with BLS/previous CPR training, provided instruction in use of an automated external defibrillator (AED), which is recommended in the 2015 AHA guidelines (pg. 19) (Lavonas et al., 2015). Two studies trained law enforcement officers on addiction treatment referral (Wagner et al., 2016, Dahlem et al., 2017)(Table 3).

Table 3.

Curriculum and duration of opioid overdose response training.

Curriculum	# of studies
Basic Life Support Training/Previous CPR Training	5
Risk factors for an opioid overdose	19
Recognition of an opioid overdose	24
Help-seeking (e.g. calling 9-1-1)	15
Naloxone administration	22
Device assembly demonstration	21
Only rescue breathing	10
AED	5*
Recovery position	12
Treatment referral	2
Duration
≤10 min	1
15–45 min	8
1 h	8
1.5 h	1
2.5–3 h	2

The following studies included the curriculum listed in Table 3: in conjunction with basic life support training/previous CPR training (Klimas et al., 2015, Berland et al., 2017, Dion, 2016, Wagner et al., 2016, Zhang et al., 2018), risk factors for an opioid overdose (Franko et al., 2019, Kobayashi et al., 2017, Ashrafioun et al., 2016, Dahlem et al., 2017, Pietrusza et al., 2018, Kwon et al., 2020, Dietze et al., 2018, Klimas et al., 2015, Petterson and Madah-Amiri, 2017, Berland et al., 2017, Dion, 2016, Wagner et al., 2016, Zhang et al., 2018, Bergeria et al., 2019, Heavey et al., 2018, Huhn et al., 2018, Pade et al., 2017, Saucier et al., 2016, Williams et al., 2014), recognition of an opioid overdose (Franko et al., 2019, Kobayashi et al., 2017, Dietze et al., 2018, Klimas et al., 2015, Petterson and Madah-Amiri, 2017, Berland et al., 2017, Dion, 2016, Wagner et al., 2016, Zhang et al., 2018, Bergeria et al., 2019, Heavey et al., 2018, Huhn et al., 2018, Pade et al., 2017, Saucier et al., 2016, Williams et al., 2014, Giordano et al., 2020, Ashrafioun et al., 2016, Behar et al., 2015, Crocker et al., 2019, Dahlem et al., 2017, Hargraves et al., 2019, Pietrusza et al., 2018, Kwon et al., 2020, Monteiro et al., 2017), help-seeking (Franko et al., 2019, Kobayashi et al., 2017, Wagner et al., 2016, Williams et al., 2014, Giordano et al., 2020, Hargraves et al., 2019, Pietrusza et al., 2018, Dietze et al., 2018, Klimas et al., 2015, Petterson and Madah-Amiri, 2017, Berland et al., 2017, Bergeria et al., 2019, Heavey et al., 2018, Huhn et al., 2018, Pade et al., 2017), device assembly demonstration (Edwards et al., 2015, Franko et al., 2019, Kobayashi et al., 2017, McDermott and Collins, 2012, Klimas et al., 2015, Petterson and Madah-Amiri, 2017, Berland et al., 2017, Dion, 2016, Wagner et al., 2016, Pade et al., 2017, Saucier et al., 2016, Williams et al., 2014, Giordano et al., 2020, Ashrafioun et al., 2016, Behar et al., 2015, Crocker et al., 2019, Dahlem et al., 2017, Hargraves et al., 2019, Pietrusza et al., 2018, Kwon et al., 2020, Monteiro et al., 2017), only rescue breathing (Kobayashi et al., 2017, Dietze et al., 2018, Petterson and Madah-Amiri, 2017, Bergeria et al., 2019, Heavey et al., 2018, Huhn et al., 2018, Pade et al., 2017, Saucier et al., 2016, Williams et al., 2014, Giordano et al., 2020), recovery position (Franko et al., 2019, Kobayashi et al., 2017, Pade et al., 2017, Williams et al., 2014, Dietze et al., 2018, Klimas et al., 2015, Petterson and Madah-Amiri, 2017, Berland et al., 2017, Wagner et al., 2016, Zhang et al., 2018, Bergeria et al., 2019, Heavey et al., 2018), and treatment referral (Wagner et al., 2016, Dahlem et al., 2017).

*By default, the studies that provided overdose prevention training in conjunction with basic life support training, or were previously trained in CPR covered AED (Klimas et al., 2015, Berland et al., 2017, Dion, 2016, Wagner et al., 2016, Zhang et al., 2018).

The following studies used the durations listed in Table 3: ≤10 min (Behar et al., 2015); 15 –45 min (Bergeria et al., 2019, Huhn et al., 2018, Ashrafioun et al., 2016, Pietrusza et al., 2018, Petterson and Madah-Amiri, 2017, Berland et al., 2017, Dion, 2016, Wagner et al., 2016); 1 h (Klimas et al., 2015, Zhang et al., 2018, Pade et al., 2017, Saucier et al., 2016, Williams et al., 2014, Crocker et al., 2019, Dahlem et al., 2017, Hargraves et al., 2019); 1. 5 h (Heavey et al., 2018), and 2.5–3 h (Dietze et al., 2018, Kwon et al., 2020).

Opioid overdose response training curriculum was adapted from different guidelines and models such as, AHA and SAMHSA guidelines, Harm Reduction Coalition’s model, and Centers for Disease Control guidelines. The duration of the training programs ranged from five minutes to 3 h, but most trainings were between 30 min and one hour long (Table 3).

2.1.3. Naloxone preparation

The provided naloxone preparations have changed over time, but considering the impact that preparation has on administration technique, this information is included from the studies (Table 4). Naloxone preparations varied, with studies including Narcan®, prefilled naloxone syringes with mucosal atomizers, and vials/syringes for IM administration. Prefilled naloxone syringes with mucosal atomizers were most often provided in opioid overdose response training studies. Four studies with simulation-based outcome measures compared rescuer performance variation based on naloxone preparation (Table 4).

Table 4.

Naloxone preparation used in opioid overdose response training.

Single device	# of studies
Mucosal Atomizer (MA)	11
Narcan®	8
IV Cannulation	1
Pre-filled syringe or Vial & Syringe (V&S)	2
Two preparations compared
MA + Evzio®	1
MA + IV Cannulation	1
MA + V&S	1
Narcan® + Evzio®	1
Three or more preparations compared
MA + Narcan® + V&S	3
MA + Narcan® + V&S + Evzio®	2

2.1.4. Validated assessment tools

Eleven studies used the Opioid Overdose Knowledge Scale (Wagner et al., 2016, Zhang et al., 2018, Heavey et al., 2018, Williams et al., 2014, Giordano et al., 2020, Kwon et al., 2020, Monteiro et al., 2017, Dietze et al., 2018, Klimas et al., 2015, Petterson and Madah-Amiri, 2017, Berland et al., 2017), eight used the Opioid Overdose Attitude Scale (Klimas et al., 2015, Berland et al., 2017, Wagner et al., 2016, Zhang et al., 2018, Heavey et al., 2018, Williams et al., 2014, Giordano et al., 2020, Kwon et al., 2020), two used the Brief Opioid Overdose Knowledge questionnaire (Bergeria et al., 2019, Huhn et al., 2018), two used the Brief Overdose and Recognition Response Assessment (Saucier et al., 2016, Behar et al., 2015), and one used the Perceived Competence Scale in conjunction with a non-validated knowledge assessment tool (Ashrafioun et al., 2016)(Table 5). Only one study (Giordano et al., 2020)used validated assessment scales (OOKS and OOAS) and a simulation rubric.

Table 5.

Assessment tools used in opioid overdose response training.

	# of studies		# of studies
OOKS	11	Critical Task Assessment	4
OOAS	8	Checklist	3
BOOK	2	Naloxone Assembly and Administration and/or Simulation Scenario Completion Time	7
BORRA	2
Perceived Competence Scale	1
Non-validated Tool
Knowledge/Recognition	7
Confidence/Self-Efficacy	4
Attitude/Comfort	4

The use of each assessment tool in the overdose response training studies is based on the following; OOKS (Wagner et al., 2016, Zhang et al., 2018, Heavey et al., 2018, Williams et al., 2014, Giordano et al., 2020, Kwon et al., 2020, Monteiro et al., 2017, Dietze et al., 2018, Klimas et al., 2015, Petterson and Madah-Amiri, 2017, Berland et al., 2017), OOAS (Klimas et al., 2015, Berland et al., 2017, Wagner et al., 2016, Zhang et al., 2018, Heavey et al., 2018, Williams et al., 2014, Giordano et al., 2020, Kwon et al., 2020), BOOK (Bergeria et al., 2019, Huhn et al., 2018), BORRA (Saucier et al., 2016, Behar et al., 2015), Perceived Competence Scale (Ashrafioun et al., 2016), non-validated knowledge tool (Dion, 2016, Pade et al., 2017, Ashrafioun et al., 2016, Crocker et al., 2019, Dahlem et al., 2017, Hargraves et al., 2019, Pietrusza et al., 2018), non-validated confidence/self-efficacy tool (Saucier et al., 2016, Crocker et al., 2019, Hargraves et al., 2019, Kwon et al., 2020), non-validated attitude/comfort tool (Pade et al., 2017, Saucier et al., 2016, Behar et al., 2015, Hargraves et al., 2019). The use of each assessment tool in the simulation studies is based on the following; Critical Task Assessment (Krieter et al., 2016, Edwards et al., 2015, Eggleston et al., 2018, Eggleston et al., 2019), Checklist (Franko et al., 2019, Kobayashi et al., 2017, Kim et al., 2016), Naloxone Administration Time (Goldberg et al., 2018, McDermott and Collins, 2012), and five studies included Naloxone Administration Time as a secondary measure (Kobayashi et al., 2017, Edwards et al., 2015, Eggleston et al., 2018, Eggleston et al., 2019, Franko et al., 2019).

2.1.5. Non-validated assessment tools

Six studies used a non-validated knowledge assessment tool (Dion, 2016, Pade et al., 2017, Crocker et al., 2019, Dahlem et al., 2017, Hargraves et al., 2019, Pietrusza et al., 2018), four used a non-validated confidence/self-efficacy assessment tool (Saucier et al., 2016, Crocker et al., 2019, Hargraves et al., 2019, Kwon et al., 2020), and four used a non-validated attitude/comfort assessment tool (Pade et al., 2017, Saucier et al., 2016, Behar et al., 2015, Hargraves et al., 2019)(Table 5).

Four studies included assessment of completing critical tasks in simulation, three included an assessment of completing indicated actions on a checklist (Franko et al., 2019, Kobayashi et al., 2017, Kim et al., 2016), and seven studies measured naloxone administration time (McDermott and Collins, 2012, Edwards et al., 2015, Eggleston et al., 2018, Eggleston et al., 2019, Franko et al., 2019, Goldberg et al., 2018, Kobayashi et al., 2017)(Table 5).

2.2. Pre-to-post opioid overdose response training outcome measures

2.2.1. Changes in knowledge

Pre-to-post percentage score changes were calculated for studies that used questionnaires, however a minimal clinically important difference was not identified. OOKS test score percentages increased $\le$10% for three studies (Klimas et al., 2015, Williams et al., 2014, Giordano et al., 2020), between 11 and 20% for five studies (Dietze et al., 2018, Petterson and Madah-Amiri, 2017, Williams et al., 2014, Kwon et al., 2020, Monteiro et al., 2017), between 21 and 30% for three studies (Wagner et al., 2016, Zhang et al., 2018, Heavey et al., 2018), and 41–50% for one study (Berland et al., 2017). Seven of these 12 studies used a modified version of the OOKS (Dietze et al., 2018, Petterson and Madah-Amiri, 2017, Berland et al., 2017, Wagner et al., 2016, Zhang et al., 2018, Heavey et al., 2018, Monteiro et al., 2017) (Table 7). One study reported the greatest increase was in the ‘risk factors’ domain (Klimas et al., 2015) (Table 7). Five studies reported the highest increase was in the ‘overdose signs’ domain (Dietze et al., 2018, Wagner et al., 2016, Zhang et al., 2018, Heavey et al., 2018, Kwon et al., 2020)(Table 7). Nine studies reported the greatest increase was in the ‘naloxone use’ domain (Petterson and Madah-Amiri, 2017, Wagner et al., 2016, Zhang et al., 2018, Williams et al., 2014, Giordano et al., 2020, Monteiro et al., 2017)(Table 7). In three studies, it was reported that participants’ OOKS pre-training scores were high (Petterson and Madah-Amiri, 2017, Williams et al., 2014, Giordano et al., 2020). Two studies using the BOOK compared an experimental online training format to a control online training format (see Table 6 for description) (Bergeria et al., 2019, Huhn et al., 2018). In both studies, BOOK test scores significantly increased in all three domains (i.e., opioid, overdose, and response knowledge) (Table 7). However, the group that received the ‘interactive’ training in Huhn et al. (2018) scored between 11 and 20% compared to the 21–30% scored by the ‘didactive’ training group. Both the ‘didactive’ and ‘interactive’ training groups in Bergeria et al. (2019) had a 21–30% improvement (Table 7). However, neither study reported significant differences between group conditions. BORRA test score percentages increased $\le$10% for both ‘overdose’ and ‘non-overdose’ identification sections in Behar et al. (2015) and only in the ‘non-overdose’ identification section in Saucier et al. (2016) (Table 7). Participants’ scores increased by 31–40% in the ‘overdose’ identification section in Saucier et al. (2016) (Table 7). In the seven studies that assessed participants pre-to-post training using a non-validated knowledge questionnaire, test score percentages increased by $\le$10% for one study (Dion, 2016), by 21–30% for three studies (Ashrafioun et al., 2016, Crocker et al., 2019, Hargraves et al., 2019), by 31–40% for two studies (Pade et al., 2017, Pietrusza et al., 2018), and $\ge$51% for one study (Dahlem et al., 2017). All seven studies reported that participants’ scores increased significantly across these domains: risk factors for an overdose, signs, symptom recognition, and use of naloxone. Seven out of 22 studies reported participants had high-baseline knowledge (i.e. ceiling effect), such that participants scored > 80% on the pre-test knowledge scales (Dietze et al., 2018, Petterson and Madah-Amiri, 2017, Wagner et al., 2016, Zhang et al., 2018, Giordano et al., 2020, Behar et al., 2015, Hargraves et al., 2019).

Table 7.

Outcome measures of opioid overdose response training studies without simulation.

Rigor	Construct	Changes			Post-training % difference
		Content Areas	Modified Scales	Greatest Significant Difference (+)	$\le$10%	11–20%	21–30%	31–40%	41–50%	$\ge$51%
Validated	Knowledge
	OOKS		7		4δ	5	3	–	1	–
	Risk factors	Risk factors		1
	Overdose signs	Overdose signs		5α
	Actions	Actions		–
	Naloxone use	Naloxone use		9α
	BOOK		0		–	1	3δ	–	–	–
	Opioid knowledge	Risk factors		2β
	Opioid overdose knowledge	Overdose signs		2β
	Opioid overdose response knowledge	Actions, naloxone use		2β
	BORRA		0		3	–	–	1	–	–
	9 Overdose events	Overdose signs, naloxone use		2γ
	7 Non-overdose events	Overdose signs, naloxone use		2γ
	Confidence
	Perceived Competence Scale	Competence	–	1	–	–	–	1	–	–
	Attitude
	OOAS		8		6δ	4	–	–	–	–
	Competence	Competence		4
	Concerns	Concerns		–
	Readiness	Readiness		–
Non-Validated	Knowledge	Recognition	–	7	1	–	3	2	–	1
	Confidence	Self-efficacy	–	4	1	–	1	1	1	–
	Attitude	Comfort	–	3	–	1	1	1	–	–

^αThe same five studies, Dietze et al. (2018), Heavey et al. (2018), Kwon et al. (Kwon et al., 2020), Wagner et al. (2016), and Zhang et al. (2018), reported that the greatest significant differences post-training were in the ‘overdose signs’ and ‘naloxone use’ OOKS domains.

^βThe same two studies, Bergeria et al. (2019) and Huhn et al. (2018), reported that the greatest significant differences were in all three of the BOOK domains.

^γThe same two studies, Behar et al. (2015) and Saucier et al. (2016), reported that the greatest significant differences were in both the overdose and non-overdose identification sections.

^δControl group percentage score change included. Bergeria et al. (2019) and Huhn et al. (2018) reported participants in the ‘didactive’ training scored a 21–30% difference pre-to-post training on the BOOK. Giordano et al. (2020) and Williams et al. (2014) reported participants in the control groups scored a $\le$10% difference pre-to-post training on the OOKS and OOAS.

Table 6.

Opioid Overdose Response Training Studies Descriptions.

Authors (Year)	Target Population and Sample Size	Study Design	Naloxone Preparation	Assessment Tool	Outcome Measures	Duration/Simulation Scenario	Training Description
Ashrafioun et al. (2016)	Total Participants = 428 (Healthcare Professionals = 93; Family/friends = 40; Others = 139; Patients = 3)	Pre-to-post; no randomization	Intranasal (mucosal atomizer, n=274) and Intramuscular (vial and syringe, n=154)	Non-validated; 7 knowledge items (scored 0 to 7); Perceived Competence Scale; 4 confidence items (scored 0 to 7)	Avg. knowledge score increased from 4.2 to 6.2 (28.6%) (p<0.001); Knowledge scores did not differ between participant type pre-to-post training (p=0.09)/ Avg. confidence score increased from 3.7 to 5.9 (31.4%) (p<0.001); Family/friends reported greater confidence than both providers (p=0.024) and “others” (p<0.001) post training/ NR	20–45 min	Overdose Prevention Training; risk factors for opioid overdose, recognition of opioid overdose, naloxone administration (demonstration)
Behar et al. (2015)	Laypersons (Syringe exchange clients) = 60	Pre-to-post; no randomization	Intranasal (mucosal atomizer)	BORRA; 16 items (scored 0 to 9 for overdose events, and 0 to 7 for non-overdose events) Non-validated; 3 comfort items (scored on a 4-point Likert scale); overdose response assessment; naloxone assembly and administration	Avg. score increased from 7.5 to 7.9 (4.4%) (p<0.02) for overdose events identified, Avg. score increased from 6.2 to 6.6 (5.7%) (p<0.02) for non-overdose events identified/ NR/ Increase in comfort identifying overdose (75% to 97%, p<0.01), managing overdose (58% to 98%, p<0.01), administering naloxone (58% to 98%, p<0.01)	5–10 min	Overdose Prevention Training; recognition of opioid overdose, naloxone administration (demonstration)
Bergeria et al. (2019)	Laypersons (Pain patients and illicit users) = 119 (‘Didactive’ = 61; ‘Interactive’ = 58)	Prospective randomized web-based intervention	Intranasal (Narcan®)	BOOK; 12 knowledge items (scored 0 to 12)	No differences between groups; Avg. knowledge score increased from 7.5 to 10.6 (25.8%) (p<0.05) for ‘Didactive’ group and 8 to 10.9 (24.2%) (p<0.05) for ‘Interactive’; Acute pain patients scored lower than chronic pain patients (p=0.001)/ NR/ NR	Avg. time: ‘Didactive’ = 27.4 min; ‘Interactive’ = 36.4 min	Overdose Prevention Training; risk factors for opioid overdose, recognition of opioid overdose, help-seeking (e.g. calling 9–1-1), naloxone administration, rescue breathing, recovery position
Berland et al. (2017)	Healthcare Professionals (1st Year Medical students) = 73	Pre-to-post; no randomization	Intranasal (mucosal atomizer/Narcan®) and Intramuscular (vial and syringe)	OOKS/OOAS; 19 knowledge items (scored 0 to 19); 16 attitude items (scored 16 to 80)	Avg. knowledge score increased from 9.73 to 17.85 (42.7%) (p<0.01)/ NR/ Avg. attitude score increased from 43.66 to 58.75 (18.9%) (p<0.01)	30 min	Basic Life Support Training; risk factors for opioid overdose, recognition of opioid overdose, help-seeking (e.g. calling 9–1-1), naloxone administration (demonstration), rescue breathing, recovery position
Crocker et al. (2019)	First Responders (27 = LEOs, 24 = Firefighters, 6 = Both) = 57	Pre-to-post; no randomization	Intranasal (Narcan®) and Intramuscular (Evzio®)	Non-validated; 7 knowledge items (scored 0 to 7); 3 confidence items (scored on a 5-point Likert scale)	Avg. knowledge score increased from 35% to 56% (p < 0.001)/ Avg. confidence score improvement of 0.52 points (10.5%)/ NR	1 h	Overdose Prevention Training; recognition of opioid overdose, naloxone administration (demonstration)
Dahlem et al. (2017)	First Responders (LEOs) = 98	Pre-to-post; no randomization	Intranasal (mucosal atomizer)	Non-validated; 6 knowledge items (scored on a 5-point Likert Scale)	Avg. score increased 2.66 to 4.7 (40.8%) (p < 0.001) for signs, symptoms, and risk factors, 1.74 to 4.82 (61.6%) (p < 0.001) for understanding what naloxone is items, 1.12 to 4.96 (76.8%) (p < 0.001) for assembly and preparation items, 1.22 to 4.95 (74.6%) (p < 0.001) for naloxone administration items/ NR/ NR	45 min to 1 h	Overdose Prevention Training; risk factors for opioid overdose, recognition of opioid overdose, naloxone administration (demonstration), treatment referral
Dietze et al. (2018)	Laypersons and PWIDs = 683 participants (Canberra = 183; Sydney = 67; Melbourne = 280; Perth = 153)	Pre-to-post; no randomization	Intramuscular (pre-filled syringes)	OOKS; 20 knowledge items (scored 0 to 20)	Avg. knowledge score increased by 14.8%, risk items by 5% (p < 0.01), sign items by 12% (p < 0.01), action items by 12.6% (p < 0.01), and naloxone use items by 29.7% (p < 0.01)/ NR/ NR	30 min to 2.5 h	Overdose Prevention Training; risk factors for opioid overdose, recognition of opioid overdose, help-seeking (e.g. calling 9–1-1), naloxone administration, rescue breathing, recovery position
Dion et al. (2016)	Healthcare Professionals (Nursing students) = 49	Pre-to-post; no randomization	Intranasal (mucosal atomizer)	Non-validated; 7 knowledge items (scored 0 to 7)	Avg. knowledge score increased from 2.7609 to 2.9038 (2%) (p < 0.001)/ NR/ NR	45 min	Overdose Prevention Training/Previous CPR Training; risk factors for opioid overdose, recognition of opioid overdose, naloxone administration (demonstration)
Edwards et al. (2015)	Laypersons = 42	Single-site, open label, randomized	Comparing Intranasal (mucosal atomizer) and Intramuscular (Evzio®)	Critical Task Errors/Simulation Scenario Completion Time	100% (Phase 3, p<0.0001) of participants correctly admin. Evzio®; 57.1% (Phase 3, p<0.0001) of participants correctly admin. MA naloxone/ 0 (Phase 3) critical errors occurred with Evzio®; 31 (Phase 3) critical errors occurred with MA/ Avg. simulation scenario completion time using Evzio® was 30 s (Phase 3); avg. simulation scenario completion time using MA was 120 s (Phase 3)	High fidelity setting; use scenario provided, home environment, mannequin located on couch, distraction added (e.g. TV)	Phase 1: no training, simulation completion; Phase 2: one-on-one training, demonstrate correct naloxone administration; Phase 3: simulation completion
Eggleston et al. (2018)	Laypersons = 138	Prospective, single-site, open label, randomized	Comparing Intranasal (mucosal atomizer/Narcan®) and Intramuscular (vial and syringe)	Critical Task Errors/Naloxone Assembly and Admin. Time	100% (n=46) of participants correctly admin. Narcan® (IM, p<0.001; MA, p=0.056); 89.1% (n=46) correctly admin. MA naloxone (IM, p=0.038); 69.6% (n=46) correctly admin. IM naloxone/ 75% of MA critical errors occurred during assembly; 85% of IM critical errors were an inability to withdraw naloxone from the vial/ Median Narcan® assembly and admin. time was 34.3 s (IM, p<0.001; MA, p<0.001); Median MA assembly and admin. time was 110.3 s (IM, p=0.1); Median IM assembly and admin. time was 99.9 s	Low fidelity setting; use scenario unknown, public environment, mannequin located on a table, distraction added (e.g. surrounding spectators)	2-minute training video (description not provided)
Eggleston et al. (2019)	Laypersons = 207	Prospective, single-site, open label, randomized	Comparing Intranasal (mucosal atomizer/Narcan®) and Intramuscular (vial and syringe)	Critical Task Errors/Naloxone Assembly and Admin. Time	66.7% (n=69) correctly admin. Narcan® (MA, p<0.001); 51.5% (n=68) correctly admin. IM naloxone (MA, p<0.001); 2.9% (n=70) correctly administered MA naloxone/ NR/ Median Narcan® (n=47) assembly and admin. time was 16 s (IM, p<0.001; MA, p<0.012); Median. IM (n=35) assembly and admin. time was 58 s; Median MA (n=2) assembly and admin. time was 113 s	Low fidelity setting; use scenario unknown, public environment, mannequin located on a table, distraction added (e.g. surrounding spectators)	No training
*Franko II et al. (2019)	Healthcare Professionals (3rd Year Pharmacy Students) = 133	Randomized	Intranasal (mucosal atomizer)	Checklist/Simulation Scenario Completion Time	Avg. 'grade' for the state training group (n=64) was 64%; 89% for the novel training group (n=69) (p<0.001)/ NR/ Median simulation scenario completion time for the state training group was 170 s; 120 s for the novel training group (p=0.31)	High fidelity setting; use scenario provided, home environment, patient actor located on floor, distraction added (e.g. panicked bystander)	Novel training; overdose prevention; risk factors for opioid overdose, recognition of opioid overdose, help-seeking (e.g. calling 9–1-1), naloxone administration (demonstration), recovery position
Giordano et al. (2020)	Healthcare Professionals (Nursing students) = 50	Quasi-experimental pre-to-post with randomization; Hybrid simulation group (n=31); Virtual reality group (n=19)	Intranasal (Narcan®)	OOKS/OOAS; 45 knowledge items (scored 0 to 45); 32 attitude items (scored 32 to 160)	No differences between groups; Avg. knowledge score was not sig. different from baseline (38.48) to follow up (38.23) for 'Hybrid simulation' group; Avg. knowledge score was not sig. different from baseline (39.63) to follow up (39.05) for 'Virtual reality' group/ NR/ Avg. attitude score was not sig. different from baseline (87.23) to follow up (85.87) for 'Hybrid simulation' group; Avg. attitude score was not sig. different from baseline (87.74) to follow up (86.05) for 'Virtual reality' group; sig. decrease in whole sample's OOAS scores by 1.48 (p=0.002)	NR	Overdose Prevention Training; recognition of opioid overdose, help-seeking (e.g., calling 9-1-1), rescue breathing, naloxone administration (demonstration), immediate recovery care after revival
Goldberg et al. (2018)	Laypersons = 50	Single-site, convenience sampling; no randomization	Intranasal (Narcan®)	Naloxone Assembly and Admin. and Simulation Scenario Completion Time	Admin. of Narcan® was 98% successful/ NR/ Median time for unlocking the public-access naloxone box was 90 s; median time for Narcan® assembly and admin. was 61 s; median simulation scenario completion time was 189 s	High fidelity setting; use scenario unknown, public environment, mannequin located on sidewalk, distraction added (e.g. surrounding spectators)	No training; provided instructions-for-use
Hargraves et al. (2019)	Healthcare Professionals (Family medicine interns) = 48	Pre-to-post; no randomization	Intranasal (mucosal atomizer/Narcan®) and Intramuscular (vial and syringe/ Evzio®)	Non-validated; 7 knowledge items (scored 0 to 7); four self-efficacy items (scored on a 4-point Likert scale); five attitude items (scored on a 4-point Likert scale)	Avg. knowledge score increased from 67.5% to 95.9% (p<0.05)/ Avg. self-efficacy score increased from 62.1% to 97.8% (p<0.05)/ Avg. attitude score increased from 71.2% to 91.2% (p<0.05)	1 h	Overdose Prevention Training; recognition of opioid overdose, help-seeking (e.g. calling 9–1-1), naloxone administration (demonstration)
Heavey et al. (2018)	Laypersons (Family/friends) = 198	Pre-to-post; no randomization	Intranasal (Narcan®)	OOKS/OOAS; 42 knowledge items (scored 0 to 42); 26 attitude items (scored 26 to 130)	Avg. knowledge score increased by 9.7 out of 42 points (23.1%, p<0.001)/ NR/ Avg. attitude score increased by 20 out of 130 points (15.4%, p<0.001)	90 min	Overdose Prevention Training; risk factors for opioid overdose, recognition of opioid overdose, help-seeking (e.g. calling 9–1-1), naloxone administration, rescue breathing, recovery position
Huhn et al. (2018)	Laypersons (Individuals prescribed opioids for pain) = 197 (‘Didactive’ = 97; ‘Interactive’ = 100)	Prospective randomized web-based intervention	Intranasal (Narcan®)	BOOK; 12 knowledge items (scored 0 to 12)	No differences between groups; Avg. knowledge score improvement of 2 points (opioid knowledge increased by 1 point (p<0.001), overdose knowledge increased by 0.6 points (p<0.001), overdose response knowledge increased by 1.6 points (p<0.001), 'I don't know' responses decreased by 2.9 points (p<0.001))/ NR/ NR	Avg. time: ‘Didactive’ = 21.5 min; ‘Interactive’ = NR	Overdose Prevention Training/64.9% had previous CPR training; risk factors for opioid overdose, recognition of opioid overdose, help-seeking (e.g. calling 9–1-1), naloxone administration, rescue breathing
Kim et al. (2016)	Healthcare Professionals (Emergency Medicine Residents) = 23	Ecological; no randomization; Comparison of Internet Trained (IT) and Reading Assignment (RA)	Intravenous (IV cannulation)	Checklist	Mean scores between the IT (0.72) and RA (0.49) participants differed by 0.23 points (p<0.05); mean scores on the time-weighted checklist between the IT (0.65) and RA (0.38) participants differed by 0.27 points (p<0.05)/ NR/ NR	High fidelity setting; use scenario unknown, emergency room environment, mannequin located in hospital bed	IT training; management of the poisoned patient, including assessment of airway, breathing, circulation, bedside glucose, and physical examination to identify a toxicologic syndrome; RA training; consisted of a recommended reading assignment in a toxicology textbook
Klimas et al. (2015)	Healthcare Professionals (General practitioner trainees) = 23	Pre-to-post; no randomization	Intranasal (mucosal atomizer)	OOKS/OOAS; 45 knowledge items (scored 0 to 45); 26 attitude items (scored 26 to 130)	Avg. knowledge score increased from 27.4 to 31.4 (8.9%) (p<0.001)/ NR/ Avg. attitude score increased from 97.4 to 108.6 (8.6%) (p<0.001)	1 h	Basic Life Support Training; risk factors for opioid overdose, recognition of opioid overdose, help-seeking (e.g. calling 9–1-1), naloxone administration (demonstration), rescue breathing, recovery position
Kobayashi et al. (2017)	Laypersons (Prison inmates) = 85	No randomization	Intranasal (mucosal atomizer)	Checklist/Naloxone Assembly and Admin. and Simulation Scenario Completion Time	51.8% (n=44) of participants correctly admin. MA naloxone; Median checklist score was 12 out of 21; 72.9% of participants called 9–1-1 via instruction to bystander (n=7) or by cellphone mockup (n=56), 65.9% of participants completed one basic airway maneuver and 91.8% of participants completed two rescue breaths/ Median time for simulation scenario completion using MA was 94.5 s; median time for MA assembly and admin. was 58 s	High fidelity setting; use scenario unknown, public environment, mannequin located on floor, distraction added (e.g. street noise recordings of an approaching police car)	Overdose prevention; risk factors for opioid overdose, recognition of opioid overdose, help-seeking (e.g. calling 9–1-1), rescue breathing, naloxone administration (demonstration), recovery position
Krieter et al. (2016)	Laypersons = 116	Study A subjects (n=63) randomized; Study B subjects (n=53)	Intranasal (Narcan®)	Critical Task Errors	90.6% (n=48) of Study B participants correctly admin. Narcan® without review of a QSG; 90.6% (n=29) of Study A (Arm 1) participants correctly admin. 2x Narcan® with review of a QSG; 90.3% (n=28) of Study A (Arm 2) participants correctly admin. 2x Narcan® without review of a QSG/ NR/ NR	High fidelity setting; use scenario unknown, home environment, mannequin located on floor, distraction added (e.g. TV and radio playing)	No training; provided instructions for use
Kwon et al. (Kwon et al., 2020)	Healthcare Professionals (3rd Year Pharmacy students) = 56	Pre-to-post; no randomization	Intranasal (mucosal atomizer/Narcan®) and Intramuscular (vial and syringe/Evzio®)	OOKS/OOAS; 45 knowledge items (scored 0 to 45); 32 attitude items (scored 32 to 160) Non-validated; 7 confidence items (scored on a 5-point Likert scale); Simulation rubric	Avg. knowledge scores increased from 33.3 to 41.9 (19.1%) (p<0.001)/ Avg. confidence scores increased for 'dispense naloxone' from 1.6 to 4.1 (50%) (p<0.001) and 'counsel on how to stimulate victim' from 1.6 to 4.5 (58%) (p<0.001)/ Avg. attitude scores increased from 93.3 to 120.4 (16.9%) (p<0.001)	3 h	Overdose Prevention Training; risk factors for opioid overdose, recognition of opioid overdose; counsel a patient/caregiver on proper use of naloxone
McDermott et al. (2012)	First Responders (Advanced Paramedics) = 18	Prospective, single-site, open label, block randomization	Comparing Intranasal (mucosal atomizer) and Intravenous (IV cannulation)	Naloxone Assembly and Admin. Time	NR/ NR/ Avg. MA assembly and admin. time was 87.1 s; Avg. IV assembly and admin. time was 178.2 s; difference in avg. assembly and admin. time was 91.1 s (p<0.0001)	Low fidelity setting; use scenario unknown, mannequin located on table	Standardized formal IV cannulation techniques; formal instruction on intranasal naloxone mucosal atomizer device
Monteiro et al. (2017)	Healthcare Professionals (Medical students) = 120	Pre-to-post; no randomization; Posttest at 12 weeks (n=51)	Intranasal (mucosal atomizer)	OOKS; 54 knowledge items (scored 0 to 54)	Avg. knowledge scores increased from 40.84 (n=120) to 47.94 (n=51, p<0.001) (13.1%)	NR	Overdose Prevention Training; recognition of opioid overdose, naloxone administration (demonstration)
Pade et al. (2016)	Laypersons (family members of opioid-dependent inpatients) = 47	Pre-to-post; no randomization	Intranasal (mucosal atomizer)	Non-validated; 1 recognition item (scored on a 5-point Likert scale); 1 comfort item (scored on a 5-point Likert scale)	Avg. recognition score increased from 2.8 to 4.6 (36%) (p<0.001)/ NR/ Avg. comfort score increased from 3.3 to 4.6 (26%) (p<0.001)	1 h	Overdose Prevention Training; risk factors for opioid overdose, recognition of opioid overdose, help-seeking (e.g. calling 9–1-1), naloxone administration (demonstration), rescue breathing, recovery position
Petterson et al. (2017)	Laypersons (Prison inmates) = 31	Pre-to-post; no randomization	Intranasal (mucosal atomizer)	OOKS; 39 knowledge items (scored 0 to 39)	Avg. knowledge score increased from 32.1 to 38.7 (16.9%) (p<0.001)/ NR/ NR	15–30 min	Overdose Prevention Training; risk factors for opioid overdose, recognition of opioid overdose, help-seeking (e.g. calling 9–1-1), naloxone administration (demonstration), rescue breathing, recovery position
Pietrusza et al. (2018)	Laypersons (Homeless adults) = 30	Pre-to-post; no randomization	Intranasal (Narcan®)	Non-validated; 6 knowledge items (scored 0 to 6)	Avg. knowledge score increased from 3.17 to 5.37 (36.7%) (p<0.0001)/ NR/ NR	15 min	Overdose Prevention Training; risk factors for opioid overdose, recognition of opioid overdose, help-seeking (e.g. calling 9–1-1), naloxone administration (demonstration)
Saucier et al. (2016)	First Responders (LEOs) = 316	Pre-to-post; no randomization	Intranasal (mucosal atomizer)	BORRA; 16-items (scored 0 to 9 for overdose events, and 0 to 7 for non-overdose events) Non-validated; 4 self-efficacy items (scored on a 6-point Likert scale); 7 attitude items (scored on a 6-point Likert scale)	Avg. score increased from 4.2 to 7.4 (35.6%) (p<0.001) for overdose events identified, Avg. score increased from 2.7 to 2.99 (4.1%) (p=0.06) for non-overdose events identified/ Avg. self-efficacy score for identification of an opioid overdose increased from 2.93 to 4.3 (22.8%) (p<0.001)/ NR	1 h	Overdose Prevention Training; risk factors for opioid overdose, recognition of opioid overdose, naloxone administration (demonstration), rescue breathing
Wagner et al (2016)	First Responders (LEOs) = 81	Pre-to-post; no randomization	Intranasal (mucosal atomizer)	OOKS/OOAS; 22 knowledge items (scored 0 to 22); 20 attitude items (scored 20 to 100)	Avg. knowledge score increased by 20.2%/ NR/ Median attitude score improved from 2.9 to 2 (18%) (p<0.001) for the competency subscale, 2.5 to 2 (10%) (p<0.001) for the concern’s subscale	30 min	Overdose Prevention Training/Previous CPR Training; risk factors for opioid overdose, recognition of opioid overdose, help-seeking (e.g. calling 9–1-1), naloxone administration (demonstration), recovery position, treatment referral
*Williams et al. (2014)	Laypersons (Family/friends) = 123	Two-group, parallel-arm, open label, randomized	Intramuscular (pre-filled syringes)	OOKS/OOAS; 45 knowledge items (scored 0 to 45); 32 attitude items (scored 32 to 160)	Avg. knowledge scores increased from 33.49 to 34.30 (1.8%) (p>0.05) for the control group, 31.91 to 38.38 (14.4%) (p<0.001) for the experimental group; Odds of experimental training increasing knowledge was 4.24 times higher/ NR/ Avg. attitude scores increased from 100.07 to 107.25 (4.5%) (p<0.001) for the control group, 100.63 to 114.73 (8.8%) (p<0.001) for the experimental group; Odds of experimental training increasing attitudes was 2.75 times higher	Experimental Group = 60 min; Control Group = 20 min	Experimental Group Training; risk factors for opioid overdose, recognition of opioid overdose, help-seeking (e.g. calling 9–1-1), naloxone administration (demonstration), rescue breathing, recovery position
Zhang et al. (2018)	First Responders (EMTs) = 117	Pre-to-post; no randomization	Intranasal (Narcan®)	OOKS/OOAS; 21 knowledge items (scored 0 to 21); 20 attitude items (scored 20 to 100)	Avg. knowledge score increased by 24%/ NR/ Median attitude score improved from 2.8 to 2.1 (14%) (p<0.0001) for the competency subscale, 2.3 to 2 (6%) (p<0.0001) for the concern’s subscale	1 h	Overdose Prevention Training/Previous CPR Training; risk factors for opioid overdose, recognition of opioid overdose, naloxone administration, recovery position

Admin, administration; Avg; average; BOOK, brief opioid overdose knowledge; BORRA, brief overdose and recognition response assessment; CPR; cardiopulmonary resuscitation; EMT; emergency medical technician; IM, intramuscular; IV, intravenous; LEO; law enforcement officer; MA, mucosal atomizer; NR, not reported; OOAS, opioid overdose attitude scale; OOKS, opioid overdose knowledge scale; PWIDs, people who inject drugs; QSG, quick start guide

*Williams et al. (2014), control training group did not view an 8-minute film about opioid overdoses and were required to read an informational pamphlet

*Franko II et al. (2019), control training group did not receive stress management techniques, nor given the opportunity to watch the management of a live overdose.

2.2.2. Changes in confidence

The Perceived Competence Scale was used in one study to assess confidence in recognizing and responding to an opioid overdose; researchers reported scores increased by 31% pre-to-post training (Ashrafioun et al., 2016)(Table 7). Four studies using non-validated confidence scales reported scores increased by $\le$10% in one study (Crocker et al., 2019), by 21–30% in one study (Saucier et al., 2016), by 31–40% in one study (Hargraves et al., 2019), and by 41–50% in one study (Kwon et al., 2020) (Table 7).

2.2.3. Changes in attitude

The Opioid Overdose Assessment Scale scores increased by $\le$10% in four studies (Klimas et al., 2015, Zhang et al., 2018, Williams et al., 2014, Giordano et al., 2020) and by 11–20% in four studies (Berland et al., 2017, Wagner et al., 2016, Heavey et al., 2018, Kwon et al., 2020) (Table 7). It is important to note that all eight studies used a modified version of the OOAS. Four studies reported the greatest significant difference was in the ‘competence’ domain (Klimas et al., 2015, Heavey et al., 2018, Williams et al., 2014, Kwon et al., 2020)(Table 7). Giordano et al. (2020) reported no change across training groups’ (see Table 6 for description) OOAS scores, but did find that there was a significant decrease in the whole samples OOAS scores by 1.48 points (p = 0.002) at a 3-week follow up. In Zhang et al. (2018) and Wagner et al. (2016) an additional seven items assessed ‘attitudes towards overdose victims’; Zhang et al. (2018) reported survey respondents were more likely to agree with the statement “People who overdose need to learn a lesson from it so they will not do it again” post-training (p = 0.0005), while Wagner et al. (2016) reported no changes. Three studies using non-validated attitude scales reported percentages increased by 11–20% in one study (Hargraves et al., 2019); 21 –30% in one study (Pade et al., 2017), and 31–40% in one study (Behar et al., 2015)(Table 7). Hargraves et al. (2019) reported that family medicine interns felt more comfortable post-training across several items asking about prescribing naloxone (p < 0.001) and administering intramuscular naloxone, intranasal naloxone, and Evzio® (p < 0.001). Pade et al. (2017) reported first responders’ perception of responding to an opioid overdose improved across two of five questions answered pre-to-post training (p = 0.02); three questions that were not statistically significant related to rural geographical area and documentation hindering response. Behar et al. (2015) reported that first-time recipients of naloxone were more comfortable pre-to-post training in identifying an overdose (p < 0.01), managing an overdose (p < 0.01), and administering naloxone (p < 0.01), while participants receiving a refill prescription felt comfortable administering naloxone (p < 0.01).

2.3. Simulated rescuer performance outcome measures

2.3.1. Critical task errors

Four studies used critical task error assessments to evaluate participants during simulated opioid overdose resuscitations (Krieter et al., 2016, Edwards et al., 2015, Eggleston et al., 2018, Eggleston et al., 2019)(Table 5). However, two of the four studies reported the utilization of different tools. Two studies were led by the same first author (Eggleston et al., 2018, Eggleston et al., 2019). Edwards et al. (2015) reported that 42 participants committed zero critical task errors using Evzio®, while the same 42 participants committed 31 critical task errors using the prefilled naloxone syringes with mucosal atomizers (Table 6). Thus, 100% of participants successfully administered naloxone using Evzio® compared to 57.1% of those who used the prefilled naloxone syringes with mucosal atomizers. The 31 critical task errors committed during preparation of the prefilled naloxone syringes with mucosal atomizers consisted of four errors attaching the naloxone cartridge to the syringe correctly, three errors attaching the atomizer to the syringe correctly, nine incidences of drug leakage during device assembly, eight failures to administer naloxone into either nostril, and 15 failures to administer naloxone into the second nostril. Eggleston et al. (2018) found that 89.1% of 46 trained laypersons correctly administered prefilled naloxone syringes with mucosal atomizers, and that 75% of the critical errors were associated to device assembly. Additionally, 69.6% of 46 trained participants correctly administered naloxone via vial/syringe. Most commonly, 85% of the errors were due to an inability to withdraw naloxone from the vial. In comparison, 100% of 46 trained participants correctly administered naloxone via Narcan®. In a similar study, Eggleston et al. (2019) reported that 66.7% of 69 untrained laypersons correctly administered Narcan®, 51.5% of 68 untrained participants correctly administered naloxone via vial/syringe, and 2.9% of 70 untrained participants correctly administered naloxone using the prefilled syringes with mucosal atomizers. Krieter et al. (2016) reported that 90% of 105 participants correctly administered Narcan®.

2.3.2. Checklist

Three studies (Franko et al., 2019, Kobayashi et al., 2017, Kim et al., 2016)monitored participants via a checklist during a simulated opioid overdose scenario (Table 5). Franko II et al. (2019) found that 69 participants trained via ‘novel’ training achieved a score of 89% on a checklist versus 64 participants who received ‘state’ training that scored 64% (see Table 6 for description). Significantly fewer participants in the ‘state’ training group performed the following actions than the ‘novel’ training group: determining if the patient had a pulse (p < 0.0001), determining if the patient was breathing (p < 0.0001), assembling the naloxone atomizer (p < 0.02), tilting the patient’s head to expose nasal passage (p < 0.0001), and properly administering naloxone (p < 0.0001) (Table 8). Kim et al. (2016) reported that 12 participants scored 72% on their checklist post-internet-training compared to 11 participants that read a toxicology textbook chapter and received a 49% (see Table 6 for description). The internet-trained participants scored 65% on the time-weighted version of the checklist compared to the 38% scored by those reading the toxicology textbook chapter. Kobayashi et al. (2018), reported that the median checklist score received was 57.1% across 21 items based on the 2015 AHA guidelines. Additionally, 51.8% of 85 participants correctly administered naloxone via mucosal atomizer nasal spray.

Table 8.

Outcome measures in opioid overdose response training studies with simulation.

Successful Administration
Evzio®	100%
Mucosal Atomizer	2.9% to 89.1%
Narcan®	66.7% to 100%
V&S	51.5% to 69.6%
Checklist Scores
Laypersons	57%
Healthcare Professionals	64% to 89%
Naloxone Assembly and Administration and/or Simulation Scenario Completion Time
Laypersons
Simulation Scenario Completion Time
Evzio®	30 seconds*
MA	94.5 secondsǂ
Narcan ®	189 secondsǂ
Naloxone Assembly and Administration Time
MA	58 to 113 secondsǂ
Narcan®	16 to 61 secondsǂ
V&S	58 to 99 secondsǂ
First Responders
Naloxone Assembly and Administration Time
MA	87.1 seconds*
IV Cannulation	178.2 seconds*
Healthcare Professionals
Simulation Scenario Completion Time
MA	120 to 170 secondsǂ

* The reported values were means.

ǂ The reported values were medians.

Successful Administration is based on the following; Narcan® (Eggleston et al., 2018, Eggleston et al., 2019, Goldberg et al., 2018, Krieter et al., 2016), MA (Kobayashi et al., 2017, Edwards et al., 2015, Eggleston et al., 2018, Eggleston et al., 2019), Evzio® (Edwards et al., 2015), and V&S (Eggleston et al., 2018, Eggleston et al., 2019). Checklist Scores are based on the following; Layperson (Kobayashi et al., 2017) and HCPs (experimental groups) (Franko et al., 2019, Kim et al., 2016). Naloxone Assembly and Administration and/or Simulation Scenario Completion Time is based on the following; Laypersons; Narcan® (Eggleston et al., 2018, Eggleston et al., 2019, Goldberg et al., 2018), MA (Kobayashi et al., 2017, Edwards et al., 2015, Eggleston et al., 2018, Eggleston et al., 2019), Evzio® (Edwards et al., 2015), V&S (Eggleston et al., 2018, Eggleston et al., 2019), First Responders; IV Cannulation (McDermott and Collins, 2012), MA (McDermott and Collins, 2012); Healthcare Professionals, MA (Franko et al., 2019).

2.3.3. Naloxone assembly and administration and/or simulation scenario completion time

Seven studies reported naloxone assembly and administration and/or simulation scenario completion times (Kim et al., 2016, Edwards et al., 2015, Eggleston et al., 2018, Eggleston et al., 2019, Franko et al., 2019). A synthesis of the naloxone assembly and administration and/or simulation scenario completion times could not be carried out (Table 8). The small sample size of studies reported means and medians. Thus, an average could not be calculated across the reported administration and scenario completion times. Edwards et al. (2015) reported it took laypersons an average of 30 s to complete simulation scenario using Evzio® (Edwards et al., 2015). In two studies, it was reported that laypersons took a median time of 94.5 and 189 s to complete a simulation scenario using prefilled naloxone syringes with mucosal atomizers and Narcan®, respectively (Goldberg et al., 2018, Kobayashi et al., 2017). The median times for layperson assembly and administration of prefilled naloxone syringes with mucosal atomizers, Narcan®, and vial/syringe ranged from 58 to 113 s, 16 to 61 s, and 58 to 99.9 s, respectively (Eggleston et al., 2018, Eggleston et al., 2019, Goldberg et al., 2018, Kobayashi et al., 2017). McDermott et al. (2012) reported advanced paramedics took 87.1 and 178 s to assemble and administer naloxone via prefilled syringes with mucosal atomizers and IV cannulation, respectively (McDermott and Collins, 2012). The difference in mean delivery times was 91.1 s (p < 0.0001) (McDermott and Collins, 2012). Lastly, Franko II et al. (2019) reported 3rd year pharmacy students that had received a ‘novel’ training or ‘state’ training took a median of 120 and 170 s to complete an overdose simulation scenario, respectively (Franko et al., 2019).

2.4. Discussion

In this review, only four studies examining training programs randomized participants to an experimental or standard education group (Bergeria et al., 2019, Huhn et al., 2018, Williams et al., 2014, Giordano et al., 2020), which limits objective comparison and the ability to draw concise conclusions. Evidence suggests that these training programs increased knowledge in the ‘overdose signs’ and ‘naloxone use’ domains most frequently, and attitude in the ‘competence’ domain on validated instruments. Seven out of 22 studies that tested knowledge reported participants scored > 80% on pre-test items (Dietze et al., 2018, Petterson and Madah-Amiri, 2017, Wagner et al., 2016, Zhang et al., 2018, Giordano et al., 2020, Behar et al., 2015, Hargraves et al., 2019), potentially highlighting that the OOKS, OOAS, and BORRA are not intended to differentiate between participants with high-degrees of knowledge and/or experience. Rather, these tools may be better suited to differentiate minimally competent participants from those who are not (Giordano et al., 2020). Minimal clinically important differences are not available for the OOKS, OOAS, BORRA, and BOOK. Additionally, across seven studies that used non-validated scales, there were more extreme percentage increases in scores compared to the use of validated scales (Table 7). The effect of training duration on rescuer performance could not be assessed due to irreconcilable differences in target population and lack of behavioral measurements. Thus, conclusions on the association between training duration, and curricula and score changes are unclear. Furthermore, the use of non-standardized assessments complicates the determination of effectiveness of different training programs and participants’ adherence to skills learned.

Current assessment tools have drawbacks including ceiling effects, applicability, and lack of evidence correlating scores with rescuer performance. The OOKS is the leading tool for knowledge assessment, but it is not ideal in its validated form for many opioid overdose response training programs. The language includes some British usage not easily generalizable to all regions (e.g. “fitting” is used to describe a seizure, which may not be understood by local populations, including in the United States). Additionally, many questions center specifically about heroin use despite a wider range of opioids currently in use. Furthermore, the OOKS does not include intranasal naloxone administration (by atomizer or Narcan, the primary formulation FDA approved for layperson use). Finally, it does not include CPR and therefore is aimed at laypersons without BLS training. This has led researchers to modify the OOKS and supplement with non-validated instruments (Table 7) (Williams et al., 2013).

Non-simulation-based outcome measures were used to assess laypersons at a ratio of 3:1 at-risk people to friends and family (Table 2). Comparatively, simulation-based outcomes have been used in a significantly smaller proportion of laypersons who are themselves at risk. Based on the studies with simulation-based outcome measures, there is varying success in the ability of trained and untrained laypersons to administer naloxone by three common naloxone preparations (Eggleston et al., 2018, Eggleston et al., 2019). There is evidence that naloxone administration is often quicker and more successful using Narcan® or Evzio® compared to prefilled naloxone syringes with mucosal atomizers. It is important to note that two studies, Edwards et al. (2015) and Krieter et al. (2016) were funded by Kaleo, Inc. (manufacturer of Evzio®) and Adapt Pharma (manufacturer of Narcan®), respectively. Regardless, according to the World Health Organization “naloxone is effective in the treatment of opioid overdose only if it is: (NIDA, 2019) available for administration (Hedegaard et al., 2020), administered correctly by the user, and (Han et al., 2019) administered in a timely fashion as early intervention is often the determinant outcome when faced with a life-threatening [overdose] event” (WHO, 2014). The reviewed validated and non-validated questionnaires fail to fully evaluate the development of the skill of naloxone administration.

Strongly influenced by budgetary considerations, there is still little consistency in the naloxone preparation used in the prehospital setting (Kerensky and Walley, 2017, Weaver et al., 2018). The variability in naloxone preparations used in the reported studies is likely attributed to the availability of new products when the study was conducted, expansion of naloxone access laws, funding, and the target populations. These factors may explain the variations in overdose reversal training content.

Conclusions about the overall effectiveness of opioid overdose response training is limited because of the lack of randomized studies or other rigorous study designs (Orkin et al., 2019). However, there are ethical concerns for conducting randomized studies, given the efficacy of naloxone in reversing opioid overdoses (Boyer, 2012). Simulation addresses ethical considerations and affords the opportunity to create a high-fidelity environment to compare rescuer performance.

Given the widespread implementation of overdose response training and variations in training and assessment, the authors believe there is a need to standardize an assessment tool that can accurately evaluate participants’ performance while responding to a simulated community-based overdose. The optimal impact of overdose response training depends on mastering the skills necessary for timely naloxone administration and other rescue maneuvers (Tobin et al., 2018). Ultimately, a standardized, validated, responsive and accurate assessment tool could help interpret the relationships between training curriculum, duration, and achieved proficiency.

2.5. Practical implications

Development and validation of a gold standard for overdose response training is difficult and utilization of simulation methodology may not be financially feasible in many cases. However, there are several components of training that require further investigation that simulation research can resolve 1) recognition of signs and symptoms in various victims (e.g., substance misuse and therapeutic use), 2) order of actions taken in response to various victim presentations (e.g., body position, pulseless victim, etc.), and 3) ability to administer naloxone in a timely manner (i.e., increase in synthetic opioid overdoses). As overdose response training programs proliferate, training guidelines will continue to update like they do in CPR (Lavonas et al., 2015)and Stop the Bleed (Goralnick et al., 2018)training, thus exploration of the behavioral markers of proficiency is warranted.

The authors are not proposing that high-fidelity simulation should be included into overdose response training. A gold standard simulation-based outcome tool would allow evaluation of lower fidelity outcome tools for deployment in conjunction with overdose response training programs. (Orkin et al., 2019), are planning to compare training programs (novel vs existing) using a high-fidelity simulation-based assessment (Orkin et al., 2019).

2.6. Limitations

This review has several limitations to consider. First, the articles reviewed might not encapsulate all relevant studies using rescuer-centered outcomes. There may have been additional search terms necessary to find all relevant articles, but the terms chosen were based on expert opinion. Second, the review was descriptive in nature. Sources were not sufficient for a statistical analysis comparing the outcome measures utilized in the studies reviewed, in part due to the use of varying subscales and non-validated scales which made it unproductive to aggregate data. Thirdly, the analysis was not pre-registered, and the results should be considered exploratory. Lastly, the scope of this review will most likely need to be reexamined in the next few years given that there has already been a 4-fold increase, since 2014, in overdose training literature. Additionally, the variability in naloxone preparation will most likely increase as other products are introduced to the market requiring further investigation.

2.7. Conclusion

There has been limited exploration of a tool or outcome measure that would ensure an individual is ‘proficient’ in naloxone administration and resuscitation. Simply having naloxone is part of the challenge, and there are opportunities to further build the public’s capability to respond to an emergent situation.

Validated multiple choice knowledge assessment tools were commonly used to assess the outcomes of training programs. It is unknown how scores on these assessment tools may correlate with actual rescuer performance responding to an overdose. Seven studies reported ceiling effects most likely attributed to participants’ background medical knowledge or experience. The inclusion of simulation-based outcome measures of performance, including the commission of critical errors and the time to naloxone administration, provided better insight into rescuer skill proficiency (Krieter et al., 2016, Edwards et al., 2015, Eggleston et al., 2018, Eggleston et al., 2019). Simulation would allow assessment of rescuer performance under varied conditions such as victim characteristics and naloxone preparation. However, checklists which are developed in conjunction with a training program may introduce a source of bias which will always favor the associated training program (as compared to a validated checklist based on best practices for overdose response).

The lack of assessment tools measuring proficiency limits the comparison of existing training programs. The authors propose a validated, responsive, applicable measure of proficiency for rescuer performance in opioid overdose response would help evaluate and improve opioid overdose response training programs, improving rescuer performance and victim outcome.

Funding

This work was supported by a Carilion Clinic and Fralin Biomedical Research Institute Center for Transformative Research on Health Behaviors Pilot Feasibility Grant.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.