Meeting science meets public health: Results from the “Stress and Violence in fire-based EMS Responders (SAVER)” Systems Checklist Consensus Conference (SC³)

Abstract

Introduction:

In order to implement a systems-level Emergency Medical Services (EMS) workplace violence intervention, input from end users was critically needed. We convened the two-day Stress and Violence in fire-based EMS Responders (SAVER)” Systems Checklist Consensus Conference (SC³) using methods from meeting science (i.e., ThinkLets) to comprehensively and efficiently gather feedback from stakeholders on the completeness and utility of the draft checklist that would comprise the intervention.

Methods:

ThinkLets, a codified facilitation technique was used to aid brainstorming, convergence, organization, evaluation, and consensus building activities on the SAVER Systems Checklist among 41 national stakeholders during a two-day conference. A qualitative and quantitative process evaluation was conducted to measure the effectiveness of conference procedures. To verify checklist feasibility results from the conference, a second feasibility assessment was conducted with the four implementation sites.

Conclusions:

The quantitative conference evaluation results indicated most participants viewed the conference process favorably. Emergent themes reflecting on conference effectiveness and suggestions for improvements are described. The re-evaluation of the checklist’s feasibility completed by the SAVER study sites confirmed prior feasibility findings. SAVER study sites cast 45.5% of votes on checklist items to be most feasible, 34.9% as less feasible, and 19.6% as extremely difficult.

Practical Applications:

Multidisciplinary collaboration between public health, occupational health psychology, and meeting science led to the development of the SAVER Systems Checklist. The checklist underscores important needs for EMS policy and training development critical to responder safety as identified and supported by over 41 diverse subject matter experts. The incorporation of a widely used meeting science method, ThinkLets, into public health intervention design proved an effective and well-received approach to bring assessment, evaluation, and consensus to the SAVER Systems Checklist. These methods may hold benefit for other industries and disciplines that may not be familiar with such facilitation and consensus-building techniques.

1. Introduction

For many organizations, improving occupational safety and health concerns of workers is an important yet challenging venture. In 2017, there were approximately 2.8 million non-fatal employee injuries and 5,147 worker fatalities in the United States according to the U.S. Bureau of Labor Statistics (Bureau of Labor Statistics, 2018). Nonfatal and fatal work-related incidents also pose organizational safety and health concerns to the workers who respond to these events. Fire and emergency medical services (EMS) occupations are among those with the greatest risk for injury and death in the line of duty (Reichard & Jackson, 2010). For U.S. paramedics and EMTs, the annual rate of non-fatal work-related injuries is three times higher than the national average (Maguire & Smith, 2013). Approximately 21,200 EMS personnel were treated in emergency departments for their work-related injuries in 2017 (Centers for Disease Control and Prevention, 2019). Consequently, opportunities to reduce injury costs and improve safety and well-being of employees and the communities served are continuously sought by emergency responder organizations (Allen, Baran, & Scott, 2010).

EMS is a high injury risk, high stress occupation at the crucial intersection of public health, public safety and healthcare systems (American College of Emergency Physicians, 2017; NHTSA Office of EMS, 2019). A significant challenge facing the occupational safety and health of EMS workers is violence from patients and bystanders (Taylor, Davis, Barnes, Lacovara, & Patel, 2015). Career exposure by EMS responders to an act of verbal and/or physical violence is estimated between 57 and 93% (Taylor & Murray, 2017). In fact, workplace violence has become so ingrained into the occupational identity, that many EMS responders have internalized the belief that violence is simply “part of the job” (Taylor et al., 2016; Taylor & Murray, 2017). This negative internalization regarding workplace violence exposures creates many challenges for EMS organizations including: reporting of violent incidents, short and long-term mental health impact, and workforce retention (Taylor & Murray, 2017). In addition to violence, an increase in community demand for services (National EMS Information System, 2016) presents major challenges for the organizational structure, mental health, and safety burden for EMS workers. To date, no evidence-based interventions exist to prepare EMS responders for violence on the job (Taylor & Murray, 2017).

However, EMS industrial trade journals frequently describe best practices to address workplace violence, specifically checklists. An extensive review of EMS industrial literature found examples supporting the development of checklists for each phase of emergency response including: traveling to the event, scene arrival/prior to entry, patient care, transport to the hospital, transfer to emergency department staff, and assessing readiness to return to service (Murray et al., 2019; Taylor et al., 2019; Taylor & Murray, 2017). Checklists work to reduce complexity in systems because they acknowledge errors as inevitable (Perrow, 1984). Checklists have seen successful in reducing errors and unsafe practices in many industries (Ashford, 1998; Berenholtz et al., 2004; Erdek & Pronovost, 2004; Gawande, 2010; National Aeronautics and Space Administration, 1990; Shappell et al., 2006). In healthcare, for example, the “Doctor’s Checklist” was created at Johns Hopkins Hospital in 2001 to avoid infections when inserting central lines into intensive care unit (ICU) patients. Checklist implementation resulted in an 11% reduction in the 10-day line infection rate and after 15 months, only two central line infections had occurred (Berenholtz et al., 2004). Another checklist focused on nursing observations of pain in the ICU saw the likelihood of patients bearing untreated pain decreased from 41% to 3% (Erdek & Pronovost, 2004). In addition, the World Health Organization’s Surgical Safety Checklist helped reduce physician error, patient safety events, and even helped decrease post-surgical mortality rates by 37% in the decade since implementation (Ramsay et al., 2019).

The “Stress and Violence in fire-based EMS Responders (SAVER)” checklist was developed in 2018 (Taylor et al., 2019). It is an innovative departure from the traditional checklists described above in that it is designed to shift the onus of safety and health from the EMS responder back to the organization for whom they work by identifying actions leadership can institute through policy, training, and environmental modifications. Individual pause points (Gawande, 2010), which are opportunities for individuals to take a brief moment to stop and consider their personal safety before continuing with patient care activities, were incorporated into the SAVER Systems Checklist. Pause points create feedback channels for EMS responders to communicate with leadership regarding their safety and health needs when serving the public (Taylor et al., 2019).

In order for a checklist to be utilized as designed, there must be a willing culture to implement it. One consideration for improving safety and health in organizations is through the development and maintenance of a climate for safety (Dunn, Scott, Allen, & Bonilla, 2016; Zohar, 2000). Safety climate, a type of organizational climate, is the degree to which employees hold shared beliefs and perceptions that management supports, rewards, and expects safe behavior and safe work practices (Hoffman & Stetzer, 1996, 1998). Introduced to the organizational science literature in the 1980s, a recent meta-analysis on safety climate has linked the construct to both increased safety behaviors and decreased occupational injuries (Clarke, 2006, 2010). Organizations with a strong safety climate often have fewer negative safety outcomes, such as near-misses, injuries, and line of duty deaths (Christian, Bradley, Wallace, & Burke, 2009; Zohar, 2000). In healthcare, hospitals with a strong safety climate not only have a positive impact on workers, but also a reduced incidence of patient safety events (Singer, Lin, Falwell, Gaba, & Baker, 2009).

One methodology to increase occupational safety climate is the use of meeting science (Tannenbaum & Cerasoli, 2013). Meeting science is a useful tool for promoting safety culture and safety behaviors by creating environments where collaborative development of intervention processes can occur. One such example to be further discussed here is the use of meeting science to overcome the fraught and rate limiting step that meetings and consensus can often pose for the development of public health interventions. Codified facilitation techniques are routinely used in other fields such as occupational health psychology (De Vreede, 2006). Yet, these methods are new and innovative to the field of public health and even more so to the fire and rescue service, the context of our work. Through the application of this meaningful convergence process, national stakeholders were able to participate and comment on the design of the SAVER Systems Checklist, a workplace violence intervention for EMS, in a highly effective and novel way.

To address the growing concern of workplace violence in EMS, a public health intervention using a combination of theory and methods from occupational health psychology and meeting science was pursued. The purpose of this manuscript is to report out on the collaborative facilitative process that utilized meeting science in the development of the SAVER Systems Checklist. Specifically, the primary goals of this paper are to: (1) Describe the evaluative results from the Systems Checklist Consensus Conference (SC³) where collaboration tools from meeting science were deployed and; (2) Describe the feasibility assessment re-evaluation results subsequently completed by the four SAVER study sites.

2. Method

2.1. Study design

The activity described herein is part of a Federal Emergency Management Agency (FEMA)-funded project, “Stress and Violence in fire-based EMS Responders (SAVER).” The first aim of SAVER was to develop a systems-level checklist for violence against fire-based EMS responders. The SAVER Systems Checklist followed a similar development process (Fig. 1) as the World Health Organization’s Patient Safety Programme used to develop the Surgical Safety Checklist (Weiser et al., 2010).

Fig. 1.

SAVER Systems Checklist Development Process.

The systematic literature review and the drafting of checklist items from Fig. 1 have been described previously (Taylor et al., 2019). The next step of revising and refining the checklist with stakeholder input is described herein.

3. Systems checklist consensus conference (SC³)

In July 2018, 41 national stakeholders and diverse subject matter experts (SMEs) convened for the two-day “Systems Checklist Consensus Conference (SC³)” to facilitate consensus-building, collaboration, and evaluation of the SAVER Systems Checklist. The 41 national stakeholders represented 27 different organizations and a wide range of service levels, including leaders and representatives from national fire service and EMS organizations, members of federal, state, and local governments, academics and researchers, fire department members of various organizational rank, and labor union representatives. Multiple members from the four SAVER study sites were invited to ensure organizational and occupational representation at the following fire department levels: leadership, labor union, EMS field supervisor, and a paramedic with 10–15 years of experience. Representatives from dispatch and law enforcement were also invited to the conference to provide consultation on the checklist, however only one dispatch representative was able to attend.

3.1. Consensus-building: ThinkLets

To create consensus on the efficacy, utility, and feasibility of the SAVER Systems Checklist, we utilized a series of structured focus groups constructed around three separate ThinkLet systems, a facilitated consensus-building facilitation technique and collaboration method (De Vreede, Kolfschoten, & Briggs, 2006). A human subject’s protocol was submitted and approved by the Drexel University Institutional Review Board, which approved the recording of ThinkLet sessions for research and quality assurance purposes. Three separate facilitated collaboration ThinkLet systems were developed based on previous research (Kolfschoten, Briggs, De Vreede, Jacobs, & Appelman, 2006) and were referred to as ThinkLets 1–3: Idea Generation, ThinkLet 4: Convergence, and ThinkLet 5: Feasibility Assessment. Idea Generation was conducted three times (ThinkLets 1–3) to achieve data saturation (Steinhauser, Read, & de Vreede, 2008; Tracy, 2013). Prior to the conference, the facilitation team received training on each ThinkLet to ensure consistency across the ThinkLet sessions and groups within each session. Each ThinkLet session was facilitated by a lead facilitator with an assistant to take notes and manage logistics.

For ThinkLets 1–3 (Idea Generation), the 41 SMEs were broken into six groups of 7–8 individuals on Day 1. Since the checklist was organized into six distinct phases of emergency response, the SMEs were broken into six groups so that during each iteration of ThinkLets 1–3, a phase of the checklist was reviewed by a new subset of SMEs. Each group participated in three ThinkLet sessions and reviewed three consecutive checklist phases. The groups were first asked to review their assigned phase of the checklist, generating as many additional ideas for items to be included on the checklist within a particular phase, and to also revise or refine the original candidate checklist items as needed. The goal was to generate checklist items that would best support EMS responders in a particular phase, rather than be stifled by organizational or resource restrictions they could anticipate. ThinkLet sessions 1–3 were 60 min long, followed by a 60-minute break. This break allowed the facilitators essential time to check in with one another on each session’s idea generations, make process adjustments, and affirm goals. The break also afforded participants with cognitive rest and networking opportunities. This iterative process allowed for constant item refinement and new item generation throughout ThinkLets 1–3. The facilitators then compiled all the changes to the checklist, including adding new items and revising current items on the checklist. At this point in the process, the fully compiled version of the checklist grew from 159 to 242 candidate items. The 242 candidate items were then used in ThinkLet 4 at the start of Day 2.

For ThinkLet 4 (Convergence), the same 41 SMEs were again placed in groups of 7–8 individuals. Groups were deliberately reorganized, in different configurations than ThinkLets 1–3, to ensure a mixture of perspective, expertise, group dynamics, and previous knowledge of each of the checklist phases. ThinkLet 4 gave all SMEs the opportunity to review all candidate checklist items across all phases of EMS response, regardless of whether they had the opportunity to comment on them in ThinkLets 1–3. ThinkLet 4 focused on evaluating the newly compiled checklist and seeking agreement on any final changes to be made. The groups were directed to review and discuss each checklist item as a group in an effort to reach consensus that the checklist was both comprehensive and appropriate for the phase identified, whether it needed to be moved to a different phase of the checklist or deleted entirely. Each group reviewed all six phases of the checklist in 15-minute intervals over a 90-minute session. ThinkLet 4 processes led to the deletion of 10 candidate checklist items based on SME consensus. A 60-minute break followed for facilitator preparation of the final session, ThinkLet 5 (Feasibility Assessment).

ThinkLet 5 allowed each participant to, individually and anonymously, rate the resultant 232 candidate checklist items according to the feasibility of implementation. Each participant was instructed to think about the timeframe they felt their organization could successfully implement each checklist item and rate each item in one of the following feasibility categories: Most Feasible (Easily Implemented within 3–6 months), Less Feasible (Reasonably Implemented within 1–2 years), Extremely Difficult (Nearly Impossible/Infeasible within 2 years to never).

Further refinements to the checklist were completed by JAT, RMM, and JAA following the close of the conference to correct grammar, inconsistencies, and remove redundant items. The final SAVER Systems Checklist encompasses 174 checklist items arranged by the six phases of EMS response: pre-event, traveling to the scene, scene arrival, patient care, assessing readiness to return to service, and post-event. Pause points for the individual EMS responders were incorporated at the end of each of phase. Overall, 47.5% of items across all phases were rated as most feasible (MF), 33.7% as less feasible (LF), and 11.6% as extremely difficult (ED) with 7.2% missing data (Taylor et al., 2019).

3.2. Process evaluation

To assess the various components of SC³, a qualitative and quantitative process evaluation was administered at the close of conference activities. Quantitative questions included some of the following:

1. What level of satisfaction do you have with this conference? (Scale: Extremely satisfied, Very satisfied, Moderately satisfied, Slightly satisfied, Not satisfied at all)

2. To what degree do you feel the conference structure promoted thoughtful interactive collaboration (length and frequency of breaks, etc.)? (Scale: Strongly agree, Agree, Neutral, Disagree, Strongly disagree)

3. To what degree do you feel the amount of material covered in this conference was appropriate for the time provided? (Scale: Strongly agree, Agree, Neutral, Disagree, Strongly disagree)

4. To what degree do you feel confident that you can share the goals of the systems-level checklist with others? (Scale: Strongly agree, Agree, Neutral, Disagree, Strongly disagree)

5. Did you feel that you were able to freely discuss your thoughts and opinions in this conference? (Yes or No)

The percent agreements were calculated for question 1 by combining ‘Extremely satisfied’ and ‘Very satisfied’ responses; for questions 2–4 by combining ‘Strongly agree’ and ‘Agree’ responses; and for question 5 by coding the open-ended responses into ‘Yes’ and ‘No’ categories. Responses that did not explicitly state ‘Yes,’ or spoke to their perceived difficulty in speaking up, were coded as ‘No’ responses. A total of 38/41 participant evaluations were collected. SMEs were also asked to provide qualitative feedback on the impact of keynote speakers, which helped ground and focus the conference on the issue at hand, which part of the conference they enjoyed most and why, and whether they felt they could freely discuss their thoughts and opinions during the conference, in addition to general comments and feedback. Evaluations were analyzed for emergent themes by JAT, RMM, ALD, and LJS. Thematic and emblematic quotes were identified without qualitative software through manual identification conducted by the research team.

4. Study site feasibility assessment re-evaluation

To verify results from the initial feasibility assessment conducted during SC³, each SAVER study site was asked to provide a re-evaluation of the checklist’s feasibility. This was done for two reasons: (a) to hone in on the fire department specific perspective, since ThinkLet 5 collected anonymized responses on the checklist’s feasibility and there were a small but concerning number of missing responses; and (b) to collect feasibility data on the checklist from the four participating SAVER study sites who would be responsible for implementation and compare it to the initial assessment.

The checklist was entered into Qualtrics, an online survey software system. Each checklist item was rated on the same feasibility scale as the initial feasibility assessment (e.g., MF, LF, ED). All fire department members from the four SAVER study sites who attended SC³ were asked to complete the secondary feasibility assessment. A total of 18/20 individuals completed the re-evaluation survey. Percent MF, percent LF, percent ED, and percent missing votes were calculated by total possible respondents and by each phase. For example, percent MF was calculated by the number of MF votes per checklist item, divided by the total possible responses (18) and divided by the total number of checklist items per that phase, multiplied by 100.

5. Results

The initial feasibility results of the checklist were published previously (Taylor et al., 2019). Reported here are the quantitative and qualitative conference process evaluation results and the secondary feasibility assessment based only on the four fire departments that will implement the checklist.

6. Quantitative consensus conference evaluation results

The quantitative portion of the conference process evaluation revealed a high percent agreement of responses, which indicates positive reactions to the conference. Most scores were 90% or higher with one exception having to do with sufficient time (Table 1).

Table 1.

SC³ Process Evaluation Results.

Question	Mean	Std. Dev.	Percent Agreement % (n)
What level of satisfaction do you have with this conference?	4.68	0.53	97.37% (37)
To what degree do you feel the conference structure promoted thoughtful interactive collaboration (length and frequency of breaks, etc.)?	4.84	0.37	100% (38)
To what degree do you feel the amount of material covered in this conference was appropriate for the time provided?	4.51	0.91	87.18% (34)
To what degree do you feel confident that you can share the goals of the systems-level checklist with others?	4.95	0.23	100% (38)
Did you feel that you were able to freely discuss your thoughts and opinions in this conference?*	1.95	0.23	94.59% (35)

7. Qualitative consensus conference evaluation results

Written feedback to “What part of this conference did you enjoy the most?” and “Other Comments” provided qualitative reflection on the conference’s effectiveness and stakeholders’ beliefs that the checklist was needed and would be useful for their industry. SMEs also highlighted areas for improvement of the conference processes. When reviewing the qualitative reflection in the “Did you feel that you were able to freely discuss your thoughts and opinions in this conference?” and “Other Comments,” several SMEs suggested modifications to the conference that could have made their review of the checklist more rich. Several salient themes emerged from the qualitative responses and representative quotes are highlighted here:

7.1. Engaging conference structure & process

Several SMEs commented on the structure and process of the conference, which was organized with structured break times interspersed throughout each day, allowing for more focused engagement during the participatory ThinkLet sessions:

“Break times allowed us to ‘be present’ during the sessions because we knew we would have time to get work calls, emails, etc. The amount of work you did to prepare the checklist made for excellent use of our time. It was very productive. Amazing opportunity and really important work.”

“Great meeting – You really got a lot of info out of 2 days of work. Longer breaks were really, really helpful for keeping people engaged and focused. Good camaraderie and team spirit. Feel like I’ve got a bunch of new friends. Great work setting things up beforehand. Best working meeting I’ve ever attended.”

7.2. Value of ThinkLets methods

Participants commented on the ThinkLets process as a new and effective tool to gather diverse perspectives, bring focus to tasks, and accomplish the intended goals of each ThinkLet session. Participants expressed enjoyment learning how ThinkLets operate and the team-centered approach of the ThinkLets methods:

“The group ThinkLets were incredible. I thoroughly enjoyed learning a new process for weeding through a lot of information and watching groups make rapid progress in a short amount of time.”

“ThinkLets were an interesting and valuable way to provide input into the tool, and I liked having teams with mixed experience and backgrounds to work together to respond.”

7.3. Networking & idea sharing opportunities

Some respondents reflected on the culture of open dialogue and idea sharing amongst the conference participants, in addition to insights gained by discussing issues with other fire departments and EMS agencies. Creating a safe environment for cross-talk amongst participants was intentional and participant feedback reflected that this goal was accomplished:

“From start to finish it was organized and proceeded according to the syllabus. No wasted time. The most [enjoyable aspect of the conference was] NETWORKING! Comparison of departments is so very important.”

“Ideas were able to be shared and opinions validated and discussed. The process truly allowed for silos to be removed within thoughts.”

7.4. Impact of conference & SAVER systems checklist

Many SMEs utilized the ‘Other Comments’ section to share their gratitude that this issue of workplace violence is being addressed, as well as the positive impact they believe the checklist will have on increasing safety for the fire and rescue service:

“To sum up the conference, this subject has been long overdue. The tracking, data collecting, collaborations between labor and management, and awareness of these issues should assist with moving forward in the development of policies and education regarding violence to EMS responders. The environment is not getting any safer, the streets are getting more dangerous.”

“This was one of the most enjoyable conferences I’ve ever been a part of. I travel all over the country giving lectures about mental health and workplace violence, but I can never tell if I’m making a difference. This is the only time I felt like true progress was made. I can’t wait to see my department enact some of the changes discussed. You are all making a true impactful difference. I hope you know that you are going to save lives. You will save people you will never get to meet. They’re lives will be saved and they won’t even know it was because of a policy and training you suggested.”

7.5. Suggested improvements: understanding of ThinkLets

A few SMEs noted that their understanding of the consensus-building process of the ThinkLets started slowly for the group, attributing this to their unfamiliarity with these particular methods. Participants expressed that while there was a brief learning curve, they were able to quickly adjust as the conference progressed:

“The ThinkLet process could/should have been explained more thoroughly at the beginning. In our first [ThinkLet session], my group initially struggled/floundered until we figured out the ground rules and goals of the exercise.”

“I believe this started off slow, but we picked up on the needs/wants pretty fast.”

7.6. Suggested improvements: time constraints

Others noted that insufficient time was provided for each ThinkLet due to the amount of material to be reviewed, understood, and then discussed in each session and therefore felt rushed:

“There were some time constraints, which prohibited long discussion, but the facilitators did an excellent job at helping us move along, and they were extremely professional.”

“I would allow more time in the ThinkLets when the group is working on a [checklist] phase that has lots of items to evaluate. We felt like we did not have enough time.”

7.7. Suggested improvements: desire to provide individualized feedback

While the ThinkLets were designed to be completed in a group format and many participants enjoyed this team-centered approach, two respondents identified specific ThinkLet sessions and their desire to have had the opportunity to provide more individualized feedback:

“During ThinkLet 5, I would have liked the opportunity to delete some checklist items without others input. It would be more interesting to see what would be deleted without more discussion with others.”

“I think it’s a disservice to give me three choices and 10 minutes to evaluate each section. I hope and know that we could have made suggestions by writing out responses. I would suggest sending out the 6 stages to everyone so they can explain their individual circumstances for each question.”

7.8. Suggested improvements: absent stakeholder perspectives

Lastly, some SMEs expressed their desire for participation from additional first responder agencies, such as the police and dispatch, whom they felt would have enriched the discussion:

“One thing that really needed answering were the policies and procedures of police departments. I strongly suggest including police in this conference next time. Some input from that side of the fence would have provided a little more clarity on the way violent patients and scenes are handled by police and what rules, regulations and restrictions do they have when interacting with EMS.”

“More inclusion of dispatchers, more medics, and more firefighters in and subsequent discussion. This way we can have more direct level information on what is needed.”

8. Study site feasibility assessment

Overall, 45.5% of votes across all phases rated items as most feasible, 34.9% as less feasible, and 19.6% as extremely difficult (Table 2). “Assessing readiness to return to service” was the only phase in which the majority of votes rated items as less feasible (37.4%). “Assessing readiness to return to service” also had the highest percentage of votes rating items as extremely difficult (32.0%). “Post-event” was evenly split between most feasible and less feasible categories (38.1% and 41.0%, respectively).

Table 2.

Study Site Feasibility Re-evaluation by Phases of SAVER Systems Checklist (n = votes).

Phase of EMS Response	Most Feasible % (n)	Less Feasible % (n)	Extremely Difficult % (n)
Pre-Event	45.5% (336)	31.9% (235)	22.6% (167)
Traveling to the scene	60.4% (174)	31.9% (92)	7.7% (22)
Scene Arrival	74.2% (187)	21.4% (54)	4.4% (11)
Patient Care	49.6% (268)	35.6% (192)	14.8% (80)
Assessing Readiness to Return to Service	30.6% (165)	37.4% (202)	32.0% (173)
Post-Event	38.1% (295)	41.0% (317)	20.9% (162)
Total Votes	45.5% (1425)	34.9% (1092)	19.6% (615)

The final SAVER checklist comprised of 174 items was re-evaluated by the study site cohort (Table 3). Table 3 provides an overview of the feasibility votes on each checklist item by the 18 participants from the 4 study sites that will implement the checklist.

Table 3.

SAVER Systems-level Checklist Study Site Feasibility Re-evaluation Results by Checklist.

Checklist Number	Phase 1. Pre-Event	M.F.	L.F.	E.D.
	Mission
1.1	Does your department have as part of its mission statement (i.e., core values, vision, organizational philosophy, etc.) that the safety and health of its members is paramount in order to provide quality community service?	14	3	1
1.1.a	Does the department have as a part of its mission statement the expressed recognition and commitment to emergency medical services?	16	1	1
1.2	Does your department implement policies, practices, and procedures that support EMS responder safety?	14	3	1
1.2.a	Does your department utilize clear definitions of violence, both physical and verbal? Does this include exposures related to violence (e.g., bites, bodily fluids)? Does your department provide training on all definitions?	9	7	2
1.2.b	Does your department have a policy for if and when EMS responders can use self-defense or other means for protecting themselves?	4	7	7
1.3	Does your department train EMS responders for potential verbal and physical violence (e.g., prevention, patient abandonment, felonious assault laws, cultural competency, simulation, self-defense, law enforcement cross-training, fit-for-duty, etc.)?	4	9	5
1.3.a	Does the training include hands-on self-defense instruction?	3	6	9
1.3.b	Does the training include the development of situational awareness?	9	5	4
1.3.c	Does the training include how to use department approved protective and defensive equipment?	3	7	8
1.4	Does your department encourage a level playing field (i.e., flattened hierarchy) among ranks when expressing safety concerns?	9	6	3
1.4.a	Does your department encourage a speak-up culture surrounding verbal and physical violence, without fear of harassment, embarrassment, or punishment?	11	5	2
1.4.b	Does your department cultivate a team-centric approach to patient care (i.e., EMS responders, firefighters, dispatchers, and leadership as equal participants of the team)?	11	6	1
1.4.c	Is there a standing EMS or labor management committee that regularly meets to discuss responder safety issues?	12	6	0
	Zero-Tolerance for Violence
1.5	Does your department express through policy that verbal and physical violence against members is not tolerated?	10	6	2
1.6	Does your department utilize a placard on the vehicle to educate the public that it is a crime to assault an EMS responder (if the law exists in your state)?	5	9	4
1.6.a	Does your department display placard in patient native languages (e.g., Spanish)?	3	13	2
	Department Practices
1.7	Does your department have a psychological evaluation as part of the hiring process?	1	4	13
1.8	Does your department select uniforms that clearly designate and separate EMS responders from other first responders (e.g., police)?	7	6	5
1.9	Does your department have a policy that dictates who may ride with the EMS responder?	16	2	0
1.9.a	Does your department have a policy concerning who may ride in the patient care compartment of the ambulance?	17	1	0
1.9.b	Does your department have a policy concerning who may ride in the cab of the ambulance?	17	1	0
1.10	Does your department have a policy or procedure that outlines when police should escort an ambulance with a violent or arrested patient?	11	6	1
1.11	Does your department have policies for securing patients for their safety and the safety of the responder?	17	0	1
1.12	Does leadership in your department ride with EMS responders to have a thorough understanding of their work environment?	5	8	5
1.13	Does your department provide ride-alongs and fire/EMS 101 for local politicians, media, researchers, clinicians, etc.?	10	3	5
1.14	Does your department have a policy to cross-train with other agencies regarding violence (e.g., Police, Dispatch, Social Work, Community Health, etc.)?	1	10	7
1.15	Does your department provide training for dispatchers on recognizing when to flag calls as specific types?	8	9	1
	Professional Behavior
1.16	Does your department have policies regarding professional behavior and communicating with patients, patient families, and bystanders as a de-escalation technique for EMS responders and dispatchers?	8	8	2
1.16.a	Does your department have trainings and simulations for EMS responders and dispatchers on professional behavior?	7	8	3
1.17	Does your department have de-escalation training for mediating violent acts?	2	9	7
	Communication
1.18	Does your department have a universal code (e.g., mayday) for distress or emergency for EMS responders?	14	2	2
1.18.a	Does your police department use the same universal code?	8	7	3
1.18.b	Does the municipality use the same universal code?	5	8	5
1.18.c	Does the state use the same universal code?	0	7	11
1.18.d	Does the entire national EMS services system use the same universal code?	0	1	17
1.19	Does your department have the ability to monitor and record violent events in the field (e.g., black box, body cameras, physiological monitoring, etc.)?	3	6	9
1.20	Does your department have a means of communicating between the cab and patient care compartment of the ambulance (e.g., window, headset, radio in windowless, video camera display, etc.)?	15	2	1
1.21	Does your city use billboards to display EMS personnel working to care for the public and remind the public to be kind when help arrives (e.g., care for those who care for you)?	1	10	7
1.22	Does your department have interagency protocols or agreements for communication and data sharing? (e.g., Law enforcement, mutual aid, etc.)	6	8	4
1.22.a	Have EMS responders been trained how to use interagency communication protocols?	6	7	5
1.22.b	Has dispatch been trained on how to communicate with police dispatch to coordinate police assistance on EMS runs?	14	3	1
Count		336	235	167
Percent		45.5%	31.9%	22.6%
Checklist Number	Phase 2. Traveling to the Scene	M.F.	L.F.	E.D.
	Dispatch
2.1	Does your department have dispatch protocols for when to launch additional resources to support scene safety?	14	4	0
2.1.a	Are dispatchers trained on when to launch additional resources?	14	4	0
2.1.b	Does your department have a policy requiring dispatchers to keep the caller on the line until EMS arrives to ensure the sharing of information?	13	3	2
2.2	Has your department operationalized a ‘flag' in your dispatch system to alert EMS responders to previously known violent locations or individuals?	10	8	0
2.2.a	Are dispatchers trained in communicating that a ‘flag' exists for previously violent locations or individuals?	11	7	0
2.2.b	Are EMS responders trained in confirming with dispatch if the location is a previously violent location or individual?	11	7	0
	En Route
2.3	Does your department have a policy regarding the use of lights and sirens (e.g., responder discretion, tiered response)?	17	1	0
2.3.a	Have EMS responders been trained on scenarios which require different uses of lights and sirens?	13	5	0
	Police Assist
2.4	Does your department have a policy for calls that require police assistance (e.g., dispatch to notify that police are en route, EMS responders have been trained to check for police en route)?	14	3	1
2.4.a	Have dispatch, police, and EMS responders been trained on police assist?	12	5	1
2.5	Does EMS receive police dispatch data on neighborhoods and locations that have known risks for violence?	4	11	3
2.5.a	Are dispatchers trained to know how to use these data to inform EMS responders about risk?	4	14	0
2.6	Does your department have the capability to share radio frequency with police?	16	0	2
2.6.a	Have EMS responders been trained on how and when to share radio frequency?	10	5	3
2.6.b	Have EMS responders been trained to understand police department terminology?	5	10	3
2.7	Are there adequate resources to have joint police and EMS response?	6	5	7
Count		174	92	22
Percent		60.4%	31.9%	7.7%
Checklist Number	Phase 3. Scene Arrival
	Body Armor
3.1	Does your department have a policy regarding body armor for EMS responders (e.g., ballistic vests, helmets, stab protection)?	10	6	2
3.1.a	Are EMS responders trained on how and when to don body armor properly and what weapons the armor protects against?	6	10	2
3.1.b	At a minimum, is the departmenťs policy compliant with national standards, such as the NFPA?	5	10	3
	Dispatch
3.2	Does your department have a policy in place for EMS responders to communicate scene conditions upon arrival?	17	0	1
3.2.a	Have EMS responders been trained on how and when to communicate scene conditions?	16	1	1
3.2.b	Has dispatch been trained on how to respond depending upon the update from EMS responders?	15	3	0
	Staging
3.3	Does your department have policies for staging ambulances during events with a strong potential for violence (e.g., underlying medical condition, drug and alcohol influence, domestic violence, suicide attempts, behavioral/mental health emergencies, civil unrest, active shooters, terrorism, etc.)	16	2	0
3.3.a	Have EMS responders been trained on staging and exiting protocols?	17	1	0
	Interagency Incident Command
3.4	Does your department have protocols on communicating field updates to dispatch and vice versa?	17	1	0
3.4.a	Has dispatch been trained on receiving and responding to field updates while fielding other calls?	15	3	0
3.4.b	Has dispatch been trained to communicate with necessary agencies if an update necessitates more EMS responders or police?	16	2	0
3.4.c	Does your department have a feedback mechanism for communication failures and breakdowns?	8	8	2
	Scene Assessment and Safety
3.5	Does your department have protocols and tools for scene assessment?	14	4	0
3.5.a	Have EMS responders been trained on protocols for scene assessment?	15	3	0
Count		187	54	11
Percent		74.2%	21.4%	4.4%
Checklist Number	Phase 4. Patient Care	M.F.	L.F.	E.D.
	De-escalation
4.1	Does your department have Standard Operating Procedures [SOPs] for specific call types as it pertains to becoming a potential threat to EMS responders (e.g., underlying medical conditions, drug and alcohol influence, domestic violence, suicide attempts, behavioral/mental health emergencies, civil unrest, active shooters, terrorism etc.)?	7	9	2
4.1.a	Have EMS responders been trained on these SOPs and how to care for patients in these specific call types?	10	6	2
4.1.b	Have EMS responders been trained on how to protect themselves in these situations?	7	7	4
4.2	Does your department have training on assessing patients and bystanders, and their environment and immediate vicinity for threats (i.e., physical, mental, or metabolic conditions, egress routes, physical barriers for bystanders, cover and concealment, potential weapons, etc.)?	4	9	5
4.3	Does your department have policies on de-escalation techniques for various patient conditions (i.e., physical, mental, or metabolic conditions)?	5	7	6
4.3.a	Have EMS responders been trained on these de-escalation techniques?	3	9	6
4.4	Does your department have policies on when to call for backup at the earliest recognition of a threat?	15	2	1
4.4.a	Does your department have a graduated response to various levels of threat recognition, both from patient and bystanders?	8	9	1
4.4.b	Have EMS responders been trained on when to call for backup?	12	5	1
	Restraints and Self-defense
4.5	Does your department have policies on when to use restraints (i.e., chemical restraints, physical restraints), and what interagency involvement is needed (e.g., Police, Medical Control, etc.)?	14	4	0
4.5.a	Are EMS responders trained on when and how to use restraints when not in contact or without approval from medical control?	14	4	0
4.6	Does your department have a policy on self-defense?	4	5	9
4.6.a	Are EMS responders trained on self-defense techniques (e.g., breakaways, disarming, evasive actions, and less lethal tactics such as taser and mace)?	2	5	11
4.6.b	Have EMS responders been trained on city and state laws related to self-defense and what is an appropriate response per department policy?	2	9	7
4.7	Does your department have a policy regarding leaving the scene - with or without the patient - when EMS responders' safety is at risk?	12	4	2
4.7.a	Have EMS responders been trained on this policy?	11	5	2
	Weapons-related Safe Actions and Practices
4.8	Does your department have policies regarding safe practices while administering care (e.g., if weapons are found on patient or bystander, etc.)?	10	6	2
4.8.a	Does your department train EMS responders on these safe practices?	9	7	2
4.8.b	Is dispatch trained to inform EMS responders if weapons are on scene?	15	1	2
4.8.c	Have EMS responders been trained on safe practices while in the ambulance and administering patient care?	11	6	1
4.9	Does your department have a policy on weapon discovery and securement when in transit?	9	6	3
4.9.a	Have EMS responders been trained on this policy?	7	8	3
4.10	Does your department use a standardized coded language to convey danger on scene with all relevant agencies (i.e., EMS, Police, Fire, Hospital, etc.)?	7	9	2
4.10.a	Are EMS responders trained in the coded language to notify dispatch of any concerns (e.g., crowd forming) and to call dispatch for backup (e.g., police assist, extra fire truck)?	6	11	1
4.10.b	Is dispatch trained in this coded language to safely communicate with EMS responders in the field?	6	11	1
4.10.c	Does your department have a 'panic button' mechanism in place when coded language or verbal communication is not an option?	15	3	0
	Transport and Transfer to the Hospital
4.11	Does your department have a policy regarding receipt of dangerous or violent patients in emergency department?	10	6	2
4.11.a	Have EMS responders been trained on this policy?	8	8	2
4.12	Is there a system in place to let the hospital know that an EMS responder has been injured?	13	5	0
4.13	Is there notification during handoff at the hospital to alert staff of patient or bystander violence?	12	6	0
Count		268	192	80
Percent		49.6%	35.6%	14.8%
Checklist Number	Phase 5. Assessing Readiness to Return to Service	M.F.	L.F.	E.D.
	Readiness to Return to Service
5.1	Does your department have a policy that gives EMS responders and supervisors the autonomy to decide what they need physically and emotionally after a call, prior to returning to service (e.g., return to quarters, peer support, Critical Incident Stress Management (CISM), Employee Assistance Program (EAP), time off before return to service, seek religious counsel, etc.)?	9	8	1
5.1.a*	Are you tracking what options are used?	3	11	4
5.1.b	Do all EMS responders (from top-down: chief, supervisors, field personnel) receive recurrent training on how to recognize acute, cumulative, and chronic stress exposures from on-duty sources and their personal lives in themselves and others?	4	8	6
5.1.c	Do all EMS responders (from top-down: chief, supervisors, field personnel) receive recurrent training on how to reflect on stress of the job and the importance of reflection as professional practice (e.g., post-incident emotional assessment)?	3	9	6
5.1.d	Are there certain calls or circumstances that result in a mandatory wellness check-in?	9	6	3
5.1.e	Do EMS personnel receive training and resources to build personal resiliency to deal with stressors outside of work?	4	10	4
5.2	Does your department have a policy that allows for recovery from work to reflect at the end of a call (e.g., post-incident emotional assessment), have breaks for food, time to use the bathroom, or rest during their shift?	5	6	7
5.2.a	Does your department have a policy regarding under what circumstances a unit can be forcibly/automatically returned to service, and who has the authority to override such an action (i.e., dispatcher, EMS responder, supervisor)?	6	7	5
5.2.b	Have dispatchers been trained on when they can and cannot call an ambulance back in service from a break?	6	6	6
5.2.c	Have dispatchers been trained on when EMS responders can override a return to service decision?	5	5	8
5.2.d	Have EMS responders been trained on how to communicate breaks to dispatch?	6	6	6
	Physical and Psychological Injury Assessment
5.3	Does your department have a policy that outlines how to support an EMS responder (physically and emotionally) who has experienced verbal or physical violence?	6	9	3
5.3.a	Do supervisors have training in stress recognition and management?	4	10	4
5.3.b	Have supervisors received training on how to identify and respond to EMS personnel expressing a need for breaks, or those suffering from stress exposure?	5	9	4
5.3.c	Does your department have a policy that allows supervisors to encourage responders to seek help?	9	8	1
5.3.d	Are the necessary support services available to EMS responders (e.g., counseling, Stress First Aid, Critical Incident Stress Management (CISM), Employee Assistance Programs (EAP), peer support programs, Crisis Response Teams (CRTs), Chaplains, etc.)?	14	3	1
5.4	Does your department have a non-punitive policy that specifies that coworkers should notify their field officer/supervisor when their partner is showing signs of stress exposure, or has experienced violence/injury?	9	8	1
5.4.a	If yes, are coworkers able to report concerns anonymously?	9	8	1
5.5	Does your department have a policy that specifies that EMS responders should notify their supervisor when they have experienced verbal or physical violence with or without injury?	8	6	4
	Staffing Policy
5.6	Does your department have policies to increase EMS responder staffing to cover overworked EMS responders as needed (e.g., having two additional EMS responders per shift to provide relief/coverage)?	1	4	13
5.6.a	Does your department have adequate staffing to support overworked or vacant positions?	2	5	11
5.6.b	Does your department have an agile overtime policy that can be implemented when someone needs to be taken out of service for emotional/physical recovery?	6	5	7
5.7	Does your department have stress pay/mental health days (i.e., day/days off) available for EMS responders?	2	2	14
5.7.a	Does your department differentiate work-related stress as an injury or a personal illness?	1	7	10
5.7.b	Can personnel use sick leave for mental health days?	9	5	4
5.7.c	Does your department clearly communicate if sick days can be utilized as mental health days?	6	8	4
5.8	Does your department have a policy/procedure to rotate EMS responders from busy stations to less busy stations for recovery time?	1	5	12
5.8.a	If yes, is it voluntary or mandatory?	2	4	12
5.9	Does your department routinely rotate responders between EMS and fire duties to provide relief from EMS overwork?	5	5	8
5.10	Are EMS responders trained on how, when, and who to ask for support and specialized resources when in need of recovery from work?	6	9	3
Count		165	202	173
Percent Checklist Number	Phase 6. Post- Event	30.6%	37.4%	32.0%
	Reporting
6.1	Does your department train on the importance of and methods associated with reporting violent events?	5	10	3
6.1.a	Does your department perpetuate a safe culture for reporting so that members will not be disrespected or dismissed for reporting a violent event (i.e., will all reports be treated with seriousness and respect)?	6	9	3
6.1.b	Does your department encourage the reporting of all incidents of violence (verbal or physical) to reporting systems like EMERG, Occupational Safety and Health Administration (OSHA) 300, National Firefighter Near Miss Reporting System (NFFNMRS), National Fire Incident Reporting System (NFIRS), state requirements, etc.?	3	11	4
6.1.c	Does your department investigate ways to administratively simplify multiple reporting systems to encourage reporting of violent events?	3	10	5
6.1.d	Does your department encourage reporting violence that leads to physical injury and a clear process that leads the EMS responder to report to Workers' Compensation?	9	6	3
6.1.e	Does your department have a way to disseminate immediate and brief information describing the violence experienced by your members?	8	6	4
6.1.f	Does your department have a policy that protects an EMS responder's time - either by going out of service or using overtime - so that they can easily report any acts of violence or exposure they experienced on a call, before they return to service and go on their next call?	5	10	3
6.1.g	Does your department training include guidelines and best practices for documentation (including appropriate terminology) that can help to support the EMS responder, should the EMS responder narrative be used in court proceedings (i.e., inclusive of appropriate documentation for use of force, self-defense, and restraints, etc.)?	5	11	2
6.1.h	Does your department train EMS responders with a checklist that describes what should be included in a patient care report narrative regarding on scene violence targeting responders?	8	7	3
6.1.i	Does your department train EMS responders on how to communicate with police or investigators regarding a violent incident, when appropriate?	6	9	3
6.1.j	Does your department have a policy for collecting data for when dispatch does not advise crews of appropriate staging? Is there a mechanism for reviewing this policy?	4	10	4
6.2	Does your department have a policy that dictates that dispatch will flag previously known violent locations as reported by EMS responders, and this information will be conveyed on future calls without inadvertently identifying individuals?	7	8	3
6.2.a	Does your department have a policy that the violence dispatch flag is included in the Quality Assurance and Quality Improvement (QA/QI) process?	5	10	3
6.2.b	Does your department have policies to regularly update the list of violent locations?	5	8	5
	Organizational Support
6.3	Does your department have accessible and timely medical oversight to clear responders to return to work without docking pay or missing shifts?	8	8	2
6.4	Does your department have a return to work policy that addresses long-term clearance by mental health professionals?	6	7	5
6.5	Does your department issue guidance (SOP/SOG) for dispatchers and supervisors on how to interact with an injured EMS responder (e.g., acknowledging the violent encounter and its impact, not blaming the EMS responder, asking if they need treatment or psychological assistance, informing EMS responders of all reporting options such as Workers' Compensation (if necessary), and assisting them with pressing charges (if desired), asking for their perspective on how this could have been prevented and what departmental resources are needed, contacting or visiting injured EMS responders at their home or medical facility by the department or IAFF local, disseminating information back to the department, providing support to injured responder)	6	8	4
6.5.a	Do your departmenťs supervisors receive recurrent training on this checklist?	4	10	4
6.5.b	Does your department have a policy that dictates immediate supervisor actions as they relate to filing reports of violent incidents?	10	4	4
6.6	Does your department utilize informal After Action Reviews (AAR) following violent events (e.g., What was your mission? What went well? What might we have done differently? What could have gone better? Who needs to know? How could this have been prevented and what resources from the department are needed?)	9	6	3
6.6.a	Are lessons learned from the informal AAR shared in a way that protects the responder's privacy	8	7	3
6.6.b	Does your department have a protocol in place for an AAR of calls that required notifications, updates, or emergency communications?	11	4	3
6.6.c	Is information that is gained after an AAR shared with the rest of the department?	9	6	3
6.6.d	Does your department change policy/SOPs from items learned in the AAR process?	7	7	4
6.7	Does your department measure organizational outcomes that are important to EMS responders (e.g., burnout, job satisfaction, engagement, intention to leave the profession, turnover)?	2	8	8
6.8	Does your department offer recurrent training to field supervisors and leadership on the importance of safety culture, safety outcomes, and organizational outcomes?	5	9	4
	Immediate Mental Health Support
6.9	Does your department have one or more post-incident support programs instituted for EMS responders who need them? (e.g., Stress First Aid, Critical Incident Stress Management (CISM), Employee Assistance Programs (EAP), peer support programs, Crisis Response Teams (CRTs), Chaplains, etc.)	14	2	2
6.9.a	Has your department identified a best practice for implementation of confidential mental health support (e.g., before returning to service, after returning to quarters, informally - when convenient and asked for by EMS responders)?	9	4	5
6.9.b	Has your department considered external resources to provide the appropriate level of support for post-incident needs (e.g., peer support group for high risk occupations, EMS responder-trained psychologists, etc.)?	9	6	3
6.9.c	Has your department considered using an outside agency to handle EAP (should not be in the same building as department administration)?	12	2	4
6.9.d	Has your department trained members on how to access these resources and/or best practices for implementation, should they need them?	10	6	2
	Long-Term Physical and Mental Health Support
6.10	Does your department have a policy that an Employee Assistance Program (EAP) representative, mental health counselor, city insurance case manager, etc., can perform mental health checks on injured EMS responders?	8	6	4
6.11	Are diverse modalities offered within the department for mental health support programs (e.g., Psychological First Aid, Critical Incident Stress Management (CISM), Complementary Alternative Medicine modalities (CAM), HeartMath, Mindfulness-based Stress Reduction programs (MSBR), etc.)?	9	5	4
6.12	Have EMS personnel, regardless of rank, been trained on recognizing signs of cumulative stress, paying particular attention to the long-term impact of this work?	6	8	4
6.13	Do those contracted to provide mental health services have demonstrated experience working with EMS responders?	8	6	4
6.13.a	Are these mental health services accepted by EMS responders?	7	8	3
6.14	Does your department provide recurrent training on adaptive skills, such as coping and resiliency?	3	8	7
6.15	Does your departmenťs training curriculum recognize and train on stress as a chronic occupational exposure, including the relationship between the EMS responder workload and its cumulative stress impact?	3	9	6
6.15.a	Are EMS personnel, regardless of rank, trained on stress as a chronic occupational exposure (i.e., trained on the physiological effects of stress, recognizing cumulative stress exposure in one's self and others)?	4	7	7
	Support for Court
6.16	Does your department train EMS responders to know that your state has criminal statutes, should they be assaulted?	4	10	4
6.17	Does your department provide support and information about available resources for court to the assaulted EMS responder as they maneuver the court/legal system?	5	8	5
6.17.a	Does a member of your department, IAFF Local, or other advocate attend court with the assaulted responder?	11	6	1
6.17.b	Does your department have a policy that specifies that preparation for the judicial process and court appearances are compensable activities?	9	7	2
Count		295	317	162
Percent		38.1%	41.0%	20.9%

9. Discussion

The SAVER Systems Checklist is a multi-level checklist that focuses on policy, training, environmental modifications, and (limited) individual actions with the goal of shifting the burden of workplace violence prevention from the individual EMS responder to the organization for whom that person works. In so doing, the SAVER Systems Checklist is assisting the fire and rescue service in making a paradigm shift, one that moves its hierarchical and often reactionary culture to an environment of flatter hierarchies and primary prevention. Moving the focus upstream has the potential to create the most impact, as it moves prevention from tertiary to primary allowing focus to be placed on organizational policies, communication of safety practices, development of training opportunities, and collaborative teamwork.

Merging disciplines of public health, occupational health psychology, and meeting science was a beneficial strategy in the developmental process of this workplace violence intervention. Techniques that are unknown or unfamiliar to public health research and practice, such as meeting science, may hold benefit for intervention design in occupational safety and health.

10. Practical benefits of methodological processes

In reflecting on the benefits of the ThinkLets process relative to the development of the checklist, the collaboration and facilitation tool presented some efficiencies worth noting here (De Vreede et al., 2006). Essentially, the consensus conference brought together many experts and the ThinkLets organized the interactions into a series of highly structured and intense collaborative meetings. To illustrate this efficiency, consider the following:

• ThinkLets 1–3: six sections reviewed three times equaling 18 separate facilitated meetings

• ThinkLet 4: six sections reviewed six times equaling 36 separate facilitated meetings

• ThinkLet 5: six sections reviewed and rated once each by all SMEs equivalent to six large meetings

By utilizing this collaborative process, the conference accomplished the work of approximately 60 separate meetings with SMEs, including the processing of all information, and the compilation of results. The efficiency gained is further highlighted when one considers the time and resources that would have been required to schedule this many meetings with SMEs who were not co-located, as well as structuring all those meetings without the facilitated processes used here. In other words, the ThinkLets themselves incorporate best practices in meeting science (Allen, Lehmann-Willenbrock, & Rogelberg, 2015) with the efficiency of co-locating SMEs for an intense 2-day conference. Furthermore, a full one-third of the checklist (30% of the final items) was developed during this conference by the SMEs via the collaboration and facilitation. Without both the process and willing participation by the SMEs, the final comprehensive checklist as it is currently constituted, would not exist.

Originally, the Systems Checklist Consensus Conference was slated to use a more technological-focused process, involving ThinkTank technologies. Early into checklist development and conference planning processes the research team decided to depart from the technology-heavy ThinkTank procedures. The procedure was essentially the same as the computer-based ThinkTank but facilitated by a person rather than a software tool. Following the non-computer-based model allowed for more face-to-face discussions among participants, promoting important cross-talk across a diverse set of stakeholders and created a safe environment for open dialogue. Notably, collaboration and discussion were two of the most highlighted aspects expressed by SMEs in their evaluation of conference processes. The evaluative responses collected revealed the positive attitudes, acceptance, and excitement through learning and participating in this new collaborative ThinkLets process. Participants also expressed wanting to bring the ThinkLets technique back to their departments to assist in decision-making procedures. Given the application of this facilitation technique and subsequent feedback, use of ThinkLets processes may be a useful methodologic consideration to explore with other diverse groups of stakeholders in other industries and disciplines.

11. Reflections on the study site feasibility assessment & next steps

A limitation of conducting the anonymous feasibility assessment at SC³, was the inability to ascertain the fire-department specific perspective. By asking members of the four SAVER study sites who attended SC³ who had a grounded understanding of the checklist and its goals to re-evaluate the checklists feasibility, we are better able to accurately determine the approximate timeline for implementation. The feasibility results from both SC³ and re-evaluation from the study sites were largely confirmatory and pointed to the majority of items as ‘most-feasible’ and therefore possible to implement within three to six months. These results provide our implementation efforts with focus and direction, as we aim to implement checklist items that are most impactful and achievable in the given the study timeframe.

The study site feasibility re-evaluation has also been helpful in providing a more holistic view of where the fire service is at in terms of organizational readiness to change regarding workplace violence. The feasibility results indicate that earlier phases of the checklist, occurring prior to patient interaction (where the majority of violent exposures occur), are the most amenable to policy and training intervention. In particular, the study sites indicated that the ‘pre-event’ phase, was the most feasible. This was surprising considering that the policy and training interventions contained within this phase are not in place in these departments. Given that violence has been discussed by the industry for over 40 years, coupled with the more recent concerns of mental health and well-being of members in the fire and rescue service, we think the lack of policy is more indicative of the challenges faced by departments in developing and implementing policy that supports EMS responders and less indicative of low prioritization placed by the organization. It is less surprising that later phases, such as “assessing readiness to return to service” and “post event,” were rated less feasible because these are areas where systems pressures may make policy invention more difficult to achieve.

To prepare for implementation, study sites and other stakeholders will be reconvened to transform the checklist items into model policies using similar facilitated consensus methods from meeting science. These resultant policies will then be translated into accompanying training modules to be implemented and extensively evaluated for impact. The evaluation of the checklist will follow a rigorous quasi-experimental repeated-measures mixed methods design, which will provide us with the evidence needed to support the checklist’s impact on organizational, mental health, and safety outcomes.

12. Limitations

Time constraints were a noted limitation throughout SC³. While the conference proceeded as scheduled throughout all sessions, the checklist development process would have benefited from additional ThinkLet sessions and additional time for de-briefing sessions for the research team. For example, in ThinkLets 1–3 (Idea Generation), the main goal was to allow SMEs an opportunity to generate any checklist items missing from the checklist phase they were reviewing. Data saturation science supported the need for only 3 Idea Generation sessions (Steinhauser et al., 2008; Tracy, 2013), however at the close of Day 1, new checklist items were still being discussed (though few) amongst SMEs. This was in part because not every SME had the opportunity to review every phase of the checklist during ThinkLets 1–3 (they reviewed 3 consecutive phases out of the 6 total). If we had additional time, we may have chosen to incorporate additional Idea Generation sessions, possibly reaching data saturation. Alternatively, each fire department may have interpreted checklist items slightly differently, depending on their organizational goals and resources with which they can dedicate to implementation. Therefore, newly generated ideas that were still percolating at the end of Day 1 may also simply have been a different interpretation of an item already on the checklist. In addition, since the primary goal of ThinkLets 1–3 was to generate new ideas, there was little time within the session for SMEs to dedicate to writing a formal and polished checklist item. Time constraints at the close of conference Day 1 also impacted the authors’ ability to significantly process and refine all newly generated checklist ideas into formal checklist items (a process the candidate items enjoyed) prior to the beginning of ThinkLet 4 the next day.

ThinkLet 4 (Convergence) also experienced some time limitations and would have benefited from a longer session - for both the SMEs doing the activity and the research team on the back end. In particular, the research team was unable to review the additions, deletions, and re-ordering of the entire checklist as a group for the start of ThinkLet 5 (Feasibility Assessment) during which SMEs voted on the resultant checklist. It was only after the conclusion of the conference that the research team was able to convene to review the checklist in its entirety. And while each facilitator was the ‘expert scientist’ of that particular checklist phase, the checklist may have reached a more polished state heading into ThinkLet 5 if the research team was able to give a detailed, more timely review to the checklist across all phases, making an additional review to identify formalities, redundancies, etc. As such, there were very similarly worded checklist items across phases present on the resultant checklist that SMEs voted on for ThinkLet 5. This resulted in the reduction of the checklist from 232 items on the resultant checklist at the close of SC³ to 174 items on the final checklist reported here. Despite these limitations, the feasibility ratings between newly generated items and the original polished checklist items did not differ, thereby helping us feel confident that there were no biases affecting original feasibility assessment (Supplementary data 1). Furthermore, the trends from the re-evaluation of the final checklist (i.e., fully polished and reduced for redundancies) by the SAVER study sites showed results almost identical to the initial feasibility assessment. Thus, we feel confident in the final SAVER Systems Checklist and our development process despite the described time constraints of the conference.

A noted limitation of ThinkLet 5 procedures was the anonymization of the feasibility assessment data which prevented stratification of responses between fire department and non-fire department conference participants (Taylor et al., 2019). The re-evaluation conducted with the four SAVER study sites allowed us to make stronger conclusions regarding the feasibility scores for checklist implementation. There were no missing data in the secondary feasibility assessment supporting our prior conclusion that missing data in the initial assessment came from participants who were not members of fire departments and therefore did not know how to cast their vote.

A final limitation was that the group of invited SMEs did not include all the dispatch and law enforcement perspectives invited. Evaluation feedback recognized this limitation of expert opinion and experience. Despite these absent viewpoints, we feel confident that the key stakeholders in attendance were diverse enough in experience that the impact of these missing stakeholders was small.

13. Conclusions

To our knowledge, the SAVER Systems Checklist addressing violence against EMS responders is the first application of a checklist intervention at a systems level. By utilizing the highly efficient and well-regarded ThinkLets process, key SMEs in the field of EMS were able to come to consensus within two-days to ensure the systems-level checklist was comprehensive for all phases of EMS response. Addressing workplace violence in EMS is a complex issue with major organizational obstacles that require a shift of the current organizational climate. In the current EMS system, fear, blame, and punishment often accompany reports of violence exposures (Taylor et al., 2016). These sentiments perpetuate the belief that violence is a “part of the job,” and a culture of silence and underreporting. The SAVER Systems Checklist seeks to change this by shifting the responsibility for safety from the individual to the organization, focusing on policy and training that prevent injuries and violence in the first place. The checklist underscores important needs for EMS policy and training development critical to responder safety as identified and supported by diverse subject matter experts.

The checklist has the potential to directly benefit EMS responders by reducing violent injury exposures, work-related mental health impacts, and improving organizational outcomes (e.g., burnout, job satisfaction, engagement). It may also lead to improved quality of care and patient outcomes. As we developed the checklist with the fire and rescue service, we will disseminate its implementation results through mechanisms they determine will give it the widest audience and the greatest opportunity for change throughout the industry. We invite those with feedback on the checklist to contact the corresponding author.

Biography

Regan Murray, MPH, EMT is a Project Manager and certified Emergency Medical Technician at the Center for Firefighter Injury Research and Safety Trends (FIRST) at the Drexel University Dornsife School of Public Health. She provides oversight to the project funded by the Federal Emergency Management Agency (FEMA): “Stress and Violence in fire-based EMS Responders (SAVER).” Ms. Murray received her Bachelor of Arts degree from St. Lawrence University and earned her Masters of Public Health degree, concentrating in Community Health and Prevention, from Drexel University.

Joseph Allen, PhD is a Professor in Industrial and Organizational (I/O) Psychology and the Director for the Center for Meeting Effectiveness at the University of Utah. He completed his doctorate in Organizational Science at the University of North Carolina at Charlotte in 2010 and received his Master of Arts degree in I/O Psychology at the UNCC in 2008. Dr. Allen’s research focuses on three major areas of inquiry including the study of workplace meetings, non-profit organizational effectiveness, and emotional labor in various service-related contexts.

Andrea L. Davis, MPH, CPH is the Senior Project Manager at the FIRST Center. She earned her Masters of Public Health degree from Drexel University in 2012 and holds the designation of Certified in Public Health from the National Board of Public Health Examiners. Ms. Davis holds a Masters of Liberal Arts from the Harvard University Extension School and Bachelor of Art from the University of Delaware.

Jennifer A. Taylor, PhD, MPH, CPPS is an injury epidemiologist and the Arthur L. and Joanne B. Frank Professor of Environmental and Occupational Health at the Drexel University Dornsife School of Public Health in Philadelphia, PA. Dr. Taylor is the founding Director of the FIRST Center at Drexel University. She received her doctorate from the Johns Hopkins Bloomberg School of Public Health, specializing in Injury Prevention and Control, where she received the Haddon Fellowship and the ERC-NIOSH Training Fellowship in Occupational Injury. Dr. Taylor’s research investigates the impact of safety climate on occupational injury and related psychosocial outcomes among first responders.