Building Capacity for Injury Prevention: A Process Evaluation of a Replication of the Cardiff Violence Prevention Program in the Southeastern United States

Abstract

Objectives:

Violence is a major public health problem in the United States. In 2016, more than 1.6 million assault-related injuries were treated in U.S. emergency departments. Unfortunately, information about the magnitude and patterns of violent incidents is often incomplete and underreported to law enforcement (LE). In an effort to identify more complete information on violence for the development of prevention programs, a cross-sectoral Cardiff Violence Prevention Programme (Cardiff Model) partnership was established at a large, urban emergency department (ED) with a level I trauma designation and local metropolitan LE agency in the Atlanta, Georgia metropolitan area. The Cardiff Model is a promising violence prevention approach that promotes combining injury data from hospitals and LE. The objective was to describe the Cardiff Model implementation and collaboration between hospital and law enforcement partners.

Methods:

The Cardiff Model was replicated in the United States. A process evaluation was conducted by reviewing project materials, nurse surveys and interviews, and ED-LE records.

Results:

Cardiff Model replication centered around four activities: (1) collaboration between the hospital and LE to form a community safety partnership locally called the United States Injury Prevention Partnership; (2) building hospital capacity for data collection; (3) data aggregation and analysis; and (4) developing and implementing violence prevention interventions based on the data.

Conclusions:

The Cardiff Model can be implemented in the U.S. for sustainable violent injury data surveillance and sharing. Key components include building a strong ED-LE partnership, communicating with each other and hospital staff, engaging in capacity building, and sustainability planning.

INTRODUCTION

Violence is a major public health problem. In 2016, there were more than 1.6 million nonfatal, assault-related injuries treated in U.S. emergency department in the U.S.¹ Despite significant progress made in preventing violence,² incomplete information about the magnitude and patterns of violence limits local prevention efforts. Over half of violent incidents go unreported to law enforcement (LE),^3–5 and this incomplete information often forms the basis of LE and community prevention or response strategies. While violent incidents are often unreported to LE, many victims seek medical treatment. Thus, these violence experiences can be captured in the medical setting.^6–8

To improve community understanding of when, where and how violent injury occurs and to integrate prevention efforts by combining hospital and LE records, the Cardiff Violence Prevention Programme (also known as the Cardiff Model), an evidence informed violence prevention intervention adopted by the National Health Service in the United Kingdom, creates a cross-sectoral partnership between local hospitals and LE.^9,10 The partnership uses LE and aggregated ED injury data to inform community violence prevention programs such as those in CDC’s technical packages which highlight best available and evidence-based violence prevention programs, policies and practices across multiple violence topics (e.g., suicide, youth violence, sexual violence, intimate partner violence, and child abuse and neglect).¹¹ Cardiff Model implementation in the U.S. may have adaptation needs. For example, the U.K. has a public sector health system (i.e., National Health Service) whereas the U.S. system is largely privatized. In the U.S., law enforcement has more local accountability and control often requiring collaboration across multiple agencies.¹²

Cardiff Model evaluations have demonstrated a 36% reduction in hospital admissions related to violence in Merseyside, England;¹³ a 42% decrease in violence-related hospital admissions in Cardiff, Wales relative to comparison cities;⁹ and significant cost-savings for health and justice systems. For every $1 spent on the model, nearly $15 in health system and over $19 in criminal justice system costs are saved.¹¹⁴ While several published articles provide evidence of the effectiveness and cost-benefit of the Cardiff Model;^9,14–19 and of the benefits of cross-sectoral relationships,²⁰ one process evaluation describes model integration into a U.K. ED,²¹ one describes the feasibility and implementation of Cardiff Model integration into a pediatric ED,²² and none document the collaborative ED and LE agency process in the U.S. Therefore, this process evaluation is the first to document the U.S. Cardiff Model adaptation, implementation and collaboration between a large, urban ED with a level I trauma designation and a local county LE agency in the Atlanta, Georgia metropolitan.²

CARDIFF MODEL ADAPTATION AND IMPLEMENTATION

A Cardiff Model replication was conducted from January 2015 through July 2017. During this time, activities focused on four primary areas: (1) collaboration between the hospital and LE to form a community safety partnership; (2) building hospital capacity for data collection; (3) data aggregation and analysis; and (4) developing and implementing violence prevention interventions based on the data.

Materials and Methods

Implementation and Evaluation Approach

A logic model (see Figure 1) was used to define and evaluate this U.S. Cardiff Model implementation and evaluation process. The ultimate outcome is a collaborative, cross-sectoral partnership that successfully engages in violence prevention interventions. This process evaluation documents implementation through short and some intermediate term outcomes. It is also important to note that financial investment (e.g., staff time) is an important consideration when adapting and implementing model, although cost-effectiveness has been documented elsewhere.¹⁴

Figure 1.

Southeastern US replication of the Cardiff Violence Prevention Model’s Logic Model. ED, emergency department; LE, law enforcement.

Design, Setting and Participation

This process evaluation was conducted in the Atlanta metropolitan area and a new partnership developed between an urban ED and a county LE agency. The CDC and model developer Dr. Jonathan Shepherd provided technical assistance. The hospital has approximately 921,000 patient visits each year; of those, approximately 140,000 patients are seen in the ED. The hospital primarily sees adults and serves the greater Atlanta metropolitan area and also sees patients from a larger geographical area as visitors or as the nearest ED with a Level I trauma designation. This teaching hospital is mainly staffed by two universities with over 1,100 active staff physicians, over 800 residents/fellows trained annually, and more than 300 medical students educated at the hospital each year. The local county served by the LE agency includes 271 square miles with almost 700,000 residents.

Process evaluation data were obtained through: document review (e.g., project proposal, recorded meeting minutes), Cardiff Model Screening Tool (CMST; see Figure 2), nurse satisfaction survey, hospital triage data, semi-structured interviews with hospital and implementation stakeholders, and financial records from Fall 2015 to Summer 2016 (see Table 1). As this Cardiff Model adaptation and implementation project was undertaken as public health practice, the resulting feasibility evaluation activities were granted exemption from institutional review board approval by the participating hospital.

Figure 2.

Cardiff Model Screening Tool items. CMST, Cardiff Model Screening Tool.

Table 1.

Data Collection Activities for Replication of the Cardiff Model in the U.S. Process Evaluation

Data Collection Method and Primary Area(s) Used	Sources (n)	Data collection focus
Document Review 1–4	Project proposal, contracts, reports, correspondence, meeting agendas and minutes, presentations, Hospital and LE financial records	Identifying stakeholders, clarifying context, processes and activities, implementation costs
Cardiff Model Screening Tool 2, 3	Cardiff Model Partnership consensus decision (inclusive of hospital and law enforcement representatives)	Identify useful data fields while decreasing burden on hospital staff
Nurse Satisfaction Survey 2	ED RNs (n=78; 51% of 153 Cardiff Model trained ED RNs)	Satisfaction, utility and integration of Cardiff Model data collection screen
Hospital ED Triage Data 2	ED Triage times (April 2015 vs. April 2016 triage chart time data)	Impact on total triage time of Cardiff Model data screen inclusion
Interviews 1–4	Hospital Administration (n=3)	Identifying processes to integrate Cardiff Model into the ED
	Hospital Implementation Team Leaders (n=3)	Implementation logistics, training, and integration into ED processes
	Hospital Technology Team (n=2)	Cardiff Model data collection survey integration into the electronic medical record
	Hospital Nurse Cardiff Model Champions (n=7)	Implementation feedback on the data collection survey and training
	Injury Prevention Partnership Members (n=8)	Capacity Building, Lessons Learned, Use of Cardiff Model data

Note: Primary areas used include: (1) collaboration between the hospital and LE to form a community safety partnership; (2) building hospital capacity for data collection; (3) data aggregation and analysis; and (4) developing and implementing violence prevention interventions based on the data

Process Evaluation Data Sources

Cardiff Model Screening Tool (CMST)

The CMST was integrated into the electronic medical record and located as a primary triage screen allowing nurses easy access (see Figure 2). There are two pre-screening questions: (1) “Do you have an injury?” and (2) “Was someone trying to hurt you?” If a patient answers yes to both, the patient is considered to have a violence-related injury. Assault method and weapon used, date and time that the injury occurred, and geographic location where the injury occurred are then collected from the patient (see Figure 3). CMST data collection ranged from 30 seconds to 2 minutes depending on information provided. Patients are not required to disclose information.

Figure 3.

Flow chart of emergency department visit and Cardiff Model Screening Tool data collection points during care. CMST, Cardiff Model Screening Tool.

Nurse satisfaction survey

An adapted 26 item nurse satisfaction survey²² was administered using paper and pencil from July 6, 2016 through September 1, 2016. For this process evaluation, six items (1-desire to see Cardiff Model reports, 2-satisfaction with the Cardiff Model, 3-commitment to the Cardiff Model, 4-Cardiff Model supportive of hospital goals and mission, 5-Cardiff Model integrated into the nurse workflow, and 6-nurses are able to collect Cardiff Model data from patients), measured on 6-point scales (very dissatisfied to very satisfied and strongly disagree to strongly agree), were used. The hospital’s violence prevention program coordinator and a Cardiff Model nurse champion facilitated survey collection at 7:00 AM and 7:00 PM during shift change meetings. Upon completion, nurses were given a small incentive (CMST branded pen, granola bar, and candy). Over half (51%, n=78 nurses of 153 CMST trained ED nurses) completed the survey

Hospital ED triage and screening data

ED triage times were collected from April 2015 (pre-Cardiff Model implementation; N=1,385) and compared to April 2016 ED triage times (post-Cardiff Model implementation when all ED nurses were trained and collecting CMST data; N=1,556). Triage times ranged from 0–30 minutes, excluding any times beyond 30 minutes as this most likely indicated an incorrect triage charting time (e.g., triage screens not closed). Triage times pre-CMST implementation (n=1234, M = 2.68 minutes, SD = 3.11) were compared to post-CMST implementation (n=1502, M = 2.95 minutes, SD =3.00). Triage times were compared using a one-way ANOVA to address the nurse concern that the CMST would be a time consuming task.

ED screening data was collected throughout the project on a weekly basis from November 2015 through May 2017. Total percentage of patients screened was calculated on a weekly basis by the hospital VPC after full implementation from total number of patients asked CMST questions divided by the total number of patients visiting the ED and converting the number to a percentage. Missed patients were largely due to need for critical care (trauma) services.

Interviews

Semi-structured, face-to-face and/or phone interviews were conducted by the project coordinator throughout the collaboration and implementation phases with key partnership personnel from the hospital and LE agency weekly during the first eight months of the partnership and during monthly partnership meetings thereafter. Interview questions were open-ended and included two questions: (1) what has been considered a success in the last week? (2) what is a challenge and how do you plan to overcome this challenge to Cardiff Model implementation? Depending on responses, probing or explanatory questions were asked. Issues discussed ranged from partnership formation, relationship building, CMST adaptation, CMST integration into the hospital electronic medical record, and staff training. Interview notes provided specific details.

Document Review

Document review included written materials such as meeting agendas and minutes, financial records, formal reports, presentations to various stakeholders (e.g., others implementing the model, community members), training materials, and e-mails.

RESULTS

Local community safety partnership

The prototype community safety partnership first convened in June 2016 with formal introductions and a two-day meeting held at the local LE agency. Core project personnel included two LE representatives, four hospital representatives, a CDC science officer, and a project coordinator. The partnership met monthly for 60–90 minutes and had 30 minute weekly technical assistance (TA) calls with a project coordinator from June to December 2015, and as-needed calls thereafter. In November 2015, the local partnership was formalized by adopting an official partnership logo and name, the United States Injury Prevention Partnership (USIPP), and developed and adopted vision, mission and values statements to guide their violence prevention efforts (see Figure 4). Throughout the project, hospital and LE leadership were informed of project status using verbal and written updates via their organization’s representatives. Leadership support is critical to Cardiff Model sustainability as it allowed for fast CMST integration, identification of nurse champions, nurse training integration for new ED nurses, and overall hospital support of the project.

Figure 4.

United States Injury Prevention Partnership logo and vision, mission and core values statements.

USIPP meetings post-CMST implementation regularly included presentations summarizing violence data (e.g., aggregated maps), exploring evidence-based violence prevention interventions,¹¹ and involving experts on innovative approaches to understand and prevent violence (e.g., social media analysis). USIPP members also met informally after meetings for socialization and relationship building. In addition to regular meetings, USIPP held two celebrations honoring hospital staff and LE (see Implementation of CMST data collection section) and three site visits with other Cardiff Model U.S. implementation sites (i.e., Philadelphia, PA and Milwaukee metropolitan, WI). These multi-site visits allowed discussion centered on challenges and solutions to implementing the Cardiff Model in the U.S. For example, there was robust discussion about which position (nurse, registrar or other employee) was best suited to collect data before deciding that nurses could efficiently and effectively collect valid data. USIPP met periodically with other hospital staff, LE officers, local government officials, and community groups to expand the Cardiff Model.

Building hospital capacity for data collection

The CMST hospital system integration and use required pre-implementation capacity building activities which were overseen by a part-time hospital violence prevention coordinator (VPC). Hospital CMST capacity building meetings involved USIPP’s hospital representative, the VPC, nursing leadership, and information technology staff. To coordinate the integration of the electronic CMST, several pre-implementation activities took place: (1) CMST item selection and adaptation by USIPP, (2) integration of the CMST into the electronic data entry screen and (3) ED nurse training.

First, USIPP decided on the CMST violence-related items (see Figure 2) and an initial screen was integrated into the electronic data screen by hospital staff. The CMST was pilot tested by seven nurse champions (selected by nursing leadership for investment in violence prevention) and modified based on their feedback for ease of use. The CMST resides among the primary triage screens and all patients are screened after triage unless in need of critical medical care (see Figure 3). From hospital records, approximately 70–86% of all ED patients were screened weekly. As of June 2017, 132,284 patients were screened, and 14,292 (11%) reported an injury, 2,942 (2%) reported a violent injury and completed the CMST screen.

Subsequently, by March 2016, 153 or 100% of nurses employed in the ED were taught face-to-face in 1:1 training to use the CMST and the CMST was considered fully implemented. A one-way ANOVA revealed no significant differences between April 2015 and April 2016 triage times F(1,2885)=0.30, p=0.58, ns.

The VPC also integrated CMST training into the ED nurse hiring process for CMST sustainability in March 2016. Quality checks were performed weekly and training refreshers were provided at least every two weeks during nurse shift change meetings and through informal meet and greets (the VPC would walk the ED floor and visit with nurses). During these times, small incentives (e.g., snacks, USIPP-branded pens, badge reels) were distributed to keep nurses engaged. Nurses, hospital staff, and the LE agency were honored at two celebrations with certificates, plaques and appreciation gifts (e.g., jackets, scarves). These celebrations kept Cardiff Model engagement high among hospital staff.

The nurse satisfaction survey data were analyzed for overall satisfaction or agreement by recoding responses to a dichotomous scale (e.g., somewhat satisfied, satisfied and very satisfied responses were combined as “satisfied”). Overall, nurse satisfaction and commitment to the Cardiff Model was high (92.9%) and nurses reported widespread ability to collect CMST data from patients (91.0%; see Figure 5).

Figure 5.

Cardiff Model nurse survey results (selected items).

Since April 2016, approximately 100% or 114 newly hired ED nurses have been trained to use the CMST (267 total ED nurses trained). Ongoing challenges include how and when to screen critical care or trauma patients.

Data aggregation and analysis

The public health agency partner (CDC) in the collaboration maintained the role of aggregating and analyzing combined data. Hospital and LE data aggregation began in December 2015 following the CMST pilot and was updated weekly when new data was available from hospital or LE partners. Aggregated data were used to create geospatial maps using free, open-source R statistical software. Maps were updated every 3 months. Data were examined to identify census block groups with elevated counts of violence and individual businesses or public places with elevated violence counts.

Violence prevention interventions

Based on the mapping activities, the block group from the LE’s jurisdiction (out of 279 potential block groups) with the highest count of violent crime over the preceding year was selected by USIPP for further violence prevention intervention. Detailed mapping allowed for the identification of precise businesses and public spaces that experienced the highest rates of violent crime within the block group. After visiting the hotspot area and several meetings with the LE precinct commander and staff, an active community representative, and a local business group; USIPP piloted a Safety Improvement Project with the precinct and three area businesses to implement violence prevention interventions. These included: supporting existing LE work (e.g., improving LE patrols), supporting local youth engagement activities (e.g., youth basketball program), business environmental improvements (e.g., cleaning the lot and adding plants), and business safety improvements (e.g., increasing lighting and security cameras). Evaluation efforts via photographs, feedback and review of maps are in process. Violence prevention efforts are ongoing as USIPP remains engaged with the local precinct and community.

DISCUSSION

This process evaluation seeks to fill a gap in the literature by describing how the Cardiff Model can be adapted and implemented in the U.S. Building cross-sectoral relationships between hospital ED and LE; building hospital capacity for data collection; aggregating and analyzing combined hospital and LE data; and developing and implementing violence prevention interventions are key steps for Cardiff Model implementation. The CDC provides multiple technical packages that provide strategies and approaches for violence prevention that are based on the best available evidence and may be used to identify strategies for community partnerships to consider.¹¹

These results show the model can successfully be adapted, implemented, and sustained by building a strong local partnership through regular and informal meetings and building hospital CMST capacity (see Figures 2 and 3). From early in the project, hiring a VPC to oversee hospital implementation and monitor data quality remained critical to Cardiff Model success. Weekly monitoring, especially in the early stages of data collection, proved to be important for training nursing staff and ensuring high-quality data. Celebrating milestones, such as CMST integration into the electronic data screen, successful data sharing, and successful mapping, were an effective and integral way to build Cardiff Model momentum and engagement. Additionally, regular VPC communication with nurses and conducting a nurse satisfaction survey were key to understanding CMST implementation, sustaining the data collection process, and identifying improvement needs.

Building hospital capacity required several meetings with key hospital staff (e.g., information technology staff, and nursing leaders) who contributed to planning and provided valuable support for CMST implementation. Meetings for approvals to implement the CMST and integrate it into the electronic data screen required months of coordination and logistics. External foundation funding provided financial support for the VPC (20 hours a week during this pilot project). The VPC is a critical role and hospitals adopting this model are encouraged to determine key staff to help train nurses and monitor data quality or find additional funding for staff support. ED nurses are an integral part of this adaptation as they collect CMST data; other hospitals may want to explore staff who may best suit their patient flow processes to collect CMST data (e.g., nurses, registrars, or other staff).

The VPC operated similarly to Violence Reduction Nurses who were part of the U.K. Cardiff Model implementation. The VPC, a public health professional (vs. a medicial clinician), supported all aspects of implementation from training nurses to coordinating with information technology staff. The VPC served as part of the ED representation on USIPP and took on many administrative tasks both within the ED and USIPP.

USIPP and hospital staff engaged in several discussions and a pilot to best determine what CMST items to include (see Figure 2). The U.S. and U.K. versions of the CMST are almost identical; both concentrate on violence-related (intentional) injuries occurring in public places. Minor U.S. adaptations include word choices (e.g. bar vs. pub) and item ordering. Other U.S.-based Cardiff Model projects may want to consider collecting information on unintentional injuries to inform public health prevention strategies (see the Wisconsin adaptation).²²

U.S. Cardiff Model adaptations may face the most difficulty in determining procedures for aggregating and analyzing data. While Cardiff Model hospital data does not contain patient names, date of birth, or social security numbers, sharing of hospital data on the location of violent incidents must be done in Health Insurance Portability and Accountability Act compliant manner. Public health agencies have the authority to receive protected health information and are the ideal partners for receiving, processing, and aggregating data from multiple sources. Summary information in the form of aggregate maps and lists of businesses experiencing high counts of violence can then be shared with the community safety partnership for injury prevention and control activities.^5,23,24 The Cardiff Model collects, aggregates and shares violence-related injury data, potentially through a public health agency, in a form that may be used by law enforcement and other partners. USIPP’s role in interpreting and reinterpreting data provides the basis for collaborative, cross-sectoral partnership that’s purpose is to prevent violence.

This Cardiff Model implementation faced many limitations and challenges. Unlike the original implementation of the Cardiff Model in Wales, the local LE partner serves a large geographical area which includes multiple hospitals and surrounds other LE jurisdictions. In large metropolitan areas, data from multiple hospitals is likely needed to fully reveal unknown hotspots. Thus, more hospitals and LE agencies, working in collaboration, are needed to create more complete violence maps. Regardless, the Cardiff Model, even in developmental stages, prompts the community safety partnership to examine available data on precise locations of violence and prompts consideration of evidence-based violence prevention efforts. Other ongoing challenges include continuing financial support for a VPC, maintaining high nursing participation in screening, and expanding the CMST to more comprehensively include all critical care and trauma patients. Nurse participation in the sastisfaction survey may be considered low (51%), although this is a similar past response rate when the survey was implemented in a different U.S. hospital.²² The hospital’s primary role is to provide care, thus, medical care needs must first be met and then CMST data collection may occur among critical care and trauma patients. Current hospital discussions include collecting CMST data from these patients prior to discharge. Ongoing patient willingness to disclose violence remains a potential limitation as there are known reasons for non-disclosure to law enforcement,³ although evidence suggests the ED captures large proportions of previously unreported violent injuries.⁵

In summary, the Cardiff Model can be adapted and implemented in the U.S. by building a strong ED-LE partnership, engaging in shared capacity building, communicating regularly both internally and with each other, and planning for sustainability. To achieve full model utility, key adaptations to improve the model included: (1) identifying a hospital staff member to oversee data collection implementation and monitor data quality, (2) identifying appropriate ED staff to screen patients, and (3) building a strong local partnership between ED, LE, and public health agencies.²⁵ Data were used to inform violence prevention efforts where USIPP worked with local community partners on safety improvement projects. These steps and continued local expansion provide the foundation for building buy-in among stakeholders and empowering the community safety partnership to engage in data-driven violence prevention.

What is known on this subject:

• Over half of violent crime in the United States goes unreported to the police.

• The Cardiff Violence Prevention Programme (Cardiff Model) has shown significant reductions in violence-related hospital admissions in the United Kingdom.

• The Cardiff Model is cost-effective: for every $1 spent, nearly $15 in health system and over $19 in criminal justice system costs are saved.

What this study adds:

• This is the first Cardiff Model process evaluation documenting the collaborative emergency department and law enforcement agency process in the United States.

• Four primary areas for Cardiff Model replication are explicated: (1) collaboration between the hospital and LE to form a community safety partnership; (2) building hospital capacity for data collection; (3) data aggregation and analysis; and (4) developing and implementing violence prevention interventions based on the data.

• This process evaluation shows successful replication of the Cardiff Model in the U.S.