Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 Sep 1.
Published in final edited form as: Res Soc Work Pract. 2020 Mar 30;30(6):678–687. doi: 10.1177/1049731520912002

Fidelity Assessment of a Social Work-Led Intervention Among Patients with Firearm Injuries

Vivian H Lyons 1,2, Lina R Benson 3, Elizabeth Griffin 2, Anthony S Floyd 2,3, Sharon W Kiche 2, Kevin P Haggerty 2,4,5, Lauren Whiteside 2,6, Sarah Conover 7,8, Daniel B Herman 7,8, Frederick P Rivara 1,2,9, Ali Rowhani-Rahbar 1,2
PMCID: PMC7508463  NIHMSID: NIHMS1626456  PMID: 32973371

Abstract

Purpose:

To support future development and refinement of social work-led intervention programs among patients with firearm injuries and to demonstrate how a fidelity assessment can be used to adjust and refine intervention delivery in an ongoing trial.

Methods:

We conducted a fidelity assessment of a randomized controlled trial of a social work-led intervention among patients with a firearm injury.

Results:

We found that our study intervention was well implemented, meeting 70% of the fidelity assessment score items, however noted lower fidelity with client-based items.

Discussion:

As a result of fidelity assessment findings, we refined intervention delivery to improve implementation fidelity including beginning to review cases of all patients each month, rather than focusing on patients in crisis. Our fidelity assessment process and findings offer insight into the challenges of implementing an intervention among patients with firearm injuries and highlights the value of monitoring intervention fidelity during an ongoing trial.

Keywords: social work intervention, critical time intervention, fidelity assessment, firearm injury

Firearm injuries represent a substantial public health problem (McLean et al., 2019). Over 36,000 Americans are killed by firearms each year, an average of 100 per day, and many more experience a non-fatal firearm injury (FI) (Centers for Disease Control WISQARS, 2019). For people who survive a FI, the road to recovery can be quite challenging, with impacted mobility, job loss and unstable housing in addition to psychological stresses associated with traumatic injury (Fowler et al., 2015; Greenspan & Kellermann, 2002; Joseph et al., 2019; Smith et al., 2018; Vella et al., 2019). In addition, people who survive a FI are at substantially higher risk of an assault-related injury, particularly firearms, as well as arrest for a firearm-related or violent crime and death from a firearm (Rowhani-Rahbar et al., 2015). Providing support and interventions designed to reduce the increased risk of injury in this population is critical to improve their health and well-being, and may also reduce the overall levels of firearm violence and its effects in the community.

Several established or emerging hospital-based and/or community-based programs have been developed to provide support to patients with a violent injury (Affinati et al., 2016). However, none of these programs have been studied exclusively among patients with firearm injuries and there is a dearth of literature rigorously examining the effectiveness of these programs. The studies that do exist offer mixed findings and have relatively small sample sizes (Aboutanos et al., 2011; Becker et al., 2004; Cheng et al., 2008; Zun, Downey, & Rosen, 2006). These programs had notable differences, and some found reductions in injury and arrests, while others found no change. The mixed findings of these studies may be due, in part, to their program implementation. These studies generally did not report implementation data in a standardized way, limiting our ability to critically review how implementation of their intervention may have impacted their findings. Were these results due to challenges in retention, specific problems with recruitment, or variation in individual practices among persons delivering the intervention? Could these programs be more effective in other populations or with different implementation strategies?

One way to create more standardized and detailed reporting of interventions is with fidelity assessments. A fidelity assessment measures implementation fidelity, or how closely an intervention’s actual implementation aligns with the intended implementation. It is a key component of translating an evidence-based program into practice, as an intervention implemented without fidelity will likely be missing core components of the intervention (Breitenstein et al., 2010), and not achieve the same effect size as the original efficacy studies. Implementation fidelity can be measured in many different ways, including direct observation, interviews, surveys and record review; a large body of work exists discussing strategies for developing a fidelity assessment for a particular intervention, as there is no one standard fidelity assessment for all interventions (Bertram et al., 2015; Carroll et al., 2007; Fixsen et al., 2005; Mahalic, 2004).

In addition to providing insight into whether conflicting findings were driven by variability in program implementation, a fidelity assessment also offers opportunities to improve program implementation or program delivery based on findings from the assessment during an ongoing trial. A fidelity assessment can also identify individual-level challenges with intervention delivery such that investigators and practitioners can anticipate future program delivery issues.

Despite their value, relatively few studies of social work interventions report fidelity measures or describe formal fidelity assessments. A review by Corley & Kim (2016) of 32 recent articles from 2009-2013 describing social work interventions found great variability in each study’s use of fidelity assessment tools to maintain consistency and provide transparent reporting. They found almost 20% of included studies did not include a description of the intervention, only 50% described strategies to monitor intervention delivery and almost 40% did not report supervision of treatment providers (Corley & Kim, 2016). This finding was consistent with two other similar studies which found fidelity measures are often not reported in the social work literature (Naleppa & Cagle, 2010; Tucker & Blythe, 2008), despite the Society for Prevention Research report highlighting the importance of implementation measurement in their Standards of Evidence in 2005 (Flay et al., 2005). To our knowledge, there have not been any published fidelity assessments on interventions offered exclusively to patients with firearm injuries. Use of fidelity assessments conducted specifically on interventions offered to patients with firearm injuries is critical as it increases chances of success for future interventions by offering opportunities to improve intervention design and anticipate challenges when beginning a new program or trial.

This paper describes a fidelity assessment implemented within a randomized controlled trial of the social work-led Helping Individuals with Firearm Injuries-Critical Time Intervention (HiFi-CTI) among patients with a firearm injury. Critical Time Intervention (CTI) was developed in New York City during the 1980s to support vulnerable people during ‘critical periods’ of transition in their lives, and first tested with a randomized controlled trial aimed at reducing recurrent homelessness among men with mental illness (Susser et al., 1997). CTI is a time-limited, case management practice delivered by a social worker trained in the CTI model. The aim is to increase support in the community during transition periods, including following discharge from institutional settings (Center for the Advancement of Critical Time Intervention, 2016; Coalition for Evidence Based Policy, 2013). CTI facilitates community integration and continuity of care by ensuring that a person has enduring ties to their community and support systems during these and other critical periods. It has been widely employed among veterans, people with mental illness, and homeless or incarcerated individuals (Angell et al., 2014; de Vet et al., 2013; Herman et al., 2011; Hwang & Burns, 2014; Lako et al., 2013; Shinn et al., 2015). CTI meets the Coalition for Evidence-based Policy’s rigorous “Top Tier” standard for interventions “shown in well-designed and implemented randomized controlled trials, preferably conducted in typical community settings, to produce sizable, sustained benefits to participants and/or society” (Coalition for Evidence Based Policy, 2013). CTI was, since its inception, thought of as an intervention that could be applied in a wide variety of contexts. The current study is the first to employ an adaption of the CTI model to assist patients with firearm injuries. As CTI is designed to respond to the specific needs of the patient, the services offered to each patient are customized based on their personal goals. This approach was especially compelling for HiFi as patients with firearm injuries faced unique challenges and, with CTI, we were able to directly address each need and recovery goal described by our patients.

We conducted a fidelity assessment to 1) measure how closely HiFi-CTI followed CTI principles, 2) identify individual-level challenges with HiFi-CTI delivery to offer insight into development of future community-based interventions among patients with firearm injuries, and 3) present fidelity assessment results that can be used to contextualize forthcoming findings on the intervention effect from the parent study and facilitate comparisons with other community-based interventions among similar patient populations. We additionally describe changes made to our program both during and following the assessment to improve intervention delivery among our patient population. This effort will help support future development and refinement of social work-led intervention programs among patients with firearm injuries.

Methods

Parent Study – Helping Individuals with Firearm Injuries [HiFi]

The HiFi study was conducted at Harborview Medical Center (HMC) in Seattle, WA. Patients were recruited following admission to HMC for a firearm injury. To be eligible, patients must have been 18 years of age or older at the time of the injury, able to provide consent within 4 weeks following hospital discharge, able to understand and speak English, able to provide at least one direct or alternate contact and planning to reside in King, Pierce, Thurston, Yakima or Snohomish counties for at least 6 months following hospital discharge.

During the nearly three year enrollment period, we enrolled 232 patients. Once enrolled, patients in the treatment group received trauma informed services, hospital- or community-based motivational interviewing, a community-based extended outreach program, and support by a multidisciplinary team of relevant community agencies to help identify relevant recovery resources. The multidisciplinary team served as an ongoing resource for our Support Specialist. The motivational interviewing (MI) component is a patient-centered behavioral technique and attempts to engage patients in order to find reasons to change behavior (Berkowitz & Johansen, 2012; Prochaska et al., 1994, Rollnick, Miller & Butler, 2008). The main aim of MI in this study was to motivate the patient to engage with the extended outreach case management provided by our Support Specialist. The community-based extended outreach program used a modified CTI approach, which we subsequently refer to as HiFi-CTI.

HiFi-CTI included three phases and began with our study Support Specialist, a trained Social Worker, developing a trusting relationship with the patient during the pre- HiFi-CTI period through MI, and identifying initial recovery goals. The Support Specialist typically worked with 10-15 active patients at a time, with multiple clients in each phase. In Phase 1, the Support Specialist got to know the patient, assessed the patient’s needs, and implemented a transition plan intended to link the patient to services and supports in the community. The plan typically included home visits and other meetings with the patient, the patient’s caregivers, and community service providers designed to teach crisis-resolution skills, provide support and advice, and mediate any conflicts. The overarching goal was to connect the patient to people and agencies that will assume the primary role of support. Main components of this phase included making home visits, engaging in collaborative assessments, meeting with existing supports, introducing the patient to new supports and giving advice to patients and caregivers.

In Phase 2, the Support Specialist monitored and adjusted the systems of support that were developed during Phase 1. This phase involved fewer meetings with the patient, as the Support Specialist encouraged the patient to problem-solve with the help of community resources and family members, and intervened only if the patient was receiving inadequate support or if a crisis occurred. The overarching goal was to monitor and strengthen a patient’s support network and skills. Main components of this phase include observing the operation of the patient’s support network, mediating conflicts between the patient and caregivers, supporting modification of the network as necessary, and encouraging the patient to take more responsibility for their care.

In Phase 3, the Support Specialist helped the patient develop and implement a plan to achieve long-term goals (e.g., employment, family reunification) and finalized the transfer of responsibilities to caregivers and community providers. The overarching goal was to terminate HiFi-CTI services with support network safely in place. Main components of this phase included stepping back to ensure that supports could function independently, developing and setting in motion a plan for long-term goals and hosting a meeting with the patient and other supporters to mark final transfer of care.

Fidelity Assessment

We partnered with experts from the Center for the Advancement of Critical Time Intervention (CACTI) to conduct the formal fidelity assessment of the intervention, as the center had already developed a CTI Fidelity Scale which has been previously employed in a number of other settings to evaluate the degree to which implementations follow CTI principles (Conover & Herman, 2017; Shaw et al., 2017). However, the HiFi-CTI varied from CTI in important ways, namely differences in intervention timeframe, geographic area and number of Support Specialists. We therefore believed it would be necessary to modify the CTI Fidelity Scale to account for these differences while still measuring how well HiFi-CTI adhered to the core CTI principles. Since we were unable to find guidelines for how to adapt an existing fidelity assessment for an evidence-based intervention to evaluate an adaptation of the intervention, we instead worked directly with CACTI personnel to adapt their CTI Fidelity Scale for HiFi-CTI using an iterative, collaborative process.

The adaptation began with an in-person training on the core CTI principles and existing fidelity assessment tools delivered at CACTI for the fidelity assessment team leader. The fidelity assessment team then reviewed each individual fidelity criterion to identify elements that needed revision in order to adequately reflect HiFi-CTI, our adapted version of CTI. Following this initial review, we met with the HiFi study team to continue the review, including identifying which of our data elements should be used to evaluate each of the fidelity assessment items as our HiFi-CTI data collection tools differed from CTI data collection tools. Over the course of four months we iteratively refined the HiFi-CTI fidelity assessment in collaboration with our CACTI partners prior to the start of the fidelity assessment.

The final adapted fidelity assessment items for HiFi-CTI are presented in Table 1. While we made some slight changes to the original CTI Fidelity Scale to fit our intervention delivery, we maintained the original CTI Fidelity Scale structure with individual fidelity items to evaluate client-based, worker-based and team-based domains with each individual fidelity item measuring one or two of the basic principles of CTI (Table 2).

Table 1:

CTI Fidelity Rating Form for the Helping Individuals with Firearm Injuries Study Adapted CTI Intervention

Assessment Domains Fidelity Measures  Measures Score*
Client-Based Intake Assessment and Early Engagement of Client Measuring: In person meeting with patient and completion of the intake form.
Assessed using: Support Specialist encounter documentation from contacts with patients during pre-CTI that included a description of goal setting.		 5
Community-Based Meetings Measuring: In person meetings, rather than phone calls, with the patient and their community support.
Assessed using: Encounter documentation.		 3
Intensive Phase 1 Measuring: Sufficient meetings with both patient and their community support providers to support the initial intensity required of Phase 1.
Assessed using: Encounter documentation.		 1
Stepping Back for Sustainability Measuring: Sufficient meetings with both patient and their community support providers to support the transfer of care required of Phase 3.
Assessed using: Progress notes.		 1
Phased Intervention Measuring: Transition between phases occurs when planned.
Assessed using: Progress notes and closing notes.		 3
Time-Limited Measuring: The intervention does not last longer than 6 months past hospital discharge.
Assessed using: Closing notes.		 4
Few Dropouts Measuring: The Support Specialist’s engagement did not end earlier than 6 months following hospital discharge.
Assessed using: Closing notes.		 4
Worker-Based Small Caseload Size Measuring: The Support Specialist’s weighted** caseload is < 20.
Assessed using: Phase-Date Form		 5
Quality of CTI Worker’s Role Measuring: The Support Specialist conducted ongoing community assessments and interventions in person, were available to patients and their community supports, monitored relationships and encouraged communication between patients and their community supports.
Assessed using: Support Specialist structured interview and Supervisor-Coordinator focus group.		 5
Quality of Phase Plans Measuring: Limited number of focus areas selected, reason for focus area described and summary of prior efforts.
Assessed using: Phase Plans.		 2
Quality of Progress Notes Measuring: Sufficient detail on encounter documentation to identify type of contact (e.g. in-person, phone call), who the contact was with (e.g. patient, community support, service provider), description of what was discussed, emerging concerns and next steps.
Assessed using: Encounter documentation.		 2
Quality of Closing Note Measuring: The Support Specialist had a final meeting with the client to discuss the end of CTI and transfer-of-care.
Assessed using: Closing notes.		 1
Team-Based Weekly Team Supervision Measuring: Weekly team discussions of patients that allow consistent review of all cases each month.
Assessed using: Team Supervision Forms.		 5
Quality of Supervision Measuring: Feedback from team members to the Support Specialist includes positive feedback when activities are consistent with the CTI model and alternative suggestions are offered when activities are inconsistent.
Assessed using: Weekly Team Meeting Observation Form		 5
Quality of Fieldwork Coordination Measuring: Phase Date Form is distributed to team regularly and CTI documentation is reviewed with Support Specialist Supervisors regularly.
Assessed using: Support Specialist structured interview and Supervisor-Coordinator focus group.		 5
Total Fidelity score and rating: 70% of all fidelity items met. Well Implemented
Open in a new tab
*

Fidelity Assessment Scoring System: if ≤40% of assessment components are met, the item receives a score of 1 and is rated as ‘Not Implemented’, if between 41-54% of assessment components are met, the item receives a score of 2 and is rated as ‘Poorly Implemented’, if between 55-69% of assessment components are met, the item receives a score of 3 and is rated as ‘Fairly Implemented’, if between 70-84% of assessment components are met, the item receives a score of 4 and is rated as ‘Well Implemented’, and if ≥84% of assessment components are met, the item receives a score of 5 and is rated as ‘Ideally Implemented’.

**

Patient caseload is weighted according to their progress through Phase 1, 2 and 3 as patients in Phase 1 require more intensive contact and engagement than patients in Phase 2 or 3.

Note: Each score corresponds to the proportion of specific items met in the worksheet evaluation for each fidelity assessment item.

Table 2:

The 12 Core Principles of the Critical Time Intervention (CTI)

CTI Principles Description
1. Time-limited The intervention should be no longer than 9 months.
2. Phased approach There should be a pre-CTI phase followed by three phases of equal duration.
3. Focused Each phase should focus on specific goals.
4. During transition The intervention should occur during a period of transition in client’s lives that is the same for all clients in program that is defined as ‘leaving from’ or ‘coming to’ (i.e. recurrent homelessness).
5. Care coordination The intervention uses a phased linking process to connect clients to community-based services.
6. Decreasing intensity The intervention intensity decreases over time with each phase as community supports assume responsibility.
7. Community-based Meetings should occur within the client’s own community, not in the program office.
8. Strengths-based The intervention should use a strengths-based, recovery-oriented approach where the Support Specialists conduct phase planning and goal setting via shared decision-making.
9. Familiarity with client Each Support Specialist should establish familiarity with the client during pre-CTI prior to beginning the intervention.
10. Continuity of care Intervention continues throughout the 9-month intervention period with the CTI Support Specialist activities tied to the linking process.
11. Small caseloads Each Support Specialist should have no more than 20 patients at a time. This should be calculated using a weighted system to account for the varying intensity of support required in each of the three CTI phases.
12. Weekly team supervision The Support Specialists should have a weekly meeting led by an experienced clinician who can provide support and troubleshoot case management approaches.
Open in a new tab

The adaptation reflected the differences between CTI and HiFi-CTI. HiFi-CTI was 6 months long, whereas the initial CTI program was 9 months long. Thus, the number of meetings per time period criteria was reduced to reflect the shorter timeline for HiFi-CTI. Additionally, HiFi was conducted over a 5 county wide geographic area that was part of the catchment area of the recruiting hospital, compared to the typical hyper-local delivery of CTI. As a result, the number of community-based, in-person meetings with the patients required per phase were modified to reflect the HiFi-CTI reliance on alternative forms of communication for intervention delivery.

Finally, the HiFi-CTI program had only one Support Specialist, instead of the multiple Support Specialists in many other CTI programs, so we modified the fidelity assessment tools to reflect a single Support Specialist. As a result, the ‘Weekly Team Supervision’ fidelity item initially included a criterion outlining Support Specialist attendance to the weekly team meetings when clients were discussed. However, in HiFi, our Support Specialist was, by definition, present for all cases discussed (as she did all case presentations), so we removed that criterion from the HiFi-Fidelity assessment. Similarly, the Team-Based fidelity domain item focused on quality of fieldwork coordination initially included criteria for regular communication between Support Specialists while in the field was removed.

To ensure that our fidelity assessment would be less subject to random variation over time in HiFi-CTI program implementation fidelity over time, we defined our fidelity assessment period from 12/1/2017-6/1/2018 and reviewed 6-months of Support Specialist notes and documentation, using the CACTI recommended sampling and scoring strategy for fidelity assessments. As the fidelity assessment focused on the implementation of HiFi-CTI, we restricted our study to patients in the intervention arm of our parent trial who were eligible for HiFi-CTI.

For each individual item in the fidelity assessment, we used the pre-specified worksheets developed by CACTI to assess how closely that specific item aligned with initial CTI goals and practices, using separate worksheets for each fidelity item. For each item worksheet, we followed the sampling strategy for that specific assessment item and reviewed the appropriate documentation including: the Support Specialist’s encounter documentation, progress notes, phase plans, phase-date forms, closing notes from the fidelity assessment period, notes from a focus group with the lead investigators of the study, an interview with the Support Specialist and observation of a weekly team supervision meeting (see Figure 1 for the data source for each fidelity item).

Encounter documentation included date of contact, intervention phase of patient at the time of the encounter, and location of contact. Progress notes for each encounter included a description of meeting including a summary of the contact, any new goals set, social support or service providers who attended, and plans for next steps in achieving the patient’s set goals. Phase plans include the patient stated goals for the upcoming CTI phase. The phase-date form indicated the patient’s current phase, Support Specialist’s impression of pattern of progress, level of interest and how useful the patient found CTI for each phase separately. Closing notes included a summary of the patient’s overall progress through CTI, description of their engagement with the Support Specialist and specific goals met through CTI. Our focus group was conducted with the Principal Investigator of the randomized trial, a Co-Investigator with specific expertise in social work interventions who provided mentorship and support to the Support Specialist and the trial’s Research Coordinator who managed the Support Specialists day to day activities and reporting. As we only had one Support Specialist, we conducted an interview with them instead of the typical focus group that other CTI Fidelity Assessments have employed. To structure the focus group, interview and meeting observation we used the pre-existing CTI Fidelity Scale facilitation guides which included questions on challenges of intervention delivery, communication with other staff and how well the interventionist followed CTI principles.

Using the formal fidelity assessment worksheets that had been adapted in consultation with CACTI staff, abstracted data were compared to item-specific CACTI set goals and benchmarks to calculate item-specific scores and the overall HiFi-CTI program fidelity score (Table 1). For each item, a score was calculated by dividing the total number of criteria met by the total number of criteria possible. Using the ‘Intake Assessment and Early Engagement of Client’ fidelity item in the Client Based Domain as an example, we used the dates of the fidelity assessment to identify 5 patients who completed Pre-CTI in the time-frame of interest using the CACTI sampling strategy for that item. The number of patients sampled for each HiFi-CTI fidelity assessment item was the same as in the CTI fidelity assessment. We measured two items for each of the 5 sampled patients, 1) if there was ≥ 1 meeting with the Support Specialist per month during the Pre-CTI phase, and 2) if the intake assessment was dated during the Pre-CTI phase. The total number of points contributed by each client (max 2) was added for all 5 patients, and then divided by 10 (the total number of possible points for this item) to get the percentage of criteria that were met for this fidelity item (see Table 1 for all fidelity items).

The overall program fidelity score was calculated by adding fidelity item scores for each item and dividing by the total number of rated items. While each fidelity assessment item was calculated using aggregate patient data, there was no patient-level fidelity score, only item specific and overall program fidelity scores. Following the fidelity assessment, we presented the findings to the HiFi-CTI study team along with recommendations for improving specific fidelity assessment item scores.

Intervention Dose

Unlike some interventions, HiFi-CTI allows patients to determine their level of engagement with the Support Specialist. As the fidelity assessment includes measures of patient engagement, enrolling many patients who elect to not participate in HiFi-CTI will result in a reduced fidelity assessment score. We additionally created a measure for patient intervention “dose” to allow a more granular examination of patient characteristics associated with participation in HiFi-CTI. We were unable to identify standards for defining intervention dose for CTI programs and related interventions in a literature review. Thus, to reflect the wide variety of patient engagement in our study, we defined a continuous intervention dose variable where each in-person contact or phone call during the 6-month intervention period following pre-HiFi-CTI was counted as a contact with our Support Specialist and increased the patient intervention dose by 1, with no upper limit. Our analysis of intervention dose was restricted to intervention patients who had completed their 6 months of CTI as their dose was liable to change up until the end of Phase 3.

Intervention patients who did not meet with or speak with our Support Specialist had an intervention dose of 0. Text-messaging, emailing and social media messaging did not contribute to increasing the intervention dose, as these methods of communication do not easily facilitate the motivational interviewing and rich discussion that characterize CTI. Although these methods of communication were not included in the dose calculation, they did facilitate communication that led to contacts captured in our dose measure.

Following development of this dose measure, we used the distribution of dose to empirically categorize patients into three groups to facilitate identification of patient characteristics associated with HiFi-CTI engagement. The three groups were: intervention patients with no dose (dose = 0), intervention patients with a low dose (1 ≥ dose ≤ 6) and intervention patients with a high dose (dose ≥ 7). Patients were grouped into one of these three groups based on their intervention dose at the end of the 6-month period.

Data Analysis

We tested for differences between intervention dose groups using the Fisher’s exact or one-way ANOVA tests, as appropriate, for each individual and injury characteristic including injury severity score and injury intent. Patient demographic and injury data were collected at study enrollment using a detailed survey and chart abstraction. Study procedures were approved by the University of Washington Human Subjects Division (Institutional Review Board), and study participants provided informed consent for all study procedures. Analyses were conducted in Stata 14 (StataCorp, 2015).

Results

Fidelity Assessment

Using the adaptation of the CTI Fidelity Scale and rating system, the overall implementation of the HiFi-CTI program was found to meet 70% of the HiFi-CTI fidelity items, corresponding to a ‘well implemented’ program (Score of 1 = Not implemented, 2 = Poorly implemented, 3 = Fairly Implemented, 4 = Well Implemented, 5 = Ideally Implemented) (Figure 1). We observed some variability in the average scores for each fidelity assessment domain. Client-based fidelity items had an average score of 3 (‘fairly implemented’), worker-based fidelity items had an average score of 3, and team-based fidelity items had an average score of 5 (‘ideally implemented’).

Intervention Dose

There were 90 intervention patients who had completed HiFi-CTI at the time of the fidelity assessment and were therefore included in this analysis. On average, patients met with our Support Specialist for HiFi-CTI intervention delivery 5 times during the 6-month intervention period. Low dose patients had a median of 3 meetings during follow up (Interquartile range [IQR]: 2-5), and high dose patients had a median of 14 meetings during follow up (IQR: 8-17). There was no significant difference between intervention dose groups with respect to age, race, employment, stable housing, injury intent, household income or injury severity score. There was a greater proportion of males in the no intervention dose and low intervention dose groups than in the high intervention dose group (100%, 86% and 73%, respectively; p < 0.01). Similarly, a greater proportion of no intervention dose and low intervention dose patients reported an arrest in the year prior to study enrollment compared to high intervention dose patients (84%, 72% and 53%, respectively, p = 0.03) (Table 3).

Table 3:

Demographic Information by Intervention Dose Group

Characteristics No Intervention Dose
N=31		 Low Intervention Dose
N=29		 High Intervention Dose
N=30		 Total

N=90		 p
Patient Age, yrs [x‒ (SD)] 29.3(10.0) 31.7(11.0) 32.5(17.0) 31.1(13.0) 0.57
Male [N(%)] 31(100.0) 25(86.2) 22(73.3) 78(86.7) 0.01
Race [N(%)] 0.74
  White 14(45.2) 11(37.9) 14(48.3) 39(43.8)
  Black 14(45.2) 14(48.3) 10(34.5) 38(42.7)
  Asian 0(0.0) 2(6.9) 2(6.9) 4(4.5)
  Hawaiian/Pacific Islander 0(0.0) 1(3.4) 1(3.5) 2(2.3)
  American Indian 3(9.7) 1(3.5) 2(6.9) 6(6.7)
Hispanic [N(%)] 3(9.7) 5(17.2) 5(16.7) 13(14.4) 0.65
Employment [N(%)] 0.53
  Working 16(55.2) 19(65.5) 21(70.0) 56(63.6)
  Laid Off 7(24.1) 4(13.8) 3(10.0) 14(15.9)
  Student 1(3.5) 3(10.3) 3(10.0) 7(8.0)
  Retired 0(0.0) 1(3.4) 0(0.0) 1(1.1)
  Other 5(17.2) 2(6.9) 3(10.0) 10(11.4)
Stable Housing [N(%)] 0.3
  Stable 19(63.3) 14(53.8) 24(80.0) 57(66.3)
  Transitional 5(16.7) 7(26.9) 3(10.0) 15(17.4)
  Unstable 6(20.0) 5(19.2) 3(10.0) 14(16.3)
Injury Intent [N(%)] 0.802
  Assault 24 (77.4) 22 (75.9) 20 (66.7) 66(73.3)
  Unintentional 5 (16.1) 6 (20.7) 7 (23.3) 18(20.0)
  Undetermined 2 (6.5) 1 (3.4) 3 (10.0) 6 (6.7)
Patient Household Income [N(%)] 0.22
  <$25K 13(61.9) 15(62.5) 10(40.0) 38(54.3)
  25-50K 7(33.3) 5(20.8) 6(24.0) 18(25.7)
  50-100K 1(4.8) 3(12.5) 6(24.0) 10(14.3)
  >100K 0(0.0) 1(4.2) 3(12.0) 4(5.7)
Injury Severity Score [x‒ (SD)] 8.8(9.0) 11.8(16.0) 11.9(15.0) 10.8(10.0) 0.32
Arrested in Past Year [N(%)] 26(83.9) 21(72.4) 16(53.3) 63(70.0) 0.03
Open in a new tab

Abbreviations: SD: Standard Deviation; Missing information: Race (n=1), Employment (n=2), Marital Status (n=23), Income (n=20), Arrest in prior year (n=1), ISS (n=12).

Note: Data presented in Table 2 were collected when patients initially enrolled in the Helping Individual’s with Firearm Injuries study. P values were obtained using Fisher’s exact text for categorical variables or one-way ANOVA tests for continuous variables. Stable housing includes individuals living in a private home, private apartment, or other dwelling (e.g. trailer) who indicated that the dwelling was their address. Transitional housing includes individuals living in a private home, private apartment, or other dwelling (e.g. trailer) who indicated that the dwelling was not their address. Unstable housing includes individuals who indicated that they were couch surfing, homeless, or in a shelter.

Discussion

Overall, the HiFi-CTI program was well implemented according to the principles and core components of the CTI model and reflects an average of our lower client- and worker-based fidelity items and high team-based scores. Our lower client scores reflect the large proportion (34%) of intervention patients who did not respond to our Support Specialist and so did not receive any intervention. Our lower worker-based scores were likely due to differences in the information captured in the HiFi-CTI and CTI phase plan, progress notes and closing notes. Rather than measure how well our Support Specialist was documenting these notes based on initial HiFi-CTI expectations, we decided to use the CTI standards of reporting for the fidelity assessment to prevent over-customization of the fidelity assessment and ensure we were measuring how well HiFi-CTI implemented CTI.

However, there were additional changes made to HiFi-CTI program implementation between the fidelity assessment training and when the fidelity assessment was conducted not reflected in our fidelity process scores. During the fidelity assessment training, we realized that the weekly team supervision measure was likely not being met and it was not tracked formally. Initially, our Support Specialist would select the patients thought to need discussion among the study team members. These clients were often selected when the Support Specialist needed help thinking through strategies to address specific challenges, if their case was particularly complex or if there was a major change in their case that changed the Support Specialist’s approach. While this allowed greater focus on patients in crisis, it resulted in infrequent discussion of other patients. Following the fidelity assessment training, we began tracking which patients were discussed at each weekly meeting to calculate the proportion of patients discussed per week and per month.

We additionally restructured our weekly meetings so that the Support Specialist had more dedicated time allotted to discuss intervention patients and made it our goal to discuss, however briefly, every intervention patient each month. In November 2017, the first month during which we tracked the proportion of patients discussed per month, we reviewed 69% of patients. In December 2017, we reviewed 86%. By January 2018, we were reviewing 100% of patients every month. Our score of 5 for the weekly team supervision fidelity assessment score is reflective of the integration of these changes. These changes allowed both more opportunity for consistent oversight and increased opportunities to discuss patients’ needs and service delivery issues.

Importantly, the process of conducting the fidelity assessment increased our program implementation fidelity and changed practice. The assessment process re-focused our attention on intervention oversight and monitoring, providing a level of scrutiny over intervention fidelity that was typical at the start of our study but had lapsed over time. This increased attention likely influenced implementation in other ways not captured in this study, potentially by increasing contact attempts, or improved charting during the fidelity assessment period. In addition, our selection of the fidelity assessment period may have influenced our results, as there is often natural variation in intervention delivery over time. Future studies could implement regular fidelity assessments to maintain high intervention fidelity and ensure fidelity assessment findings are not reflective of temporal trends.

The assessment process also identified individual patient level factors that influenced overall program implementation fidelity with lower client-based fidelity items, primarily driven by patient desire to connect with our Support Specialist and engage with other community supports and service providers. This directly impacted five of the seven client-based fidelity assessment items (Figure 1). Our development of the intervention dose measure was a first attempt to identify patients who might require more proactive engagement during the pre-HiFi-CTI phase. This measure could be refined and employed in future CTI adaptions with similar patient populations to improve patient engagement with the intervention.

Our patients changed their phone numbers frequently, moved frequently, and often did not provide a home address or forwarding information. While this could be indicative of houmelessness and instability, it may also reflect fear-driven systems avoidance, especially amongst patients with an assault-related firearm injury. Assault patients in our study voiced concern over their shooter attacking them again and discussed employing multiple strategies to ensure their safety including changing their phone number, moving frequently and not disclosing their location to family or friends. In addition, a greater proportion of patients who reported an arrest in the past year had no intervention dose compared to patients with a high intervention dose. Recent engagement in the criminal justice system may have also led patients to eschew communication with our Support Specialist and study team for fear of reporting illegal behavior to law enforcement. These behaviors increase the difficulty of meeting regularly with the Support Specialist and increase likelihood of dropouts. Future violence intervention programs, especially programs like HiFi-CTI that require many in-person contacts among patients with firearm injuries, should proactively consider how to structure their patient contacts to counteract systems avoidance. One strategy could include adopting a ‘neighborhood storefront’ approach to the staff office, as was done in an early CTI trial, where the office welcomes patients to drop in, have a snack and connect with a CTI Support Specialist, even without an appointment (Conover et al., 1997). Having a physical space where patients can come to reliably connect with study staff and receive services would provide a good alternative communication strategy for patients who do not feel safe leaving contact phone numbers or addresses. This was attempted for HiFi, but the space available was a conference room in the hospital, not an easily accessible space in the community. It was never utilized by patients.

While conducting the fidelity assessment resulted in increasing weekly team supervision and patient case discussion, the goals and components of the HiFi-CTI intervention remain unchanged. Most importantly, we do not believe that those changes impacted our ability to evaluate the HiFi study as variations in HiFi-CTI delivery would be functionally similar to those observed in intervention delivery between patients during the course of the study. While variation in HiFi-CTI delivery would impede our ability to examine the effect of a single HiFi-CTI intervention delivery scheme, we are still able to measure the efficacy of treatment assignment with an intent-to-treat analysis. If the fidelity assessment did influence intervention delivery or patient engagement with the Support Specialist (e.g. increased frequency of client case discussion led to increased contact attempts and increased contacts), that can be captured in a per-protocol-effect analysis that accounted for patient adherence to the intervention using the number of contacts per phase. If any change in intervention implementation was of concern given specific parameters of the intervention, future studies could employ fidelity assessments regularly to maintain consistent intervention delivery.

Limitations

This is one of the first fidelity assessments of a Social Work-informed intervention to have been conducted exclusively amongst patients with firearm injuries, and the first adaptation of the CTI program to this unique patient population. However, it should be noted that we assessed fidelity at only one point in time. As a result, we are unable to determine if the results observed would have differed had the assessment occurred during a different period in the study. Future studies should consider implementing multiple fidelity assessments conducted periodically to address this limitation. Additionally, the fidelity assessment for HiFi-CTI had not been systematically tested for validity or reliability when it was implemented. However, we collaborated closely with CACTI and made use of the pre-existing fidelity assessment tool developed for CTI, which had been tested.

Discussion and Applications to Practice

Core to the National Association of Social Workers Code of Ethics is the importance of using evidence-based practices when they are available (National Association of Social Workers, 2017). In order to effectively apply those practices, fidelity assessment is critical to ensure proper implementation and adherence. This ensures quality and guards against implementation of adaptations of evidence-based programs which no longer contain their critical core components. It can also offer insight into inconclusive study findings, as a fidelity assessment conducted during the study period can identify which intervention delivery components were not optimally implemented and might have influenced the overall study findings. Finally, improvements in intervention fidelity that we implemented as a result of fidelity assessment efforts conducted midway through our study demonstrates the value of fidelity monitoring to ensure intervention quality in a randomized trial.

Patients with firearm injuries are an especially vulnerable population, and social workers are uniquely positioned to understand the social justice issues that are core to disparities in firearm violence while providing appropriate support. Given the high workload of social workers, program monitoring which includes regular review of all clients, as was implemented with HiFi as a result of our fidelity assessment, may ensure fewer clients fall through the cracks. Similarly, periodic fidelity assessments of ongoing programs may provide timely opportunities to reallocate resources to ensure program goals are met at the client, worker and team levels and ensure program fidelity is maintained as the delivery of new interventions expand in the community.

Acknowledgments:

The authors thank Alvaro Martinez, Manal Jmaileh and Navya Gunaje for their valuable contributions to this study. Funding for this study was provided by the City of Seattle (contracts DA16-1570, DA17-1570, DA18-1570 and DA19-1570), Arnold Ventures, and the National Center for Advancing Translational Sciences of the National Institutes of Health (UL1 TR002319 and TL1 TR002318). The views expressed in this report are the authors’ and do not necessarily reflect the views of the funders.

References

  1. Aboutanos MB, Jordan A, Cohen R, Foster RL, Goodman K, Halfond RW, Poindexter R, Charles R, Smith SC, Wolfe LG, Hogue B, & Ivatury RR (2011). Brief violence interventions with community case management services are effective for high-risk trauma patients. The Journal of Trauma, 71, 227–228. doi: 10.1097/TA.0b013e31821e0c86 [DOI] [PubMed] [Google Scholar]
  2. Affinati S, Patton D, Hansen L, Ranney M, Christmas AB, Violano P, Sodhi A, Robinson B, & Crandall M (2016). Hospital-based violence intervention programs targeting adult populations: an Eastern Association for the Surgery of Trauma evidence-based review. Trauma Surgery & Acute Care Open, 1(1), 1–7. doi: 10.1136/tsaco-2016-000024 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Angell B, Matthews E, Barrenger S, Watson AC, & Draine J (2014). Engagement processes in model programs for community reentry from prison for people with serious mental illness. International Journal of Law and Psychiatry, 37, 490–500. doi: 10.1016/j.ijlp.2014.02.022 [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Becker MG, Hall JS, Ursic CM, Jain S, & Calhoun D (2004). Caught in the Crossfire: the effects of a peer-based intervention program for violently injured youth. The Journal of Adolescent Health : Official Publication of the Society for Adolescent Medicine, 34, 177–183. doi: 10.1016/j.jadohealth.2003.04.001 [DOI] [PubMed] [Google Scholar]
  5. Berkowitz SA, & Johansen KL (2012). Does motivational interviewing improve outcomes? Archives of Internal Medicine, 172, 463–464. doi: 10.1001/archinternmed.2012.155 [DOI] [PubMed] [Google Scholar]
  6. Bertram RM, Blase KA, & Fixsen DL (2015). Improving programs and outcomes: Implementation frameworks and organization change. Research on Social Work Practice, 25, 477–487. doi: 10.1177/1049731514537687 [DOI] [Google Scholar]
  7. Breitenstein SM, Gross D, Garvey CA, Hill C, Fogg L, & Resnick B (2010). Implementation fidelity in community-based interventions. Research in Nursing & Health, 33, 164–173. doi: 10.1002/nur.20373 [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Carroll C, Patterson M, Wood S, Booth A, Rick J, & Balain S (2007). A conceptual framework for implementation fidelity. Implementation Science, 2(1), 40–49. doi: 10.1186/1748-5908-2-40 [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Center for the Advancement of Critical Time Intervention. (2016). Critical Time Intervention Model. Avaliable from: https://www.criticaltime.org/
  10. Cheng TL, Haynie D, Brenner R, Wright JL, Chung S, & Simons-Morton B (2008). Effectiveness of a mentor-implemented, violence prevention intervention for assault-injured youths presenting to the emergency department: results of a randomized trial. Pediatrics, 122, 938–946. doi: 10.1542/peds.2007-2096 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Coalition for Evidence Based Policy (2018). Evidence Summary for the Critical Time Intervention Social Programs That Work. Laura and John Arnold Foundation; https://evidencebasedprograms.org/document/critical-time-intervention-evidence-summary/ [Google Scholar]
  12. Conover S, Berkman A, Gheith A, Jahiel R, Stanley DA, Geller PA, Valencia E, & Susser E (1997). Methods for successful follow-up of elusive urban populations: an ethnographic approach with homeless men. Bulletin of the New York Academy of Medicine, 74, 90–108. doi: 9211004 [PMC free article] [PubMed] [Google Scholar]
  13. Conover S, & Herman D (2017). Critical Time Intervention Fidelity Scale. Unpublished manuscript, Department of Epidemiology and the Center for Homelessness Prevention Studies, Columbia University Mailman School of Public Health, New York. [Google Scholar]
  14. Centers for Disease Control and Prevention, National Center for Injury Prevention and Control. Web-based Injury Statistics Query and Reporting System (WISQARS) [online]. (2017) [cited 2019 Dec 20]. Available from URL: www.cdc.gov/injury/wisqars
  15. Corley NA, & Kim I (2016). An Assessment of Intervention Fidelity in Published Social Work Intervention Research Studies. Research on Social Work Practice, 26, 53–60. doi: 10.1177/1049731515579419 [DOI] [Google Scholar]
  16. de Vet R, van Luijtelaar MJA, Brilleslijper-Kater SN, Vanderplasschen W, Beijersbergen MD, & Wolf JRLM (2013). Effectiveness of case management for homeless persons: a systematic review. American Journal of Public Health, 103(10), e13–26. doi: 10.2105/AJPH.2013.301491 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Flay BR, Biblan A, Boruch RF, González Castro F, Gottfredson D, Kellam S, Moscicki EK, Schinke S, Valentine JC & Ji P (2005) Standards of Evidence: Criteria for Efficacy, Effectiveness and Dissemination. Prevention Science, 6, 151–175. doi: 10.1007/s11121-005-5553-y [DOI] [PubMed] [Google Scholar]
  18. Fixsen D, Naoom S, Blase K, Friedman R, Wallace F (2005). Implementation Research: A Synthesis of the Literature Tamps, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network. [Google Scholar]
  19. Fowler KA, Dahlberg LL, Haileyesus T, & Annest JL (2015). Firearm injuries in the United States. Preventive Medicine, 79, 5–14. doi: 10.1016/j.ypmed.2015.06.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Greenspan AI, & Kellermann AL (2002). Physical and psychological outcomes 8 months after serious gunshot injury. The Journal of Trauma, 53, 709–716. doi: 10.1097/01.TA.0000022350.03761.F8 [DOI] [PubMed] [Google Scholar]
  21. Herman D, Conover S, Gorroochurn P, Hinterland K, Hoepner L, & Susser E (2011). Randomized trial of Critical Time Intervention to prevent homelessness after hospital discharge. Psychiatric Services, 62, 713–719. doi: 10.1176/appi.ps.62.7.713 [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Hwang SW, & Burns T (2014). Health interventions for people who are homeless. Lancet (London, England), 384, 1541–1547. doi: 10.1016/S0140-6736(14)61133-8 [DOI] [PubMed] [Google Scholar]
  23. Joseph B, Hanna K, Callcut RA, Coleman JJ, Sakran JV, & Neumayer LA (2019). The Hidden Burden of Mental Health Outcomes Following Firearm-related Injures. Annals of Surgery, 270, 593–601. doi: 10.1097/SLA.0000000000003473 [DOI] [PubMed] [Google Scholar]
  24. Lako DAM, de Vet R, Beijersbergen MD, Herman DB, van Hemert AM, & Wolf JRLM (2013). The effectiveness of critical time intervention for abused women and homeless people leaving Dutch shelters: study protocol of two randomised controlled trials. BMC Public Health, 13, 555. doi: 10.1186/1471-2458-13-555 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Mahalic S (2004). The importance of implementation fidelity. Emotional and Behavioral Disorders in Youth, 4, 83–105. [Google Scholar]
  26. McLean RM, Harris P, Cullen J, Maier RV, Yasuda KE, Schwartz BJ, Benjamin GC (2019) Firearm-related injury and death in the United States: A call to action from the Nation’s Leading Physician and Public Health Professional Organizations. Annals of Internal Medicine. 171, 573–577. doi: 10.7326/M19-2441 [DOI] [PubMed] [Google Scholar]
  27. Naleppa MJ, & Cagle JG (2010). Treatment Fidelity in Social Work Intervention Research: A Review of Published Studies. Research on Social Work Practice, 20, 674–681. doi: 10.1177/1049731509352088 [DOI] [Google Scholar]
  28. National Association of Social Workers. (2017). NASW code of ethics. Retrieved Jan 4, 2020, from https://www.socialworkers.org/About/Ethics/Code-of-Ethics/Code-of-Ethics-English
  29. Prochaska JO, Velicer WF, Rossi JS, Goldstein MG, Marcus BH, Rakowski W, Fiore C, Harlow LL, Redding CA, & Rosenbloom D (1994). Stages of change and decisional balance for 12 problem behaviors. Health Psychology, 13, 39–46. doi: 10.1037/0278-6133.13.1.39 [DOI] [PubMed] [Google Scholar]
  30. Rollnick S, Miller WR, & Butler C (2008). Motivational interviewing in health care: helping patients change behavior. Guilford Press. [Google Scholar]
  31. Rowhani-Rahbar A, Zatzick D, Wang J, Mills BM, Simonetti JA, Fan MD, & Rivara FP (2015). Firearm-related hospitalization and risk for subsequent violent injury, death, or crime perpetration: a cohort study. Annals of Internal Medicine, 162, 492–500. doi: 10.7326/M14-2362 [DOI] [PubMed] [Google Scholar]
  32. Shaw J, Conover S, Herman D, & Al E (2017). Chapter 2, Quantitative methodology In Critical time intervention for severely mentally ill prisoners (CrISP): a randomized controlled trial. NIHR Journals Library; Avaliable from: https://www.ncbi.nlm.nih.gov/books/NBK424465/#s2-8 [PubMed] [Google Scholar]
  33. Shinn M, Samuels J, Fischer SN, Thompkins A, & Fowler PJ (2015). Longitudinal Impact of a Family Critical Time Intervention on Children in High-Risk Families Experiencing Homelessness: A Randomized Trial. American Journal of Community Psychology, 56, 205–216. doi: 10.1007/s10464-015-9742-y [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Smith RN, Seamon MJ, Kumar V, Robinson A, Shults J, Reilly PM, & Richmond TS (2018). Lasting impression of violence: Retained bullets and depressive symptoms. Injury, 49, 135–140. doi: 10.1016/j.injury.2017.08.057 [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. StataCorp. (2015). Stata Statistical Software: Release 14. College Station, TX: StataCorp LP. [Google Scholar]
  36. Tucker AR, & Blythe B (2008). Attention to treatment fidelity in social work outcomes: A review of the literature from the 1990s. Social Work Research, 32, 185–190. doi: 10.1093/swr/32.3.185 [DOI] [Google Scholar]
  37. Vella MA, Warshauer A, Tortorello G, Fernandez-Moure J, Giacolone J, Chen B, Cabulong A, Chreiman K, Sims C, Schwab CW, Reilly PM, Lane-Fall M, & Seamon MJ (2019). Long-term Functional, Psychological, Emotional, and Social Outcomes in Survivors of Firearm Injuries. JAMA Surgery. 155, 51–59. doi: 10.1001/jamasurg.2019.4533 [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Zun LS, Downey LV, & Rosen J (2006). The effectiveness of an ED-based violence prevention program. American Journal of Emergency Medicine, 24, 8–13. doi: 10.1016/j.ajem.2005.05.009 [DOI] [PubMed] [Google Scholar]

RESOURCES