Equitable Implementation of S.A.F.E. Firearm: A Multi-Method Pilot Study

Abstract

Attention to health equity is critical in the implementation of firearm safety efforts. We present our operationalization of equity-oriented recommendations in preparation for launch of a hybrid effectiveness-implementation trial focused on firearm safety promotion in pediatric primary care as a universal suicide prevention strategy. In Step 1 of our process, pre-trial engagement with stakeholders and literature review alerted us that delivery of a firearm safety program may vary by patients’ medical complexity, race, and ethnicity. In Step 2, we selected the Health Equity Implementation Framework to inform our understanding of contextual determinants (i.e., barriers and facilitators). In Step 3, we leveraged an implementation pilot across 5 pediatric primary care clinics in 2 health system sites to study signals of inequities. Eligible well-child visits for 694 patients and 47 clinicians were included. Our results suggested that medical complexity was not associated with program delivery. We did see potential signals of inequities by race and ethnicity but must interpret with caution. Though we did not initially plan to examine differences by sex, we discovered that clinicians may be more likely to deliver the program to parents of males than females. Seven qualitative interviews with clinicians provided additional context. In Step 4, we interrogated equity considerations (e.g., why and how do these inequities exist). In Step 5, we will develop a plan to monitor and mitigate potential inequities related to race, ethnicity, and sex in the fully powered trial. Our process highlights that prospective, rigorous, exploratory work is vital for equity-informed implementation trials.

1. Introduction

Attention to health equity in implementation ensures that evidence-based interventions reach and benefit all individuals.^1-4 Scrutinizing unjust and avoidable differences in implementation is imperative for firearm violence prevention efforts given steeply rising rates of firearm suicide over the past decade in American Indian/Alaska Native, Asian/Pacific Islander, Hispanic/Latino, and Black/African American youth.^5,6 Through descriptive pilot work, we examined potential inequities prior to launch of a multi-health system hybrid effectiveness-implementation trial focused on firearm safety promotion in pediatric primary care. Our team operationalized equity-oriented recommendations for implementation science^2-4,7-10 and pragmatic trials^11,12 through a 5-step process: (1) engage with stakeholders and critically review the literature, (2) select conceptual framework(s) to inform our understanding of contextual determinants (i.e., barriers and facilitators) of implementation, (3) leverage a pilot to study signals of inequities, (4) interrogate equity considerations (e.g., why and how do these inequities exist), and (5) develop a plan to monitor and mitigate inequities during the trial (Figure 1). Though health outcomes and opportunities are stratified across many characteristics, we focused on medical complexity, race and ethnicity, and sex assigned at birth in this analysis.

Figure 1.

Five-step process of adding an equity lens to planning for an implementation trial.

2. Background

2.1. ASPIRE trial

Adolescent and child Suicide Prevention in Routine clinical Encounters (ASPIRE) (R01MH123491-01; PI Beidas) is a hybrid effectiveness-implementation trial that tests the most effective way to implement S.A.F.E. (Suicide and Accident prevention through Family Education) Firearm, an adapted evidence-based safe firearm storage program, as a universal suicide prevention strategy in pediatric primary care.^13-15 S.A.F.E. Firearm includes a brief discussion on secure firearm storage and distribution of free cable locks to all parents or guardians (hereafter referred to as parents) during well-child visits (WCVs) for children aged 5-17.¹⁴ The two implementation strategies tested in the trial include (1) an electronic health record (EHR)-nudge (i.e., prompt) incorporated into standard WCV templates (Nudge) and (2) the EHR-nudge plus practice facilitation (Nudge+). Facilitation is a well-established approach that involves partnering with clinicians to overcome site-specific barriers through a number of methods (e.g., audit and feedback, problem-solving) with the goal of building organizational capacity for improvement.^16-18 In the full trial, thirty clinics across Kaiser Permanente Colorado and Henry Ford Health (Michigan) are randomized to Nudge or Nudge+.

2.2. Presenting problem and response

During pre-trial engagement work, clinician stakeholders shared that S.A.F.E. Firearm delivery may vary by patients’ medical complexity. Children with complex chronic disease have significant chronic conditions in two or more body systems.¹⁹ Stakeholders indicated that multiple diagnoses can compete for attention in time-constrained WCVs, crowding out preventative counseling.²⁰ Literature suggests disparities in the receipt of screening and preventative counseling among youth identifying as Black and Hispanic.^21-23 Moreover, racially and ethnically minoritized children are disproportionately affected by illness and disability in the U.S.^24,25 Systemic racism, comprised of both structural and individual-level racism, drives inequitable outcomes. Structural racism is encapsulated in the cultural norms, laws, policies, and practices that design health care systems for the majority, while individual-level racism includes interpersonal forms of racism, like implicit bias.^26-28 Cumulatively, these inputs suggested the possibility of variation in program delivery by patients’ race and ethnicity (Step 1).

Our goal is that all children have an opportunity to receive and benefit from S.A.F.E. Firearm. Potential health gains are predicated on program receipt. Key ethical considerations arise if implementation exacerbates health disparities.²⁹ While our team systematically adapted S.A.F.E. Firearm in partnership with diverse constituencies and developed implementation strategies to support uniform delivery,^14,30-32 the trial is inherently situated amidst broader social forces that influence implementation determinants and processes.^33,34 Stakeholder feedback presented an opportunity to carry out a “real world,” a priori evaluation of signals of inequities in program delivery prior to trial launch.

We leveraged a pilot in both health systems to examine whether S.A.F.E. Firearm delivery differed by patients’ medical complexity and race and ethnicity. We hypothesized that clinicians would have higher odds of delivering S.A.F.E. Firearm to parents of youth classified as medically non-complex compared to parents of youth classified as medically complex. We also hypothesized that clinicians would have higher odds of delivering S.A.F.E. Firearm to parents of youth in non-Hispanic/Latino (NH) White groups compared to parents of youth from NH-Black/African American and Hispanic/Latino groups. We conceptualized race and ethnicity variables as proxies for “social stratification achieved through systemic racism.”^{26(p. 2)} Additionally, we interviewed a subset of clinicians participating in the pilot to better understand the local context, as well as factors that enabled or hindered equitable implementation. We anticipated that findings from the quantitative and qualitative data could inform potential adaptations to the Nudge+ arm of the trial to optimize equitable implementation and clinical outcomes.

3. Methods

3.1. Conceptual framework

We used the Health Equity Implementation Framework, a multilevel framework that assesses equity-focused determinants that may promote or hinder implementation, to guide our design and analysis (Step 2).³³ Elements include recipients (i.e., patient and provider factors), characteristics of the innovation, clinical encounter, inner context, and societal influence.

3.2. Ethics

The pilot study was approved by the University of Pennsylvania IRB, which served as the single IRB. Informed consent was obtained prior to interview participation.

3.3. Setting

Site 1 is an integrated, nonprofit health care system that covers a large metropolitan area in a midwestern U.S. state and serves over 1.25 million patients. Site 2 is an integrated health care system in a western state with a salary-based physician model that serves 525,000 patients. This study took place in 5 pediatric and family medicine clinics across both health systems (Site 1: n=3, Site 2: n=2). The site investigators and local clinic champions selected clinics with the goal of piloting implementation procedures (e.g., documentation, training). In response to stakeholder feedback, we harnessed this planned pilot to rigorously study equitable implementation.

3.4. S.A.F.E. Firearm intervention

S.A.F.E. Firearm consists of 2 components: (1) a brief discussion with parents led by primary care clinicians on secure firearm storage, and (2) free cable locks offered to all parents. The discussion is informed by motivational interviewing, emphasizing a nonjudgmental approach while supporting incremental changes toward more secure storage with the shared goal of preventing injury.^13,14

3.5. EHR-nudge implementation strategy

At both sites, an update to the WCV documentation template included a new “firearm safety” item with a list of choices to document secure firearm storage discussion and cable lock offers. To close the patient encounter in the EHR, clinicians needed to respond to the EHR-nudge choices (discussed secure firearm storage, offered a cable lock) or delete if preferred.

3.6. Participants

Eligible WCVs were from youth ages 5-17 seen by pediatric and family medicine clinicians who conducted pediatric primary care visits, had access to S.A.F.E. Firearm training materials, and had the EHR-nudge in their WCV template during the pilot period. Interview participants were sampled from the clinicians who engaged in the pilot.

3.7. Design and procedures

Pilot initiation activities included parent handout and cable lock delivery to each clinic, EHR-nudge activation, and clinician training (Step 3). At each health system, the research team led a 20-30 minute virtual training session with an overview of the trial, the S.A.F.E. Firearm program, and EHR documentation practices. Trainers also suggested that clinicians complete the American Academy of Pediatrics “Safer: Storing Firearms Prevents Harm” course.³⁵ Clinicians received the S.A.F.E. Firearm Clinician Guide, a resource with detailed guidance.

Site 1 data were collected over a 4-week period from October through December 2021. Due to a slower ramp up period, 3-months of data from November 2021 through early February 2022 were included from Site 2 clinics. Variables of interest were extracted from the EHR.

For pilot clinician interviews, clinician champions within each health system collaborated with the research team for recruitment in February 2022. Using a convenience sampling approach, champions were invited to participate in a brief interview and extended the invitation to pilot clinician colleagues. To facilitate recruitment, the research team also invited Site 2 pilot clinicians at a pediatric stakeholder meeting.

The semi-structured interview guide was informed by the Health Equity Implementation Framework (Appendix A). Two research team members conducted the audio-recorded interviews by telephone or video conference. Afterwards, interviewers debriefed and wrote field notes to capture key impressions. Audio files were professionally transcribed. Participants at Site 1 received $50 for participation. Due to health system policies, participants at Site 2 were ineligible for payment.

3.8. Variables

Clinician-documented program delivery, or reach, is the primary implementation outcome in the trial. This was defined as a composite outcome; clinicians needed to document “yes” to both program components (i.e., discussion and lock offer) to meet criteria for delivering the full S.A.F.E. Firearm program. Each program component, discussion and lock offer, was also analyzed separately as a dependent outcome. Based on discussion with stakeholders at each site and conventional documentation standards, we assumed no delivery when the EHR prompts were not answered or deleted.

Independent variables included patients’ medical complexity, race and ethnicity, and sex. Medical complexity was determined by the Pediatric Medical Complexity Algorithm (PMCA), version 3.0.³⁶ The PMCA uses International Classification of Diseases (ICD)-9 and ICD-10 codes to stratify each patient as non-chronic, non-complex chronic, and complex chronic.³⁶ We included ICD codes for the pilot period WCV and preceding encounters. We dichotomized the variable as (a) non-chronic or non-complex chronic versus (b) complex chronic.

Race and ethnicity were parent-reported and recorded as discrete categories in the EHR. Due to sample size, we collapsed the variables to the following 4 classifications: Hispanic/Latino, NH-Black/African American, NH-White, or NH-Other (which included Asian, American Indian/Alaska Native, Native Hawaiian or Other Pacific Islander, multiracial, and other as reported in the EHR).

Sex assigned at birth was recorded in the EHR and coded as male, female, other, or unknown. Both systems recently started collecting more representative gender identity variables (e.g., transgender, non-binary), but those data are very incomplete at this time and thus not included.

3.9. Quantitative Analyses

SAS 9.4 was used. To investigate associations between outcomes and covariates of interest, general estimating equations (GEE) models were created with clustering by clinician and an exchangeable correlation structure. The model included main effects of site, 4-level race and ethnicity, and medical complexity. During model development, our team suspected that patient sex might be an important covariate for program delivery, particularly given literature suggesting boys are more likely than girls to report having handled a gun in the home,³⁷ and male teenagers use a firearm 51% of the time when they die by suicide compared to 25% of females.³⁸ Thus, we controlled for sex by including it as a main effect in all GEE models. All 2-way interactions were tested between site, medical complexity, and sex in the main model; however, due to sample size limitations, we were unable to reliably include 2-way interactions with 4-level race and ethnicity. Instead, potential interactions including race and ethnicity were assessed in a separate model with a binary race and ethnicity variable (NH-White and Non-White). Since there was no evidence of dependence between covariates (i.e., no interactions were significant), the final model for all outcomes included only the fixed effects of site, race and ethnicity, medical complexity, and sex. Marginal probabilities were computed by averaging over the distribution of covariates. Proc GENMOD was used for all model constructions.

Of the 752 eligible WCVs, 2 (0.003%) were missing patient race and/or ethnicity, 2 (0.003%) were missing sex, and 2 (0.0003%) were missing ICD codes to derive the medical complexity variable. Race was indicated as “unknown” in 56 (7.4%) WCVs. We used a complete case analysis including only those WCVs with known race and ethnicity for a final sample size of 694 observations. All GEE models included 47 clinician clusters, and the intraclass correlation (ICC) was defined by the estimated exchangeable parameter.

3.10. Qualitative Analyses

Clinician interview data were analyzed using rapid qualitative methods^39,40 to gain a preliminary understanding of feedback in light of the short timeline between the pilot and trial launch.⁴¹ Two research team members created templated summaries with neutral domain names corresponding to each interview question, as well as illustrative quotations. They then reviewed the transcripts and condensed key responses within each domain. Summaries were then transferred into a data matrix in Excel. The neutral domain names comprised the rows, and participant IDs comprised the columns. The matrix enabled the identification of trends in responses across the participants.³⁹ Similar concepts were grouped and mapped to domains of the Health Equity Implementation Framework. Analytic memos were maintained to facilitate connections.

4. Results

4.1. Quantitative

4.1.1. Patient characteristics

The final sample was comprised of 694 WCV observations (Table 1). Approximately half of patients were female (49.7%), and 23.8% were medically complex. Patients from racially and ethnically diverse groups represented 51% of the sample.

Table 1.

Characteristics of youth included in the pilot dataset.

Demographic Characteristics	Health System Sites
	Site 1 (n=398)	Site 2 (n=296)	Combined (N = 694)
Sex, n (%)
Female	199 (50%)	146 (49.3%)	345 (49.7%)
Male	199 (50%)	150 (50.7%)	349 (50.3%)
Race and ethnicity, n (%)
Hispanic/Latino	11 (2.8%)	32 (10.8%)	43 (6.2%)
Non-Hispanic Black/African American	180 (45.2%)	10 (3.4%)	190 (27.4%)
Non-Hispanic White	132 (33.2%)	208 (70.3%)	340 (48.9%)
Non-Hispanic Other	75 (18.8%)	46 (15.5%)	121 (17.4%)
American Indian/Alaska Native	1 (0.3%)	0 (0%)	1 (0.1%)
Asian	41 (10.3%)	22 (7.4%)	63 (9.1%)
Multiracial	12 (3.0%)	13 (4.4%)	25 (3.6%)
Native Hawaiian/Other Pacific Islander	0 (0%)	2 (0.7%)	2 (0.3%)
Other	21 (5.3%)	9 (3.0%)	30 (4.3%)
Medical complexity, n (%)
Non-complex	320 (80.4%)	209 (70.6%)	529 (76.2%)
Complex	78 (19.6%)	87 (29.4%)	165 (23.8%)

4.1.2. Reach outcome

Reach (i.e., yes to both discussion and locks) was documented for 40.8% of patients overall, with 54.8% at Site 1 and 22.0% at Site 2. Clinicians documented reach for 36.5% of female and 45.0% of male patients, and 35.2% of medically complex and 42.5% of medically non-complex children. By race and ethnicity, 27.9% of Hispanic/Latino, 58.4% of NH-Black/African American, 36.2% of NH-White, and 30.6% of children in the NH-Other group had documented reach.

The odds of documented reach were 5.91 times higher at Site 1 than Site 2 (95% CI [2.14, 16.37], P < 0.001). Sex was significant; the odds of reach for females was 32.6% less than males (OR 0.67, 95% CI [0.47, 0.97], P = 0.03). Medical complexity was not significantly associated with reach (OR 1.24, 95% CI [0.77, 2.01], P = 0.38) (Table 2). Although the type III test for 4-level race and ethnicity was non-significant (P = 0.196), the odds of documented reach differed between NH-White and NH-Other patients (OR 1.72, 95% CI [1.02, 2.88], P = 0.04). The odds of documented reach by all other race and ethnicity comparisons (e.g., Hispanic/Latino and NH-Black/AA) were non-significantly different (Figure 2). Please see Figure 3 for estimated marginal probability comparisons.

Table 2.

Estimated odds of S.A.F.E. Firearm delivery by site, sex, and medical complexity.

Group 1	Group 2	Outcome	OR	95% CI
Site 1	Site 2	Reach	5.91*	2.14	16.37
		Discussion	5.27*	1.87	14.86
		Lock	5.86*	2.12	16.19
Non-complex	Complex	Reach	1.24	0.77	2.01
		Discussion	1.26	0.84	1.90
		Lock	1.14	0.74	1.77
Female	Male	Reach	0.67*	0.47	0.97
		Discussion	0.70	0.46	1.06
		Lock	0.72	0.51	1.01

^*Individual odds ratio significant at P < 0.05.

Odds ratios represent the rate of delivery for Group 1 relative to Group 2. Adjusted odds ratios for each outcome are estimated using a GEE model that includes covariates for site, medical complexity, race and ethnicity, and sex.

Figure 2. Comparison of adjusted odds of S.A.F.E. Firearm delivery between racial and ethnic groups.

Odds ratios represent the rate of delivery for Group 1 relative to Group 2. Adjusted odds ratios for each outcome are estimated using a GEE model that includes covariates for site, medical complexity, race and ethnicity, and sex. Type III tests for race and ethnicity were nonsignificant for reach, discussion, and lock models (p-values = 0.196, 0.180, and 0.205, respectively).

Non-Hispanic (NH) label was removed to promote clarity.

Figure 3. Estimated marginal probabilities for all three implementation outcomes by race and ethnicity.

Marginal probabilities for each outcome are estimated using a GEE model that includes covariates for site, medical complexity, race and ethnicity, and sex.

Non-Hispanic (NH) label was removed in the legend to promote clarity.

4.1.3. Discussion of secure firearm storage outcome

Overall, clinicians documented discussion of secure firearm storage for 55.8% of patients, with 68.3% at Site 1 and 38.9% at Site 2. Clinicians documented discussion for 60.2% of male and 51.3% of female patients. They documented discussion for 49.1% of medically complex and 57.8% of medically non-complex children. Discussion was documented for 44.2% of Hispanic/Latino, 69.0% of NH-Black/African American, 53.2% of NH-White children, and 46.3% of children in the NH-Other group.

Clinicians at Site 1 had 5.27 greater odds of documenting discussion compared to clinicians at Site 2 (95% CI [1.87, 14.86], P < 0.001). Neither sex (OR = 0.70, 95% CI [0.46, 1.06], P = 0.09) nor medical complexity (OR = 1.26, 95% CI [0.84, 1.90], P = 0.26) was significantly associated with discussion (Table 2). Although the type III test for 4-level race and ethnicity was non-significant (P = 0.18), the odds of clinicians documenting discussion was 1.72 times higher (95% CI [1.06, 2.80], P = 0.03) for patients from the NH-White group compared to patients from the NH-Other group. The odds of clinicians documenting discussion did not differ significantly between other racial and ethnic group comparisons (Figure 2).

4.1.4. Lock outcome

Overall, clinicians documented offering locks at 41.6% of WCVs, with 56.0% at Site 1 in contrast to 22.3% at Site 2. Clinicians documented lock offers for 45.3% of male and 38.0% of female patients. Locks were documented for 37.0% of medically complex and 43.1% of medically non-complex children. 27.9% of Hispanic/Latino, 60% of NH-Black/African American, 36.8% of NH-White children, and 31.4% of in the NH-Other group had documented lock offers.

There was a significant difference in the odds of documented lock offers by site. Clinicians at Site 1 had 5.86 greater odds of documenting lock offers compared to clinicians at Site 2 (95% CI [2.12, 16.19], P < 0.001). As with discussion, lock offers were not significantly associated with sex (OR = 0.72, 95% CI [0.51, 1.01], P = 0.06) or medical complexity (OR = 1.14, 95% CI [0.74, 1.77], P = 0.54) (Table 2). Although the type III test for 4-level race and ethnicity was non-significant (P = 0.21), the odds of clinicians documenting lock offers was 1.66 times higher (95% CI [1.01, 2.72], P = 0.045) for patients from the NH-White group compared to patients from the NH-Other group. The odds of clinicians documenting lock offers did not differ significantly between other racial and ethnic group comparisons (Figure 2).

4.2. Qualitative

4.2.1. Clinician characteristics

Seven clinicians participated in qualitative interviews (Site 1: n = 4, Site 2: n = 3). Six were physicians and 1 was a nurse practitioner. Four participants identified as female; 1 identified as Hispanic/Latino, 1 identified as Black/African American, and 5 identified as White.

4.2.2. Findings

Findings are presented across domains of the Health Equity Implementation Framework, with illustrative quotations noted in Table 3.

Table 3.

Determinants and Illustrative quotations from clinician interviews organized by Health Equity Implementation Framework domains.

Domain	Illustrative Quotations
Characteristics of the Innovation Explicit non-judgmental approach Relative advantage of universal delivery Free/no-cost/low barrier	“I take care of all kinds of people. So not to make them think that I'm the gun police or making any judgment. I'm not asking. I don't really want to know. I just want to know, ‘Do you want one [free cable firearm lock]? Because we're giving them out for free. And so it's free, so I'm not screening you.’” (Site 1, P42)
Recipients: Patient (and Parent) Factors Parent preferences Medical conditions	“I don't think that there was really much association between any of their underlying conditions and the conversation with the safe firearms, just because the way I had it in our template. We were discussing any of their underlying medical disorders, and then, when we talk about safety, it just came underneath that safety sort of category. So I don't think that there was much association in my mind." (Site 1, P41)
Recipients: Provider Factors Comfort and self-efficacy Communication skills	“I think initially, I had that fear that there was going to be-- the people were either going to get upset and-- in either direction, that there was going to be people who were going to feel like, "Well, why is he asking me about guns?" Like, "I have a gun, I'm going to be judged." Or like, "What kind of family do you think we are that we would have guns?" So initially, I had this preconceived idea that I didn't want to upset people or insult people on either side of the spectrum. But actually fairly quickly, now I feel like that is a fear that has gone to the side because I don't see that that's happened at all.” (Site 2, P111)
Clinical Encounter Competing demands Clinician reliance on the EHR Sensitive topic normalization Use of interpreter services	"But I think having that prompt is really important, and that's part of medicine, fortunately these days, because we can be a lot more consistent with our application of our care if we utilize technology to help us make sure that we're doing that." (Site 2, P40) I think there's been no difference in who we deliver that [firearm safety] message to. And I would say there's been no consistent differences in terms of reception across multiple different variations of parents and patients. And I think it's been pretty uniform. (Site 2, P112) “I'm asking everyone, regardless of any underlying race or ethnicity.” (Site 1, P40) “[…] I don't feel like I personally change the way that I deliver the information based on socioeconomic status, race, whatever. I still try to at least get that one-liner in there, more so because, I think, it's in my note and I need to click through it. So […] it's like, "I just have to address it." Whether I spend a lot of time on it and whether the patient wants to know more about it is different, and I can kind of extend or shorten the amount of counseling that I do. But I think just knowing that it's in the note, I try to just address it for everybody. So the prompting, I think, works.” (Site 1, P31) "And I think [families are] much more inclined to listen to that additional topic of conversation when it's included in that part of the guidance that we provide, as opposed to kind of incorporating it into discussion in a way that makes it feel, I think, hyper-focused or hypersensitive to families, because they may or may not feel singled out. So I think normalizing the discussion has been a really big help for a lot of our providers when it comes to tailoring their own discussions with families." (Site 2, P112)
Local & Organizational Level Lock location and accessibility Organizational culture	“[Equitable program delivery] really depends upon the culture of the company, because if equity and care delivery is important to the company and they emphasize that and they look at the disparities that exist, then I think that you're a lot more likely to be able to achieve closing that gap. If the company doesn't consider it a priority, then I think you're a lot less likely to see those impacts.” (Site 2, P40)
Social Influence Local high school shooting COVID-19 pandemic and mental health sequelae	"And so I live the next town over from [town]. So where we live, we were very impacted. And actually, our school was closed for a few days after the shooting […] We are literally like the next district over. It's probably ten minutes to [town] from my house. And so a lot of the mental health issues in regards to that have been, just in our local community, very difficult." (Site 1, P41)

Characteristics of the innovation.

Clinicians described the non-judgmental motivational interviewing-informed approach and framing of the program for all families as relative advantages of S.A.F.E. Firearm over usual practices. The fact that locks were free was a noted facilitator.

Recipients: patient (and parent) factors.

Clinicians reported that families presented with overall positive responses, though many parents volunteered they were not firearm owners. These disclosures led some clinicians to tailor discussions spontaneously to emphasize firearm safety beyond the primary household (e.g., friends, co-parents). The presence of multiple chronic conditions did not appear to impede program delivery, though reduced discussion time in a few scenarios. Clinicians described variation in socioeconomic status, political affiliation, and firearm ownership among the families served. They denied that families’ social needs (e.g., food insecurity, neighborhood violence) impacted delivery.

Recipients: provider factors.

Clinicians reported that among their colleagues, initial apprehension improved with training and practice.

Clinical encounter.

After an initial adjustment period, clinicians adapted to workflow changes. All participants emphasized the value of the EHR-nudge to promote consistent initiation of program delivery for all families. Location of the EHR-nudge in the safety section of the template facilitated routinization. Moreover, embedding firearm safety into broader safety counseling (e.g., car seats) supported normalization of the conversation, in contrast to what could be perceived as targeted delivery (i.e., parents feeling singled out due to household firearms or stereotyped behaviors). Several clinicians acknowledged topic sensitivity; they described taking cues from parents and gauging willingness to engage. Limited interest from parents abbreviated the discussion. Key factors that contributed to not delivering the program included running out of time, distractions, and forgetting. No clinicians observed differences in program delivery by patients’ race and ethnicity during the pilot period, and all endorsed the belief that delivery was uniform. Notably, Site 2 clinicians emphasized the value of interpreter services for Spanish-speaking families to ensure consistent, high-quality delivery.

Local and organizational level.

Clinicians noted the importance of the lock locations within each clinic; ready availability reduced friction. One clinician emphasized that company culture (i.e., the organization’s emphasis on equity, care delivery, and closing gaps) is important for equitable program delivery.

Social influence.

Participants from Site 1 identified that a school shooting took place during the pilot in their community. The shooting was followed by copycat threats and district-wide shutdowns. Additionally, clinicians at both sites described increases in mental health symptoms among their patients since the onset of the COVID-19 pandemic, with concerns amplified at Site 1 following this school shooting. Surges in mental health concerns increased the salience of suicide prevention efforts.

5. Discussion

Our process, further described below, is an exemplar of grounding an implementation trial with an equity lens prior to launch. We engaged with local stakeholders and examined the literature (Step 1), then leveraged a pilot to study signals of inequities in S.A.F.E. Firearm delivery (Step 2). Our results suggest that medical complexity was not associated with S.A.F.E. Firearm reach, discussions, or lock offers (Step 3). We found potential inequities in reach by race and ethnicity, specifically greater odds of documented reach, discussions, and lock offers for NH-White than NH-Other groups. This finding must be interpreted with caution because of a nonsignificant type III test of race and ethnicity. We also discovered that clinicians may be more likely to deliver the composite program to parents of male than female patients. The reach model had the lowest ICC and thus the highest statistical power compared to the discussion and lock offer models. Differences in the statistical significance of sex across outcome models may be attributable to differences in ICC. Overall, the pilot afforded us the opportunity to identify potential differences in implementation, highlighting that prospective, rigorous work is vital for equity-informed implementation trials.

Explanatory power is limited in this preliminary examination⁴² (Step 4). Findings of higher odds of documented program delivery to parents of NH-White youth compared to parents of youth from NH-Other groups is important to consider clinically, particularly given the heterogeneity of the NH-Other group and surges in firearm suicide among American Indian/Alaska Native and Asian youth.^5,6 Notably, Hispanic/Latino patients had lower observed proportions of each outcome than patients in the NH-Other group, but the difference was not statistically significant in our model. The difference between observed and modeled probabilities may be because the observed outcomes do not account for clustering (i.e., behavior of a few clinicians within a particular clinic could drive findings) and the distribution of race and ethnicity across sites. It is also possible that the sample size was insufficient for the model to detect a small but clinically meaningful effect, highlighting the need to not dismiss null results given the consequence of Type II errors.¹¹ Questions remain about the extent to which factors like spoken language, recent immigration, or social capital might illuminate inequities impacting clinical care.⁴³ Our brief qualitative interviews highlighted facilitators of program delivery (e.g., relative advantage) but did not uncover embedded power structures that may influence program receipt in the clinical encounter. Future work should examine complex interdependencies between social categories and systems of care, as well as the role of implicit bias.^11,28,44

The finding of differences in reach by sex assigned at birth also requires further probing. Clinicians’ knowledge, motivations, or assumptions (e.g., related to gendered cultural norms) may have contributed to this variation. Overall rates of firearm violence have risen during the COVID-19 pandemic, and firearms are now the leading cause of death among youth ages 1-19.⁴⁵ Between 2019 and 2020, mortality has increased among both male and female youth.⁴⁵ While overall rates of firearm injury are higher among males than females, program delivery should not be limited by population-level data.⁴⁶ It is critical that clinicians consistently provide guideline-concordant care so that girls are not marginalized in well-intentioned suicide prevention efforts.

Next steps will be to monitor for potential inequities during the active trial by descriptively assessing observed rates of delivery by race, ethnicity, and sex pooled across arms (Step 5). Our team is considering options for adapting facilitation, the implementation strategy being tested in the Nudge+ arm of the trial, to increase the likelihood of equitable outcomes. Facilitation is a process of interactive problem-solving that occurs in the context of a supportive interpersonal relationship.¹⁷ Used widely in primary care settings to enhance the implementation of new innovations, facilitation is primed for data-driven adaptation, with real-time responsiveness in the implementation approach.^17,33 Additionally, our team is initiating a qualitative sub-study focused on firearm suicide prevention in Black youth. These findings will inform further program and strategy refinement. We aim to weave attention to equity throughout the trial and engage in continuous learning.

Our study has limitations. First, due to sample size, we collapsed WCVs from Asian, American Indian/Alaska Native, Native Hawaiian or Other Pacific Islander, multiracial, and other groups into a NH-Other category for the race and ethnicity variable. We also were unable to include those with unrecorded race and ethnicity, a potentially unique subpopulation. The main trial analysis will account for greater heterogeneity in race given potentially racialized experiences of care receipt and the different historical and social contexts of these groups. We will strive to maintain granularity in future work. Second, as noted above it is possible that we were unable to detect small but meaningful effects given the power for each outcome. Third, race, ethnicity, sex, and ICD codes were derived from the EHR and are subject to misclassification. Patient gender was not available. Fourth, we did not have data on parents’ or pilot clinicians’ race, ethnicity, and gender, which may have relevance to the interaction within the clinical encounter.^47,48 Fifth, we were not able to confirm whether a parent was in attendance at each WCV to receive the program. Lastly, due to recruitment challenges, our qualitative sample was small and possibly influenced by selection bias. We did not reach informational redundancy⁴⁹ across all domains, but this tradeoff was acceptable given that our analytic goal was to glean a preliminary understanding of determinants. Due to the timeline between pilot and trial launch, we initiated interviews before the quantitative data analysis was complete and were unable to probe on differences in program delivery by sex due to the variable’s late addition to the models.

Overall, our findings emphasize the value of measuring implementation outcomes such as reach through an equity lens.^2,4 These key intermediate outcomes are fundamentally intertwined with clinical outcomes (i.e., effectiveness) and implementation success or failure.^4,50

6. Conclusion

Our 5-step process enhanced attention to health equity in our trial. We call on other investigators to innovate in this space, as implementation trials must focus on disaggregating implementation outcomes across disadvantaged subgroups in order to understand distributional effects.⁵¹ Beyond clinical research, our work highlights the potential for health systems to engage in data-driven monitoring for inequities⁵² to ensure that implementation of firearm violence prevention programs translates into meaningful impact for all families.

Highlights.

• We present a case exemplar for adding an equity lens to an implementation trial

• We leveraged a pilot to study signals of inequities prior to trial launch

• Rigorous exploratory work is vital for equity-informed implementation trials

Appendix A

Pilot Clinician Interview Guide

1. Tell me how it went delivering S.A.F.E. Firearm.

   1. How did patients’ medical conditions impact S.A.F.E. Firearm delivery?
      [If clarification is needed]: For example, concerns about multiple chronic conditions like asthma, diabetes, or depression?
   2. How did patients’ social needs impact S.A.F.E. Firearm delivery?
      [If clarification is needed]: For example, concerns about food insecurity, neighborhood violence, educational supports?
2. Can you describe a patient encounter in which you were unable to deliver the program?

   1. Can you walk me through details that precluded delivering the S.A.F.E. Firearm program?
   2. Can you describe the characteristics of the patient and family?
3. Are there other factors that influence your decision to deliver the S.A.F.E. Firearm program?

   1. Probe: Factors like topic sensitivity, concern for stigma, etc.?

4. Our initial findings from the pilot suggest no differences in program delivery by patients’ medical complexity. How does this resonate with you?

5. Our initial findings also suggest no differences in program delivery by patients’ race or ethnicity. How does this resonate with you?

6. Do you mind sharing more about local climate as it relates to delivery of firearm safety?

7. Is there anything else that you’d like to share to help us understand equitable program delivery?

8. If you are comfortable, would you be willing to answer a few questions about your demographic characteristics? These are entirely optional.

What is your current gender identity? You may select all that apply.

• Male

• Female

• Trans male/trans man

• Trans female/trans woman

• Genderqueer/gender non-conforming

• Different identity (please state): _____________________

• Prefer not to answer

What is your race? You may select all that apply.

• American Indian or Alaska Native

• Asian

• Native Hawaiian or Other Pacific Islander

• Black or African American

• White

• Other (please state): _______

• Prefer not to answer

Do you identify as Hispanic or Latino/a/x?

• Yes

• No

• Prefer not to answer

Thank you for participating in this interview today!