Content assessment and readability of online injury prevention resources on firearm safety: a cross-sectional analysis of hospital and national injury prevention literature

Abstract

Background

Firearm-related injuries are a preventable public health epidemic and the leading cause of pediatric death in America. Online injury prevention resources (OIPRs) offer potential for educating the public on firearm safety. National public health organizations recommend a sixth-grade reading level for these resources. We hypothesize that OIPRs for firearm safety may not meet this standard and are inconsistent in content.

Methods

We analyzed firearm injury OIPRs from three sources: verified trauma centers (TCs), national health organizations, and gun violence prevention advocacy groups. We assessed readability using reading time, Flesch-Kincaid grade level, and Flesch reading ease. We also assessed whether OIPRs included child safety, safe handling, and safe storage of firearms.

Results

Among 587 TCs, 105 had publicly accessible OIPRs. After removing duplicates, we analyzed 53 unique hospital OIPRs, 25 from national organizations, and 8 from advocacy groups. The mean reading time of hospital-based OIPRs was 2 min and 49 s, and 5 min and 30 s for advocacy organizations. The average Flesch-Kincaid Grade Level for hospital OIPRs was 8.2, national organizations 8.4, and advocacy groups 9.7. Only 21% of hospital and 22% of national OIPRs met the sixth-grade level; none of the advocacy groups met this standard. 79% of hospital-based OIPRs content related to child safety, compared with 44% of national organizations and none of the advocacy groups. Only 21% of TCs and no advocacy groups provided information on safe handling practices.

Conclusion

Few OIPRs meet recommended readability guidelines and often fail to address key topics such as child safety or safe handling of firearms. This gap in accessible educational information highlights the need for standardized resources to reduce firearm injury. Future research should aim to improve these resources to ensure usability and effective outreach to our communities

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Online injury prevention resources (OIPRs) have the potential to educate the public on firearm safety, but their readability and content often fail to meet recommended standards.

WHAT THIS STUDY ADDS

• This study reveals that most OIPRs exceed the sixth-grade reading level and inconsistently address crucial topics like child safety and safe handling practices.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• These findings highlight the need for standardized, accessible OIPRs, prompting healthcare institutions and policymakers to improve educational materials to better prevent firearm injuries.

Introduction

Firearm-related injuries are a preventable public health epidemic, claiming nearly 49,000 American lives in 2022, the majority from suicide.¹ They are the leading cause of death in children and a newly designated public health crisis by the US Surgeon General.^{1 2} Although evidence-based strategies exist for primary prevention of firearm injury, they are underused.^{3 4} Two-thirds of unintentional child fatalities involving firearms could be prevented with improved storage practices, and only 46% of firearm owners follow recommendations for safe storage.^{5 6} Physicians have been identified as key counselors by professional societies, but this role has not been widely adopted in clinical practice. Currently 62.3% of trauma surgeons report not counseling their patients on safe firearm storage, and 61% of primary care providers rarely or never address firearm safety in health maintenance visits.7,9 This gap in clinical practice likely propels patients towards online resources for guidance on injury prevention.¹⁰

Online injury prevention resources (OIPRs) may offer valuable information on reducing injuries from a range of preventable causes, including firearms. Online health education can improve health knowledge and attitudes, particularly in communities facing social barriers.^{11 12} OIPRs have shown potential in improving attitudes toward and preventing suicide, a leading cause of firearm deaths, by raising awareness and providing education.13,15 Accessible firearm OIPRs materials do have the potential to positively impact health behaviors and prevent unintentional injuries and suicides.¹⁶ However, if OIPRs are not developed intentionally to overcome health literacy challenges, their utility may be limited.

The American Medical Association recommends that all public health materials ought to meet a maximum sixth-grade reading level for accessibility to the general public.¹⁷ Failures to meet this standard have been identified in patient education materials across multiple fields, including trauma and injury prevention.^{18 19} Currently, the readability of OIPRs for firearm injury prevention remains uncertain, and we hypothesize that a significant number of these resources may fail to meet the recommended standard.²⁰ There are also no existing standardizations for the content of OIPRs. We therefore seek to characterize the content of these resources around three key topics: child safety, safe storage, and safe handling of firearms. These three topics were chosen because they are each instrumental driving forces of the public health crisis of firearm violence.²¹ In so doing, we seek to understand how accessible and useful OIPRs are to impacted communities, aiming to identify gaps which can ultimately be remedied to reduce firearm violence.

Methods

Data acquisition and inclusion criteria

To assess the accessibility of OIPRs, we conducted an analysis of materials available through three main sources: trauma centers (TCs) verified by the American College of Surgeons (ACS); national health organizations including government public health information and non-governmental organizations (NGOs) such as non-profit or advocacy groups aimed at reducing gun violence. Our focus on content from these groups aimed to reflect the avenues through which the public is most likely to access evidence-based injury prevention information.

For TCs, we referenced the ACS’ current directory to identify all verified adult and pediatric TCs.⁵ A standardized Google search was performed for each center by inputting the full hospital name combined with the term “firearm safety” to locate relevant online resources directly associated with these institutions. Only the first page of search results was considered, based on evidence indicating a significant decrease in user engagement beyond this point.^{22 23} To avoid search bias of cookies and user account information, we navigated websites on incognito mode and disabled search location. TCs were included if they provided any firearm safety information on their websites in the form of instructions or guidance. Exclusions were applied to hospitals that referred users to external sources for injury prevention information, as well as those that described active research or committee activity on gun violence without specific resources aimed at their patient population. Duplicates of information from TCs within the same hospital systems that share online resources were identified and excluded from the readability analysis but were considered in the descriptive analysis to portray overall accessibility of information. This approach follows previously published methodology for readability assessments of online health information.¹⁸

For national organizations, we examined the first 50 Google search results for ‘firearm safety resources’, ensuring search conditions were identical to those used for TCs to prevent bias. Resources were aggregated, including information that was deemed reputable based on their origin from state or federal departments, or non-profits focused on gun violence reduction. Exclusions were applied to non-authoritative sources such as news articles, blogs, magazines, and social media content. Similar conditions were used for advocacy organizations, with the search term ‘gun violence prevention advocacy organization’. If a search yielded the home pages of organizations overall, a second level of search was used within each organization’s website with the terms “firearm safety”, “gun safety”, and “safe storage”. Though this may reflect lower utility to patients seeking injury prevention information, we included information from these organizations as they are authoritative bodies that patients may still turn to for accurate injury prevention information online.

Content and readability assessments of online injury prevention resources

We extracted and converted the content of online resources into plain text for analysis. This process involved removing non-text elements (advertisements, copyright notices, hyperlinks, disclaimers, etc) as well as converting numerals and symbols to text form to accurately reflect the materials’ reading demands.²⁴ The content was appraised by identifying the intended audience and evaluating whether three key topics were covered in each resource, whereas readability was assessed using established readability metrics.

For the content appraisal of OIPRs, we analyzed whether each resource covered the three key topic areas of child safety, safe storage, and safe handling of firearms. Information around child safety included ways to speak to children about firearm safety, guidelines for protecting children from firearms both in their own homes and the homes of others, and ensuring firearms are not accessible to children. Information about safe storage included methods for locking away firearms, and guidance regarding storing firearms separately from ammunition. For safe handling, resources needed to mention one of the four general principles of safe use to be included: keeping a gun locked, unloaded, storing ammunition locked, and in a separate location.²⁵ We performed a general count of how many resources in each category (TC, national organization, and NGO) contained content on these topics. If the resource provided any recommendations around a topic we deemed the content present and counted that entity as positive. We then report the total findings as absolutes (n) as well as percentages. We did not perform a qualitative inductive thematic analysis.

We examined the reading time of cleaned text based on common estimates of a 200 words-per-minute reading speed.^{26 27} Given the lack of consensus on a singular best readability metric, we opted for the most widely used and validated tool available, the Flesch-Kincaid (FK) model, aligning our methodology with established best practices in readability research.²⁸ This model assesses readability based on sentence length and syllable count, gauging materials’ overall complexity in terms of US education grade levels.^{29 30} To analyze the appropriateness of the reading level for the general public, we adopted the American Medical Association’s recommendation, using a sixth-grade reading level as the threshold in the FK Grade Level.¹⁷ The Flesch Reading Ease score, which also uses sentence length and syllable count, rates texts on a 0–100 scale. For this reading ease score, higher scores indicate easier readability. Texts scoring above 80 are considered suitable for a below sixth grade reading level, our benchmark for this study. Scores below 30 suggest college graduate-level complexity. ^{28 31}

Statistical assessment

Our analysis was performed using R software (V.4.3.1), using the koRpus package for assessing grade-level readability.³² For each group, we calculated and presented mean scores, SD, and p values, ultimately using analysis of variance testing to assess the significance of overall variability between the three groups, with post-hoc Tukey HSD (Honestly Significant Difference) testing to further analyze the difference between each head-to-head pairing of groups (TC compared with national organizations; TC compared with advocacy organizations; national organizations compared with advocacy organizations).

Results

Resource categories and characteristics

We identified a total of 587 validated TCs as of March 13, 2024. Of these, 105 offered firearm injury prevention resources. These resources varied across TC verification levels. Level 1 TCs were more likely to provide resources for both adult (18% availability) and pediatric populations (49% availability) compared with Level 2 (7%, adults; 18%, pediatrics) and Level 3 TCs (16%, adults). In total, 53 unique OIPRs were isolated for analysis. (table 1)

Table 1. Availability of OIPRs via US trauma centers.

	Adults		Pediatric
	Did not provide OIPR	Provided OIPR	Did not provide OIPR	Provided OIPR
Level 1 trauma center	222	36	71	33
Level 2 trauma center	216	16	63	11
Level 3 trauma center	129	20	N/A	N/A

OIPRs, online injury prevention resources.

Among national organization resources, we identified 25 distinct OIPRs from a range of state and national health organizations, as well as think tanks. We found nine organizations that are working in the firearm space, encompassing both national associations and non-profits. One federal resource dedicated to firearm injury prevention was identified. Finally, we identified eight advocacy organizations with information on firearm injury prevention and safe use.

We studied a total of 85 OIPRs. After the removal of duplicates, our search yielded 53 unique hospital-based OIPRs, 25 unique national OIPRs, and 8 unique advocacy group OIPRs. The process identified several duplications between TC resources, primarily stemming from a common external provider that supplies firearm safety information to multiple hospitals and hospital groups. No entirely duplicated content was observed among the national or advocacy OIPRs, although many of the advocacy organizations appeared to pull from similar research talking points regarding safe storage of firearms.

In evaluating the intended audience of the OIPRs, 42 out of the 53 hospital-based OIPRs (79%) included specific injury prevention information about firearm safety around children, whereas 11 of the 25 national organization OIPRs (44%) included this information. None of the eight advocacy organizations OIPRs contained information specific to the topic of safety around children. All TCs and advocacy organizations discussed safe storage in their materials, and 22 of 25 national organizations (88%) discussed this component of firearm injury prevention. Conversely, safe handling of firearms was discussed in a much smaller percentage of OIPRs across groups: 11 of 53 TCs (21%) mentioned pointers for safe handling; 14 of 25 national organizations (56%); and 0 of 8 advocacy groups (0%).

Quantitative findings revealed that the mean reading time of hospital-based OIPRs was 2 min 49 s (SD=2 min 24 s). This was similar to the mean reading time of national OIPRs of 3 min 26 s (SD=3 min 47 s), whereas advocacy organization reading time was much longer, with a mean of 5 min 30 s (SD=3 min 10 s; p=0.057) (figure 1a). The average FK Grade Level for hospital OIPRs was 8.2 (SD=2.7); for national OIPRs was 8.4 (SD=2.3); and for advocacy groups was 9.7 (SD=1.3; p=0.31) (figure 1b). 11 of 53 TC-based OIPRs (21%) and 5 of 23 national OIPRs (22%) met the recommended sixth grade reading level of health literacy. None of the advocacy organizations met this requirement.

Figure 1. Readability Rtatistics, by Resource Type. a: Mean Reading Time of OIPRs by Group, b: Average Flesch Kincaid Ease of OIPRs by Group, c: Average FK Reading Ease of OIPRs by Group.

Flesch Reading Ease, where higher scores indicate easier readability, was the only metric in which the groups were significantly different: hospital OIPRs had a mean score of 63 (SD=13) compared with national OIPRs with a mean of 56 (SD=14), and advocacy groups with a mean of 54 (SD=6; p=0.04) (figure 1c). All of these fell around a ninth grade reading level. Tukey post hoc testing showed that the only statistically significant head-to-head difference between all groups, in all categories (reading time, FK grade level, and FK reading ease) was between the reading time of hospitals and advocacy groups (p=0.047). (table 2). To illustrate these differences in readability, online supplemental material provides examples of firearm safety resources with both high and low readability scores.

Table 2. Average readability metrics of OIPRs by group type.

Reading time	Mean	SD	N
Hospital	2 min 49 s	2 min 24 s	53
National	3 min 26 s	3 min 47 s	25
Advocacy	5 min 30 s	3 min 10 s	8
FK
Hospital	8.2	2.7	53
National	8.4	2.4	25
Advocacy	9.6	1.3	8
Ease
Hospital	63.1	12.9	53
National	56.4	14.1	25
Advocacy	54.2	6.2	8

FK, Flesch-Kincaid; OIPRs, online injury prevention resources.

Discussion

Our study suggests that OIPRs for firearm safety from organizations represented are generally written at a higher grade level than recommended for health education literature, and show variable content. Only 21% of hospital-based OIPRs, 22% from national organizations, and none from advocacy groups met the nationally recommended sixth grade reading level for patient education materials. Additionally, only 44% of national organizations and no advocacy groups provided resources tailored for safety around pediatric populations, despite the fact that firearm injury is the leading cause of pediatric death in the USA.⁴ Although the majority of resources discussed safe storage practices, only 21% of TCs and no advocacy groups provided any information on safe handling of firearms.

TCs and providers have an opportunity to provide community-oriented educational materials. Injury prevention programs are required by the ACS for verification as a Level I TC, and are strongly recommended for Levels II–IV.³³ Community education is a key part of injury prevention programming, and OIPRs likely serve as one of the central components of this educational missive. However, publicly available content for firearm injury prevention was found in only 18% of TCs. The content of OIPRs is also not standardized, varying in readability and discussions of pediatric prevention and safe use. As the initial point of engagement for firearm injury victims, TCs have a unique opportunity to provide prevention in violence-affected communities.³⁴ Failure to meet suggested standards of readability and lack of knowledge around the most important content to include likely decreases the efficacy of these educational programs for community injury prevention. Committed initiatives taken up by journals such as Trauma Surgery and Acute Care Open to increase public education represent a step in the right direction.³⁵

The lack of accessibility and utility of OIPRs is concerning more broadly, given the lack of other centralized resources for such counseling. In schools, firearm safety education often focuses on active shooter trainings, despite no empirical evidence of their effectiveness, rather than safe storage and injury prevention.³⁶ Similarly, firearm manufacturers do not have a standardized or legally enforced approach to educating consumers on safe handling or safe storage. Healthcare providers fail to consistently provide injury prevention counseling, for reasons ranging from perceived lack of time to fears concerning damaging patient relationships and the legality of this counseling.37,39 Additionally, the consequences of traumatic injury are multigenerational; children with injured parents demonstrate 130% higher odds of themselves experiencing injuries, creating cycles of trauma that comprehensive prevention could interrupt.⁴⁰ Simplifying and optimizing online resources to meet recommended reading levels is thus a crucial facet of patient education, in lieu of standardized and consistent counseling from other sources.

Our analysis also demonstrates that OIPRs do not sufficiently address the prevention of pediatric injury or safe handling practices. Currently no advocacy groups discuss either of these topics in their online materials, and safe handling and safe use principles are omitted from nearly 80% of hospital-based resources. Although there is currently no consensus on the optimal content of OIPRs, studies have shown that the presence of an unlocked firearm is associated with higher risks of suicide and unintentional firearm injury among children and adolescents, and that 67% of unintentional firearm deaths among children and adolescents (ages 0–17) occurred and children were playing with or showing the firearm to others when it discharged.^{6 25} Meanwhile, the weapons with which children commit suicide also vary based on season and location.⁴¹ All of these findings point to a clear need for more targeted counseling aimed at protecting children from the lethality of firearms, securing weapons in the home, and teaching safe handling practices to firearm owners. Our study is the first to highlight this gap, and, as a result, establish the need for the development of a standardized approach to the content of OIPRs, expanding on the recently published guidance by the US Surgeon General’s office on firearm violence in America, which suggests the need for more research on optimal injury prevention strategies.²

This study includes some clear limitations which merit further review. These include, first and foremost, the exclusion of video-based and image-based resources which may supplement text-based resources outside an individual’s literacy level. Our study focused on healthcare and public health sources of firearm safety information, but did not comprehensively assess commercial sources such as those from gun manufacturers or distributors. Our study is further limited by our approach to content assessment, where OIPRs were counted as meeting the safe handling criterion if they mentioned any one of the four established principles. This method may overestimate the proportion of resources that comprehensively address safe handling practices. The distinction between resources with minimal versus comprehensive coverage remains unexplored and represents an opportunity for more detailed content analysis in future studies. Moreover, the applicability of our study may be limited by the fact that the majority (83%) of health information seekers online use general search engines to find information, and only 15% turn to specific health websites, although we did study the resources at the top of our search strategy using common search engines.¹⁰ Finally, no sufficient behavioral analyses exist around the efficacy of preventative counseling and written injury prevention resources in effecting behavioral change around firearm use, even controlling for the readability of the resources’ contents. In other words, more evidence is needed to demonstrate that even when preventive counseling and written resources are thoroughly comprehensible, they effectively engender behavioral change around the use of firearms. However, our rigorous assessment of the readability of OIPRs is the first assessment of its kind and an obvious first step in ensuring that those who may be motivated and open to changing their behaviors around firearm use are adequately equipped with the resources to do so.

Conclusion

This study shows that most OIPRs regarding firearm injury prevention do not meet recommended readability guidelines and that a significant proportion do not adequately address safety for pediatric injury prevention, nor safe handling of firearms. Taken along with the knowledge that the communities most vulnerable to gun violence also have the least access to primary care and preventive counseling, as well as lower health literacy on average, this failure in the accessibility and utility of online resources represents a clear gap in firearm injury prevention strategy in this country that may be directly addressed and corrected. Future research should identify the utility of video and image-based resources, resources provided by the firearm industry to purchasers of firearms, as well as behavioral changes associated with text-based injury prevention resources as compared with oral counseling by healthcare providers.

Data availability statement

Data are available upon request. All data were publicly sourced from the internet.